mirror of
https://github.com/sqlite/sqlite.git
synced 2026-10-23 21:49:02 +03:00
Updates to the testrunner documentation.
FossilOrigin-Name: bcac99f962c6403fddfcaeef309ea9f825fa661fee0b3e3fd24f04c861be1d1b
This commit is contained in:
drh
@@ -81,24 +81,21 @@ Both of the commands above accomplish about the same thing, but the second
|
||||
one has the advantage of not requiring "watch" to be installed on your
|
||||
system.
|
||||
|
||||
Sometimes testrunner.tcl uses the `testfixture` binary that it is run with
|
||||
to run tests (see "Binary Tests" below). Sometimes it builds testfixture and
|
||||
Sometimes testrunner.tcl uses the `testfixture` and `sqlite3` binaries that
|
||||
are in the directory from which testrunner.tcl is run.
|
||||
(see "Binary Tests" below). Sometimes it builds testfixture and
|
||||
other binaries in specific configurations to test (see "Source Tests").
|
||||
|
||||
<a name=binary_tests></a>
|
||||
# 2. Binary Tests
|
||||
|
||||
The commands described in this section all run various combinations of the Tcl
|
||||
test scripts using the `testfixture` binary used to run the testrunner.tcl
|
||||
script (i.e. they do not invoke the compiler to build new binaries, or the
|
||||
`make` command to run tests that are not Tcl scripts). The procedure to run
|
||||
these tests is therefore:
|
||||
|
||||
1. Build the "testfixture" (or "testfixture.exe" for windows) binary using
|
||||
whatever method seems convenient.
|
||||
|
||||
2. Test the binary built in step 1 by running testrunner.tcl with it,
|
||||
perhaps with various options.
|
||||
test scripts using whatever `testfixture` binary (and maybe also the `sqlite3`
|
||||
binary, depending on the test) that is found in the directory from which
|
||||
testrunner.tcl is launched. So typically, one must first run something
|
||||
like "`make testfixture sqlite3`" before launching binary tests. In other
|
||||
words, testrunner.tcl does not build the binaries under test itself; it
|
||||
expects them to be available already.
|
||||
|
||||
The following sub-sections describe the various options that can be
|
||||
passed to testrunner.tcl to test binary testfixture builds.
|
||||
@@ -140,22 +137,22 @@ are defined in file *testrunner_data.tcl*.
|
||||
To run the "veryquick" test set, use either of the following:
|
||||
|
||||
```
|
||||
./testfixture $TESTDIR/testrunner.tcl
|
||||
./testfixture $TESTDIR/testrunner.tcl veryquick
|
||||
tclsh $TESTDIR/testrunner.tcl
|
||||
tclsh $TESTDIR/testrunner.tcl veryquick
|
||||
```
|
||||
|
||||
To run the "full" test suite:
|
||||
|
||||
```
|
||||
./testfixture $TESTDIR/testrunner.tcl full
|
||||
tclsh $TESTDIR/testrunner.tcl full
|
||||
```
|
||||
|
||||
To run the subset of the "full" test suite for which the test file name matches
|
||||
a specified pattern (e.g. all tests that start with "fts5"), either of:
|
||||
|
||||
```
|
||||
./testfixture $TESTDIR/testrunner.tcl fts5%
|
||||
./testfixture $TESTDIR/testrunner.tcl 'fts5*'
|
||||
tclsh $TESTDIR/testrunner.tcl fts5%
|
||||
tclsh $TESTDIR/testrunner.tcl 'fts5*'
|
||||
```
|
||||
|
||||
Strictly speaking, for a test to be run the pattern must match the script
|
||||
@@ -167,9 +164,14 @@ characters specified as part of the pattern are transformed to "\*".
|
||||
To run "all" tests (full + permutations):
|
||||
|
||||
```
|
||||
./testfixture $TESTDIR/testrunner.tcl all
|
||||
tclsh $TESTDIR/testrunner.tcl all
|
||||
```
|
||||
|
||||
Note that in all of the previous examples, when running on Unix, the
|
||||
initial "<tt>tclsh</tt>" on the command-line may be omitted. The
|
||||
testrunner.tcl is marked as executable and begins with an appropriate
|
||||
"<tt>#!</tt>" line to invoke tclsh automatically.
|
||||
|
||||
<a name=binary_test_failures></a>
|
||||
## 2.3. Investigating Binary Test Failures
|
||||
|
||||
@@ -180,16 +182,21 @@ be retrieved from either *testrunner.log* or *testrunner.db*.
|
||||
If there is no permutation, the individual test script may be run with:
|
||||
|
||||
```
|
||||
./testfixture $PATH_TO_SCRIPT
|
||||
tclsh $PATH_TO_SCRIPT
|
||||
```
|
||||
|
||||
Or, if the failure occured as part of a permutation:
|
||||
|
||||
```
|
||||
./testfixture $TESTDIR/testrunner.tcl $PERMUTATION $PATH_TO_SCRIPT
|
||||
tclsh $TESTDIR/testrunner.tcl $PERMUTATION $PATH_TO_SCRIPT
|
||||
```
|
||||
|
||||
TODO: An example instead of "$PERMUTATION" and $PATH\_TO\_SCRIPT?
|
||||
One can also run all tests that failed in the previous invocation by
|
||||
typing:
|
||||
|
||||
```
|
||||
tclsh $TESTDIR/testrunner.tcl retest
|
||||
```
|
||||
|
||||
<a name=source_code_tests></a>
|
||||
# 3. Source Code Tests
|
||||
@@ -204,11 +211,9 @@ other tests. The advantages of this are that:
|
||||
* it ensures that tests are always run using binaries created with the
|
||||
same set of compiler options.
|
||||
|
||||
The testrunner.tcl commands described in this section may be run using
|
||||
either a *testfixture* (or testfixture.exe) build, or with any other Tcl
|
||||
shell that supports SQLite 3.31.1 or newer via "package require sqlite3".
|
||||
|
||||
TODO: ./configure + Makefile.msc build systems.
|
||||
The testrunner.tcl commands described in this section do not require that
|
||||
the testfixture and/or sqlite3 binaries be built ahead of time. Those
|
||||
binaries will be constructed automatically.
|
||||
|
||||
<a name=commands_to_run_tests></a>
|
||||
## 3.1. Commands to Run SQLite Tests
|
||||
@@ -227,13 +232,13 @@ In other words, it is equivalent to running:
|
||||
$TOP/configure --enable-all --enable-debug
|
||||
make fuzztest
|
||||
make testfixture
|
||||
./testfixture $TOP/test/testrunner.tcl veryquick
|
||||
tclsh $TOP/test/testrunner.tcl veryquick
|
||||
|
||||
# Then, after removing files created by the tests above:
|
||||
$TOP/configure --enable-all OPTS="-O0"
|
||||
make fuzztest
|
||||
make testfixture
|
||||
./testfixture $TOP/test/testrunner.tcl veryquick
|
||||
tclsh $TOP/test/testrunner.tcl veryquick
|
||||
```
|
||||
|
||||
The **sdevtest** command is identical to the mdevtest command, except that the
|
||||
@@ -380,7 +385,7 @@ When running either binary or source code tests, testrunner.tcl reports the
|
||||
number of jobs it intends to use to stdout. e.g.
|
||||
|
||||
```
|
||||
$ ./testfixture $TESTDIR/testrunner.tcl
|
||||
$ tclsh $TESTDIR/testrunner.tcl
|
||||
splitting work across 16 jobs
|
||||
... more output ...
|
||||
```
|
||||
@@ -390,7 +395,7 @@ of real cores on the machine. This can be overridden using the "--jobs" (or -j)
|
||||
switch:
|
||||
|
||||
```
|
||||
$ ./testfixture $TESTDIR/testrunner.tcl --jobs 8
|
||||
$ tclsh $TESTDIR/testrunner.tcl --jobs 8
|
||||
splitting work across 8 jobs
|
||||
... more output ...
|
||||
```
|
||||
@@ -400,5 +405,5 @@ running by exucuting the following command from the directory containing the
|
||||
testrunner.log and testrunner.db files:
|
||||
|
||||
```
|
||||
$ ./testfixture $TESTDIR/testrunner.tcl njob $NEW_NUMBER_OF_JOBS
|
||||
$ tclsh $TESTDIR/testrunner.tcl njob $NEW_NUMBER_OF_JOBS
|
||||
```
|
||||
|
||||
12
manifest
12
manifest
@@ -1,5 +1,5 @@
|
||||
C Verify\sthe\seffect\sof\s"--titles\son"\sin\sthe\s"insert"\smode\sin\sthe\sCLI.
|
||||
D 2025-12-31T20:43:31.772
|
||||
C Updates\sto\sthe\stestrunner\sdocumentation.
|
||||
D 2026-01-01T15:41:50.965
|
||||
F .fossil-settings/binary-glob 61195414528fb3ea9693577e1980230d78a1f8b0a54c78cf1b9b24d0a409ed6a x
|
||||
F .fossil-settings/empty-dirs dbb81e8fc0401ac46a1491ab34a7f2c7c0452f2f06b54ebb845d024ca8283ef1
|
||||
F .fossil-settings/ignore-glob 35175cdfcf539b2318cb04a9901442804be81cd677d8b889fcc9149c21f239ea
|
||||
@@ -63,7 +63,7 @@ F doc/jsonb.md acd77fc3a709f51242655ad7803510c886aa8304202fa9cf2abc5f5c4e9d7ae5
|
||||
F doc/lemon.html 2085fda0a90a94fe92159a79dccc5c30d5a2313524887a31659cd66162f17233
|
||||
F doc/pager-invariants.txt 83aa3a4724b2d7970cc3f3461f0295c46d4fc19a835a5781cbb35cb52feb0577
|
||||
F doc/tcl-extension-testing.md b88861804fc1eaf83249f8e206334189b61e150c360e1b80d0dcf91af82354f5
|
||||
F doc/testrunner.md 5ee928637e03f136a25fef852c5ed975932e31927bd9b05a574424ae18c31019
|
||||
F doc/testrunner.md 1232d8095a15a6aa6fbb6685313558038dde49e6b0b7b1daebd5cbeec5d0abf9
|
||||
F doc/trusted-schema.md 33625008620e879c7bcfbbfa079587612c434fa094d338b08242288d358c3e8a
|
||||
F doc/vdbesort-memory.md 4da2639c14cd24a31e0af694b1a8dd37eaf277aff3867e9a8cc14046bc49df56
|
||||
F doc/vfs-shm.txt 1a55f3f0e7b6745931b117ba5c9df3640d7a0536f532ef0052563100f4416f86
|
||||
@@ -2189,8 +2189,8 @@ F tool/warnings-clang.sh bbf6a1e685e534c92ec2bfba5b1745f34fb6f0bc2a362850723a9ee
|
||||
F tool/warnings.sh d924598cf2f55a4ecbc2aeb055c10bd5f48114793e7ba25f9585435da29e7e98
|
||||
F tool/win/sqlite.vsix deb315d026cc8400325c5863eef847784a219a2f
|
||||
F tool/winmain.c 00c8fb88e365c9017db14c73d3c78af62194d9644feaf60e220ab0f411f3604c
|
||||
P d2c5df41285a9c8a66e9bdea4e28d86fe10665ad01cfbab5a7ce43b8e39b4e34
|
||||
R 0430ecb6b12c6a794879456e6400ecec
|
||||
P a5f922b0832d9544b083f3d55d4029aa80e320bf5505547c5a1ddb21f4685e3e
|
||||
R 9df0730403c6e3e10d8224307f93708f
|
||||
U drh
|
||||
Z 30b7479fb2fd2c8bcd423b72d9fa94de
|
||||
Z 4765444ce4544ead9159cfe583142a35
|
||||
# Remove this line to create a well-formed Fossil manifest.
|
||||
|
||||
@@ -1 +1 @@
|
||||
a5f922b0832d9544b083f3d55d4029aa80e320bf5505547c5a1ddb21f4685e3e
|
||||
bcac99f962c6403fddfcaeef309ea9f825fa661fee0b3e3fd24f04c861be1d1b
|
||||
|
||||
Reference in New Issue
Block a user