Merge the latest trunk enhancments into the reuse-schema branch.

FossilOrigin-Name: 52262ac9205e7e4f19855a47955a5df10d53c222b4872d2ed1e5ca5230034a1d
2025-08-08 14:02:16 +03:00 · 2023-10-23 19:27:44 +00:00
parent 1d8366292e 4fecc81e20
commit 7c897ff2d6
297 changed files with 23786 additions and 10492 deletions
--- a/doc/testrunner.md
+++ b/doc/testrunner.md
@@ -0,0 +1,284 @@
+
+
+# The testrunner.tcl Script
+
+# 1. Overview
+
+testrunner.tcl is a Tcl script used to run multiple SQLite tests using 
+multiple jobs. It supports the following types of tests:
+
+  *  Tcl test scripts.
+
+  *  Tests run with [make] commands. Specifically, at time of writing, 
+     [make fuzztest], [make mptest], [make sourcetest] and [make threadtest].
+
+testrunner.tcl pipes the output of all tests and builds run into log file
+**testrunner.log**, created in the cwd directory. Searching this file for
+"failed" is a good way to find the output of a failed test.
+
+testrunner.tcl also populates SQLite database **testrunner.db**. This database
+contains details of all tests run, running and to be run. A useful query
+might be:
+
+```
+  SELECT * FROM script WHERE state='failed'
+```
+
+Running the command:
+
+```
+  ./testfixture $(TESTDIR)/testrunner.tcl status
+```
+
+in the directory containing the testrunner.db database runs various queries
+to produce a succinct report on the state of a running testrunner.tcl script.
+Running:
+
+```
+  watch ./testfixture $(TESTDIR)/testrunner.tcl status
+```
+
+in another terminal is a good way to keep an eye on a long running test.
+
+Sometimes testrunner.tcl uses the [testfixture] binary that it is run with
+to run tests (see "Binary Tests" below). Sometimes it builds testfixture and
+other binaries in specific configurations to test (see "Source Tests").
+
+# 2. Binary Tests
+
+The commands described in this section all run various combinations of the Tcl
+test scripts using the [testfixture] binary used to run the testrunner.tcl
+script (i.e. they do not invoke the compiler to build new binaries, or the
+[make] command to run tests that are not Tcl scripts). The procedure to run
+these tests is therefore:
+
+  1. Build the "testfixture" (or "testfixture.exe" for windows) binary using
+     whatever method seems convenient.
+
+  2. Test the binary built in step 1 by running testrunner.tcl with it, 
+     perhaps with various options.
+
+The following sub-sections describe the various options that can be
+passed to testrunner.tcl to test binary testfixture builds.
+
+## 2.1. Organization of Tcl Tests
+
+Tcl tests are stored in files that match the pattern *\*.test*. They are
+found in both the $TOP/test/ directory, and in the various sub-directories
+of the $TOP/ext/ directory of the source tree. Not all *\*.test* files
+contain Tcl tests - a handful are Tcl scripts designed to invoke other
+*\*.test* files.
+
+The **veryquick** set of tests is a subset of all Tcl test scripts in the
+source tree. In includes most tests, but excludes some that are very slow.
+Almost all fault-injection tests (those that test the response of the library
+to OOM or IO errors) are excluded. It is defined in source file 
+*test/permutations.test*.
+
+The **full** set of tests includes all Tcl test scripts in the source tree.
+To run a "full" test is to run all Tcl test scripts that can be found in the
+source tree.
+
+File *permutations.test* defines various test "permutations". A permutation
+consists of:
+
+  *  A subset of Tcl test scripts, and 
+
+  *  Runtime configuration to apply before running each test script 
+     (e.g. enabling auto-vacuum, or disable lookaside).
+
+Running **all** tests is to run all tests in the full test set, plus a dozen
+or so permutations. The specific permutations that are run as part of "all"
+are defined in file *testrunner_data.tcl*.
+
+## 2.2. Commands to Run Tests
+
+To run the "veryquick" test set, use either of the following:
+
+```
+  ./testfixture $TESTDIR/testrunner.tcl
+  ./testfixture $TESTDIR/testrunner.tcl veryquick
+```
+
+To run the "full" test suite:
+
+```
+  ./testfixture $TESTDIR/testrunner.tcl full
+```
+
+To run the subset of the "full" test suite for which the test file name matches
+a specified pattern (e.g. all tests that start with "fts5"), either of:
+
+```
+  ./testfixture $TESTDIR/testrunner.tcl fts5%
+  ./testfixture $TESTDIR/testrunner.tcl 'fts5*'
+```
+
+To run "all" tests (full + permutations):
+
+```
+  ./testfixture $TESTDIR/testrunner.tcl all
+```
+
+<a name=binary_test_failures></a>
+## 2.3. Investigating Binary Test Failures
+
+If a test fails, testrunner.tcl reports name of the Tcl test script and, if
+applicable, the name of the permutation, to stdout. This information can also
+be retrieved from either *testrunner.log* or *testrunner.db*.
+
+If there is no permutation, the individual test script may be run with:
+
+```
+  ./testfixture $PATH_TO_SCRIPT
+```
+
+Or, if the failure occured as part of a permutation:
+
+```
+  ./testfixture $TESTDIR/testrunner.tcl $PERMUTATION $PATH_TO_SCRIPT
+```
+
+TODO: An example instead of "$PERMUTATION" and $PATH\_TO\_SCRIPT?
+
+# 3. Source Code Tests
+
+The commands described in this section invoke the C compiler to build 
+binaries from the source tree, then use those binaries to run Tcl and
+other tests. The advantages of this are that:
+
+  *  it is possible to test multiple build configurations with a single
+     command, and 
+
+  *  it ensures that tests are always run using binaries created with the
+     same set of compiler options.
+
+The testrunner.tcl commands described in this section may be run using
+either a *testfixture* (or testfixture.exe) build, or with any other Tcl
+shell that supports SQLite 3.31.1 or newer via "package require sqlite3".
+
+TODO: ./configure + Makefile.msc build systems.
+
+## Commands to Run SQLite Tests
+
+The **mdevtest** command is equivalent to running the veryquick tests and
+the [make fuzztest] target once for each of two --enable-all builds - one 
+with debugging enabled and one without:
+
+```
+  tclsh $TESTDIR/testrunner.tcl mdevtest
+```
+
+In other words, it is equivalent to running:
+
+```
+  $TOP/configure --enable-all --enable-debug
+  make fuzztest
+  make testfixture
+  ./testfixture $TOP/test/testrunner.tcl veryquick
+
+  # Then, after removing files created by the tests above:
+  $TOP/configure --enable-all OPTS="-O0"
+  make fuzztest
+  make testfixture
+  ./testfixture $TOP/test/testrunner.tcl veryquick
+```
+
+The **sdevtest** command is identical to the mdevtest command, except that the
+second of the two builds is a sanitizer build. Specifically, this means that
+OPTS="-fsanitize=address,undefined" is specified instead of OPTS="-O0":
+
+```
+  tclsh $TESTDIR/testrunner.tcl sdevtest
+```
+
+The **release** command runs lots of tests under lots of builds. It runs
+different combinations of builds and tests depending on whether it is run
+on Linux, Windows or OSX. Refer to *testrunner\_data.tcl* for the details
+of the specific tests run.
+
+```
+  tclsh $TESTDIR/testrunner.tcl release
+```
+
+## Running ZipVFS Tests
+
+testrunner.tcl can build a zipvfs-enabled testfixture and use it to run
+tests from the Zipvfs project with the following command:
+
+```
+  tclsh $TESTDIR/testrunner.tcl --zipvfs $PATH_TO_ZIPVFS
+```
+
+This can be combined with any of "mdevtest", "sdevtest" or "release" to
+test both SQLite and Zipvfs with a single command:
+
+```
+  tclsh $TESTDIR/testrunner.tcl --zipvfs $PATH_TO_ZIPVFS mdevtest
+```
+
+## Investigating Source Code Test Failures
+
+Investigating a test failure that occurs during source code testing is a
+two step process:
+
+  1. Recreating the build configuration in which the test failed, and
+
+  2. Re-running the actual test.
+
+To recreate a build configuration, use the testrunner.tcl **script** command
+to create a build script. A build script is a bash script on Linux or OSX, or
+a dos \*.bat file on windows. For example:
+
+```
+  # Create a script that recreates build configuration "Device-One" on 
+  # Linux or OSX:
+  tclsh $TESTDIR/testrunner.tcl script Device-One > make.sh 
+
+  # Create a script that recreates build configuration "Have-Not" on Windows:
+  tclsh $TESTDIR/testrunner.tcl script Have-Not > make.bat 
+```
+
+The generated bash or \*.bat file script accepts a single argument - a makefile
+target to build. This may be used either to run a [make] command test directly,
+or else to build a testfixture (or testfixture.exe) binary with which to
+run a Tcl test script, as <a href=#binary_test_failures>described above</a>.
+
+
+
+# 4. Controlling CPU Core Utilization
+
+When running either binary or source code tests, testrunner.tcl reports the
+number of jobs it intends to use to stdout. e.g.
+
+```
+  $ ./testfixture $TESTDIR/testrunner.tcl
+  splitting work across 16 jobs
+  ... more output ...
+```
+
+By default, testfixture.tcl attempts to set the number of jobs to the number 
+of real cores on the machine. This can be overridden using the "--jobs" (or -j)
+switch:
+
+```
+  $ ./testfixture $TESTDIR/testrunner.tcl --jobs 8
+  splitting work across 8 jobs
+  ... more output ...
+```
+
+The number of jobs may also be changed while an instance of testrunner.tcl is
+running by exucuting the following command from the directory containing the
+testrunner.log and testrunner.db files:
+
+```
+  $ ./testfixture $TESTDIR/testrunner.tcl njob $NEW_NUMBER_OF_JOBS
+```
+
+
+
+
+
+
+
+