sqlite/doc/testrunner.md

The testrunner.tcl Script

• 1. Overview
• 2. Binary Tests
  • 2.1. Organization of Tcl Tests
  • 2.2. Commands to Run Tests
  • 2.3. Investigating Binary Test Failures
• 3. Source Tests
  • 3.1. Commands to Run SQLite Tests
  • 3.2. Running ZipVFS Tests
  • 3.3. Investigating Source Code Test Failures
  • 3.4. External Fuzzcheck Databases
• 4. Extra testrunner.tcl Options
• 5. Controlling CPU Core Utilization

1. Overview

The testrunner.tcl program is a Tcl script used to run multiple SQLite tests in parallel, thus reducing testing time on multi-core machines. It supports the following types of tests:

• Tcl test scripts.

• Fuzzcheck tests, including using an external fuzzcheck database.

• Tests run with make commands. Examples:

  • make devtest
  • make releasetest
  • make sdevtest
  • make testrunner

The testrunner.tcl program stores output of all tests and builds run in log file testrunner.log, created in the current working directory. Search this file to find details of errors. Suggested search commands:

• grep "^!" testrunner.log
• grep failed testrunner.log

The testrunner.tcl program also populates SQLite database testrunner.db. This database contains details of all tests run, running and to be run. A useful query might be:

  SELECT * FROM script WHERE state='failed'

You can get a summary of errors in a prior run by invoking commands like these:

  tclsh $(TESTDIR)/testrunner.tcl errors
  tclsh $(TESTDIR)/testrunner.tcl errors -v

Running the command:

  tclsh $(TESTDIR)/testrunner.tcl status

in the directory containing the testrunner.db database runs various queries to produce a succinct report on the state of a running testrunner.tcl script. A good way to keep and eye on test progress is to run either of the two following commands:

  watch tclsh $(TESTDIR)/testrunner.tcl status
  tclsh $(TESTDIR)/testrunner.tcl status -d 2

Both of the commands above accomplish about the same thing, but the second one has the advantage of not requiring "watch" to be installed on your system.

Sometimes testrunner.tcl uses the testfixture binary that it is run with to run tests (see "Binary Tests" below). Sometimes it builds testfixture and other binaries in specific configurations to test (see "Source Tests").

2. Binary Tests

The commands described in this section all run various combinations of the Tcl test scripts using the testfixture binary used to run the testrunner.tcl script (i.e. they do not invoke the compiler to build new binaries, or the make command to run tests that are not Tcl scripts). The procedure to run these tests is therefore:

1. Build the "testfixture" (or "testfixture.exe" for windows) binary using whatever method seems convenient.

2. Test the binary built in step 1 by running testrunner.tcl with it, perhaps with various options.

The following sub-sections describe the various options that can be passed to testrunner.tcl to test binary testfixture builds.

2.1. Organization of Tcl Tests

Tcl tests are stored in files that match the pattern *.test. They are found in both the $TOP/test/ directory, and in the various sub-directories of the $TOP/ext/ directory of the source tree. Not all *.test files contain Tcl tests - a handful are Tcl scripts designed to invoke other *.test files.

The veryquick set of tests is a subset of all Tcl test scripts in the source tree. In includes most tests, but excludes some that are very slow. Almost all fault-injection tests (those that test the response of the library to OOM or IO errors) are excluded. It is defined in source file test/permutations.test.

The full set of tests includes all Tcl test scripts in the source tree. To run a "full" test is to run all Tcl test scripts that can be found in the source tree.

File permutations.test defines various test "permutations". A permutation consists of:

• A subset of Tcl test scripts, and

• Runtime configuration to apply before running each test script (e.g. enabling auto-vacuum, or disable lookaside).

Running all tests is to run all tests in the full test set, plus a dozen or so permutations. The specific permutations that are run as part of "all" are defined in file testrunner_data.tcl.

2.2. Commands to Run Tests

To run the "veryquick" test set, use either of the following:

  ./testfixture $TESTDIR/testrunner.tcl
  ./testfixture $TESTDIR/testrunner.tcl veryquick

To run the "full" test suite:

  ./testfixture $TESTDIR/testrunner.tcl full

To run the subset of the "full" test suite for which the test file name matches a specified pattern (e.g. all tests that start with "fts5"), either of:

  ./testfixture $TESTDIR/testrunner.tcl fts5%
  ./testfixture $TESTDIR/testrunner.tcl 'fts5*'

Strictly speaking, for a test to be run the pattern must match the script filename, not including the directory, using the rules of Tcl's [string match] command. Except that before the matching is done, any "%" characters specified as part of the pattern are transformed to "*".

To run "all" tests (full + permutations):

  ./testfixture $TESTDIR/testrunner.tcl all

2.3. Investigating Binary Test Failures

If a test fails, testrunner.tcl reports name of the Tcl test script and, if applicable, the name of the permutation, to stdout. This information can also be retrieved from either testrunner.log or testrunner.db.

If there is no permutation, the individual test script may be run with:

  ./testfixture $PATH_TO_SCRIPT

Or, if the failure occured as part of a permutation:

  ./testfixture $TESTDIR/testrunner.tcl $PERMUTATION $PATH_TO_SCRIPT

TODO: An example instead of "$PERMUTATION" and $PATH_TO_SCRIPT?

3. Source Code Tests

The commands described in this section invoke the C compiler to build binaries from the source tree, then use those binaries to run Tcl and other tests. The advantages of this are that:

• it is possible to test multiple build configurations with a single command, and

• it ensures that tests are always run using binaries created with the same set of compiler options.

The testrunner.tcl commands described in this section may be run using either a testfixture (or testfixture.exe) build, or with any other Tcl shell that supports SQLite 3.31.1 or newer via "package require sqlite3".

TODO: ./configure + Makefile.msc build systems.

3.1. Commands to Run SQLite Tests

The mdevtest command is equivalent to running the veryquick tests and the make fuzztest target once for each of two --enable-all builds - one with debugging enabled and one without:

  tclsh $TESTDIR/testrunner.tcl mdevtest

In other words, it is equivalent to running:

  $TOP/configure --enable-all --enable-debug
  make fuzztest
  make testfixture
  ./testfixture $TOP/test/testrunner.tcl veryquick

  # Then, after removing files created by the tests above:
  $TOP/configure --enable-all OPTS="-O0"
  make fuzztest
  make testfixture
  ./testfixture $TOP/test/testrunner.tcl veryquick

The sdevtest command is identical to the mdevtest command, except that the second of the two builds is a sanitizer build. Specifically, this means that OPTS="-fsanitize=address,undefined" is specified instead of OPTS="-O0":

  tclsh $TESTDIR/testrunner.tcl sdevtest

The release command runs lots of tests under lots of builds. It runs different combinations of builds and tests depending on whether it is run on Linux, Windows or OSX. Refer to testrunner_data.tcl for the details of the specific tests run.

  tclsh $TESTDIR/testrunner.tcl release

As with source code tests, one or more patterns may be appended to any of the above commands (mdevtest, sdevtest or release). Pattern matching is used for both Tcl tests and fuzz tests.

  tclsh $TESTDIR/testrunner.tcl release rtree%

3.2. Running ZipVFS Tests

testrunner.tcl can build a zipvfs-enabled testfixture and use it to run tests from the Zipvfs project with the following command:

  tclsh $TESTDIR/testrunner.tcl --zipvfs $PATH_TO_ZIPVFS

This can be combined with any of "mdevtest", "sdevtest" or "release" to test both SQLite and Zipvfs with a single command:

  tclsh $TESTDIR/testrunner.tcl --zipvfs $PATH_TO_ZIPVFS mdevtest

3.3. Investigating Source Code Test Failures

Investigating a test failure that occurs during source code testing is a two step process:

1. Recreating the build configuration in which the test failed, and

2. Re-running the actual test.

To recreate a build configuration, use the testrunner.tcl script command to create a build script. A build script is a bash script on Linux or OSX, or a dos *.bat file on windows. For example:

  # Create a script that recreates build configuration "Device-One" on 
  # Linux or OSX:
  tclsh $TESTDIR/testrunner.tcl script Device-One > make.sh 

  # Create a script that recreates build configuration "Have-Not" on Windows:
  tclsh $TESTDIR/testrunner.tcl script Have-Not > make.bat 

The generated bash or *.bat file script accepts a single argument - a makefile target to build. This may be used either to run a make command test directly, or else to build a testfixture (or testfixture.exe) binary with which to run a Tcl test script, as described above.

3.4 External Fuzzcheck Databases

Testrunner.tcl will also run fuzzcheck against an external (out of tree) database, for example fuzzcheck databases generated by dbsqlfuzz. To do this, simply add the "--fuzzdb FILENAME" command-line option or set the FUZZDB environment variable to the name of the external database. For large external databases, testrunner.tcl will automatically use the "--slice" command-line option of fuzzcheck to divide the work up into multiple jobs, to increase parallelism.

Thus, for example, to run a full releasetest including an external dbsqlfuzz database, run a command like one of these:

  tclsh test/testrunner.tcl releasetest --fuzzdb ../fuzz/20250415.db
  FUZZDB=../fuzz/20250415.db make releasetest
  nmake /f Makefile.msc FUZZDB=../fuzz/20250415.db releasetest

The patternlist option to testrunner.tcl will match against fuzzcheck databases. So if you want to run only tests involving the external database, you can use a command something like this:

  tclsh test/testrunner.tcl releasetest 20250415 --fuzzdb ../fuzz/20250415.db

4. Extra testrunner.tcl Options

The testrunner.tcl script options in this section may be used with both source code and binary tests.

The --buildonly option instructs testrunner.tcl just to build the binaries required by a test, not to run any actual tests. For example:

  # Build binaries required by release test.
  tclsh $TESTDIR/testrunner.tcl --buildonly release"

The --dryrun option prevents testrunner.tcl from building any binaries or running any tests. Instead, it just writes the shell commands that it would normally execute into the testrunner.log file. Example:

  # Log the shell commmands that make up the mdevtest test.
  tclsh $TESTDIR/testrunner.tcl --dryrun mdevtest"

The --explain option is similar to --dryrun in that it prevents testrunner.tcl from building any binaries or running any tests. The difference is that --explain prints on standard output a human-readable summary of all the builds and tests that would have been run.

  # Show what builds and tests would have been run
  tclsh $TESTDIR/testrunner.tcl --explain mdevtest

The --status option uses VT100 escape sequences to display the test status full-screen. This is similar to running "watch test/testrunner status" in a separate window, just more convenient. Unfortunately, this option does not work correctly on Windows, due to the sketchy implementation of VT100 escapes on the Windows console.

5. Controlling CPU Core Utilization

When running either binary or source code tests, testrunner.tcl reports the number of jobs it intends to use to stdout. e.g.

  $ ./testfixture $TESTDIR/testrunner.tcl
  splitting work across 16 jobs
  ... more output ...

By default, testfixture.tcl attempts to set the number of jobs to the number of real cores on the machine. This can be overridden using the "--jobs" (or -j) switch:

  $ ./testfixture $TESTDIR/testrunner.tcl --jobs 8
  splitting work across 8 jobs
  ... more output ...

The number of jobs may also be changed while an instance of testrunner.tcl is running by exucuting the following command from the directory containing the testrunner.log and testrunner.db files:

  $ ./testfixture $TESTDIR/testrunner.tcl njob $NEW_NUMBER_OF_JOBS