Migrate the rest of the test harness from Perl to Python.

test.pl becomes test.py and ci.pl becomes ci.py, so an invocation only changes from .pl to .py. Running the tests needs no command since that is what a developer does, and the unit and coverage commands that a run drives are commands of test.py because a test runs inside a container where only the repository copy is available.

The Perl library in test/lib is gone. The modules the documentation depends on move to doc/lib, with ExecuteTest becoming Execute, and are trimmed to what the documentation calls, which removes the storage manifest, info, and owner along with everything else the tests were the only user of. release.pl no longer needs the vm definitions since all it did with them was compare two vm names.

Nothing in a test container or on a CI runner runs on Perl now so none of it is installed in either place, which was a substantial part of a base image. The packaging tools go with it since they are written in Perl and nothing builds a package, and ca-certificates is installed explicitly because it was arriving as a dependency of a package that is no longer there. Every base image is rebuilt as a result.

The generated Dockerfile is otherwise unchanged. The one difference is that the PostgreSQL repository for RHEL is installed for the architecture being built rather than for the architecture of the host, which only differs when cross building.

The list of tests to run is unchanged. It was compared against the Perl across the vm, module, test, run, and PostgreSQL version options before the Perl was removed.

The vm definitions, the user lookup, the test list, the container build, the job, the test run itself, and the code format are covered by tests, which the Perl equivalents never were. The container build is checked by comparing the generated script in full, since the hash of that script is the cache key.

Make the code format a command rather than two options of a test run, and fold in the black check that CI has been running as a step of its own. It gets the files to format from git rather than from the repository copy, so it no longer depends on a test run having made that copy and is told nothing about where either path is.

Wire up --log-level-test, which the driver has been collecting and discarding. Remove --psql-bin, --gen-check, --no-coverage-report, --build-max, --log-level-test-file, --no-optimize, --no-debug, and --debug-test-trace, none of which have done anything for some time.

Allow --vm=all only for --vm-build. Building every container at once works but running the tests on every vm never has, since a test needs the binary and the build for the vm it runs on and there is one of each.

.github/workflows/test-arch.yml

@@ -28,7 +28,7 @@ jobs:
         run: echo "${{secrets.GITHUB_TOKEN}}" | docker login ghcr.io -u ${{github.actor}} --password-stdin
 
       - name: Run Test
-        run: cd ${HOME?} && ${GITHUB_WORKSPACE?}/pgbackrest/test/ci.pl test --vm=u22 --param=c-only --param=no-coverage --param=no-back-trace --param=build-max=2
+        run: cd ${HOME?} && ${GITHUB_WORKSPACE?}/pgbackrest/test/ci.py test --vm=u22 --param=c-only --param=no-coverage --param=no-back-trace
 
   # Basic tests on other architectures using emulation. The emulation is so slow that running all the unit tests would be too
   # expensive, but this at least shows that the build works and some of the more complex tests run. In particular, it is good to
@@ -59,7 +59,7 @@ jobs:
       - name: Install
         run: |
           sudo apt-get update
-          sudo DEBCONF_NONINTERACTIVE_SEEN=true DEBIAN_FRONTEND=noninteractive apt-get install -y perl sudo libxml-checker-perl libyaml-perl python3-yaml rsync zlib1g-dev libssl-dev libxml2-dev libpq-dev libyaml-dev pkg-config make gcc ccache meson git liblz4-dev liblz4-tool zstd libzstd-dev bzip2 libbz2-dev
+          sudo DEBCONF_NONINTERACTIVE_SEEN=true DEBIAN_FRONTEND=noninteractive apt-get install -y sudo python3-yaml rsync zlib1g-dev libssl-dev libxml2-dev libpq-dev libyaml-dev pkg-config make gcc ccache meson git liblz4-dev liblz4-tool zstd libzstd-dev bzip2 libbz2-dev
           docker run --privileged --rm tonistiigi/binfmt --install all
 
       - name: Login to GHCR
@@ -67,7 +67,7 @@ jobs:
         run: echo "${{secrets.GITHUB_TOKEN}}" | docker login ghcr.io -u ${{github.actor}} --password-stdin
 
       - name: Build VM
-        run: ${GITHUB_WORKSPACE?}/pgbackrest/test/test.pl --vm-build --vm=u22 --vm-arch=${{matrix.arch}}
+        run: ${GITHUB_WORKSPACE?}/pgbackrest/test/test.py --vm-build --vm=u22 --vm-arch=${{matrix.arch}}
 
       # Compiling under emulation is slow enough that a warm ccache saves a lot of time
       - name: Restore ccache
@@ -79,16 +79,16 @@ jobs:
 
       - name: Run Test
         run: |
-          ${GITHUB_WORKSPACE?}/pgbackrest/test/test.pl --vm=u22 --vm-arch=${{matrix.arch}} --no-valgrind --no-coverage --no-optimize --build-max=2 --module=common --test=type-pack --test=crypto --test=compress
-          ${GITHUB_WORKSPACE?}/pgbackrest/test/test.pl --vm=u22 --vm-arch=${{matrix.arch}} --no-valgrind --no-coverage --no-optimize --build-max=2 --module=postgres --test=interface
-          ${GITHUB_WORKSPACE?}/pgbackrest/test/test.pl --vm=u22 --vm-arch=${{matrix.arch}} --no-valgrind --no-coverage --no-optimize --build-max=2 --module=command --test=backup --test=restore --test=archive-get --test=archive-push --test=verify
+          ${GITHUB_WORKSPACE?}/pgbackrest/test/test.py --vm=u22 --vm-arch=${{matrix.arch}} --no-valgrind --no-coverage --module=common --test=type-pack --test=crypto --test=compress
+          ${GITHUB_WORKSPACE?}/pgbackrest/test/test.py --vm=u22 --vm-arch=${{matrix.arch}} --no-valgrind --no-coverage --module=postgres --test=interface
+          ${GITHUB_WORKSPACE?}/pgbackrest/test/test.py --vm=u22 --vm-arch=${{matrix.arch}} --no-valgrind --no-coverage --module=command --test=backup --test=restore --test=archive-get --test=archive-push --test=verify
 
       # Run the smoke test against the arch build: a real PostgreSQL driven through a full backup/restore/verify cycle, which
       # exercises checksums end-to-end on big-endian. It reuses the pgbackrest binary already built for the arch above and the
-      # single PostgreSQL version baked into the arch base image. A fresh container is started (test.pl removes its own) with the
+      # single PostgreSQL version baked into the arch base image. A fresh container is started (test.py removes its own) with the
       # SSH entrypoint overridden and run as root so smoke.py can drop to the postgres user, as the misc smoke job does.
       #
-      # The binary is copied to /tmp first: test.pl creates the test tree mode 0770 owned by the build user, so the unprivileged
+      # The binary is copied to /tmp first: test.py creates the test tree mode 0770 owned by the build user, so the unprivileged
       # postgres user that smoke.py drops to cannot traverse it to execute the binary. /tmp is world-accessible -- it is where
       # smoke.py puts its own work dirs -- so the copy there is reachable.
       - name: Run Smoke Test

.github/workflows/test-bsd.yml

@@ -15,7 +15,6 @@ jobs:
     env:
       LDFLAGS: -L/opt/homebrew/opt/openssl@3/lib -L/opt/homebrew/opt/libpq/lib -L/opt/homebrew/opt/libxml2/lib -L/opt/homebrew/opt/libyaml/lib
       CPPFLAGS: -I/opt/homebrew/opt/openssl@3/include -I/opt/homebrew/opt/libpq/include -I/opt/homebrew/opt/libxml2/include/libxml2 -I/opt/homebrew/opt/libyaml/include
-      PERL5LIB: /opt/homebrew/opt/perl5/lib/perl5
       PKG_CONFIG_PATH: /opt/homebrew/opt/libpq/lib/pkgconfig:/opt/homebrew/opt/openssl@3/lib/pkgconfig
 
     steps:
@@ -73,7 +72,7 @@ jobs:
           ls -lah ${GITHUB_WORKSPACE?}
 
       - name: Run Test
-        run: cd .. && ${GITHUB_WORKSPACE?}/test/test.pl --vm-max=2 --no-coverage --no-valgrind  --module=command --test=backup --test=info --test=archive-push
+        run: cd .. && ${GITHUB_WORKSPACE?}/test/test.py --vm-max=2 --no-coverage --no-valgrind  --module=command --test=backup --test=info --test=archive-push
 
       # The integration tests cannot run here because there are no containers, but the smoke test drives a backup/restore cycle
       # against each PostgreSQL installed above, which covers the basics
@@ -139,8 +138,7 @@ jobs:
 
           # postgresql*-client provides the libpq to build against and postgresql*-server provides the tooling the smoke test needs
           # to create a cluster
-          sudo pkg install -y bash git ${pgServer%-server}-client ${pgServer} pkgconf libxml2 gmake perl5 libyaml \
-            p5-YAML-LibYAML rsync meson
+          sudo pkg install -y bash git ${pgServer%-server}-client ${pgServer} pkgconf libxml2 gmake libyaml rsync meson
 
           # PyYAML for the test harness. The package is py<version>-pyyaml, renamed from py<version>-yaml in 2024 so both names are
           # searched, and it is built only for the python flavors the ports tree selects, which is not necessarily the version the
@@ -157,7 +155,7 @@ jobs:
 
       - name: Run Test
         shell: cpa.sh {0}
-        run: cd .. && perl ${GITHUB_WORKSPACE?}/test/test.pl --vm-max=2 --no-coverage --no-valgrind  --module=command --test=backup --test=info --test=archive-push
+        run: cd .. && ${GITHUB_WORKSPACE?}/test/test.py --vm-max=2 --no-coverage --no-valgrind  --module=command --test=backup --test=info --test=archive-push
 
       # The integration tests cannot run here because there are no containers, but the smoke test drives a backup/restore cycle
       # against the PostgreSQL installed above, which covers the basics

.github/workflows/test-doc.yml

@@ -44,4 +44,4 @@ jobs:
         run: echo "${{secrets.GITHUB_TOKEN}}" | docker login ghcr.io -u ${{github.actor}} --password-stdin
 
       - name: Build Documentation
-        run: cd ${HOME?} && ${GITHUB_WORKSPACE?}/pgbackrest/test/ci.pl doc --vm=${{matrix.vm}}
+        run: cd ${HOME?} && ${GITHUB_WORKSPACE?}/pgbackrest/test/ci.py doc --vm=${{matrix.vm}}

.github/workflows/test-linux.yml

@@ -69,7 +69,7 @@ jobs:
         run: echo "${{secrets.GITHUB_TOKEN}}" | docker login ghcr.io -u ${{github.actor}} --password-stdin
 
       - name: Run Test
-        run: cd ${HOME?} && ${GITHUB_WORKSPACE?}/pgbackrest/test/ci.pl ${{matrix.param}} --param=build-max=2
+        run: cd ${HOME?} && ${GITHUB_WORKSPACE?}/pgbackrest/test/ci.py ${{matrix.param}}
 
       # Upload the coverage report on failure in case the failure was caused by lack of coverage so uncovered lines can be
       # inspected from the run's artifacts

.github/workflows/test-misc.yml

@@ -43,25 +43,13 @@ jobs:
         with:
           path: pgbackrest
 
-      # Check that the Python is formatted. On failure the exact command to fix it is printed so it can be copied and run locally.
-      # This runs before the C format check because that check runs ci.pl, which removes passwordless sudo, and installing black
-      # needs it.
-      - name: Check Python Format
+      # The formatters and the PyYAML the harness needs to load. Nothing else is required since the check only reads the source.
+      - name: Install
         run: |
           sudo apt-get update
-          DEBCONF_NONINTERACTIVE_SEEN=true DEBIAN_FRONTEND=noninteractive sudo apt-get install -y black
+          DEBCONF_NONINTERACTIVE_SEEN=true DEBIAN_FRONTEND=noninteractive sudo apt-get install -y uncrustify black \
+            python3-yaml
 
-          cd ${GITHUB_WORKSPACE?}/pgbackrest
-
-          if ! black --check --diff --line-length=132 test/smoke.py test/test.py test/lib test/src/module/test; then
-              echo
-              echo "ERROR: Python is not formatted correctly. Run this from the pgbackrest path to fix it:"
-              echo
-              echo "    black --line-length=132 test/smoke.py test/test.py test/lib test/src/module/test"
-              echo
-              exit 1
-          fi
-
-      - name: Check C Format
-        run: |
-          cd ${HOME?} && ${GITHUB_WORKSPACE?}/pgbackrest/test/ci.pl test --param=code-format-check
+      # On failure the check says how to fix what it found so it can be run locally
+      - name: Check
+        run: ${GITHUB_WORKSPACE?}/pgbackrest/test/test.py code-format --check

CODING.md

@@ -2,9 +2,9 @@
 
 ## Formatting with uncrustify
 
-pgBackRest uses uncrustify to check/update the code formatting. If the `code-format` test fails in CI then reformat the code:
+pgBackRest uses uncrustify for C and black for Python to check/update the code formatting. If the `code-format` test fails in CI then reformat the code:
 ```
-pgbackrest/test/test.pl --code-format
+pgbackrest/test/test.py code-format
 ```
 
 Also review the standards in the following sections below. Some standards require further explanation and others are not enforced by uncrustify.

CONTRIBUTING.md

@@ -26,14 +26,14 @@ Coding standards are defined in [CODING.md](https://github.com/pgbackrest/pgback
 
 ## Building a Development Environment
 
-This example is based on Ubuntu 22.04, but it should work on many versions of Debian and Ubuntu.
+This example is based on Ubuntu 24.04, but it should work on many versions of Debian and Ubuntu.
 
 pgbackrest-dev => Install development tools
 ```
-sudo apt-get install rsync git devscripts build-essential valgrind autoconf \
-       autoconf-archive libssl-dev zlib1g-dev libxml2-dev libpq-dev pkg-config \
-       libxml-checker-perl libyaml-perl libdbd-pg-perl liblz4-dev liblz4-tool \
-       zstd libzstd-dev bzip2 libbz2-dev libyaml-dev ccache python3-distutils meson
+sudo apt-get install rsync git build-essential valgrind libssl-dev zlib1g-dev \
+       libxml2-dev libpq-dev pkg-config libxml-checker-perl liblz4-dev liblz4-tool \
+       zstd libzstd-dev bzip2 libbz2-dev libyaml-dev ccache meson python3-yaml \
+       python3-coverage
 ```
 
 Some unit tests and all the integration tests require Docker. Running in containers allows us to simulate multiple hosts, test on different distributions and versions of PostgreSQL, and use sudo without affecting the host system.
@@ -51,7 +51,7 @@ pgbackrest-dev => Clone pgBackRest repository
 git clone https://github.com/pgbackrest/pgbackrest.git
 ```
 
-If using a RHEL-based system, the CPAN XML parser is required to run `test.pl` and `doc.pl`. Instructions for installing Docker and the XML parser can be found in the `README.md` file of the pgBackRest [doc](https://github.com/pgbackrest/pgbackrest/blob/main/doc) directory in the section "The following is a sample RHEL 7 configuration that can be used for building the documentation".
+If using a RHEL-based system, the CPAN XML parser is required to run `doc.pl`. Instructions for installing Docker and the XML parser can be found in the `README.md` file of the pgBackRest [doc](https://github.com/pgbackrest/pgbackrest/blob/main/doc) directory in the section "The following is a sample RHEL 7 configuration that can be used for building the documentation".
 
 ## Coding
 
@@ -191,12 +191,12 @@ myObjToLog(const MyObj *this)
 
 A list of all possible test combinations can be viewed by running:
 ```
-pgbackrest/test/test.pl --dry-run
+pgbackrest/test/test.py --dry-run
 ```
 
 While some files are automatically generated during `make`, others are generated by running the test harness as follows:
 ```
-pgbackrest/test/test.pl --gen-only
+pgbackrest/test/test.py --gen-only
 ```
 
 Prior to any submission, the html version of the documentation should also be run and the output checked by viewing the generated html on the local file system under `pgbackrest/doc/output/html`. More details can be found in the pgBackRest [doc/README.md](https://github.com/pgbackrest/pgbackrest/blob/main/doc/README.md) file.
@@ -218,7 +218,7 @@ Examples of test runs are provided in the following sections. There are several
 
 For more options, run the test or documentation engine with the `--help` option:
 ```
-pgbackrest/test/test.pl --help
+pgbackrest/test/test.py --help
 pgbackrest/doc/doc.pl --help
 ```
 
@@ -228,28 +228,31 @@ If Docker is not installed, then the available tests can be listed using `--dry-
 
 pgbackrest-dev => List tests that don't require a container
 ```
-pgbackrest/test/test.pl --dry-run
+pgbackrest/test/test.py --dry-run
 
 --- output ---
 
-    P00   INFO: test begin on x86_64 - log level info
+    P00   INFO: test command begin 2.60.0dev: --dry-run --no-log-timestamp
+    P00   INFO: test begin on aarch64 - log level info
     P00   INFO: clean autogenerate code
---> P00   INFO: 83 tests selected
+--> P00   INFO: 104 tests selected
                 
-    P00   INFO: P1-T01/83 - vm=none, module=common, test=error
-           [filtered 80 lines of output]
-    P00   INFO: P1-T82/83 - vm=none, module=performance, test=type
-    P00   INFO: P1-T83/83 - vm=none, module=performance, test=storage
+    P00   INFO: P1-T001/104 - vm=none, module=common, test=error
+           [filtered 101 lines of output]
+    P00   INFO: P1-T103/104 - vm=none, module=performance, test=type
+    P00   INFO: P1-T104/104 - vm=none, module=performance, test=storage
 --> P00   INFO: DRY RUN COMPLETED SUCCESSFULLY
+    P00   INFO: test command end: completed successfully
 ```
 
 pgbackrest-dev => Run a test
 ```
-pgbackrest/test/test.pl --vm-out --module=common --test=wait
+pgbackrest/test/test.py --vm-out --module=common --test=wait
 
 --- output ---
 
-    P00   INFO: test begin on x86_64 - log level info
+    P00   INFO: test command begin 2.60.0dev: --vm-out --module=common --test=wait --no-log-timestamp
+    P00   INFO: test begin on aarch64 - log level info
     P00   INFO: cleanup old data
     P00   INFO: autogenerate code
     P00   INFO: build for none (/home/vagrant/test/build/none)
@@ -257,8 +260,8 @@ pgbackrest/test/test.pl --vm-out --module=common --test=wait
                 
     P00   INFO: P1-T1/1 - vm=none, module=common, test=wait
                 
-        P00   INFO: test command begin 2.59.0: [common/wait] --log-level=info --no-log-timestamp --repo-path=/home/vagrant/test/repo --scale=1 --test-path=/home/vagrant/test --vm=none --vm-id=0
-        P00   INFO: test command end: completed successfully
+        P00   INFO: unit command begin 2.60.0dev: unit --repo-path=/home/vagrant/test/repo --test-path=/home/vagrant/test --log-level=info --log-level-test=off --vm=none --vm-id=0 --no-log-timestamp --scale=1 common/wait
+        P00   INFO: unit command end: completed successfully
         run 1 - waitNew(), waitMore, and waitFree()
                       L0018     expect AssertError: assertion 'waitTime <= 999999000' failed
         
@@ -312,17 +315,19 @@ pgbackrest/test/test.pl --vm-out --module=common --test=wait
     P00   INFO: P1-T1/1 - vm=none, module=common, test=wait
     P00   INFO: tested modules have full coverage
     P00   INFO: TESTS COMPLETED SUCCESSFULLY
+    P00   INFO: test command end: completed successfully
 ```
 
 An entire module can be run by using only the `--module` option.
 
 pgbackrest-dev => Run a module
 ```
-pgbackrest/test/test.pl --module=postgres
+pgbackrest/test/test.py --module=postgres
 
 --- output ---
 
-    P00   INFO: test begin on x86_64 - log level info
+    P00   INFO: test command begin 2.60.0dev: --module=postgres --no-log-timestamp
+    P00   INFO: test begin on aarch64 - log level info
     P00   INFO: cleanup old data
     P00   INFO: autogenerate code
     P00   INFO: build for none (/home/vagrant/test/build/none)
@@ -332,40 +337,45 @@ pgbackrest/test/test.pl --module=postgres
     P00   INFO: P1-T2/2 - vm=none, module=postgres, test=interface
     P00   INFO: tested modules have full coverage
     P00   INFO: TESTS COMPLETED SUCCESSFULLY
+    P00   INFO: test command end: completed successfully
 ```
 
 #### With Docker
 
-Build a container to run tests. The vm must be pre-configured but a variety are available. A vagrant file is provided in the test directory as an example of running in a virtual environment. The vm names are all three character abbreviations, e.g. `u22` for Ubuntu 22.04.
+Build a container to run tests. The vm must be pre-configured but a variety are available. A vagrant file is provided in the test directory as an example of running in a virtual environment. The vm names are all three character abbreviations, e.g. `u24` for Ubuntu 24.04.
 
 pgbackrest-dev => Build a VM
 ```
-pgbackrest/test/test.pl --vm-build --vm=u22
+pgbackrest/test/test.py --vm-build --vm=u24
 
 --- output ---
 
-    P00   INFO: test begin on x86_64 - log level info
-    P00   INFO: Checking cache ghcr.io/pgbackrest/test:u22-base-x86_64-20260717A-bbc27d6dee5c ...
-    P00   INFO: Using cached ghcr.io/pgbackrest/test:u22-base-x86_64-20260717A-bbc27d6dee5c image
-    P00   INFO: Building ghcr.io/pgbackrest/test:u22-test-x86_64 image ...
+    P00   INFO: test command begin 2.60.0dev: --vm-build --vm=u24 --no-log-timestamp
+    P00   INFO: test begin on aarch64 - log level info
+    P00   INFO: Checking cache ghcr.io/pgbackrest/test:u24-base-aarch64-20260710A-ccbab513fcbd ...
+    P00   INFO: Building ghcr.io/pgbackrest/test:u24-base-aarch64 image (ghcr.io/pgbackrest/test:u24-base-aarch64-20260710A-ccbab513fcbd) ...
+    P00   INFO: Building ghcr.io/pgbackrest/test:u24-test-aarch64 image ...
     P00   INFO: Build Complete
+    P00   INFO: test command end: completed successfully
 ```
-> **NOTE:** to build all the vms, just omit the `--vm` option above.
+> **NOTE:** to build all the vms, use `--vm=all` above.
 
 pgbackrest-dev => Run a Specific Test Run
 ```
-pgbackrest/test/test.pl --vm=u22 --module=postgres --test=interface --run=2
+pgbackrest/test/test.py --vm=u24 --module=postgres --test=interface --run=2
 
 --- output ---
 
-    P00   INFO: test begin on x86_64 - log level info
+    P00   INFO: test command begin 2.60.0dev: --vm=u24 --module=postgres --test=interface --run=2 --no-log-timestamp
+    P00   INFO: test begin on aarch64 - log level info
     P00   INFO: cleanup old data and containers
     P00   INFO: autogenerate code
-    P00   INFO: clean build for u22 (/home/vagrant/test/build/u22)
+    P00   INFO: clean build for u24 (/home/vagrant/test/build/u24)
     P00   INFO: 1 test selected
                 
-    P00   INFO: P1-T1/1 - vm=u22, module=postgres, test=interface, run=2
+    P00   INFO: P1-T1/1 - vm=u24, module=postgres, test=interface, run=2
     P00   INFO: TESTS COMPLETED SUCCESSFULLY
+    P00   INFO: test command end: completed successfully
 ```
 
 ### Writing a Unit Test
@@ -514,7 +524,7 @@ TEST_ERROR(cmdCheck(), ConfigError, "primary database not found\nHINT: check ind
 
 Unit tests are run for all files that are listed in `define.yaml` and a coverage report generated for each file listed under the tag `coverage:`. Note that some files are listed in multiple `coverage:` sections for a module; in this case, each test for the file being modified should be specified for the module in which the file exists (e.g. `--module=storage --test=posix --test=gcs`, etc.) or, alternatively, simply run the module without the `--test` option. It is recommended that a `--vm` be specified since running the same test for multiple vms is unnecessary for coverage. The following example would run the test set from the **define.yaml** section detailed above.
 ```
-pgbackrest/test/test.pl --vm-out --module=command --test=check --vm=u22
+pgbackrest/test/test.py --vm-out --module=command --test=check --vm=u24
 ```
 > **NOTE:** Not all systems perform at the same speed, so if a test is timing out, try rerunning with another vm.
 
@@ -526,7 +536,7 @@ If 100 percent code coverage has not been achieved, an error message will be dis
 
 Sometimes it is useful to look at files that were generated during the test. The default for running any test is that, at the start/end of the test, the test harness will clean up all files and directories created. To override this behavior, a single test run must be specified and the option `--no-cleanup` provided. Again, continuing with the check command, from **define.yaml** above, there are four tests. Below, test one will be run and nothing will be cleaned up so that the files and directories in `test/test-0` can be inspected.
 ```
-pgbackrest/test/test.pl --vm-out --module=command --test=check --run=1 --no-cleanup
+pgbackrest/test/test.py --vm-out --module=command --test=check --run=1 --no-cleanup
 ```
 
 ### Understanding Test Output
@@ -616,7 +626,7 @@ repo-test-type:
 - `command` - list each command for which the option is valid. If a command is not listed, then the option is not valid for the command and an error will be thrown if it is attempted to be used for that command. In this case the valid commands are `backup` and `restore`.
 - `command-role` - defines the processes for which the option is valid. `main` indicates the option will be used by the main process and not be passed on to other local/remote processes.
 
-When `test.pl` is run the `config.auto.h` file will be generated to contain the constants used for options in the code. For the C enums, any dashes in the option name will be removed, camel-cased and prefixed with `cfgOpt`, e.g. `repo-path` becomes `cfgOptRepoPath`.
+When `test.py` is run the `config.auto.h` file will be generated to contain the constants used for options in the code. For the C enums, any dashes in the option name will be removed, camel-cased and prefixed with `cfgOpt`, e.g. `repo-path` becomes `cfgOptRepoPath`.
 
 ### help.xml
 
@@ -638,12 +648,12 @@ To add an option, add the following to the `<option-list>` section; if it does n
 
 It is important to run the `help` command unit test after adding an option in case a change is required:
 ```
-pgbackrest/test/test.pl --module=command --test=help --vm-out
+pgbackrest/test/test.py --module=command --test=help --vm-out
 ```
 
 To verify the `help` command output, build the pgBackRest executable:
 ```
-pgbackrest/test/test.pl --build-only
+pgbackrest/test/test.py --build-only
 ```
 
 Use the pgBackRest executable to test the help output:

doc/doc.pl

@@ -22,11 +22,10 @@ use Storable;
 use lib dirname(abs_path($0)) . '/lib';
 use lib dirname(dirname(abs_path($0))) . '/lib';
 use lib dirname(dirname(abs_path($0))) . '/build/lib';
-use lib dirname(dirname(abs_path($0))) . '/test/lib';
 
-use pgBackRestTest::Common::ExecuteTest;
-use pgBackRestTest::Common::Storage;
-use pgBackRestTest::Common::StoragePosix;
+use pgBackRestDoc::Common::Execute;
+use pgBackRestDoc::Common::Storage;
+use pgBackRestDoc::Common::StoragePosix;
 
 use pgBackRestDoc::Common::Doc;
 use pgBackRestDoc::Common::DocContainer;
@@ -190,8 +189,8 @@ eval
     # Get the base path
     my $strBasePath = abs_path(dirname($0));
 
-    my $oStorageDoc = new pgBackRestTest::Common::Storage(
-        $strBasePath, new pgBackRestTest::Common::StoragePosix({bFileSync => false, bPathSync => false}));
+    my $oStorageDoc = new pgBackRestDoc::Common::Storage(
+        $strBasePath, new pgBackRestDoc::Common::StoragePosix({bFileSync => false, bPathSync => false}));
 
     if (!defined($strDocPath))
     {

doc/lib/pgBackRestDoc/Common/DocContainer.pm

@@ -15,7 +15,7 @@ use Digest::SHA qw(sha1_hex);
 use Exporter qw(import);
     our @EXPORT = qw();
 
-use pgBackRestTest::Common::ExecuteTest;
+use pgBackRestDoc::Common::Execute;
 
 use pgBackRestDoc::Common::Log;
 use pgBackRestDoc::ProjectInfo;
@@ -55,7 +55,7 @@ sub dockerBuildCached
     {
         &log(INFO, "Checking cache ${strPullTag} ...");
 
-        my $oPull = new pgBackRestTest::Common::ExecuteTest(
+        my $oPull = new pgBackRestDoc::Common::Execute(
             "docker pull ${strPullTag}", {bSuppressStdErr => true, bSuppressError => true});
         $oPull->begin();
 
@@ -81,7 +81,7 @@ sub dockerBuildCached
         my $strPushTag = defined($strForkCacheTag) ? $strForkCacheTag : $strCacheTag;
         executeTest("docker tag ${strImage} ${strPushTag}", {bSuppressError => true});
 
-        my $oPush = new pgBackRestTest::Common::ExecuteTest(
+        my $oPush = new pgBackRestDoc::Common::Execute(
             "docker push ${strPushTag}", {bSuppressStdErr => true, bSuppressError => true});
         $oPush->begin();
 

doc/lib/pgBackRestDoc/Common/DocExecute.pm

@@ -15,7 +15,7 @@ use Exporter qw(import);
 use File::Basename qw(dirname);
 use Storable qw(dclone);
 
-use pgBackRestTest::Common::ExecuteTest;
+use pgBackRestDoc::Common::Execute;
 
 use pgBackRestDoc::Common::DocManifest;
 use pgBackRestDoc::Common::Exception;

doc/lib/pgBackRestDoc/Common/Execute.pm

@@ -1,7 +1,7 @@
 ####################################################################################################################################
-# ExecuteTest.pm - Module to execute external commands
+# Execute.pm - Module to execute external commands
 ####################################################################################################################################
-package pgBackRestTest::Common::ExecuteTest;
+package pgBackRestDoc::Common::Execute;
 
 ####################################################################################################################################
 # Perl includes
@@ -19,9 +19,9 @@ use Symbol 'gensym';
 
 use pgBackRestDoc::Common::Log;
 
-use pgBackRestTest::Common::Io::Handle;
-use pgBackRestTest::Common::Io::Buffered;
-use pgBackRestTest::Common::Wait;
+use pgBackRestDoc::Common::Io::Handle;
+use pgBackRestDoc::Common::Io::Buffered;
+use pgBackRestDoc::Common::Wait;
 
 ####################################################################################################################################
 # new
@@ -57,11 +57,9 @@ sub new
     $self->{bSuppressError} = defined($self->{bSuppressError}) ? $self->{bSuppressError} : false;
     $self->{bSuppressStdErr} = defined($self->{bSuppressStdErr}) ? $self->{bSuppressStdErr} : false;
     $self->{bOutLogOnError} = defined($self->{bOutLogOnError}) ? $self->{bOutLogOnError} : true;
-    $self->{bShowOutput} = defined($self->{bShowOutput}) ? $self->{bShowOutput} : false;
     $self->{bShowOutputAsync} = defined($self->{bShowOutputAsync}) ? $self->{bShowOutputAsync} : false;
     $self->{iExpectedExitStatus} = defined($self->{iExpectedExitStatus}) ? $self->{iExpectedExitStatus} : 0;
     $self->{iRetrySeconds} = defined($self->{iRetrySeconds}) ? $self->{iRetrySeconds} : undef;
-    $self->{bLogOutput} = defined($self->{bLogOutput}) ? $self->{bLogOutput} : true;
 
     # Return from function and log return values if any
     return logDebugReturn
@@ -84,16 +82,6 @@ sub begin
     $self->{strErrorLog} = '';
     $self->{strOutLog} = '';
 
-    if (defined($self->{oTestLog}))
-    {
-        if (defined($self->{strComment}))
-        {
-            $self->{strFullLog} = $self->{oTestLog}->regExpAll($self->{strComment}) . "\n";
-        }
-
-        $self->{strFullLog} .= '> ' . $self->{oTestLog}->regExpAll($self->{strCommand}) . "\n" . ('-' x '132') . "\n";
-    }
-
     &log(DETAIL, "executing command: $self->{strCommand}");
 
     # Execute the command
@@ -102,11 +90,11 @@ sub begin
     $self->{pId} = open3(undef, $self->{hOut}, $self->{hError}, $self->{strCommand});
 
     # Create buffered read object
-    $self->{oIo} = new pgBackRestTest::Common::Io::Buffered(new pgBackRestTest::Common::Io::Handle('exec test', $self->{hOut}), 0, 65536);
+    $self->{oIo} = new pgBackRestDoc::Common::Io::Buffered(new pgBackRestDoc::Common::Io::Handle('exec test', $self->{hOut}), 0, 65536);
 
     # Create buffered error object
-    $self->{oIoError} = new pgBackRestTest::Common::Io::Buffered(
-        new pgBackRestTest::Common::Io::Handle('exec test', $self->{hError}), 0, 65536);
+    $self->{oIoError} = new pgBackRestDoc::Common::Io::Buffered(
+        new pgBackRestDoc::Common::Io::Handle('exec test', $self->{hError}), 0, 65536);
 
     # Record start time and set process timeout
     $self->{iProcessTimeout} = 300;
@@ -251,11 +239,6 @@ sub endRetry
         confess &log(ERROR, "STDOUT:\n$self->{strOutLog}\n\noutput found on STDERR:\n$self->{strErrorLog}");
     }
 
-    if ($self->{bShowOutput})
-    {
-        print "output:\n$self->{strOutLog}\n";
-    }
-
     # Return from function and log return values if any
     return logDebugReturn
     (
@@ -331,7 +314,7 @@ sub executeTest
     my $strCommand = shift;
     my $oParam = shift;
 
-    my $oExec = new pgBackRestTest::Common::ExecuteTest($strCommand, $oParam);
+    my $oExec = new pgBackRestDoc::Common::Execute($strCommand, $oParam);
     $oExec->begin();
     $oExec->end();
 

doc/lib/pgBackRestDoc/Common/Host.pm

@@ -17,7 +17,7 @@ use Exporter qw(import);
 use pgBackRestDoc::Common::Log;
 use pgBackRestDoc::Common::String;
 
-use pgBackRestTest::Common::ExecuteTest;
+use pgBackRestDoc::Common::Execute;
 
 ####################################################################################################################################
 # new
@@ -137,7 +137,7 @@ sub execute
 
     $strCommand =~ s/'/'\\''/g;
 
-    my $oExec = new pgBackRestTest::Common::ExecuteTest(
+    my $oExec = new pgBackRestDoc::Common::Execute(
         "docker exec -u ${strUser} $self->{strContainer}" .
         ($bBashWrap ? " bash" . ($bLoadEnv ? ' -l' : '') . " -c '${strCommand}'" : " ${strCommand}"), $oParam);
 

doc/lib/pgBackRestDoc/Common/HostGroup.pm

@@ -17,7 +17,7 @@ use Exporter qw(import);
 use pgBackRestDoc::Common::Log;
 use pgBackRestDoc::Common::String;
 
-use pgBackRestTest::Common::ExecuteTest;
+use pgBackRestDoc::Common::Execute;
 
 ####################################################################################################################################
 # Global host group variable

doc/lib/pgBackRestDoc/Common/Io/Base.pm

@@ -1,7 +1,7 @@
 ####################################################################################################################################
 # Base IO Module
 ####################################################################################################################################
-package pgBackRestTest::Common::Io::Base;
+package pgBackRestDoc::Common::Io::Base;
 
 use strict;
 use warnings FATAL => qw(all);

doc/lib/pgBackRestDoc/Common/Io/Buffered.pm

@@ -1,8 +1,8 @@
 ####################################################################################################################################
 # Buffered Handle IO
 ####################################################################################################################################
-package pgBackRestTest::Common::Io::Buffered;
-use parent 'pgBackRestTest::Common::Io::Filter';
+package pgBackRestDoc::Common::Io::Buffered;
+use parent 'pgBackRestDoc::Common::Io::Filter';
 
 use strict;
 use warnings FATAL => qw(all);
@@ -17,9 +17,9 @@ use Time::HiRes qw(gettimeofday);
 use pgBackRestDoc::Common::Exception;
 use pgBackRestDoc::Common::Log;
 
-use pgBackRestTest::Common::Io::Base;
-use pgBackRestTest::Common::Io::Handle;
-use pgBackRestTest::Common::Wait;
+use pgBackRestDoc::Common::Io::Base;
+use pgBackRestDoc::Common::Io::Handle;
+use pgBackRestDoc::Common::Wait;
 
 ####################################################################################################################################
 # Package name constant
@@ -232,18 +232,6 @@ sub readLine
     return $strLine;
 }
 
-####################################################################################################################################
-# writeLine - write a string and \n terminate it
-####################################################################################################################################
-sub writeLine
-{
-    my $self = shift;
-    my $strBuffer = shift;
-
-    $strBuffer .= "\n";
-    return $self->parent()->write(\$strBuffer);
-}
-
 ####################################################################################################################################
 # Getters/Setters
 ####################################################################################################################################

doc/lib/pgBackRestDoc/Common/Io/Filter.pm

@@ -1,7 +1,7 @@
 ####################################################################################################################################
 # Base Filter Module
 ####################################################################################################################################
-package pgBackRestTest::Common::Io::Filter;
+package pgBackRestDoc::Common::Io::Filter;
 
 use strict;
 use warnings FATAL => qw(all);

doc/lib/pgBackRestDoc/Common/Io/Handle.pm

@@ -1,8 +1,8 @@
 ####################################################################################################################################
 # Basic Handle IO
 ####################################################################################################################################
-package pgBackRestTest::Common::Io::Handle;
-use parent 'pgBackRestTest::Common::Io::Base';
+package pgBackRestDoc::Common::Io::Handle;
+use parent 'pgBackRestDoc::Common::Io::Base';
 
 use strict;
 use warnings FATAL => qw(all);

doc/lib/pgBackRestDoc/Common/Storage.pm

@@ -1,8 +1,8 @@
 ####################################################################################################################################
 # Implements storage functionality using drivers.
 ####################################################################################################################################
-package pgBackRestTest::Common::Storage;
-use parent 'pgBackRestTest::Common::StorageBase';
+package pgBackRestDoc::Common::Storage;
+use parent 'pgBackRestDoc::Common::StorageBase';
 
 use strict;
 use warnings FATAL => qw(all);
@@ -15,7 +15,7 @@ use pgBackRestDoc::Common::Exception;
 use pgBackRestDoc::Common::Log;
 use pgBackRestDoc::Common::String;
 
-use pgBackRestTest::Common::StorageBase;
+use pgBackRestDoc::Common::StorageBase;
 
 ####################################################################################################################################
 # new
@@ -70,85 +70,6 @@ sub new
     );
 }
 
-####################################################################################################################################
-# Copy a file. If special encryption settings are required, then the file objects from openRead/openWrite must be passed instead of
-# file names.
-####################################################################################################################################
-sub copy
-{
-    my $self = shift;
-
-    # Assign function parameters, defaults, and log debug info
-    my
-    (
-        $strOperation,
-        $xSourceFile,
-        $xDestinationFile,
-        $bSourceOpen,
-    ) =
-        logDebugParam
-        (
-            __PACKAGE__ . '->copy', \@_,
-            {name => 'xSourceFile', required => false},
-            {name => 'xDestinationFile'},
-            {name => 'bSourceOpen', optional => true, default => false},
-        );
-
-    # Is source/destination an IO object or a file expression?
-    my $oSourceFileIo = defined($xSourceFile) ? (ref($xSourceFile) ? $xSourceFile : $self->openRead($xSourceFile)) : undef;
-
-    # Does the source file exist?
-    my $bResult = false;
-
-    # Copy if the source file exists
-    if (defined($oSourceFileIo))
-    {
-        my $oDestinationFileIo = ref($xDestinationFile) ? $xDestinationFile : $self->openWrite($xDestinationFile);
-
-        # Use C copy if source and destination are C objects
-        if (defined($oSourceFileIo->{oStorageCRead}) && defined($oDestinationFileIo->{oStorageCWrite}))
-        {
-            $bResult = $self->{oStorageC}->copy(
-                $oSourceFileIo->{oStorageCRead}, $oDestinationFileIo->{oStorageCWrite}) ? true : false;
-        }
-        else
-        {
-            # Open the source file if it is a C object
-            $bResult = defined($oSourceFileIo->{oStorageCRead}) ? ($bSourceOpen || $oSourceFileIo->open()) : true;
-
-            if ($bResult)
-            {
-                # Open the destination file if it is a C object
-                if (defined($oDestinationFileIo->{oStorageCWrite}))
-                {
-                    $oDestinationFileIo->open();
-                }
-
-                # Copy the data
-                do
-                {
-                    # Read data
-                    my $tBuffer = '';
-
-                    $oSourceFileIo->read(\$tBuffer, $self->{lBufferMax});
-                    $oDestinationFileIo->write(\$tBuffer);
-                }
-                while (!$oSourceFileIo->eof());
-
-                # Close files
-                $oSourceFileIo->close();
-                $oDestinationFileIo->close();
-            }
-        }
-    }
-
-    return logDebugReturn
-    (
-        $strOperation,
-        {name => 'bResult', value => $bResult, trace => true},
-    );
-}
-
 ####################################################################################################################################
 # exists - check if file exists
 ####################################################################################################################################
@@ -235,38 +156,6 @@ sub get
     );
 }
 
-####################################################################################################################################
-# info - get information for path/file
-####################################################################################################################################
-sub info
-{
-    my $self = shift;
-
-    # Assign function parameters, defaults, and log debug info
-    my
-    (
-        $strOperation,
-        $strPathFileExp,
-        $bIgnoreMissing,
-    ) =
-        logDebugParam
-        (
-            __PACKAGE__ . '::fileStat', \@_,
-            {name => 'strPathFileExp'},
-            {name => 'bIgnoreMissing', optional => true, default => false},
-        );
-
-    # Stat the path/file
-    my $oInfo = $self->driver()->info($self->pathGet($strPathFileExp), {bIgnoreMissing => $bIgnoreMissing});
-
-    # Return from function and log return values if any
-    return logDebugReturn
-    (
-        $strOperation,
-        {name => 'oInfo', value => $oInfo, trace => true}
-    );
-}
-
 ####################################################################################################################################
 # list - list all files/paths in path
 ####################################################################################################################################
@@ -321,37 +210,6 @@ sub list
     );
 }
 
-####################################################################################################################################
-# manifest - build path/file/link manifest starting with base path and including all subpaths
-####################################################################################################################################
-sub manifest
-{
-    my $self = shift;
-
-    # Assign function parameters, defaults, and log debug info
-    my
-    (
-        $strOperation,
-        $strPathExp,
-        $strFilter,
-    ) =
-        logDebugParam
-        (
-            __PACKAGE__ . '->manifest', \@_,
-            {name => 'strPathExp'},
-            {name => 'strFilter', optional => true, trace => true},
-        );
-
-    my $hManifest = $self->driver()->manifest($self->pathGet($strPathExp), {strFilter => $strFilter});
-
-    # Return from function and log return values if any
-    return logDebugReturn
-    (
-        $strOperation,
-        {name => 'hManifest', value => $hManifest, trace => true}
-    );
-}
-
 ####################################################################################################################################
 # move - move path/file
 ####################################################################################################################################
@@ -431,8 +289,6 @@ sub openWrite
         $strOperation,
         $xFileExp,
         $strMode,
-        $strUser,
-        $strGroup,
         $lTimestamp,
         $bAtomic,
         $bPathCreate,
@@ -442,8 +298,6 @@ sub openWrite
             __PACKAGE__ . '->openWrite', \@_,
             {name => 'xFileExp'},
             {name => 'strMode', optional => true, default => $self->{strDefaultFileMode}},
-            {name => 'strUser', optional => true},
-            {name => 'strGroup', optional => true},
             {name => 'lTimestamp', optional => true},
             {name => 'bAtomic', optional => true, default => false},
             {name => 'bPathCreate', optional => true, default => false},
@@ -451,8 +305,7 @@ sub openWrite
 
     # Open the file
     my $oFileIo = $self->driver()->openWrite($self->pathGet($xFileExp),
-        {strMode => $strMode, strUser => $strUser, strGroup => $strGroup, lTimestamp => $lTimestamp, bPathCreate => $bPathCreate,
-            bAtomic => $bAtomic});
+        {strMode => $strMode, lTimestamp => $lTimestamp, bPathCreate => $bPathCreate, bAtomic => $bAtomic});
 
     # Return from function and log return values if any
     return logDebugReturn
@@ -462,39 +315,6 @@ sub openWrite
     );
 }
 
-####################################################################################################################################
-# owner - change ownership of path/file
-####################################################################################################################################
-sub owner
-{
-    my $self = shift;
-
-    # Assign function parameters, defaults, and log debug info
-    my
-    (
-        $strOperation,
-        $strPathFileExp,
-        $strUser,
-        $strGroup
-    ) =
-        logDebugParam
-        (
-            __PACKAGE__ . '->owner', \@_,
-            {name => 'strPathFileExp'},
-            {name => 'strUser', required => false},
-            {name => 'strGroup', required => false}
-        );
-
-    # Set ownership
-    $self->driver()->owner($self->pathGet($strPathFileExp), {strUser => $strUser, strGroup => $strGroup});
-
-    # Return from function and log return values if any
-    return logDebugReturn
-    (
-        $strOperation
-    );
-}
-
 ####################################################################################################################################
 # pathCreate - create path
 ####################################################################################################################################
@@ -531,36 +351,6 @@ sub pathCreate
     );
 }
 
-####################################################################################################################################
-# pathExists - check if path exists
-####################################################################################################################################
-sub pathExists
-{
-    my $self = shift;
-
-    # Assign function parameters, defaults, and log debug info
-    my
-    (
-        $strOperation,
-        $strPathExp,
-    ) =
-        logDebugParam
-        (
-            __PACKAGE__ . '->pathExists', \@_,
-            {name => 'strPathExp'},
-        );
-
-    # Check exists
-    my $bExists = $self->driver()->pathExists($self->pathGet($strPathExp));
-
-    # Return from function and log return values if any
-    return logDebugReturn
-    (
-        $strOperation,
-        {name => 'bExists', value => $bExists}
-    );
-}
-
 ####################################################################################################################################
 # pathGet - resolve a path expression into an absolute path
 ####################################################################################################################################
@@ -634,31 +424,6 @@ sub pathGet
     );
 }
 
-####################################################################################################################################
-# Sync path so newly added file entries are not lost
-####################################################################################################################################
-sub pathSync
-{
-    my $self = shift;
-
-    # Assign function parameters, defaults, and log debug info
-    my
-    (
-        $strOperation,
-        $strPathExp,
-    ) =
-        logDebugParam
-        (
-            __PACKAGE__ . '->pathSync', \@_,
-            {name => 'strPathExp'},
-        );
-
-    $self->driver()->pathSync($self->pathGet($strPathExp));
-
-    # Return from function and log return values if any
-    return logDebugReturn($strOperation);
-}
-
 ####################################################################################################################################
 # put - writes a buffer out to storage all at once
 ####################################################################################################################################
@@ -761,6 +526,5 @@ sub remove
 ####################################################################################################################################
 sub pathBase {shift->{strPathBase}}
 sub driver {shift->{oDriver}}
-sub type {shift->{oDriver}->type()}
 
 1;

doc/lib/pgBackRestDoc/Common/StorageBase.pm

@@ -1,7 +1,7 @@
 ####################################################################################################################################
 # Base Storage Module
 ####################################################################################################################################
-package pgBackRestTest::Common::StorageBase;
+package pgBackRestDoc::Common::StorageBase;
 
 use strict;
 use warnings FATAL => qw(all);
@@ -12,13 +12,7 @@ use Exporter qw(import);
     our @EXPORT = qw();
 use pgBackRestDoc::Common::Log;
 
-use pgBackRestTest::Common::Io::Base;
-
-####################################################################################################################################
-# Storage constants
-####################################################################################################################################
-use constant STORAGE_POSIX                                          => 'posix';
-    push @EXPORT, qw(STORAGE_POSIX);
+use pgBackRestDoc::Common::Io::Base;
 
 ####################################################################################################################################
 # new

doc/lib/pgBackRestDoc/Common/StoragePosix.pm

@@ -0,0 +1,443 @@
+####################################################################################################################################
+# Posix Storage
+#
+# Implements storage functions for Posix-compliant file systems.
+####################################################################################################################################
+package pgBackRestDoc::Common::StoragePosix;
+
+use strict;
+use warnings FATAL => qw(all);
+use Carp qw(confess);
+use English '-no_match_vars';
+
+use Exporter qw(import);
+    our @EXPORT = qw();
+use File::Basename qw(basename dirname);
+use Fcntl qw(:mode);
+use File::stat qw{lstat};
+
+use pgBackRestDoc::Common::Exception;
+use pgBackRestDoc::Common::Log;
+
+use pgBackRestDoc::Common::StorageBase;
+use pgBackRestDoc::Common::StoragePosixRead;
+use pgBackRestDoc::Common::StoragePosixWrite;
+
+####################################################################################################################################
+# new
+####################################################################################################################################
+sub new
+{
+    my $class = shift;
+
+    # Create the class hash
+    my $self = {};
+    bless $self, $class;
+
+    # Assign function parameters, defaults, and log debug info
+    (
+        my $strOperation,
+        $self->{bFileSync},
+        $self->{bPathSync},
+    ) =
+        logDebugParam
+        (
+            __PACKAGE__ . '->new', \@_,
+            {name => 'bFileSync', optional => true, default => true},
+            {name => 'bPathSync', optional => true, default => true},
+        );
+
+    # Set default temp extension
+    $self->{strTempExtension} = 'tmp';
+
+    # Return from function and log return values if any
+    return logDebugReturn
+    (
+        $strOperation,
+        {name => 'self', value => $self, trace => true}
+    );
+}
+
+####################################################################################################################################
+# exists - check if a path or file exists
+####################################################################################################################################
+sub exists
+{
+    my $self = shift;
+
+    # Assign function parameters, defaults, and log debug info
+    my
+    (
+        $strOperation,
+        $strFile,
+    ) =
+        logDebugParam
+        (
+            __PACKAGE__ . '->exists', \@_,
+            {name => 'strFile', trace => true},
+        );
+
+    # Does the path/file exist?
+    my $bExists = true;
+    my $oStat = lstat($strFile);
+
+    # Use stat to test if file exists
+    if (defined($oStat))
+    {
+        # Check that it is actually a file
+        $bExists = !S_ISDIR($oStat->mode) ? true : false;
+    }
+    else
+    {
+        # If the error is not entry missing, then throw error
+        if (!$OS_ERROR{ENOENT})
+        {
+            logErrorResult(ERROR_FILE_EXISTS, "unable to test if file '${strFile}' exists", $OS_ERROR);
+        }
+
+        $bExists = false;
+    }
+
+    # Return from function and log return values if any
+    return logDebugReturn
+    (
+        $strOperation,
+        {name => 'bExists', value => $bExists, trace => true}
+    );
+}
+
+####################################################################################################################################
+# list - list all files/paths in path
+####################################################################################################################################
+sub list
+{
+    my $self = shift;
+
+    # Assign function parameters, defaults, and log debug info
+    my
+    (
+        $strOperation,
+        $strPath,
+        $bIgnoreMissing,
+    ) =
+        logDebugParam
+        (
+            __PACKAGE__ . '->list', \@_,
+            {name => 'strPath', trace => true},
+            {name => 'bIgnoreMissing', optional => true, default => false, trace => true},
+        );
+
+    # Working variables
+    my @stryFileList;
+    my $hPath;
+
+    # Attempt to open the path
+    if (opendir($hPath, $strPath))
+    {
+        @stryFileList = grep(!/^(\.|\.\.)$/m, readdir($hPath));
+        close($hPath);
+    }
+    # Else process errors
+    else
+    {
+        # Ignore the error if the file is missing and missing files should be ignored
+        if (!($OS_ERROR{ENOENT} && $bIgnoreMissing))
+        {
+            logErrorResult($OS_ERROR{ENOENT} ? ERROR_FILE_MISSING : ERROR_FILE_OPEN, "unable to read path '${strPath}'", $OS_ERROR);
+        }
+    }
+
+    # Return from function and log return values if any
+    return logDebugReturn
+    (
+        $strOperation,
+        {name => 'stryFileList', value => \@stryFileList, ref => true, trace => true}
+    );
+}
+
+####################################################################################################################################
+# move - move path/file
+####################################################################################################################################
+sub move
+{
+    my $self = shift;
+
+    # Assign function parameters, defaults, and log debug info
+    my
+    (
+        $strOperation,
+        $strSourceFile,
+        $strDestinationFile,
+        $bPathCreate,
+    ) =
+        logDebugParam
+        (
+            __PACKAGE__ . '->move', \@_,
+            {name => 'strSourceFile', trace => true},
+            {name => 'strDestinationFile', trace => true},
+            {name => 'bPathCreate', default => false, trace => true},
+        );
+
+    # Get source and destination paths
+    my $strSourcePathFile = dirname($strSourceFile);
+    my $strDestinationPathFile = dirname($strDestinationFile);
+
+    # Move the file
+    if (!rename($strSourceFile, $strDestinationFile))
+    {
+        my $strMessage = "unable to move '${strSourceFile}'";
+
+        # If something is missing determine if it is the source or destination
+        if ($OS_ERROR{ENOENT})
+        {
+            if (!$self->exists($strSourceFile))
+            {
+                logErrorResult(ERROR_FILE_MISSING, "${strMessage} because it is missing");
+            }
+
+            if ($bPathCreate)
+            {
+                # Attempt to create the path - ignore exists here in case another process creates it first
+                $self->pathCreate($strDestinationPathFile, {bCreateParent => true, bIgnoreExists => true});
+
+                # Try move again
+                $self->move($strSourceFile, $strDestinationFile);
+            }
+            else
+            {
+                logErrorResult(ERROR_PATH_MISSING, "${strMessage} to missing path '${strDestinationPathFile}'");
+            }
+        }
+        # Else raise the error
+        else
+        {
+            logErrorResult(ERROR_FILE_MOVE, "${strMessage} to '${strDestinationFile}'", $OS_ERROR);
+        }
+    }
+
+    # Return from function and log return values if any
+    return logDebugReturn($strOperation);
+}
+
+####################################################################################################################################
+# openRead - open file for reading
+####################################################################################################################################
+sub openRead
+{
+    my $self = shift;
+
+    # Assign function parameters, defaults, and log debug info
+    my
+    (
+        $strOperation,
+        $strFile,
+        $bIgnoreMissing,
+    ) =
+        logDebugParam
+    (
+        __PACKAGE__ . '->openRead', \@_,
+        {name => 'strFile', trace => true},
+        {name => 'bIgnoreMissing', optional => true, default => false, trace => true},
+    );
+
+    my $oFileIO = new pgBackRestDoc::Common::StoragePosixRead($self, $strFile, {bIgnoreMissing => $bIgnoreMissing});
+
+    # Return from function and log return values if any
+    return logDebugReturn
+    (
+        $strOperation,
+        {name => 'oFileIO', value => $oFileIO, trace => true},
+    );
+}
+
+####################################################################################################################################
+# openWrite - open file for writing
+####################################################################################################################################
+sub openWrite
+{
+    my $self = shift;
+
+    # Assign function parameters, defaults, and log debug info
+    my
+    (
+        $strOperation,
+        $strFile,
+        $strMode,
+        $lTimestamp,
+        $bPathCreate,
+        $bAtomic,
+    ) =
+        logDebugParam
+    (
+        __PACKAGE__ . '->openWrite', \@_,
+        {name => 'strFile', trace => true},
+        {name => 'strMode', optional => true, trace => true},
+        {name => 'lTimestamp', optional => true, trace => true},
+        {name => 'bPathCreate', optional => true, trace => true},
+        {name => 'bAtomic', optional => true, trace => true},
+    );
+
+    my $oFileIO = new pgBackRestDoc::Common::StoragePosixWrite(
+        $self, $strFile,
+        {strMode => $strMode, lTimestamp => $lTimestamp, bPathCreate => $bPathCreate, bAtomic => $bAtomic,
+            bSync => $self->{bFileSync}});
+
+    # Return from function and log return values if any
+    return logDebugReturn
+    (
+        $strOperation,
+        {name => 'oFileIO', value => $oFileIO, trace => true},
+    );
+}
+
+####################################################################################################################################
+# pathCreate - create path
+####################################################################################################################################
+sub pathCreate
+{
+    my $self = shift;
+
+    # Assign function parameters, defaults, and log debug info
+    my
+    (
+        $strOperation,
+        $strPath,
+        $strMode,
+        $bIgnoreExists,
+        $bCreateParent,
+    ) =
+        logDebugParam
+        (
+            __PACKAGE__ . '->pathCreate', \@_,
+            {name => 'strPath', trace => true},
+            {name => 'strMode', optional => true, default => '0750', trace => true},
+            {name => 'bIgnoreExists', optional => true, default => false, trace => true},
+            {name => 'bCreateParent', optional => true, default => false, trace => true},
+        );
+
+    # Attempt to create the directory
+    if (!mkdir($strPath, oct($strMode)))
+    {
+        my $strMessage = "unable to create path '${strPath}'";
+
+        # If parent path is missing
+        if ($OS_ERROR{ENOENT})
+        {
+            if (!$bCreateParent)
+            {
+                confess &log(ERROR, "${strMessage} because parent does not exist", ERROR_PATH_MISSING);
+            }
+
+            # Create parent path
+            $self->pathCreate(dirname($strPath), {strMode => $strMode, bIgnoreExists => true, bCreateParent => $bCreateParent});
+
+            # Create path
+            $self->pathCreate($strPath, {strMode => $strMode, bIgnoreExists => true});
+        }
+        # Else if path already exists
+        elsif ($OS_ERROR{EEXIST})
+        {
+            if (!$bIgnoreExists)
+            {
+                confess &log(ERROR, "${strMessage} because it already exists", ERROR_PATH_EXISTS);
+            }
+        }
+        else
+        {
+            logErrorResult(ERROR_PATH_CREATE, ${strMessage}, $OS_ERROR);
+        }
+    }
+
+    # Return from function and log return values if any
+    return logDebugReturn($strOperation);
+}
+
+####################################################################################################################################
+# remove - remove path/file
+####################################################################################################################################
+sub remove
+{
+    my $self = shift;
+
+    # Assign function parameters, defaults, and log debug info
+    my
+    (
+        $strOperation,
+        $xstryPathFile,
+        $bIgnoreMissing,
+        $bRecurse,
+    ) =
+        logDebugParam
+        (
+            __PACKAGE__ . '->remove', \@_,
+            {name => 'xstryPathFile', trace => true},
+            {name => 'bIgnoreMissing', optional => true, default => false, trace => true},
+            {name => 'bRecurse', optional => true, default => false, trace => true},
+        );
+
+    # Working variables
+    my $bRemoved = true;
+
+    # Remove a tree
+    if ($bRecurse)
+    {
+        my $oManifest = $self->manifest($xstryPathFile, {bIgnoreMissing => true});
+
+        # Iterate all files in the manifest
+        foreach my $strFile (sort({$b cmp $a} keys(%{$oManifest})))
+        {
+            # remove directory
+            if ($oManifest->{$strFile}{type} eq 'd')
+            {
+                my $xstryPathFileRemove = $strFile eq '.' ? $xstryPathFile : "${xstryPathFile}/${strFile}";
+
+                if (!rmdir($xstryPathFileRemove))
+                {
+                    # Throw error if this is not an ignored missing path
+                    if (!($OS_ERROR{ENOENT} && $bIgnoreMissing))
+                    {
+                        logErrorResult(ERROR_PATH_REMOVE, "unable to remove path '${strFile}'", $OS_ERROR);
+                    }
+                }
+            }
+            # Remove file
+            else
+            {
+                $self->remove("${xstryPathFile}/${strFile}", {bIgnoreMissing => true});
+            }
+        }
+    }
+    # Only remove the specified file
+    else
+    {
+        foreach my $strFile (ref($xstryPathFile) ? @{$xstryPathFile} : ($xstryPathFile))
+        {
+            if (unlink($strFile) != 1)
+            {
+                $bRemoved = false;
+
+                # Throw error if this is not an ignored missing file
+                if (!($OS_ERROR{ENOENT} && $bIgnoreMissing))
+                {
+                    logErrorResult(
+                        $OS_ERROR{ENOENT} ? ERROR_FILE_MISSING : ERROR_FILE_OPEN, "unable to remove file '${strFile}'", $OS_ERROR);
+                }
+            }
+        }
+    }
+
+    # Return from function and log return values if any
+    return logDebugReturn
+    (
+        $strOperation,
+        {name => 'bRemoved', value => $bRemoved, trace => true}
+    );
+}
+
+####################################################################################################################################
+# Getters/Setters
+####################################################################################################################################
+sub tempExtension {shift->{strTempExtension}}
+sub tempExtensionSet {my $self = shift; $self->{strTempExtension} = shift}
+
+1;

doc/lib/pgBackRestDoc/Common/StoragePosixRead.pm

@@ -1,8 +1,8 @@
 ####################################################################################################################################
 # Posix File Read
 ####################################################################################################################################
-package pgBackRestTest::Common::StoragePosixRead;
-use parent 'pgBackRestTest::Common::Io::Handle';
+package pgBackRestDoc::Common::StoragePosixRead;
+use parent 'pgBackRestDoc::Common::Io::Handle';
 
 use strict;
 use warnings FATAL => qw(all);

doc/lib/pgBackRestDoc/Common/StoragePosixWrite.pm

@@ -1,8 +1,8 @@
 ####################################################################################################################################
 # Posix File Write
 ####################################################################################################################################
-package pgBackRestTest::Common::StoragePosixWrite;
-use parent 'pgBackRestTest::Common::Io::Handle';
+package pgBackRestDoc::Common::StoragePosixWrite;
+use parent 'pgBackRestDoc::Common::Io::Handle';
 
 use strict;
 use warnings FATAL => qw(all);
@@ -15,8 +15,8 @@ use File::Basename qw(dirname);
 use pgBackRestDoc::Common::Exception;
 use pgBackRestDoc::Common::Log;
 
-use pgBackRestTest::Common::Io::Handle;
-use pgBackRestTest::Common::StorageBase;
+use pgBackRestDoc::Common::Io::Handle;
+use pgBackRestDoc::Common::StorageBase;
 
 ####################################################################################################################################
 # CONSTRUCTOR
@@ -32,8 +32,6 @@ sub new
         $oDriver,
         $strName,
         $strMode,
-        $strUser,
-        $strGroup,
         $lTimestamp,
         $bPathCreate,
         $bAtomic,
@@ -45,8 +43,6 @@ sub new
             {name => 'oDriver', trace => true},
             {name => 'strName', trace => true},
             {name => 'strMode', optional => true, trace => true},
-            {name => 'strUser', optional => true, trace => true},
-            {name => 'strGroup', optional => true, trace => true},
             {name => 'lTimestamp', optional => true, trace => true},
             {name => 'bPathCreate', optional => true, default => false, trace => true},
             {name => 'bAtomic', optional => true, default => false, trace => true},
@@ -61,8 +57,6 @@ sub new
     $self->{oDriver} = $oDriver;
     $self->{strName} = $strName;
     $self->{strMode} = $strMode;
-    $self->{strUser} = $strUser;
-    $self->{strGroup} = $strGroup;
     $self->{lTimestamp} = $lTimestamp;
     $self->{bPathCreate} = $bPathCreate;
     $self->{bAtomic} = $bAtomic;
@@ -114,9 +108,6 @@ sub open
     # Set file mode to binary
     binmode($self->{fhFile});
 
-    # Set the owner
-    $self->{oDriver}->owner($strFile, {strUser => $self->{strUser}, strGroup => $self->{strGroup}});
-
     # Set handle
     $self->handleWriteSet($self->{fhFile});
 

doc/lib/pgBackRestDoc/Common/Wait.pm

@@ -1,7 +1,7 @@
 ####################################################################################################################################
 # COMMON WAIT MODULE
 ####################################################################################################################################
-package pgBackRestTest::Common::Wait;
+package pgBackRestDoc::Common::Wait;
 
 use strict;
 use warnings FATAL => qw(all);

doc/lib/pgBackRestDoc/Custom/DocCustomRelease.pm

@@ -333,8 +333,8 @@ sub docGet
     my $strOperation = logDebugParam(__PACKAGE__ . '->docGet');
 
     # Load the git history
-    my $oStorageDoc = new pgBackRestTest::Common::Storage(
-        dirname(abs_path($0)), new pgBackRestTest::Common::StoragePosix({bFileSync => false, bPathSync => false}));
+    my $oStorageDoc = new pgBackRestDoc::Common::Storage(
+        dirname(abs_path($0)), new pgBackRestDoc::Common::StoragePosix({bFileSync => false, bPathSync => false}));
     my @hyGitLog = @{(JSON::PP->new()->allow_nonref())->decode(${$oStorageDoc->get("resource/git-history.cache")})};
 
     # Get renderer

doc/lib/pgBackRestDoc/Html/DocHtmlSite.pm

@@ -16,7 +16,7 @@ use File::Copy;
 use POSIX qw(strftime);
 use Storable qw(dclone);
 
-use pgBackRestTest::Common::ExecuteTest;
+use pgBackRestDoc::Common::Execute;
 
 use pgBackRestDoc::Common::DocManifest;
 use pgBackRestDoc::Common::Exception;

doc/lib/pgBackRestDoc/Markdown/DocMarkdown.pm

@@ -15,7 +15,7 @@ use File::Copy;
 use POSIX qw(strftime);
 use Storable qw(dclone);
 
-use pgBackRestTest::Common::ExecuteTest;
+use pgBackRestDoc::Common::Execute;
 
 use pgBackRestDoc::Common::DocManifest;
 use pgBackRestDoc::Common::Log;

doc/lib/pgBackRestDoc/ProjectInfo.pm

@@ -38,11 +38,11 @@ push @EXPORT, qw(PROJECT_VERSION);
 ####################################################################################################################################
 # Load project info from src/version.h
 ####################################################################################################################################
-require pgBackRestTest::Common::Storage;
-require pgBackRestTest::Common::StoragePosix;
+require pgBackRestDoc::Common::Storage;
+require pgBackRestDoc::Common::StoragePosix;
 
-my $strProjectInfo = ${new pgBackRestTest::Common::Storage(
-    dirname(dirname(abs_path($0))), new pgBackRestTest::Common::StoragePosix())->get('src/version.h')};
+my $strProjectInfo = ${new pgBackRestDoc::Common::Storage(
+    dirname(dirname(abs_path($0))), new pgBackRestDoc::Common::StoragePosix())->get('src/version.h')};
 
 foreach my $strLine (split("\n", $strProjectInfo))
 {

doc/release.pl

@@ -22,12 +22,10 @@ use Storable;
 use lib dirname($0) . '/lib';
 use lib dirname(dirname($0)) . '/build/lib';
 use lib dirname(dirname($0)) . '/lib';
-use lib dirname(dirname($0)) . '/test/lib';
 
-use pgBackRestTest::Common::ExecuteTest;
-use pgBackRestTest::Common::Storage;
-use pgBackRestTest::Common::StoragePosix;
-use pgBackRestTest::Common::VmTest;
+use pgBackRestDoc::Common::Execute;
+use pgBackRestDoc::Common::Storage;
+use pgBackRestDoc::Common::StoragePosix;
 
 use pgBackRestDoc::Common::Doc;
 use pgBackRestDoc::Common::DocManifest;
@@ -129,10 +127,10 @@ eval
     my $strDocPath = dirname(abs_path($0));
     my $strDocHtml = "${strDocPath}/output/html";
     my $strDocExe = "${strDocPath}/doc.pl";
-    my $strTestExe = dirname($strDocPath) . "/test/test.pl";
+    my $strTestExe = dirname($strDocPath) . "/test/test.py";
 
-    my $oStorageDoc = new pgBackRestTest::Common::Storage(
-        $strDocPath, new pgBackRestTest::Common::StoragePosix({bFileSync => false, bPathSync => false}));
+    my $oStorageDoc = new pgBackRestDoc::Common::Storage(
+        $strDocPath, new pgBackRestDoc::Common::StoragePosix({bFileSync => false, bPathSync => false}));
 
     # Determine if this is a dev release
     my $bDev = PROJECT_VERSION =~ /dev$/;
@@ -238,7 +236,7 @@ eval
         executeTest('docker rm -f $(docker ps -a -q)', {bSuppressError => true});
 
         # Generate deployment docs for RHEL
-        if (!defined($strVm) || $strVm eq VM_RH9)
+        if (!defined($strVm) || $strVm eq 'rh9')
         {
             &log(INFO, "Generate RHEL documentation");
 
@@ -246,7 +244,7 @@ eval
         }
 
         # Generate deployment docs for Debian
-        if (!defined($strVm) || $strVm eq VM_U24)
+        if (!defined($strVm) || $strVm eq 'u24')
         {
             &log(INFO, "Generate Debian/Ubuntu documentation");
 

doc/xml/coding.xml

@@ -7,10 +7,10 @@
     <section id="uncrustify">
         <title>Formatting with <proper>uncrustify</proper></title>
 
-        <p><backrest/> uses <proper>uncrustify</proper> to check/update the code formatting. If the <id>code-format</id> test fails in CI then reformat the code:</p>
+        <p><backrest/> uses <proper>uncrustify</proper> for <proper>C</proper> and <proper>black</proper> for <proper>Python</proper> to check/update the code formatting. If the <id>code-format</id> test fails in CI then reformat the code:</p>
 
         <code-block>
-            pgbackrest/test/test.pl --code-format
+            pgbackrest/test/test.py code-format
         </code-block>
 
         <p>Also review the standards in the following sections below. Some standards require further explanation and others are not enforced by <proper>uncrustify</proper>.</p>

doc/xml/contributing.xml

@@ -7,7 +7,7 @@
     <variable-list>
         <variable key="host-user-id" eval="y">use English; $UID</variable>
 
-        <!-- Options to pass to test.pl so log output is reproducible -->
+        <!-- Options to pass to test.py so log output is reproducible -->
         <variable key="test-cmd-extra">--no-log-timestamp</variable>
 
         <variable key="host-contrib-id">contrib</variable>
@@ -91,11 +91,10 @@
 
             <execute user="root" pre="y">
                 <exe-cmd>
-                    apt-get install rsync git devscripts build-essential valgrind autoconf
-                    autoconf-archive libssl-dev zlib1g-dev libxml2-dev libpq-dev pkg-config
-                    libxml-checker-perl libyaml-perl libdbd-pg-perl liblz4-dev liblz4-tool
-                    zstd libzstd-dev bzip2 libbz2-dev libyaml-dev ccache meson
-                    python3-yaml python3-coverage
+                    apt-get install rsync git build-essential valgrind libssl-dev zlib1g-dev
+                    libxml2-dev libpq-dev pkg-config libxml-checker-perl liblz4-dev liblz4-tool
+                    zstd libzstd-dev bzip2 libbz2-dev libyaml-dev ccache meson python3-yaml
+                    python3-coverage
                 </exe-cmd>
                 <exe-cmd-extra>-y 2>&amp;1</exe-cmd-extra>
             </execute>
@@ -103,7 +102,7 @@
             <!-- Clean so the tests give consistent output -->
             <execute show="n">
                 <exe-cmd>
-                    pgbackrest/test/test.pl --clean-only
+                    pgbackrest/test/test.py --clean-only
                 </exe-cmd>
             </execute>
         </execute-list>
@@ -144,7 +143,7 @@
             </execute>
         </execute-list>
 
-        <p>If using a RHEL-based system, the CPAN XML parser is required to run <file>test.pl</file> and <file>doc.pl</file>. Instructions for installing Docker and the XML parser can be found in the <file>README.md</file> file of the <backrest/> <link url="{[github-url-main]}/doc">doc</link> directory in the section <quote>The following is a sample RHEL 7 configuration that can be used for building the documentation</quote>.</p>
+        <p>If using a RHEL-based system, the CPAN XML parser is required to run <file>doc.pl</file>. Instructions for installing Docker and the XML parser can be found in the <file>README.md</file> file of the <backrest/> <link url="{[github-url-main]}/doc">doc</link> directory in the section <quote>The following is a sample RHEL 7 configuration that can be used for building the documentation</quote>.</p>
     </section>
 
     <!-- ======================================================================================================================= -->
@@ -303,13 +302,13 @@ myObjToLog(const MyObj *this)
 
         <p>A list of all possible test combinations can be viewed by running:</p>
         <code-block>
-            pgbackrest/test/test.pl --dry-run
+            pgbackrest/test/test.py --dry-run
         </code-block>
 
         <p>While some files are automatically generated during <code>make</code>, others are generated by running the test harness as follows:</p>
 
         <code-block>
-            pgbackrest/test/test.pl --gen-only
+            pgbackrest/test/test.py --gen-only
         </code-block>
 
         <p>Prior to any submission, the html version of the documentation should also be run and the output checked by viewing the generated html on the local file system under <code>pgbackrest/doc/output/html</code>. More details can be found in the <backrest/> <link url="{[github-url-main]}/doc/README.md">doc/README.md</link> file.</p>
@@ -337,7 +336,7 @@ myObjToLog(const MyObj *this)
             <p>For more options, run the test or documentation engine with the <setting>--help</setting> option:</p>
 
             <code-block>
-pgbackrest/test/test.pl --help
+pgbackrest/test/test.py --help
 pgbackrest/doc/doc.pl --help
             </code-block>
 
@@ -351,7 +350,7 @@ pgbackrest/doc/doc.pl --help
                     <title>List tests that don't require a container</title>
 
                     <execute output="y">
-                        <exe-cmd>pgbackrest/test/test.pl --dry-run</exe-cmd>
+                        <exe-cmd>pgbackrest/test/test.py --dry-run</exe-cmd>
                         <exe-cmd-extra>{[test-cmd-extra]}</exe-cmd-extra>
                         <exe-highlight>[0-9]+ tests selected|DRY RUN COMPLETED SUCCESSFULLY</exe-highlight>
                     </execute>
@@ -361,7 +360,7 @@ pgbackrest/doc/doc.pl --help
                     <title>Run a test</title>
 
                     <execute output="y">
-                        <exe-cmd>pgbackrest/test/test.pl --vm-out --module=common --test=wait</exe-cmd>
+                        <exe-cmd>pgbackrest/test/test.py --vm-out --module=common --test=wait</exe-cmd>
                         <exe-cmd-extra>{[test-cmd-extra]}</exe-cmd-extra>
                     </execute>
                 </execute-list>
@@ -372,7 +371,7 @@ pgbackrest/doc/doc.pl --help
                     <title>Run a module</title>
 
                     <execute output="y">
-                        <exe-cmd>pgbackrest/test/test.pl --module=postgres</exe-cmd>
+                        <exe-cmd>pgbackrest/test/test.py --module=postgres</exe-cmd>
                         <exe-cmd-extra>{[test-cmd-extra]}</exe-cmd-extra>
                     </execute>
                 </execute-list>
@@ -388,18 +387,18 @@ pgbackrest/doc/doc.pl --help
                     <title>Build a VM</title>
 
                     <execute output="y">
-                        <exe-cmd>pgbackrest/test/test.pl --vm-build --vm=u24</exe-cmd>
+                        <exe-cmd>pgbackrest/test/test.py --vm-build --vm=u24</exe-cmd>
                         <exe-cmd-extra>{[test-cmd-extra]}</exe-cmd-extra>
                     </execute>
                 </execute-list>
 
-                <admonition type="note">to build all the vms, just omit the <setting>--vm</setting> option above.</admonition>
+                <admonition type="note">to build all the vms, use <setting>--vm=all</setting> above.</admonition>
 
                 <execute-list host="{[host-contrib]}">
                     <title>Run a Specific Test Run</title>
 
                     <execute output="y">
-                        <exe-cmd>pgbackrest/test/test.pl {[dash]}-vm=u24 {[dash]}-module=postgres {[dash]}-test=interface {[dash]}-run=2</exe-cmd>
+                        <exe-cmd>pgbackrest/test/test.py {[dash]}-vm=u24 {[dash]}-module=postgres {[dash]}-test=interface {[dash]}-run=2</exe-cmd>
                         <exe-cmd-extra>{[test-cmd-extra]}</exe-cmd-extra>
                     </execute>
                 </execute-list>
@@ -593,7 +592,7 @@ TEST_ERROR(cmdCheck(), ConfigError, "primary database not found\nHINT: check ind
             <p>Unit tests are run for all files that are listed in <file>define.yaml</file> and a coverage report generated for each file listed under the tag <code>coverage:</code>. Note that some files are listed in multiple <code>coverage:</code> sections for a module; in this case, each test for the file being modified should be specified for the module in which the file exists (e.g. <code>--module=storage --test=posix --test=gcs</code>, etc.) or, alternatively, simply run the module without the <code>--test</code> option. It is recommended that a <code>--vm</code> be specified since running the same test for multiple vms is unnecessary for coverage. The following example would run the test set from the <b>define.yaml</b> section detailed above.</p>
 
             <code-block>
-pgbackrest/test/test.pl --vm-out --module=command --test=check --vm=u24
+pgbackrest/test/test.py --vm-out --module=command --test=check --vm=u24
             </code-block>
 
 <admonition type="note">Not all systems perform at the same speed, so if a test is timing out, try rerunning with another vm.</admonition>
@@ -607,7 +606,7 @@ pgbackrest/test/test.pl --vm-out --module=command --test=check --vm=u24
             <p>Sometimes it is useful to look at files that were generated during the test. The default for running any test is that, at the start/end of the test, the test harness will clean up all files and directories created. To override this behavior, a single test run must be specified and the option <code>--no-cleanup</code> provided. Again, continuing with the check command, from <b>define.yaml</b> above, there are four tests. Below, test one will be run and nothing will be cleaned up so that the files and directories in <path>test/test-0</path> can be inspected.</p>
 
             <code-block>
-pgbackrest/test/test.pl --vm-out --module=command --test=check --run=1 --no-cleanup
+pgbackrest/test/test.py --vm-out --module=command --test=check --run=1 --no-cleanup
             </code-block>
         </section>
 
@@ -720,7 +719,7 @@ run 8/1 ------------- L2285 no current backups
                 <list-item><id>command-role</id> - defines the processes for which the option is valid. <code>main</code> indicates the option will be used by the main process and not be passed on to other local/remote processes.</list-item>
             </list>
 
-            <p>When <file>test.pl</file> is run the <file>config.auto.h</file> file will be generated to contain the constants used for options in the code. For the C enums, any dashes in the option name will be removed, camel-cased and prefixed with <code>cfgOpt</code>, e.g. <code>repo-path</code> becomes <code>cfgOptRepoPath</code>.</p>
+            <p>When <file>test.py</file> is run the <file>config.auto.h</file> file will be generated to contain the constants used for options in the code. For the C enums, any dashes in the option name will be removed, camel-cased and prefixed with <code>cfgOpt</code>, e.g. <code>repo-path</code> becomes <code>cfgOptRepoPath</code>.</p>
         </section>
 
         <!-- =================================================================================================================== -->
@@ -750,13 +749,13 @@ run 8/1 ------------- L2285 no current backups
             <p>It is important to run the <cmd>help</cmd> command unit test after adding an option in case a change is required:</p>
 
             <code-block>
-pgbackrest/test/test.pl --module=command --test=help --vm-out
+pgbackrest/test/test.py --module=command --test=help --vm-out
             </code-block>
 
             <p>To verify the <cmd>help</cmd> command output, build the <backrest/> executable:</p>
 
             <code-block>
-pgbackrest/test/test.pl --build-only
+pgbackrest/test/test.py --build-only
             </code-block>
 
             <p>Use the <backrest/> executable to test the help output:</p>

test/ci.pl

@@ -1,255 +0,0 @@
-#!/usr/bin/perl
-####################################################################################################################################
-# CI Test Wrapper
-####################################################################################################################################
-
-####################################################################################################################################
-# Perl includes
-####################################################################################################################################
-use strict;
-use warnings FATAL => qw(all);
-use Carp qw(confess longmess);
-use English '-no_match_vars';
-
-# Convert die to confess to capture the stack trace
-$SIG{__DIE__} = sub { Carp::confess @_ };
-
-use File::Basename qw(dirname);
-use Getopt::Long qw(GetOptions);
-use Cwd qw(abs_path);
-use Pod::Usage qw(pod2usage);
-
-use lib dirname($0) . '/lib';
-use lib dirname(dirname($0)) . '/lib';
-use lib dirname(dirname($0)) . '/doc/lib';
-
-use pgBackRestDoc::Common::Exception;
-use pgBackRestDoc::Common::Log;
-use pgBackRestDoc::ProjectInfo;
-
-use pgBackRestTest::Common::ContainerTest;
-use pgBackRestTest::Common::ExecuteTest;
-use pgBackRestTest::Common::VmTest;
-
-####################################################################################################################################
-# Usage
-####################################################################################################################################
-
-=head1 NAME
-
-ci.pl - CI Test Wrapper
-
-=head1 SYNOPSIS
-
-test.pl [options] doc|test
-
- VM Options:
-   --vm                 docker container to build/test
-   --param              parameters to pass to test.pl
-   --sudo               test requires sudo
-   --no-tempfs          do not create tempfs mount
-
- General Options:
-   --help               display usage and exit
-=cut
-
-####################################################################################################################################
-# Command line parameters
-####################################################################################################################################
-my $strVm = "none";
-my $strVmArch;
-my @stryParam;
-my $bNoTempFs;
-my $bSudo;
-my $bHelp;
-
-GetOptions ('help' => \$bHelp,
-            'param=s@' => \@stryParam,
-            'no-tempfs' => \$bNoTempFs,
-            'sudo' => \$bSudo,
-            'vm=s' => \$strVm,
-            'vm-arch=s' => \$strVmArch)
-    or pod2usage(2);
-
-####################################################################################################################################
-# Begin/end functions to track timing
-####################################################################################################################################
-my $lProcessBegin;
-my $strProcessTitle;
-
-sub processBegin
-{
-    $strProcessTitle = shift;
-
-    &log(INFO, "Begin ${strProcessTitle}");
-    $lProcessBegin = time();
-}
-
-sub processExec
-{
-    my $strCommand = shift;
-    my $rhParam = shift;
-
-    &log(INFO, "    Exec ${strCommand}");
-    executeTest($strCommand, $rhParam);
-}
-
-sub processEnd
-{
-    &log(INFO, "    End ${strProcessTitle} (" . (time() - $lProcessBegin) . 's)');
-}
-
-####################################################################################################################################
-# Run in eval block to catch errors
-####################################################################################################################################
-eval
-{
-    # Display version and exit if requested
-    if ($bHelp)
-    {
-        syswrite(*STDOUT, "CI Test Wrapper\n");
-
-        syswrite(*STDOUT, "\n");
-        pod2usage();
-
-        exit 0;
-    }
-
-    if (@ARGV != 1)
-    {
-        syswrite(*STDOUT, "test|doc required\n\n");
-        pod2usage();
-    }
-
-    ################################################################################################################################
-    # Paths
-    ################################################################################################################################
-    my $strBackRestBase = dirname(dirname(abs_path($0)));
-    my $strReleaseExe = "${strBackRestBase}/doc/release.pl";
-    my $strTestExe = "${strBackRestBase}/test/test.pl";
-
-    logLevelSet(INFO, INFO, OFF);
-
-    processBegin('install common packages');
-    processExec('sudo apt-get -qq update', {bSuppressStdErr => true, bSuppressError => true});
-    processExec(
-        'sudo DEBIAN_FRONTEND=noninteractive apt-get install -y meson libxml-checker-perl libyaml-perl python3-yaml',
-        {bSuppressStdErr => true});
-    processEnd();
-
-    if (!$bNoTempFs)
-    {
-        processBegin('mount tmpfs');
-        processExec('mkdir -p -m 770 test');
-        processExec('sudo mount -t tmpfs -o size=2048m tmpfs test');
-        processExec('df -h test', {bShowOutputAsync => true});
-        processEnd();
-    }
-
-    ################################################################################################################################
-    # Build documentation
-    ################################################################################################################################
-    my $strUser = getpwuid($UID);
-
-    if ($ARGV[0] eq 'doc')
-    {
-        processBegin('remove sudo');
-        processExec("sudo rm /etc/sudoers.d/${strUser}");
-        processEnd();
-
-        processBegin('create link from home to repo for contributing doc');
-        processExec("ln -s ${strBackRestBase} \${HOME?}/" . PROJECT_EXE);
-        processEnd();
-
-        processBegin('release documentation');
-        processExec("${strReleaseExe} --build --no-gen --vm=${strVm}", {bShowOutputAsync => true, bOutLogOnError => false});
-        processEnd();
-    }
-
-    ################################################################################################################################
-    # Run test
-    ################################################################################################################################
-    elsif ($ARGV[0] eq 'test')
-    {
-        # Build list of packages that need to be installed
-        my $strPackage =
-            "gcc ccache git rsync zlib1g-dev libssl-dev libxml2-dev libpq-dev libyaml-dev pkg-config uncrustify" .
-            " libssh2-1-dev valgrind";
-
-        # Extra packages required when testing without containers
-        if ($strVm eq VM_NONE)
-        {
-            $strPackage .= " liblz4-dev liblz4-tool zstd libzstd-dev bzip2 libbz2-dev";
-        }
-        # Else packages needed for integration tests on containers
-        else
-        {
-            $strPackage .= " libdbd-pg-perl";
-        }
-
-        processBegin('/tmp/pgbackrest owned by root so tests cannot use it');
-        processExec('sudo mkdir -p /tmp/pgbackrest && sudo chown root:root /tmp/pgbackrest && sudo chmod 700 /tmp/pgbackrest');
-        processEnd();
-
-        processBegin('install test packages');
-        processExec(
-            "sudo DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -y ${strPackage}",
-            {bSuppressStdErr => true});
-        processEnd();
-
-        if (!$bSudo)
-        {
-            processBegin('remove sudo');
-            processExec("sudo rm /etc/sudoers.d/${strUser}");
-            processEnd();
-        }
-
-        # Build the container
-        my $strVmArchParam = defined($strVmArch) ? " --vm-arch=${strVmArch}" : '';
-
-        if ($strVm ne VM_NONE)
-        {
-            processBegin("${strVm} build");
-            processExec(
-                "${strTestExe} --vm-build --vm=${strVm}${strVmArchParam}", {bShowOutputAsync => true, bOutLogOnError => false});
-            processEnd();
-        }
-
-        processBegin(($strVm eq VM_NONE ? "no container" : $strVm) . ' test');
-        processExec(
-            "${strTestExe} --gen-check --log-level-test-file=off --no-coverage-report --vm-max=2 --vm=${strVm}${strVmArchParam}" .
-                (@stryParam != 0 ? " --" . join(" --", @stryParam) : ''),
-            {bShowOutputAsync => true, bOutLogOnError => false});
-        processEnd();
-    }
-
-    ################################################################################################################################
-    # Catch error
-    ################################################################################################################################
-    else
-    {
-        confess &log(ERROR, 'invalid command ' . $ARGV[0]);
-    }
-
-    &log(INFO, "CI Complete");
-
-    # Exit with success
-    exit 0;
-}
-
-####################################################################################################################################
-# Check for errors
-####################################################################################################################################
-or do
-{
-    # If a backrest exception then return the code
-    exit $EVAL_ERROR->code() if (isException(\$EVAL_ERROR));
-
-    # Else output the unhandled error
-    print $EVAL_ERROR;
-    exit ERROR_UNHANDLED;
-};
-
-# It shouldn't be possible to get here
-&log(ASSERT, 'execution reached invalid location in ' . __FILE__ . ', line ' . __LINE__);
-exit ERROR_ASSERT;

test/ci.py

@@ -0,0 +1,203 @@
+#!/usr/bin/env python3
+"""pgBackRest CI Wrapper.
+
+Prepares a CI runner and then runs the tests or builds the documentation on it. A separate program from the test harness because it
+is about the runner rather than about the tests, and because it runs before the runner has what the harness needs.
+
+This is the only place that knows what a runner is missing, since a runner starts from a stock image rather than from one of the
+test images the tests themselves run in.
+
+All output, including errors, goes to stdout so the run reads in the order it happened."""
+
+####################################################################################################################################
+import argparse
+import os
+import signal
+import sys
+import time
+import traceback
+
+# Send everything written to stderr to stdout instead so the output is in the order it happened
+sys.stderr = sys.stdout
+
+# Do not cache bytecode. The harness runs from a copy of the repository that the linter then scans, so a __pycache__ written during
+# import would be reported as an unexpected binary file. This must be set before the harness modules are imported below.
+sys.dont_write_bytecode = True
+
+# The library lives beside this script. Insert it first so the harness modules are found before anything else on the path.
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), "lib"))
+
+from common.error import EXIT_ERROR, TestError  # noqa: E402
+from common.exec import exec_one  # noqa: E402
+from common.log import *  # noqa: E402
+from common.user import user_name  # noqa: E402
+from common.vm import *  # noqa: E402
+from config.config import PROJECT_EXE, project_version  # noqa: E402
+
+
+####################################################################################################################################
+def cfg_load(arg_list, path_repo):
+    """Parse the command line and initialize logging."""
+
+    parser = argparse.ArgumentParser(prog="ci.py", description="pgBackRest CI Wrapper")
+
+    parser.add_argument("--version", action="version", version="pgBackRest %s CI Wrapper" % project_version(path_repo))
+    parser.add_argument("target", choices=("doc", "test"), help="run the tests or build the documentation")
+    parser.add_argument("--vm", default=VM_NONE, metavar="VM", help="vm to run on")
+    parser.add_argument("--vm-arch", metavar="ARCH", help="vm architecture (defaults to the host architecture)")
+    parser.add_argument("--param", action="append", default=[], metavar="OPTION", help="option to pass to the test harness")
+    parser.add_argument("--sudo", action="store_true", help="leave passwordless sudo in place for the run")
+    parser.add_argument("--no-tempfs", dest="tempfs", action="store_false", help="do not mount a tmpfs for the test path")
+
+    config = parser.parse_args(arg_list)
+    config.repo_path = path_repo
+
+    log_init(INFO, True)
+
+    return config
+
+
+####################################################################################################################################
+def step(title, command_list, show_output=False):
+    """Run the commands that make up one step of the run and report how long the step took."""
+
+    time_begin = time.time()
+
+    log(INFO, "Begin %s" % title)
+
+    for command in command_list:
+        log(INFO, "    Exec %s" % command)
+        exec_one(command, show_output=show_output)
+
+    log(INFO, "    End %s (%us)" % (title, int(time.time() - time_begin)))
+
+
+####################################################################################################################################
+def sudo_remove():
+    """Remove passwordless sudo so a test cannot depend on having it.
+
+    A runner grants it to every user, which the test containers do not."""
+
+    step("remove sudo", ["sudo rm /etc/sudoers.d/%s" % user_name()])
+
+
+####################################################################################################################################
+def doc_build(config):
+    """Build the documentation."""
+
+    # The documentation is the only thing left that runs on Perl
+    step("install documentation packages", ["sudo DEBIAN_FRONTEND=noninteractive apt-get install -y libxml-checker-perl"])
+
+    sudo_remove()
+
+    step(
+        "create link from home to repo for contributing doc",
+        ["ln -s %s %s/%s" % (config.repo_path, os.environ["HOME"], PROJECT_EXE)],
+    )
+
+    step(
+        "release documentation",
+        ["%s/doc/release.pl --build --no-gen --vm=%s" % (config.repo_path, config.vm)],
+        show_output=True,
+    )
+
+
+####################################################################################################################################
+def test_run(config):
+    """Run the tests."""
+
+    vm_arch = "" if config.vm_arch is None else " --vm-arch=%s" % config.vm_arch
+
+    # Packages the tests need that the test images already have
+    package = "gcc ccache git rsync zlib1g-dev libssl-dev libxml2-dev libpq-dev libyaml-dev pkg-config libssh2-1-dev valgrind"
+
+    # Extra packages required when testing without containers
+    if config.vm == VM_NONE:
+        package += " liblz4-dev liblz4-tool zstd libzstd-dev bzip2 libbz2-dev"
+
+    step(
+        "/tmp/pgbackrest owned by root so tests cannot use it",
+        ["sudo mkdir -p /tmp/%s && sudo chown root:root /tmp/%s && sudo chmod 700 /tmp/%s" % ((PROJECT_EXE,) * 3)],
+    )
+
+    step(
+        "install test packages",
+        ["sudo DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -y %s" % package],
+    )
+
+    if not config.sudo:
+        sudo_remove()
+
+    # Build the containers
+    if config.vm != VM_NONE:
+        step(
+            "%s build" % config.vm,
+            ["%s/test/test.py --vm-build --vm=%s%s" % (config.repo_path, config.vm, vm_arch)],
+            show_output=True,
+        )
+
+    step(
+        ("no container" if config.vm == VM_NONE else config.vm) + " test",
+        [
+            "%s/test/test.py --vm-max=2 --vm=%s%s%s"
+            % (config.repo_path, config.vm, vm_arch, "".join(" --%s" % param for param in config.param))
+        ],
+        show_output=True,
+    )
+
+
+####################################################################################################################################
+def command_run(config):
+    """Prepare the runner and then run what was asked for on it."""
+
+    step(
+        "install common packages",
+        [
+            # A stale package index is not worth failing the run over since the install below will report what is missing
+            "sudo apt-get -qq update || true",
+            "sudo DEBIAN_FRONTEND=noninteractive apt-get install -y meson python3-yaml",
+        ],
+    )
+
+    # The test path is a tmpfs so the tests are not limited by the runner's disk, which is slow and shared
+    if config.tempfs:
+        step(
+            "mount tmpfs",
+            ["mkdir -p -m 770 test", "sudo mount -t tmpfs -o size=2048m tmpfs test", "df -h test"],
+            show_output=True,
+        )
+
+    if config.target == "doc":
+        doc_build(config)
+    else:
+        test_run(config)
+
+    log(INFO, "CI Complete")
+
+    return 0
+
+
+####################################################################################################################################
+def main():
+    """Main."""
+
+    # Die silently on SIGPIPE as C programs do, rather than raising when output is piped to a command that exits early
+    signal.signal(signal.SIGPIPE, signal.SIG_DFL)
+
+    path_repo = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
+    try:
+        return command_run(cfg_load(sys.argv[1:], path_repo))
+    except TestError as error:
+        log(ERROR, error)
+
+        return error.status
+    except Exception:
+        # An unexpected exception is a bug here rather than a problem with the run, so show the traceback
+        print(traceback.format_exc())
+
+        return EXIT_ERROR
+
+
+if __name__ == "__main__":
+    sys.exit(main())

test/define.yaml

@@ -1089,6 +1089,18 @@ unit:
         coverage:
           - test/common/render
 
+      # ----------------------------------------------------------------------------------------------------------------------------
+      - name: common/user
+
+        coverage:
+          - test/common/user
+
+      # ----------------------------------------------------------------------------------------------------------------------------
+      - name: common/vm
+
+        coverage:
+          - test/common/vm
+
       # ----------------------------------------------------------------------------------------------------------------------------
       - name: config/cli
 
@@ -1115,12 +1127,42 @@ unit:
           - test/command/lint/string_id
           - test/command/lint/lint
 
+      # ----------------------------------------------------------------------------------------------------------------------------
+      - name: command/code/format
+
+        coverage:
+          - test/command/code/format
+
       # ----------------------------------------------------------------------------------------------------------------------------
       - name: command/test/python
 
         coverage:
           - test/command/test/python
 
+      # ----------------------------------------------------------------------------------------------------------------------------
+      - name: command/test/list
+
+        coverage:
+          - test/command/test/list
+
+      # ----------------------------------------------------------------------------------------------------------------------------
+      - name: command/test/container
+
+        coverage:
+          - test/command/test/container
+
+      # ----------------------------------------------------------------------------------------------------------------------------
+      - name: command/test/job
+
+        coverage:
+          - test/command/test/job
+
+      # ----------------------------------------------------------------------------------------------------------------------------
+      - name: command/test/test
+
+        coverage:
+          - test/command/test/test
+
       # ----------------------------------------------------------------------------------------------------------------------------
       - name: command/coverage/report
 

test/lib/command/code/format.py

@@ -0,0 +1,106 @@
+"""Code Format Command.
+
+Formats the source to the project standards, or checks that it is already formatted, which is what CI runs. C is formatted with
+uncrustify and python with black, and the execute permissions are checked afterward since a stray execute bit is easy to introduce
+and hard to notice.
+
+Only version controlled files are formatted, which is what git reports here, so scratch work in the working tree is left alone."""
+
+####################################################################################################################################
+import os
+
+from common.error import TestError
+from common.exec import exec_one
+from common.log import *
+
+# Line length black formats python to, which is the line length the rest of the project uses
+_LINE_LENGTH = 132
+
+# Files that are allowed to be executable. Everything else that is version controlled must not be, since a stray execute bit is
+# usually the result of an editor or a file system that does not preserve modes.
+_FILE_EXECUTABLE_LIST = (
+    "doc/doc.pl",
+    "doc/release.pl",
+    "src/build/dist.sh",
+    "test/ci.py",
+    "test/smoke.py",
+    "test/test.py",
+)
+
+
+####################################################################################################################################
+def _file_list(path_repo):
+    """The version controlled files, which are the ones that get formatted.
+
+    Only regular files are returned. A file that is in the index but no longer in the working tree has nothing to format, and git
+    reports a symlink to a directory as a single entry rather than descending into it."""
+
+    file_list = exec_one("git -C %s ls-files -c --others --exclude-standard" % path_repo).splitlines()
+
+    return [file for file in file_list if os.path.isfile(os.path.join(path_repo, file))]
+
+
+####################################################################################################################################
+def _format_c(path_repo, file_list, check):
+    """Format the C with uncrustify."""
+
+    command = "uncrustify -c %s/test/uncrustify.cfg%s" % (path_repo, " --check" if check else " --replace --no-backup")
+
+    for file in file_list:
+        # Skip anything that is not C
+        if not file.endswith((".c", ".h", ".c.inc")):
+            continue
+
+        # Skip the test template, which does not format correctly, and vendorized and generated files
+        if file == "test/src/test.c" or file.endswith((".vendor.h", ".vendor.c.inc", ".auto.c.inc")):
+            continue
+
+        command += " %s/%s" % (path_repo, file)
+
+    exec_one(command)
+
+
+####################################################################################################################################
+def _format_python(path_repo, file_list, check):
+    """Format the python with black."""
+
+    command = "black --line-length=%u%s" % (_LINE_LENGTH, " --check --diff" if check else "")
+
+    for file in file_list:
+        if file.endswith(".py"):
+            command += " %s/%s" % (path_repo, file)
+
+    exec_one(command)
+
+
+####################################################################################################################################
+def _permission_check(path_repo, file_list):
+    """Check that nothing is executable that is not meant to be, which formatting can munge."""
+
+    for file in file_list:
+        mode = os.stat(os.path.join(path_repo, file)).st_mode & 0o777
+
+        if mode & 0o111 and file not in _FILE_EXECUTABLE_LIST:
+            raise TestError("expected mode '%04o' for '%s' but found '%04o'" % (mode & 0o666, file, mode))
+
+
+####################################################################################################################################
+def cmd_code_format(config):
+    """Format the source, or check that it is already formatted."""
+
+    path_repo = config.repo_path
+    file_list = _file_list(path_repo)
+
+    log(INFO, "code format" + (" check" if config.check else ""))
+
+    try:
+        _format_c(path_repo, file_list, config.check)
+        _format_python(path_repo, file_list, config.check)
+    except TestError as error:
+        if not config.check:
+            raise
+
+        # The check is what CI runs, so it says how to fix what it found rather than only what it found
+        raise TestError("%s\nHINT: run 'test.py code-format' to format the code" % error)
+
+    _permission_check(path_repo, file_list)

test/lib/command/coverage/coverage.py

@@ -9,7 +9,7 @@ import os
 
 from command.test.define import TEST_LANG_C, TEST_LANG_PYTHON, test_def_find, test_def_parse
 from common.error import check
-from common.log import WARN, log
+from common.log import *
 from common.storage import file_read, file_write, path_list
 
 

test/lib/command/lint/ascii.py

@@ -5,7 +5,7 @@ bidirectional, or homoglyph characters. Intentional non-ASCII byte values must b
 the source."""
 
 ####################################################################################################################################
-from common.log import WARN, log
+from common.log import *
 
 # Bytes that may appear in source: printable 7-bit ASCII plus tab and linefeed
 _BYTE_ALLOW = bytes(range(0x20, 0x7F)) + b"\t\n"

test/lib/command/lint/lint.py

@@ -9,7 +9,7 @@ import os
 from command.lint.ascii import lint_ascii
 from command.lint.string_id import lint_str_id
 from common.error import TestError
-from common.log import WARN, log
+from common.log import *
 from common.storage import path_list_recurse
 
 # Files that are exempt from the content checks. A binary file is an unscannable place to hide content, so each entry must be a

test/lib/command/lint/string_id.py

@@ -7,7 +7,7 @@ can be used in a switch, which means a hand-edited string and value can drift ap
 import re
 
 from common.error import TestError
-from common.log import WARN, log
+from common.log import *
 from common.render import bld_str_id_seq
 
 # Macro invocations to check, e.g. STRID5("test", 0x2a7250) or STRID6S("test", 1, 0x1e2a7250)

test/lib/command/test/build.py

@@ -11,9 +11,10 @@ import os
 
 from command.test.define import TEST_TYPE_INTEGRATION, TEST_TYPE_PERFORMANCE, TEST_TYPE_UNIT
 from common.error import check
-from common.log import log_level_enum
+from common.log import *
 from common.render import bld_enum
 from common.storage import file_read, file_remove, file_write_differs, path_list, path_list_recurse
+from common.user import group_id, group_name, user_id, user_name
 
 # Comment block used to divide the generated meson.build, matching the width used in the rest of the project
 MESON_COMMENT_BLOCK = "#" * 132
@@ -435,10 +436,10 @@ class TestBuild:
             "{[C_TEST_CONTAINER]}": "true" if self.vm != "none" else "false",
             # User/group info
             "{[C_TEST_GROUP]}": group_name(),
-            "{[C_TEST_GROUP_ID]}": "%u" % os.getgid(),
+            "{[C_TEST_GROUP_ID]}": "%u" % group_id(),
             "{[C_TEST_USER]}": user_name(),
             "{[C_TEST_USER_LEN]}": "%u" % len(user_name()),
-            "{[C_TEST_USER_ID]}": "%u" % os.getuid(),
+            "{[C_TEST_USER_ID]}": "%u" % user_id(),
             # VM for integration testing
             "{[C_TEST_VM]}": self.vm_int,
             # PostgreSQL version for integration testing
@@ -478,37 +479,3 @@ class TestBuild:
                 continue
 
             file_remove(os.path.join(self.path_unit, name), error_on_missing=True)
-
-
-####################################################################################################################################
-# User and group the test runs as. Looked up once since the values are used several times and never change.
-_user_name = None
-_group_name = None
-
-
-####################################################################################################################################
-def user_name():
-    """User the test runs as, looked up once since it is used several times and never changes."""
-
-    global _user_name
-
-    if _user_name is None:
-        import pwd
-
-        _user_name = pwd.getpwuid(os.getuid()).pw_name
-
-    return _user_name
-
-
-####################################################################################################################################
-def group_name():
-    """Group the test runs as, looked up once since it is used several times and never changes."""
-
-    global _group_name
-
-    if _group_name is None:
-        import grp
-
-        _group_name = grp.getgrgid(os.getgid()).gr_name
-
-    return _group_name

test/lib/command/test/container.py

@@ -0,0 +1,647 @@
+"""Container Build.
+
+Generates a Dockerfile for each vm and builds it. Every vm has a base image with the packages the tests need and a test image that
+adds the test user, ssh, and the certificate authority, so a change to the test setup does not rebuild the packages.
+
+Base images are cached in a container registry keyed by a hash of the generated Dockerfile combined with the manual revision in
+test/container.yaml. The revision forces a rebuild when the Dockerfile is unchanged but the upstream packages have changed, e.g. a
+new PostgreSQL beta."""
+
+####################################################################################################################################
+import hashlib
+import os
+import re
+
+import yaml
+
+from common.error import check
+from common.exec import exec_one, exec_status
+from common.log import *
+from common.storage import file_read, file_write, path_create
+from common.user import group_id, group_name, user_id, user_name
+from common.vm import *
+
+# Separator between sections of the generated script. Every section is a continuation of a single RUN so the image has one layer.
+_SECTION = " && \\\n\n"
+
+# The pgbackrest namespace is the canonical cache that every environment pulls from and only pgbackrest CI can push to. A fork
+# pushes to and pulls from its own namespace instead.
+_REPO_OWNER = "pgbackrest"
+
+
+####################################################################################################################################
+def container_repo(owner=None):
+    """Container registry the images are stored in.
+
+    Pass an owner to build the tag for that owner's own cache. Images are always tagged in the pgbackrest namespace locally so
+    docker calls do not need to know which cache they came from."""
+
+    return "ghcr.io/%s/test" % (_REPO_OWNER if owner is None else owner)
+
+
+####################################################################################################################################
+def _ssh_setup(user, group):
+    """Install the test key pair so tests and the documentation can ssh between containers."""
+
+    path_user = "/%s" % user if user == "root" else "/home/%s" % user
+
+    key_list = [
+        "-----BEGIN RSA PRIVATE KEY-----",
+        "MIIEowIBAAKCAQEA5VySpEan5Rn7QQ5XP13YkXxTg+Xp9N+ecyDhD92OQk0VZOFn",
+        "EKlXUaMXH0WVjRRgjgbF78JHFbEQjUHimbzr9ev1IuDaIS42E1nUkBaHggnHNAW2",
+        "aqtczUDSnLGcXqd0XeDtwpjj1I0xWOeJli/xjU+eSwjzqNgX6ZX2P0uTXflu254u",
+        "SjOF3IARo9FiDEKQdksoxNOAhCTYgDAynmkcfJ9e87FjiTGmfnZO3H6gU2kidSFX",
+        "rwdHtdN3Qlv6RgCroLGAZZpmtqBorTwShWRScWAAg/OqWwLphfNbXIIHZO4EJ4S2",
+        "JLgsgJHb20SJ2XEI2Iln9sj+oCWgLJ2m+RLvWwIDAQABAoIBAArBC0EiQkyxf1Xe",
+        "txmKAWsE4iI85/oqzVJG7YvhuVdY0j16J2vLvNk05T3P9JdPqB1QqlGNEZSDSlbi",
+        "isjm55tkFl4tyRx61F9A5tYLWwwuVYWWFPutuuVcJOPi8gWAItUkruaLu6GjgyJQ",
+        "143QA//lBp4sYRxUEX71defO19iKkDz+xEuOzYMd16j76OKMcmbnog7hbMrXR4Yi",
+        "kNXuhnwBadutaXLve3mZ0JowrZyHKfTUWOHgvuULpCVD5su3NdbpE2AVn1idcF8V",
+        "jaj6p0vtcvEnXEC69XwX+yL0TgvOE4Vu/OWg8lDWQdetONIHGbElJZvB6eyTF1nl",
+        "IIgLc2ECgYEA7PL3soOfH3dMNUt4KGSw1cK/kwvy7UsT6QrAPi1Cc/A4szluk6/O",
+        "5YhKfTjzHW5WDmsTTAcT29MLmW8dQXeUAe/1BtIATubsav+uSelfBmUAnQj9fvKT",
+        "ieJ6JMf20OTbS6XODA3+jJAdApLCu61Lv6nePOuNzLY/uqSMWu9kO/sCgYEA981v",
+        "YIUaadFaHnPnmax0+jJMs8S5AIEjSfSIxR8oNOWNUxBBvwFd4zWTApVfZqKjmI83",
+        "Ng5tISxspzHseakyrIpoDqzxRQPxZF7RTO6VUX+ZQj6iIXVp9FDqWAjvDACuSky2",
+        "mGAiiA+fWZ1za62opgoYQZ17O17SyHF9/vJ7XCECgYANwNyXxAQMc4Q847CJx65r",
+        "+e3cvyjOlTkGodUexsnAqQThgkfk0qOTtyF7uz6BStI77AMmupJwhAN8WHK+Rg6V",
+        "PjRevPm/mq/GVijrqVwWpu4uL0NnhvUBX9/vGpw868u+zFT1ZiqMRiEo8RPUiO6I",
+        "pXd82b9VTo7Mapiq/pI22QKBgC8Kb586BUabOGlZhVi11Ur9q3Pg32HKIgHTCveo",
+        "r4BDJ23iQyjYQJN2Qx8VbhPUwgue/FMlr+/BOCsRHhwGU5lPeOt4RyDb28I7Aa6C",
+        "CBR9jYF21F5XpLJ9fc8SexajNnLiVzNb5JJBrPVdH2EMiVxjxDEIjTE7EfZ9HPb9",
+        "3w8hAoGBALyik3jr0W6DxzqOW0jPXPLDqp1JvzieC03nZD1scWeI8qIzUOpLX7cc",
+        "jaMU/8QMBRvyEcZK82Cedilm30nLf+C/FR5TsUmftS7IcjoC4Z2ZXWNOhMv22TUJ",
+        "Ml6z//+WSZ3qVZ5rvAeo4obwiBfe+Uh+AyyprEGgmimF9qDejcwc",
+        "-----END RSA PRIVATE KEY-----",
+    ]
+
+    public_key = (
+        "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlXJKkRqflGftBDlc/XdiRfFOD5en0355zIOEP3Y5CTRVk4WcQqVdRoxcfRZWNFGCOBsXvwk"
+        "cVsRCNQeKZvOv16/Ui4NohLjYTWdSQFoeCCcc0BbZqq1zNQNKcsZxep3Rd4O3CmOPUjTFY54mWL/GNT55LCPOo2BfplfY/S5Nd+W7bni5KM4XcgBGj0"
+        "WIMQpB2SyjE04CEJNiAMDKeaRx8n17zsWOJMaZ+dk7cfqBTaSJ1IVevB0e103dCW/pGAKugsYBlmma2oGitPBKFZFJxYACD86pbAumF81tcggdk7gQn"
+        "hLYkuCyAkdvbRInZcQjYiWf2yP6gJaAsnab5Eu9b user@pgbackrest-test"
+    )
+
+    result = _SECTION + "# Setup SSH\n" + "    mkdir %s/.ssh && \\\n" % path_user
+
+    for idx, key in enumerate(key_list):
+        result += "    echo '%s' %s %s/.ssh/id_rsa && \\\n" % (key, ">" if idx == 0 else ">>", path_user)
+
+    result += (
+        "    echo '%s' > %s/.ssh/authorized_keys && \\\n" % (public_key, path_user)
+        + "    echo 'Host *' > %s/.ssh/config && \\\n" % path_user
+        + "    echo '    StrictHostKeyChecking no' >> %s/.ssh/config && \\\n" % path_user
+        + "    cp %s/.ssh/authorized_keys %s/.ssh/id_rsa.pub && \\\n" % (path_user, path_user)
+        + "    chown -R %s:%s %s/.ssh && \\\n" % (user, group, path_user)
+        + "    chmod 700 %s/.ssh && \\\n" % path_user
+        + "    chmod 600 %s/.ssh/*" % path_user
+    )
+
+    return result
+
+
+####################################################################################################################################
+def _file_copy(path_repo, file_source, file_dest):
+    """Copy a text file into the container.
+
+    This does not work if the file contains single quotes."""
+
+    result = ""
+
+    # Trailing newlines are dropped so the file does not end up with a blank line appended to it
+    for line in file_read(os.path.join(path_repo, file_source)).rstrip("\n").split("\n"):
+        result += "    echo '%s' %s %s && \\\n" % (line, ">>" if result else ">", file_dest)
+
+    return result
+
+
+####################################################################################################################################
+def _ca_setup(os_base, path_repo, file_ca):
+    """Install the certificate authority the test certificates are signed with."""
+
+    if os_base == VM_OS_BASE_RHEL:
+        file_cert = "/etc/pki/ca-trust/source/anchors"
+    else:
+        file_cert = "/usr/local/share/ca-certificates"
+
+    file_cert += "/pgbackrest-test-ca.crt"
+
+    result = _SECTION + "# Install CA\n" + _file_copy(path_repo, file_ca, file_cert) + "    chmod 644 %s && \\\n" % file_cert
+
+    if os_base == VM_OS_BASE_RHEL:
+        result += "    update-ca-trust extract"
+    else:
+        result += "    update-ca-certificates"
+
+    return result
+
+
+####################################################################################################################################
+def _group_create(os_base, name, id):
+    """Command to create a group."""
+
+    if os_base == VM_OS_BASE_ALPINE:
+        return "getent group %s >/dev/null || addgroup -g%u %s" % (name, id, name)
+
+    return "groupadd -f -g%u %s" % (id, name)
+
+
+####################################################################################################################################
+def _user_create(os_base, name, id, group):
+    """Command to create a user."""
+
+    if os_base == VM_OS_BASE_RHEL:
+        return "adduser -g%s -u%u -N %s" % (group, id, name)
+
+    return 'adduser --uid=%u --ingroup=%s --disabled-password --gecos "" %s' % (id, group, name)
+
+
+####################################################################################################################################
+def _entry_point_setup(os_base):
+    """Start ssh when the container starts."""
+
+    result = "\n\n# Start SSH when container starts\n" + "ENTRYPOINT "
+
+    if os_base == VM_OS_BASE_RHEL:
+        result += "rm -rf /run/nologin && /usr/sbin/sshd -D"
+    elif os_base == VM_OS_BASE_DEBIAN:
+        result += "service ssh restart && bash"
+    else:
+        result += "/usr/sbin/sshd -D"
+
+    return result
+
+
+####################################################################################################################################
+def _script_package(name, vm):
+    """Install the packages the tests need."""
+
+    result = _SECTION + "# Install packages\n"
+
+    if vm.os_base == VM_OS_BASE_RHEL:
+        if name == VM_RH8:
+            result += (
+                "    dnf install -y dnf-plugins-core && \\\n"
+                + "    dnf config-manager --set-enabled powertools && \\\n"
+                + "    dnf -y install epel-release && \\\n"
+                + "    crb enable && \\\n"
+            )
+        elif name in (VM_RH9, VM_RH10):
+            result += (
+                "    dnf install -y dnf-plugins-core && \\\n" + "    dnf -y install epel-release && \\\n" + "    crb enable && \\\n"
+            )
+
+        result += (
+            "    yum -y update && \\\n"
+            + "    yum -y install openssh-server openssh-clients sudo git ca-certificates gcc make ccache meson \\\n"
+            + "        openssl openssl-devel libyaml-devel zlib-devel libxml2-devel lz4-devel lz4 bzip2-devel bzip2 \\\n"
+            + "        libssh2-devel zstd libzstd-devel systemd-devel"
+        )
+
+        if name != VM_RH8:
+            result += " valgrind"
+
+        # Coverage for the python tests. rh10 and later package a version new enough while rh9 packages one that predates the
+        # per line branch detail the report needs, so there it comes from pip below.
+        if name == VM_RH9:
+            result += " python3-pip"
+        elif name != VM_RH8:
+            result += " python3-coverage"
+
+        # The test harness needs a python3 that PyYAML is packaged for. On RH8 that is not the platform python, which meson
+        # installs and which claims the python3 alternative at a priority nothing else can outrank, so point it explicitly.
+        if name == VM_RH8:
+            result += " python3.12 python3.12-pyyaml && \\\n" + "    alternatives --set python3 /usr/bin/python3.12"
+        else:
+            result += " python3-pyyaml"
+    elif vm.os_base == VM_OS_BASE_DEBIAN:
+        result += (
+            "    export DEBCONF_NONINTERACTIVE_SEEN=true DEBIAN_FRONTEND=noninteractive && \\\n"
+            + "    apt-get update && \\\n"
+            + "    apt-get install -y --no-install-recommends openssh-server sudo gcc make git \\\n"
+            + "        ca-certificates libssl-dev tzdata zlib1g-dev libxml2-dev pkg-config \\\n"
+            + "        libbz2-dev bzip2 libyaml-dev liblz4-dev liblz4-tool gnupg lsb-release ccache meson \\\n"
+            + "        libssh2-1-dev libcurl4-openssl-dev libsystemd-dev python3-yaml"
+        )
+
+        if name != VM_D12:
+            result += " valgrind"
+
+        # Coverage for the python tests, which are run on u24 and later
+        if name not in (VM_D12, VM_U22):
+            result += " python3-coverage"
+
+        result += " zstd libzstd-dev"
+    else:
+        result += (
+            "    apk update && \\\n"
+            + "    apk add --no-cache sudo openssh git rsync tzdata openssh ca-certificates openrc bash && \\\n"
+            + "    rc-update add sshd && \\\n"
+            + "    apk add --no-cache meson build-base libpq-dev openssl-dev libxml2-dev pkgconfig lz4-dev bzip2-dev\\\n"
+            + "        openssh-keygen zlib-dev yaml-dev libssh2-dev valgrind lz4 zstd zstd-dev py3-yaml"
+        )
+
+    # Coverage for python tests where the packaged version predates the per line branch detail the report needs, so it comes from
+    # pip rather than from the distribution
+    if name == VM_RH9:
+        result += _SECTION + "# Install python coverage\n" + "    pip3 install --no-cache-dir 'coverage>=6.5'"
+
+    return result
+
+
+####################################################################################################################################
+def _script_pg(name, vm, arch):
+    """Install PostgreSQL."""
+
+    result = ""
+
+    if vm.os_base in (VM_OS_BASE_RHEL, VM_OS_BASE_DEBIAN):
+        result += _SECTION + "# Install PostgreSQL packages\n"
+
+    if vm.os_base == VM_OS_BASE_RHEL:
+        # RHEL 10's rpm-sequoia OpenPGP backend rejects the SHA-1 binding signature in the PGDG GPG key, and the crypto-policy
+        # SHA1 subpolicy that could re-enable it was removed in EL-10. So rh10 skips the key import and installs the repo/packages
+        # with gpg checks disabled (acceptable for a throwaway test container).
+        if name != VM_RH10 and vm.pg_repo:
+            result += "    rpm --import https://download.postgresql.org/pub/repos/yum/keys/RPM-GPG-KEY-PGDG && \\\n"
+
+        if not vm.pg_repo:
+            # Enable the native PostgreSQL application stream instead of adding the PGDG repo
+            result += "    dnf -y module enable postgresql:%s && \\\n" % vm.db_list[-1]
+        elif name == VM_RH9:
+            result += (
+                "    rpm -ivh \\\n"
+                + "        https://download.postgresql.org/pub/repos/yum/reporpms/EL-9-%s/" % arch
+                + "pgdg-redhat-repo-latest.noarch.rpm && \\\n"
+                + "    dnf -qy module disable postgresql && \\\n"
+                + "    yum -y install libcurl-devel && \\\n"
+            )
+        elif name == VM_RH10:
+            # Install the repo without a gpg check and disable gpgcheck on the PGDG repos (see the SHA-1 note above). RHEL 10 also
+            # dropped dnf modularity, so there is no postgresql module to disable.
+            result += (
+                "    dnf -y install --nogpgcheck \\\n"
+                + "        https://download.postgresql.org/pub/repos/yum/reporpms/EL-10-%s/" % arch
+                + "pgdg-redhat-repo-latest.noarch.rpm && \\\n"
+                + "    sed -i 's/gpgcheck=1/gpgcheck=0/g' /etc/yum.repos.d/pgdg-redhat-all.repo && \\\n"
+                + "    yum -y install libcurl-devel && \\\n"
+            )
+        # Every rhel vm that installs from the PostgreSQL repo is one of the three above, so there is nothing to fall through to
+        elif name == VM_F44:  # {uncoverable_branch}
+            result += (
+                "    rpm -ivh \\\n"
+                + "        https://download.postgresql.org/pub/repos/yum/reporpms/F-44-%s/" % arch
+                + "pgdg-fedora-repo-latest.noarch.rpm && \\\n"
+                + "    yum -y install libcurl-devel && \\\n"
+            )
+
+        result += "    yum -y install postgresql-devel"
+    elif vm.os_base == VM_OS_BASE_DEBIAN:
+        # Install repo from apt.postgresql.org
+        if vm.pg_repo:
+            result += (
+                "    apt-get install -y --no-install-recommends postgresql-common && \\\n"
+                + "    /usr/share/postgresql-common/pgdg/apt.postgresql.org.sh -y"
+                + (" -c 19" if name == VM_U24 and arch in (VM_ARCH_AARCH64, VM_ARCH_X86_64) else "")
+                + " && \\\n"
+            )
+
+        result += (
+            "    apt-get install -y --no-install-recommends postgresql-common libpq-dev && \\\n"
+            + "    sed -i 's/^\\#create\\_main\\_cluster.*$/create\\_main\\_cluster \\= false/' "
+            + "/etc/postgresql-common/createcluster.conf"
+        )
+
+    if vm.db_list and arch in (VM_ARCH_AARCH64, VM_ARCH_X86_64, VM_ARCH_I386):
+        result += _SECTION + "# Install PostgreSQL\n"
+
+        if vm.os_base == VM_OS_BASE_RHEL:
+            result += "    yum -y install"
+        elif vm.os_base == VM_OS_BASE_DEBIAN:
+            result += "    apt-get install -y --no-install-recommends"
+        else:
+            result += "    apk add --no-cache"
+
+        # Construct list of databases to install
+        for db_version in vm.db_list:
+            if vm.os_base == VM_OS_BASE_RHEL:
+                # Native RHEL installs the unversioned server package from the application stream
+                if not vm.pg_repo:
+                    result += " postgresql-server"
+                else:
+                    db_version_no_dot = db_version.replace(".", "")
+
+                    result += " postgresql%s-server" % db_version_no_dot
+
+                    # Add development package for the latest version of postgres
+                    if db_version == vm.db_list[-1]:
+                        result += " postgresql%s-devel" % db_version_no_dot
+            elif vm.os_base == VM_OS_BASE_DEBIAN:
+                result += " postgresql-%s" % db_version
+            else:
+                result += " postgresql%s" % db_version
+    # On other architectures (e.g. ppc64le, s390x) install a single PostgreSQL version for the smoke test via the postgresql
+    # metapackage (its default from the configured repo). The full version matrix is skipped to keep the emulated build fast; one
+    # version is enough to check the build and exercise checksums end-to-end on big-endian.
+    elif vm.db_list and vm.os_base == VM_OS_BASE_DEBIAN:
+        result += (
+            _SECTION
+            + "# Install a single PostgreSQL version for the smoke test\n"
+            + "    apt-get install -y --no-install-recommends postgresql"
+        )
+
+    # Add path to latest version of postgres (PGDG installs to a versioned path; native packages use /usr/bin)
+    if vm.os_base == VM_OS_BASE_RHEL and vm.pg_repo:
+        result += (
+            "\n\nENV PATH=/usr/pgsql-%s/bin:$PATH\n" % vm.db_list[-1]
+            + "ENV PKG_CONFIG_PATH=/usr/pgsql-%s/lib/pkgconfig:$PKG_CONFIG_PATH\n" % vm.db_list[-1]
+        )
+
+    return result
+
+
+####################################################################################################################################
+def script_base(name, arch):
+    """Generate the script for the base image, which holds everything installed from a package."""
+
+    vm = vm_get(name)
+    result = _script_package(name, vm)
+
+    result += (
+        _SECTION
+        + "# Regenerate SSH keys\n"
+        + "    rm -f /etc/ssh/ssh_host_rsa_key* && \\\n"
+        + "    ssh-keygen -t rsa -b 2048 -f /etc/ssh/ssh_host_rsa_key"
+    )
+
+    if vm.os_base == VM_OS_BASE_DEBIAN:
+        result += _SECTION + "# Fix root tty\n" + "    sed -i 's/^mesg n/tty -s \\&\\& mesg n/g' /root/.profile"
+        result += _SECTION + "# Suppress dpkg interactive output\n" + "    rm /etc/apt/apt.conf.d/70debconf"
+
+    result += _script_pg(name, vm, arch)
+
+    if vm.os_base == VM_OS_BASE_DEBIAN:
+        result += (
+            _SECTION
+            + "# Cleanup\n"
+            + "    apt-get autoremove -y && \\\n"
+            + "    apt-get clean && \\\n"
+            + "    rm -rf /var/lib/apt/lists/*"
+        )
+
+    return result
+
+
+####################################################################################################################################
+def script_test(name, path_repo):
+    """Generate the script for the test image, which adds the test user and everything it needs."""
+
+    vm = vm_get(name)
+    result = ""
+
+    if vm.os_base == VM_OS_BASE_RHEL and host_arch() == VM_ARCH_AARCH64:
+        result += _SECTION + "# Remove unneeded language setup\n" + "    rm /etc/profile.d/lang.sh"
+
+    result += _ca_setup(vm.os_base, path_repo, "test/certificate/pgbackrest-test-ca.crt")
+
+    result += (
+        _SECTION
+        + "# Create banner to make sure pgBackRest ignores it\n"
+        + "    echo '***********************************************' >  /etc/issue.net && \\\n"
+        + "    echo 'Sample banner to make sure banners are skipped.' >> /etc/issue.net && \\\n"
+        + "    echo ''                                                >> /etc/issue.net && \\\n"
+        + "    echo 'More banner after a blank line.'                 >> /etc/issue.net && \\\n"
+        + "    echo '***********************************************' >> /etc/issue.net && \\\n"
+        + "    echo 'Banner /etc/issue.net'                           >> /etc/ssh/sshd_config"
+    )
+
+    if name in (VM_U22, VM_U24, VM_A321):
+        result += (
+            _SECTION
+            + "    echo '# Add PubkeyAcceptedAlgorithms (required for SFTP)'              >> /etc/ssh/sshd_config && \\\n"
+            + "    echo 'HostKeyAlgorithms=+ssh-rsa,ssh-rsa-cert-v01@openssh.com'        >> /etc/ssh/sshd_config && \\\n"
+            + "    echo 'PubkeyAcceptedAlgorithms=+ssh-rsa,ssh-rsa-cert-v01@openssh.com' >> /etc/ssh/sshd_config"
+        )
+
+    # Rename existing group that would conflict with our group name. This is pretty hacky but should be OK since we are the only
+    # thing running in the container.
+    result += (
+        _SECTION
+        + "# Rename conflicting group\n"
+        + "    sed -i 's/^[^:]*:x:%u:.*/%s:x:%u:/' /etc/group" % (group_id(), group_name(), group_id())
+    )
+
+    result += (
+        _SECTION
+        + "# Create test user\n"
+        + "    %s && \\\n" % _group_create(vm.os_base, group_name(), group_id())
+        + "    %s && \\\n" % _user_create(vm.os_base, user_name(), user_id(), group_name())
+        + "    mkdir -m 750 /home/%s/test && \\\n" % user_name()
+        + "    chown %s:%s /home/%s/test" % (user_name(), group_name(), user_name())
+    )
+
+    # On Alpine the test account must have a password for SSH logon
+    if vm.os_base == VM_OS_BASE_ALPINE:
+        result += " && \\\n" + "    passwd -d '%s' %s" % (user_name(), user_name())
+
+    result += _SECTION + "# Configure sudo\n"
+
+    if vm.os_base == VM_OS_BASE_RHEL:
+        result += (
+            # Don't allow sudo to disable core dump (suppresses errors, see https://github.com/sudo-project/sudo/issues/42)
+            '    echo "Set disable_coredump false" >> /etc/sudo.conf && \\\n'
+            + "    echo '%%%s        ALL=(ALL)       NOPASSWD: ALL' > /etc/sudoers.d/%s && \\\n" % (group_name(), group_name())
+            + "    sed -i 's/^Defaults    requiretty$/\\# Defaults    requiretty/' /etc/sudoers"
+        )
+    else:
+        result += "    echo '%%%s ALL=(ALL) NOPASSWD: ALL' >> /etc/sudoers" % group_name()
+
+    result += _ssh_setup(user_name(), group_name())
+
+    result += _SECTION + "# Make %s home dir readable\n" % user_name() + "    chmod g+r,g+x /home/%s" % user_name()
+
+    result += _entry_point_setup(vm.os_base)
+
+    return result
+
+
+####################################################################################################################################
+def container_script(title, image_parent, script):
+    """Wrap a script in the Dockerfile that builds it."""
+
+    return "# %s Container\n" % title + "FROM %s" % image_parent + ("\n\nRUN echo 'OPTIMIZED BUILD'" + script if script else "")
+
+
+####################################################################################################################################
+def _revision_get(revision, name, arch):
+    """Revision for a container.
+
+    A container uses the revision for its vm and architecture if it has one, else the revision for its vm, else the required "all"
+    revision. So bumping "all" rebuilds only the containers without a more specific revision."""
+
+    return revision.get("%s-%s" % (name, arch), revision.get(name, revision[VM_ALL]))
+
+
+####################################################################################################################################
+def revision_check(revision):
+    """Check the revisions read from container.yaml.
+
+    A revision is keyed by "all", a vm, or a vm qualified with an architecture, e.g. "u22-x86_64". Without this an invalid key
+    would silently fall back to a less specific revision and the expected rebuild would never happen."""
+
+    # The "all" revision is required so it can never be accidentally unset, which would change every image hash at once. It can
+    # only be bumped.
+    check(isinstance(revision, dict) and VM_ALL in revision, "the 'all' revision is required in test/container.yaml")
+
+    for key in sorted(revision):
+        check(revision[key] is not None, "revision '%s' in test/container.yaml must be set to a value" % key)
+
+        if key == VM_ALL:
+            continue
+
+        key_vm, _, key_arch = key.partition("-")
+
+        check(vm_valid(key_vm) and key_vm != VM_NONE, "revision '%s' in test/container.yaml has invalid vm '%s'" % (key, key_vm))
+        check(
+            key_arch == "" or key_arch in VM_ARCH_LIST,
+            "revision '%s' in test/container.yaml has invalid architecture '%s'" % (key, key_arch),
+        )
+
+
+####################################################################################################################################
+def _container_write(path_repo, name, arch, title, image_parent, image, script, revision_map, force, show_output):
+    """Write the Dockerfile for an image and build it, using the cache when the image is already there."""
+
+    path_temp = os.path.join(path_repo, "test/result/docker")
+    tag = "%s:%s" % (container_repo(), image)
+    script = container_script(title, image_parent, script)
+
+    # Only a base image is cached since a test image is cheap to build and depends on the user that built it
+    cached = False
+    cache_tag = None
+    fork_cache_tag = None
+
+    if "-base-" in image:
+        revision = _revision_get(revision_map, name, arch)
+        cache_image = "%s-%s-%s" % (image, revision, hashlib.sha1((script + revision).encode()).hexdigest()[:12])
+        cache_tag = "%s:%s" % (container_repo(), cache_image)
+
+        # On a fork also use the fork owner's own cache, which it can write to
+        owner = os.environ.get("GITHUB_REPOSITORY_OWNER", "").lower() or _REPO_OWNER
+        fork_cache_tag = "%s:%s" % (container_repo(owner), cache_image) if owner != _REPO_OWNER else None
+
+        # Pull from the pgbackrest cache first, then the fork's own cache. A failed pull (image missing or no read access) falls
+        # through to the next cache or a local build.
+        if not force:
+            for pull_tag in [cache_tag] + ([] if fork_cache_tag is None else [fork_cache_tag]):
+                log(INFO, "Checking cache %s ..." % pull_tag)
+
+                if exec_status("docker pull %s" % pull_tag) == 0:
+                    log(INFO, "Using cached %s image" % pull_tag)
+                    exec_one("docker tag %s %s" % (pull_tag, tag))
+                    cached = True
+
+                    break
+
+    if cached:
+        return
+
+    log(INFO, "Building %s image%s ..." % (tag, "" if cache_tag is None else " (%s)" % cache_tag))
+
+    # Write and build the image
+    file_write(os.path.join(path_temp, image), script.strip() + "\n")
+    exec_one(
+        "docker build --platform linux/%s%s -f %s -t %s %s"
+        % (arch, " --no-cache" if force else "", os.path.join(path_temp, image), tag, os.path.join(path_repo, "test")),
+        show_output=show_output,
+    )
+
+    # Push the base image to the cache this environment can write to (the fork's own cache on a fork, else the pgbackrest cache)
+    # and log when it succeeds. Best effort since it requires write access, which is not available locally or on pull requests from
+    # forks; those just use the image they built.
+    if cache_tag is not None:
+        push_tag = cache_tag if fork_cache_tag is None else fork_cache_tag
+        exec_status("docker tag %s %s" % (tag, push_tag))
+
+        if exec_status("docker push %s" % push_tag) == 0:
+            log(INFO, "Cached %s image" % push_tag)
+
+
+####################################################################################################################################
+def container_remove(expression):
+    """Remove the containers whose name matches an expression."""
+
+    regexp = re.compile(expression)
+
+    for container in sorted(exec_one('docker ps -a --format "{{.Names}}"').strip().split("\n")):
+        if regexp.search(container):
+            exec_status("docker rm -f %s" % container)
+
+
+####################################################################################################################################
+def _image_remove(expression):
+    """Remove the images whose name matches an expression."""
+
+    regexp = re.compile(expression)
+
+    for image in sorted(exec_one('docker images --format "{{.Repository}}:{{.Tag}}"').strip().split("\n")):
+        if regexp.search(image):
+            log(INFO, "Removing %s image..." % image)
+            exec_one("docker rmi -f %s" % image)
+
+
+####################################################################################################################################
+def container_build(config):
+    """Build the containers for one vm or for all of them."""
+
+    path_repo = config.repo_path
+    path_temp = os.path.join(path_repo, "test/result/docker")
+    arch = config.vm_arch if config.vm_arch is not None else host_arch()
+    show_output = config.log_level >= DETAIL
+
+    path_create(path_temp, mode=0o770)
+
+    with open(os.path.join(path_repo, "test/container.yaml"), "r") as file:
+        revision_map = yaml.safe_load(file)
+
+    revision_check(revision_map)
+
+    # Remove old images on force
+    if config.vm_force:
+        expression = "^" + container_repo() + ("" if config.vm == VM_ALL else ":%s-" % config.vm)
+
+        exec_one("rm -f %s/%s" % (path_temp, "*" if config.vm == VM_ALL else "%s-*" % config.vm))
+        _image_remove(expression + ".*")
+
+    for name in VM_LIST if config.vm == VM_ALL else [config.vm]:
+        # Base image
+        image_base = "%s-base-%s" % (name, arch)
+
+        _container_write(
+            path_repo,
+            name,
+            arch,
+            "Base",
+            vm_get(name).image,
+            image_base,
+            script_base(name, arch),
+            revision_map,
+            config.vm_force,
+            show_output,
+        )
+
+        # Test image
+        _container_write(
+            path_repo,
+            name,
+            arch,
+            "Test",
+            "%s:%s" % (container_repo(), image_base),
+            "%s-test-%s" % (name, arch),
+            script_test(name, path_repo),
+            revision_map,
+            config.vm_force,
+            show_output,
+        )
+
+    log(INFO, "Build Complete")

test/lib/command/test/job.py

@@ -0,0 +1,318 @@
+"""Test Job.
+
+Runs one test and monitors its progress. The test runs in another process, and in a container of its own when the vm is not none,
+so several tests can run at once. The harness is invoked there with the unit command, which builds the test and runs it.
+
+The job is polled rather than waited on so the driver can keep every vm busy, which is why begin() only starts the test and end()
+reports whether it has finished."""
+
+####################################################################################################################################
+import math
+import os
+import time
+
+from command.test.container import container_remove
+from command.test.define import TEST_LANG_C, TEST_TYPE_UNIT
+from common.exec import Exec, exec_one
+from common.log import *
+from common.storage import path_create
+from common.user import user_name
+from common.vm import *
+
+
+####################################################################################################################################
+class TestJob:
+    """Run a test and monitor its progress."""
+
+    def __init__(self, config, run, vm_idx, vm_max, test_idx, test_max, image, show_output):
+        self.config = config
+        self.run = run
+        self.vm_idx = vm_idx
+        self.vm_max = vm_max
+        self.test_idx = test_idx
+        self.test_max = test_max
+        self.image = image  # Container image the test runs in
+        self.show_output = show_output  # Write the test output as it arrives?
+
+        self.try_idx = 0
+        self.log_level = config.log_level
+        self.coverage = False
+        self.exec = None
+        self.description = None
+        self.time_begin = None
+
+        # A unit test is built for the vm it runs on but an integration test runs the binary that was built for none
+        self.path_unit = os.path.join(config.test_path, "unit-%u" % vm_idx, VM_NONE if run.integration else run.vm)
+        self.path_data = os.path.join(config.test_path, "data-%u" % vm_idx)
+        self.path_repo = os.path.join(config.test_path, "repo")
+        self.path_host = os.path.join(config.test_path, "test-%u" % vm_idx)
+        self.container = "test-%u" % vm_idx
+
+    ################################################################################################################################
+    def _exec_container(self, command):
+        """Run a command in the test container when the test has one."""
+
+        if self.run.vm == VM_NONE:
+            return command
+
+        return "docker exec -i -u %s %s %s" % (user_name(), self.container, command)
+
+    ################################################################################################################################
+    def _container_run(self):
+        """Start the container the test runs in."""
+
+        config = self.config
+        path_build = os.path.join(config.test_path, "build", self.run.vm)
+        path_ccache = os.path.join(config.test_path, "ccache")
+
+        # An integration test starts its own containers so it never gets here, which means the unit build and the ccache that is
+        # shared with the host and every other container are always mounted
+        mount = " -v %s:%s -v %s:%s -v %s:%s:ro -v %s:%s -e CCACHE_DIR=%s" % (
+            self.path_unit,
+            self.path_unit,
+            self.path_data,
+            self.path_data,
+            path_build,
+            path_build,
+            path_ccache,
+            path_ccache,
+            path_ccache,
+        )
+
+        exec_one(
+            "docker run%s -itd -h %s-test --name=%s -v %s:%s%s -v %s:%s -v %s:%s %s"
+            % (
+                "" if config.vm_arch is None else " --platform linux/%s" % config.vm_arch,
+                self.run.vm,
+                self.container,
+                self.path_host,
+                self.path_host,
+                mount,
+                config.repo_path,
+                config.repo_path,
+                self.path_repo,
+                self.path_repo,
+                self.image,
+            )
+        )
+
+    ################################################################################################################################
+    def _command(self):
+        """Build the command that runs the test."""
+
+        config = self.config
+        run = self.run
+
+        # An integration test runs the binary that was built for none rather than one built for the vm
+        vm = VM_NONE if run.integration else run.vm
+
+        command = (
+            "python3 %s/test/test.py unit --repo-path=%s --test-path=%s --log-level=%s --log-level-test=%s --vm=%s"
+            % (
+                self.path_repo,
+                self.path_repo,
+                config.test_path,
+                LEVEL_NAME[self.log_level],
+                LEVEL_NAME[config.log_level_test],
+                run.vm,
+            )
+            + ("" if config.vm_arch is None else " --vm-arch=%s" % config.vm_arch)
+            + " --vm-id=%u" % self.vm_idx
+            + (" --profile" if config.profile else "")
+            + "".join(" --test=%u" % test for test in run.run_list or [])
+            + ("" if config.log_timestamp else " --no-log-timestamp")
+            + ("" if config.tz is None else " --tz='%s'" % config.tz)
+            + " --scale=%u" % config.scale
+            + ("" if run.pg_version is None else " --pg-version=%s" % run.pg_version)
+            + ("" if config.back_trace else " --no-back-trace")
+            + ("" if self.coverage else " --no-coverage")
+        )
+
+        # A python test writes its own coverage since there is no gcov step for it in end()
+        if run.module.lang != TEST_LANG_C and self.coverage:
+            command += " --coverage-file=%s/test/result/coverage/raw/%s.json" % (config.repo_path, run.coverage_name)
+
+        command += " " + run.module.name
+
+        # A python test is run by the harness above so there is no binary to run here
+        if run.module.lang == TEST_LANG_C:
+            valgrind = ""
+
+            if config.valgrind and not run.performance:
+                path_suppress = os.path.join(self.path_repo, "test/src/valgrind.suppress.%s" % vm)
+
+                valgrind = (
+                    "valgrind -q --gen-suppressions=all"
+                    + (" --suppressions=%s" % path_suppress if os.path.exists(path_suppress) else "")
+                    + " --exit-on-first-error=yes --leak-check=full --leak-resolution=high --error-exitcode=25 "
+                )
+
+            # Copy stderr to both stderr and stdout so it is displayed and detected as an error. Piping to tee would mask the test
+            # exit status (the pipe returns tee's status) so save the status to a file and exit with it explicitly. Otherwise a
+            # test failure (e.g. a valgrind error) would be hidden and reported as missing coverage instead.
+            command += (
+                " && \\\nexec 3>&1 && \\\n{ %s%s/build/test-unit 2>&1 1>&3; echo $? > %s/result; } | tee /dev/stderr && \\\n"
+                % (valgrind, self.path_unit, self.path_unit)
+                + "exit $(cat %s/result)" % self.path_unit
+            )
+
+        # Run in the container when the test has one
+        if vm != VM_NONE:
+            command = "docker exec -i -u %s %s bash -l -c '\\\n%s'" % (user_name(), self.container, command)
+
+        return command
+
+    ################################################################################################################################
+    def begin(self):
+        """Start the test and report whether it was started.
+
+        A test that has used up its retries is not started again, which is how the driver knows to give up on it."""
+
+        config = self.config
+        run = self.run
+
+        self.time_begin = time.time()
+        self.try_idx += 1
+
+        if self.try_idx > config.retry + 1:
+            return False
+
+        # Raise the log level for the last try so there is something to look at when it fails again
+        if self.try_idx != 1 and self.try_idx == config.retry + 1:
+            self.log_level = DEBUG
+
+        self.description = (
+            "P%0*d-T%0*d/%0*d - vm=%s, module=%s, test=%s"
+            % (
+                len(str(self.vm_max)),
+                self.vm_idx + 1,
+                len(str(self.test_max)),
+                self.test_idx + 1,
+                len(str(self.test_max)),
+                self.test_max,
+                run.vm,
+                run.group,
+                run.test,
+            )
+            + ("" if run.run_list is None else ", run=%s" % ",".join(str(test) for test in sorted(run.run_list)))
+            + ("" if run.pg_version is None else ", pg-version=%s" % run.pg_version)
+            + ("" if self.try_idx == 1 else " (retry %u)" % (self.try_idx - 1))
+        )
+
+        # A dry run lists the tests it would run, so the list goes to the log rather than the detail the tests write
+        dry_run = config.dry_run and not config.vm_out
+
+        # A blank line separates the test from the output that follows it, which is only shown when it was asked for
+        separator = "\n" if (not config.dry_run and config.vm_out) or self.show_output else ""
+
+        log(INFO if dry_run or self.show_output else DETAIL, self.description + separator)
+
+        if dry_run:
+            return False
+
+        path_create(self.path_host, mode=0o770)
+
+        if not run.integration:
+            path_create(self.path_unit, mode=0o770)
+
+            if run.module.type == TEST_TYPE_UNIT:
+                path_create(self.path_data, mode=0o770)
+
+            if run.vm != VM_NONE:
+                self._container_run()
+
+        # Coverage is collected when the vm supports it, except for a performance or profile run where it would skew the timing
+        self.coverage = vm_get(run.vm).coverage_c and config.coverage and not run.performance and not config.profile
+
+        self.exec = Exec(self._command(), show_output=self.show_output)
+        self.exec.begin()
+
+        return True
+
+    ################################################################################################################################
+    def _profile(self):
+        """Write the profile the test generated."""
+
+        path_profile = os.path.join(self.config.repo_path, "test/result/profile")
+
+        exec_one(
+            self._exec_container(
+                "gprof %s/build/test-unit %s/build/gmon.out > %s/gprof.txt" % (self.path_unit, self.path_unit, self.path_unit)
+            )
+        )
+
+        path_create(path_profile)
+        exec_one("cp %s/gprof.txt %s/gprof.txt" % (self.path_unit, path_profile))
+
+    ################################################################################################################################
+    def _coverage(self):
+        """Write the coverage the test generated.
+
+        Meson names the directory it compiled the test into and the name has changed between versions, so look for both."""
+
+        path_object = os.path.join(self.path_unit, "build/test-unit.p")
+
+        if not os.path.exists(path_object):
+            path_object = os.path.join(self.path_unit, "build/test-unit@exe")
+
+        exec_one(
+            self._exec_container(
+                "gcov --json-format --stdout --branch-probabilities %s/test.c.gcda > %s/test/result/coverage/raw/%s.json"
+                % (path_object, self.config.repo_path, self.run.coverage_name)
+            )
+        )
+
+    ################################################################################################################################
+    def end(self):
+        """Report whether the test has finished and whether it failed.
+
+        The test is waited on when it is the only one running, since there is nothing else to do until it is done."""
+
+        config = self.config
+        status = self.exec.end(wait=self.vm_max == 1)
+
+        if status is None:
+            return False, False
+
+        if self.show_output:
+            print("")
+
+        # Write the profile and coverage a C test left behind. A python test wrote its own coverage while it ran.
+        if status == 0 and self.run.module.lang == TEST_LANG_C and not self.run.integration:
+            if config.profile:
+                self._profile()
+
+            if self.coverage:
+                self._coverage()
+
+        elapsed = math.ceil((time.time() - self.time_begin) * 100) / 100
+
+        # Anything written to stderr is a failure whatever the exit status was, e.g. a valgrind error
+        fail = status != 0 or self.exec.error != ""
+
+        if fail:
+            output = self.exec.output.strip()
+
+            log(
+                ERROR,
+                self.description
+                + " (err%d%s)" % (status, "-%.2fs" % elapsed if config.log_timestamp else "")
+                + ("" if self.show_output else ":\n\n%s\n" % (output if output else "NO OUTPUT ON STDOUT OR STDERR")),
+            )
+        else:
+            log(
+                INFO,
+                self.description
+                + (" (%.2fs)" % elapsed if config.log_timestamp else "")
+                + (":\n\n%s\n" % self.exec.output.strip() if config.vm_out and not self.show_output else ""),
+            )
+
+        if config.cleanup:
+            # An integration test starts its own containers and names them after the job, so those are removed here too. Anything
+            # still running in one would keep writing to the test path while it is being removed.
+            if self.run.vm != VM_NONE:
+                container_remove("^%s($|-)" % self.container)
+
+            exec_one("chmod -R 700 %s/* 2>&1;rm -rf %s" % (self.path_host, self.path_host))
+
+        return True, fail

test/lib/command/test/list.py

@@ -0,0 +1,125 @@
+"""Test List.
+
+Selects the tests to run from the definitions and the options. A test module produces one run per vm, and an integration module
+produces one run per sub-test and PostgreSQL version as well, since each of those needs its own container."""
+
+####################################################################################################################################
+from command.test.define import TEST_TYPE_INTEGRATION, TEST_TYPE_PERFORMANCE
+from common.vm import *
+
+
+####################################################################################################################################
+class TestRun:
+    """A test module to run on one vm, for one PostgreSQL version and one sub-test where those apply."""
+
+    def __init__(self, module, vm, pg_version, run_list):
+        self.module = module
+        self.vm = vm
+        self.pg_version = pg_version  # PostgreSQL version to test against, None when the test does not need one
+        self.run_list = run_list  # Sub-tests to run, None for all of them
+
+        # The module name is the group and the test in it, e.g. the common group and the error test in it
+        self.group, _, self.test = module.name.partition("/")
+
+    ################################################################################################################################
+    @property
+    def integration(self):
+        """Is this an integration test?
+
+        An integration test runs the pgbackrest binary rather than a unit test, so there is nothing to build for it and it starts
+        its own containers."""
+
+        return self.module.type == TEST_TYPE_INTEGRATION
+
+    ################################################################################################################################
+    @property
+    def performance(self):
+        """Is this a performance test?
+
+        A performance test is built and run like a unit test but is timed, so debugging and coverage are left out of it."""
+
+        return self.module.type == TEST_TYPE_PERFORMANCE
+
+    ################################################################################################################################
+    @property
+    def coverage_name(self):
+        """Name of the raw coverage file for this test.
+
+        A test name contains a path separator so flatten it, since the coverage command reads the raw files from one directory."""
+
+        return self.module.name.replace("/", "-")
+
+
+####################################################################################################################################
+def _selected(name_list, name):
+    """Was this module or test selected on the command line?
+
+    An empty list selects everything, which is what running with no --module or --test does."""
+
+    return not name_list or name.lower() in [selected.lower() for selected in name_list]
+
+
+####################################################################################################################################
+def test_list_get(module_list, config):
+    """Build the list of tests to run."""
+
+    result = []
+
+    for vm_name in VM_LIST if config.vm == VM_ALL else [config.vm]:
+        vm = vm_get(vm_name)
+
+        for module in module_list:
+            # The module name is the group and the test in it, e.g. the common group and the error test in it
+            group, _, test = module.name.partition("/")
+
+            if not _selected(config.module, group) or not _selected(config.test, test):
+                continue
+
+            # Skip this test if it does not run on this vm
+            if module.vm_list and vm_name not in module.vm_list:
+                continue
+
+            integration = module.type == TEST_TYPE_INTEGRATION
+
+            # Skip this test if only C tests were requested. An integration test runs the pgbackrest binary rather than a unit
+            # test, so it is the only kind that is not C.
+            if config.c_only and integration:
+                continue
+
+            # Skip this test if it needs a container and there is none. Integration tests always need one.
+            if config.vm == VM_NONE and (integration or module.container_required):
+                continue
+
+            # Skip this test if it is a performance test and those were not requested
+            if not config.performance and module.type == TEST_TYPE_PERFORMANCE:
+                continue
+
+            # Skip this test if it does not need a container and only tests that need one were requested
+            if config.container_only and not integration and not module.container_required:
+                continue
+
+            # Skip this test if only tests that provide coverage were requested
+            if config.coverage_only and not module.coverage_list:
+                continue
+
+            # PostgreSQL versions to test against, most recent first. Only an integration module declares that it needs them.
+            pg_version_list = list(reversed(vm.db_test_list)) if module.pg_required else [None]
+
+            for pg_version in pg_version_list:
+                if pg_version is not None and config.pg_version not in ("all", "minimal", pg_version):
+                    continue
+
+                # An integration test runs each sub-test in its own container, so each one is a separate run here. Every other
+                # test runs all of its sub-tests at once and passes on whatever --run selected.
+                if not integration:
+                    result.append(TestRun(module, vm_name, pg_version, config.run if config.run else None))
+
+                    continue
+
+                for run in range(1, module.total + 1):
+                    if config.run and run not in config.run:
+                        continue
+
+                    result.append(TestRun(module, vm_name, pg_version, [run]))
+
+    return result

test/lib/command/test/test.py

@@ -0,0 +1,432 @@
+"""Test Command.
+
+Selects the tests to run, generates the code they need, builds the pgbackrest binary when a test uses it, and runs the tests. Each
+test runs in a separate process (and its own container when the vm is not none) so several can run at once.
+
+This is what a developer runs, which is why it needs no command. The unit and coverage commands are the steps it drives, and
+they are commands of their own because a test runs inside a container where only the repository copy is available."""
+
+####################################################################################################################################
+import os
+import re
+import time
+
+from command.test.container import container_build, container_remove, container_repo
+from command.test.define import test_def_parse
+from command.test.job import TestJob
+from command.test.list import test_list_get
+from common.error import TestError, check
+from common.exec import Exec, exec_one
+from common.log import *
+from common.storage import file_read, file_write, file_write_differs, path_create
+from common.user import user_name
+from common.vm import *
+
+
+####################################################################################################################################
+def _version_update(path_repo):
+    """Update the version in src/version.h and meson.build.
+
+    The version components in src/version.h are the source of truth, so the version string, the version number, and the version
+    meson builds with are all generated from them rather than being kept in sync by hand."""
+
+    path_version = os.path.join(path_repo, "src/version.h")
+    part = {}
+
+    for name in ("MAJOR", "MINOR", "PATCH", "SUFFIX"):
+        match = re.search(r"^#define PROJECT_VERSION_%s\s+(\S+)$" % name, file_read(path_version), re.M)
+
+        check(match is not None, "unable to find PROJECT_VERSION_%s in src/version.h" % name)
+
+        part[name] = match.group(1)
+
+    version = "%s.%s.%s%s" % (part["MAJOR"], part["MINOR"], part["PATCH"], part["SUFFIX"].strip('"'))
+    version_num = "%d%03d%03d" % (int(part["MAJOR"]), int(part["MINOR"]), int(part["PATCH"]))
+
+    # Version defines in src/version.h
+    result = ""
+
+    for line in file_read(path_version).rstrip("\n").split("\n"):
+        if line.startswith("#define PROJECT_VERSION "):
+            line = "#define PROJECT_VERSION" + " " * 45 + '"%s"' % version
+        elif line.startswith("#define PROJECT_VERSION_NUM "):
+            line = "#define PROJECT_VERSION_NUM" + " " * 41 + version_num
+
+        result += line + "\n"
+
+    file_write_differs(path_version, result)
+
+    # Version meson builds with
+    path_meson = os.path.join(path_repo, "meson.build")
+    result = ""
+
+    for line in file_read(path_meson).rstrip("\n").split("\n"):
+        if line.startswith("    version: '"):
+            line = "    version: '%s'," % version
+
+        result += line + "\n"
+
+    file_write_differs(path_meson, result)
+
+    return version
+
+
+####################################################################################################################################
+def _repo_copy(config):
+    """Copy the repository so a test can tell which files it changed.
+
+    Only version controlled files are copied, and the generated files that the code generation below writes into the copy are left
+    out so a stale copy of them cannot be used."""
+
+    exec_one(
+        "git -C %s ls-files -c --others --exclude-standard |" % config.repo_path
+        + " rsync -rLtW --delete --files-from=- --exclude=test/result"
+        + " --exclude=src/postgres/interface.auto.c.inc --exclude=src/command/help/help.auto.c.inc"
+        # This option is not supported on MacOS. The eventual plan is to remove the need for it.
+        + ("" if exec_one("uname").strip() == "Darwin" else " --ignore-missing-args")
+        + " %s/ %s" % (config.repo_path, os.path.join(config.test_path, "repo"))
+    )
+
+
+####################################################################################################################################
+def _code_generate(config, path_repo_copy):
+    """Generate the code that is built from the definitions in the source.
+
+    This runs on the host rather than in a container since the generator is built for the host either way."""
+
+    path_repo = config.repo_path
+    path_build = os.path.join(config.test_path, "build", VM_NONE)
+    build_clean = not os.path.exists(os.path.join(path_build, "build.ninja"))
+
+    log(INFO, ("clean " if build_clean else "") + "autogenerate code")
+
+    command = "ninja -C %s src/build-code 2>&1" % path_build
+
+    # A dry run does the minimum required, i.e. only what building the test list depends on
+    if not config.dry_run:
+        for generate in ("config", "error", "postgres-version"):
+            command += " && \\\n%s/src/build-code %s %s/src" % (path_build, generate, path_repo)
+
+    command += " && \\\n%s/src/build-code postgres %s/src %s" % (path_build, path_repo, path_repo_copy)
+
+    if build_clean:
+        command = (
+            "meson setup -Dwerror=true -Dfatal-errors=true -Dbuildtype=debug %s %s && \\\n" % (path_build, path_repo) + command
+        )
+
+    exec_one(command)
+
+
+####################################################################################################################################
+def _build(config, bin_required):
+    """Build the pgbackrest binary when a test needs it.
+
+    A unit test is built from the repository copy by the unit command so there is nothing to build for it here, but its build path
+    is mounted into the container so the path must exist either way."""
+
+    path_build = os.path.join(config.test_path, "build", config.vm)
+    build_clean = not os.path.exists(os.path.join(path_build, "build.ninja"))
+
+    log(INFO, ("clean " if build_clean else "") + "build for %s (%s)" % (config.vm, path_build))
+
+    command = "ninja -C %s src/pgbackrest 2>&1" % path_build if bin_required else None
+
+    if build_clean:
+        command = "meson setup -Dwerror=true -Dfatal-errors=true -Dbuildtype=debug %s %s" % (path_build, config.repo_path) + (
+            "" if command is None else " && \\\n" + command
+        )
+
+    # There is nothing to do when the build is already set up and no binary is needed
+    if command is None:
+        return
+
+    if config.vm != VM_NONE:
+        command = "docker exec -i -u %s test-build bash -c '%s'" % (user_name(), command)
+
+    exec_one(command, show_output=config.log_level >= DETAIL)
+
+
+####################################################################################################################################
+def _run(config, test_list, image):
+    """Run the tests, keeping every vm busy, and report how many failed and how many were retried."""
+
+    process_list = [None] * config.vm_max
+    show_output = config.vm_out and (len(test_list) == 1 or config.vm_max == 1) and not config.dry_run
+    count_fail = 0
+    count_retry = 0
+    idx_test = 0
+
+    while True:
+        # Wait for a vm to free up
+        while True:
+            active = 0
+
+            for idx in range(config.vm_max):
+                job = process_list[idx]
+
+                if job is None:
+                    continue
+
+                done, fail = job.end()
+
+                if not done:
+                    active += 1
+
+                    continue
+
+                process_list[idx] = None
+
+                if not fail:
+                    continue
+
+                # Start the test again when it has a retry left, else count it as a failure
+                if job.begin():
+                    process_list[idx] = job
+                    count_retry += 1
+                    active += 1
+                else:
+                    count_fail += 1
+
+            # Only wait when every vm is busy or every test has been assigned, otherwise there is something to do
+            if active == config.vm_max or idx_test == len(test_list):
+                time.sleep(0.05)
+
+            if active != config.vm_max:
+                break
+
+        # Assign tests to the vms that are free
+        for idx in range(config.vm_max):
+            if process_list[idx] is not None or idx_test == len(test_list):
+                continue
+
+            job = TestJob(config, test_list[idx_test], idx, config.vm_max, idx_test, len(test_list), image, show_output)
+            idx_test += 1
+            active += 1
+
+            if job.begin():
+                process_list[idx] = job
+
+        if active == 0:
+            break
+
+    return count_fail, count_retry
+
+
+####################################################################################################################################
+def _coverage(config, test_list):
+    """Merge the coverage the tests produced and write the report, reporting whether any module is missing coverage."""
+
+    command = Exec(
+        "python3 %s/test/test.py coverage --log-level=warn --vm=%s --repo-path=%s --test-path=%s%s %s"
+        % (
+            config.repo_path,
+            config.vm,
+            config.repo_path,
+            config.test_path,
+            " --coverage-summary" if config.coverage_summary else "",
+            " ".join(run.module.name for run in test_list),
+        ),
+        show_output=True,
+    )
+    command.begin()
+    status = command.end()
+
+    # The coverage command reports incomplete coverage with a status of one, which is not an error here since the tests all passed
+    if status > 1:
+        raise TestError("coverage command failed", status)
+
+    if status == 0:
+        log(INFO, "tested modules have full coverage")
+
+    return status == 1
+
+
+####################################################################################################################################
+def cmd_test(config):
+    """Run the tests."""
+
+    time_begin = time.time()
+    path_repo = config.repo_path
+    path_repo_copy = os.path.join(config.test_path, "repo")
+
+    # Set a neutral umask so tests work as expected
+    os.umask(0)
+
+    log(INFO, "test begin on %s - log level %s" % (host_arch(), LEVEL_NAME[config.log_level]))
+
+    # The coverage summary is generated from the C tests that provide coverage, so it selects them
+    if config.coverage_summary:
+        config.coverage_only = True
+        config.c_only = True
+
+    # Profiling needs a build without the instrumentation that would skew the timing
+    if config.profile:
+        config.back_trace = False
+        config.valgrind = False
+        config.coverage = False
+
+    # Check the options that the parser cannot
+    check(len(config.run) <= 1, "only one --run can be provided")
+    check(not config.test or len(config.module) == 1, "only one --module can be provided when --test is specified")
+    check(not config.run or len(config.test) == 1, "only one --test can be provided when --run is specified")
+    check(not config.vm_build or config.vm != VM_NONE, "select a vm to build, or all of them")
+
+    # Every vm can be built at once but a test runs on one vm, since it needs the binary and the build for the vm it runs on
+    check(config.vm != VM_ALL or config.vm_build, "select a single vm to test on")
+
+    # Check the vm now so a typo is reported before anything is generated or built
+    if config.vm != VM_ALL:
+        vm_get(config.vm)
+
+    # The test path holds everything the tests generate, which would be a mess to sort out from the repository
+    check(
+        not (config.test_path + "/").startswith(path_repo + "/"),
+        "test path '%s' may not be in the repo path '%s'\n" % (config.test_path, path_repo)
+        + "HINT: was test.py run in '%s'?\n" % path_repo
+        + "HINT: use --test-path to set a test path\n"
+        + "HINT: run test.py from outside the repo, e.g. 'pgbackrest/test/test.py'",
+    )
+
+    # Clean working and result paths
+    if config.clean or config.clean_only:
+        log(INFO, "clean working (%s) and result (%s/test/result) paths" % (config.test_path, path_repo))
+
+        for path in (config.test_path, os.path.join(path_repo, "test/result")):
+            if os.path.exists(path):
+                exec_one("find %s -mindepth 1 -print0 | xargs -0 rm -rf" % path)
+
+        if config.clean_only:
+            return 0
+
+    # Build the containers
+    if config.vm_build:
+        container_build(config)
+
+        return 0
+
+    # Load the test definitions
+    module_list = test_def_parse(path_repo)
+
+    # Clean up data left by the prior run
+    coverage_c = vm_get(config.vm).coverage_c
+
+    if not config.dry_run or config.vm_out:
+        log(INFO, "cleanup old data" + (" and containers" if config.vm != VM_NONE else ""))
+
+        if config.vm != VM_NONE:
+            container_remove("test-([0-9]+|build)")
+
+        exec_one(
+            "chmod 700 -R %s/test-* 2>&1 || true && rm -rf %s/temp %s/test-* %s/data-*"
+            % (config.test_path, config.test_path, config.test_path, config.test_path)
+        )
+        path_create(os.path.join(config.test_path, "temp"), mode=0o770)
+
+        # Overwrite the C coverage report so it will load but not show old coverage
+        path_create(os.path.join(path_repo, "test/result/coverage"), mode=0o770)
+        file_write(
+            os.path.join(path_repo, "test/result/coverage/coverage.html"),
+            "<center>[ %s ]</center>" % ("Generating Coverage Report" if config.coverage else "No Coverage Testing"),
+        )
+
+        # Clear the coverage the prior run produced
+        if coverage_c and not config.dry_run:
+            exec_one("rm -rf %s/test/result/coverage/raw/*" % path_repo)
+            path_create(os.path.join(path_repo, "test/result/coverage/raw"), mode=0o770)
+
+    _version_update(path_repo)
+
+    # A single ccache is shared by the host and all containers so a file compiled in one is a hit in the others, e.g. a module
+    # compiled for one unit test is a hit for every other unit test that includes it. ccache is safe for concurrent access so there
+    # is no need to keep a separate cache per vm or vm index. Keep the cache in the test path so it is removed by --clean.
+    path_ccache = os.path.join(config.test_path, "ccache")
+    path_create(path_ccache, mode=0o770)
+
+    # Set for builds that run on the host. Containers are passed the path explicitly since docker does not pass the environment.
+    os.environ["CCACHE_DIR"] = path_ccache
+
+    # Start the build container when the tests run in one
+    image = "%s:%s-test-%s" % (container_repo(), config.vm, config.vm_arch if config.vm_arch is not None else host_arch())
+
+    if config.vm != VM_NONE:
+        # The cache does not need to be mounted since it is in the test path, which is mounted below
+        exec_one(
+            "docker run%s -itd -h test-build --name=test-build -v %s:%s -v %s:%s -e CCACHE_DIR=%s %s"
+            % (
+                "" if config.vm_arch is None else " --platform linux/%s" % config.vm_arch,
+                path_repo,
+                path_repo,
+                config.test_path,
+                config.test_path,
+                path_ccache,
+                image,
+            )
+        )
+
+    # Create the repository copy path first so an obvious rsync error is reported rather than a confusing one
+    path_create(path_repo_copy, mode=0o770)
+
+    _code_generate(config, path_repo_copy)
+
+    if config.gen_only:
+        return 0
+
+    _repo_copy(config)
+
+    # Determine which tests to run and what they need built
+    test_list = []
+    bin_required = config.build_only
+    unit_required = config.build_only
+
+    if not config.build_only:
+        test_list = test_list_get(module_list, config)
+
+        # The binary is required for an integration test and for any test that asks for it. The build path is mounted into the
+        # container of every other test, so it must exist even when nothing is built there.
+        bin_required = any(run.integration or run.module.bin_required for run in test_list)
+        unit_required = any(not run.integration for run in test_list)
+
+    if not config.dry_run:
+        if bin_required or unit_required:
+            _build(config, bin_required)
+
+        # Shut down the build container
+        if config.vm != VM_NONE:
+            exec_one("docker rm -f test-build")
+
+        if config.build_only:
+            return 0
+
+    if not test_list:
+        raise TestError("no tests were selected")
+
+    log(INFO, "%u test%s selected\n" % (len(test_list), "" if len(test_list) == 1 else "s"))
+
+    # Preserving the results of more than one test is not possible since they share a path
+    check(config.cleanup or len(test_list) == 1, "--no-cleanup is not valid when more than one test will run")
+
+    # Only use one vm for a dry run so the tests are listed in order
+    if config.dry_run:
+        config.vm_max = 1
+
+    count_fail, count_retry = _run(config, test_list, image)
+
+    # Write the coverage report. There is nothing to report when a single run was selected since that cannot cover a module.
+    uncovered = False
+
+    if coverage_c and config.coverage and not config.dry_run and count_fail == 0 and not config.run:
+        uncovered = _coverage(config, test_list)
+
+    log(
+        INFO,
+        ("DRY RUN COMPLETED" if config.dry_run else "TESTS COMPLETED")
+        + (
+            " SUCCESSFULLY" + (" WITH MODULE(S) MISSING COVERAGE" if uncovered else "")
+            if count_fail == 0
+            else " WITH %u FAILURE(S)" % count_fail
+        )
+        + ("" if count_retry == 0 else ", %u RETRY(IES)" % count_retry)
+        + ("" if not config.log_timestamp else " (%us)" % int(time.time() - time_begin)),
+    )
+
+    return 1 if count_fail > 0 or (uncovered and not config.coverage_summary) else 0

test/lib/command/test/unit.py

@@ -16,21 +16,9 @@ from command.test.build import TestBuild
 from command.test.define import TEST_LANG_PYTHON, TEST_TYPE_PERFORMANCE, test_def_find, test_def_parse
 from common.error import TestError, check
 from common.exec import exec_one
-from common.log import DETAIL, WARN, log
+from common.log import *
 from common.storage import file_remove, path_create, path_list
-
-
-####################################################################################################################################
-def _arch_fix(arch):
-    """Normalize the architecture reported by uname to the names the project uses."""
-
-    if arch == "i686":
-        return "i386"
-
-    if arch == "arm64":
-        return "aarch64"
-
-    return arch
+from common.vm import *
 
 
 ####################################################################################################################################
@@ -116,10 +104,7 @@ def cmd_unit(config):
         return
 
     # Get test architecture
-    architecture = config.vm_arch
-
-    if architecture is None:
-        architecture = _arch_fix(exec_one("uname -m").strip())
+    architecture = config.vm_arch if config.vm_arch is not None else host_arch()
 
     test_build = TestBuild(config, module, architecture)
     path_unit = test_build.path_unit

test/lib/common/exec.py

@@ -1,28 +1,134 @@
 """Execute Process.
 
-Commands run through the shell with stderr folded into stdout, matching execOneExpectP() in src/build/common/exec.c. The combined
-output is returned on success and carried in the error on failure, since a failed meson or ninja run says what went wrong there
-rather than in its exit status."""
+Exec runs a command in the background so several can run at once, which is how the test driver keeps every vm busy. It keeps stdout
+and stderr apart because a test that writes anything to stderr has failed whatever its exit status, e.g. a valgrind error.
+
+exec_one() runs a command and waits for it, with stderr folded into the output as execOneExpectP() in src/build/common/exec.c does.
+The combined output is returned on success and carried in the error on failure, since a failed meson or ninja run says what went
+wrong there rather than in its exit status."""
 
 ####################################################################################################################################
+import os
+import select
 import subprocess
+import sys
 
 from common.error import TestError
 
 
 ####################################################################################################################################
-def exec_one(command, result_expect=0):
+class Exec:
+    """Run a command in the background and collect its output."""
+
+    def __init__(self, command, show_output=False, merge_error=False):
+        self.command = command
+        self.show_output = show_output  # Write output as it arrives rather than only keeping it?
+        self.merge_error = merge_error  # Fold stderr into the output rather than keeping it apart?
+        self.output = ""
+        self.error = ""
+
+        self._process = None
+        self._fd_out = None
+        self._fd_list = []
+        self._show_idx = 0  # How much of the output has been written when show_output is set
+
+    ################################################################################################################################
+    def begin(self):
+        """Start the command."""
+
+        self._process = subprocess.Popen(
+            ["sh", "-c", self.command],
+            stdout=subprocess.PIPE,
+            stderr=subprocess.STDOUT if self.merge_error else subprocess.PIPE,
+        )
+        self._fd_out = self._process.stdout.fileno()
+        self._fd_list = [self._fd_out] + ([] if self.merge_error else [self._process.stderr.fileno()])
+
+    ################################################################################################################################
+    def _show(self):
+        """Write the complete lines that have arrived since the last call, indented the way the log indents output."""
+
+        idx = self.output.rfind("\n") + 1
+
+        if idx > self._show_idx:
+            for line in self.output[self._show_idx : idx].splitlines():
+                sys.stdout.write("    %s\n" % line)
+
+            sys.stdout.flush()
+            self._show_idx = idx
+
+    ################################################################################################################################
+    def _drain(self, timeout):
+        """Read whatever is available on stdout and stderr, dropping a stream when it reaches end of file.
+
+        A timeout of zero returns immediately and None waits for data, which is only safe once the command has exited."""
+
+        for fd in select.select(self._fd_list, [], [], timeout)[0]:
+            data = os.read(fd, 65536)
+
+            # No data on a ready stream means end of file
+            if not data:
+                self._fd_list.remove(fd)
+
+                continue
+
+            if fd == self._fd_out:
+                self.output += data.decode(errors="replace")
+
+                if self.show_output:
+                    self._show()
+            else:
+                self.error += data.decode(errors="replace")
+
+    ################################################################################################################################
+    def end(self, wait=True):
+        """Return the exit status, or None when the command is still running and wait is not set.
+
+        Output is drained while the command runs so a full pipe buffer cannot block it."""
+
+        while self._process.poll() is None:
+            self._drain(0.05 if wait else 0)
+
+            if not wait:
+                return None
+
+        # The command is done so read what is left. Waiting for data is safe here since the only writer has exited.
+        while self._fd_list:
+            self._drain(None)
+
+        self._process.stdout.close()
+
+        if not self.merge_error:
+            self._process.stderr.close()
+
+        return self._process.returncode
+
+
+####################################################################################################################################
+def exec_one(command, result_expect=0, show_output=False):
     """Run a command through the shell and return its combined output.
 
     Raise TestError when the exit status is not the expected one."""
 
-    process = subprocess.run(
-        ["sh", "-c", command + " 2>&1"], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True
-    )
+    process = Exec(command, show_output=show_output, merge_error=True)
+    process.begin()
+    status = process.end()
 
-    if process.returncode != result_expect:
-        output = process.stdout.strip()
+    if status != result_expect:
+        output = process.output.strip()
 
-        raise TestError("%s terminated unexpectedly [%d]%s%s" % (command, process.returncode, ": " if output else "", output))
+        raise TestError("%s terminated unexpectedly [%d]%s%s" % (command, status, ": " if output else "", output))
 
-    return process.stdout
+    return process.output
+
+
+####################################################################################################################################
+def exec_status(command):
+    """Run a command through the shell and return its exit status, discarding the output.
+
+    Used where a command is expected to fail, e.g. pulling an image that is not in the cache."""
+
+    process = Exec(command)
+    process.begin()
+
+    return process.end()

test/lib/common/user.py

@@ -0,0 +1,37 @@
+"""User and Group.
+
+The user the tests run as. The container is built with this user so files created inside it are owned by the user on the host, and
+the generated test.c is given the same names so a test can check ownership against what it will find at runtime."""
+
+####################################################################################################################################
+import grp
+import os
+import pwd
+
+
+####################################################################################################################################
+def user_id():
+    """Id of the user the tests run as."""
+
+    return os.getuid()
+
+
+####################################################################################################################################
+def user_name():
+    """Name of the user the tests run as."""
+
+    return pwd.getpwuid(user_id()).pw_name
+
+
+####################################################################################################################################
+def group_id():
+    """Id of the group the tests run as."""
+
+    return os.getgid()
+
+
+####################################################################################################################################
+def group_name():
+    """Name of the group the tests run as."""
+
+    return grp.getgrgid(group_id()).gr_name

test/lib/common/vm.py

@@ -0,0 +1,153 @@
+"""Virtual Machine Definitions.
+
+The vms tests can run on, the PostgreSQL versions each one provides, and what the container build needs to know about them. A vm
+name is what --vm accepts, e.g. u24 for Ubuntu 24.04.
+
+The default list is checked when this module is imported: every supported PostgreSQL version must be tested on exactly one default
+vm and at least one default vm must collect C coverage. Without the check a new version could be added and silently never tested."""
+
+####################################################################################################################################
+import platform
+
+from common.error import TestError, check
+
+# Os base a vm is built on, which determines its package manager and where PostgreSQL is installed
+VM_OS_BASE_ALPINE = "alpine"
+VM_OS_BASE_DEBIAN = "debian"
+VM_OS_BASE_RHEL = "rhel"
+
+# Architectures a container can be built for
+VM_ARCH_AARCH64 = "aarch64"
+VM_ARCH_I386 = "i386"
+VM_ARCH_PPC64LE = "ppc64le"
+VM_ARCH_S390X = "s390x"
+VM_ARCH_X86_64 = "x86_64"
+
+VM_ARCH_LIST = (VM_ARCH_AARCH64, VM_ARCH_I386, VM_ARCH_PPC64LE, VM_ARCH_S390X, VM_ARCH_X86_64)
+
+# All default vms and no vm at all, i.e. run directly on the host
+VM_ALL = "all"
+VM_NONE = "none"
+
+# Vms
+VM_A321 = "a321"
+VM_A324 = "a324"
+VM_D12 = "d12"
+VM_F44 = "f44"
+VM_RH8 = "rh8"
+VM_RH9 = "rh9"
+VM_RH10 = "rh10"
+VM_U22 = "u22"
+VM_U24 = "u24"
+
+# Default vms, i.e. the ones --vm=all selects
+VM_LIST = (VM_D12, VM_RH8, VM_RH9, VM_RH10, VM_U22, VM_U24, VM_A321, VM_A324)
+
+# PostgreSQL versions the project supports. Each one must be tested on a default vm, which vm_check() enforces.
+PG_VERSION_LIST = ("9.6", "10", "11", "12", "13", "14", "15", "16", "17", "18", "19")
+
+
+####################################################################################################################################
+class Vm:
+    """A vm definition."""
+
+    def __init__(self, os_base, db_list, image=None, db_test_list=None, pg_repo=True, coverage_c=False):
+        self.os_base = os_base
+        self.image = image  # Image the container is built from, None when the vm has no container
+        self.pg_repo = pg_repo  # Install PostgreSQL from the PGDG repo rather than from the distribution?
+        self.coverage_c = coverage_c  # Is C coverage collected on this vm?
+        self.db_list = db_list  # PostgreSQL versions installed
+        self.db_test_list = db_list if db_test_list is None else db_test_list  # Versions integration tests run against
+
+
+####################################################################################################################################
+# fmt: off
+_VM = {
+    # No container, i.e. run directly on the host
+    VM_NONE: Vm(VM_OS_BASE_DEBIAN, ["10"], coverage_c=True),
+
+    # Alpine 3.21
+    VM_A321: Vm(VM_OS_BASE_ALPINE, ["15", "16", "17"], image="alpine:3.21", db_test_list=["16"], pg_repo=False),
+
+    # Alpine 3.24
+    VM_A324: Vm(VM_OS_BASE_ALPINE, ["16", "17", "18"], image="alpine:3.24", db_test_list=["17"], pg_repo=False),
+
+    # Debian 12
+    VM_D12: Vm(VM_OS_BASE_DEBIAN, ["15"], image="debian:12", pg_repo=False),
+
+    # RHEL 8
+    VM_RH8: Vm(VM_OS_BASE_RHEL, ["10"], image="rockylinux/rockylinux:8", pg_repo=False),
+
+    # RHEL 9
+    VM_RH9: Vm(VM_OS_BASE_RHEL, ["14", "15", "16", "17", "18"], image="rockylinux/rockylinux:9", db_test_list=["14"]),
+
+    # RHEL 10
+    VM_RH10: Vm(VM_OS_BASE_RHEL, ["14", "15", "16", "17", "18"], image="rockylinux/rockylinux:10", db_test_list=["18"]),
+
+    # Fedora 44
+    VM_F44: Vm(
+        VM_OS_BASE_RHEL, ["14", "15", "16", "17", "18"], image="fedora:44", db_test_list=["15"], coverage_c=True),
+
+    # Ubuntu 22.04
+    VM_U22: Vm(
+        VM_OS_BASE_DEBIAN, ["9.6", "10", "11", "12", "13", "14"], image="ubuntu:22.04", db_test_list=["9.6", "11"],
+        coverage_c=True),
+
+    # Ubuntu 24.04
+    VM_U24: Vm(
+        VM_OS_BASE_DEBIAN, ["9.6", "10", "11", "12", "13", "14", "15", "16", "17", "18", "19"], image="ubuntu:24.04",
+        db_test_list=["12", "13", "19"], coverage_c=True),
+}
+# fmt: on
+
+
+####################################################################################################################################
+def vm_valid(vm):
+    """Is there a definition for this vm?"""
+
+    return vm in _VM
+
+
+####################################################################################################################################
+def vm_get(vm):
+    """The definition for a vm, which also validates that the vm exists."""
+
+    if not vm_valid(vm):
+        raise TestError("no definition for vm '%s'" % vm)
+
+    return _VM[vm]
+
+
+####################################################################################################################################
+def vm_check(vm_map):
+    """Check that C coverage and every supported PostgreSQL version are tested on a default vm.
+
+    Called below on the definitions in this module, so a version that no vm tests is an error rather than a silent gap."""
+
+    check(any(vm_map[vm].coverage_c for vm in VM_LIST), "C coverage is not configured to run on a default vm")
+
+    for version in PG_VERSION_LIST:
+        vm_found = None
+
+        for vm in VM_LIST:
+            if version in vm_map[vm].db_test_list:
+                check(vm_found is None, "PostgreSQL %s is already configured to run on default vm %s" % (version, vm_found))
+
+                vm_found = vm
+
+        check(vm_found is not None, "PostgreSQL %s is not configured to run on a default vm" % version)
+
+
+####################################################################################################################################
+def host_arch():
+    """Architecture of the host, using the names the project uses.
+
+    Mac reports arm64 where Linux reports aarch64 and 32-bit x86 reports i686 where the project uses i386."""
+
+    arch = platform.machine()
+
+    return {"arm64": VM_ARCH_AARCH64, "i686": VM_ARCH_I386}.get(arch, arch)
+
+
+####################################################################################################################################
+vm_check(_VM)

test/lib/config/cli.py

@@ -1,15 +1,19 @@
 """Command Line Interface.
 
-The commands are the steps the Perl test framework drives:
+Running the tests is what a developer does so it needs no command, e.g. test.py --vm=u24 --module=common. The commands are the
+steps that run underneath it and the ones that are not about running tests at all:
 
-unit        build the unit test for one test module, run once per test inside its container
-coverage    merge the coverage produced by the unit tests and write the report
-lint        run the source linter on its own"""
+unit        build the unit test for one test module and run it, run once per test by the test run above
+coverage    merge the coverage the unit tests produced and write the report
+lint        run the source linter on its own
+code-format format the source to the project standards, or check that it is
+
+A command parses into the same configuration as the test run, so a command option always wins over one given before the command."""
 
 ####################################################################################################################################
 import argparse
 
-from common.log import LEVEL_NAME
+from common.log import *
 
 
 ####################################################################################################################################
@@ -18,7 +22,6 @@ def _parser_common():
 
     result = argparse.ArgumentParser(add_help=False)
 
-    result.add_argument("--repo-path", default="pgbackrest", metavar="PATH", help="code repository path")
     result.add_argument(
         "--log-level", default="info", choices=sorted(LEVEL_NAME.values()), metavar="LEVEL", help="console log level"
     )
@@ -27,6 +30,19 @@ def _parser_common():
     return result
 
 
+####################################################################################################################################
+def _parser_repo(parent):
+    """Options for the commands that are told where the repository is.
+
+    A test run is not told, since it works on the repository it is part of rather than on the copy the tests are built from."""
+
+    result = argparse.ArgumentParser(add_help=False, parents=[parent])
+
+    result.add_argument("--repo-path", default="pgbackrest", metavar="PATH", help="code repository path")
+
+    return result
+
+
 ####################################################################################################################################
 def _parser_test_path(parent):
     """Options shared by the commands that work in the test path."""
@@ -39,38 +55,94 @@ def _parser_test_path(parent):
     return result
 
 
+####################################################################################################################################
+def _parser_test(result):
+    """Options for running the tests, which select the tests, build what they need, and run them.
+
+    These are on the main parser rather than on a command of their own so the tests can be run without naming a command."""
+
+    # Test selection
+    result.add_argument("--module", action="append", default=[], metavar="MODULE", help="module to test")
+    result.add_argument("--test", action="append", default=[], metavar="TEST", help="test to run in a module")
+    result.add_argument("--run", action="append", type=int, default=[], metavar="RUN", help="run to execute in a test")
+    result.add_argument("--pg-version", default="minimal", metavar="VERSION", help="pg version integration tests run against")
+    result.add_argument("--c-only", action="store_true", help="only run C tests")
+    result.add_argument("--container-only", action="store_true", help="only run tests that require a container")
+    result.add_argument("--coverage-only", action="store_true", help="only run tests that provide coverage")
+    result.add_argument("--no-performance", dest="performance", action="store_false", help="do not run performance tests")
+    result.add_argument("--dry-run", action="store_true", help="show the tests that would run without running them")
+
+    # What to build and how
+    result.add_argument("--build-only", action="store_true", help="build the binary but do not run tests")
+    result.add_argument("--gen-only", action="store_true", help="only generate code")
+    result.add_argument("--no-back-trace", dest="back_trace", action="store_false", help="do not build with back trace")
+    result.add_argument("--no-valgrind", dest="valgrind", action="store_false", help="do not run the C tests with valgrind")
+    result.add_argument("--no-coverage", dest="coverage", action="store_false", help="do not collect coverage")
+    result.add_argument("--coverage-summary", action="store_true", help="write the coverage summary for the documentation")
+    result.add_argument("--profile", action="store_true", help="build with profiling and write the profile")
+    result.add_argument("--scale", type=int, default=1, metavar="FACTOR", help="scale performance tests")
+    result.add_argument("--tz", metavar="TZ", help="run the tests in the specified timezone")
+
+    # Paths and logging
+    result.add_argument("--clean", action="store_true", help="clean the working and result paths before running")
+    result.add_argument("--clean-only", action="store_true", help="clean the working and result paths and exit")
+    result.add_argument("--no-cleanup", dest="cleanup", action="store_false", help="do not clean up after a test, for debugging")
+    result.add_argument(
+        "--log-level-test", default="off", choices=sorted(LEVEL_NAME.values()), metavar="LEVEL", help="test log level"
+    )
+    result.add_argument("-q", "--quiet", action="store_true", help="equivalent to --log-level=off")
+
+    # Vm
+    result.add_argument("--vm-arch", metavar="ARCH", help="vm architecture (defaults to the host architecture)")
+    result.add_argument("--vm-build", action="store_true", help="build the vm containers and exit")
+    result.add_argument("--vm-force", action="store_true", help="force a rebuild of the vm containers")
+    result.add_argument("--vm-out", action="store_true", help="show the output of the tests")
+    result.add_argument("--vm-max", type=int, default=1, metavar="COUNT", help="max vms to run in parallel")
+    result.add_argument("--retry", type=int, default=0, metavar="COUNT", help="retry a failed test this many times")
+
+
+####################################################################################################################################
+def _parser_unit(command, parent):
+    """The unit command, which builds one test module and runs it when it is python."""
+
+    result = command.add_parser(
+        "unit", parents=[parent], help="build the unit test for a test module", description="Build a unit test."
+    )
+
+    result.add_argument("module", metavar="module/test", help="test module to build")
+    result.add_argument("--vm-id", type=int, default=0, metavar="ID", help="0-based id of the vm the test runs on")
+    result.add_argument("--vm-arch", metavar="ARCH", help="vm architecture (defaults to the host architecture)")
+    result.add_argument("--pg-version", default="invalid", metavar="VERSION", help="pg version for integration tests")
+    result.add_argument("--test", type=int, metavar="RUN", help="run only the specified test run")
+    result.add_argument("--test-name", metavar="LIST", help="comma separated tests to run in a python test module")
+    result.add_argument("--coverage-file", metavar="PATH", help="write coverage for a python test module to this path")
+    result.add_argument("--scale", type=int, default=1, metavar="FACTOR", help="scale performance tests")
+    result.add_argument("--tz", metavar="TZ", help="run the test in the specified timezone")
+    result.add_argument(
+        "--log-level-test", default="off", choices=sorted(LEVEL_NAME.values()), metavar="LEVEL", help="test log level"
+    )
+    result.add_argument("--no-coverage", dest="coverage", action="store_false", help="do not build with coverage")
+    result.add_argument("--no-back-trace", dest="back_trace", action="store_false", help="do not build with back trace")
+    result.add_argument("--optimize", action="store_true", help="build with optimization")
+    result.add_argument("--profile", action="store_true", help="build with profiling")
+
+
 ####################################################################################################################################
 def cli_parse(arg_list, version):
     """Build the parser and parse the command line."""
 
     common = _parser_common()
-    test_path = _parser_test_path(common)
+    repo = _parser_repo(common)
+    test_path = _parser_test_path(repo)
 
-    parser = argparse.ArgumentParser(prog="test.py", description="pgBackRest Test Harness")
+    parser = argparse.ArgumentParser(prog="test.py", description="pgBackRest Test Harness", parents=[_parser_test_path(common)])
     parser.add_argument("--version", action="version", version="pgBackRest %s Test Harness" % version)
-    command = parser.add_subparsers(dest="command", metavar="command", required=True)
 
-    # Unit
-    # ---------------------------------------------------------------------------------------------------------------------------
-    unit = command.add_parser(
-        "unit", parents=[test_path], help="build the unit test for a test module", description="Build a unit test."
-    )
-    unit.add_argument("module", metavar="module/test", help="test module to build")
-    unit.add_argument("--vm-id", type=int, default=0, metavar="ID", help="0-based id of the vm the test runs on")
-    unit.add_argument("--vm-arch", metavar="ARCH", help="vm architecture (defaults to the host architecture)")
-    unit.add_argument("--pg-version", default="invalid", metavar="VERSION", help="pg version for integration tests")
-    unit.add_argument("--test", type=int, metavar="RUN", help="run only the specified test run")
-    unit.add_argument("--test-name", metavar="LIST", help="comma separated tests to run in a python test module")
-    unit.add_argument("--coverage-file", metavar="PATH", help="write coverage for a python test module to this path")
-    unit.add_argument("--scale", type=int, default=1, metavar="FACTOR", help="scale performance tests")
-    unit.add_argument("--tz", metavar="TZ", help="run the test in the specified timezone")
-    unit.add_argument(
-        "--log-level-test", default="off", choices=sorted(LEVEL_NAME.values()), metavar="LEVEL", help="test log level"
-    )
-    unit.add_argument("--no-coverage", dest="coverage", action="store_false", help="do not build with coverage")
-    unit.add_argument("--no-back-trace", dest="back_trace", action="store_false", help="do not build with back trace")
-    unit.add_argument("--optimize", action="store_true", help="build with optimization")
-    unit.add_argument("--profile", action="store_true", help="build with profiling")
+    _parser_test(parser)
+
+    command = parser.add_subparsers(dest="command", metavar="command")
+
+    _parser_unit(command, test_path)
 
     # Coverage
     # ---------------------------------------------------------------------------------------------------------------------------
@@ -85,6 +157,16 @@ def cli_parse(arg_list, version):
 
     # Lint
     # ---------------------------------------------------------------------------------------------------------------------------
-    command.add_parser("lint", parents=[common], help="lint the source", description="Run the source linter.")
+    command.add_parser("lint", parents=[repo], help="lint the source", description="Run the source linter.")
+
+    # Code format
+    # ---------------------------------------------------------------------------------------------------------------------------
+    code_format = command.add_parser(
+        "code-format",
+        parents=[common],
+        help="format the source to project standards",
+        description="Format the source to project standards, or check that it is already formatted.",
+    )
+    code_format.add_argument("--check", action="store_true", help="check the formatting rather than changing it")
 
     return parser.parse_args(arg_list)

test/lib/config/config.py

@@ -6,10 +6,13 @@ import os
 import re
 
 from common.error import TestError
-from common.log import log_init, log_level_parse
+from common.log import *
 from common.storage import file_read
 from config.cli import cli_parse
 
+# Name of the project executable, which is also the name of the repository and of the container registry namespace
+PROJECT_EXE = "pgbackrest"
+
 
 ####################################################################################################################################
 def project_version(path_repo):
@@ -31,22 +34,19 @@ def cfg_load(arg_list, path_harness):
 
     config = cli_parse(arg_list, project_version(path_harness))
 
-    # Make relative paths absolute so the harness does not depend on the working directory
-    config.repo_path = os.path.abspath(config.repo_path)
+    # A test run works on the repository it is part of. A command is told where the repository is, since it runs from the copy the
+    # tests are built from. Relative paths are made absolute so nothing depends on the working directory.
+    config.repo_path = os.path.abspath(config.repo_path) if hasattr(config, "repo_path") else path_harness
+    config.test_path = os.path.abspath(config.test_path)
 
     # The repository is only ever read from so it must already exist. Checking here keeps a mistyped path from looking like success
     # rather than an error, e.g. the linter would find no files to check and report that everything passed.
     if not os.path.isdir(config.repo_path):
         raise TestError("repo path '%s' does not exist" % config.repo_path)
 
-    if hasattr(config, "test_path"):
-        config.test_path = os.path.abspath(config.test_path)
-
     # Convert log levels to ids now that they are known to be valid
-    config.log_level = log_level_parse(config.log_level)
-
-    if hasattr(config, "log_level_test"):
-        config.log_level_test = log_level_parse(config.log_level_test)
+    config.log_level = OFF if config.quiet else log_level_parse(config.log_level)
+    config.log_level_test = log_level_parse(config.log_level_test)
 
     log_init(config.log_level, config.log_timestamp)
 

test/lib/pgBackRestTest/Common/BuildTest.pm

@@ -1,57 +0,0 @@
-####################################################################################################################################
-# Build Binaries and Auto-Generate Code
-####################################################################################################################################
-package pgBackRestTest::Common::BuildTest;
-
-####################################################################################################################################
-# Perl includes
-####################################################################################################################################
-use strict;
-use warnings FATAL => qw(all);
-use Carp qw(confess);
-use English '-no_match_vars';
-
-# use Cwd qw(abs_path);
-use Exporter qw(import);
-    our @EXPORT = qw();
-
-use pgBackRestDoc::Common::Log;
-use pgBackRestDoc::Common::String;
-
-####################################################################################################################################
-# Save contents to a file if the file is missing or the contents are different. This saves write IO and prevents the timestamp from
-# changing.
-####################################################################################################################################
-sub buildPutDiffers
-{
-    my $oStorage = shift;
-    my $strFile = shift;
-    my $strContentNew = shift;
-
-    # Attempt to load the file
-    my $bSave = true;
-    my $oFile = $oStorage->openRead($strFile, {bIgnoreMissing => true});
-
-    # If file was found see if the content is the same
-    if (defined($oFile))
-    {
-        my $strContentFile = ${$oStorage->get($oFile)};
-
-        if ((defined($strContentFile) ? $strContentFile : '') eq (defined($strContentNew) ? $strContentNew : ''))
-        {
-            $bSave = false;
-        }
-    }
-
-    # Save if the contents are different or missing
-    if ($bSave)
-    {
-        $oStorage->put($oStorage->openWrite($strFile, {bPathCreate => true}), $strContentNew);
-    }
-
-    # Was the file saved?
-    return $bSave;
-}
-
-push @EXPORT, qw(buildPutDiffers);
-

test/lib/pgBackRestTest/Common/ContainerTest.pm

@@ -1,880 +0,0 @@
-####################################################################################################################################
-# ContainerTest.pm - Build containers for testing and documentation
-####################################################################################################################################
-package pgBackRestTest::Common::ContainerTest;
-
-####################################################################################################################################
-# Perl includes
-####################################################################################################################################
-use strict;
-use warnings FATAL => qw(all);
-use Carp qw(confess longmess);
-use English '-no_match_vars';
-
-use Cwd qw(abs_path);
-use Digest::SHA qw(sha1_hex);
-use Exporter qw(import);
-    our @EXPORT = qw();
-use File::Basename qw(dirname);
-use Getopt::Long qw(GetOptions);
-
-use pgBackRestDoc::Common::Log;
-use pgBackRestDoc::Common::String;
-use pgBackRestDoc::ProjectInfo;
-
-use pgBackRestTest::Common::ExecuteTest;
-use pgBackRestTest::Common::VmTest;
-
-####################################################################################################################################
-# User/group definitions
-####################################################################################################################################
-use constant TEST_USER                                              => getpwuid($UID) . '';
-    push @EXPORT, qw(TEST_USER);
-use constant TEST_USER_ID                                           => $UID;
-    push @EXPORT, qw(TEST_USER_ID);
-use constant TEST_GROUP                                             => getgrgid((getpwnam(TEST_USER))[3]) . '';
-    push @EXPORT, qw(TEST_GROUP);
-use constant TEST_GROUP_ID                                          => getgrnam(TEST_GROUP) . '';
-    push @EXPORT, qw(TEST_GROUP_ID);
-
-####################################################################################################################################
-# Container Debug - speeds container debugging by splitting each section into a separate intermediate container
-####################################################################################################################################
-use constant CONTAINER_DEBUG                                        => false;
-
-####################################################################################################################################
-# Store cache container checksums
-####################################################################################################################################
-my $hContainerCache;
-
-####################################################################################################################################
-# Container repo - defines the container registry where the containers will be located
-####################################################################################################################################
-sub containerRepo
-{
-    # The pgbackrest namespace is the canonical cache that every environment pulls from and only pgbackrest CI can push to. Pass an
-    # owner to build the tag for that owner's own cache instead (a fork can push to and pull from it). Images are always tagged in
-    # the pgbackrest namespace locally so Docker calls do not need to know which cache they came from.
-    my $strOwner = shift;
-
-    return 'ghcr.io/' . (defined($strOwner) ? $strOwner : PROJECT_EXE) . '/test';
-}
-
-push @EXPORT, qw(containerRepo);
-
-####################################################################################################################################
-# Section header
-####################################################################################################################################
-sub sectionHeader
-{
-    return (CONTAINER_DEBUG ? "\n\nRUN \\\n" : " && \\\n\n");
-}
-
-####################################################################################################################################
-# Write the Docker container
-####################################################################################################################################
-sub containerWrite
-{
-    my $oStorageDocker = shift;
-    my $strTempPath = shift;
-    my $strOS = shift;
-    my $strArch = shift;
-    my $strTitle = shift;
-    my $strImageParent = shift;
-    my $strImage = shift;
-    my $strCopy = shift;
-    my $strScript = shift;
-    my $bForce = shift;
-
-    my $strTag = containerRepo() . ":${strImage}";
-
-    $strScript =
-        "# ${strTitle} Container\n" .
-        "FROM ${strImageParent}" .
-        (defined($strCopy) ? "\n\n${strCopy}" : '') .
-        (defined($strScript) && $strScript ne ''?
-            "\n\nRUN echo '" . (CONTAINER_DEBUG ? 'DEBUG' : 'OPTIMIZED') . " BUILD'" . $strScript : '');
-
-    # Base images are cached in the container registry, keyed by a hash of the generated Dockerfile combined with the manual
-    # revision from container.yaml. Images are pulled from the pgbackrest cache first, then from the repository owner's own cache
-    # when running on a fork, and built locally if neither has them. A build is pushed to the cache the environment can write to
-    # (the fork's own cache on a fork, else the pgbackrest cache). The image is always tagged in the pgbackrest namespace locally,
-    # so downstream Docker calls reference a single name regardless of where it came from.
-    my $bCached = false;
-    my $strCacheTag;
-    my $strForkCacheTag;
-
-    if ($strImage =~ /\-base\-/)
-    {
-        # The revision forces a rebuild when the Dockerfile is unchanged but the upstream packages have changed, e.g. a new
-        # PostgreSQL beta. A container uses the revision for its vm and architecture if it has one, else the revision for its vm,
-        # else the required "all" revision (all validated in containerBuild). So bumping "all" rebuilds only the containers without
-        # a more specific revision; a container with a more specific revision is rebuilt by bumping that revision (or deleting its
-        # cached tag). The revision is also placed in the tag for reference.
-        my $strRevision =
-            defined($hContainerCache->{"${strOS}-${strArch}"}) ? $hContainerCache->{"${strOS}-${strArch}"} :
-                (defined($hContainerCache->{$strOS}) ? $hContainerCache->{$strOS} : $hContainerCache->{&VM_ALL});
-        my $strCacheImage = "${strImage}-${strRevision}-" . substr(sha1_hex($strScript . $strRevision), 0, 12);
-
-        $strCacheTag = containerRepo() . ":${strCacheImage}";
-
-        # On a fork also use the fork owner's own cache, which it can write to
-        my $strOwner = $ENV{GITHUB_REPOSITORY_OWNER} ? lc($ENV{GITHUB_REPOSITORY_OWNER}) : PROJECT_EXE;
-        $strForkCacheTag = $strOwner ne PROJECT_EXE ? containerRepo($strOwner) . ":${strCacheImage}" : undef;
-
-        # Pull from the pgbackrest cache first, then the fork's own cache. A failed pull (image missing or no read access) falls
-        # through to the next cache or a local build.
-        if (!(defined($bForce) && $bForce))
-        {
-            foreach my $strPullTag ($strCacheTag, defined($strForkCacheTag) ? $strForkCacheTag : ())
-            {
-                &log(INFO, "Checking cache ${strPullTag} ...");
-
-                my $oPull = new pgBackRestTest::Common::ExecuteTest(
-                    "docker pull ${strPullTag}", {bSuppressStdErr => true, bSuppressError => true});
-                $oPull->begin();
-
-                if ($oPull->end() == 0)
-                {
-                    &log(INFO, "Using cached ${strPullTag} image");
-                    executeTest("docker tag ${strPullTag} ${strTag}");
-                    $bCached = true;
-                    last;
-                }
-            }
-        }
-    }
-
-    if (!$bCached)
-    {
-        &log(INFO, "Building ${strTag} image" . (defined($strCacheTag) ? " (${strCacheTag})" : '') . ' ...');
-
-        # Write and build the image
-        $oStorageDocker->put("${strTempPath}/${strImage}", trim($strScript) . "\n");
-        executeTest(
-            'docker build' . (defined($strArch) ? " --platform linux/${strArch}" : '') .
-            (defined($bForce) && $bForce ? ' --no-cache' : '') . " -f ${strTempPath}/${strImage} -t ${strTag} " .
-                $oStorageDocker->pathGet('test'),
-            {bSuppressStdErr => true, bShowOutputAsync => (logLevel())[1] eq DETAIL});
-
-        # Push the base image to the cache this environment can write to (the fork's own cache on a fork, else the pgbackrest
-        # cache) and log when it succeeds. Best effort since it requires write access, which is not available locally or on pull
-        # requests from forks; those just use the image they built.
-        if (defined($strCacheTag))
-        {
-            my $strPushTag = defined($strForkCacheTag) ? $strForkCacheTag : $strCacheTag;
-            executeTest("docker tag ${strTag} ${strPushTag}", {bSuppressError => true});
-
-            my $oPush = new pgBackRestTest::Common::ExecuteTest(
-                "docker push ${strPushTag}", {bSuppressStdErr => true, bSuppressError => true});
-            $oPush->begin();
-
-            if ($oPush->end() == 0)
-            {
-                &log(INFO, "Cached ${strPushTag} image");
-            }
-        }
-    }
-}
-
-####################################################################################################################################
-# User/group creation
-####################################################################################################################################
-sub groupCreate
-{
-    my $strOS = shift;
-    my $strName = shift;
-    my $iId = shift;
-
-    my $oVm = vmGet();
-
-    if ($$oVm{$strOS}{&VM_OS_BASE} eq VM_OS_BASE_RHEL || $$oVm{$strOS}{&VM_OS_BASE} eq VM_OS_BASE_DEBIAN)
-    {
-        return "groupadd -f -g${iId} ${strName}";
-    }
-    elsif ($$oVm{$strOS}{&VM_OS_BASE} eq VM_OS_BASE_ALPINE)
-    {
-        return "getent group ${strName} >/dev/null || addgroup -g${iId} ${strName}";
-    }
-}
-
-sub userCreate
-{
-    my $strOS = shift;
-    my $strName = shift;
-    my $iId = shift;
-    my $strGroup = shift;
-
-    my $oVm = vmGet();
-
-    if ($$oVm{$strOS}{&VM_OS_BASE} eq VM_OS_BASE_RHEL)
-    {
-        return "adduser -g${strGroup} -u${iId} -N ${strName}";
-    }
-    elsif ($$oVm{$strOS}{&VM_OS_BASE} eq VM_OS_BASE_DEBIAN || $$oVm{$strOS}{&VM_OS_BASE} eq VM_OS_BASE_ALPINE)
-    {
-        return "adduser --uid=${iId} --ingroup=${strGroup} --disabled-password --gecos \"\" ${strName}";
-    }
-
-    confess &log(ERROR, "unable to create user for os '${strOS}'");
-}
-
-####################################################################################################################################
-# Setup SSH
-####################################################################################################################################
-sub sshSetup
-{
-    my $strOS = shift;
-    my $strUser = shift;
-    my $strGroup = shift;
-
-    my $strUserPath = $strUser eq 'root' ? "/${strUser}" : "/home/${strUser}";
-
-    my $strScript = sectionHeader() .
-        "# Setup SSH\n" .
-        "    mkdir ${strUserPath}/.ssh && \\\n" .
-        "    echo '-----BEGIN RSA PRIVATE KEY-----' > ${strUserPath}/.ssh/id_rsa && \\\n" .
-        "    echo 'MIIEowIBAAKCAQEA5VySpEan5Rn7QQ5XP13YkXxTg+Xp9N+ecyDhD92OQk0VZOFn' >> ${strUserPath}/.ssh/id_rsa && \\\n" .
-        "    echo 'EKlXUaMXH0WVjRRgjgbF78JHFbEQjUHimbzr9ev1IuDaIS42E1nUkBaHggnHNAW2' >> ${strUserPath}/.ssh/id_rsa && \\\n" .
-        "    echo 'aqtczUDSnLGcXqd0XeDtwpjj1I0xWOeJli/xjU+eSwjzqNgX6ZX2P0uTXflu254u' >> ${strUserPath}/.ssh/id_rsa && \\\n" .
-        "    echo 'SjOF3IARo9FiDEKQdksoxNOAhCTYgDAynmkcfJ9e87FjiTGmfnZO3H6gU2kidSFX' >> ${strUserPath}/.ssh/id_rsa && \\\n" .
-        "    echo 'rwdHtdN3Qlv6RgCroLGAZZpmtqBorTwShWRScWAAg/OqWwLphfNbXIIHZO4EJ4S2' >> ${strUserPath}/.ssh/id_rsa && \\\n" .
-        "    echo 'JLgsgJHb20SJ2XEI2Iln9sj+oCWgLJ2m+RLvWwIDAQABAoIBAArBC0EiQkyxf1Xe' >> ${strUserPath}/.ssh/id_rsa && \\\n" .
-        "    echo 'txmKAWsE4iI85/oqzVJG7YvhuVdY0j16J2vLvNk05T3P9JdPqB1QqlGNEZSDSlbi' >> ${strUserPath}/.ssh/id_rsa && \\\n" .
-        "    echo 'isjm55tkFl4tyRx61F9A5tYLWwwuVYWWFPutuuVcJOPi8gWAItUkruaLu6GjgyJQ' >> ${strUserPath}/.ssh/id_rsa && \\\n" .
-        "    echo '143QA//lBp4sYRxUEX71defO19iKkDz+xEuOzYMd16j76OKMcmbnog7hbMrXR4Yi' >> ${strUserPath}/.ssh/id_rsa && \\\n" .
-        "    echo 'kNXuhnwBadutaXLve3mZ0JowrZyHKfTUWOHgvuULpCVD5su3NdbpE2AVn1idcF8V' >> ${strUserPath}/.ssh/id_rsa && \\\n" .
-        "    echo 'jaj6p0vtcvEnXEC69XwX+yL0TgvOE4Vu/OWg8lDWQdetONIHGbElJZvB6eyTF1nl' >> ${strUserPath}/.ssh/id_rsa && \\\n" .
-        "    echo 'IIgLc2ECgYEA7PL3soOfH3dMNUt4KGSw1cK/kwvy7UsT6QrAPi1Cc/A4szluk6/O' >> ${strUserPath}/.ssh/id_rsa && \\\n" .
-        "    echo '5YhKfTjzHW5WDmsTTAcT29MLmW8dQXeUAe/1BtIATubsav+uSelfBmUAnQj9fvKT' >> ${strUserPath}/.ssh/id_rsa && \\\n" .
-        "    echo 'ieJ6JMf20OTbS6XODA3+jJAdApLCu61Lv6nePOuNzLY/uqSMWu9kO/sCgYEA981v' >> ${strUserPath}/.ssh/id_rsa && \\\n" .
-        "    echo 'YIUaadFaHnPnmax0+jJMs8S5AIEjSfSIxR8oNOWNUxBBvwFd4zWTApVfZqKjmI83' >> ${strUserPath}/.ssh/id_rsa && \\\n" .
-        "    echo 'Ng5tISxspzHseakyrIpoDqzxRQPxZF7RTO6VUX+ZQj6iIXVp9FDqWAjvDACuSky2' >> ${strUserPath}/.ssh/id_rsa && \\\n" .
-        "    echo 'mGAiiA+fWZ1za62opgoYQZ17O17SyHF9/vJ7XCECgYANwNyXxAQMc4Q847CJx65r' >> ${strUserPath}/.ssh/id_rsa && \\\n" .
-        "    echo '+e3cvyjOlTkGodUexsnAqQThgkfk0qOTtyF7uz6BStI77AMmupJwhAN8WHK+Rg6V' >> ${strUserPath}/.ssh/id_rsa && \\\n" .
-        "    echo 'PjRevPm/mq/GVijrqVwWpu4uL0NnhvUBX9/vGpw868u+zFT1ZiqMRiEo8RPUiO6I' >> ${strUserPath}/.ssh/id_rsa && \\\n" .
-        "    echo 'pXd82b9VTo7Mapiq/pI22QKBgC8Kb586BUabOGlZhVi11Ur9q3Pg32HKIgHTCveo' >> ${strUserPath}/.ssh/id_rsa && \\\n" .
-        "    echo 'r4BDJ23iQyjYQJN2Qx8VbhPUwgue/FMlr+/BOCsRHhwGU5lPeOt4RyDb28I7Aa6C' >> ${strUserPath}/.ssh/id_rsa && \\\n" .
-        "    echo 'CBR9jYF21F5XpLJ9fc8SexajNnLiVzNb5JJBrPVdH2EMiVxjxDEIjTE7EfZ9HPb9' >> ${strUserPath}/.ssh/id_rsa && \\\n" .
-        "    echo '3w8hAoGBALyik3jr0W6DxzqOW0jPXPLDqp1JvzieC03nZD1scWeI8qIzUOpLX7cc' >> ${strUserPath}/.ssh/id_rsa && \\\n" .
-        "    echo 'jaMU/8QMBRvyEcZK82Cedilm30nLf+C/FR5TsUmftS7IcjoC4Z2ZXWNOhMv22TUJ' >> ${strUserPath}/.ssh/id_rsa && \\\n" .
-        "    echo 'Ml6z//+WSZ3qVZ5rvAeo4obwiBfe+Uh+AyyprEGgmimF9qDejcwc' >> ${strUserPath}/.ssh/id_rsa && \\\n" .
-        "    echo '-----END RSA PRIVATE KEY-----' >> ${strUserPath}/.ssh/id_rsa && \\\n" .
-        "    echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlXJKkRqflGftBDlc/XdiRfFOD5en0355zIOEP3Y5CTRVk4WcQqVdRoxcfRZWNFGCOBsXvwk" .
-             "cVsRCNQeKZvOv16/Ui4NohLjYTWdSQFoeCCcc0BbZqq1zNQNKcsZxep3Rd4O3CmOPUjTFY54mWL/GNT55LCPOo2BfplfY/S5Nd+W7bni5KM4XcgBGj0" .
-             "WIMQpB2SyjE04CEJNiAMDKeaRx8n17zsWOJMaZ+dk7cfqBTaSJ1IVevB0e103dCW/pGAKugsYBlmma2oGitPBKFZFJxYACD86pbAumF81tcggdk7gQn" .
-             "hLYkuCyAkdvbRInZcQjYiWf2yP6gJaAsnab5Eu9b user\@pgbackrest-test' > ${strUserPath}/.ssh/authorized_keys && \\\n" .
-        "    echo 'Host *' > ${strUserPath}/.ssh/config && \\\n" .
-        "    echo '    StrictHostKeyChecking no' >> ${strUserPath}/.ssh/config && \\\n";
-
-    $strScript .=
-        "    cp ${strUserPath}/.ssh/authorized_keys ${strUserPath}/.ssh/id_rsa.pub && \\\n" .
-        "    chown -R ${strUser}:${strGroup} ${strUserPath}/.ssh && \\\n" .
-        "    chmod 700 ${strUserPath}/.ssh && \\\n" .
-        "    chmod 600 ${strUserPath}/.ssh/*";
-
-    return $strScript;
-}
-
-####################################################################################################################################
-# Copy text file into container. Note that this will not work if the file contains single quotes.
-####################################################################################################################################
-sub fileCopy
-{
-    my $oStorage = shift;
-    my $strSourceFile = shift;
-    my $strDestFile = shift;
-
-    my $strScript;
-
-    foreach my $strLine (split("\n", ${$oStorage->get($strSourceFile)}))
-    {
-        $strScript .= "    echo '${strLine}' " . (defined($strScript) ? '>>' : '>') . " ${strDestFile} && \\\n";
-    }
-
-    return $strScript;
-}
-
-####################################################################################################################################
-# CA Setup
-####################################################################################################################################
-sub caSetup
-{
-    my $strOS = shift;
-    my $oStorage = shift;
-    my $strCaFile = shift;
-
-    my $strOsBase = vmGet()->{$strOS}{&VM_OS_BASE};
-
-    # Determine CA location
-    my $strCertFile = undef;
-
-    if ($strOsBase eq VM_OS_BASE_RHEL)
-    {
-        $strCertFile = '/etc/pki/ca-trust/source/anchors';
-    }
-    elsif ($strOsBase eq VM_OS_BASE_DEBIAN || $strOsBase eq VM_OS_BASE_ALPINE)
-    {
-        $strCertFile = '/usr/local/share/ca-certificates';
-    }
-    else
-    {
-        confess &log(ERROR, "unable to install CA for ${strOsBase}");
-    }
-
-    $strCertFile .= '/pgbackrest-test-ca.crt';
-
-    # Write CA
-    my $strScript =
-        sectionHeader() .
-        "# Install CA\n" .
-        fileCopy($oStorage, $strCaFile, $strCertFile) .
-        "    chmod 644 ${strCertFile} && \\\n";
-
-    # Install CA
-    if ($strOsBase eq VM_OS_BASE_RHEL)
-    {
-        $strScript .=
-            "    update-ca-trust extract";
-    }
-    elsif ($strOsBase eq VM_OS_BASE_DEBIAN || $strOsBase eq VM_OS_BASE_ALPINE)
-    {
-        $strScript .=
-            "    update-ca-certificates";
-    }
-
-    return $strScript;
-}
-
-####################################################################################################################################
-# Entry point setup
-####################################################################################################################################
-sub entryPointSetup
-{
-    my $strOS = shift;
-
-    my $strScript =
-        "\n\n# Start SSH when container starts\n" .
-        'ENTRYPOINT ';
-
-    my $oVm = vmGet();
-
-    if ($oVm->{$strOS}{&VM_OS_BASE} eq VM_OS_BASE_RHEL)
-    {
-        $strScript .= 'rm -rf /run/nologin && /usr/sbin/sshd -D';
-    }
-    elsif ($oVm->{$strOS}{&VM_OS_BASE} eq VM_OS_BASE_DEBIAN)
-    {
-        $strScript .= 'service ssh restart && bash';
-    }
-    elsif ($oVm->{$strOS}{&VM_OS_BASE} eq VM_OS_BASE_ALPINE)
-    {
-        $strScript .= '/usr/sbin/sshd -D';
-    }
-
-    return $strScript;
-}
-
-####################################################################################################################################
-# Build containers
-####################################################################################################################################
-sub containerBuild
-{
-    my $oStorageDocker = shift;
-    my $strVm = shift;
-    my $strArch = shift;
-    my $bVmForce = shift;
-
-    # Create temp path
-    my $strTempPath = $oStorageDocker->pathGet('test/result/docker');
-    $oStorageDocker->pathCreate($strTempPath, {strMode => '0770', bIgnoreExists => true, bCreateParent => true});
-
-    # Load container definitions from yaml
-    require YAML::XS;
-    YAML::XS->import(qw(Load));
-
-    $hContainerCache = Load(${$oStorageDocker->get($oStorageDocker->pathGet('test/container.yaml'))});
-
-    # The "all" revision is required so it can never be accidentally unset, which would change every image hash at once. It can
-    # only be bumped.
-    if (ref($hContainerCache) ne 'HASH' || !defined($hContainerCache->{&VM_ALL}))
-    {
-        confess &log(ERROR, "the 'all' revision is required in test/container.yaml");
-    }
-
-    # Validate the revisions. A revision is keyed by "all", a vm, or a vm qualified with an architecture, e.g. "u22-x86_64". Without
-    # this an invalid key would silently fall back to a less specific revision and the expected rebuild would never happen.
-    foreach my $strKey (sort(keys(%{$hContainerCache})))
-    {
-        if (!defined($hContainerCache->{$strKey}) || ref($hContainerCache->{$strKey}) ne '')
-        {
-            confess &log(ERROR, "revision '${strKey}' in test/container.yaml must be set to a value");
-        }
-
-        next if $strKey eq VM_ALL;
-
-        my ($strKeyVm, $strKeyArch) = split('-', $strKey, 2);
-
-        if (!defined(vmGet()->{$strKeyVm}) || $strKeyVm eq VM_NONE)
-        {
-            confess &log(ERROR, "revision '${strKey}' in test/container.yaml has invalid vm '${strKeyVm}'");
-        }
-
-        if (defined($strKeyArch) && !grep($_ eq $strKeyArch, VM_ARCH_LIST))
-        {
-            confess &log(ERROR, "revision '${strKey}' in test/container.yaml has invalid architecture '${strKeyArch}'");
-        }
-    }
-
-    # Remove old images on force
-    if ($bVmForce)
-    {
-        my $strRegExp = '^' . containerRepo();
-
-        if ($strVm ne 'all')
-        {
-            $strRegExp .= "\:${strVm}-";
-        }
-
-        executeTest("rm -f ${strTempPath}/" . ($strVm eq 'all' ? '*' : "${strVm}-*"));
-        imageRemove("${strRegExp}.*");
-    }
-
-    # VM Images
-    ################################################################################################################################
-    my $oVm = vmGet();
-
-    foreach my $strOS ($strVm eq 'all' ? VM_LIST : ($strVm))
-    {
-        my $oOS = $oVm->{$strOS};
-        my $bDocBuild = false;
-
-        # Deprecated OSs can only be used to build packages
-        my $bDeprecated = $oVm->{$strOS}{&VM_DEPRECATED} ? true : false;
-
-        # Base image
-        ###########################################################################################################################
-        my $strImageParent = $oVm->{$strOS}{&VM_IMAGE};
-        $strArch = defined($strArch) ? $strArch : hostArch();
-        my $strImage = "${strOS}-base" . (defined($strArch) ? "-${strArch}" : '-' . hostArch());
-        my $strCopy = undef;
-
-        #---------------------------------------------------------------------------------------------------------------------------
-        my $strScript = sectionHeader() .
-            "# Install packages\n";
-
-        if ($$oVm{$strOS}{&VM_OS_BASE} eq VM_OS_BASE_RHEL)
-        {
-            if ($strOS eq VM_RH8)
-            {
-                $strScript .=
-                    "    dnf install -y dnf-plugins-core && \\\n" .
-                    "    dnf config-manager --set-enabled powertools && \\\n" .
-                    "    dnf -y install epel-release && \\\n" .
-                    "    crb enable && \\\n";
-            }
-            elsif ($strOS eq VM_RH9 || $strOS eq VM_RH10)
-            {
-                $strScript .=
-                    "    dnf install -y dnf-plugins-core && \\\n" .
-                    "    dnf -y install epel-release && \\\n" .
-                    "    crb enable && \\\n";
-            }
-
-            $strScript .=
-                "    yum -y update && \\\n" .
-                "    yum -y install openssh-server openssh-clients sudo git \\\n" .
-                "        perl perl-Digest-SHA perl-DBD-Pg perl-YAML-LibYAML openssl \\\n" .
-                "        gcc make perl-ExtUtils-MakeMaker perl-Test-Simple openssl-devel perl-ExtUtils-Embed rpm-build \\\n" .
-                "        libyaml-devel zlib-devel libxml2-devel lz4-devel lz4 bzip2-devel bzip2 perl-JSON-PP ccache meson \\\n" .
-                "        libssh2-devel zstd libzstd-devel systemd-devel";
-
-            if ($strOS ne VM_RH8)
-            {
-                $strScript .= " valgrind";
-            }
-
-            # Coverage for the python tests. rh10 and later package a version new enough while rh9 packages one that predates the
-            # per line branch detail the report needs, so there it comes from pip below.
-            if ($strOS eq VM_RH9)
-            {
-                $strScript .= " python3-pip";
-            }
-            elsif ($strOS ne VM_RH8)
-            {
-                $strScript .= " python3-coverage";
-            }
-
-            # The test harness needs a python3 that PyYAML is packaged for. On RH8 that is not the platform python, which meson
-            # installs and which claims the python3 alternative at a priority nothing else can outrank, so point it explicitly.
-            if ($strOS eq VM_RH8)
-            {
-                $strScript .=
-                    " python3.12 python3.12-pyyaml && \\\n" .
-                    "    alternatives --set python3 /usr/bin/python3.12";
-            }
-            else
-            {
-                $strScript .= " python3-pyyaml";
-            }
-        }
-        elsif ($$oVm{$strOS}{&VM_OS_BASE} eq VM_OS_BASE_DEBIAN)
-        {
-            $strScript .=
-                "    export DEBCONF_NONINTERACTIVE_SEEN=true DEBIAN_FRONTEND=noninteractive && \\\n" .
-                "    apt-get update && \\\n" .
-                "    apt-get install -y --no-install-recommends openssh-server sudo gcc make git \\\n" .
-                "        libdbd-pg-perl libhtml-parser-perl libssl-dev libperl-dev \\\n" .
-                "        libyaml-libyaml-perl tzdata devscripts lintian libxml-checker-perl txt2man debhelper \\\n" .
-                "        libppi-html-perl libtemplate-perl libtest-differences-perl zlib1g-dev libxml2-dev pkg-config \\\n" .
-                "        libbz2-dev bzip2 libyaml-dev libjson-pp-perl liblz4-dev liblz4-tool gnupg lsb-release ccache meson \\\n" .
-                "        libssh2-1-dev libcurl4-openssl-dev libsystemd-dev python3-yaml";
-
-            if ($strOS ne VM_D12)
-            {
-                $strScript .= " valgrind";
-            }
-
-            # Coverage for the python tests, which are run on u24 and later
-            if ($strOS ne VM_D12 && $strOS ne VM_U22)
-            {
-                $strScript .= " python3-coverage";
-            }
-
-            $strScript .= ' zstd libzstd-dev';
-        }
-        elsif ($$oVm{$strOS}{&VM_OS_BASE} eq VM_OS_BASE_ALPINE)
-        {
-            $strScript .=
-                "    apk update && \\\n" .
-                "    apk add --no-cache sudo openssh git rsync tzdata openssh ca-certificates openrc bash && \\\n" .
-                "    rc-update add sshd && \\\n" .
-                "    apk add --no-cache meson build-base libpq-dev openssl-dev libxml2-dev pkgconfig lz4-dev bzip2-dev\\\n" .
-                "        openssh-keygen zlib-dev yaml-dev libssh2-dev perl perl-yaml-libyaml valgrind lz4 zstd zstd-dev py3-yaml";
-        }
-
-        # Coverage for python tests where the packaged version predates the per line branch detail the report needs, so it comes
-        # from pip rather than from the distribution
-        if ($strOS eq VM_RH9)
-        {
-            $strScript .= sectionHeader() .
-                "# Install python coverage\n" .
-                "    pip3 install --no-cache-dir 'coverage>=6.5'";
-        }
-
-        #---------------------------------------------------------------------------------------------------------------------------
-        $strScript .= sectionHeader() .
-            "# Regenerate SSH keys\n" .
-            "    rm -f /etc/ssh/ssh_host_rsa_key* && \\\n" .
-            "    ssh-keygen -t rsa -b 2048 -f /etc/ssh/ssh_host_rsa_key";
-
-        if ($$oVm{$strOS}{&VM_OS_BASE} eq VM_OS_BASE_DEBIAN)
-        {
-            $strScript .= sectionHeader() .
-                "# Fix root tty\n" .
-                "    sed -i 's/^mesg n/tty -s \\&\\& mesg n/g' /root/.profile";
-
-            $strScript .= sectionHeader() .
-                "# Suppress dpkg interactive output\n" .
-                "    rm /etc/apt/apt.conf.d/70debconf";
-        }
-
-        #---------------------------------------------------------------------------------------------------------------------------
-        if (!$bDeprecated)
-        {
-            if ($$oVm{$strOS}{&VM_OS_BASE} eq VM_OS_BASE_RHEL || $$oVm{$strOS}{&VM_OS_BASE} eq VM_OS_BASE_DEBIAN)
-            {
-                $strScript .= sectionHeader() .
-                    "# Install PostgreSQL packages\n";
-            }
-
-            if ($$oVm{$strOS}{&VM_OS_BASE} eq VM_OS_BASE_RHEL)
-            {
-                # RHEL 10's rpm-sequoia OpenPGP backend rejects the SHA-1 binding signature in the PGDG GPG key, and the
-                # crypto-policy SHA1 subpolicy that could re-enable it was removed in EL-10. So rh10 skips the key import
-                # and installs the repo/packages with gpg checks disabled (acceptable for a throwaway test container).
-                if ($strOS ne VM_RH10 && vmPgRepo($strOS))
-                {
-                    $strScript .=
-                        "    rpm --import https://download.postgresql.org/pub/repos/yum/keys/RPM-GPG-KEY-PGDG && \\\n";
-                }
-
-                if (!vmPgRepo($strOS))
-                {
-                    # Enable the native PostgreSQL application stream instead of adding the PGDG repo
-                    $strScript .=
-                        "    dnf -y module enable postgresql:" . @{$oOS->{&VM_DB}}[-1] . " && \\\n";
-                }
-                elsif ($strOS eq VM_RH9)
-                {
-                    $strScript .=
-                        "    rpm -ivh \\\n" .
-                        "        https://download.postgresql.org/pub/repos/yum/reporpms/EL-9-" . hostArch() . "/" .
-                            "pgdg-redhat-repo-latest.noarch.rpm && \\\n" .
-                        "    dnf -qy module disable postgresql && \\\n" .
-                        "    yum -y install libcurl-devel && \\\n";
-                }
-                elsif ($strOS eq VM_RH10)
-                {
-                    # Install the repo without a gpg check and disable gpgcheck on the PGDG repos (see the SHA-1 note
-                    # above). RHEL 10 also dropped dnf modularity, so there is no postgresql module to disable.
-                    $strScript .=
-                        "    dnf -y install --nogpgcheck \\\n" .
-                        "        https://download.postgresql.org/pub/repos/yum/reporpms/EL-10-" . hostArch() . "/" .
-                            "pgdg-redhat-repo-latest.noarch.rpm && \\\n" .
-                        "    sed -i 's/gpgcheck=1/gpgcheck=0/g' /etc/yum.repos.d/pgdg-redhat-all.repo && \\\n" .
-                        "    yum -y install libcurl-devel && \\\n";
-                }
-                elsif ($strOS eq VM_F44)
-                {
-                    $strScript .=
-                        "    rpm -ivh \\\n" .
-                        "        https://download.postgresql.org/pub/repos/yum/reporpms/F-44-" . hostArch() . "/" .
-                            "pgdg-fedora-repo-latest.noarch.rpm && \\\n" .
-                        "    yum -y install libcurl-devel && \\\n"
-                }
-
-                $strScript .= "    yum -y install postgresql-devel";
-            }
-            elsif ($$oVm{$strOS}{&VM_OS_BASE} eq VM_OS_BASE_DEBIAN)
-            {
-                # Install repo from apt.postgresql.org
-                if (vmPgRepo($strVm))
-                {
-                    $strScript .=
-                        "    apt-get install -y --no-install-recommends postgresql-common && \\\n" .
-                        "    /usr/share/postgresql-common/pgdg/apt.postgresql.org.sh -y" .
-                            ($strOS eq VM_U24 && ($strArch eq VM_ARCH_AARCH64 || $strArch eq VM_ARCH_X86_64) ? ' -c 19' : '') .
-                            " && \\\n";
-                }
-
-                $strScript .=
-                    "    apt-get install -y --no-install-recommends postgresql-common libpq-dev && \\\n" .
-                    "    sed -i 's/^\\#create\\_main\\_cluster.*\$/create\\_main\\_cluster \\= false/' " .
-                        "/etc/postgresql-common/createcluster.conf";
-            }
-
-            if (defined($oOS->{&VM_DB}) && @{$oOS->{&VM_DB}} > 0 &&
-                ($strArch eq VM_ARCH_AARCH64 || $strArch eq VM_ARCH_X86_64 || $strArch eq VM_ARCH_I386))
-            {
-                $strScript .= sectionHeader() .
-                    "# Install PostgreSQL\n";
-
-                if ($$oVm{$strOS}{&VM_OS_BASE} eq VM_OS_BASE_RHEL)
-                {
-                    $strScript .= "    yum -y install";
-                }
-                elsif ($$oVm{$strOS}{&VM_OS_BASE} eq VM_OS_BASE_DEBIAN)
-                {
-                    $strScript .= "    apt-get install -y --no-install-recommends";
-                }
-                elsif ($$oVm{$strOS}{&VM_OS_BASE} eq VM_OS_BASE_ALPINE)
-                {
-                    $strScript .= "    apk add --no-cache";
-                }
-
-                # Construct list of databases to install
-                foreach my $strDbVersion (@{$oOS->{&VM_DB}})
-                {
-                    if ($$oVm{$strOS}{&VM_OS_BASE} eq VM_OS_BASE_RHEL)
-                    {
-                        # Native RHEL installs the unversioned server package from the application stream
-                        if (!vmPgRepo($strOS))
-                        {
-                            $strScript .= " postgresql-server";
-                        }
-                        else
-                        {
-                            my $strDbVersionNoDot = $strDbVersion;
-                            $strDbVersionNoDot =~ s/\.//;
-
-                            $strScript .= " postgresql${strDbVersionNoDot}-server";
-
-                            # Add development package for the latest version of postgres
-                            if ($strDbVersion eq @{$oOS->{&VM_DB}}[-1])
-                            {
-                                $strScript .= " postgresql${strDbVersionNoDot}-devel";
-                            }
-                        }
-                    }
-                    elsif ($$oVm{$strOS}{&VM_OS_BASE} eq VM_OS_BASE_DEBIAN)
-                    {
-                        $strScript .= " postgresql-${strDbVersion}";
-                    }
-                    elsif ($$oVm{$strOS}{&VM_OS_BASE} eq VM_OS_BASE_ALPINE)
-                    {
-                        $strScript .= " postgresql${strDbVersion}";
-                    }
-                }
-            }
-            # On other architectures (e.g. ppc64le, s390x) install a single PostgreSQL version for the smoke test via the
-            # postgresql metapackage (its default from the configured repo). The full version matrix is skipped to keep the
-            # emulated build fast; one version is enough to check the build and exercise checksums end-to-end on big-endian.
-            elsif (defined($oOS->{&VM_DB}) && @{$oOS->{&VM_DB}} > 0 && $$oVm{$strOS}{&VM_OS_BASE} eq VM_OS_BASE_DEBIAN)
-            {
-                $strScript .= sectionHeader() .
-                    "# Install a single PostgreSQL version for the smoke test\n" .
-                    "    apt-get install -y --no-install-recommends postgresql";
-            }
-        }
-
-        # Add path to latest version of postgres (PGDG installs to a versioned path; native packages use /usr/bin)
-        if ($$oVm{$strOS}{&VM_OS_BASE} eq VM_OS_BASE_RHEL && vmPgRepo($strOS))
-        {
-            $strScript .=
-                "\n\nENV PATH=/usr/pgsql-" . @{$oOS->{&VM_DB}}[-1] . "/bin:\$PATH\n" .
-                "ENV PKG_CONFIG_PATH=/usr/pgsql-" . @{$oOS->{&VM_DB}}[-1] . "/lib/pkgconfig:\$PKG_CONFIG_PATH\n";
-        }
-
-        #---------------------------------------------------------------------------------------------------------------------------
-        if ($$oVm{$strOS}{&VM_OS_BASE} eq VM_OS_BASE_DEBIAN)
-        {
-        $strScript .= sectionHeader() .
-            "# Cleanup\n";
-
-            $strScript .=
-                "    apt-get autoremove -y && \\\n" .
-                "    apt-get clean && \\\n" .
-                "    rm -rf /var/lib/apt/lists/*";
-        }
-
-        containerWrite(
-            $oStorageDocker, $strTempPath, $strOS, $strArch, 'Base', $strImageParent, $strImage, $strCopy, $strScript, $bVmForce);
-
-        # Test image
-        ########################################################################################################################
-        if (!$bDeprecated)
-        {
-            $strImageParent = containerRepo() . ":${strImage}";
-            $strImage = "${strOS}-test" . (defined($strArch) ? "-${strArch}" : '-' . hostArch());
-
-            $strCopy = undef;
-            $strScript = '';
-
-            #-----------------------------------------------------------------------------------------------------------------------
-            if ($oVm->{$strOS}{&VM_OS_BASE} eq VM_OS_BASE_RHEL && hostArch() eq VM_ARCH_AARCH64)
-            {
-                $strScript .= sectionHeader() .
-                    "# Remove unneeded language setup\n" .
-                    "    rm /etc/profile.d/lang.sh";
-            }
-
-            #-----------------------------------------------------------------------------------------------------------------------
-            $strScript .= caSetup($strOS, $oStorageDocker, "test/certificate/pgbackrest-test-ca.crt");
-
-            #-----------------------------------------------------------------------------------------------------------------------
-            $strScript .= sectionHeader() .
-                "# Create banner to make sure pgBackRest ignores it\n" .
-                "    echo '***********************************************' >  /etc/issue.net && \\\n" .
-                "    echo 'Sample banner to make sure banners are skipped.' >> /etc/issue.net && \\\n" .
-                "    echo ''                                                >> /etc/issue.net && \\\n" .
-                "    echo 'More banner after a blank line.'                 >> /etc/issue.net && \\\n" .
-                "    echo '***********************************************' >> /etc/issue.net && \\\n" .
-                "    echo 'Banner /etc/issue.net'                           >> /etc/ssh/sshd_config";
-
-            if ($strOS eq VM_U22 || $strOS eq VM_U24 || $strOS eq VM_A321)
-            {
-                $strScript .= sectionHeader() .
-                    "    echo '# Add PubkeyAcceptedAlgorithms (required for SFTP)'              >> /etc/ssh/sshd_config && \\\n" .
-                    "    echo 'HostKeyAlgorithms=+ssh-rsa,ssh-rsa-cert-v01\@openssh.com'        >> /etc/ssh/sshd_config && \\\n" .
-                    "    echo 'PubkeyAcceptedAlgorithms=+ssh-rsa,ssh-rsa-cert-v01\@openssh.com' >> /etc/ssh/sshd_config";
-            }
-
-            # Rename existing group that would conflict with our group name. This is pretty hacky but should be OK since we are the
-            # only thing running in the container.
-            $strScript .= sectionHeader() .
-                "# Rename conflicting group\n" .
-                "    sed -i 's/^[^:]*:x:" . TEST_GROUP_ID . ":.*/" . TEST_GROUP . ":x:" . TEST_GROUP_ID . ":/' /etc/group";
-
-            $strScript .= sectionHeader() .
-                "# Create test user\n" .
-                '    ' . groupCreate($strOS, TEST_GROUP, TEST_GROUP_ID) . " && \\\n" .
-                '    ' . userCreate($strOS, TEST_USER, TEST_USER_ID, TEST_GROUP) . " && \\\n" .
-                '    mkdir -m 750 /home/' . TEST_USER . "/test && \\\n" .
-                '    chown ' . TEST_USER . ':' . TEST_GROUP . ' /home/' . TEST_USER . "/test";
-
-            # On Alpine the test account must have a password for SSH logon
-            if ($$oVm{$strOS}{&VM_OS_BASE} eq VM_OS_BASE_ALPINE)
-            {
-                $strScript .=
-                    " && \\\n" .
-                    "    passwd -d '" . TEST_USER . "' " . TEST_USER;
-            }
-
-            $strScript .= sectionHeader() .
-                "# Configure sudo\n";
-
-            if ($$oVm{$strOS}{&VM_OS_BASE} eq VM_OS_BASE_RHEL)
-            {
-                $strScript .=
-                    # Don't allow sudo to disable core dump (suppresses errors, see https://github.com/sudo-project/sudo/issues/42)
-                    "    echo \"Set disable_coredump false\" >> /etc/sudo.conf && \\\n" .
-                    "    echo '%" . TEST_GROUP . "        ALL=(ALL)       NOPASSWD: ALL' > /etc/sudoers.d/" . TEST_GROUP .
-                        " && \\\n" .
-                    "    sed -i 's/^Defaults    requiretty\$/\\# Defaults    requiretty/' /etc/sudoers";
-            }
-            else
-            {
-                $strScript .=
-                    "    echo '%" . TEST_GROUP . " ALL=(ALL) NOPASSWD: ALL' >> /etc/sudoers";
-            }
-
-            $strScript .= sshSetup($strOS, TEST_USER, TEST_GROUP);
-
-            $strScript .= sectionHeader() .
-                "# Make " . TEST_USER . " home dir readable\n" .
-                '    chmod g+r,g+x /home/' . TEST_USER;
-
-            $strScript .= entryPointSetup($strOS);
-
-            containerWrite(
-                $oStorageDocker, $strTempPath, $strOS, $strArch, 'Test', $strImageParent, $strImage, $strCopy, $strScript,
-                $bVmForce);
-        }
-    }
-
-    &log(INFO, "Build Complete");
-}
-
-push @EXPORT, qw(containerBuild);
-
-####################################################################################################################################
-# containerRemove
-#
-# Remove containers that match a regexp.
-####################################################################################################################################
-sub containerRemove
-{
-    my $strRegExp = shift;
-
-    foreach my $strContainer (sort(split("\n", trim(executeTest('docker ps -a --format "{{.Names}}"')))))
-    {
-        if ($strContainer =~ $strRegExp)
-        {
-            executeTest("docker rm -f ${strContainer}", {bSuppressError => true});
-        }
-    }
-}
-
-push @EXPORT, qw(containerRemove);
-
-####################################################################################################################################
-# imageRemove
-#
-# Remove images that match a regexp.
-####################################################################################################################################
-sub imageRemove
-{
-    my $strRegExp = shift;
-
-    foreach my $strImage (sort(split("\n", trim(executeTest('docker images --format "{{.Repository}}:{{.Tag}}"')))))
-    {
-        if ($strImage =~ $strRegExp)
-        {
-            &log(INFO, "Removing ${strImage} image...");
-            executeTest("docker rmi -f ${strImage}");
-        }
-    }
-}
-
-1;

test/lib/pgBackRestTest/Common/DbVersion.pm

@@ -1,64 +0,0 @@
-####################################################################################################################################
-# DB VERSION MODULE
-####################################################################################################################################
-package pgBackRestTest::Common::DbVersion;
-
-use strict;
-use warnings FATAL => qw(all);
-use Carp qw(confess);
-
-use Exporter qw(import);
-    our @EXPORT = qw();
-
-use pgBackRestDoc::Common::Log;
-
-####################################################################################################################################
-# PostgreSQL version numbers
-####################################################################################################################################
-use constant PG_VERSION_96                                          => '9.6';
-    push @EXPORT, qw(PG_VERSION_96);
-use constant PG_VERSION_10                                          => '10';
-    push @EXPORT, qw(PG_VERSION_10);
-use constant PG_VERSION_11                                          => '11';
-    push @EXPORT, qw(PG_VERSION_11);
-use constant PG_VERSION_12                                          => '12';
-    push @EXPORT, qw(PG_VERSION_12);
-use constant PG_VERSION_13                                          => '13';
-    push @EXPORT, qw(PG_VERSION_13);
-use constant PG_VERSION_14                                          => '14';
-    push @EXPORT, qw(PG_VERSION_14);
-use constant PG_VERSION_15                                          => '15';
-    push @EXPORT, qw(PG_VERSION_15);
-use constant PG_VERSION_16                                          => '16';
-    push @EXPORT, qw(PG_VERSION_16);
-use constant PG_VERSION_17                                          => '17';
-    push @EXPORT, qw(PG_VERSION_17);
-use constant PG_VERSION_18                                          => '18';
-    push @EXPORT, qw(PG_VERSION_18);
-use constant PG_VERSION_19                                          => '19';
-    push @EXPORT, qw(PG_VERSION_19);
-
-####################################################################################################################################
-# versionSupport
-#
-# Returns an array of the supported Postgres versions.
-####################################################################################################################################
-sub versionSupport
-{
-    # Assign function parameters, defaults, and log debug info
-    my ($strOperation) = logDebugParam(__PACKAGE__ . '->versionSupport');
-
-    my @strySupportVersion = (PG_VERSION_96, PG_VERSION_10, PG_VERSION_11, PG_VERSION_12, PG_VERSION_13, PG_VERSION_14,
-                              PG_VERSION_15, PG_VERSION_16, PG_VERSION_17, PG_VERSION_18, PG_VERSION_19);
-
-    # Return from function and log return values if any
-    return logDebugReturn
-    (
-        $strOperation,
-        {name => 'strySupportVersion', value => \@strySupportVersion}
-    );
-}
-
-push @EXPORT, qw(versionSupport);
-
-1;

test/lib/pgBackRestTest/Common/DefineTest.pm

@@ -1,285 +0,0 @@
-####################################################################################################################################
-# DefineTest.pm - Defines all tests that can be run
-####################################################################################################################################
-package pgBackRestTest::Common::DefineTest;
-
-####################################################################################################################################
-# Perl includes
-####################################################################################################################################
-use strict;
-use warnings FATAL => qw(all);
-use Carp qw(confess);
-
-use Exporter qw(import);
-    our @EXPORT = qw();
-
-use pgBackRestDoc::Common::Log;
-use pgBackRestDoc::Common::String;
-
-use pgBackRestTest::Common::VmTest;
-
-################################################################################################################################
-# Test definition constants
-#
-# Documentation for these constants is in test/define.yaml.
-################################################################################################################################
-use constant TESTDEF_INTEGRATION                                    => 'integration';
-    push @EXPORT, qw(TESTDEF_INTEGRATION);
-use constant TESTDEF_PERFORMANCE                                    => 'performance';
-    push @EXPORT, qw(TESTDEF_PERFORMANCE);
-use constant TESTDEF_UNIT                                           => 'unit';
-    push @EXPORT, qw(TESTDEF_UNIT);
-
-use constant TESTDEF_MODULE                                         => 'module';
-    push @EXPORT, qw(TESTDEF_MODULE);
-use constant TESTDEF_NAME                                           => 'name';
-    push @EXPORT, qw(TESTDEF_NAME);
-use constant TESTDEF_TEST                                           => 'test';
-    push @EXPORT, qw(TESTDEF_TEST);
-
-use constant TESTDEF_DB                                             => 'db';
-    push @EXPORT, qw(TESTDEF_DB);
-use constant TESTDEF_CONTAINER                                      => 'container';
-    push @EXPORT, qw(TESTDEF_CONTAINER);
-use constant TESTDEF_CONTAINER_REQUIRED                             => 'containerReq';
-    push @EXPORT, qw(TESTDEF_CONTAINER_REQUIRED);
-use constant TESTDEF_COVERAGE                                       => 'coverage';
-    push @EXPORT, qw(TESTDEF_COVERAGE);
-use constant TESTDEF_C                                              => 'c';
-    push @EXPORT, qw(TESTDEF_C);
-use constant TESTDEF_LANG                                           => 'lang';
-    push @EXPORT, qw(TESTDEF_LANG);
-use constant TESTDEF_LANG_PYTHON                                    => 'python';
-    push @EXPORT, qw(TESTDEF_LANG_PYTHON);
-use constant TESTDEF_PYTHON                                         => 'python';
-    push @EXPORT, qw(TESTDEF_PYTHON);
-use constant TESTDEF_INCLUDE                                        => 'include';
-    push @EXPORT, qw(TESTDEF_INCLUDE);
-use constant TESTDEF_INDIVIDUAL                                     => 'individual';
-    push @EXPORT, qw(TESTDEF_INDIVIDUAL);
-use constant TESTDEF_TOTAL                                          => 'total';
-    push @EXPORT, qw(TESTDEF_TOTAL);
-use constant TESTDEF_TYPE                                           => 'type';
-    push @EXPORT, qw(TESTDEF_TYPE);
-use constant TESTDEF_BIN_REQ                                        => 'binReq';
-    push @EXPORT, qw(TESTDEF_BIN_REQ);
-use constant TESTDEF_VM                                             => 'vm';
-    push @EXPORT, qw(TESTDEF_VM);
-
-use constant TESTDEF_COVERAGE_FULL                                  => 'full';
-    push @EXPORT, qw(TESTDEF_COVERAGE_FULL);
-
-####################################################################################################################################
-# Process normalized data into a more queryable form
-####################################################################################################################################
-my $hTestDefHash;                                                   # An easier way to query hash version of the above
-my @stryModule;                                                     # Ordered list of modules
-my $hModuleTest;                                                    # Ordered list of tests for each module
-my $hCoverageType;                                                  # Coverage type for each code module (full/partial)
-my $hCoverageList;                                                  # Tests required for full code module coverage (if type full)
-
-sub testDefLoad
-{
-    my $strDefineYaml = shift;
-
-    # Load test definitions from yaml
-    require YAML::XS;
-    YAML::XS->import(qw(Load));
-
-    my $hTestDef = Load($strDefineYaml);
-
-    # Keep a list of all harnesses added so far. These will make up the harness list for subsequent tests.
-    my @rhyHarnessFile = ();
-
-    # Keep a list of all modules added for coverage so far. These will make up the core list for subsequent tests.
-    my @stryCoreFile = ();
-
-    # Keep a list of modules that are test before this one so we know what is available
-    my $strTestDefine = '';
-
-    # Iterate each test type
-    foreach my $strModuleType (TESTDEF_UNIT, TESTDEF_INTEGRATION, TESTDEF_PERFORMANCE)
-    {
-        my $hModuleType = $hTestDef->{$strModuleType};
-
-        my $bContainer = true;                                      # By default run tests in a single container
-        my $bIndividual = false;                                    # By default runs are all executed in the same container
-
-        if ($strModuleType eq TESTDEF_INTEGRATION)
-        {
-            $bContainer = false;                                    # Integration tests can run in multiple containers
-            $bIndividual = true;                                    # Integration tests can change containers on each run
-        }
-
-        # Iterate each module
-        foreach my $hModule (@{$hModuleType})
-        {
-            # Push the module onto the ordered list
-            my $strModule = $hModule->{&TESTDEF_NAME};
-            push(@stryModule, $strModule);
-
-            # Iterate each test
-            my @stryModuleTest;
-
-            foreach my $hModuleTest (@{$hModule->{&TESTDEF_TEST}})
-            {
-                # Push the test on the order list
-                my $strTest = $hModuleTest->{&TESTDEF_NAME};
-                push(@stryModuleTest, $strTest);
-
-                # Resolve variables that can be set in the module or the test
-                foreach my $strVar (
-                    TESTDEF_DB, TESTDEF_BIN_REQ, TESTDEF_VM, TESTDEF_CONTAINER_REQUIRED)
-                {
-                    $hTestDefHash->{$strModule}{$strTest}{$strVar} = coalesce(
-                        $hModuleTest->{$strVar}, $hModule->{$strVar}, $strVar eq TESTDEF_VM ? undef : false);
-
-                    # Make false = 0 for debugging
-                    if ($strVar ne TESTDEF_VM && $hTestDefHash->{$strModule}{$strTest}{$strVar} eq '')
-                    {
-                        $hTestDefHash->{$strModule}{$strTest}{$strVar} = false;
-                    }
-                }
-
-                # Set module type variables. A python test runs natively like a C test, i.e. it does not require a container, but it
-                # is not built so it is flagged separately. Language may be set for the whole module and overridden by a test.
-                my $strLang = coalesce($hModuleTest->{&TESTDEF_LANG}, $hModule->{&TESTDEF_LANG});
-
-                $hTestDefHash->{$strModule}{$strTest}{&TESTDEF_TYPE} = $strModuleType;
-                $hTestDefHash->{$strModule}{$strTest}{&TESTDEF_PYTHON} =
-                    defined($strLang) && $strLang eq TESTDEF_LANG_PYTHON ? true : false;
-                $hTestDefHash->{$strModule}{$strTest}{&TESTDEF_C} =
-                    $strModuleType ne TESTDEF_INTEGRATION && $strTest !~ /perl$/ ? true : false;
-                $hTestDefHash->{$strModule}{$strTest}{&TESTDEF_INTEGRATION} = $strModuleType eq TESTDEF_INTEGRATION ? true : false;
-                $hTestDefHash->{$strModule}{$strTest}{&TESTDEF_CONTAINER} = $bContainer;
-                $hTestDefHash->{$strModule}{$strTest}{&TESTDEF_INDIVIDUAL} = $bIndividual;
-
-                # Set test count
-                $hTestDefHash->{$strModule}{$strTest}{&TESTDEF_TOTAL} = $hModuleTest->{&TESTDEF_TOTAL};
-
-                # If this is a C test then add the test module to coverage
-                if ($hModuleTest->{&TESTDEF_C})
-                {
-                    my $strTestFile = "module/${strModule}/${strTest}Test";
-
-                    $hModuleTest->{&TESTDEF_COVERAGE}{$strTestFile} = TESTDEF_COVERAGE_FULL;
-                }
-
-                # Concatenate coverage for tests
-                foreach my $xCodeModule (@{$hModuleTest->{&TESTDEF_COVERAGE}})
-                {
-                    my $strCodeModule = undef;
-                    my $strCoverage = undef;
-
-                    if (ref($xCodeModule))
-                    {
-                        $strCodeModule = (keys(%{$xCodeModule}))[0];
-                        $strCoverage = $xCodeModule->{$strCodeModule};
-                    }
-                    else
-                    {
-                        $strCodeModule = $xCodeModule;
-                        $strCoverage = TESTDEF_COVERAGE_FULL;
-                    }
-
-                    $hTestDefHash->{$strModule}{$strTest}{&TESTDEF_COVERAGE}{$strCodeModule} = $strCoverage;
-
-                    # Build coverage type hash and make sure coverage type does not change
-                    if (!defined($hCoverageType->{$strCodeModule}))
-                    {
-                        $hCoverageType->{$strCodeModule} = $strCoverage;
-                    }
-                    elsif ($hCoverageType->{$strCodeModule} ne $strCoverage)
-                    {
-                        confess &log(ASSERT, "cannot mix coverage types for ${strCodeModule}");
-                    }
-
-                    # Add to coverage list
-                    push(@{$hCoverageList->{$strCodeModule}}, {strModule=> $strModule, strTest => $strTest});
-
-                    # Check if this module is already in the core list
-                    if (!$hTestDefHash->{$strModule}{$strTest}{&TESTDEF_INTEGRATION} && !grep(/^$strCodeModule$/i, @stryCoreFile))
-                    {
-                        push(@stryCoreFile, $strCodeModule);
-                    }
-                }
-
-                # Set include list
-                @{$hTestDefHash->{$strModule}{$strTest}{&TESTDEF_INCLUDE}} = ();
-
-                if (defined($hModuleTest->{&TESTDEF_INCLUDE}))
-                {
-                    push(@{$hTestDefHash->{$strModule}{$strTest}{&TESTDEF_INCLUDE}}, @{$hModuleTest->{&TESTDEF_INCLUDE}});
-                }
-            }
-
-            $hModuleTest->{$strModule} = \@stryModuleTest;
-        }
-    }
-}
-
-push @EXPORT, qw(testDefLoad);
-
-####################################################################################################################################
-# testDefModuleList
-####################################################################################################################################
-sub testDefModuleList
-{
-    return @stryModule;
-}
-
-push @EXPORT, qw(testDefModuleList);
-
-####################################################################################################################################
-# testDefModule
-####################################################################################################################################
-sub testDefModule
-{
-    my $strModule = shift;
-
-    if (!defined($hTestDefHash->{$strModule}))
-    {
-        confess &log(ASSERT, "unable to find module ${strModule}");
-    }
-
-    return $hTestDefHash->{$strModule};
-}
-
-push @EXPORT, qw(testDefModule);
-
-####################################################################################################################################
-# testDefModuleTestList
-####################################################################################################################################
-sub testDefModuleTestList
-{
-    my $strModule = shift;
-
-    if (!defined($hModuleTest->{$strModule}))
-    {
-        confess &log(ASSERT, "unable to find module ${strModule}");
-    }
-
-    return @{$hModuleTest->{$strModule}};
-}
-
-push @EXPORT, qw(testDefModuleTestList);
-
-####################################################################################################################################
-# testDefModuleTest
-####################################################################################################################################
-sub testDefModuleTest
-{
-    my $strModule = shift;
-    my $strModuleTest = shift;
-
-    if (!defined($hTestDefHash->{$strModule}{$strModuleTest}))
-    {
-        confess &log(ASSERT, "unable to find module ${strModule}, test ${strModuleTest}");
-    }
-
-    return $hTestDefHash->{$strModule}{$strModuleTest};
-}
-
-push @EXPORT, qw(testDefModuleTest);
-
-1;

test/lib/pgBackRestTest/Common/JobTest.pm

@@ -1,430 +0,0 @@
-####################################################################################################################################
-# JobTest.pm - Run a test job and monitor progress
-####################################################################################################################################
-package pgBackRestTest::Common::JobTest;
-
-####################################################################################################################################
-# Perl includes
-####################################################################################################################################
-use strict;
-use warnings FATAL => qw(all);
-use Carp qw(confess);
-use English '-no_match_vars';
-
-use Cwd qw(abs_path);
-use Exporter qw(import);
-    our @EXPORT = qw();
-use File::Basename qw(dirname basename);
-use POSIX qw(ceil);
-use Time::HiRes qw(gettimeofday usleep);
-
-use pgBackRestDoc::Common::Exception;
-use pgBackRestDoc::Common::Log;
-use pgBackRestDoc::Common::String;
-use pgBackRestDoc::ProjectInfo;
-
-use pgBackRestTest::Common::BuildTest;
-use pgBackRestTest::Common::ContainerTest;
-use pgBackRestTest::Common::DbVersion;
-use pgBackRestTest::Common::DefineTest;
-use pgBackRestTest::Common::ExecuteTest;
-use pgBackRestTest::Common::ListTest;
-use pgBackRestTest::Common::VmTest;
-
-####################################################################################################################################
-# new
-####################################################################################################################################
-sub new
-{
-    my $class = shift;          # Class name
-
-    # Create the class hash
-    my $self = {};
-    bless $self, $class;
-
-    # Assign function parameters, defaults, and log debug info
-    (
-        my $strOperation,
-        $self->{oStorageTest},
-        $self->{strBackRestBase},
-        $self->{strTestPath},
-        $self->{oTest},
-        $self->{bDryRun},
-        $self->{bVmOut},
-        $self->{strPlatform},
-        $self->{strVmArch},
-        $self->{strImage},
-        $self->{iVmIdx},
-        $self->{iVmMax},
-        $self->{iTestIdx},
-        $self->{iTestMax},
-        $self->{strLogLevel},
-        $self->{strLogLevelTest},
-        $self->{strLogLevelTestFile},
-        $self->{bLogTimestamp},
-        $self->{bShowOutputAsync},
-        $self->{bNoCleanup},
-        $self->{iRetry},
-        $self->{bBackTraceUnit},
-        $self->{bValgrindUnit},
-        $self->{bCoverageUnit},
-        $self->{bCoverageSummary},
-        $self->{bOptimize},
-        $self->{bProfile},
-        $self->{iScale},
-        $self->{strTimeZone},
-        $self->{bDebug},
-        $self->{bDebugTestTrace},
-        $self->{iBuildMax},
-    ) =
-        logDebugParam
-        (
-            __PACKAGE__ . '->new', \@_,
-            {name => 'oStorageTest'},
-            {name => 'strBackRestBase'},
-            {name => 'strTestPath'},
-            {name => 'oTest'},
-            {name => 'bDryRun'},
-            {name => 'bVmOut'},
-            {name => 'strPlatform'},
-            {name => 'strVmArch', required => false},
-            {name => 'strImage'},
-            {name => 'iVmIdx'},
-            {name => 'iVmMax'},
-            {name => 'iTestIdx'},
-            {name => 'iTestMax'},
-            {name => 'strLogLevel'},
-            {name => 'strLogLevelTest'},
-            {name => 'strLogLevelTestFile'},
-            {name => 'bLogTimestamp'},
-            {name => 'bShowOutputAsync'},
-            {name => 'bNoCleanup'},
-            {name => 'iRetry'},
-            {name => 'bBackTraceUnit'},
-            {name => 'bValgrindUnit'},
-            {name => 'bCoverageUnit'},
-            {name => 'bCoverageSummary'},
-            {name => 'bOptimize'},
-            {name => 'bProfile'},
-            {name => 'iScale'},
-            {name => 'strTimeZone', required => false},
-            {name => 'bDebug'},
-            {name => 'bDebugTestTrace'},
-            {name => 'iBuildMax'},
-        );
-
-    # Set try to 0
-    $self->{iTry} = 0;
-
-    # Setup the path where unit test will be built
-    $self->{strUnitPath} =
-        "$self->{strTestPath}/unit-$self->{iVmIdx}/" .
-        ($self->{oTest}->{&TEST_TYPE} eq TESTDEF_INTEGRATION ? 'none' : $self->{oTest}->{&TEST_VM});
-    $self->{strDataPath} = "$self->{strTestPath}/data-$self->{iVmIdx}";
-    $self->{strRepoPath} = "$self->{strTestPath}/repo";
-
-    # Return from function and log return values if any
-    return logDebugReturn
-    (
-        $strOperation,
-        {name => 'self', value => $self, trace => true}
-    );
-}
-
-####################################################################################################################################
-# run
-####################################################################################################################################
-sub run
-{
-    my $self = shift;
-
-    # Assign function parameters, defaults, and log debug info
-    (my $strOperation) = logDebugParam (__PACKAGE__ . '->run', \@_,);
-
-    # Start the test timer
-    my $fTestStartTime = gettimeofday();
-
-    # Was the job run?
-    my $bRun = false;
-
-    # Should the job be run?
-    $self->{iTry}++;
-
-    if ($self->{iTry} <= ($self->{iRetry} + 1))
-    {
-        if ($self->{iTry} != 1 && $self->{iTry} == ($self->{iRetry} + 1))
-        {
-            $self->{strLogLevel} = lc(DEBUG);
-        }
-
-        my $strTest = sprintf(
-            'P%0' . length($self->{iVmMax}) . 'd-T%0' . length($self->{iTestMax}) . 'd/%0' .
-            length($self->{iTestMax}) . "d - ", $self->{iVmIdx} + 1, $self->{iTestIdx} + 1, $self->{iTestMax}) .
-            'vm=' . $self->{oTest}->{&TEST_VM} .
-            ', module=' . $self->{oTest}->{&TEST_MODULE} .
-            ', test=' . $self->{oTest}->{&TEST_NAME} .
-            (defined($self->{oTest}->{&TEST_RUN}) ? ', run=' . join(',', sort(@{$self->{oTest}->{&TEST_RUN}})) : '') .
-            (defined($self->{oTest}->{&TEST_DB}) ? ', pg-version=' . $self->{oTest}->{&TEST_DB} : '') .
-            ($self->{iTry} > 1 ? ' (retry ' . ($self->{iTry} - 1) . ')' : '');
-
-        my $strImage = 'test-' . $self->{iVmIdx};
-        my $strDbVersion = (defined($self->{oTest}->{&TEST_DB}) ? $self->{oTest}->{&TEST_DB} : PG_VERSION_96);
-        $strDbVersion =~ s/\.//;
-
-        &log($self->{bDryRun} && !$self->{bVmOut} || $self->{bShowOutputAsync} ? INFO : DETAIL, "${strTest}" .
-             (!($self->{bDryRun} || !$self->{bVmOut}) || $self->{bShowOutputAsync} ? "\n" : ''));
-
-        my $strHostTestPath = "$self->{strTestPath}/${strImage}";
-        my $strVmTestPath = $strHostTestPath;
-
-        # Don't create the container if this is a dry run unless output from the VM is required. Output can be requested
-        # to get more information about the specific tests that will be run.
-        if (!$self->{bDryRun} || $self->{bVmOut})
-        {
-            # Create host test directory
-            $self->{oStorageTest}->pathCreate($strHostTestPath, {strMode => '0770'});
-
-            # Create unit directory
-            if ($self->{oTest}->{&TEST_C} && !$self->{oStorageTest}->pathExists($self->{strUnitPath}))
-            {
-                $self->{oStorageTest}->pathCreate($self->{strUnitPath}, {strMode => '0770', bCreateParent => true});
-            }
-
-            # Create data directory
-            if ($self->{oTest}->{&TEST_TYPE} eq TESTDEF_UNIT && !$self->{oStorageTest}->pathExists($self->{strDataPath}))
-            {
-                $self->{oStorageTest}->pathCreate($self->{strDataPath}, {strMode => '0770'});
-            }
-
-            # ccache path shared with the host and the build container. This is created by test.pl so it is guaranteed to exist
-            # before being mounted below, which keeps docker from creating it as root.
-            my $strCCachePath = "$self->{strTestPath}/ccache";
-
-            if ($self->{oTest}->{&TEST_CONTAINER})
-            {
-                if ($self->{oTest}->{&TEST_VM} ne VM_NONE)
-                {
-                    my $strBuildPath = $self->{strTestPath} . '/build/' . $self->{oTest}->{&TEST_VM};
-
-                    executeTest(
-                        'docker run' . $self->{strPlatform} . ' -itd -h ' .
-                        $self->{oTest}->{&TEST_VM} . "-test --name=${strImage} -v ${strHostTestPath}:${strVmTestPath}" .
-                        ($self->{oTest}->{&TEST_C} ? " -v $self->{strUnitPath}:$self->{strUnitPath}" : '') .
-                        ($self->{oTest}->{&TEST_C} ? " -v $self->{strDataPath}:$self->{strDataPath}" : '') .
-                        " -v $self->{strBackRestBase}:$self->{strBackRestBase}" .
-                        " -v $self->{strRepoPath}:$self->{strRepoPath}" .
-                        ($self->{oTest}->{&TEST_C} ? " -v ${strBuildPath}:${strBuildPath}:ro" : '') .
-                        ($self->{oTest}->{&TEST_C} ? " -v ${strCCachePath}:${strCCachePath}" : '') .
-                        ($self->{oTest}->{&TEST_C} ? " -e CCACHE_DIR=${strCCachePath}" : '') .
-                        ' ' . $self->{strImage},
-                        {bSuppressStdErr => true});
-                }
-            }
-        }
-
-        # Disable debug/coverage for performance and profile tests
-        my $bPerformance = $self->{oTest}->{&TEST_TYPE} eq TESTDEF_PERFORMANCE;
-
-        if ($bPerformance || $self->{bProfile})
-        {
-            $self->{bDebug} = false;
-            $self->{bDebugTestTrace} = false;
-            $self->{bCoverageUnit} = false;
-        }
-
-        # Is coverage being tested?
-        my $bCoverage = vmCoverageC($self->{oTest}->{&TEST_VM}) && $self->{bCoverageUnit};
-
-        # Name of the raw coverage file for this test. A python test name may contain a path separator so flatten it, since the
-        # coverage command reads the raw files from a single directory.
-        my $strCoverageName = $self->{oTest}->{&TEST_MODULE} . '-' . $self->{oTest}->{&TEST_NAME};
-        $strCoverageName =~ s/\//-/g;
-
-        # Create run parameters
-        my $strCommandRunParam = '';
-
-        foreach my $iRunIdx (@{$self->{oTest}->{&TEST_RUN}})
-        {
-            $strCommandRunParam .= ' --test=' . $iRunIdx;
-        }
-
-        if (!$self->{bDryRun} || $self->{bVmOut})
-        {
-            my $bValgrind = $self->{bValgrindUnit} && $self->{oTest}->{&TEST_TYPE} ne TESTDEF_PERFORMANCE;
-            my $strValgrindSuppress =
-                $self->{strRepoPath} . '/test/src/valgrind.suppress.' .
-                ($self->{oTest}->{&TEST_TYPE} eq TESTDEF_INTEGRATION ? VM_NONE : $self->{oTest}->{&TEST_VM});
-            my $strVm = $self->{oTest}->{&TEST_TYPE} eq TESTDEF_INTEGRATION ? VM_NONE : $self->{oTest}->{&TEST_VM};
-
-            my $strCommand =
-                ($strVm ne VM_NONE ? "docker exec -i -u ${\TEST_USER} ${strImage} bash -l -c '\\\n" : '') .
-                'python3 ' . $self->{strRepoPath} . '/test/test.py unit' .
-                    ' --repo-path=' . $self->{strRepoPath} . ' --test-path=' . $self->{strTestPath} .
-                    " --log-level=$self->{strLogLevel}" . ' --vm=' . $self->{oTest}->{&TEST_VM} .
-                    (defined($self->{strVmArch}) ? ' --vm-arch=' . $self->{strVmArch} : '') .
-                    ' --vm-id=' . $self->{iVmIdx} . ($self->{bProfile} ? ' --profile' : '') . $strCommandRunParam .
-                    ($self->{bLogTimestamp} ? '' : ' --no-log-timestamp') .
-                    ($self->{strTimeZone} ? " --tz='$self->{strTimeZone}'" : '') .
-                    ($self->{iScale} ? " --scale=$self->{iScale}" : '') .
-                    (defined($self->{oTest}->{&TEST_DB}) ? ' --pg-version=' . $self->{oTest}->{&TEST_DB} : '') .
-                    ($self->{bBackTraceUnit} ? '' : ' --no-back-trace') . ($bCoverage ? '' : ' --no-coverage') .
-                    # A python test writes its own coverage since there is no gcov step for it below
-                    ($self->{oTest}->{&TEST_PYTHON} && $bCoverage ?
-                        ' --coverage-file=' . $self->{strBackRestBase} . '/test/result/coverage/raw/' .
-                        $strCoverageName . '.json' : '') . ' ' .
-                    $self->{oTest}->{&TEST_MODULE} . '/' . $self->{oTest}->{&TEST_NAME} .
-                # A python test is run by the harness above so there is no binary to run here
-                ($self->{oTest}->{&TEST_PYTHON} ? '' :
-                    " && \\\n" .
-                    # Copy stderr to both stderr and stdout so it is displayed and detected as an error. Piping to tee would mask
-                    # the test exit status (the pipe returns tee's status) so save the status to a file and exit with it
-                    # explicitly. Otherwise a test failure (e.g. a valgrind error) would be hidden and reported as missing
-                    # coverage instead.
-                    "exec 3>&1 && \\\n" .
-                    '{ ' .
-                    # Test with valgrind when requested
-                    ($bValgrind ?
-                        'valgrind -q --gen-suppressions=all' .
-                        ($self->{oStorageTest}->exists($strValgrindSuppress) ? " --suppressions=${strValgrindSuppress}" : '') .
-                        " --exit-on-first-error=yes --leak-check=full --leak-resolution=high --error-exitcode=25" . ' ' : '') .
-                        "$self->{strUnitPath}/build/test-unit 2>&1 1>&3; echo \$? > $self->{strUnitPath}/result; } | " .
-                        "tee /dev/stderr && \\\n" .
-                    "exit \$(cat $self->{strUnitPath}/result)") .
-                ($strVm ne VM_NONE ? "'" : '');
-
-            my $oExec = new pgBackRestTest::Common::ExecuteTest(
-                $strCommand, {bSuppressError => true, bShowOutputAsync => $self->{bShowOutputAsync}});
-
-            $oExec->begin();
-
-            $self->{oProcess} =
-            {
-                exec => $oExec,
-                test => $strTest,
-                start_time => $fTestStartTime,
-            };
-
-            $bRun = true;
-        }
-    }
-
-    # Return from function and log return values if any
-    return logDebugReturn
-    (
-        $strOperation,
-        {name => 'bRun', value => $bRun, trace => true}
-    );
-}
-
-####################################################################################################################################
-# end
-####################################################################################################################################
-sub end
-{
-    my $self = shift;
-
-    # Assign function parameters, defaults, and log debug info
-    (my $strOperation) = logDebugParam (__PACKAGE__ . '->run', \@_,);
-
-    # Is the job done?
-    my $bDone = false;
-    my $bFail = false;
-
-    my $oExecDone = $self->{oProcess}{exec};
-    my $strTestDone = $self->{oProcess}{test};
-    my $iTestDoneIdx = $self->{oProcess}{idx};
-
-    my $iExitStatus = $oExecDone->end($self->{iVmMax} == 1);
-
-    if (defined($iExitStatus))
-    {
-        my $strImage = 'test-' . $self->{iVmIdx};
-
-        if ($self->{bShowOutputAsync})
-        {
-            syswrite(*STDOUT, "\n");
-        }
-
-        # If C code generate profile info
-        if ($iExitStatus == 0 && $self->{oTest}->{&TEST_C} && !$self->{oTest}->{&TEST_PYTHON} && $self->{bProfile})
-        {
-            executeTest(
-                ($self->{oTest}->{&TEST_VM} ne VM_NONE ? 'docker exec -i -u ' . TEST_USER . " ${strImage} " : '') .
-                    "gprof $self->{strUnitPath}/build/test-unit $self->{strUnitPath}/build/gmon.out >" .
-                    " $self->{strUnitPath}/gprof.txt");
-
-            $self->{oStorageTest}->pathCreate(
-                "$self->{strBackRestBase}/test/result/profile", {strMode => '0750', bIgnoreExists => true, bCreateParent => true});
-            $self->{oStorageTest}->copy(
-                "$self->{strUnitPath}/gprof.txt", "$self->{strBackRestBase}/test/result/profile/gprof.txt");
-        }
-
-        # If C code generate coverage info. A python test wrote its own coverage while it ran.
-        if ($iExitStatus == 0 && $self->{oTest}->{&TEST_C} && !$self->{oTest}->{&TEST_PYTHON} &&
-            vmCoverageC($self->{oTest}->{&TEST_VM}) && $self->{bCoverageUnit})
-        {
-            my $strCoverageName = $self->{oTest}->{&TEST_MODULE} . '-' . $self->{oTest}->{&TEST_NAME};
-            $strCoverageName =~ s/\//-/g;
-
-            executeTest(
-                ($self->{oTest}->{&TEST_VM} ne VM_NONE ? 'docker exec -i -u ' . TEST_USER . " ${strImage} " : '') .
-                    "gcov --json-format --stdout --branch-probabilities " .
-                    (-e "$self->{strUnitPath}/build/test-unit.p" ?
-                        "$self->{strUnitPath}/build/test-unit.p" : "$self->{strUnitPath}/build/test-unit\@exe") .
-                        '/test.c.gcda > ' . $self->{strBackRestBase} . '/test/result/coverage/raw/' .
-                        $strCoverageName . '.json');
-        }
-
-        # Record elapsed time
-        my $fTestElapsedTime = ceil((gettimeofday() - $self->{oProcess}{start_time}) * 100) / 100;
-
-        # Output error
-        if ($iExitStatus != 0 || (defined($oExecDone->{strErrorLog}) && $oExecDone->{strErrorLog} ne ''))
-        {
-            # Get stdout (no need to get stderr since stderr is redirected to stdout)
-            my $strOutput = trim($oExecDone->{strOutLog}) ? "STDOUT:\n" . trim($oExecDone->{strOutLog}) : '';
-
-            # If no stdout or stderr output something rather than a blank line
-            if ($strOutput eq '')
-            {
-                $strOutput = 'NO OUTPUT ON STDOUT OR STDERR';
-            }
-
-            &log(ERROR, "${strTestDone} (err${iExitStatus}" . ($self->{bLogTimestamp} ? "-${fTestElapsedTime}s)" : '') .
-                 (defined($oExecDone->{strOutLog}) && !$self->{bShowOutputAsync} ? ":\n\n${strOutput}\n" : ''), undef, undef, 4);
-
-            $bFail = true;
-        }
-        # Output success
-        else
-        {
-            &log(INFO, "${strTestDone}" . ($self->{bLogTimestamp} ? " (${fTestElapsedTime}s)" : '').
-                 ($self->{bVmOut} && !$self->{bShowOutputAsync} ?
-                     ":\n\n" . trim($oExecDone->{strOutLog}) . "\n" : ''), undef, undef, 4);
-        }
-
-        if (!$self->{bNoCleanup})
-        {
-            my $strHostTestPath = "$self->{strTestPath}/${strImage}";
-
-            if ($self->{oTest}->{&TEST_VM} ne VM_NONE)
-            {
-                containerRemove("test-$self->{iVmIdx}");
-            }
-
-            executeTest("chmod -R 700 ${strHostTestPath}/* 2>&1;rm -rf ${strHostTestPath}");
-        }
-
-        $bDone = true;
-    }
-
-    # Return from function and log return values if any
-    return logDebugReturn
-    (
-        $strOperation,
-        {name => 'bDone', value => $bDone, trace => true},
-        {name => 'bFail', value => $bFail, trace => true}
-    );
-}
-
-1;

test/lib/pgBackRestTest/Common/ListTest.pm

@@ -1,200 +0,0 @@
-####################################################################################################################################
-# ListTest.pm - Creates a list of tests to be run based on input criteria
-####################################################################################################################################
-package pgBackRestTest::Common::ListTest;
-
-####################################################################################################################################
-# Perl includes
-####################################################################################################################################
-use strict;
-use warnings FATAL => qw(all);
-use Carp qw(confess);
-
-use Exporter qw(import);
-    our @EXPORT = qw();
-
-use pgBackRestDoc::Common::Log;
-use pgBackRestDoc::Common::String;
-
-use pgBackRestTest::Common::DefineTest;
-use pgBackRestTest::Common::VmTest;
-
-################################################################################################################################
-# Test constants
-################################################################################################################################
-use constant TEST_DB                                                => 'db';
-    push @EXPORT, qw(TEST_DB);
-use constant TEST_C                                                 => 'c';
-    push @EXPORT, qw(TEST_C);
-use constant TEST_CONTAINER                                         => 'container';
-    push @EXPORT, qw(TEST_CONTAINER);
-use constant TEST_MODULE                                            => 'module';
-    push @EXPORT, qw(TEST_MODULE);
-use constant TEST_NAME                                              => 'test';
-    push @EXPORT, qw(TEST_NAME);
-use constant TEST_BIN_REQ                                           => 'bin-req';
-    push @EXPORT, qw(TEST_BIN_REQ);
-use constant TEST_PGSQL_BIN                                         => 'psql-bin';
-    push @EXPORT, qw(TEST_PGSQL_BIN);
-use constant TEST_INTEGRATION                                       => 'integration';
-    push @EXPORT, qw(TEST_INTEGRATION);
-use constant TEST_TYPE                                              => 'type';
-    push @EXPORT, qw(TEST_TYPE);
-use constant TEST_RUN                                               => 'run';
-    push @EXPORT, qw(TEST_RUN);
-use constant TEST_VM                                                => 'os';
-    push @EXPORT, qw(TEST_VM);
-use constant TEST_PYTHON                                            => 'python';
-    push @EXPORT, qw(TEST_PYTHON);
-
-####################################################################################################################################
-# testListGet
-####################################################################################################################################
-sub testListGet
-{
-    my $strVm = shift;
-    my $stryModule = shift;
-    my $stryModuleTest = shift;
-    my $iyModuleTestRun = shift;
-    my $strDbVersion = shift;
-    my $bCoverageOnly = shift;
-    my $bCOnly = shift;
-    my $bContainerOnly = shift;
-    my $bNoPerformance = shift;
-
-    my $oyVm = vmGet();
-    my $oyTestRun = [];
-
-    if ($strVm ne 'all' && !defined($${oyVm}{$strVm}))
-    {
-        confess &log(ERROR, "${strVm} is not a valid VM");
-    }
-
-    my @stryTestOS = VM_LIST;
-
-    if ($strVm ne 'all')
-    {
-        @stryTestOS = ($strVm);
-    }
-
-    foreach my $strTestOS (@stryTestOS)
-    {
-        foreach my $strModule (testDefModuleList())
-        {
-            my $hModule = testDefModule($strModule);
-
-            if (@{$stryModule} == 0 || grep(/^$strModule$/i, @{$stryModule}))
-            {
-                foreach my $strModuleTest (testDefModuleTestList($strModule))
-                {
-                    my $hTest = testDefModuleTest($strModule, $strModuleTest);
-
-                    if (@{$stryModuleTest} == 0 || grep(/^$strModuleTest$/i, @{$stryModuleTest}))
-                    {
-                        my $iDbVersionMin = -1;
-                        my $iDbVersionMax = -1;
-
-                        # Database versions to test
-                        if (defined($hTest->{&TESTDEF_DB}) && $hTest->{&TESTDEF_DB})
-                        {
-                            $iDbVersionMin = 0;
-                            $iDbVersionMax = @{$$oyVm{$strTestOS}{&VM_DB_TEST}} - 1;
-                        }
-
-                        my $bFirstDbVersion = true;
-
-                        # Skip this test if it can't run on this VM
-                        next if (defined($hTest->{&TESTDEF_VM}) && grep(/^$strTestOS$/i, @{$hTest->{&TESTDEF_VM}}) == 0);
-
-                        # Skip this test if only C tests are requested and this is not a C test
-                        next if ($bCOnly && !$hTest->{&TESTDEF_C});
-
-                        # Skip this test if it is integration and vm=none
-                        next if ($strVm eq VM_NONE && $hTest->{&TESTDEF_TYPE} eq TESTDEF_INTEGRATION);
-
-                        # Skip this test if it is performance and no performance is specified
-                        next if ($bNoPerformance && $hTest->{&TESTDEF_TYPE} eq TESTDEF_PERFORMANCE);
-
-                        # Skip this test if it is not C and vm=none. Perl tests require Docker which is not supported.
-                        next if ($strVm eq VM_NONE && !$hTest->{&TESTDEF_C});
-
-                        # Skip this test if a container is required and vm=none.
-                        next if ($strVm eq VM_NONE && $hTest->{&TESTDEF_CONTAINER_REQUIRED});
-
-                        # Skip this if it does not require a container and container only tests are required.
-                        next if ($bContainerOnly && $hTest->{&TESTDEF_C} && !$hTest->{&TESTDEF_CONTAINER_REQUIRED});
-
-                        for (my $iDbVersionIdx = $iDbVersionMax; $iDbVersionIdx >= $iDbVersionMin; $iDbVersionIdx--)
-                        {
-                            if ($iDbVersionIdx == -1 || $strDbVersion eq 'all' || $strDbVersion eq 'minimal' ||
-                                ($strDbVersion ne 'all' &&
-                                    $strDbVersion eq ${$$oyVm{$strTestOS}{&VM_DB_TEST}}[$iDbVersionIdx]))
-                            {
-                                # Individual tests will be each be run in a separate container. This is the default.
-                                my $bTestIndividual =
-                                    !defined($hTest->{&TESTDEF_INDIVIDUAL}) || $hTest->{&TESTDEF_INDIVIDUAL} ? true : false;
-
-                                my $iTestRunMin = $bTestIndividual ? 1 : -1;
-                                my $iTestRunMax = $bTestIndividual ? $hTest->{&TESTDEF_TOTAL} : -1;
-
-                                for (my $iTestRunIdx = $iTestRunMin; $iTestRunIdx <= $iTestRunMax; $iTestRunIdx++)
-                                {
-                                    # Skip this run if a list was provided and this test is not in the list
-                                    next if (
-                                        $bTestIndividual && @{$iyModuleTestRun} != 0 &&
-                                            !grep(/^$iTestRunIdx$/i, @{$iyModuleTestRun}));
-
-                                    # Skip this run if only coverage tests are requested and this test does not provide coverage
-                                    next if ($bCoverageOnly && !defined($hTest->{&TESTDEF_COVERAGE}));
-
-                                    my $strDbVersion = $iDbVersionIdx == -1 ? undef :
-                                                           ${$$oyVm{$strTestOS}{&VM_DB_TEST}}[$iDbVersionIdx];
-
-                                    my $strPgSqlBin = $$oyVm{$strTestOS}{&VMDEF_PGSQL_BIN};
-
-                                    if (defined($strDbVersion))
-                                    {
-                                        $strPgSqlBin =~ s/\{\[version\]\}/$strDbVersion/g;
-                                    }
-                                    else
-                                    {
-                                        $strPgSqlBin =~ s/\{\[version\]\}/9\.4/g;
-                                    }
-
-                                    my $oTestRun =
-                                    {
-                                        &TEST_TYPE => $hTest->{&TESTDEF_TYPE},
-                                        &TEST_VM => $strTestOS,
-                                        &TEST_C => coalesce($hTest->{&TESTDEF_C}, $hModule->{&TESTDEF_C}, false),
-                                        &TEST_CONTAINER => defined($hTest->{&TESTDEF_CONTAINER}) ?
-                                            $hTest->{&TESTDEF_CONTAINER} : $hModule->{&TESTDEF_CONTAINER},
-                                        &TEST_PGSQL_BIN => $strPgSqlBin,
-                                        &TEST_BIN_REQ => $hTest->{&TESTDEF_BIN_REQ},
-                                        &TEST_INTEGRATION => $hTest->{&TESTDEF_INTEGRATION},
-                                        &TEST_MODULE => $strModule,
-                                        &TEST_NAME => $strModuleTest,
-                                        &TEST_PYTHON => $hTest->{&TESTDEF_PYTHON},
-                                        &TEST_RUN =>
-                                            $iTestRunIdx == -1 ? (@{$iyModuleTestRun} == 0 ? undef : $iyModuleTestRun) :
-                                                [$iTestRunIdx],
-                                        &TEST_DB => $strDbVersion
-                                    };
-
-                                    push(@{$oyTestRun}, $oTestRun);
-                                }
-
-                                $bFirstDbVersion = false;
-                            }
-                        }
-                    }
-                }
-            }
-        }
-    }
-
-    return $oyTestRun;
-}
-
-push @EXPORT, qw(testListGet);
-
-1;

test/lib/pgBackRestTest/Common/StoragePosix.pm

@@ -1,915 +0,0 @@
-####################################################################################################################################
-# Posix Storage
-#
-# Implements storage functions for Posix-compliant file systems.
-####################################################################################################################################
-package pgBackRestTest::Common::StoragePosix;
-
-use strict;
-use warnings FATAL => qw(all);
-use Carp qw(confess);
-use English '-no_match_vars';
-
-use Exporter qw(import);
-    our @EXPORT = qw();
-use File::Basename qw(basename dirname);
-use Fcntl qw(:mode);
-use File::stat qw{lstat};
-
-use pgBackRestDoc::Common::Exception;
-use pgBackRestDoc::Common::Log;
-
-use pgBackRestTest::Common::StorageBase;
-use pgBackRestTest::Common::StoragePosixRead;
-use pgBackRestTest::Common::StoragePosixWrite;
-
-####################################################################################################################################
-# Package name constant
-####################################################################################################################################
-use constant STORAGE_POSIX_DRIVER                                      => __PACKAGE__;
-    push @EXPORT, qw(STORAGE_POSIX_DRIVER);
-
-####################################################################################################################################
-# new
-####################################################################################################################################
-sub new
-{
-    my $class = shift;
-
-    # Create the class hash
-    my $self = {};
-    bless $self, $class;
-
-    # Assign function parameters, defaults, and log debug info
-    (
-        my $strOperation,
-        $self->{bFileSync},
-        $self->{bPathSync},
-    ) =
-        logDebugParam
-        (
-            __PACKAGE__ . '->new', \@_,
-            {name => 'bFileSync', optional => true, default => true},
-            {name => 'bPathSync', optional => true, default => true},
-        );
-
-    # Set default temp extension
-    $self->{strTempExtension} = 'tmp';
-
-    # Return from function and log return values if any
-    return logDebugReturn
-    (
-        $strOperation,
-        {name => 'self', value => $self, trace => true}
-    );
-}
-
-####################################################################################################################################
-# exists - check if a path or file exists
-####################################################################################################################################
-sub exists
-{
-    my $self = shift;
-
-    # Assign function parameters, defaults, and log debug info
-    my
-    (
-        $strOperation,
-        $strFile,
-    ) =
-        logDebugParam
-        (
-            __PACKAGE__ . '->exists', \@_,
-            {name => 'strFile', trace => true},
-        );
-
-    # Does the path/file exist?
-    my $bExists = true;
-    my $oStat = lstat($strFile);
-
-    # Use stat to test if file exists
-    if (defined($oStat))
-    {
-        # Check that it is actually a file
-        $bExists = !S_ISDIR($oStat->mode) ? true : false;
-    }
-    else
-    {
-        # If the error is not entry missing, then throw error
-        if (!$OS_ERROR{ENOENT})
-        {
-            logErrorResult(ERROR_FILE_EXISTS, "unable to test if file '${strFile}' exists", $OS_ERROR);
-        }
-
-        $bExists = false;
-    }
-
-    # Return from function and log return values if any
-    return logDebugReturn
-    (
-        $strOperation,
-        {name => 'bExists', value => $bExists, trace => true}
-    );
-}
-
-####################################################################################################################################
-# info - get information for path/file
-####################################################################################################################################
-sub info
-{
-    my $self = shift;
-
-    # Assign function parameters, defaults, and log debug info
-    my
-    (
-        $strOperation,
-        $strPathFile,
-        $bIgnoreMissing,
-    ) =
-        logDebugParam
-        (
-            __PACKAGE__ . '->info', \@_,
-            {name => 'strFile', trace => true},
-            {name => 'bIgnoreMissing', optional => true, default => false, trace => true},
-        );
-
-    # Stat the path/file
-    my $oInfo = lstat($strPathFile);
-
-    # Check for errors
-    if (!defined($oInfo))
-    {
-        if (!($OS_ERROR{ENOENT} && $bIgnoreMissing))
-        {
-            logErrorResult($OS_ERROR{ENOENT} ? ERROR_FILE_MISSING : ERROR_FILE_OPEN, "unable to stat '${strPathFile}'", $OS_ERROR);
-        }
-    }
-
-    # Return from function and log return values if any
-    return logDebugReturn
-    (
-        $strOperation,
-        {name => 'oInfo', value => $oInfo, trace => true}
-    );
-}
-
-####################################################################################################################################
-# linkDestination - get destination of symlink
-####################################################################################################################################
-sub linkDestination
-{
-    my $self = shift;
-
-    # Assign function parameters, defaults, and log debug info
-    my
-    (
-        $strOperation,
-        $strLink,
-    ) =
-        logDebugParam
-        (
-            __PACKAGE__ . '->linkDestination', \@_,
-            {name => 'strLink', trace => true},
-        );
-
-    # Get link destination
-    my $strLinkDestination = readlink($strLink);
-
-    # Check for errors
-    if (!defined($strLinkDestination))
-    {
-        logErrorResult(
-            $OS_ERROR{ENOENT} ? ERROR_FILE_MISSING : ERROR_FILE_OPEN, "unable to get destination for link ${strLink}", $OS_ERROR);
-    }
-
-    # Return from function and log return values if any
-    return logDebugReturn
-    (
-        $strOperation,
-        {name => 'strLinkDestination', value => $strLinkDestination, trace => true}
-    );
-}
-
-####################################################################################################################################
-# list - list all files/paths in path
-####################################################################################################################################
-sub list
-{
-    my $self = shift;
-
-    # Assign function parameters, defaults, and log debug info
-    my
-    (
-        $strOperation,
-        $strPath,
-        $bIgnoreMissing,
-    ) =
-        logDebugParam
-        (
-            __PACKAGE__ . '->list', \@_,
-            {name => 'strPath', trace => true},
-            {name => 'bIgnoreMissing', optional => true, default => false, trace => true},
-        );
-
-    # Working variables
-    my @stryFileList;
-    my $hPath;
-
-    # Attempt to open the path
-    if (opendir($hPath, $strPath))
-    {
-        @stryFileList = grep(!/^(\.|\.\.)$/m, readdir($hPath));
-        close($hPath);
-    }
-    # Else process errors
-    else
-    {
-        # Ignore the error if the file is missing and missing files should be ignored
-        if (!($OS_ERROR{ENOENT} && $bIgnoreMissing))
-        {
-            logErrorResult($OS_ERROR{ENOENT} ? ERROR_FILE_MISSING : ERROR_FILE_OPEN, "unable to read path '${strPath}'", $OS_ERROR);
-        }
-    }
-
-    # Return from function and log return values if any
-    return logDebugReturn
-    (
-        $strOperation,
-        {name => 'stryFileList', value => \@stryFileList, ref => true, trace => true}
-    );
-}
-
-####################################################################################################################################
-# manifest - build path/file/link manifest starting with base path and including all subpaths
-####################################################################################################################################
-sub manifest
-{
-    my $self = shift;
-
-    # Assign function parameters, defaults, and log debug info
-    my
-    (
-        $strOperation,
-        $strPath,
-        $bIgnoreMissing,
-        $strFilter,
-    ) =
-        logDebugParam
-        (
-            __PACKAGE__ . '->manifest', \@_,
-            {name => 'strPath', trace => true},
-            {name => 'bIgnoreMissing', optional => true, default => false, trace => true},
-            {name => 'strFilter', optional => true, trace => true},
-        );
-
-    # Generate the manifest
-    my $hManifest = {};
-    $self->manifestRecurse($strPath, undef, 0, $hManifest, $bIgnoreMissing, $strFilter);
-
-    # Return from function and log return values if any
-    return logDebugReturn
-    (
-        $strOperation,
-        {name => 'hManifest', value => $hManifest, trace => true}
-    );
-}
-
-sub manifestRecurse
-{
-    my $self = shift;
-
-    # Assign function parameters, defaults, and log debug info
-    my
-    (
-        $strOperation,
-        $strPath,
-        $strSubPath,
-        $iDepth,
-        $hManifest,
-        $bIgnoreMissing,
-        $strFilter,
-    ) =
-        logDebugParam
-        (
-            __PACKAGE__ . '::manifestRecurse', \@_,
-            {name => 'strPath', trace => true},
-            {name => 'strSubPath', required => false, trace => true},
-            {name => 'iDepth', default => 0, trace => true},
-            {name => 'hManifest', required => false, trace => true},
-            {name => 'bIgnoreMissing', required => false, default => false, trace => true},
-            {name => 'strFilter', required => false, trace => true},
-        );
-
-    # Set operation and debug strings
-    my $strPathRead = $strPath . (defined($strSubPath) ? "/${strSubPath}" : '');
-    my $hPath;
-
-    # If this is the top level stat the path to discover if it is actually a file
-    my $oPathInfo = $self->info($strPathRead, {bIgnoreMissing => $bIgnoreMissing});
-
-    if (defined($oPathInfo))
-    {
-        # If the initial path passed is a file then generate the manifest for just that file
-        if ($iDepth == 0 && !S_ISDIR($oPathInfo->mode()))
-        {
-            $hManifest->{basename($strPathRead)} = $self->manifestStat($strPathRead);
-        }
-        # Else read as a normal directory
-        else
-        {
-            # Get a list of all files in the path (including .)
-            my @stryFileList = @{$self->list($strPathRead, {bIgnoreMissing => $iDepth != 0})};
-            unshift(@stryFileList, '.');
-            my $hFileStat = $self->manifestList($strPathRead, \@stryFileList, $strFilter);
-
-            # Loop through all subpaths/files in the path
-            foreach my $strFile (keys(%{$hFileStat}))
-            {
-                my $strManifestFile = $iDepth == 0 ? $strFile : ($strSubPath . ($strFile eq qw(.) ? '' : "/${strFile}"));
-                $hManifest->{$strManifestFile} = $hFileStat->{$strFile};
-
-                # Recurse into directories
-                if ($hManifest->{$strManifestFile}{type} eq 'd' && $strFile ne qw(.))
-                {
-                    $self->manifestRecurse($strPath, $strManifestFile, $iDepth + 1, $hManifest);
-                }
-            }
-        }
-    }
-
-    # Return from function and log return values if any
-    return logDebugReturn($strOperation);
-}
-
-sub manifestList
-{
-    my $self = shift;
-
-    # Assign function parameters, defaults, and log debug info
-    my
-    (
-        $strOperation,
-        $strPath,
-        $stryFile,
-        $strFilter,
-    ) =
-        logDebugParam
-        (
-            __PACKAGE__ . '->manifestList', \@_,
-            {name => 'strPath', trace => true},
-            {name => 'stryFile', trace => true},
-            {name => 'strFilter', required => false, trace => true},
-        );
-
-    my $hFileStat = {};
-
-    foreach my $strFile (@{$stryFile})
-    {
-        if ($strFile ne '.' && defined($strFilter) && $strFilter ne $strFile)
-        {
-            next;
-        }
-
-        $hFileStat->{$strFile} = $self->manifestStat("${strPath}" . ($strFile eq qw(.) ? '' : "/${strFile}"));
-
-        if (!defined($hFileStat->{$strFile}))
-        {
-            delete($hFileStat->{$strFile});
-        }
-    }
-
-    # Return from function and log return values if any
-    return logDebugReturn
-    (
-        $strOperation,
-        {name => 'hFileStat', value => $hFileStat, trace => true}
-    );
-}
-
-sub manifestStat
-{
-    my $self = shift;
-
-    # Assign function parameters, defaults, and log debug info
-    my
-    (
-        $strOperation,
-        $strFile,
-    ) =
-        logDebugParam
-        (
-            __PACKAGE__ . '->manifestStat', \@_,
-            {name => 'strFile', trace => true},
-        );
-
-    # Stat the path/file, ignoring any that are missing
-    my $oStat = $self->info($strFile, {bIgnoreMissing => true});
-
-    # Generate file data if stat succeeded (i.e. file exists)
-    my $hFile;
-
-    if (defined($oStat))
-    {
-        # Check for regular file
-        if (S_ISREG($oStat->mode))
-        {
-            $hFile->{type} = 'f';
-
-            # Get size
-            $hFile->{size} = $oStat->size;
-
-            # Get modification time
-            $hFile->{modification_time} = $oStat->mtime;
-        }
-        # Check for directory
-        elsif (S_ISDIR($oStat->mode))
-        {
-            $hFile->{type} = 'd';
-        }
-        # Check for link
-        elsif (S_ISLNK($oStat->mode))
-        {
-            $hFile->{type} = 'l';
-            $hFile->{link_destination} = $self->linkDestination($strFile);
-        }
-        # Not a recognized type
-        else
-        {
-            confess &log(ERROR, "${strFile} is not of type directory, file, or link", ERROR_FILE_INVALID);
-        }
-
-        # Get user name
-        $hFile->{user} = getpwuid($oStat->uid);
-
-        # Get group name
-        $hFile->{group} = getgrgid($oStat->gid);
-
-        # Get mode
-        if ($hFile->{type} ne 'l')
-        {
-            $hFile->{mode} = sprintf('%04o', S_IMODE($oStat->mode));
-        }
-    }
-
-    # Return from function and log return values if any
-    return logDebugReturn
-    (
-        $strOperation,
-        {name => 'hFile', value => $hFile, trace => true}
-    );
-}
-
-####################################################################################################################################
-# move - move path/file
-####################################################################################################################################
-sub move
-{
-    my $self = shift;
-
-    # Assign function parameters, defaults, and log debug info
-    my
-    (
-        $strOperation,
-        $strSourceFile,
-        $strDestinationFile,
-        $bPathCreate,
-    ) =
-        logDebugParam
-        (
-            __PACKAGE__ . '->move', \@_,
-            {name => 'strSourceFile', trace => true},
-            {name => 'strDestinationFile', trace => true},
-            {name => 'bPathCreate', default => false, trace => true},
-        );
-
-    # Get source and destination paths
-    my $strSourcePathFile = dirname($strSourceFile);
-    my $strDestinationPathFile = dirname($strDestinationFile);
-
-    # Move the file
-    if (!rename($strSourceFile, $strDestinationFile))
-    {
-        my $strMessage = "unable to move '${strSourceFile}'";
-
-        # If something is missing determine if it is the source or destination
-        if ($OS_ERROR{ENOENT})
-        {
-            if (!$self->exists($strSourceFile))
-            {
-                logErrorResult(ERROR_FILE_MISSING, "${strMessage} because it is missing");
-            }
-
-            if ($bPathCreate)
-            {
-                # Attempt to create the path - ignore exists here in case another process creates it first
-                $self->pathCreate($strDestinationPathFile, {bCreateParent => true, bIgnoreExists => true});
-
-                # Try move again
-                $self->move($strSourceFile, $strDestinationFile);
-            }
-            else
-            {
-                logErrorResult(ERROR_PATH_MISSING, "${strMessage} to missing path '${strDestinationPathFile}'");
-            }
-        }
-        # Else raise the error
-        else
-        {
-            logErrorResult(ERROR_FILE_MOVE, "${strMessage} to '${strDestinationFile}'", $OS_ERROR);
-        }
-    }
-
-    # Return from function and log return values if any
-    return logDebugReturn($strOperation);
-}
-
-####################################################################################################################################
-# openRead - open file for reading
-####################################################################################################################################
-sub openRead
-{
-    my $self = shift;
-
-    # Assign function parameters, defaults, and log debug info
-    my
-    (
-        $strOperation,
-        $strFile,
-        $bIgnoreMissing,
-    ) =
-        logDebugParam
-    (
-        __PACKAGE__ . '->openRead', \@_,
-        {name => 'strFile', trace => true},
-        {name => 'bIgnoreMissing', optional => true, default => false, trace => true},
-    );
-
-    my $oFileIO = new pgBackRestTest::Common::StoragePosixRead($self, $strFile, {bIgnoreMissing => $bIgnoreMissing});
-
-    # Return from function and log return values if any
-    return logDebugReturn
-    (
-        $strOperation,
-        {name => 'oFileIO', value => $oFileIO, trace => true},
-    );
-}
-
-####################################################################################################################################
-# openWrite - open file for writing
-####################################################################################################################################
-sub openWrite
-{
-    my $self = shift;
-
-    # Assign function parameters, defaults, and log debug info
-    my
-    (
-        $strOperation,
-        $strFile,
-        $strMode,
-        $strUser,
-        $strGroup,
-        $lTimestamp,
-        $bPathCreate,
-        $bAtomic,
-    ) =
-        logDebugParam
-    (
-        __PACKAGE__ . '->openWrite', \@_,
-        {name => 'strFile', trace => true},
-        {name => 'strMode', optional => true, trace => true},
-        {name => 'strUser', optional => true, trace => true},
-        {name => 'strGroup', optional => true, trace => true},
-        {name => 'lTimestamp', optional => true, trace => true},
-        {name => 'bPathCreate', optional => true, trace => true},
-        {name => 'bAtomic', optional => true, trace => true},
-    );
-
-    my $oFileIO = new pgBackRestTest::Common::StoragePosixWrite(
-        $self, $strFile,
-        {strMode => $strMode, strUser => $strUser, strGroup => $strGroup, lTimestamp => $lTimestamp, bPathCreate => $bPathCreate,
-            bAtomic => $bAtomic, bSync => $self->{bFileSync}});
-
-    # Return from function and log return values if any
-    return logDebugReturn
-    (
-        $strOperation,
-        {name => 'oFileIO', value => $oFileIO, trace => true},
-    );
-}
-
-####################################################################################################################################
-# owner - change ownership of path/file
-####################################################################################################################################
-sub owner
-{
-    my $self = shift;
-
-    # Assign function parameters, defaults, and log debug info
-    my
-    (
-        $strOperation,
-        $strFilePath,
-        $strUser,
-        $strGroup,
-    ) =
-        logDebugParam
-        (
-            __PACKAGE__ . '->owner', \@_,
-            {name => 'strFilePath', trace => true},
-            {name => 'strUser', optional => true, trace => true},
-            {name => 'strGroup', optional => true, trace => true},
-        );
-
-    # Only proceed if user or group was specified
-    if (defined($strUser) || defined($strGroup))
-    {
-        my $strMessage = "unable to set ownership for '${strFilePath}'";
-        my $iUserId;
-        my $iGroupId;
-
-        # If the user or group is not defined then get it by stat'ing the file. This is because the chown function requires that
-        # both user and group be set.
-        my $oStat = $self->info($strFilePath);
-
-        if (!defined($strUser))
-        {
-            $iUserId = $oStat->uid;
-        }
-
-        if (!defined($strGroup))
-        {
-            $iGroupId = $oStat->gid;
-        }
-
-        # Lookup user if specified
-        if (defined($strUser))
-        {
-            $iUserId = getpwnam($strUser);
-
-            if (!defined($iUserId))
-            {
-                logErrorResult(ERROR_FILE_OWNER, "${strMessage} because user '${strUser}' does not exist");
-            }
-        }
-
-        # Lookup group if specified
-        if (defined($strGroup))
-        {
-            $iGroupId = getgrnam($strGroup);
-
-            if (!defined($iGroupId))
-            {
-                logErrorResult(ERROR_FILE_OWNER, "${strMessage} because group '${strGroup}' does not exist");
-            }
-        }
-
-        # Set ownership on the file if the user or group would be changed
-        if ($iUserId != $oStat->uid || $iGroupId != $oStat->gid)
-        {
-            if (!chown($iUserId, $iGroupId, $strFilePath))
-            {
-                logErrorResult(ERROR_FILE_OWNER, "${strMessage}", $OS_ERROR);
-            }
-        }
-    }
-
-    # Return from function and log return values if any
-    return logDebugReturn($strOperation);
-}
-
-####################################################################################################################################
-# pathCreate - create path
-####################################################################################################################################
-sub pathCreate
-{
-    my $self = shift;
-
-    # Assign function parameters, defaults, and log debug info
-    my
-    (
-        $strOperation,
-        $strPath,
-        $strMode,
-        $bIgnoreExists,
-        $bCreateParent,
-    ) =
-        logDebugParam
-        (
-            __PACKAGE__ . '->pathCreate', \@_,
-            {name => 'strPath', trace => true},
-            {name => 'strMode', optional => true, default => '0750', trace => true},
-            {name => 'bIgnoreExists', optional => true, default => false, trace => true},
-            {name => 'bCreateParent', optional => true, default => false, trace => true},
-        );
-
-    # Attempt to create the directory
-    if (!mkdir($strPath, oct($strMode)))
-    {
-        my $strMessage = "unable to create path '${strPath}'";
-
-        # If parent path is missing
-        if ($OS_ERROR{ENOENT})
-        {
-            if (!$bCreateParent)
-            {
-                confess &log(ERROR, "${strMessage} because parent does not exist", ERROR_PATH_MISSING);
-            }
-
-            # Create parent path
-            $self->pathCreate(dirname($strPath), {strMode => $strMode, bIgnoreExists => true, bCreateParent => $bCreateParent});
-
-            # Create path
-            $self->pathCreate($strPath, {strMode => $strMode, bIgnoreExists => true});
-        }
-        # Else if path already exists
-        elsif ($OS_ERROR{EEXIST})
-        {
-            if (!$bIgnoreExists)
-            {
-                confess &log(ERROR, "${strMessage} because it already exists", ERROR_PATH_EXISTS);
-            }
-        }
-        else
-        {
-            logErrorResult(ERROR_PATH_CREATE, ${strMessage}, $OS_ERROR);
-        }
-    }
-
-    # Return from function and log return values if any
-    return logDebugReturn($strOperation);
-}
-
-####################################################################################################################################
-# pathExists - check if path exists
-####################################################################################################################################
-sub pathExists
-{
-    my $self = shift;
-
-    # Assign function parameters, defaults, and log debug info
-    my
-    (
-        $strOperation,
-        $strPath,
-    ) =
-        logDebugParam
-        (
-            __PACKAGE__ . '->pathExists', \@_,
-            {name => 'strPath', trace => true},
-        );
-
-    # Does the path/file exist?
-    my $bExists = true;
-    my $oStat = lstat($strPath);
-
-    # Use stat to test if path exists
-    if (defined($oStat))
-    {
-        # Check that it is actually a path
-        $bExists = S_ISDIR($oStat->mode) ? true : false;
-    }
-    else
-    {
-        # If the error is not entry missing, then throw error
-        if (!$OS_ERROR{ENOENT})
-        {
-            logErrorResult(ERROR_FILE_EXISTS, "unable to test if path '${strPath}' exists", $OS_ERROR);
-        }
-
-        $bExists = false;
-    }
-
-    # Return from function and log return values if any
-    return logDebugReturn
-    (
-        $strOperation,
-        {name => 'bExists', value => $bExists, trace => true}
-    );
-}
-
-####################################################################################################################################
-# pathSync - perform fsync on path
-####################################################################################################################################
-sub pathSync
-{
-    my $self = shift;
-
-    # Assign function parameters, defaults, and log debug info
-    my
-    (
-        $strOperation,
-        $strPath,
-    ) =
-        logDebugParam
-        (
-            __PACKAGE__ . '->pathSync', \@_,
-            {name => 'strPath', trace => true},
-        );
-
-    open(my $hPath, "<", $strPath)
-        or confess &log(ERROR, "unable to open '${strPath}' for sync", ERROR_PATH_OPEN);
-    open(my $hPathDup, ">&", $hPath)
-        or confess &log(ERROR, "unable to duplicate '${strPath}' handle for sync", ERROR_PATH_OPEN);
-
-    $hPathDup->sync()
-        or confess &log(ERROR, "unable to sync path '${strPath}'", ERROR_PATH_SYNC);
-
-    close($hPathDup);
-    close($hPath);
-
-    # Return from function and log return values if any
-    return logDebugReturn($strOperation);
-}
-
-####################################################################################################################################
-# remove - remove path/file
-####################################################################################################################################
-sub remove
-{
-    my $self = shift;
-
-    # Assign function parameters, defaults, and log debug info
-    my
-    (
-        $strOperation,
-        $xstryPathFile,
-        $bIgnoreMissing,
-        $bRecurse,
-    ) =
-        logDebugParam
-        (
-            __PACKAGE__ . '->remove', \@_,
-            {name => 'xstryPathFile', trace => true},
-            {name => 'bIgnoreMissing', optional => true, default => false, trace => true},
-            {name => 'bRecurse', optional => true, default => false, trace => true},
-        );
-
-    # Working variables
-    my $bRemoved = true;
-
-    # Remove a tree
-    if ($bRecurse)
-    {
-        my $oManifest = $self->manifest($xstryPathFile, {bIgnoreMissing => true});
-
-        # Iterate all files in the manifest
-        foreach my $strFile (sort({$b cmp $a} keys(%{$oManifest})))
-        {
-            # remove directory
-            if ($oManifest->{$strFile}{type} eq 'd')
-            {
-                my $xstryPathFileRemove = $strFile eq '.' ? $xstryPathFile : "${xstryPathFile}/${strFile}";
-
-                if (!rmdir($xstryPathFileRemove))
-                {
-                    # Throw error if this is not an ignored missing path
-                    if (!($OS_ERROR{ENOENT} && $bIgnoreMissing))
-                    {
-                        logErrorResult(ERROR_PATH_REMOVE, "unable to remove path '${strFile}'", $OS_ERROR);
-                    }
-                }
-            }
-            # Remove file
-            else
-            {
-                $self->remove("${xstryPathFile}/${strFile}", {bIgnoreMissing => true});
-            }
-        }
-    }
-    # Only remove the specified file
-    else
-    {
-        foreach my $strFile (ref($xstryPathFile) ? @{$xstryPathFile} : ($xstryPathFile))
-        {
-            if (unlink($strFile) != 1)
-            {
-                $bRemoved = false;
-
-                # Throw error if this is not an ignored missing file
-                if (!($OS_ERROR{ENOENT} && $bIgnoreMissing))
-                {
-                    logErrorResult(
-                        $OS_ERROR{ENOENT} ? ERROR_FILE_MISSING : ERROR_FILE_OPEN, "unable to remove file '${strFile}'", $OS_ERROR);
-                }
-            }
-        }
-    }
-
-    # Return from function and log return values if any
-    return logDebugReturn
-    (
-        $strOperation,
-        {name => 'bRemoved', value => $bRemoved, trace => true}
-    );
-}
-
-####################################################################################################################################
-# Getters/Setters
-####################################################################################################################################
-sub className {STORAGE_POSIX_DRIVER}
-sub tempExtension {shift->{strTempExtension}}
-sub tempExtensionSet {my $self = shift; $self->{strTempExtension} = shift}
-sub type {STORAGE_POSIX}
-
-1;

test/lib/pgBackRestTest/Common/VmTest.pm

@@ -1,459 +0,0 @@
-####################################################################################################################################
-# VmTest.pm - Vm constants and data
-####################################################################################################################################
-package pgBackRestTest::Common::VmTest;
-
-####################################################################################################################################
-# Perl includes
-####################################################################################################################################
-use strict;
-use warnings FATAL => qw(all);
-use Carp qw(confess);
-
-use Exporter qw(import);
-    our @EXPORT = qw();
-
-use pgBackRestDoc::Common::Exception;
-use pgBackRestDoc::Common::Log;
-use pgBackRestDoc::Common::String;
-
-use pgBackRestTest::Common::DbVersion;
-
-####################################################################################################################################
-# VM hash keywords
-####################################################################################################################################
-use constant VM_DB                                                  => 'db';
-    push @EXPORT, qw(VM_DB);
-use constant VM_DB_TEST                                             => 'db-test';
-    push @EXPORT, qw(VM_DB_TEST);
-# Will coverage testing be run for C?
-use constant VMDEF_COVERAGE_C                                       => 'coverage-c';
-use constant VM_DEPRECATED                                          => 'deprecated';
-    push @EXPORT, qw(VM_DEPRECATED);
-use constant VM_IMAGE                                               => 'image';
-    push @EXPORT, qw(VM_IMAGE);
-use constant VM_OS_BASE                                             => 'os-base';
-    push @EXPORT, qw(VM_OS_BASE);
-use constant VMDEF_PG_REPO                                          => 'pg-repo';
-use constant VMDEF_PGSQL_BIN                                        => 'psql-bin';
-    push @EXPORT, qw(VMDEF_PGSQL_BIN);
-
-####################################################################################################################################
-# Valid OS base List
-####################################################################################################################################
-use constant VM_OS_BASE_ALPINE                                      => 'alpine';
-    push @EXPORT, qw(VM_OS_BASE_ALPINE);
-use constant VM_OS_BASE_DEBIAN                                      => 'debian';
-    push @EXPORT, qw(VM_OS_BASE_DEBIAN);
-use constant VM_OS_BASE_RHEL                                        => 'rhel';
-    push @EXPORT, qw(VM_OS_BASE_RHEL);
-
-####################################################################################################################################
-# Valid architecture list
-####################################################################################################################################
-use constant VM_ARCH_AARCH64                                        => 'aarch64';
-    push @EXPORT, qw(VM_ARCH_AARCH64);
-use constant VM_ARCH_I386                                           => 'i386';
-    push @EXPORT, qw(VM_ARCH_I386);
-use constant VM_ARCH_PPC64LE                                        => 'ppc64le';
-    push @EXPORT, qw(VM_ARCH_PPC64LE);
-use constant VM_ARCH_S390X                                          => 's390x';
-    push @EXPORT, qw(VM_ARCH_S390X);
-use constant VM_ARCH_X86_64                                         => 'x86_64';
-    push @EXPORT, qw(VM_ARCH_X86_64);
-
-# List of architectures that a container can be built for
-use constant VM_ARCH_LIST                                           =>
-    (VM_ARCH_AARCH64, VM_ARCH_I386, VM_ARCH_PPC64LE, VM_ARCH_S390X, VM_ARCH_X86_64);
-    push @EXPORT, qw(VM_ARCH_LIST);
-
-####################################################################################################################################
-# Valid VM list
-####################################################################################################################################
-use constant VM_ALL                                                 => 'all';
-    push @EXPORT, qw(VM_ALL);
-
-use constant VM_NONE                                                => 'none';
-    push @EXPORT, qw(VM_NONE);
-
-use constant VM_A321                                                 => 'a321';
-    push @EXPORT, qw(VM_A321);
-use constant VM_A324                                                 => 'a324';
-    push @EXPORT, qw(VM_A324);
-use constant VM_D12                                                 => 'd12';
-    push @EXPORT, qw(VM_D12);
-use constant VM_RH8                                                 => 'rh8';
-    push @EXPORT, qw(VM_RH8);
-use constant VM_RH9                                                 => 'rh9';
-    push @EXPORT, qw(VM_RH9);
-use constant VM_RH10                                                => 'rh10';
-    push @EXPORT, qw(VM_RH10);
-use constant VM_F44                                                 => 'f44';
-    push @EXPORT, qw(VM_F44);
-use constant VM_U22                                                 => 'u22';
-    push @EXPORT, qw(VM_U22);
-use constant VM_U24                                                 => 'u24';
-    push @EXPORT, qw(VM_U24);
-
-# List of default test VMs
-use constant VM_LIST                                                =>
-    (VM_D12, VM_RH8, VM_RH9, VM_RH10, VM_U22, VM_U24, VM_A321, VM_A324);
-    push @EXPORT, qw(VM_LIST);
-
-my $oyVm =
-{
-    # None
-    &VM_NONE =>
-    {
-        &VM_OS_BASE => VM_OS_BASE_DEBIAN,
-        &VMDEF_COVERAGE_C => true,
-        &VMDEF_PGSQL_BIN => '/usr/lib/postgresql/{[version]}/bin',
-
-        &VM_DB =>
-        [
-            PG_VERSION_10,
-        ],
-
-        &VM_DB_TEST =>
-        [
-            PG_VERSION_10,
-        ],
-    },
-
-    # Alpine 3.21
-    &VM_A321 =>
-    {
-        &VM_OS_BASE => VM_OS_BASE_ALPINE,
-        &VM_IMAGE => 'alpine:3.21',
-        &VMDEF_PG_REPO => false,
-        &VMDEF_PGSQL_BIN => '/usr/libexec/postgresql{[version]}',
-
-        &VM_DB =>
-        [
-            PG_VERSION_15,
-            PG_VERSION_16,
-            PG_VERSION_17,
-        ],
-
-        &VM_DB_TEST =>
-        [
-            PG_VERSION_16,
-        ],
-    },
-
-    # Alpine 3.24
-    &VM_A324 =>
-    {
-        &VM_OS_BASE => VM_OS_BASE_ALPINE,
-        &VM_IMAGE => 'alpine:3.24',
-        &VMDEF_PG_REPO => false,
-        &VMDEF_PGSQL_BIN => '/usr/libexec/postgresql{[version]}',
-
-        &VM_DB =>
-        [
-            PG_VERSION_16,
-            PG_VERSION_17,
-            PG_VERSION_18,
-        ],
-
-        &VM_DB_TEST =>
-        [
-            PG_VERSION_17,
-        ],
-    },
-
-    # Debian 12
-    &VM_D12 =>
-    {
-        &VM_OS_BASE => VM_OS_BASE_DEBIAN,
-        &VM_IMAGE => 'debian:12',
-        &VMDEF_PG_REPO => false,
-        &VMDEF_PGSQL_BIN => '/usr/lib/postgresql/{[version]}/bin',
-
-        &VM_DB =>
-        [
-            PG_VERSION_15,
-        ],
-
-        &VM_DB_TEST =>
-        [
-            PG_VERSION_15,
-        ],
-    },
-
-    # RHEL 8
-    &VM_RH8 =>
-    {
-        &VM_OS_BASE => VM_OS_BASE_RHEL,
-        &VM_IMAGE => 'rockylinux/rockylinux:8',
-        &VMDEF_PG_REPO => false,
-        &VMDEF_PGSQL_BIN => '/usr/bin',
-
-        &VM_DB =>
-        [
-            PG_VERSION_10,
-        ],
-
-        &VM_DB_TEST =>
-        [
-            PG_VERSION_10,
-        ],
-    },
-
-    # RHEL 9
-    &VM_RH9 =>
-    {
-        &VM_OS_BASE => VM_OS_BASE_RHEL,
-        &VM_IMAGE => 'rockylinux/rockylinux:9',
-        &VMDEF_PGSQL_BIN => '/usr/pgsql-{[version]}/bin',
-
-        &VM_DB =>
-        [
-            PG_VERSION_14,
-            PG_VERSION_15,
-            PG_VERSION_16,
-            PG_VERSION_17,
-            PG_VERSION_18,
-        ],
-
-        &VM_DB_TEST =>
-        [
-            PG_VERSION_14,
-        ],
-    },
-
-    # RHEL 10
-    &VM_RH10 =>
-    {
-        &VM_OS_BASE => VM_OS_BASE_RHEL,
-        &VM_IMAGE => 'rockylinux/rockylinux:10',
-        &VMDEF_PGSQL_BIN => '/usr/pgsql-{[version]}/bin',
-
-        &VM_DB =>
-        [
-            PG_VERSION_14,
-            PG_VERSION_15,
-            PG_VERSION_16,
-            PG_VERSION_17,
-            PG_VERSION_18,
-        ],
-
-        &VM_DB_TEST =>
-        [
-            PG_VERSION_18,
-        ],
-    },
-
-    # Fedora 44
-    &VM_F44 =>
-    {
-        &VM_OS_BASE => VM_OS_BASE_RHEL,
-        &VM_IMAGE => 'fedora:44',
-        &VMDEF_PGSQL_BIN => '/usr/pgsql-{[version]}/bin',
-        &VMDEF_COVERAGE_C => true,
-
-        &VM_DB =>
-        [
-            PG_VERSION_14,
-            PG_VERSION_15,
-            PG_VERSION_16,
-            PG_VERSION_17,
-            PG_VERSION_18,
-        ],
-
-        &VM_DB_TEST =>
-        [
-            PG_VERSION_15,
-        ],
-    },
-
-    # Ubuntu 22.04
-    &VM_U22 =>
-    {
-        &VM_OS_BASE => VM_OS_BASE_DEBIAN,
-        &VM_IMAGE => 'ubuntu:22.04',
-        &VMDEF_COVERAGE_C => true,
-        &VMDEF_PGSQL_BIN => '/usr/lib/postgresql/{[version]}/bin',
-
-        &VM_DB =>
-        [
-            PG_VERSION_96,
-            PG_VERSION_10,
-            PG_VERSION_11,
-            PG_VERSION_12,
-            PG_VERSION_13,
-            PG_VERSION_14,
-        ],
-
-        &VM_DB_TEST =>
-        [
-            PG_VERSION_96,
-            PG_VERSION_11,
-        ],
-    },
-
-    # Ubuntu 24.04
-    &VM_U24 =>
-    {
-        &VM_OS_BASE => VM_OS_BASE_DEBIAN,
-        &VM_IMAGE => 'ubuntu:24.04',
-        &VMDEF_COVERAGE_C => true,
-        &VMDEF_PGSQL_BIN => '/usr/lib/postgresql/{[version]}/bin',
-
-        &VM_DB =>
-        [
-            PG_VERSION_96,
-            PG_VERSION_10,
-            PG_VERSION_11,
-            PG_VERSION_12,
-            PG_VERSION_13,
-            PG_VERSION_14,
-            PG_VERSION_15,
-            PG_VERSION_16,
-            PG_VERSION_17,
-            PG_VERSION_18,
-            PG_VERSION_19,
-        ],
-
-        &VM_DB_TEST =>
-        [
-            PG_VERSION_12,
-            PG_VERSION_13,
-            PG_VERSION_19,
-        ],
-    },
-};
-
-####################################################################################################################################
-# Set VM_DB_TEST to VM_DB if it is not defined so it doesn't have to be checked everywhere
-####################################################################################################################################
-foreach my $strVm (sort(keys(%{$oyVm})))
-{
-    if (!defined($oyVm->{$strVm}{&VM_DB_TEST}))
-    {
-        $oyVm->{$strVm}{&VM_DB_TEST} = $oyVm->{$strVm}{&VM_DB};
-    }
-}
-
-####################################################################################################################################
-# Verify that each version of PostgreSQL is represented in one and only one default VM
-####################################################################################################################################
-foreach my $strPgVersion (versionSupport())
-{
-    my $strVmPgVersionRun;
-    my $bVmCoverageC = false;
-
-    foreach my $strVm (VM_LIST)
-    {
-        if (vmCoverageC($strVm))
-        {
-            $bVmCoverageC = true;
-        }
-
-        foreach my $strVmPgVersion (@{$oyVm->{$strVm}{&VM_DB_TEST}})
-        {
-            if ($strPgVersion eq $strVmPgVersion)
-            {
-                if (defined($strVmPgVersionRun))
-                {
-                    confess &log(ASSERT, "PostgreSQL $strPgVersion is already configured to run on default vm $strVm");
-                }
-
-                $strVmPgVersionRun = $strVm;
-            }
-        }
-    }
-
-    my $strErrorSuffix = 'is not configured to run on a default vm';
-
-    if (!$bVmCoverageC)
-    {
-        confess &log(ASSERT, "C coverage ${strErrorSuffix}");
-    }
-
-    if (!defined($strVmPgVersionRun))
-    {
-        confess &log(ASSERT, "PostgreSQL ${strPgVersion} ${strErrorSuffix}");
-    }
-}
-
-####################################################################################################################################
-# vmValid
-####################################################################################################################################
-sub vmValid
-{
-    my $strVm = shift;
-
-    if (!defined($oyVm->{$strVm}))
-    {
-        confess &log(ERROR, "no definition for vm '${strVm}'");
-    }
-}
-
-push @EXPORT, qw(vmValid);
-
-####################################################################################################################################
-# vmPgRepo
-####################################################################################################################################
-sub vmPgRepo
-{
-    my $strVm = shift;
-
-    vmValid($strVm);
-
-    if (!defined($oyVm->{$strVm}{&VMDEF_PG_REPO}))
-    {
-        return true;
-    }
-
-    return $oyVm->{$strVm}{&VMDEF_PG_REPO};
-}
-
-push @EXPORT, qw(vmPgRepo);
-
-####################################################################################################################################
-# vmGet
-####################################################################################################################################
-sub vmGet
-{
-    return $oyVm;
-}
-
-push @EXPORT, qw(vmGet);
-
-####################################################################################################################################
-# vmCoverageC
-####################################################################################################################################
-sub vmCoverageC
-{
-    my $strVm = shift;
-
-    return $oyVm->{$strVm}{&VMDEF_COVERAGE_C} ? true : false;
-}
-
-push @EXPORT, qw(vmCoverageC);
-
-####################################################################################################################################
-# Get host architecture
-####################################################################################################################################
-my $strHostArch = undef;
-
-sub hostArch
-{
-    if (!defined($strHostArch))
-    {
-        $strHostArch = trim(`uname -m`);
-
-        # Mac M1 reports arm64 but we generally need aarch64 (which Linux reports)
-        if ($strHostArch eq 'arm64')
-        {
-            $strHostArch = VM_ARCH_AARCH64;
-        }
-    }
-
-    return $strHostArch;
-}
-
-push @EXPORT, qw(hostArch);
-
-1;

test/src/module/test/command/code/format_test.py

@@ -0,0 +1,203 @@
+"""Test Code Format Command.
+
+The formatters are captured rather than run, since what this module does is decide which files to format with which formatter. The
+repository is written here so a file can be left out of the working tree while git still reports it."""
+
+####################################################################################################################################
+import io
+import os
+import tempfile
+from contextlib import redirect_stdout
+from unittest.mock import patch
+
+from harness.test import *
+
+from command.code.format import *
+from common.error import *
+from common.log import *
+from common.storage import file_write, path_create
+
+# What git reports for the repository, i.e. everything that is version controlled. src/gone.c is in the index but has been removed
+# from the working tree and local is a path rather than a file, so neither is written as a file below.
+FILE_LIST = (
+    "README.md",
+    "doc/doc.pl",
+    "local",
+    "src/build/help.auto.c.inc",
+    "src/gone.c",
+    "src/main.c",
+    "src/main.h",
+    "src/module.vendor.c.inc",
+    "src/vendor.vendor.h",
+    "test/lib/common/log.py",
+    "test/src/test.c",
+    "test/test.py",
+)
+
+
+####################################################################################################################################
+class Config:
+    """What the command reads from the command line."""
+
+    def __init__(self, repo_path, check=False):
+        self.repo_path = repo_path
+        self.check = check
+
+
+####################################################################################################################################
+def _repo_create(path):
+    """Write the repository, i.e. everything git reports that is a file that is still there."""
+
+    for file in FILE_LIST:
+        if file != "src/gone.c":
+            file_write(os.path.join(path, file), "")
+
+    # A path where git reported one, which is what it does for a symlink to a directory
+    os.remove(os.path.join(path, "local"))
+    path_create(os.path.join(path, "local"))
+
+    return path
+
+
+####################################################################################################################################
+def _cmd_code_format(config, fail=None):
+    """Run the command with the formatters captured rather than run.
+
+    Returns the commands and the error raised, if any. Naming a formatter in fail makes it report that it found something."""
+
+    command_list = []
+    error = None
+
+    def exec_fake(command, result_expect=0, show_output=False):
+        command_list.append(command)
+
+        if command.startswith("git "):
+            return "\n".join(FILE_LIST) + "\n"
+
+        if fail is not None and command.startswith(fail):
+            raise TestError("%s terminated unexpectedly [1]" % fail)
+
+        return ""
+
+    output = io.StringIO()
+    log_init(INFO, False)
+
+    try:
+        with patch("command.code.format.exec_one", exec_fake):
+            with redirect_stdout(output):
+                try:
+                    cmd_code_format(config)
+                except TestError as exception:
+                    error = str(exception)
+    finally:
+        log_init(INFO, True)
+
+    return command_list, error, output.getvalue()
+
+
+####################################################################################################################################
+def test_code_format():
+    """Every version controlled file goes to the formatter for its language and nothing else does."""
+
+    with tempfile.TemporaryDirectory() as path:
+        path_repo = _repo_create(path)
+
+        command_list, error, output = _cmd_code_format(Config(path_repo))
+
+        assert_is_none(error)
+        assert_equal(output, "P00   INFO: code format\n")
+
+        # Only the version controlled files are formatted, which is what git reports
+        assert_equal(command_list[0], "git -C %s ls-files -c --others --exclude-standard" % path_repo)
+
+        # The C, which is formatted in place
+        assert_equal(
+            command_list[1],
+            "uncrustify -c %s/test/uncrustify.cfg --replace --no-backup %s/src/main.c %s/src/main.h" % ((path_repo,) * 3),
+        )
+
+        # The python, which is formatted to the same line length as the rest of the project
+        assert_equal(
+            command_list[2],
+            "black --line-length=132 %s/test/lib/common/log.py %s/test/test.py" % ((path_repo,) * 2),
+        )
+
+
+####################################################################################################################################
+def test_code_format_skip():
+    """A file that is not formatted is left out of both formatters."""
+
+    with tempfile.TemporaryDirectory() as path:
+        path_repo = _repo_create(path)
+
+        command_list, error, output = _cmd_code_format(Config(path_repo))
+        command = " ".join(command_list)
+
+        # The test template does not format correctly since it is not valid C until the harness fills it in
+        assert_not_in("test/src/test.c", command)
+
+        # Nothing vendorized or generated, since neither is ours to format
+        assert_not_in("help.auto.c.inc", command)
+        assert_not_in("vendor.vendor.h", command)
+        assert_not_in("module.vendor.c.inc", command)
+
+        # Nothing in a language neither formatter handles
+        assert_not_in("README.md", command)
+        assert_not_in("doc.pl", command)
+
+        # A file that git reports but is no longer in the working tree, since there is nothing there to format
+        assert_not_in("gone.c", command)
+
+        # An entry that is a path rather than a file, which is also not something to check the permissions of
+        assert_not_in("local", command)
+
+
+####################################################################################################################################
+def test_code_format_check():
+    """The check reports what is not formatted rather than formatting it, and says how to fix what it found."""
+
+    with tempfile.TemporaryDirectory() as path:
+        path_repo = _repo_create(path)
+
+        command_list, error, output = _cmd_code_format(Config(path_repo, check=True))
+
+        assert_is_none(error)
+        assert_equal(output, "P00   INFO: code format check\n")
+        assert_in("uncrustify -c %s/test/uncrustify.cfg --check " % path_repo, command_list[1])
+
+        # Black shows what it would change so the failure says what is wrong rather than only that something is
+        assert_in("black --line-length=132 --check --diff ", command_list[2])
+
+        # A check that finds something says how to fix it, since CI is where it usually runs
+        for formatter in ("uncrustify", "black"):
+            command_list, error, output = _cmd_code_format(Config(path_repo, check=True), fail=formatter)
+
+            assert_equal(error, "%s terminated unexpectedly [1]\nHINT: run 'test.py code-format' to format the code" % formatter)
+
+        # Formatting rather than checking reports what the formatter said and nothing else, since there is nothing to fix by hand
+        command_list, error, output = _cmd_code_format(Config(path_repo), fail="uncrustify")
+
+        assert_equal(error, "uncrustify terminated unexpectedly [1]")
+
+
+####################################################################################################################################
+def test_code_format_permission():
+    """Nothing is executable that is not meant to be, which formatting can munge."""
+
+    with tempfile.TemporaryDirectory() as path:
+        path_repo = _repo_create(path)
+
+        # The files that are meant to be executable are
+        for file in ("doc/doc.pl", "test/test.py"):
+            os.chmod(os.path.join(path_repo, file), 0o755)
+
+        command_list, error, output = _cmd_code_format(Config(path_repo))
+
+        assert_is_none(error)
+
+        # Nothing else is
+        os.chmod(os.path.join(path_repo, "src/main.c"), 0o750)
+
+        command_list, error, output = _cmd_code_format(Config(path_repo))
+
+        assert_equal(error, "expected mode '0640' for 'src/main.c' but found '0750'")

test/src/module/test/command/coverage/filter_py_test.py

@@ -5,7 +5,7 @@ from harness.test import *
 
 from command.coverage.coverage import Coverage, CoverageFile, CoverageLine
 from command.coverage.filter_py import *
-from command.test.define import TEST_LANG_C, TEST_LANG_PYTHON
+from command.test.define import TEST_LANG_PYTHON
 
 # Coverage as coverage.py reports it, i.e. the lines that ran, the lines that did not, and the arcs between them
 RAW = {

test/src/module/test/command/test/container_test.py

@@ -0,0 +1,450 @@
+"""Test Container Build.
+
+The script for one vm is compared in full, since its hash is the cache key and any change to it rebuilds every image built from it.
+The user and group are whoever runs the tests so the expected script names them with tokens, and the certificate and key data is
+collapsed because it is copied in a line at a time and listing every line would only make the expected script harder to read.
+
+Docker is not run here. What matters is the script that goes into an image and the tag it is looked up under, so the commands are
+captured and checked instead."""
+
+####################################################################################################################################
+import os
+import re
+import tempfile
+from unittest.mock import patch
+
+from harness.test import *
+
+from command.test.container import *
+from common.error import *
+from common.log import *
+from common.storage import file_read, file_write
+from common.user import group_id, group_name, user_id, user_name
+from common.vm import *
+
+# Revisions as test/container.yaml declares them
+REVISION = {"all": "1", VM_U24: "2", "%s-%s" % (VM_U24, VM_ARCH_X86_64): "3"}
+
+# The base script for a vm that installs from the PostgreSQL repo and collects coverage, i.e. one that exercises every option the
+# debian build has
+SCRIPT_BASE_U24 = """ && \\
+
+# Install packages
+    export DEBCONF_NONINTERACTIVE_SEEN=true DEBIAN_FRONTEND=noninteractive && \\
+    apt-get update && \\
+    apt-get install -y --no-install-recommends openssh-server sudo gcc make git \\
+        ca-certificates libssl-dev tzdata zlib1g-dev libxml2-dev pkg-config \\
+        libbz2-dev bzip2 libyaml-dev liblz4-dev liblz4-tool gnupg lsb-release ccache meson \\
+        libssh2-1-dev libcurl4-openssl-dev libsystemd-dev python3-yaml valgrind python3-coverage zstd libzstd-dev && \\
+
+# Regenerate SSH keys
+    rm -f /etc/ssh/ssh_host_rsa_key* && \\
+    ssh-keygen -t rsa -b 2048 -f /etc/ssh/ssh_host_rsa_key && \\
+
+# Fix root tty
+    sed -i 's/^mesg n/tty -s \\&\\& mesg n/g' /root/.profile && \\
+
+# Suppress dpkg interactive output
+    rm /etc/apt/apt.conf.d/70debconf && \\
+
+# Install PostgreSQL packages
+    apt-get install -y --no-install-recommends postgresql-common && \\
+    /usr/share/postgresql-common/pgdg/apt.postgresql.org.sh -y -c 19 && \\
+    apt-get install -y --no-install-recommends postgresql-common libpq-dev && \\
+    sed -i 's/^\\#create\\_main\\_cluster.*$/create\\_main\\_cluster \\= false/' /etc/postgresql-common/createcluster.conf && \\
+
+# Install PostgreSQL
+    apt-get install -y --no-install-recommends postgresql-9.6 postgresql-10 postgresql-11 postgresql-12 postgresql-13 \
+postgresql-14 postgresql-15 postgresql-16 postgresql-17 postgresql-18 postgresql-19 && \\
+
+# Cleanup
+    apt-get autoremove -y && \\
+    apt-get clean && \\
+    rm -rf /var/lib/apt/lists/*"""
+
+# The test script for the same vm
+SCRIPT_TEST_U24 = """ && \\
+
+# Install CA
+    echo '-----BEGIN CERTIFICATE-----' > /usr/local/share/ca-certificates/pgbackrest-test-ca.crt && \\
+[DATA]
+    echo '-----END CERTIFICATE-----' >> /usr/local/share/ca-certificates/pgbackrest-test-ca.crt && \\
+    chmod 644 /usr/local/share/ca-certificates/pgbackrest-test-ca.crt && \\
+    update-ca-certificates && \\
+
+# Create banner to make sure pgBackRest ignores it
+    echo '***********************************************' >  /etc/issue.net && \\
+    echo 'Sample banner to make sure banners are skipped.' >> /etc/issue.net && \\
+    echo ''                                                >> /etc/issue.net && \\
+    echo 'More banner after a blank line.'                 >> /etc/issue.net && \\
+    echo '***********************************************' >> /etc/issue.net && \\
+    echo 'Banner /etc/issue.net'                           >> /etc/ssh/sshd_config && \\
+
+    echo '# Add PubkeyAcceptedAlgorithms (required for SFTP)'              >> /etc/ssh/sshd_config && \\
+    echo 'HostKeyAlgorithms=+ssh-rsa,ssh-rsa-cert-v01@openssh.com'        >> /etc/ssh/sshd_config && \\
+    echo 'PubkeyAcceptedAlgorithms=+ssh-rsa,ssh-rsa-cert-v01@openssh.com' >> /etc/ssh/sshd_config && \\
+
+# Rename conflicting group
+    sed -i 's/^[^:]*:x:[GROUP_ID]:.*/[GROUP]:x:[GROUP_ID]:/' /etc/group && \\
+
+# Create test user
+    groupadd -f -g[GROUP_ID] [GROUP] && \\
+    adduser --uid=[USER_ID] --ingroup=[GROUP] --disabled-password --gecos "" [USER] && \\
+    mkdir -m 750 /home/[USER]/test && \\
+    chown [USER]:[GROUP] /home/[USER]/test && \\
+
+# Configure sudo
+    echo '%[GROUP] ALL=(ALL) NOPASSWD: ALL' >> /etc/sudoers && \\
+
+# Setup SSH
+    mkdir /home/[USER]/.ssh && \\
+    echo '-----BEGIN RSA PRIVATE KEY-----' > /home/[USER]/.ssh/id_rsa && \\
+[DATA]
+    echo '-----END RSA PRIVATE KEY-----' >> /home/[USER]/.ssh/id_rsa && \\
+    echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlXJKkRqflGftBDlc/XdiRfFOD5en0355zIOEP3Y5CTRVk4WcQqVdRoxcfRZWNFGCOBsXvwk\
+cVsRCNQeKZvOv16/Ui4NohLjYTWdSQFoeCCcc0BbZqq1zNQNKcsZxep3Rd4O3CmOPUjTFY54mWL/GNT55LCPOo2BfplfY/S5Nd+W7bni5KM4XcgBGj0\
+WIMQpB2SyjE04CEJNiAMDKeaRx8n17zsWOJMaZ+dk7cfqBTaSJ1IVevB0e103dCW/pGAKugsYBlmma2oGitPBKFZFJxYACD86pbAumF81tcggdk7gQn\
+hLYkuCyAkdvbRInZcQjYiWf2yP6gJaAsnab5Eu9b user@pgbackrest-test' > /home/[USER]/.ssh/authorized_keys && \\
+    echo 'Host *' > /home/[USER]/.ssh/config && \\
+    echo '    StrictHostKeyChecking no' >> /home/[USER]/.ssh/config && \\
+    cp /home/[USER]/.ssh/authorized_keys /home/[USER]/.ssh/id_rsa.pub && \\
+    chown -R [USER]:[GROUP] /home/[USER]/.ssh && \\
+    chmod 700 /home/[USER]/.ssh && \\
+    chmod 600 /home/[USER]/.ssh/* && \\
+
+# Make [USER] home dir readable
+    chmod g+r,g+x /home/[USER]
+
+# Start SSH when container starts
+ENTRYPOINT service ssh restart && bash"""
+
+
+####################################################################################################################################
+def _expect(script):
+    """Fill in the user and group the tests run as."""
+
+    return (
+        script.replace("[USER_ID]", str(user_id()))
+        .replace("[GROUP_ID]", str(group_id()))
+        .replace("[USER]", user_name())
+        .replace("[GROUP]", group_name())
+    )
+
+
+####################################################################################################################################
+def _collapse(script):
+    """Collapse the certificate and key data, which is copied in a line at a time and is not generated."""
+
+    return re.sub(r"(    echo '[A-Za-z0-9+/=]{40,}' >> [^\n]+\n)+", "[DATA]\n", script)
+
+
+####################################################################################################################################
+def _repo_write(path):
+    """Write the part of the repository the container build reads."""
+
+    file_write(os.path.join(path, "test/container.yaml"), "all: '1'\n%s: '2'\n%s-%s: '3'\n" % (VM_U24, VM_U24, VM_ARCH_X86_64))
+    file_write(
+        os.path.join(path, "test/certificate/pgbackrest-test-ca.crt"),
+        "-----BEGIN CERTIFICATE-----\nQUJDREVGR0hJSktMTU5PUFFSU1RVVldYWVphYmNkZWZnaGlqa2xtbm9w\n-----END CERTIFICATE-----\n",
+    )
+
+
+####################################################################################################################################
+class _Config:
+    """The options the container build reads."""
+
+    def __init__(self, path_repo, vm, vm_arch=VM_ARCH_X86_64, vm_force=False, log_level=INFO):
+        self.repo_path = path_repo
+        self.vm = vm
+        self.vm_arch = vm_arch
+        self.vm_force = vm_force
+        self.log_level = log_level
+
+
+####################################################################################################################################
+class _Docker:
+    """Capture the commands the build would run and report whatever status the test asked for."""
+
+    def __init__(self, status=0, status_push=None):
+        self.command_list = []
+        self.status = status  # Status reported for a command that is allowed to fail, e.g. a pull from a cache
+        self.status_push = status if status_push is None else status_push
+
+    ################################################################################################################################
+    def one(self, command, result_expect=0, show_output=False):
+        self.command_list.append(command)
+
+        if command.startswith("docker images"):
+            return "%s:%s-base-%s\nubuntu:24.04\n" % (container_repo(), VM_U24, VM_ARCH_X86_64)
+
+        if command.startswith("docker ps"):
+            return "test-0\ntest-build\nunrelated\n"
+
+        return ""
+
+    ################################################################################################################################
+    def status_get(self, command):
+        self.command_list.append(command)
+
+        return self.status_push if command.startswith("docker push") else self.status
+
+    ################################################################################################################################
+    def match(self, expression):
+        """The commands that match an expression."""
+
+        return [command for command in self.command_list if re.search(expression, command)]
+
+
+####################################################################################################################################
+def test_container_repo():
+    """Images are named in the pgbackrest namespace unless an owner asks for its own."""
+
+    assert_equal(container_repo(), "ghcr.io/pgbackrest/test")
+    assert_equal(container_repo("someone"), "ghcr.io/someone/test")
+
+
+####################################################################################################################################
+def test_container_script():
+    """The generated script is what goes into the image."""
+
+    with tempfile.TemporaryDirectory() as path:
+        _repo_write(path)
+
+        assert_equal(script_base(VM_U24, VM_ARCH_X86_64), SCRIPT_BASE_U24)
+        assert_equal(_collapse(script_test(VM_U24, path)), _expect(SCRIPT_TEST_U24))
+
+        # The script is wrapped in a Dockerfile that runs it as a single layer
+        assert_equal(
+            container_script("Base", "ubuntu:24.04", " && \\\n\n# Install packages\n    apt-get update"),
+            "# Base Container\nFROM ubuntu:24.04\n\nRUN echo 'OPTIMIZED BUILD' && \\\n\n# Install packages\n    apt-get update",
+        )
+
+        # A script with nothing in it is just the image it came from
+        assert_equal(container_script("Base", "ubuntu:24.04", ""), "# Base Container\nFROM ubuntu:24.04")
+
+
+####################################################################################################################################
+def test_container_script_vm():
+    """Each vm installs what its os base and PostgreSQL repository need."""
+
+    # RHEL 8 has no valgrind or coverage package and needs a python that PyYAML is packaged for
+    script = script_base(VM_RH8, VM_ARCH_X86_64)
+
+    assert_true("--set-enabled powertools" in script)
+    assert_true("valgrind" not in script)
+    assert_true("python3.12-pyyaml" in script)
+
+    # RHEL 8 installs PostgreSQL from the application stream rather than from the PostgreSQL repo
+    assert_true("dnf -y module enable postgresql:10" in script)
+    assert_true("yum -y install postgresql-server" in script)
+    assert_true("ENV PATH=" not in script)
+
+    # RHEL 9 packages a coverage too old to report the branch detail the report needs, so it comes from pip
+    script = script_base(VM_RH9, VM_ARCH_X86_64)
+
+    assert_true("python3-pip" in script)
+    assert_true("pip3 install --no-cache-dir 'coverage>=6.5'" in script)
+    assert_true("RPM-GPG-KEY-PGDG" in script)
+    assert_true("reporpms/EL-9-%s/" % VM_ARCH_X86_64 in script)
+    assert_true("ENV PATH=/usr/pgsql-18/bin:$PATH" in script)
+
+    # RHEL 10 cannot check the PostgreSQL repo signature so it skips the key and disables the check
+    script = script_base(VM_RH10, VM_ARCH_X86_64)
+
+    assert_true("python3-coverage" in script)
+    assert_true("RPM-GPG-KEY-PGDG" not in script)
+    assert_true("s/gpgcheck=1/gpgcheck=0/g" in script)
+
+    # Fedora installs from the Fedora build of the PostgreSQL repo
+    assert_true("reporpms/F-44-%s/" % VM_ARCH_X86_64 in script_base(VM_F44, VM_ARCH_X86_64))
+
+    # Debian 12 installs PostgreSQL from the distribution and runs neither valgrind nor coverage
+    script = script_base(VM_D12, VM_ARCH_X86_64)
+
+    assert_true("valgrind" not in script)
+    assert_true("python3-coverage" not in script)
+    assert_true("apt.postgresql.org.sh" not in script)
+
+    # Ubuntu 22.04 is the last release the python tests do not run on
+    assert_true("python3-coverage" not in script_base(VM_U22, VM_ARCH_X86_64))
+
+    # Alpine installs everything from one package manager and needs no PostgreSQL repo
+    script = script_base(VM_A321, VM_ARCH_X86_64)
+
+    assert_true("apk add --no-cache postgresql15 postgresql16 postgresql17" in script)
+    assert_true("Install PostgreSQL packages" not in script)
+
+    # An architecture the PostgreSQL repo does not build for installs one version for the smoke test
+    script = script_base(VM_U24, VM_ARCH_PPC64LE)
+
+    assert_true("apt.postgresql.org.sh -y &&" in script)
+    assert_true("Install a single PostgreSQL version for the smoke test" in script)
+
+    # There is nothing to install for alpine on that architecture, since it has no smoke test
+    assert_true("Install PostgreSQL" not in script_base(VM_A324, VM_ARCH_PPC64LE))
+
+
+####################################################################################################################################
+def test_container_script_test_vm():
+    """The test image adds the test user and what it needs on each os base."""
+
+    with tempfile.TemporaryDirectory() as path:
+        _repo_write(path)
+
+        # RHEL puts the certificate authority elsewhere, needs sudo configured differently, and starts ssh differently. The
+        # architecture is the host architecture rather than the one being built for, so it is set here to get the same script
+        # whichever machine the tests are run on.
+        with patch("command.test.container.host_arch", return_value=VM_ARCH_X86_64):
+            script = script_test(VM_RH9, path)
+
+        assert_true("/etc/pki/ca-trust/source/anchors/pgbackrest-test-ca.crt" in script)
+        assert_true("update-ca-trust extract" in script)
+        assert_true("adduser -g%s -u%u -N %s" % (group_name(), user_id(), user_name()) in script)
+        assert_true('echo "Set disable_coredump false" >> /etc/sudo.conf' in script)
+        assert_true("ENTRYPOINT rm -rf /run/nologin && /usr/sbin/sshd -D" in script)
+
+        # The language setup only gets in the way on aarch64
+        assert_true("rm /etc/profile.d/lang.sh" not in script)
+
+        with patch("command.test.container.host_arch", return_value=VM_ARCH_AARCH64):
+            assert_true("rm /etc/profile.d/lang.sh" in script_test(VM_RH9, path))
+
+        # Alpine creates the group differently, needs a password for the test user, and starts ssh without a service
+        script = script_test(VM_A321, path)
+
+        assert_true("getent group %s >/dev/null || addgroup" % group_name() in script)
+        assert_true("passwd -d '%s' %s" % (user_name(), user_name()) in script)
+        assert_true("ENTRYPOINT /usr/sbin/sshd -D" in script)
+
+        # Only the releases with an ssh that rejects the test key by default need the algorithms added back
+        assert_true("PubkeyAcceptedAlgorithms" not in script_test(VM_D12, path))
+
+
+####################################################################################################################################
+def test_container_revision():
+    """A revision must be readable and name a vm that exists, else a rebuild would silently never happen."""
+
+    assert_is_none(revision_check(REVISION))
+
+    with assert_raises(TestError) as error:
+        revision_check(["1"])
+
+    assert_equal(str(error.exception), "the 'all' revision is required in test/container.yaml")
+
+    with assert_raises(TestError) as error:
+        revision_check({VM_U24: "1"})
+
+    assert_equal(str(error.exception), "the 'all' revision is required in test/container.yaml")
+
+    with assert_raises(TestError) as error:
+        revision_check({VM_ALL: "1", VM_U24: None})
+
+    assert_equal(str(error.exception), "revision '%s' in test/container.yaml must be set to a value" % VM_U24)
+
+    with assert_raises(TestError) as error:
+        revision_check({VM_ALL: "1", "bogus": "1"})
+
+    assert_equal(str(error.exception), "revision 'bogus' in test/container.yaml has invalid vm 'bogus'")
+
+    # There is no container for none so a revision for it could never be used
+    with assert_raises(TestError) as error:
+        revision_check({VM_ALL: "1", "none": "1"})
+
+    assert_equal(str(error.exception), "revision 'none' in test/container.yaml has invalid vm 'none'")
+
+    with assert_raises(TestError) as error:
+        revision_check({VM_ALL: "1", "%s-bogus" % VM_U24: "1"})
+
+    assert_equal(str(error.exception), "revision '%s-bogus' in test/container.yaml has invalid architecture 'bogus'" % VM_U24)
+
+
+####################################################################################################################################
+def test_container_build():
+    """A base image is pulled from the cache when it is there and built and pushed when it is not."""
+
+    with tempfile.TemporaryDirectory() as path:
+        _repo_write(path)
+
+        image_base = "%s:%s-base-%s" % (container_repo(), VM_U24, VM_ARCH_X86_64)
+        image_test = "%s:%s-test-%s" % (container_repo(), VM_U24, VM_ARCH_X86_64)
+
+        # The base image is in the cache so only the test image is built. The revision for the vm and architecture is the most
+        # specific one so it is the one in the cache tag.
+        docker = _Docker(status=0)
+
+        with patch("command.test.container.exec_one", docker.one):
+            with patch("command.test.container.exec_status", docker.status_get):
+                container_build(_Config(path, VM_U24))
+
+        assert_equal(len(docker.match("^docker pull .*-base-.*-3-")), 1)
+        assert_equal(docker.match("^docker tag"), ["docker tag %s-3-%s %s" % (image_base, _hash(docker), image_base)])
+        assert_equal(len(docker.match("^docker build")), 1)
+        assert_true(docker.match("^docker build")[0].endswith("-t %s %s" % (image_test, os.path.join(path, "test"))))
+
+        # The Dockerfile is written where the build can find it and is the script with the trailing separator removed
+        assert_true(file_read(os.path.join(path, "test/result/docker/%s-test-%s" % (VM_U24, VM_ARCH_X86_64))).endswith("bash\n"))
+
+        # Nothing is in the cache so the base image is built and pushed. A fork pushes to its own cache since it cannot write to
+        # the pgbackrest one.
+        docker = _Docker(status=1, status_push=0)
+
+        with patch("command.test.container.exec_one", docker.one):
+            with patch("command.test.container.exec_status", docker.status_get):
+                with patch.dict(os.environ, {"GITHUB_REPOSITORY_OWNER": "Someone"}):
+                    container_build(_Config(path, VM_U24, log_level=DETAIL))
+
+        assert_equal(len(docker.match("^docker pull %s:" % container_repo())), 1)
+        assert_equal(len(docker.match("^docker pull %s:" % container_repo("someone"))), 1)
+        assert_equal(len(docker.match("^docker build")), 2)
+        assert_equal(len(docker.match("^docker push %s:" % container_repo("someone"))), 1)
+
+        # A push that fails is not reported, since write access is not available everywhere
+        docker = _Docker(status=1)
+
+        with patch("command.test.container.exec_one", docker.one):
+            with patch("command.test.container.exec_status", docker.status_get):
+                container_build(_Config(path, VM_D12))
+
+        assert_equal(len(docker.match("^docker push %s:" % container_repo())), 1)
+
+        # Force removes what was built before and builds without the cache, both the images and the files they were built from
+        docker = _Docker(status=1)
+
+        with patch("command.test.container.exec_one", docker.one):
+            with patch("command.test.container.exec_status", docker.status_get):
+                container_build(_Config(path, VM_U24, vm_force=True))
+
+        assert_equal(docker.match("^rm -f"), ["rm -f %s/test/result/docker/%s-*" % (path, VM_U24)])
+        assert_equal(docker.match("^docker rmi"), ["docker rmi -f %s" % image_base])
+        assert_equal(len(docker.match("^docker pull")), 0)
+        assert_equal(len(docker.match("--no-cache")), 2)
+
+        # Every vm is built when all of them are asked for, and force then removes every file and image
+        docker = _Docker(status=1)
+
+        with patch("command.test.container.exec_one", docker.one):
+            with patch("command.test.container.exec_status", docker.status_get):
+                container_build(_Config(path, VM_ALL, vm_force=True))
+
+        assert_equal(docker.match("^rm -f"), ["rm -f %s/test/result/docker/*" % path])
+        assert_equal(len(docker.match("^docker build")), len(VM_LIST) * 2)
+
+
+####################################################################################################################################
+def _hash(docker):
+    """The hash the build put in the cache tag, which is a hash of the script so it cannot be written out here."""
+
+    return re.search(r"-base-[^ ]*-3-([0-9a-f]{12})", docker.match("^docker pull")[0]).group(1)
+
+
+####################################################################################################################################
+def test_container_remove():
+    """Containers are removed by name and an image that does not match is left alone."""
+
+    docker = _Docker()
+
+    with patch("command.test.container.exec_one", docker.one):
+        with patch("command.test.container.exec_status", docker.status_get):
+            container_remove("^test-([0-9]+|build)$")
+
+    assert_equal(docker.match("^docker rm"), ["docker rm -f test-0", "docker rm -f test-build"])

test/src/module/test/command/test/job_test.py

@@ -0,0 +1,490 @@
+"""Test Test Job.
+
+The job decides what to run and reports what happened rather than running anything itself, so what it would have run is captured
+instead, the same way the container build test captures docker. The test path and the user vary by machine so both are replaced
+with tokens before comparing, as the build test does."""
+
+####################################################################################################################################
+import io
+import os
+import re
+import tempfile
+from contextlib import redirect_stdout
+from unittest.mock import patch
+
+from harness.test import *
+
+from command.test.define import TEST_LANG_C, TEST_LANG_PYTHON, TEST_TYPE_INTEGRATION, TEST_TYPE_PERFORMANCE, TEST_TYPE_UNIT
+from command.test.define import TestDefModule
+from command.test.job import *
+from command.test.list import TestRun
+from common.log import *
+from common.storage import file_write
+from common.user import user_name
+from common.vm import *
+
+# Image a test that runs in a container is started from
+IMAGE = "ghcr.io/pgbackrest/test:u24-test-x86_64"
+
+# What a C unit test runs, i.e. the harness that builds it and then the binary itself under valgrind. The binary is run rather than
+# left to the harness so the run can be wrapped in valgrind and its output copied to both streams.
+COMMAND_C = """python3 [PATH]/repo/test/test.py unit --repo-path=[PATH]/repo --test-path=[PATH] --log-level=info \
+--log-level-test=off --vm=none --vm-id=0 --no-log-timestamp --scale=1 common/error && \\
+exec 3>&1 && \\
+{ valgrind -q --gen-suppressions=all --exit-on-first-error=yes --leak-check=full --leak-resolution=high --error-exitcode=25 \
+[PATH]/unit-0/none/build/test-unit 2>&1 1>&3; echo $? > [PATH]/unit-0/none/result; } | tee /dev/stderr && \\
+exit $(cat [PATH]/unit-0/none/result)"""
+
+# What a python test runs. There is no binary so the harness runs the test itself, and it writes its own coverage since there is no
+# gcov step for it afterward.
+COMMAND_PYTHON = """python3 [PATH]/repo/test/test.py unit --repo-path=[PATH]/repo --test-path=[PATH] --log-level=info \
+--log-level-test=off --vm=none --vm-id=0 --no-log-timestamp --scale=1 \
+--coverage-file=[PATH]/repo/test/result/coverage/raw/test-common-log.json test/common/log"""
+
+
+####################################################################################################################################
+class Config:
+    """What the job reads from the command line."""
+
+    def __init__(self, test_path, **option):
+        self.repo_path = os.path.join(test_path, "repo")
+        self.test_path = test_path
+        self.vm_arch = None
+        self.log_level = INFO
+        self.log_level_test = OFF
+        self.log_timestamp = False
+        self.dry_run = False
+        self.vm_out = False
+        self.cleanup = True
+        self.retry = 0
+        self.scale = 1
+        self.tz = None
+        self.profile = False
+        self.valgrind = True
+        self.coverage = True
+        self.back_trace = True
+
+        for name, value in option.items():
+            setattr(self, name, value)
+
+
+####################################################################################################################################
+class _Exec:
+    """Stand in for the process that runs the test and report what the test asked for.
+
+    The result is set on the class since the job creates the process itself, so there is nothing to pass it through."""
+
+    status = 0
+    output = ""
+    error = ""
+    poll = 0  # How many times the process reports that it is still running before it finishes
+
+    def __init__(self, command, show_output=False):
+        self.command = command
+        self.show_output = show_output
+
+    ################################################################################################################################
+    def begin(self):
+        pass
+
+    ################################################################################################################################
+    def end(self, wait=True):
+        # Report that the process is still running as many times as the test asked for
+        if self.poll > 0:
+            self.poll -= 1
+
+            return None
+
+        return self.status
+
+
+####################################################################################################################################
+def _run(name="common/error", vm=VM_NONE, type=TEST_TYPE_UNIT, lang=TEST_LANG_C, pg_version=None, run_list=None, total=0):
+    """Build a test to run."""
+
+    module = TestDefModule(name, type)
+    module.lang = lang
+    module.total = total
+
+    return TestRun(module, vm, pg_version, run_list)
+
+
+####################################################################################################################################
+def _job(config, run=None, vm_idx=0, vm_max=1, test_idx=0, test_max=1, show_output=False):
+    """Build a job to run a test."""
+
+    return TestJob(config, _run() if run is None else run, vm_idx, vm_max, test_idx, test_max, IMAGE, show_output)
+
+
+####################################################################################################################################
+def _capture(job, action, status=0, output="", error="", poll=0):
+    """Run something on a job with the commands it would run captured rather than run.
+
+    Returns what the action returned, the commands, and what was written to the log."""
+
+    command_list = []
+    log_output = io.StringIO()
+
+    def exec_fake(command, result_expect=0, show_output=False):
+        command_list.append(command)
+
+        return ""
+
+    _Exec.status, _Exec.output, _Exec.error, _Exec.poll = status, output, error, poll
+
+    # Timestamps are suppressed so the log can be compared, and the level is raised so the detail a test writes is included
+    log_init(DETAIL, False)
+
+    try:
+        with patch("command.test.job.exec_one", exec_fake), patch("command.test.job.Exec", _Exec):
+            with patch("command.test.job.container_remove", lambda expression: command_list.append("remove " + expression)):
+                with redirect_stdout(log_output):
+                    result = action(job)
+    finally:
+        log_init(INFO, True)
+
+    return result, command_list, log_output.getvalue()
+
+
+####################################################################################################################################
+def _expect(text, config):
+    """Fill in the test path and the user the tests run as."""
+
+    return text.replace("[PATH]", config.test_path).replace("[USER]", user_name())
+
+
+####################################################################################################################################
+def test_job_command():
+    """A C test builds itself through the harness and then runs the binary it built."""
+
+    with tempfile.TemporaryDirectory() as path:
+        config = Config(path)
+        job = _job(config)
+
+        started, command_list, output = _capture(job, lambda job: job.begin())
+
+        assert_true(started)
+        assert_equal(job.exec.command, _expect(COMMAND_C, config))
+
+        # Nothing is run to start the test since there is no container for it
+        assert_equal(command_list, [])
+
+        # The test is listed at detail, since at info the log would be a list of tests that have not finished yet
+        assert_equal(output, "P00 DETAIL: P1-T1/1 - vm=none, module=common, test=error\n")
+
+        # The paths the test writes to are created, and the data path only for a unit test
+        assert_true(os.path.isdir(os.path.join(path, "test-0")))
+        assert_true(os.path.isdir(os.path.join(path, "unit-0/none")))
+        assert_true(os.path.isdir(os.path.join(path, "data-0")))
+
+        # A python test has nothing to build so the harness runs it, and it writes its own coverage
+        job = _job(config, _run(name="test/common/log", lang=TEST_LANG_PYTHON))
+
+        _capture(job, lambda job: job.begin())
+
+        assert_equal(job.exec.command, _expect(COMMAND_PYTHON, config))
+
+
+####################################################################################################################################
+def test_job_command_option():
+    """The options a test was given are passed on to the harness that runs it."""
+
+    with tempfile.TemporaryDirectory() as path:
+        config = Config(path, vm_arch="ppc64le", tz="America/New_York", scale=4, back_trace=False, log_timestamp=True)
+        job = _job(config, _run(pg_version="15", run_list=[2, 1]))
+
+        _capture(job, lambda job: job.begin())
+
+        assert_in(" --vm-arch=ppc64le --vm-id=0 --test=2 --test=1 --tz='America/New_York' --scale=4", job.exec.command)
+        assert_in(" --pg-version=15 --no-back-trace ", job.exec.command)
+
+        # Timestamps are only suppressed when they were suppressed here, since the point is a reproducible log
+        assert_not_in("--no-log-timestamp", job.exec.command)
+
+        # A run is listed with the sub-tests and the PostgreSQL version it was selected for
+        assert_equal(job.description, "P1-T1/1 - vm=none, module=common, test=error, run=1,2, pg-version=15")
+
+        # Coverage is not collected for a profile run since the instrumentation would skew the timing
+        job = _job(Config(path, profile=True), _run())
+
+        _capture(job, lambda job: job.begin())
+
+        assert_in(" --profile ", job.exec.command)
+        assert_in(" --no-coverage ", job.exec.command)
+
+        # Nor for a performance test, for the same reason
+        job = _job(Config(path), _run(name="performance/type", type=TEST_TYPE_PERFORMANCE))
+
+        _capture(job, lambda job: job.begin())
+
+        assert_in(" --no-coverage ", job.exec.command)
+
+        # Nor on a vm that cannot collect it
+        job = _job(Config(path), _run(vm=VM_D12))
+
+        _capture(job, lambda job: job.begin())
+
+        assert_in(" --no-coverage ", job.exec.command)
+
+
+####################################################################################################################################
+def test_job_valgrind():
+    """A C test runs under valgrind, with the suppressions for its vm when there are any."""
+
+    with tempfile.TemporaryDirectory() as path:
+        config = Config(path)
+
+        # A performance test is timed so valgrind would make the result meaningless
+        job = _job(config, _run(name="performance/type", type=TEST_TYPE_PERFORMANCE))
+
+        _capture(job, lambda job: job.begin())
+
+        assert_not_in("valgrind", job.exec.command)
+
+        # Valgrind can be turned off to save time
+        job = _job(Config(path, valgrind=False))
+
+        _capture(job, lambda job: job.begin())
+
+        assert_not_in("valgrind", job.exec.command)
+
+        # The suppressions for the vm are used when the repository has them
+        path_suppress = os.path.join(path, "repo/test/src/valgrind.suppress.none")
+        file_write(path_suppress, "")
+
+        job = _job(config)
+
+        _capture(job, lambda job: job.begin())
+
+        assert_in("--gen-suppressions=all --suppressions=%s --exit-on-first-error" % path_suppress, job.exec.command)
+
+
+####################################################################################################################################
+def test_job_container():
+    """A test on a vm runs in a container of its own, which is started here and removed when the test is done."""
+
+    with tempfile.TemporaryDirectory() as path:
+        config = Config(path, vm_arch="x86_64")
+        job = _job(config, _run(vm=VM_U24), vm_idx=1, vm_max=2, test_idx=3, test_max=12)
+
+        started, command_list, output = _capture(job, lambda job: job.begin())
+
+        assert_true(started)
+        assert_equal(
+            command_list,
+            [
+                _expect(
+                    "docker run --platform linux/x86_64 -itd -h u24-test --name=test-1 -v [PATH]/test-1:[PATH]/test-1"
+                    " -v [PATH]/unit-1/u24:[PATH]/unit-1/u24 -v [PATH]/data-1:[PATH]/data-1"
+                    " -v [PATH]/build/u24:[PATH]/build/u24:ro -v [PATH]/ccache:[PATH]/ccache -e CCACHE_DIR=[PATH]/ccache"
+                    " -v [PATH]/repo:[PATH]/repo -v [PATH]/repo:[PATH]/repo " + IMAGE,
+                    config,
+                )
+            ],
+        )
+
+        # The test itself runs in the container it was started in
+        assert_true(job.exec.command.startswith(_expect("docker exec -i -u [USER] test-1 bash -l -c '\\\n", config)))
+
+        # The vm and the test are numbered from one and padded out so the list lines up
+        assert_equal(job.description, "P2-T04/12 - vm=u24, module=common, test=error")
+
+        # An integration test starts its own containers and has nothing built for it, so neither is done here
+        job = _job(config, _run(name="integration/all", type=TEST_TYPE_INTEGRATION, vm=VM_U24, run_list=[1]))
+
+        started, command_list, output = _capture(job, lambda job: job.begin())
+
+        assert_equal(command_list, [])
+        assert_false(os.path.exists(os.path.join(path, "unit-0/u24")))
+
+        # It runs the binary that was built for none rather than one built for the vm, and outside a container
+        assert_true(job.exec.command.startswith("python3 "))
+        assert_in(" --vm=u24 ", job.exec.command)
+        assert_in(_expect("[PATH]/unit-0/none/build/test-unit", config), job.exec.command)
+
+
+####################################################################################################################################
+def test_job_dry_run():
+    """A dry run lists the test rather than running it, unless the output was asked for."""
+
+    with tempfile.TemporaryDirectory() as path:
+        job = _job(Config(path, dry_run=True))
+
+        started, command_list, output = _capture(job, lambda job: job.begin())
+
+        assert_false(started)
+        assert_equal(output, "P00   INFO: P1-T1/1 - vm=none, module=common, test=error\n")
+
+        # Nothing is created since nothing will run
+        assert_false(os.path.exists(os.path.join(path, "test-0")))
+
+        # Asking for the output of a dry run runs the test after all, which is a way to see more about what it would do
+        job = _job(Config(path, dry_run=True, vm_out=True))
+
+        started, command_list, output = _capture(job, lambda job: job.begin())
+
+        assert_true(started)
+        assert_equal(output, "P00 DETAIL: P1-T1/1 - vm=none, module=common, test=error\n")
+
+
+####################################################################################################################################
+def test_job_end():
+    """A test that passed is reported with how long it took."""
+
+    with tempfile.TemporaryDirectory() as path:
+        config = Config(path)
+        job = _job(config)
+
+        _capture(job, lambda job: job.begin())
+
+        # The test is not done until the process is, which is how the driver knows it can start another one
+        (done, fail), command_list, output = _capture(job, lambda job: job.end(), poll=1)
+
+        assert_false(done)
+        assert_false(fail)
+        assert_equal(command_list, [])
+
+        (done, fail), command_list, output = _capture(job, lambda job: job.end())
+
+        assert_true(done)
+        assert_false(fail)
+        assert_equal(output, "P00   INFO: P1-T1/1 - vm=none, module=common, test=error\n")
+
+        # The coverage the test left behind is written where the report is built from, and then the test path is removed
+        assert_equal(
+            command_list,
+            [
+                _expect(
+                    "gcov --json-format --stdout --branch-probabilities [PATH]/unit-0/none/build/test-unit@exe/test.c.gcda"
+                    " > [PATH]/repo/test/result/coverage/raw/common-error.json",
+                    config,
+                ),
+                _expect("chmod -R 700 [PATH]/test-0/* 2>&1;rm -rf [PATH]/test-0", config),
+            ],
+        )
+
+        # Meson has named the directory it compiled the test into both ways, so the other name is used when it is there
+        file_write(os.path.join(path, "unit-0/none/build/test-unit.p/test.c.gcda"), "")
+
+        job = _job(config)
+
+        _capture(job, lambda job: job.begin())
+        (done, fail), command_list, output = _capture(job, lambda job: job.end())
+
+        assert_in("[PATH]/unit-0/none/build/test-unit.p/test.c.gcda".replace("[PATH]", path), command_list[0])
+
+
+####################################################################################################################################
+def test_job_end_output():
+    """What the test wrote is reported when it was asked for and when the test failed."""
+
+    with tempfile.TemporaryDirectory() as path:
+        config = Config(path, vm_out=True, coverage=False, cleanup=False, log_timestamp=True)
+        job = _job(config)
+
+        _capture(job, lambda job: job.begin())
+        (done, fail), command_list, output = _capture(job, lambda job: job.end(), output="all good\n")
+
+        assert_true(done)
+        assert_false(fail)
+
+        # Nothing is cleaned up so the result can be looked at, and the elapsed time is reported
+        assert_equal(command_list, [])
+        assert_true(re.match(r"^P00   INFO: P1-T1/1 - vm=none, module=common, test=error \(\d+\.\d\ds\):", output))
+        assert_in("all good", output)
+
+        # Output that arrived as it was written is not written again
+        job = _job(config, show_output=True)
+
+        _capture(job, lambda job: job.begin())
+        (done, fail), command_list, output = _capture(job, lambda job: job.end(), output="all good\n")
+
+        assert_not_in("all good", output)
+
+
+####################################################################################################################################
+def test_job_end_fail():
+    """A test that failed is reported with what it wrote, whatever its exit status was."""
+
+    with tempfile.TemporaryDirectory() as path:
+        config = Config(path)
+        job = _job(config, _run(vm=VM_U24))
+
+        _capture(job, lambda job: job.begin())
+        (done, fail), command_list, output = _capture(job, lambda job: job.end(), status=25, output="it broke\n")
+
+        assert_true(done)
+        assert_true(fail)
+        assert_equal(output.split("\n")[0], "P00  ERROR: P1-T1/1 - vm=u24, module=common, test=error (err25):")
+        assert_in("it broke", output)
+
+        # Nothing is written where coverage would go since the test did not finish, and the container is removed
+        assert_equal(
+            command_list,
+            ["remove ^test-0($|-)", _expect("chmod -R 700 [PATH]/test-0/* 2>&1;rm -rf [PATH]/test-0", config)],
+        )
+
+        # Anything on stderr is a failure whatever the exit status was, e.g. a valgrind error the binary did not report
+        job = _job(config)
+
+        _capture(job, lambda job: job.begin())
+        (done, fail), command_list, output = _capture(job, lambda job: job.end(), error="valgrind said no\n")
+
+        assert_true(fail)
+
+        # A test that wrote nothing at all still reports something
+        assert_in("NO OUTPUT ON STDOUT OR STDERR", output)
+
+
+####################################################################################################################################
+def test_job_profile():
+    """A profile run writes the profile the test generated."""
+
+    with tempfile.TemporaryDirectory() as path:
+        config = Config(path, profile=True)
+        job = _job(config, _run(vm=VM_U24))
+
+        _capture(job, lambda job: job.begin())
+
+        # The object directory meson compiled the test into is found under either name it has had
+        file_write(os.path.join(path, "unit-0/u24/build/test-unit.p/test.c.gcda"), "")
+
+        (done, fail), command_list, output = _capture(job, lambda job: job.end())
+
+        assert_equal(
+            command_list[:2],
+            [
+                _expect(
+                    "docker exec -i -u [USER] test-0 gprof [PATH]/unit-0/u24/build/test-unit"
+                    " [PATH]/unit-0/u24/build/gmon.out > [PATH]/unit-0/u24/gprof.txt",
+                    config,
+                ),
+                _expect("cp [PATH]/unit-0/u24/gprof.txt [PATH]/repo/test/result/profile/gprof.txt", config),
+            ],
+        )
+
+
+####################################################################################################################################
+def test_job_retry():
+    """A test that failed is started again until it runs out of retries."""
+
+    with tempfile.TemporaryDirectory() as path:
+        job = _job(Config(path, retry=1))
+
+        started, command_list, output = _capture(job, lambda job: job.begin())
+
+        assert_true(started)
+        assert_in(" --log-level=info ", job.exec.command)
+
+        # The last try runs at a higher log level so there is something to look at when it fails again
+        started, command_list, output = _capture(job, lambda job: job.begin())
+
+        assert_true(started)
+        assert_in(" --log-level=debug ", job.exec.command)
+        assert_equal(job.description, "P1-T1/1 - vm=none, module=common, test=error (retry 1)")
+
+        # There are no tries left so the driver is told to give up on it
+        started, command_list, output = _capture(job, lambda job: job.begin())
+
+        assert_false(started)
+        assert_equal(output, "")

test/src/module/test/command/test/list_test.py

@@ -0,0 +1,175 @@
+"""Test Test List.
+
+The modules are built here rather than read from define.yaml so the selection can be checked against definitions that do not exist
+yet, e.g. a test that only runs on one vm."""
+
+####################################################################################################################################
+from harness.test import *
+
+from command.test.define import TEST_LANG_PYTHON, TEST_TYPE_INTEGRATION, TEST_TYPE_PERFORMANCE, TEST_TYPE_UNIT, TestDefModule
+from command.test.list import *
+from common.vm import *
+
+
+####################################################################################################################################
+class _Config:
+    """The options the selection reads, with the defaults a run with no options would have."""
+
+    def __init__(self, **option):
+        self.vm = VM_NONE
+        self.module = []
+        self.test = []
+        self.run = []
+        self.pg_version = "minimal"
+        self.c_only = False
+        self.container_only = False
+        self.coverage_only = False
+        self.performance = True
+
+        self.__dict__.update(option)
+
+
+####################################################################################################################################
+def _module(name, type=TEST_TYPE_UNIT, **field):
+    """Build a test module definition."""
+
+    result = TestDefModule(name, type)
+
+    for key, value in field.items():
+        setattr(result, key, value)
+
+    return result
+
+
+####################################################################################################################################
+def _name_list(test_list):
+    """The name of each selected test, which is what most of the selection rules decide."""
+
+    return [run.module.name for run in test_list]
+
+
+# Modules that cover every kind of test the selection has a rule for
+MODULE_LIST = [
+    _module("common/error", coverage_list=["common/error"]),
+    _module("common/exec"),
+    _module("common/socket", container_required=True),
+    _module("doc/build", vm_list=[VM_U24]),
+    _module("test/common/log", lang=TEST_LANG_PYTHON, coverage_list=["test/common/log"]),
+    _module("integration/all", type=TEST_TYPE_INTEGRATION, total=3, pg_required=True),
+    _module("performance/type", type=TEST_TYPE_PERFORMANCE, total=2),
+]
+
+
+####################################################################################################################################
+def test_list_all():
+    """Everything that can run on a vm is selected when nothing is asked for."""
+
+    # Integration tests need a container so none of them run without one, and neither does a test that requires one
+    assert_equal(
+        _name_list(test_list_get(MODULE_LIST, _Config())),
+        ["common/error", "common/exec", "test/common/log", "performance/type"],
+    )
+
+    # A vm with a container runs everything, and the integration test runs once per sub-test and PostgreSQL version
+    test_list = test_list_get(MODULE_LIST, _Config(vm=VM_U24))
+
+    assert_equal(
+        _name_list(test_list),
+        ["common/error", "common/exec", "common/socket", "doc/build", "test/common/log"]
+        + ["integration/all"] * 3 * len(vm_get(VM_U24).db_test_list)
+        + ["performance/type"],
+    )
+
+    # A test that only runs on one vm is left out of the others
+    assert_true("doc/build" not in _name_list(test_list_get(MODULE_LIST, _Config(vm=VM_D12))))
+
+    # Every default vm runs its own copy of the list
+    assert_equal(len(set(run.vm for run in test_list_get(MODULE_LIST, _Config(vm=VM_ALL)))), len(VM_LIST))
+
+
+####################################################################################################################################
+def test_list_select():
+    """A module and a test in it are selected by name, ignoring case."""
+
+    assert_equal(_name_list(test_list_get(MODULE_LIST, _Config(module=["common"]))), ["common/error", "common/exec"])
+    assert_equal(_name_list(test_list_get(MODULE_LIST, _Config(module=["COMMON"], test=["Error"]))), ["common/error"])
+
+    # A python test is named for the group it is in and the path of the module it covers
+    assert_equal(_name_list(test_list_get(MODULE_LIST, _Config(module=["test"], test=["common/log"]))), ["test/common/log"])
+
+    # A name that matches nothing selects nothing rather than being an error, since the test command reports that
+    assert_equal(_name_list(test_list_get(MODULE_LIST, _Config(module=["bogus"]))), [])
+
+
+####################################################################################################################################
+def test_list_filter():
+    """The filters leave out the kinds of test they name."""
+
+    # Only integration tests are not C, so this is the only thing --c-only leaves out
+    assert_true("integration/all" not in _name_list(test_list_get(MODULE_LIST, _Config(vm=VM_U24, c_only=True))))
+
+    # Only tests that must have a container, which includes every integration test
+    assert_equal(
+        _name_list(test_list_get(MODULE_LIST, _Config(vm=VM_U24, container_only=True))),
+        ["common/socket"] + ["integration/all"] * 3 * len(vm_get(VM_U24).db_test_list),
+    )
+
+    # Only tests that provide coverage, which is what the documentation summary is built from
+    assert_equal(_name_list(test_list_get(MODULE_LIST, _Config(coverage_only=True))), ["common/error", "test/common/log"])
+
+    # Performance tests are timed so they are left out of a run that is only checking behavior
+    assert_true("performance/type" not in _name_list(test_list_get(MODULE_LIST, _Config(performance=False))))
+
+
+####################################################################################################################################
+def test_list_pg_version():
+    """An integration test runs against the PostgreSQL versions the vm tests, most recent first."""
+
+    version_list = vm_get(VM_U24).db_test_list
+
+    assert_equal(
+        [run.pg_version for run in test_list_get(MODULE_LIST, _Config(vm=VM_U24, module=["integration"]))],
+        [version for version in reversed(version_list) for _ in range(3)],
+    )
+
+    # A single version can be selected, which is how a version specific problem is chased down
+    assert_equal(
+        [run.pg_version for run in test_list_get(MODULE_LIST, _Config(vm=VM_U24, module=["integration"], pg_version="12"))],
+        ["12"] * 3,
+    )
+
+    # A version that the vm does not test selects nothing
+    assert_equal(
+        test_list_get(MODULE_LIST, _Config(vm=VM_U24, module=["integration"], pg_version="8.0")),
+        [],
+    )
+
+
+####################################################################################################################################
+def test_list_run():
+    """Each sub-test of an integration test is a run of its own, since each one needs its own containers."""
+
+    test_list = test_list_get(MODULE_LIST, _Config(vm=VM_U24, module=["integration"], pg_version="12"))
+
+    assert_equal([run.run_list for run in test_list], [[1], [2], [3]])
+    assert_equal(test_list[0].coverage_name, "integration-all")
+    assert_true(test_list[0].integration)
+    assert_false(test_list[0].performance)
+
+    # A run can be selected, which is the only way to get one container out of an integration test
+    assert_equal(
+        [run.run_list for run in test_list_get(MODULE_LIST, _Config(vm=VM_U24, module=["integration"], pg_version="12", run=[2]))],
+        [[2]],
+    )
+
+    # Every other kind of test runs all of its sub-tests at once, so the selection is passed on for the test itself to apply
+    test_list = test_list_get(MODULE_LIST, _Config(module=["performance"]))
+
+    assert_equal([run.run_list for run in test_list], [None])
+    assert_true(test_list[0].performance)
+    assert_false(test_list[0].integration)
+
+    assert_equal([run.run_list for run in test_list_get(MODULE_LIST, _Config(module=["performance"], run=[2]))], [[2]])
+
+    # A python test name contains a path separator, which is flattened for the coverage file since they share a path
+    assert_equal(test_list_get(MODULE_LIST, _Config(module=["test"]))[0].coverage_name, "test-common-log")

test/src/module/test/command/test/test_test.py

@@ -0,0 +1,535 @@
+"""Test Test Command.
+
+The command decides what to generate, build, and run rather than doing any of it, so the commands it would run are captured the way
+the container build and job tests capture theirs, and the jobs are replaced with ones that report whatever the test asked for.
+
+The repository it works on is written here, since the command reads the version and the test definitions from it and writes the
+version back."""
+
+####################################################################################################################################
+import io
+import os
+import tempfile
+from contextlib import redirect_stdout
+from unittest.mock import patch
+
+from harness.test import *
+
+from command.test.test import *
+from common.error import *
+from common.log import *
+from common.storage import file_read, file_write
+from common.vm import *
+
+# Version as the project writes it, i.e. the components the generated defines are built from
+VERSION_H = """/***********************************************************************************************************************************
+Version Numbers and Names
+***********************************************************************************************************************************/
+#define PROJECT_VERSION_MAJOR                                       2
+#define PROJECT_VERSION_MINOR                                       60
+#define PROJECT_VERSION_PATCH                                       0
+#define PROJECT_VERSION_SUFFIX                                      "dev"
+
+#define PROJECT_VERSION                                             "old"
+#define PROJECT_VERSION_NUM                                         0
+"""
+
+# The version defines as they are generated from the components above
+VERSION_DEFINE = """#define PROJECT_VERSION                                             "2.60.0dev"
+#define PROJECT_VERSION_NUM                                         2060000"""
+
+# Just enough of the meson build for the version to be updated in it
+MESON_BUILD = """project(
+    'pgbackrest',
+    ['c'],
+    version: 'old',
+    license: 'MIT',
+)
+"""
+
+# Test definitions with a test that provides coverage, one that does not, and an integration test that needs the binary
+DEFINE = """
+unit:
+  - name: common
+
+    test:
+      - name: error
+        total: 1
+
+        coverage:
+          - common/error
+
+      - name: exec
+        total: 1
+
+integration:
+  - name: integration
+    db: true
+
+    test:
+      - name: all
+        total: 1
+        binReq: true
+
+performance:
+  - name: performance
+
+    test:
+      - name: type
+        total: 1
+"""
+
+
+####################################################################################################################################
+class Config:
+    """What the command reads from the command line."""
+
+    def __init__(self, repo_path, test_path, **option):
+        self.repo_path = repo_path
+        self.test_path = test_path
+        self.vm = VM_NONE
+        self.vm_arch = None
+        self.vm_build = False
+        self.vm_force = False
+        self.vm_out = False
+        self.vm_max = 1
+        self.module = []
+        self.test = []
+        self.run = []
+        self.pg_version = "minimal"
+        self.c_only = False
+        self.container_only = False
+        self.coverage_only = False
+        self.coverage_summary = False
+        self.performance = True
+        self.dry_run = False
+        self.build_only = False
+        self.gen_only = False
+        self.clean = False
+        self.clean_only = False
+        self.cleanup = True
+        self.back_trace = True
+        self.valgrind = True
+        self.coverage = True
+        self.profile = False
+        self.scale = 1
+        self.tz = None
+        self.retry = 0
+        self.log_level = INFO
+        self.log_level_test = OFF
+        self.log_timestamp = False
+
+        for name, value in option.items():
+            setattr(self, name, value)
+
+
+####################################################################################################################################
+class _Job:
+    """Stand in for the job that runs a test and report what the test asked for."""
+
+    fail = {}  # How many tries a test fails for, by test name
+    start = True  # Does the job start, i.e. a dry run does not start one
+    poll = 0  # How many times a job reports that it is still running before it finishes
+    started = []  # Tests that were started, in the order they were started
+
+    def __init__(self, config, run, vm_idx, vm_max, test_idx, test_max, image, show_output):
+        self.name = run.module.name
+        self.retry = config.retry
+        self.try_idx = 0
+        self.poll_idx = _Job.poll
+
+    ################################################################################################################################
+    def begin(self):
+        self.try_idx += 1
+
+        # A test that has used up its retries is not started again, and neither is one on a dry run
+        if self.try_idx > self.retry + 1 or not _Job.start:
+            return False
+
+        _Job.started.append(self.name)
+
+        return True
+
+    ################################################################################################################################
+    def end(self):
+        if self.poll_idx > 0:
+            self.poll_idx -= 1
+
+            return False, False
+
+        return True, _Job.fail.get(self.name, 0) >= self.try_idx
+
+
+####################################################################################################################################
+class _Exec:
+    """Stand in for the coverage command, which reports whether every module was covered."""
+
+    status = 0
+    command = None
+
+    def __init__(self, command, show_output=False):
+        _Exec.command = command
+
+    ################################################################################################################################
+    def begin(self):
+        pass
+
+    ################################################################################################################################
+    def end(self, wait=True):
+        return _Exec.status
+
+
+####################################################################################################################################
+def _repo_create(path, define=DEFINE):
+    """Write the repository the command reads, i.e. the version, the meson build, and the test definitions."""
+
+    result = os.path.join(path, "repo")
+
+    file_write(os.path.join(result, "src/version.h"), VERSION_H)
+    file_write(os.path.join(result, "meson.build"), MESON_BUILD)
+    file_write(os.path.join(result, "test/define.yaml"), define)
+    file_write(os.path.join(result, "test/uncrustify.cfg"), "")
+
+    return result
+
+
+####################################################################################################################################
+def _cmd_test(config, exec_result=None, job_fail=None, job_start=True, job_poll=0, coverage_status=0):
+    """Run the command with everything it would run captured rather than run.
+
+    Returns the status, the commands, the tests that were started, and what was written to the log."""
+
+    command_list = []
+    output = io.StringIO()
+
+    def exec_fake(command, result_expect=0, show_output=False):
+        command_list.append(command)
+        result = exec_result.pop(0) if exec_result else ""
+
+        if isinstance(result, Exception):
+            raise result
+
+        return result
+
+    _Job.fail, _Job.start, _Job.poll, _Job.started = {} if job_fail is None else job_fail, job_start, job_poll, []
+    _Exec.status, _Exec.command = coverage_status, None
+
+    log_init(INFO, False)
+
+    try:
+        with patch("command.test.test.exec_one", exec_fake), patch("command.test.test.TestJob", _Job):
+            with patch("command.test.test.Exec", _Exec), patch("command.test.test.container_build") as container_build:
+                with patch("command.test.test.container_remove") as container_remove:
+                    with redirect_stdout(output):
+                        try:
+                            status = cmd_test(config)
+                        except TestError as error:
+                            status = str(error)
+
+                    _cmd_test.container_build = container_build
+                    _cmd_test.container_remove = container_remove
+    finally:
+        log_init(INFO, True)
+
+    return status, command_list, list(_Job.started), output.getvalue()
+
+
+####################################################################################################################################
+def test_test_version():
+    """The version is generated from the components in the version header and written where it is needed."""
+
+    with tempfile.TemporaryDirectory() as path:
+        path_repo = _repo_create(path)
+
+        status, command_list, started, output = _cmd_test(Config(path_repo, os.path.join(path, "test"), gen_only=True))
+
+        assert_equal(status, 0)
+        assert_in(VERSION_DEFINE, file_read(os.path.join(path_repo, "src/version.h")))
+        assert_in("    version: '2.60.0dev',", file_read(os.path.join(path_repo, "meson.build")))
+
+        # A version header without the components it is generated from is an error rather than a version of nothing
+        file_write(os.path.join(path_repo, "src/version.h"), '#define PROJECT_VERSION "2.60.0dev"\n')
+
+        status, command_list, started, output = _cmd_test(Config(path_repo, os.path.join(path, "test"), gen_only=True))
+
+        assert_equal(status, "unable to find PROJECT_VERSION_MAJOR in src/version.h")
+
+
+####################################################################################################################################
+def test_test_generate():
+    """Code generation runs before anything else since everything else is built from what it writes."""
+
+    with tempfile.TemporaryDirectory() as path:
+        path_repo = _repo_create(path)
+        path_test = os.path.join(path, "test")
+        config = Config(path_repo, path_test, gen_only=True)
+
+        status, command_list, started, output = _cmd_test(config)
+
+        assert_equal(status, 0)
+
+        # There is nothing built yet so the build is set up first, and the generator is run for everything it generates
+        assert_in(
+            "meson setup -Dwerror=true -Dfatal-errors=true -Dbuildtype=debug %s/build/none %s" % (path_test, path_repo),
+            command_list[-1],
+        )
+        assert_in("ninja -C %s/build/none src/build-code" % path_test, command_list[-1])
+
+        for generate in ("config", "error", "postgres-version", "postgres"):
+            assert_in("/src/build-code %s %s/src" % (generate, path_repo), command_list[-1])
+
+        assert_in("clean autogenerate code", output)
+
+        # A dry run only generates what building the test list depends on, since nothing will be built from the rest
+        file_write(os.path.join(path_test, "build/none/build.ninja"), "")
+
+        status, command_list, started, output = _cmd_test(Config(path_repo, path_test, gen_only=True, dry_run=True))
+
+        assert_not_in("build-code config", command_list[-1])
+        assert_in("build-code postgres ", command_list[-1])
+
+        # The build is only set up when it is not there already
+        assert_not_in("meson setup", command_list[-1])
+        assert_not_in("clean autogenerate code", output)
+
+
+####################################################################################################################################
+def test_test_clean():
+    """The working and result paths are emptied so a run starts from nothing."""
+
+    with tempfile.TemporaryDirectory() as path:
+        path_repo = _repo_create(path)
+        path_test = os.path.join(path, "test")
+
+        # Only the paths that are there are cleaned
+        file_write(os.path.join(path_test, "leftover"), "")
+
+        status, command_list, started, output = _cmd_test(Config(path_repo, path_test, clean_only=True))
+
+        assert_equal(status, 0)
+        assert_equal(command_list, ["find %s -mindepth 1 -print0 | xargs -0 rm -rf" % path_test])
+
+        # Cleaning without clean-only carries on with the run
+        file_write(os.path.join(path_repo, "test/result/report"), "")
+
+        status, command_list, started, output = _cmd_test(Config(path_repo, path_test, clean=True, gen_only=True))
+
+        assert_equal(status, 0)
+        assert_in("find %s/test/result -mindepth 1 -print0 | xargs -0 rm -rf" % path_repo, command_list)
+        assert_in("autogenerate code", output)
+
+
+####################################################################################################################################
+def test_test_option_error():
+    """An option that cannot be checked by the parser is checked here."""
+
+    with tempfile.TemporaryDirectory() as path:
+        path_repo = _repo_create(path)
+        path_test = os.path.join(path, "test")
+
+        def error(**option):
+            return _cmd_test(Config(path_repo, path_test, **option))[0]
+
+        assert_equal(error(run=[1, 2]), "only one --run can be provided")
+        assert_equal(error(test=["error"]), "only one --module can be provided when --test is specified")
+        assert_equal(error(module=["common"], run=[1]), "only one --test can be provided when --run is specified")
+        assert_equal(error(vm_build=True), "select a vm to build, or all of them")
+        assert_equal(error(vm=VM_ALL), "select a single vm to test on")
+        assert_equal(error(vm="bogus"), "no definition for vm 'bogus'")
+
+        # The test path may not be in the repository since the tests would then be part of what is being tested
+        assert_in(
+            "test path '%s' may not be in the repo path '%s'" % (os.path.join(path_repo, "test"), path_repo),
+            _cmd_test(Config(path_repo, os.path.join(path_repo, "test")))[0],
+        )
+
+        # A selection that matches nothing is an error rather than a run that does nothing
+        assert_equal(error(module=["bogus"]), "no tests were selected")
+
+        # Only one test can be run when nothing is cleaned up, since they would overwrite each other
+        assert_equal(error(cleanup=False), "--no-cleanup is not valid when more than one test will run")
+
+
+####################################################################################################################################
+def test_test_build():
+    """The binary is built when a test needs it and the build path is set up either way."""
+
+    with tempfile.TemporaryDirectory() as path:
+        path_repo = _repo_create(path)
+        path_test = os.path.join(path, "test")
+
+        # A unit test needs no binary but its build path is mounted into the container, so the path is still set up
+        status, command_list, started, output = _cmd_test(Config(path_repo, path_test, module=["common"], build_only=False))
+
+        assert_equal(status, 0)
+        assert_in(
+            "meson setup -Dwerror=true -Dfatal-errors=true -Dbuildtype=debug %s/build/none %s" % (path_test, path_repo),
+            command_list,
+        )
+        assert_not_in("ninja -C %s/build/none src/pgbackrest 2>&1" % path_test, " ".join(command_list))
+        assert_equal(started, ["common/error", "common/exec"])
+
+        # An integration test runs the binary so it is built. The build is already set up so only the binary is built.
+        status, command_list, started, output = _cmd_test(Config(path_repo, path_test, vm=VM_U24, module=["integration"]))
+
+        assert_in("ninja -C %s/build/u24 src/pgbackrest 2>&1" % path_test, " ".join(command_list))
+
+        # The build runs in the build container, which is started and shut down around it
+        assert_in("docker run -itd -h test-build --name=test-build", " ".join(command_list))
+        assert_in("docker rm -f test-build", command_list)
+
+        # Building only builds the binary and runs nothing
+        status, command_list, started, output = _cmd_test(Config(path_repo, path_test, build_only=True))
+
+        assert_equal(status, 0)
+        assert_equal(started, [])
+
+        # A build that is already set up and needs no binary has nothing to do
+        file_write(os.path.join(path_test, "build/none/build.ninja"), "")
+
+        status, command_list, started, output = _cmd_test(Config(path_repo, path_test, module=["common"]))
+
+        assert_not_in("meson setup", " ".join(command_list))
+        assert_not_in("src/pgbackrest", " ".join(command_list))
+
+
+####################################################################################################################################
+def test_test_container():
+    """The containers are built when they were asked for and cleaned up before a run."""
+
+    with tempfile.TemporaryDirectory() as path:
+        path_repo = _repo_create(path)
+        path_test = os.path.join(path, "test")
+
+        status, command_list, started, output = _cmd_test(Config(path_repo, path_test, vm=VM_U24, vm_build=True))
+
+        assert_equal(status, 0)
+        assert_equal(_cmd_test.container_build.call_count, 1)
+
+        # Nothing else is done since the containers are all that was asked for
+        assert_equal(command_list, [])
+
+        # Every vm can be built at once, which is the only thing that can be done for all of them
+        status, command_list, started, output = _cmd_test(Config(path_repo, path_test, vm=VM_ALL, vm_build=True))
+
+        assert_equal(status, 0)
+        assert_equal(_cmd_test.container_build.call_count, 1)
+
+        # A run on a vm removes whatever the last run left behind
+        status, command_list, started, output = _cmd_test(Config(path_repo, path_test, vm=VM_U24, module=["common"]))
+
+        assert_equal(_cmd_test.container_remove.call_args[0][0], "test-([0-9]+|build)")
+        assert_in("cleanup old data and containers", output)
+
+
+####################################################################################################################################
+def test_test_run():
+    """Every vm is kept busy until there is nothing left to run."""
+
+    with tempfile.TemporaryDirectory() as path:
+        path_repo = _repo_create(path)
+        path_test = os.path.join(path, "test")
+
+        # More tests than vms, so a vm picks up the next test as soon as it is free
+        status, command_list, started, output = _cmd_test(Config(path_repo, path_test, module=["common"], vm_max=2), job_poll=1)
+
+        assert_equal(status, 0)
+        assert_equal(started, ["common/error", "common/exec"])
+        assert_in("2 tests selected", output)
+        assert_in("TESTS COMPLETED SUCCESSFULLY", output)
+
+        # A test that fails is started again until it runs out of retries, and then the run has failed
+        status, command_list, started, output = _cmd_test(
+            Config(path_repo, path_test, module=["common"], test=["error"], retry=1), job_fail={"common/error": 1}
+        )
+
+        assert_equal(status, 0)
+        assert_equal(started, ["common/error", "common/error"])
+        assert_in("1 test selected", output)
+        assert_in("TESTS COMPLETED SUCCESSFULLY, 1 RETRY(IES)", output)
+
+        status, command_list, started, output = _cmd_test(
+            Config(path_repo, path_test, module=["common"], test=["error"]), job_fail={"common/error": 9}
+        )
+
+        assert_equal(status, 1)
+        assert_in("TESTS COMPLETED WITH 1 FAILURE(S)", output)
+
+        # A dry run lists the tests rather than running them
+        status, command_list, started, output = _cmd_test(Config(path_repo, path_test, dry_run=True), job_start=False)
+
+        assert_equal(status, 0)
+        assert_equal(started, [])
+        assert_in("DRY RUN COMPLETED SUCCESSFULLY", output)
+
+
+####################################################################################################################################
+def test_test_coverage():
+    """The coverage the tests produced is merged into one report once they have all passed."""
+
+    with tempfile.TemporaryDirectory() as path:
+        path_repo = _repo_create(path)
+        path_test = os.path.join(path, "test")
+
+        status, command_list, started, output = _cmd_test(Config(path_repo, path_test, module=["common"]))
+
+        assert_equal(status, 0)
+        assert_in("test.py coverage --log-level=warn --vm=none", _Exec.command)
+        assert_in("common/error common/exec", _Exec.command)
+        assert_in("tested modules have full coverage", output)
+
+        # A module that is not fully covered fails the run, and the report says which one so it is not repeated here
+        status, command_list, started, output = _cmd_test(Config(path_repo, path_test, module=["common"]), coverage_status=1)
+
+        assert_equal(status, 1)
+        assert_in("SUCCESSFULLY WITH MODULE(S) MISSING COVERAGE", output)
+
+        # The summary is generated for the documentation, where incomplete coverage is not a failure
+        status, command_list, started, output = _cmd_test(
+            Config(path_repo, path_test, module=["common"], coverage_summary=True), coverage_status=1
+        )
+
+        assert_equal(status, 0)
+        assert_in(" --coverage-summary ", _Exec.command)
+
+        # Anything worse than incomplete coverage is an error
+        status, command_list, started, output = _cmd_test(Config(path_repo, path_test, module=["common"]), coverage_status=2)
+
+        assert_equal(status, "coverage command failed")
+
+        # There is nothing to report when coverage was not collected, when a single run was selected since that cannot cover a
+        # module, or on a vm that does not collect it
+        _cmd_test(Config(path_repo, path_test, module=["common"], coverage=False))
+
+        assert_is_none(_Exec.command)
+
+        _cmd_test(Config(path_repo, path_test, module=["common"], test=["error"], run=[1]))
+
+        assert_is_none(_Exec.command)
+
+        _cmd_test(Config(path_repo, path_test, vm=VM_D12, module=["common"]))
+
+        assert_is_none(_Exec.command)
+
+
+####################################################################################################################################
+def test_test_profile():
+    """Profiling turns off everything that would skew the timing."""
+
+    with tempfile.TemporaryDirectory() as path:
+        path_repo = _repo_create(path)
+        config = Config(path_repo, os.path.join(path, "test"), module=["common"], profile=True)
+
+        _cmd_test(config)
+
+        assert_false(config.back_trace)
+        assert_false(config.valgrind)
+        assert_false(config.coverage)
+
+        # The coverage summary is built from the C tests that provide coverage, so it selects them
+        config = Config(path_repo, os.path.join(path, "test"), module=["common"], coverage_summary=True)
+
+        _cmd_test(config)
+
+        assert_true(config.coverage_only)
+        assert_true(config.c_only)

test/src/module/test/command/test/unit_test.py

@@ -229,29 +229,21 @@ def test_unit_build():
     with tempfile.TemporaryDirectory() as path:
         path_repo = _repo_create(path)
 
-        command_list, output, error = _cmd_unit(Config(path_repo, path, "common/error"), ["x86_64\n"])
+        command_list, output, error = _cmd_unit(Config(path_repo, path, "common/error"))
 
         assert_is_none(error)
 
-        # The architecture comes from the machine, then meson is set up since there is nothing built yet, then ninja builds it
-        assert_equal(command_list[0], "uname -m")
-        assert_in("meson setup -Dwerror=true -Dfatal-errors=true -Dbuildtype=debug -Db_coverage=true", command_list[1])
-        assert_in("ninja -C '%s/unit-0/none/build'" % path, command_list[2])
+        # Meson is set up since there is nothing built yet, then ninja builds it
+        assert_in("meson setup -Dwerror=true -Dfatal-errors=true -Dbuildtype=debug -Db_coverage=true", command_list[0])
+        assert_in("ninja -C '%s/unit-0/none/build'" % path, command_list[1])
 
-        # The architecture is passed to the build
-        assert_equal(_cmd_unit.build.call_args[0][2], "x86_64")
+        # The architecture comes from the machine when it was not given
+        assert_equal(_cmd_unit.build.call_args[0][2], host_arch())
 
-        # A machine that reports an architecture under another name is normalized to the name the project uses
-        for report, expect in (("i686\n", "i386"), ("arm64\n", "aarch64"), ("aarch64\n", "aarch64")):
-            _cmd_unit(Config(path_repo, path, "common/error"), [report])
-
-            assert_equal(_cmd_unit.build.call_args[0][2], expect, report)
-
-        # An architecture that was given is used as it is, so the machine is not asked
-        command_list, output, error = _cmd_unit(Config(path_repo, path, "common/error", vm_arch="ppc64le"))
+        # An architecture that was given is used as it is
+        _cmd_unit(Config(path_repo, path, "common/error", vm_arch="ppc64le"))
 
         assert_equal(_cmd_unit.build.call_args[0][2], "ppc64le")
-        assert_not_in("uname -m", command_list)
 
 
 ####################################################################################################################################

test/src/module/test/common/exec_test.py

@@ -1,6 +1,10 @@
 """Test Execute Process."""
 
 ####################################################################################################################################
+import io
+import time
+from contextlib import redirect_stdout
+
 from harness.test import *
 
 from common.error import *
@@ -46,3 +50,60 @@ def test_exec_expect():
         exec_one("true", 3)
 
     assert_equal(str(error.exception), "true terminated unexpectedly [0]")
+
+
+####################################################################################################################################
+def test_exec_status():
+    """A command that is expected to fail reports its status rather than raising."""
+
+    assert_equal(exec_status("true"), 0)
+    assert_equal(exec_status("exit 9"), 9)
+
+
+####################################################################################################################################
+def test_exec_async():
+    """A command runs in the background and what it wrote to each stream is kept apart."""
+
+    process = Exec("echo out; echo err 1>&2")
+    process.begin()
+
+    assert_equal(process.end(), 0)
+    assert_equal(process.output, "out\n")
+
+    # Anything on stderr means the command failed whatever its status, so it is reported separately
+    assert_equal(process.error, "err\n")
+
+
+####################################################################################################################################
+def test_exec_async_poll():
+    """A command that is still running reports no status, which is how the driver keeps every vm busy."""
+
+    process = Exec("sleep 0.2; echo done")
+    process.begin()
+
+    poll = 0
+
+    while process.end(wait=False) is None:
+        poll += 1
+        time.sleep(0.01)
+
+    # The command was polled at least once while it was still running
+    assert_true(poll > 0)
+    assert_equal(process.output, "done\n")
+
+
+####################################################################################################################################
+def test_exec_async_show():
+    """Output is written as it arrives, a line at a time, so a long command shows progress."""
+
+    # The first write has no line ending so there is nothing to show until the rest of the line arrives
+    process = Exec("printf partial; sleep 0.2; echo ' line'; echo next", show_output=True)
+    output = io.StringIO()
+
+    with redirect_stdout(output):
+        process.begin()
+
+        assert_equal(process.end(), 0)
+
+    assert_equal(output.getvalue(), "    partial line\n    next\n")
+    assert_equal(process.output, "partial line\nnext\n")

test/src/module/test/common/user_test.py

@@ -0,0 +1,21 @@
+"""Test User and Group."""
+
+####################################################################################################################################
+import grp
+import os
+import pwd
+
+from harness.test import *
+
+from common.user import *
+
+
+####################################################################################################################################
+def test_user():
+    """The user and group are whoever the tests are running as."""
+
+    assert_equal(user_id(), os.getuid())
+    assert_equal(user_name(), pwd.getpwuid(os.getuid()).pw_name)
+
+    assert_equal(group_id(), os.getgid())
+    assert_equal(group_name(), grp.getgrgid(os.getgid()).gr_name)

test/src/module/test/common/vm_test.py

@@ -0,0 +1,91 @@
+"""Test Virtual Machine Definitions.
+
+The definitions in the module are checked when it is imported, so a broken definition would fail before any test here runs. The
+checks are exercised again with definitions built here, since the real ones can only ever be valid."""
+
+####################################################################################################################################
+from harness.test import *
+
+from common.error import *
+from common.vm import *
+
+
+####################################################################################################################################
+def _vm_map(db_test_list=None, coverage_c=True):
+    """Build a vm map that covers every default vm, which is what the check looks at.
+
+    Every supported version is tested on the first vm unless the caller says otherwise."""
+
+    result = {vm: Vm(VM_OS_BASE_DEBIAN, []) for vm in VM_LIST}
+    result[VM_LIST[0]] = Vm(
+        VM_OS_BASE_DEBIAN, list(PG_VERSION_LIST) if db_test_list is None else db_test_list, coverage_c=coverage_c
+    )
+
+    return result
+
+
+####################################################################################################################################
+def test_vm_get():
+    """A vm definition is found by name and a name with no definition is an error."""
+
+    assert_true(vm_valid(VM_U24))
+    assert_false(vm_valid("bogus"))
+
+    vm = vm_get(VM_U24)
+
+    assert_equal(vm.os_base, VM_OS_BASE_DEBIAN)
+    assert_equal(vm.image, "ubuntu:24.04")
+    assert_true(vm.pg_repo)
+    assert_true(vm.coverage_c)
+
+    # A vm that does not install from the PostgreSQL repo and tests every version it installs
+    vm = vm_get(VM_D12)
+
+    assert_false(vm.pg_repo)
+    assert_false(vm.coverage_c)
+    assert_equal(vm.db_list, vm.db_test_list)
+
+    # There is no container for none, since it runs on the host
+    assert_is_none(vm_get(VM_NONE).image)
+
+    with assert_raises(TestError) as error:
+        vm_get("bogus")
+
+    assert_equal(str(error.exception), "no definition for vm 'bogus'")
+
+
+####################################################################################################################################
+def test_vm_check():
+    """Definitions that leave a version or C coverage untested are an error."""
+
+    assert_is_none(vm_check(_vm_map()))
+
+    # C coverage must be collected somewhere, else a change could stop being checked without anything reporting it
+    with assert_raises(TestError) as error:
+        vm_check(_vm_map(coverage_c=False))
+
+    assert_equal(str(error.exception), "C coverage is not configured to run on a default vm")
+
+    # A version tested on more than one vm is an error since the extra run is just slower with nothing added
+    vm_map = _vm_map()
+    vm_map[VM_LIST[1]] = Vm(VM_OS_BASE_DEBIAN, [PG_VERSION_LIST[0]])
+
+    with assert_raises(TestError) as error:
+        vm_check(vm_map)
+
+    assert_equal(
+        str(error.exception), "PostgreSQL %s is already configured to run on default vm %s" % (PG_VERSION_LIST[0], VM_LIST[0])
+    )
+
+    # A version tested on no vm is an error, which is what catches a version that was added but never wired up
+    with assert_raises(TestError) as error:
+        vm_check(_vm_map(db_test_list=list(PG_VERSION_LIST[1:])))
+
+    assert_equal(str(error.exception), "PostgreSQL %s is not configured to run on a default vm" % PG_VERSION_LIST[0])
+
+
+####################################################################################################################################
+def test_vm_arch():
+    """The host architecture is reported with the name the project uses."""
+
+    assert_true(host_arch() in VM_ARCH_LIST)

test/src/module/test/config/cli_test.py

@@ -129,27 +129,56 @@ def test_cli_coverage():
     assert_true(config.coverage_summary)
 
 
+####################################################################################################################################
+def test_cli_test():
+    """A command line with no command runs the tests, which is what a developer types."""
+
+    config = cli_parse(["--vm=u24", "--module=common", "--module=postgres", "--test=error", "--vm-max=2"], VERSION)
+
+    assert_is_none(config.command)
+    assert_equal(config.vm, "u24")
+    assert_equal(config.module, ["common", "postgres"])
+    assert_equal(config.test, ["error"])
+    assert_equal(config.vm_max, 2)
+
+    # Defaults for what a plain run does
+    assert_equal(config.test_path, "test")
+    assert_equal(config.run, [])
+    assert_equal(config.pg_version, "minimal")
+    assert_equal(config.retry, 0)
+    assert_equal(config.scale, 1)
+    assert_is_none(config.tz)
+    assert_true(config.cleanup)
+    assert_true(config.coverage)
+    assert_true(config.valgrind)
+    assert_true(config.back_trace)
+    assert_true(config.performance)
+    assert_false(config.dry_run)
+    assert_false(config.quiet)
+
+    # A run works on the repository it is part of so it is never told where that is
+    assert_false(hasattr(config, "repo_path"))
+
+
 ####################################################################################################################################
 def test_cli_lint():
-    """The lint command reads the repository only, so it has none of the test path options."""
+    """The lint command reads the repository only."""
 
     config = cli_parse(["lint", "--repo-path=/repo"], VERSION)
 
     assert_equal(config.command, "lint")
     assert_equal(config.repo_path, "/repo")
-    assert_false(hasattr(config, "test_path"))
-    assert_false(hasattr(config, "vm"))
 
 
 ####################################################################################################################################
 def test_cli_error():
     """A command line that does not parse reports and exits."""
 
-    # A command is required since there is no default one
-    status, output = _cli_parse_exit([])
+    # An option that is not defined is rejected rather than passed on to a test
+    status, output = _cli_parse_exit(["--bogus"])
 
     assert_equal(status, 2)
-    assert_in("the following arguments are required: command", output)
+    assert_in("unrecognized arguments: --bogus", output)
 
     # A level outside the log levels is rejected here rather than later when it is converted
     status, output = _cli_parse_exit(["unit", "common/error", "--log-level=bogus"])

test/src/module/test/config/config_test.py

@@ -85,9 +85,26 @@ def test_config_load():
         assert_equal(config.test_path, path)
 
 
+####################################################################################################################################
+def test_config_load_test():
+    """A run with no command runs the tests, on the repository it is part of rather than on the copy they are built from."""
+
+    with tempfile.TemporaryDirectory() as path:
+        _version_write(path)
+
+        config = _cfg_load(["--test-path=%s/test" % path], path)
+
+        assert_is_none(config.command)
+        assert_equal(config.repo_path, path)
+        assert_equal(config.log_level, INFO)
+
+        # Quiet is a shorthand for turning the log off, which is how the documentation build runs the tests
+        assert_equal(_cfg_load(["--test-path=%s/test" % path, "--quiet"], path).log_level, OFF)
+
+
 ####################################################################################################################################
 def test_config_load_lint():
-    """The lint command has neither a test path nor a test log level, so neither is applied."""
+    """The lint command is told where the repository is, since it runs from the copy the tests are built from."""
 
     with tempfile.TemporaryDirectory() as path:
         _version_write(path)
@@ -95,9 +112,8 @@ def test_config_load_lint():
         config = _cfg_load(["lint", "--repo-path=%s" % path], path)
 
         assert_equal(config.command, "lint")
+        assert_equal(config.repo_path, path)
         assert_equal(config.log_level, INFO)
-        assert_false(hasattr(config, "test_path"))
-        assert_false(hasattr(config, "log_level_test"))
 
 
 ####################################################################################################################################

test/test.pl

@@ -1,871 +0,0 @@
-#!/usr/bin/perl
-####################################################################################################################################
-# test.pl - pgBackRest Unit Tests
-####################################################################################################################################
-
-####################################################################################################################################
-# Perl includes
-####################################################################################################################################
-use strict;
-use warnings FATAL => qw(all);
-use Carp qw(confess longmess);
-use English '-no_match_vars';
-
-# Convert die to confess to capture the stack trace
-$SIG{__DIE__} = sub { Carp::confess @_ };
-
-use Digest::SHA qw(sha1_hex);
-use File::Basename qw(dirname);
-use File::stat;
-use Getopt::Long qw(GetOptions);
-use Cwd qw(abs_path cwd);
-use JSON::PP;
-use Pod::Usage qw(pod2usage);
-use POSIX qw(ceil strftime);
-use Time::HiRes qw(gettimeofday);
-
-use lib dirname($0) . '/lib';
-use lib dirname(dirname($0)) . '/lib';
-use lib dirname(dirname($0)) . '/build/lib';
-use lib dirname(dirname($0)) . '/doc/lib';
-
-use pgBackRestDoc::Common::Exception;
-use pgBackRestDoc::Common::Log;
-use pgBackRestDoc::Common::String;
-use pgBackRestDoc::ProjectInfo;
-
-use pgBackRestTest::Common::BuildTest;
-use pgBackRestTest::Common::ContainerTest;
-use pgBackRestTest::Common::DefineTest;
-use pgBackRestTest::Common::ExecuteTest;
-use pgBackRestTest::Common::JobTest;
-use pgBackRestTest::Common::ListTest;
-use pgBackRestTest::Common::Storage;
-use pgBackRestTest::Common::StoragePosix;
-use pgBackRestTest::Common::VmTest;
-use pgBackRestTest::Common::Wait;
-
-####################################################################################################################################
-# Usage
-####################################################################################################################################
-
-=head1 NAME
-
-test.pl - pgBackRest Unit Tests
-
-=head1 SYNOPSIS
-
-test.pl [options]
-
- Test Options:
-   --module             test module to execute
-   --test               execute the specified test in a module
-   --run                execute only the specified test run
-   --dry-run            show only the tests that would be executed but don't execute them
-   --no-cleanup         don't cleanup after the last test is complete - useful for debugging
-   --clean              clean working and result paths for a completely fresh build
-   --clean-only         execute --clean and exit
-   --pg-version         version of postgres to test (all, defaults to minimal)
-   --build-only         build the binary but don't run tests
-   --build-max          max processes to use for builds (default 4)
-   --c-only             only run C tests
-   --container-only     only run tests that must be run in a container
-   --no-performance     do not run performance tests
-   --gen-only           only run auto-generation
-   --gen-check          check that auto-generated files are correct (used in CI to detect changes)
-   --no-back-trace      don't run backrace on C unit tests (may be slow with valgrind)
-   --no-valgrind        don't run valgrind on C unit tests (saves time)
-   --no-coverage        don't run coverage on C unit tests (saves time)
-   --no-coverage-report run coverage but don't generate coverage report (saves time)
-   --no-optimize        don't do compile optimization for C (saves compile time)
-   --profile            generate profile info
-   --no-debug           don't generate a debug build
-   --scale              scale performance tests
-   --tz                 test with the specified timezone
-   --debug-test-trace   test stack trace for low-level functions (slow, esp w/valgrind, may cause timeouts)
-
- Code Format Options:
-   --code-format        format code to project standards -- this may overwrite files!
-   --code-format-check  check that code is formatted to project standards
-
- Report Options:
-   --coverage-summary   generate a coverage summary report for the documentation
-   --coverage-only      only run coverage tests (as a subset of selected tests) for the documentation
-
- Configuration Options:
-   --psql-bin           path to the psql executables (e.g. /usr/lib/postgresql/16/bin/)
-   --test-path          path where tests are executed (defaults to ./test)
-   --log-level          log level to use for test harness (and Perl tests) (defaults to INFO)
-   --log-level-test     log level to use for C tests (defaults to OFF)
-   --log-level-test-file log level to use for file logging in integration tests (defaults to TRACE)
-   --no-log-timestamp   suppress timestamps, timings, etc. Used to generate documentation.
-   --quiet, -q          equivalent to --log-level=off
-
- VM Options:
-   --vm                 docker container to build/test (e.g. rh8)
-   --vm-arch            docker container architecture
-   --vm-build           build Docker containers
-   --vm-force           force a rebuild of Docker containers
-   --vm-out             Show VM output (default false)
-   --vm-max             max VMs to run in parallel (default 1)
-
- General Options:
-   --version            display version and exit
-   --help               display usage and exit
-=cut
-
-####################################################################################################################################
-# Command line parameters
-####################################################################################################################################
-my $bClean;
-my $bCleanOnly;
-my $strLogLevel = lc(INFO);
-my $strLogLevelTest = lc(OFF);
-my $strLogLevelTestFile = lc(DEBUG);
-my $bNoLogTimestamp = false;
-my $bVmOut = false;
-my @stryModule;
-my @stryModuleTest;
-my @iyModuleTestRun;
-my $iVmMax = 1;
-my $bDryRun = false;
-my $bCodeFormat = false;
-my $bCodeFormatCheck = false;
-my $bNoCleanup = false;
-my $strPgSqlBin;
-my $strTestPath;
-my $bVersion = false;
-my $bHelp = false;
-my $bQuiet = false;
-my $strPgVersion = 'minimal';
-my $strVm = VM_NONE;
-my $strVmArch;
-my $bVmBuild = false;
-my $bVmForce = false;
-my $bBuildOnly = false;
-my $iBuildMax = 4;
-my $bCoverageOnly = false;
-my $bCoverageSummary = false;
-my $bNoCoverage = false;
-my $bNoCoverageReport = false;
-my $bCOnly = false;
-my $bContainerOnly = false;
-my $bNoPerformance = false;
-my $bGenOnly = false;
-my $bGenCheck = false;
-my $bProfile = false;
-my $bNoBackTrace = false;
-my $bNoValgrind = false;
-my $bNoOptimize = false;
-my $bNoDebug = false;
-my $iScale = 1;
-my $bDebugTestTrace = false;
-my $iRetry = 0;
-my $strTimeZone = undef;
-
-my @cmdOptions = @ARGV;
-
-GetOptions ('q|quiet' => \$bQuiet,
-            'version' => \$bVersion,
-            'help' => \$bHelp,
-            'clean' => \$bClean,
-            'clean-only' => \$bCleanOnly,
-            'psql-bin=s' => \$strPgSqlBin,
-            'test-path=s' => \$strTestPath,
-            'log-level=s' => \$strLogLevel,
-            'log-level-test=s' => \$strLogLevelTest,
-            'log-level-test-file=s' => \$strLogLevelTestFile,
-            'no-log-timestamp' => \$bNoLogTimestamp,
-            'vm=s' => \$strVm,
-            'vm-arch=s' => \$strVmArch,
-            'vm-out' => \$bVmOut,
-            'vm-build' => \$bVmBuild,
-            'vm-force' => \$bVmForce,
-            'module=s@' => \@stryModule,
-            'test=s@' => \@stryModuleTest,
-            'run=s@' => \@iyModuleTestRun,
-            'vm-max=s' => \$iVmMax,
-            'dry-run' => \$bDryRun,
-            'no-cleanup' => \$bNoCleanup,
-            'pg-version=s' => \$strPgVersion,
-            'build-only' => \$bBuildOnly,
-            'build-max=s' => \$iBuildMax,
-            'coverage-only' => \$bCoverageOnly,
-            'coverage-summary' => \$bCoverageSummary,
-            'no-coverage' => \$bNoCoverage,
-            'no-coverage-report' => \$bNoCoverageReport,
-            'c-only' => \$bCOnly,
-            'container-only' => \$bContainerOnly,
-            'no-performance' => \$bNoPerformance,
-            'gen-only' => \$bGenOnly,
-            'gen-check' => \$bGenCheck,
-            'code-format' => \$bCodeFormat,
-            'code-format-check' => \$bCodeFormatCheck,
-            'profile' => \$bProfile,
-            'no-back-trace' => \$bNoBackTrace,
-            'no-valgrind' => \$bNoValgrind,
-            'no-optimize' => \$bNoOptimize,
-            'no-debug', => \$bNoDebug,
-            'scale=s' => \$iScale,
-            'tz=s', => \$strTimeZone,
-            'debug-test-trace', => \$bDebugTestTrace,
-            'retry=s' => \$iRetry)
-    or pod2usage(2);
-
-####################################################################################################################################
-# Run in eval block to catch errors
-####################################################################################################################################
-eval
-{
-    # Record the start time
-    my $lStartTime = time();
-
-    # Display version and exit if requested
-    if ($bVersion || $bHelp)
-    {
-        syswrite(*STDOUT, PROJECT_NAME . ' ' . PROJECT_VERSION . " Test Engine\n");
-
-        if ($bHelp)
-        {
-            syswrite(*STDOUT, "\n");
-            pod2usage();
-        }
-
-        exit 0;
-    }
-
-    if (@ARGV > 0)
-    {
-        syswrite(*STDOUT, "invalid parameter\n\n");
-        pod2usage();
-    }
-
-    ################################################################################################################################
-    # Update options for --coverage-summary
-    ################################################################################################################################
-    if ($bCoverageSummary)
-    {
-        $bCoverageOnly = true;
-        $bCOnly = true;
-    }
-
-    ################################################################################################################################
-    # Update options for --profile
-    ################################################################################################################################
-    if ($bProfile)
-    {
-        $bNoBackTrace = true;
-        $bNoValgrind = true;
-        $bNoCoverage = true;
-    }
-
-    ################################################################################################################################
-    # Setup
-    ################################################################################################################################
-    # Set a neutral umask so tests work as expected
-    umask(0);
-
-    # Set console log level
-    if ($bQuiet)
-    {
-        $strLogLevel = 'off';
-    }
-
-    logLevelSet(uc($strLogLevel), uc($strLogLevel), OFF, !$bNoLogTimestamp);
-    &log(INFO, 'test begin on ' . hostArch() . " - log level ${strLogLevel}");
-
-    if (@iyModuleTestRun > 1)
-    {
-        confess "Only one --run can be provided";
-    }
-
-    if (@stryModuleTest != 0 && @stryModule != 1)
-    {
-        confess "Only one --module can be provided when --test is specified";
-    }
-
-    if (@iyModuleTestRun != 0 && @stryModuleTest != 1)
-    {
-        confess "Only one --test can be provided when --run is specified";
-    }
-
-    # Check vm
-    vmValid($strVm);
-
-    # Set test path if not explicitly set
-    if (!defined($strTestPath))
-    {
-        $strTestPath = cwd() . '/test';
-    }
-
-    my $oStorageTest = new pgBackRestTest::Common::Storage(
-        $strTestPath, new pgBackRestTest::Common::StoragePosix({bFileSync => false, bPathSync => false}));
-
-    if ($bCoverageOnly)
-    {
-        if (!defined($strVm))
-        {
-            confess &log(ERROR, "select a VM for coverage testing");
-        }
-        elsif ($strVm eq VM_ALL)
-        {
-            confess &log(ERROR, "select a single VM for coverage testing");
-        }
-    }
-
-    # Get the base backrest path
-    my $strBackRestBase = dirname(dirname(abs_path($0)));
-    my $strVagrantPath = "${strBackRestBase}/test/.vagrant";
-
-    my $oStorageBackRest = new pgBackRestTest::Common::Storage(
-        $strBackRestBase, new pgBackRestTest::Common::StoragePosix({bFileSync => false, bPathSync => false}));
-
-    # Check that the test path is not in the git repo path
-    if (index("${strTestPath}/", "${strBackRestBase}/") != -1)
-    {
-        confess &log(
-            ERROR,
-            "test path '${strTestPath}' may not be in the repo path '${strBackRestBase}'\n" .
-            "HINT: was test.pl run in '${strBackRestBase}'?\n" .
-            "HINT: use --test-path to set a test path\n" .
-            "HINT: run test.pl from outside the repo, e.g. 'pgbackrest/test/test.pl'");
-    }
-
-    ################################################################################################################################
-    # Clean working and result paths
-    ################################################################################################################################
-    if ($bClean || $bCleanOnly)
-    {
-        &log(INFO, "clean working (${strTestPath}) and result (${strBackRestBase}/test/result) paths");
-
-        if ($oStorageTest->pathExists($strTestPath))
-        {
-            executeTest("find ${strTestPath} -mindepth 1 -print0 | xargs -0 rm -rf");
-        }
-
-        if ($oStorageTest->pathExists("${strBackRestBase}/test/result"))
-        {
-            executeTest("find ${strBackRestBase}/test/result -mindepth 1 -print0 | xargs -0 rm -rf");
-        }
-
-        # Exit when clean-only
-        exit 0 if $bCleanOnly;
-    }
-
-    ################################################################################################################################
-    # Build Docker containers
-    ################################################################################################################################
-    if ($bVmBuild)
-    {
-        containerBuild($oStorageBackRest, $strVm, $strVmArch, $bVmForce);
-        exit 0;
-    }
-
-    ################################################################################################################################
-    # Load test definition
-    ################################################################################################################################
-    testDefLoad(${$oStorageBackRest->get("test/define.yaml")});
-
-    ################################################################################################################################
-    # Start VM and run
-    ################################################################################################################################
-
-    # Clean up
-    #-------------------------------------------------------------------------------------------------------------------------------
-    my $iTestFail = 0;
-    my $iTestRetry = 0;
-    my $oyProcess = [];
-    my $strCodePath = "${strBackRestBase}/test/result/coverage/raw";
-
-    if (!$bDryRun || $bVmOut)
-    {
-        &log(INFO, "cleanup old data" . ($strVm ne VM_NONE ? " and containers" : ''));
-
-        if ($strVm ne VM_NONE)
-        {
-            containerRemove('test-([0-9]+|build)');
-        }
-
-        for (my $iVmIdx = 0; $iVmIdx < 8; $iVmIdx++)
-        {
-            push(@{$oyProcess}, undef);
-        }
-
-        executeTest(
-            "chmod 700 -R ${strTestPath}/test-* 2>&1 || true && rm -rf ${strTestPath}/temp ${strTestPath}/test-*" .
-                " ${strTestPath}/data-*");
-        $oStorageTest->pathCreate("${strTestPath}/temp", {strMode => '0770', bIgnoreExists => true, bCreateParent => true});
-
-        # Overwrite the C coverage report so it will load but not show old coverage
-        $oStorageTest->pathCreate(
-            "${strBackRestBase}/test/result/coverage", {strMode => '0770', bIgnoreExists => true, bCreateParent => true});
-        $oStorageBackRest->put(
-            "${strBackRestBase}/test/result/coverage/coverage.html",
-            "<center>[ " . ($bNoCoverage ? "No Coverage Testing" : "Generating Coverage Report") . " ]</center>");
-
-        # Copy C code for coverage tests
-        if (vmCoverageC($strVm) && !$bDryRun)
-        {
-            executeTest("rm -rf ${strBackRestBase}/test/result/coverage/raw/*");
-            $oStorageTest->pathCreate("${strCodePath}", {strMode => '0770', bIgnoreExists => true, bCreateParent => true});
-        }
-    }
-
-    # Auto-generate version for root meson.build script
-    #-------------------------------------------------------------------------------------------------------------------------------
-    my $strMesonBuildOld = ${$oStorageTest->get("${strBackRestBase}/meson.build")};
-    my $strMesonBuildNew;
-
-    foreach my $strLine (split("\n", $strMesonBuildOld))
-    {
-        if ($strLine =~ /^    version\: '/)
-        {
-            $strLine = "    version: '" . PROJECT_VERSION . "',";
-        }
-
-        $strMesonBuildNew .= "${strLine}\n";
-    }
-
-    buildPutDiffers($oStorageBackRest, "${strBackRestBase}/meson.build", $strMesonBuildNew);
-
-    # Auto-generate version for version.h
-    #-------------------------------------------------------------------------------------------------------------------------------
-    my $strVersionCOld = ${$oStorageTest->get("${strBackRestBase}/src/version.h")};
-    my $strVersionCNew;
-
-    foreach my $strLine (split("\n", $strVersionCOld))
-    {
-        if ($strLine =~ /^#define PROJECT_VERSION /)
-        {
-            $strLine = "#define PROJECT_VERSION" . (' ' x 45) . '"' . PROJECT_VERSION . '"';
-        }
-
-        if ($strLine =~ /^#define PROJECT_VERSION_NUM /)
-        {
-            $strLine =
-                "#define PROJECT_VERSION_NUM" . (' ' x 41) .
-                sprintf('%d%03d%03d', PROJECT_VERSION_MAJOR, PROJECT_VERSION_MINOR, PROJECT_VERSION_PATCH);
-        }
-
-        $strVersionCNew .= "${strLine}\n";
-    }
-
-    buildPutDiffers($oStorageBackRest, "${strBackRestBase}/src/version.h", $strVersionCNew);
-
-    # Set the ccache path
-    #-------------------------------------------------------------------------------------------------------------------------------
-    # A single cache is shared by the host and all containers so a file compiled in one is a hit in the others, e.g. a module
-    # compiled for one unit test is a hit for every other unit test that includes it. ccache is safe for concurrent access so there
-    # is no need to keep a separate cache per vm or vm index. Keep the cache in the test path so it is removed by --clean.
-    my $strCCachePath = "${strTestPath}/ccache";
-
-    if (!$oStorageTest->pathExists($strCCachePath))
-    {
-        $oStorageTest->pathCreate($strCCachePath, {strMode => '0770', bCreateParent => true});
-    }
-
-    # Set for builds that run on the host. Containers are passed the path explicitly since docker does not pass the environment.
-    $ENV{CCACHE_DIR} = $strCCachePath;
-
-    # Start build container if vm is not none
-    #-------------------------------------------------------------------------------------------------------------------------------
-    my $strPlatform = defined($strVmArch) ? " --platform linux/${strVmArch}" : '';
-    my $strImage = containerRepo() . ":${strVm}-test" . (defined($strVmArch) ? "-${strVmArch}" : '-' . hostArch());
-
-    if ($strVm ne VM_NONE)
-    {
-        # The cache does not need to be mounted since it is in the test path, which is mounted below
-        executeTest(
-            "docker run${strPlatform} -itd -h test-build --name=test-build" .
-                " -v ${strBackRestBase}:${strBackRestBase} -v ${strTestPath}:${strTestPath}" .
-                " -e CCACHE_DIR=${strCCachePath} ${strImage}",
-            {bSuppressStdErr => true});
-    }
-
-    # Create path for repo cache
-    #-------------------------------------------------------------------------------------------------------------------------------
-    my $strRepoCachePath = "${strTestPath}/repo";
-
-    # Create the repo path -- this should hopefully prevent obvious rsync errors below
-    $oStorageTest->pathCreate($strRepoCachePath, {strMode => '0770', bIgnoreExists => true, bCreateParent => true});
-
-    # Auto-generate code files (if --dry-run specified then do minimum required)
-    #-------------------------------------------------------------------------------------------------------------------------------
-    my $strBuildPath = "${strTestPath}/build/none";
-    my $strBuildNinja = "${strBuildPath}/build.ninja";
-
-    &log(INFO, (!-e $strBuildNinja ? 'clean ' : '') . 'autogenerate code');
-
-    # Setup build if it does not exist
-    my $strGenerateCommand =
-        "ninja -C ${strBuildPath} src/build-code 2>&1" .
-        ($bDryRun ? '' : " && \\\n${strBuildPath}/src/build-code config ${strBackRestBase}/src") .
-        ($bDryRun ? '' : " && \\\n${strBuildPath}/src/build-code error ${strBackRestBase}/src") .
-        ($bDryRun ? '' : " && \\\n${strBuildPath}/src/build-code postgres-version ${strBackRestBase}/src") .
-        " && \\\n${strBuildPath}/src/build-code postgres ${strBackRestBase}/src ${strRepoCachePath}";
-
-    if (!-e $strBuildNinja)
-    {
-        $strGenerateCommand =
-            "meson setup -Dwerror=true -Dfatal-errors=true -Dbuildtype=debug ${strBuildPath} ${strBackRestBase} && \\\n" .
-            $strGenerateCommand;
-    }
-
-    # Build code
-    executeTest($strGenerateCommand);
-
-    if ($bGenOnly)
-    {
-        exit 0;
-    }
-
-    # Make a copy of the repo to track which files have been changed
-    #-------------------------------------------------------------------------------------------------------------------------------
-    executeTest(
-        "git -C ${strBackRestBase} ls-files -c --others --exclude-standard |" .
-            " rsync -rLtW --delete --files-from=- --exclude=test/result" .
-            " --exclude=src/postgres/interface.auto.c.inc --exclude=src/command/help/help.auto.c.inc" .
-            # This option is not supported on MacOS. The eventual plan is to remove the need for it.
-            (trim(`uname`) ne 'Darwin' ? ' --ignore-missing-args' : '') .
-            " ${strBackRestBase}/ ${strRepoCachePath}");
-
-    # Format code with uncrustify and check permissions
-    #-------------------------------------------------------------------------------------------------------------------------------
-    if ($bCodeFormat || $bCodeFormatCheck)
-    {
-        &log(INFO, 'code format' . ($bCodeFormatCheck ? ' check' : ''));
-
-        my $hRepoManifest = $oStorageTest->manifest($strRepoCachePath);
-        my $hManifest = $oStorageBackRest->manifest('');
-        my $strCommand =
-            "uncrustify -c ${strBackRestBase}/test/uncrustify.cfg" .
-            ($bCodeFormatCheck ? ' --check' : ' --replace --no-backup');
-
-        foreach my $strFile (sort(keys(%{$hManifest})))
-        {
-            # Skip non-C files
-            next if $hManifest->{$strFile}{type} ne 'f' || ($strFile !~ /\.c.inc$/ && $strFile !~ /\.c$/ && $strFile !~ /\.h$/);
-
-            # Skip files that do are not version controlled
-            next if !defined($hRepoManifest->{$strFile});
-
-            # Skip specific file
-            next if
-                # Does not format correctly because it is a template
-                $strFile eq 'test/src/test.c' ||
-                # Skip vendorized and automatically generated files
-                $strFile =~ /\.vendor\.h$/ ||
-                $strFile =~ /\.(vendor|auto)\.c\.inc$/;
-
-            $strCommand .= " ${strBackRestBase}/${strFile}";
-        }
-
-        executeTest($strCommand . " 2>&1");
-
-        # Check execute permissions to make sure nothing got munged
-        foreach my $strFile (sort(keys(%{$hManifest})))
-        {
-            next if ($strFile eq '.') || !defined($hRepoManifest->{$strFile});
-
-            my $strExpectedMode = sprintf('%04o', oct($hManifest->{$strFile}{mode}) & 0666);
-
-            if ($strFile eq 'doc/doc.pl' ||
-                $strFile eq 'doc/release.pl' ||
-                $strFile eq 'test/ci.pl' ||
-                $strFile eq 'test/smoke.py' ||
-                $strFile eq 'test/test.pl' ||
-                $strFile eq 'test/test.py' ||
-                $strFile eq 'src/build/dist.sh' ||
-                $hManifest->{$strFile}{type} eq 'd')
-            {
-                $strExpectedMode = sprintf('%04o', oct($hManifest->{$strFile}{mode}) & 0777);
-            }
-
-            if ($hManifest->{$strFile}{mode} ne $strExpectedMode)
-            {
-                confess &log(
-                    ERROR,
-                    "expected mode '${strExpectedMode}' for '${strFile}' but found '" . $hManifest->{$strFile}{mode} . "'");
-            }
-        }
-
-        exit 0;
-    }
-
-    # Determine which tests to run
-    #-------------------------------------------------------------------------------------------------------------------------------
-    my $oyTestRun;
-    my $bBinRequired = $bBuildOnly;
-    my $bUnitRequired = $bBuildOnly;
-
-    # Only get the test list when they can run
-    if (!$bBuildOnly)
-    {
-        # Get the test list
-        $oyTestRun = testListGet(
-            $strVm, \@stryModule, \@stryModuleTest, \@iyModuleTestRun, $strPgVersion, $bCoverageOnly, $bCOnly, $bContainerOnly,
-            $bNoPerformance);
-
-        # Determine if the C binary needs to be built
-        foreach my $hTest (@{$oyTestRun})
-        {
-            # The build path is mounted into the test container so it must exist for unit tests even when nothing is built there
-            if ($hTest->{&TEST_C})
-            {
-                $bUnitRequired = true;
-            }
-
-            # Bin build required for integration tests and when specified
-            if (!$hTest->{&TEST_C} || $hTest->{&TEST_BIN_REQ})
-            {
-                $bBinRequired = true;
-            }
-        }
-    }
-
-    # Build the binary
-    #-------------------------------------------------------------------------------------------------------------------------------
-    if (!$bDryRun)
-    {
-        my $oVm = vmGet();
-
-        # Build the binary
-        #---------------------------------------------------------------------------------------------------------------------------
-        if ($bBinRequired || $bUnitRequired)
-        {
-            # Loop through VMs to do the C bin/integration builds
-            my $bLogDetail = $strLogLevel eq 'detail';
-            my @stryBuildVm = $strVm eq VM_ALL ? VM_LIST : ($strVm);
-
-            foreach my $strBuildVM (@stryBuildVm)
-            {
-                my $strBuildPath = "${strTestPath}/build/${strBuildVM}";
-                my $strBuildNinja = "${strBuildPath}/build.ninja";
-
-                foreach my $strBuildVM (@stryBuildVm)
-                {
-                    my $strBuildPath = "${strTestPath}/build/${strBuildVM}";
-                    my $strBuildNinja = "${strBuildPath}/build.ninja";
-
-                    &log(INFO, (!-e $strBuildNinja ? 'clean ' : '') . "build for ${strBuildVM} (${strBuildPath})");
-
-                    # Setup build if it does not exist. Unit tests are built from the repo copy by the test harness so there is
-                    # nothing to build here for them, but the build path is still mounted into their container so it must exist.
-                    my $strBuildCommand = $bBinRequired ? "ninja -C ${strBuildPath} src/pgbackrest 2>&1" : undef;
-
-                    if (!-e $strBuildNinja)
-                    {
-                        $strBuildCommand =
-                            "meson setup -Dwerror=true -Dfatal-errors=true -Dbuildtype=debug ${strBuildPath}" .
-                                " ${strBackRestBase}" . (defined($strBuildCommand) ? " && \\\n" . $strBuildCommand : '');
-                    }
-
-                    # Build code
-                    if (defined($strBuildCommand))
-                    {
-                        executeTest(
-                            ($strBuildVM ne VM_NONE ? 'docker exec -i -u ' . TEST_USER . " test-build bash -c '" : '') .
-                                $strBuildCommand . ($strBuildVM ne VM_NONE ? "'" : ''),
-                            {bShowOutputAsync => $bLogDetail});
-                    }
-                }
-            }
-        }
-
-        # Shut down the build vm
-        #---------------------------------------------------------------------------------------------------------------------------
-        if ($strVm ne VM_NONE)
-        {
-            executeTest("docker rm -f test-build");
-        }
-
-        # Exit if only testing builds
-        exit 0 if $bBuildOnly;
-    }
-
-    # Perform static source code analysis
-    #-------------------------------------------------------------------------------------------------------------------------------
-    if (!$bDryRun)
-    {
-        logFileSet($oStorageTest, cwd() . "/test");
-    }
-
-    # Run the tests
-    #-------------------------------------------------------------------------------------------------------------------------------
-    if (@{$oyTestRun} == 0)
-    {
-        confess &log(ERROR, 'no tests were selected');
-    }
-
-    &log(INFO, @{$oyTestRun} . ' test' . (@{$oyTestRun} > 1 ? 's': '') . " selected\n");
-
-    # Don't allow --no-cleanup when more than one test will run. How would the prior results be preserved?
-    if ($bNoCleanup && @{$oyTestRun} > 1)
-    {
-        confess &log(ERROR, '--no-cleanup is not valid when more than one test will run')
-    }
-
-    # Disable file logging for integration tests when there is more than one test since it will be overwritten
-    if (@{$oyTestRun} > 1)
-    {
-        $strLogLevelTestFile = lc(OFF);
-    }
-
-    # Don't allow --no-cleanup when more than one test will run. How would the prior results be preserved?
-
-    # Only use one vm for dry run so results are printed in order
-    if ($bDryRun)
-    {
-        $iVmMax = 1;
-    }
-
-    my $iTestIdx = 0;
-    my $iVmTotal;
-    my $iTestMax = @{$oyTestRun};
-    my $bShowOutputAsync = $bVmOut && (@{$oyTestRun} == 1 || $iVmMax == 1) && ! $bDryRun ? true : false;
-
-    do
-    {
-        do
-        {
-            $iVmTotal = 0;
-
-            for (my $iVmIdx = 0; $iVmIdx < $iVmMax; $iVmIdx++)
-            {
-                if (defined($$oyProcess[$iVmIdx]))
-                {
-                    my ($bDone, $bFail) = $$oyProcess[$iVmIdx]->end();
-
-                    if ($bDone)
-                    {
-                        if ($bFail)
-                        {
-                            if ($oyProcess->[$iVmIdx]->run())
-                            {
-                                $iTestRetry++;
-                                $iVmTotal++;
-                            }
-                            else
-                            {
-                                $iTestFail++;
-                                $$oyProcess[$iVmIdx] = undef;
-                            }
-                        }
-                        else
-                        {
-                            $$oyProcess[$iVmIdx] = undef;
-                        }
-                    }
-                    else
-                    {
-                        $iVmTotal++;
-                    }
-                }
-            }
-
-            # Only wait when all VMs are running or all tests have been assigned. Otherwise, there is something to do.
-            if ($iVmTotal == $iVmMax || $iTestIdx == @{$oyTestRun})
-            {
-                waitHiRes(.05);
-            }
-        }
-        while ($iVmTotal == $iVmMax);
-
-        for (my $iVmIdx = 0; $iVmIdx < $iVmMax; $iVmIdx++)
-        {
-            if (!defined($$oyProcess[$iVmIdx]) && $iTestIdx < @{$oyTestRun})
-            {
-                my $oJob = new pgBackRestTest::Common::JobTest(
-                    $oStorageTest, $strBackRestBase, $strTestPath, $$oyTestRun[$iTestIdx], $bDryRun, $bVmOut, $strPlatform,
-                    $strVmArch, $strImage, $iVmIdx, $iVmMax, $iTestIdx, $iTestMax, $strLogLevel, $strLogLevelTest,
-                    $strLogLevelTestFile, !$bNoLogTimestamp, $bShowOutputAsync, $bNoCleanup, $iRetry, !$bNoBackTrace, !$bNoValgrind,
-                    !$bNoCoverage, $bCoverageSummary, !$bNoOptimize, $bProfile, $iScale, $strTimeZone, !$bNoDebug, $bDebugTestTrace,
-                    $iBuildMax / $iVmMax < 1 ? 1 : int($iBuildMax / $iVmMax));
-                $iTestIdx++;
-
-                if ($oJob->run())
-                {
-                    $$oyProcess[$iVmIdx] = $oJob;
-                }
-
-                $iVmTotal++;
-            }
-        }
-    }
-    while ($iVmTotal > 0);
-
-    # Write out coverage info and test coverage
-    #-------------------------------------------------------------------------------------------------------------------------------
-    my $bUncoveredCodeModule = false;
-
-    if (vmCoverageC($strVm) && !$bNoCoverage && !$bDryRun && $iTestFail == 0 && @iyModuleTestRun == 0)
-    {
-        my $strModuleList;
-
-        foreach my $hTest (@{$oyTestRun})
-        {
-            $strModuleList .= ' ' . $hTest->{&TEST_MODULE} . '/' . $hTest->{&TEST_NAME};
-        }
-
-        my $oExec = new pgBackRestTest::Common::ExecuteTest(
-            "python3 ${strBackRestBase}/test/test.py coverage --log-level=warn --vm=${strVm}" .
-            " --repo-path=${strBackRestBase} --test-path=${strTestPath}" .
-            ($bCoverageSummary ? ' --coverage-summary' : '') . $strModuleList,
-            {bShowOutputAsync => true, bSuppressError => true});
-        $oExec->begin();
-        my $iResult = $oExec->end();
-
-        if ($iResult <= 1)
-        {
-            $bUncoveredCodeModule = $iResult == 1;
-
-            if (!$bUncoveredCodeModule)
-            {
-                &log(INFO, "tested modules have full coverage");
-            }
-
-        }
-        else
-        {
-            &log(ERROR, "TEST SUBCOMMAND RETURNED ERROR CODE $iResult");
-            exit $iResult;
-        }
-    }
-
-    # Print test info and exit
-    #-------------------------------------------------------------------------------------------------------------------------------
-    &log(INFO,
-        ($bDryRun ? 'DRY RUN COMPLETED' : 'TESTS COMPLETED') . ($iTestFail == 0 ? ' SUCCESSFULLY' .
-            (!$bUncoveredCodeModule ? '' : " WITH MODULE(S) MISSING COVERAGE") :
-        " WITH ${iTestFail} FAILURE(S)") . ($iTestRetry == 0 ? '' : ", ${iTestRetry} RETRY(IES)") .
-            ($bNoLogTimestamp ? '' : ' (' . (time() - $lStartTime) . 's)'));
-
-    exit 1 if ($iTestFail > 0 || ($bUncoveredCodeModule && !$bCoverageSummary));
-
-    exit 0;
-}
-
-####################################################################################################################################
-# Check for errors
-####################################################################################################################################
-or do
-{
-    # If a backrest exception then return the code
-    if (isException(\$EVAL_ERROR))
-    {
-        if ($EVAL_ERROR->code() != ERROR_OPTION_INVALID_VALUE)
-        {
-            syswrite(*STDOUT, $EVAL_ERROR->message() . "\n" . $EVAL_ERROR->trace());
-        }
-
-        exit $EVAL_ERROR->code();
-    }
-
-    # Else output the unhandled error
-    syswrite(*STDOUT, $EVAL_ERROR);
-    exit ERROR_UNHANDLED;
-};
-
-# It shouldn't be possible to get here
-&log(ASSERT, 'execution reached invalid location in ' . __FILE__ . ', line ' . __LINE__);
-exit ERROR_ASSERT;

test/test.py

@@ -1,11 +1,12 @@
 #!/usr/bin/env python3
 """pgBackRest Test Harness.
 
-Builds and runs the unit tests and reports coverage. Driven by test.pl, which selects the tests to run, manages the containers, and
-calls this harness for each test.
+Builds and runs the tests and reports coverage. A run with no command selects the tests, manages the containers, and calls this
+harness again with the unit command for each test, which is why a run works on the repository it is part of while every command is
+told where the repository copy is.
 
 Exit status: 0 = success, 1 = the coverage command found modules missing coverage, greater = error. All output, including errors,
-goes to stdout since the Perl test framework fails a test when anything is written to stderr."""
+goes to stdout since a test that writes anything to stderr has failed."""
 
 ####################################################################################################################################
 import os
@@ -14,8 +15,8 @@ import sys
 import time
 import traceback
 
-# Send everything written to stderr to stdout instead. The Perl test framework reports only what a test writes to stdout, so an
-# error raised before the handler in main() is installed, e.g. while importing below, would otherwise be lost entirely.
+# Send everything written to stderr to stdout instead. Anything a test writes to stderr is a failure, so an error raised before
+# the handler in main() is installed, e.g. while importing below, would otherwise look like one.
 sys.stderr = sys.stdout
 
 # Do not cache bytecode. The harness runs from a copy of the repository that the linter then scans, so a __pycache__ written during
@@ -30,10 +31,12 @@ from common.error import EXIT_ERROR, TestError  # noqa: E402
 # PyYAML is the only thing these imports need outside the standard library, so a failure here is almost always a missing package.
 # Coverage is also required but only by the test runner, which is a separate process and reports a missing one itself.
 try:
+    from command.code.format import cmd_code_format  # noqa: E402
     from command.coverage.coverage import cmd_coverage  # noqa: E402
     from command.lint.lint import cmd_lint  # noqa: E402
+    from command.test.test import cmd_test  # noqa: E402
     from command.test.unit import cmd_unit  # noqa: E402
-    from common.log import ERROR, INFO, log  # noqa: E402
+    from common.log import *  # noqa: E402
     from config.config import cfg_load, project_version  # noqa: E402
 except ImportError as error:
     print("unable to load the test harness: %s" % error)
@@ -47,10 +50,16 @@ except ImportError as error:
 def command_run(config):
     """Run the requested command."""
 
+    # Running the tests is what a developer does, so it is what happens when no command is given
+    if config.command is None:
+        return cmd_test(config)
+
     if config.command == "unit":
         cmd_unit(config)
     elif config.command == "coverage":
         return cmd_coverage(config)
+    elif config.command == "code-format":
+        cmd_code_format(config)
     else:
         cmd_lint(config.repo_path)
 
@@ -75,13 +84,16 @@ def main():
 
         return EXIT_ERROR
 
-    log(INFO, "%s command begin %s: %s" % (config.command, project_version(path_harness), " ".join(sys.argv[1:])))
+    # A run with no command runs the tests, which is what it is called in the log
+    command = "test" if config.command is None else config.command
+
+    log(INFO, "%s command begin %s: %s" % (command, project_version(path_harness), " ".join(sys.argv[1:])))
 
     try:
         result = command_run(config)
     except TestError as error:
         log(ERROR, error)
-        log(INFO, "%s command end: aborted with exception" % config.command)
+        log(INFO, "%s command end: aborted with exception" % command)
 
         return error.status
     except Exception:
@@ -90,7 +102,11 @@ def main():
 
         return EXIT_ERROR
 
-    log(INFO, "%s command end: completed successfully (%ums)" % (config.command, int((time.time() - time_begin) * 1000)))
+    # How long the command took is only reported when timestamps are, since the documentation runs the tests to generate output
+    # that must be the same every time
+    elapsed = " (%ums)" % int((time.time() - time_begin) * 1000) if config.log_timestamp else ""
+
+    log(INFO, "%s command end: completed successfully%s" % (command, elapsed))
 
     return result