The db-test list spreads the supported versions over the default vms so that a full run covers every version exactly once, but it also limited what --pg-version was able to select. That made it impossible to test a version on a vm that installs it but does not test it by default, e.g. --vm=u24 --pg-version=18 reported that no tests were selected even though u24 has PostgreSQL 18 installed.
Select from the versions the vm installs when a version is named and make --pg-version=all mean every version installed rather than an alias for the default selection. A version the project does not support is now an error rather than a selection of nothing, since that is a typo just as an unknown module name is.
A ctrl-c raised KeyboardInterrupt out of main() and python printed the traceback, which is the whole path from main() down to the select() the harness happened to be waiting in and says nothing about a run that was stopped on purpose. Catch it and log "terminated on signal SIGINT" as the C does, exiting with the status the C exits with for TermError.
An unexpected exception is a bug in the tool so the trace is what gets it fixed, but the trace ends with the message that says what went wrong, which is where the message gets lost. Show the message first and keep only the innermost frames, since the frames above them are the path from main() into the tool and are much the same for every error. The frame count says how many were dropped and how to get them, and the complete trace, which is what python printed before, is shown at debug level the same way the C shows a stack trace only at debug level.
The rendering is error_trace() in common/error.py, beside ToolError, so every tool reports a bug in itself the same way. The CI wrapper asks for the complete trace since a run there cannot be repeated at debug level to get it, and the code builder handles the interrupt but not the trace since it has no log level to raise.
The documentation defines the containers it runs its commands in, so it is built for a distribution rather than on a vm. Naming a vm meant mapping u24 to Debian/Ubuntu and rh9 to RHEL, a mapping that had no purpose except to be updated whenever those vms were retired.
Accept --distro=debian or --distro=rhel instead, which are the os types the documentation already generates commands for, with no default so omitting the option still builds every distribution. The CI wrapper and the doc workflow pass the distro through.
The definitions move out of common/vm.py into a vm section at the top of test/container.yaml, next to the container revisions that go with them, and are read and checked when the module is imported as they were before. The revisions move under a revision key to make room.
The declaration is read with the loader the other declarations use, so a version keeps the text it was written as rather than arriving as a float. A new yaml_map_dict() reads a mapping as a dict where a key is looked up by name, which also reports a key that was written twice instead of silently keeping the last one.
The constants that named each vm are gone. A vm is named by the string that --vm accepts, which is how the CI workflows and the documentation build already name them. A name used as a lookup is validated by vm_get(), and a name used in a comparison is protected by branch coverage since a comparison that can never match leaves a branch unhittable. VM_NONE, VM_ALL, and the os base and architecture constants remain because they name kinds rather than vms. VM_LIST is now the vms declared as default, so it is in declaration order.
The container build no longer tests a vm by name. What a vm installs and configures is declared instead: valgrind, epel, powertools, python, coverage-python, pg-repo-key, pg-repo-release, dnf-module, and ssh-rsa. The three near identical blocks that added the PostgreSQL repo collapse into one, which removes the uncoverable branch annotation on the fedora branch. Every generated script is unchanged, so no image is rebuilt.
The supported PostgreSQL versions are generated from build/postgres.yaml rather than declared a second time. A new bld_pg_version_list() reads just the versions, since a tool that only needs to know which ones are supported has no use for the interface that bld_pg_parse() scans the vendored headers for. The release flag comes with them, so the beta repository is added for whichever installed version has not been released rather than for a hardcoded vm and version.
A vm with no container declares only default and coverage-c, since every other key says how a container is built. The os base declared for none was never read, and the os the tests run on with --vm=none is whatever the host is rather than the debian it claimed. The PostgreSQL versions declared for it were never read either, since only an integration test runs against a version and those need a container.
A PostgreSQL version installed on a vm must now be one that build/postgres.yaml declares, and a version tested on a vm must be one installed there, both of which were only true by inspection before.
The containers are built by a command of its own rather than by an option on a test run, e.g. test.py vm-build --vm=u24. The code that builds them moves from command/test/container.py to command/vm/build.py.
The --vm-force option becomes --no-cache to match the option docker uses for the same thing. Since the point of it is to build without the cache rather than to clean up, it no longer removes the images that were built before, which leaves them in place to go back to.
The --vm-arch option keeps its name since it also applies to a test run.
The linter ran in the unit command, which runs once per test module, so a run scanned the entire repository again for every test in it. Move the call to the test command, which runs once, and lint the repository copy since that is what the tests are built and run from.
The lint command was confusing to run because it linted whatever repo path it was given rather than the copy the tests are built from. Replace it with --lint-only, which does what a run does and stops once the source has been linted.
The coverage command was only ever run by the test command, as a subprocess, so call it directly and pass it the test modules that ran. An error in coverage is now reported as itself rather than as a status the caller has to translate back into an error. The coverage tests move ahead of the test command test in define.yaml since it now depends on them.
The test command is now a real command, listed in the help and noted as the default, but a run still never has to name it. Its options remain on the main parser as well since running the tests with no command is by far the most common usage, so the help must show them without being asked for the command.
The unit command is left out of the command list since the test run is what calls it, in a container where only the repository copy is available, rather than something to run by hand.
The linter ran in the unit command, which runs once per test module, so a run scanned the entire repository again for every test in it. Move the call to the test command, which runs once, and lint the repository copy there since that is what the tests are built and run from.
Running the unit command directly no longer lints, but the test run does, which is what CI runs.
A test module that is not declared in define.yaml is never built or run and nothing reports it, since the file simply goes unread. This was the case for common/type/c, which is added to the definition here.
The name of the file a test module lives in moves to define.py so the linter and the build cannot disagree about it.
The tests that cover this code require a container for sudo access, e.g. to add entries to /etc/hosts, so they are only built when TEST_CONTAINER_REQUIRED is defined. Without the vm_covered exception the common/io/tls test reports coverage errors when run outside a container.
Wrap a few statements to make room for the annotation, which requires braces where a single statement was sufficient before.
The --module option now takes a test module or the path of a group of them, e.g. --module=common/type/string runs the string test while --module=common/type runs every test under it. The --test option that named a test within a module is gone, since the module name now says which test it is. A name that matches no module is an error rather than a selection of nothing, so a typo in one of several --module options is reported instead of quietly running the rest.
The option that selects a single test within a module is --test rather than --run, e.g. --module=command/backup --test=2, since --run has been a source of confusion. It requires a single --module naming a module, since the path of a group of them has nothing to apply it to. The test harness labels its output "test N" to match.
C test modules move into subdirectories that match the code they cover, e.g. common/typeStringTest.c becomes common/type/stringTest.c, so that a path selects a group that makes sense.
The test definitions are a flat list of modules per type, each named for the path of its test module, rather than tests nested under a module group. An entry that sets only options and defines no test is not a module but the options that every module under its path inherits, and it must be declared before them.
The python tests for the build, documentation, and test tools move to a new tool type at the end of the definitions, where most development never has to look at them. The lang keyword goes away since a tool is python and everything else is C, and vm is now valid only for a tool since the tools are the same everywhere while a C test is exercising the platform it runs on.
A run that ends with a module missing coverage names the module but not where to see what is missing. The report path is now logged with a file:// prefix so it can be pasted into a browser.
Apply the dnf mirror failover tuning from dd2e0ce8 to the test containers and the dist test, which pull from the same mirrors and see the same throughput variance. Raise the rate floor at which a mirror is abandoned, shorten the timeout, sort mirrors by latency, and fetch in parallel.
The repository disable that accompanied that change is not carried over. It needs an enable list that tracks what each container installs, which is noisier than it is worth here, and the failover tuning is what addresses the stalls.
The option names are the same for dnf 4 and dnf 5, so one block covers Rocky 8, 9, and 10 as well as Fedora. Reading the effective configuration back in each image confirms all five options are applied.
The container revision is left alone since the cache key hashes the generated Dockerfile and these edits invalidate it on their own.
The check ran on 22.04 for uncrustify 0.72, which meant a development container on 24.04 had to build 0.72 from source to get the same result. It now runs on 24.04 against the packaged uncrustify 0.78, so the container can install it.
Uncrustify 0.78 maps .inc to Pawn, which fails to parse the .c.inc files and aborts the run, so file_ext maps the extension back to C.
Comments in the body of a multiline #define were reindented. pp_multiline_define_body_indent arrived in 0.76 with a default of 8 and is documented as requiring pp_ignore_define_body=false, but 0.78 applies it to comments in the body whichever way that option is set, so the comments moved out to eight columns and pushed the line continuations after them past 132 characters. Setting it to indent_columns leaves them where they are.
A case label generated by a macro was indented at the level of the switch braces rather than the level of the other case labels, which is what indent_switch_body sets and what 0.73 added it for.
Six options changed from true/false to a number. The numeric form means what the boolean did and every version accepts it, so the formatting is unchanged, but 0.78 reported an error for each line and fell back to the default.
A ternary in a switch expression makes 0.78 tag the colon of the first case label as the colon of the ternary, so it wrote case cfgCmdAnnotate : and indented the statement below it. No option avoids this, so main.c assigns the ternary to a variable and switches on that.
Uncrustify rejects an option it does not know and exits, so the two options added here mean the configuration requires 0.78. This is noted in the coding standards.
Black is not affected. Both the 21.12b0 packaged with 22.04 and the 24.2.0 packaged with 24.04 report every Python file unchanged, so the job moves without reformatting anything.
Apart from main.c no source file changes, so what 0.78 produces is the formatting the project already had, and running the format again after the check produces nothing further.
doc.pl and release.pl become doc.py and release.py and the Perl library in doc/lib/pgBackRestDoc is gone, which is 32 files and 11,059 lines replaced by about 3,850 lines of Python. Nothing in the project runs on Perl now.
doc.pl and doc.py merge into one tool rather than one running the other. doc.py built the reference documents and the preprocessed user guide, wrote them to doc/output/xml, and doc.pl read them back; they are now passed in memory and those files are not written at all. That also removes a duplication the split had hidden: the preprocessor added a document's variables to its own store and left the declarations in the file, so the Perl parsed them again into a store of its own and the same variable existed twice with different values.
Most of Doc.pm does not come across. Its parse step rebuilt the parsed tree into a structure of its own and its build step invented a second way to address that structure, so a document was reached through three accessors where the tree has one. What survives is about ninety lines in common/xml.py: a field is the text of a child that holds nothing else, the text of a node is the node itself for the tags that are text and a text child for the rest, and asking for a child rather than the list of them now reports a second one as the mistake in the document that it is.
External entities are expanded before the parser sees a document, since the standard library does not fetch them. The release list is assembled from one file per release this way, 122 of them, and a declaration that is never referenced is reported rather than leaving the file it names silently unread.
Nothing validated against a document type, so the document type declarations and the two dtd files are removed. A declaration that is not enforced does not get maintained. The two documents that assemble themselves from parts keep the declaration because the parts are declared inside it.
The release notes become a document the tool builds, like the command and configuration references, rather than something the renderer reaches into a custom module to produce.
A variable may still be evaluated rather than written out, but it is a Python expression against a short list of names rather than a Perl program. The three that ask for a fact about the environment keep doing so. The four that read the project version, the current stable version, and the release date are facts the tool computes, since working them out involved constructing library objects and reading the release list a second time from inside a variable body.
exec and user move down from test/lib to doc/lib because the documentation needs both, the project version splits out into config/project, and eval and var_store move to common now that the manifest uses them as much as the preprocessor. The version is built from the parts in src/version.h rather than read from the string generated from them, which is what the Perl did and what a release needs, since a release clears the development suffix and the string is only regenerated by a test run.
libxml-checker-perl is no longer installed on CI runners, in the development container, or in the documentation images, and the RHEL image no longer installs cpanminus and a toolchain to build XML::Checker::Parser from CPAN. The user guide never ran Perl, and the Debian image never had a parser and built the same guide.
Every module in doc/lib is declared in define.yaml and has full line and branch coverage.
Ubuntu 24.04 ships a user and a group at id 1000, so a build by a user with that id could not create the test user. The test image removes whatever holds the id before creating it, as it already did for the group, and the development container image removes both because adduser gives a new group the id of the user.
The coverage summary listed the python under build/lib as modules with mangled names, since it stripped a fixed src/ prefix from paths that no longer start with one. It now summarizes what is under src, which is what the metric page says it reports.
Deploying the website no longer removes the links at the top of the site or CNAME. The Perl used find -type f, which does not match a symbolic link, and the port used a test that follows one, which took out the guides kept under retired platform names. CNAME is a regular file and the Perl removed it every deploy, which was never intended.
The rendered documentation is byte identical to what the Perl produced, for every page, the manual page, and the generated markdown. The user guide was also built end to end under Docker on both documented platforms, and rendering it again from the cache that build wrote produces the same pages, which exercises the whole path rather than only what can be checked without containers. A release was then simulated from beginning to end.
The build tool in src/build becomes build/build.py with its library in build/lib, and the C part of the documentation tool in doc/src becomes doc/doc.py with its library in doc/lib. They migrate together because doc/src was built on src/build, linking its config and help parsing along with its extensions to the core types. doc.pl runs doc.py rather than building and running a C binary, and meson runs build.py for the code it generates as it builds.
The three libraries form the hierarchy the C include path did: build/lib is the base, doc/lib sits on it, and test/lib sits on both. They are implicit namespace packages so a package spans them, and a module lives in exactly one library and holds the union of what the tools need from it, since Python resolves a module name to a single file and cannot extend one the way the C included a core module and added to it. define.yaml declares which libraries each tool may import and the test runner puts only those on the path, so a test of the build library cannot reach the doc or test library any more than the build tool can.
The declarations move with the code. config.yaml, error.yaml, and postgres.yaml are now at the root of build, and help.xml becomes doc/xml/reference.xml since it is documentation that the help happens to be generated from, so the generator is told where it is rather than knowing. dist.sh moves to build as well. The documentation tool's own option declarations are gone because they existed only to generate the option parser for the C binary and doc.py parses its command line directly.
Three modules in src/build/common were never build tool code. exec, regExp, and string extend the core types for the test harness and the unit tests, which were their only users, so they move to test/src/harness. The json and xml extensions had no users outside the two tools and are removed with them, and the tests for both are trimmed to the core functions that remain.
The help unit test built its fixture by calling the generator's C functions directly, which was the last thing outside the tools that depended on them. It now runs build.py help-data to write the packed help and loads that, so it still runs against the current declarations without needing the generated C, which may not have been written when the test runs.
The manual page wrapping is a port of the wrapper the binary uses for its own help, so the manual page and pgbackrest help break lines the same way, including where a word longer than a line is left unbroken.
libyaml is no longer needed to build pgBackRest since the only thing that read yaml was the C generator. It is dropped from the development container and from every CI platform, and meson no longer looks for it. Code generation needs nothing compiled first either, so a test run no longer builds a generator before it can select the tests.
The common source list is merged into the pgbackrest source list since it existed only so the generator could be built from the same files, and the entries that were out of alphabetical order are put in order while the list is being rewritten anyway.
Every file the generator writes is byte identical to what the C wrote, for the project declarations and for the documentation's own. The documentation is checked the same way: the HTML rendered on this branch is byte identical to the HTML rendered on integration, every file of it, which exercises the whole pipeline on the real content rather than only the four files the tool writes.
Every module in both libraries is declared in define.yaml and has full line and branch coverage. The coverage list is keyed by file rather than by module name, since the same name in two languages is two files and one of them would have silently lost its requirement while both existed.
TestError becomes ToolError, since it is the error type for three tools now rather than for the harness alone.
test.pl becomes test.py and ci.pl becomes ci.py, so an invocation only changes from .pl to .py. Running the tests needs no command since that is what a developer does, and the unit and coverage commands that a run drives are commands of test.py because a test runs inside a container where only the repository copy is available.
The Perl library in test/lib is gone. The modules the documentation depends on move to doc/lib, with ExecuteTest becoming Execute, and are trimmed to what the documentation calls, which removes the storage manifest, info, and owner along with everything else the tests were the only user of. release.pl no longer needs the vm definitions since all it did with them was compare two vm names.
Nothing in a test container or on a CI runner runs on Perl now so none of it is installed in either place, which was a substantial part of a base image. The packaging tools go with it since they are written in Perl and nothing builds a package, and ca-certificates is installed explicitly because it was arriving as a dependency of a package that is no longer there. Every base image is rebuilt as a result.
The generated Dockerfile is otherwise unchanged. The one difference is that the PostgreSQL repository for RHEL is installed for the architecture being built rather than for the architecture of the host, which only differs when cross building.
The list of tests to run is unchanged. It was compared against the Perl across the vm, module, test, run, and PostgreSQL version options before the Perl was removed.
The vm definitions, the user lookup, the test list, the container build, the job, the test run itself, and the code format are covered by tests, which the Perl equivalents never were. The container build is checked by comparing the generated script in full, since the hash of that script is the cache key.
Make the code format a command rather than two options of a test run, and fold in the black check that CI has been running as a step of its own. It gets the files to format from git rather than from the repository copy, so it no longer depends on a test run having made that copy and is told nothing about where either path is.
Wire up --log-level-test, which the driver has been collecting and discarding. Remove --psql-bin, --gen-check, --no-coverage-report, --build-max, --log-level-test-file, --no-optimize, --no-debug, and --debug-test-trace, none of which have done anything for some time.
Allow --vm=all only for --vm-build. Building every container at once works but running the tests on every vm never has, since a test needs the binary and the build for the vm it runs on and there is one of each.
The C harness in test/src/command/test is replaced by test/test.py with its library in test/lib, and test.pl runs that per test rather than building and running a C binary. Python is already required by meson so the only new dependency is PyYAML, and the harness no longer needs to be compiled before any test can run.
The harness is now tested the same way the rest of the code is. Test modules live in test/src/module/test and are declared in define.yaml with lang: python, so they run through the existing test.pl --module and --test interface. A python test is a plain function rather than a run inside testBegin(), since these tests are fast enough that selecting a single run is not useful, and --test-name selects one function when debugging.
Coverage from both languages merges into a single report, so a run that covered C and python produces one result. Full line and branch coverage is required and the uncovered and uncoverable annotations work as they do for C. Python reports the lines and arcs it knows about rather than everything a module contains, and which arcs a run reports depends on which ones it took, so python coverage merges by line number and by where an arc goes rather than by position.
A python test may import only the library modules it declares, which is what the C link step enforced by compiling in only what was declared. An import that is already in sys.modules never reaches an import hook, so the test module is loaded in its own interpreter with the hook installed ahead of the library.
Fix coverage for a shimmed .c.inc module, which has been missing from the report rather than reported as uncovered. The build writes the shimmed copy to the include path so it resolves ahead of the repository copy, which puts it outside src, and the filter only matched the path such a module is normally compiled from. In practice this is command/backup/process.inc, the only shimmed .c.inc.
Install PyYAML wherever the harness runs and a coverage module wherever the python tests run. rh8 installs python3.12 since PyYAML is not packaged for the platform python there, u24 and rh10 package a coverage new enough to report branch detail per line, and rh9 installs one with pip since the packaged version predates it. The jobs that run outside a container install PyYAML as each platform allows, with pip on macOS since homebrew has no formula for it.
Check the format of the harness and its tests with black, which previously covered only the smoke test.
The RHEL documentation builds have been slow and intermittently failing. The mirrors are not down: three consecutive full metadata refreshes from the same host measured 73, 124, and 157 seconds with a full mirror list and no dead mirrors, so the variance is mirror throughput rather than availability. Every documentation container pays this cost because Rocky sets metadata_expire to six hours and the cached base image is normally older than that by the time it is used.
Tune dnf to fail over from a slow mirror. By default a mirror is only abandoned when it delivers less than 1000 bytes per second for 30 seconds, so a mirror that is crawling rather than failing is retried instead of dropped. Raise the rate floor to 50k, shorten the timeout, sort mirrors by latency, and fetch in parallel.
Disable repositories the documentation does not use so their metadata is not refreshed in every container. The epel-release package adds an openh264 repository and the PGDG repository enables one repository per PostgreSQL version, of which only the documented version and the upgrade version are needed. This takes the enabled repository count from twelve to eight. The enable list is driven from the existing version variables so it follows a version bump.
After both changes the same three-run measurement was 81, 53, and 51 seconds, though the mirror variance being addressed also makes that comparison noisy.
The container revision is left alone since the cache key hashes the generated Dockerfile and these edits invalidate it on their own.
Add a u24 vm and split the PostgreSQL versions between it and u22 so the integration tests run as two jobs instead of one. u22 tests 9.6 and 11 and no longer installs anything above 14, while u24 installs the full range and tests 12, 13, and 19. The unit test jobs (coverage/backtrace and valgrind), the coverage summary generated for a release, and the Debian/Ubuntu documentation build all move to u24, so the PostgreSQL 19 beta repository is now enabled there rather than on u22. The manual container revision for the beta is dropped since the u24 base image is new and has no cached tag to invalidate.
Build the contributing guide development environment and the user guide host from ubuntu:24.04 rather than ubuntu:22.04.
Remove the guard that skipped PostgreSQL 18 unless the container was being built for a particular vm and architecture. It dates from when 18 was in beta and not built for ppc64le and s390x, and those architectures no longer reach it since they install a single version via the postgresql metapackage.
Drop python3-distutils, which no longer exists on Ubuntu 24.04. Nothing here uses distutils and on the older distributions it is pulled in as a meson dependency anyway.
Install the AWS CLI from the vendor installer on the Debian/Ubuntu documentation host since awscli is no longer packaged for Ubuntu 24.04. This matches what the RHEL host already does with awscli2 and means ca_bundle must now be set for both, since the vendored CLI ships its own certificate bundle instead of using the system store.
Leave the arch workflow and the code format check on u22 and ubuntu-22.04. Ubuntu 22.04 is still supported and uncrustify formats differently on 24.04, which is a separate change.
Banner comments become docstrings with a separator before each top level definition and each method, matching the rest of Python and making a description reachable by an editor rather than only by reading the file.
Two of the banners introduced a group of helpers rather than the function that followed, so those titles are dropped and each helper describes itself. A few functions had no description at all and were given one.
YAML mappings are unordered, but shim key order determined the order the modules were included in the harness, e.g. the fd harness includes common/io/fd before common/io/fdWrite. A sequence makes that order explicit instead of relying on parser behavior.
A shim entry is now either a bare module name or a map with a name and a function list, matching how harness entries are already written, and the null placeholder the map form needed is gone.
The harness integration flag is also applied to its shims after the harness has been parsed, so it no longer depends on integration appearing before shim.
The restore file module is only used internally by the restore protocol handler so move it into a file.c.inc that is included directly into protocol.c, matching the approach used for the other restore modules. This makes restoreFile() static and avoids maintaining an extra header and build entry.
Move the RestoreResult enum to protocol.h since it is still needed by process.c.inc and the restore tests but file.h no longer exists to provide it.
backupFile() is only called by the backup protocol handler so include file.c into protocol.c directly, matching how the rest of the backup module is included into backup.c. This makes backupFile() static and removes the file.h header and build entry. The BackupFile/BackupFileResult structs are only used by backupFile() and the protocol handler so they become file-private as well.
The BackupCopyResult enum is still needed by the backup process to interpret protocol job results so move it to protocol.h.
The backup module has grown large so split it into .c.inc files that are included directly into backup.c, matching the approach used for the restore module. These functions are only used internally by backup so this avoids maintaining extra header and build files and keeps them static.
Since backupProcess() is shimmed for testing and now lives in an included module, the shim uses the inc option to locate and shim it there.
The shim mechanism renames a function in the shim module's C file so the harness can supply an alternate implementation for testing, which required the shimmed function to be defined in the shim module itself. Now that large modules are being split into included .c.inc files a function to be shimmed may be defined outside the shim module.
Add an inc option to a shim function that names the included module where the function is defined. That module is shimmed in addition to the shim module and written to the include path so the shim module finds the shimmed copy (via the unit include directory) before the repo copy.
The Ceph RGW host was slow and unreliable to start, frequently returning connection failures and requiring long startup waits during documentation builds and integration tests.
SeaweedFS starts in a few seconds, ships a much smaller multi-arch image, and supports S3 object versioning, which the repository target time feature requires. The image is pinned to 4.40 to include recent versioned-listing fixes.
A single weed server command runs the master, volume, filer, and S3 gateway with TLS on 443, host-style addressing via s3.domainName, and the demo credentials in a mounted identity config, so no bootstrap script is needed. The entrypoint is overridden to run weed as root because the default drops to an unprivileged user that cannot bind 443.
After a stanza delete on the versioned repository, SeaweedFS keeps listing the now-empty stanza prefix, so info reports missing stanza data rather than missing stanza path; the user guide output is updated to match.
The documentation build builds a base host image per distribution before running the commands the docs contain, installing PostgreSQL, the Azure and AWS CLIs, and other packages from scratch on every run. On a fresh CI runner nothing is cached, so this rebuild made the doc builds the slowest jobs in CI.
Cache each base image in a new ghcr.io/pgbackrest/doc registry package, keyed by a hash of the generated Dockerfile combined with a manual revision on the host-define. A run pulls the image when the key matches and only builds and pushes it on a miss, the same approach the test containers already use; forks pull from the pgbackrest cache and push to their own.
The revision is set date-style (e.g. 20260721A) like the test container revisions in test/container.yaml and forces a rebuild when the Dockerfile is unchanged but the upstream packages have changed, e.g. a new PostgreSQL minor.
Factor the cache logic into pgBackRestDoc::Common::DocContainer to keep doc.pl tidy, grant the doc CI job packages:write and log it in to GHCR so a build can be pushed, and prune the doc package on the same schedule as the other caches.
During restore, files bundled together in the repository are read back one file or range at a time, and each new read from an object store pays the latency of establishing a request. The restore copy loop worked around this by manually coalescing adjacent bundled files into a single read, but it could neither hide request latency nor skip over unneeded data within a bundle.
Add StorageReadMulti, which is given the full list of files and ranges up front and reads them back in order. Requests are submitted ahead of time (prefetch) so the latency of the next request overlaps reading the current one, and ranges separated by a small gap (e.g. a file that already exists on the host) are merged into a single request by skipping over the gap ("read over") rather than closing and reopening at a new offset, which is faster and cheaper than a new request for most storage classes.
To support prefetch, the S3, Azure, and GCS read drivers gain an async open. The HTTP request is submitted when the read is opened and the response is deferred until the first read via a new optional openAsync driver function, so multiple requests can be in flight at once. storageReadNew() gains an async flag; the async open is completed lazily on first read and is never combined with ignore missing.
The default StorageReadMulti driver queues reads directly on the storage. When the repository is remote a dedicated remote driver is used instead so the entire request list is forwarded to the remote in a single open and prefetch and read over run there, closer to the storage. The remote read and remote read-multi share stream state and chunk handling.
Two options control the behavior: repo-storage-prefetch (default 3, range 0-1024) sets how many requests to queue ahead, and repo-storage-read-over (default 64KiB, range 0B-1MiB) sets the largest gap to read over rather than issue a new request. The number of open requests per read is one greater than prefetch since the initial read cannot be prefetched.
Rewrite the restore copy loop to add each file that will be copied to a StorageReadMulti and read through it, removing the manual offset and limit tracking across bundled files.
Only same-page section links were verified, so a page/section link with a bad section or a missing leading slash would silently render a broken anchor.
Require a link section to begin with a slash and, after all pages have been rendered, verify that each page/section link points at a section that exists on the target page. Links to a page that is not part of the build (e.g. a partial build) are skipped.
Also fix the news link to the 2.59.0 release notes, which was missing the leading slash and pointed at a broken anchor.
The distribution README (DIST.md) duplicated the Debian build dependency list from the user guide, so the two could drift. Move it into a shared variable used by both, drop python3-distutils, and link DIST.md to the user guide build section.
The variable is split with a backslash continuation so each line stays within the command line length limit. The doc executor now keeps an existing continuation rather than adding its own.
doc/DIST.md, the README shipped in the distribution tarball, was maintained by hand. Source it from doc/xml/distribution.xml and render it with the doc tool as is done for the other generated markdown, and gitignore the output like NEWS.md.
release.pl --dist renders DIST.md so meson dist can copy it into the tarball as README.md.
List items were separated by a blank line, which reads poorly when the markdown is viewed as plain text.
Regenerate README.md, CODING.md, and CONTRIBUTING.md.
The paragraph following a code block was rendered directly against the closing fence, which reads poorly when the markdown is viewed as plain text.
Regenerate CODING.md and CONTRIBUTING.md.
This command occasionally failed even though the preceding bucket creation had succeeded, presumably because the new bucket was not yet available. Retry for up to 30 seconds to give it time to appear.
Also lower the bucket creation retry from 60 to 30 seconds since it normally completes well within that.
Meson 0.55 and later name the unit test private build directory test-unit.p rather than test-unit@exe, so the harness no longer needs to check for both when clearing old coverage data.
All supported platforms ship a newer meson (RHEL 8, the oldest, has 0.58.2), so raise the minimum to 0.55.3 and always use test-unit.p.
Debian 11 (bullseye) reaches end of life before the next release, so replace it with Debian 12 (bookworm) in the distribution build matrix.
Debian 12 ships a newer meson that configures the Python installation without setuptools, so remove the debian:11-only setuptools workaround.
MEM_CONTEXT_PRIOR_END() and MEM_CONTEXT_OBJ_END() are both defined as MEM_CONTEXT_END(), so closing a MEM_CONTEXT_PRIOR_BEGIN() or MEM_CONTEXT_OBJ_BEGIN() block with the base MEM_CONTEXT_END() expands identically and cannot be detected by the compiler. Eight sites had drifted to the base closer: four in info.c, one in io/session.c, and one each in the s3, azure, and gcs storage drivers.
The headers document the matching END as the intended pairing, so use it. Preprocessed output was verified byte-identical before and after, so the change is a no-op.
These were found while prototyping a lint check for block-macro pairing.
The test and test-ccache packages grow without bound: test gains a new tag every time a base image Dockerfile changes, and test-ccache orphans the previous digest every time a per-architecture tag is overwritten. Both are pure caches that are rebuilt on demand, so add a weekly workflow to prune the stale entries, with a dry-run input for manual runs.
The test package keys base images by a hash of the Dockerfile, so a change abandons the old tag while leaving it tagged and out of reach of an untagged-only sweep. Age these out by deleting every tag older than two months; anything still in use is simply rebuilt and re-pushed on the next run. The test-ccache package overwrites its per-architecture tags on every save, leaving the previous digest untagged and referenced by nothing, so delete all untagged versions there.
The owner defaults to the account the workflow runs as, so a fork prunes its own caches rather than the canonical pgbackrest ones, mirroring where each environment pushes.
NOTE TO PACKAGERS: A new distribution tarball is available which simplifies the build process by providing pregenerated HTML documentation, man page, and code. The tarball is attached to each release as an asset and is named pgbackrest-{version}.tar.gz. See the README.md in the tarball for details. Please use the new distribution tarball to avoid the additional build tooling required to generate code in future releases.
NOTE TO PACKAGERS: There is a new optional dependency on libsystemd.
IMPORTANT NOTE: Starting with this release, only the restore command may be run as root by default. Use allow-root to run other commands as root (though this is not recommended).
Bug Fixes:
* Fix expire removing a backup in progress on another host. (Reviewed by Douglas J Hunley. Reported by Dmitrii.)
* Fix archive-push-queue-max not enforced when a WAL file errors. (Reviewed by Stefan Fercot. Reported by vut12.)
* Fix async archive-get failure during recovery across a timeline switch. (Reviewed by Douglas J Hunley. Reported by Jimmy Yih.)
* Fix potential buffer overrun in error module. (Fixed by Christophe Pettus. Reviewed by David Steele.)
* Fix use-after-free of parallel protocol client. (Reviewed by Douglas J Hunley. Reported by Georgy Shelkovy.)
* Fix heap overflow when a pg option is set without pg1. (Reviewed by Douglas J Hunley. Reported by Naveen.)
Features:
* PostgreSQL 19beta2 support. (Reviewed by Stefan Fercot.)
* Add archive-expire-before option to clean up WAL archive. (Contributed by Stefan Fercot. Reviewed by David Steele.)
* Add batch delete for Azure storage. (Reviewed by Douglas J Hunley, Crispy.)
* Add support for S3 Outposts. (Contributed by Shiva Kumar Ambigi. Reviewed by David Steele, Roberto Mello.)
* Add S3 process authentication. (Reviewed by Andrew Charlton. Suggested by Andrew Charlton.)
Improvements:
* Reconnect SFTP storage after the server drops an idle connection. (Reviewed by Stefan Fercot. Suggested by mustafa0x, tisserat.)
* Add user/group caching for faster manifest build. (Contributed by Gunnar Lindholm. Reviewed by David Steele.)
* Add per-repo backup progress to info command output. (Contributed by Will Morland. Reviewed by David Steele, Stefan Fercot.)
* Add backup.info checks to verify command. (Contributed by Denis Garsh. Reviewed by David Steele, Douglas J Hunley.)
* Allow the S3 STS endpoint to be configured. (Contributed by Simon Gratton. Reviewed by David Steele.)
* Add archive-push-batch-size to limit WAL pushed per asynchronous run. (Reviewed by Stefan Fercot.)
* Exit async archive-push on first error. (Reviewed by Stefan Fercot. Suggested by Lardière Sébastien.)
* Harden HTTP chunked response parsing. (Contributed by Shubham. Reviewed by David Steele.)
* Error when running as root unless allow-root is enabled. (Reviewed by Douglas J Hunley.)
* Fewer binary searches when building manifest. (Contributed by Gunnar Lindholm. Reviewed by David Steele.)
* Improve seek performance during block incremental delta restore. (Reviewed by David Christensen, Douglas J Hunley.)
* Bundle backup files in order rather than scanning for a better fit. (Reviewed by Stefan Fercot.)
* Report the underlying error when a query fails to complete. (Reviewed by Andrew Pogrebnoi. Suggested by Marco Fontana.)
* Report the actual error when an S3 credential request fails. (Reviewed by Douglas J Hunley. Suggested by Mitchell Grice.)
* Report archive-push spool path errors to the PostgreSQL log. (Reviewed by Douglas J Hunley. Suggested by Don Seiler.)
* Hint that pgBackRest may be out of date when a version is unsupported.
* Add systemd notify integration. (Contributed by Andrew Jackson. Reviewed by David Steele.)
* Suppress unused parameter errors in meson compiler probes. (Contributed by Jörg Plate. Reviewed by David Steele.)
* C11 is now the minimum C standard. (Reviewed by Mohammad Ali Nazir Kosar.)
Documentation Improvements:
* Make stanza option internal for the repo-* commands. (Reviewed by Stefan Fercot.)
* Document that info reads encryption settings from global section. (Reviewed by Stefan Fercot. Suggested by Ron Johnson.)
* Document that expire-auto uses the backup command configuration. (Reviewed by Stefan Fercot. Suggested by Lardière Sébastien.)
* Document adjusting the logrotate su directive for a dedicated user. (Reviewed by Douglas J Hunley. Suggested by lkanbus.)
* Document that end-of-line comments are not supported in config files. (Reviewed by Douglas J Hunley. Suggested by Alex Richman.)
Test Suite Improvements:
* Fix Alpine group conflicts in CI containers. (Contributed by Artur Zakirov. Reviewed by David Steele.)
A markdown link that specified both a page and a section failed to render when the target page was itself rendered as markdown, because markdown cannot reliably anchor into another generated file.
Resolve these links to the website URL with the section anchor, matching how links to pages that are not rendered as markdown already behave, so a markdown document can link to a section of another.
The site now serves over HTTPS at the apex domain, so switch from http://www.pgbackrest.org to https://pgbackrest.org.
Change the backrest-url-base variable that generates the documentation links and the URL hardcoded in the man page SEE ALSO section, then regenerate README.md and CONTRIBUTING.md and update the doc build test to match.
NEWS.md is the release announcement copied to a postgresql.org news item. It was maintained by hand for each release, but that content now lives in news.xml and is rendered to NEWS.md by the documentation build.
Remove the committed file and add it to .gitignore so it is treated as generated output and regenerated when an announcement is needed.
The distribution tarball ships the generated source and the rendered documentation so it builds without the code generation or documentation tooling that a git checkout requires. Add a workflow that exercises this contract: build the tarball, then on a range of distributions build and smoke test it in a fresh container that has only the required packages, which is both a test and a record of what a packager needs on each.
The tarball is built once, since it is distribution-independent by design, and uploaded as an artifact; a matrix job then consumes it per distribution and drives the built binary through a full backup, restore, and verify cycle against that distribution's native PostgreSQL. The containers are run with docker directly rather than as job containers because the runner's glibc Node.js cannot execute the checkout and artifact actions inside a musl container, which keeps one path for glibc and musl alike.
Debian, Ubuntu, Rocky, Fedora, and Alpine are covered, and the tarball is built with -Dwerror so a compiler warning on any of them fails the build. A separate check confirms the tarball ships exactly the expected generated and rendered files and that none of them is empty, catching a broken code generation or documentation render before it is uploaded.
Also remove the smoke job from the misc workflow, since building and smoke testing the tarball now covers the check a packager would run, and move its format check for smoke.py, the only Python file, into the code-format job with the other formatting checks.
The bsd test group cannot run the integration tests because there are no containers available, but the smoke test drives a backup/restore cycle against each installed PostgreSQL and so provides basic coverage on these platforms. Teach the smoke test the Homebrew and FreeBSD bin paths so it can find the installations there.
Discover the PostgreSQL versions to test at install time rather than naming them so the tests do not break when a version becomes unavailable. Read the supported versions from the generated header, intersect them with what the platform offers, and test the oldest and newest of the result.
On MacOS only keg-only formulae that are neither deprecated nor disabled are considered since the binaries in a formula that is not keg-only load libpq from the linked prefix and fail unless the formula is linked.
On FreeBSD the ports all install into the same prefix and conflict, so only one version can be installed at a time and the oldest goes on the older OS while the newest goes on the newer OS. Install the versioned client and server packages instead of postgresql-libpqxx, which depends on the default PostgreSQL version.
Now that a persistent ccache has made compiling under emulation affordable, broaden the tests run there to better check that checksums are correct on big-endian. Add the command restore, archive-get, archive-push, and verify tests alongside backup, and the common type-pack, crypto, and compress tests, which exercise the serialization, checksum, and compression code that a byte-order bug would break.
Also add a smoke test that drives a real PostgreSQL through a full backup, restore, and verify cycle, checking checksums end-to-end rather than against synthetic harness data. This needs a PostgreSQL server, so install a single version in the base image for these architectures; the full version matrix is skipped to keep the emulated build fast.
Consolidate the per-vm and per-vm-index ccache directories into a single cache shared by the host and every container, passed with CCACHE_DIR instead of mounted at ~/.ccache. A file compiled in one place is then a hit in the others, e.g. test-pgbackrest is built both on the host and in the build container, and since ccache is safe for concurrent access there is no need to split the cache. It stays in the test path so --clean removes it.
Store and restore that cache in GHCR via a new composite action. Unlike actions/cache a registry is not branch-scoped, so a short-lived branch starts warm from the last run on the fork rather than cold, which matters most for the emulated architecture jobs where compiling is slow. The action pulls the fork's own cache first and falls back to the canonical pgbackrest cache for a fork that has not run yet; the push is best effort so pull requests from forks that cannot write just use what they built.
The smoke test now reads the ELF header of the pgbackrest binary being tested and prints its machine architecture and endianness, e.g. "s390x (64-bit big-endian)", right after the user it runs as. This makes it easy to confirm which architecture is actually being exercised, which matters when a binary built for a foreign architecture is run under emulation.
test.yml had grown large enough to be awkward to navigate, but splitting it into independent workflows would have produced a separate run in the Actions UI for each file and required the push/pull_request trigger to be duplicated in every one.
Instead test.yml becomes a dispatcher that calls reusable workflows, so it remains a readable index of what runs while still producing a single run. Only test.yml has a trigger and the called workflows are all on: workflow_call. Permissions are granted in the dispatcher since a called workflow can only maintain or reduce the permissions of the token it is given, never elevate them.
The jobs are grouped by platform into linux, arch, and bsd, with doc for the documentation builds and misc for the checks that do not fit any of them.
The documentation builds are worth separating because they have little in common with the tests they were mixed in with. They do not use the test containers, so they need no GHCR login or packages: write permission to maintain the container cache, and they generate no coverage, so there is no report to upload on failure. The build-max parameter was already ignored for doc since ci.pl only passes parameters through on the test path, so it is not carried over.
Job names gain the calling job as a prefix, e.g. linux / test, which is the cost of getting a single run.
Also move symbol.out into a data directory since it is not a workflow and GitHub only scans the top level of .github/workflows.
These instructions are long out of date and it does not seem worth updating them since packagers should now be using the pre-generated documentation in the distribution tarball introduced in 8fcfef5d.
