The build tool in src/build becomes build/build.py with its library in build/lib, and the C part of the documentation tool in doc/src becomes doc/doc.py with its library in doc/lib. They migrate together because doc/src was built on src/build, linking its config and help parsing along with its extensions to the core types. doc.pl runs doc.py rather than building and running a C binary, and meson runs build.py for the code it generates as it builds.
The three libraries form the hierarchy the C include path did: build/lib is the base, doc/lib sits on it, and test/lib sits on both. They are implicit namespace packages so a package spans them, and a module lives in exactly one library and holds the union of what the tools need from it, since Python resolves a module name to a single file and cannot extend one the way the C included a core module and added to it. define.yaml declares which libraries each tool may import and the test runner puts only those on the path, so a test of the build library cannot reach the doc or test library any more than the build tool can.
The declarations move with the code. config.yaml, error.yaml, and postgres.yaml are now at the root of build, and help.xml becomes doc/xml/reference.xml since it is documentation that the help happens to be generated from, so the generator is told where it is rather than knowing. dist.sh moves to build as well. The documentation tool's own option declarations are gone because they existed only to generate the option parser for the C binary and doc.py parses its command line directly.
Three modules in src/build/common were never build tool code. exec, regExp, and string extend the core types for the test harness and the unit tests, which were their only users, so they move to test/src/harness. The json and xml extensions had no users outside the two tools and are removed with them, and the tests for both are trimmed to the core functions that remain.
The help unit test built its fixture by calling the generator's C functions directly, which was the last thing outside the tools that depended on them. It now runs build.py help-data to write the packed help and loads that, so it still runs against the current declarations without needing the generated C, which may not have been written when the test runs.
The manual page wrapping is a port of the wrapper the binary uses for its own help, so the manual page and pgbackrest help break lines the same way, including where a word longer than a line is left unbroken.
libyaml is no longer needed to build pgBackRest since the only thing that read yaml was the C generator. It is dropped from the development container and from every CI platform, and meson no longer looks for it. Code generation needs nothing compiled first either, so a test run no longer builds a generator before it can select the tests.
The common source list is merged into the pgbackrest source list since it existed only so the generator could be built from the same files, and the entries that were out of alphabetical order are put in order while the list is being rewritten anyway.
Every file the generator writes is byte identical to what the C wrote, for the project declarations and for the documentation's own. The documentation is checked the same way: the HTML rendered on this branch is byte identical to the HTML rendered on integration, every file of it, which exercises the whole pipeline on the real content rather than only the four files the tool writes.
Every module in both libraries is declared in define.yaml and has full line and branch coverage. The coverage list is keyed by file rather than by module name, since the same name in two languages is two files and one of them would have silently lost its requirement while both existed.
TestError becomes ToolError, since it is the error type for three tools now rather than for the harness alone.
test.pl becomes test.py and ci.pl becomes ci.py, so an invocation only changes from .pl to .py. Running the tests needs no command since that is what a developer does, and the unit and coverage commands that a run drives are commands of test.py because a test runs inside a container where only the repository copy is available.
The Perl library in test/lib is gone. The modules the documentation depends on move to doc/lib, with ExecuteTest becoming Execute, and are trimmed to what the documentation calls, which removes the storage manifest, info, and owner along with everything else the tests were the only user of. release.pl no longer needs the vm definitions since all it did with them was compare two vm names.
Nothing in a test container or on a CI runner runs on Perl now so none of it is installed in either place, which was a substantial part of a base image. The packaging tools go with it since they are written in Perl and nothing builds a package, and ca-certificates is installed explicitly because it was arriving as a dependency of a package that is no longer there. Every base image is rebuilt as a result.
The generated Dockerfile is otherwise unchanged. The one difference is that the PostgreSQL repository for RHEL is installed for the architecture being built rather than for the architecture of the host, which only differs when cross building.
The list of tests to run is unchanged. It was compared against the Perl across the vm, module, test, run, and PostgreSQL version options before the Perl was removed.
The vm definitions, the user lookup, the test list, the container build, the job, the test run itself, and the code format are covered by tests, which the Perl equivalents never were. The container build is checked by comparing the generated script in full, since the hash of that script is the cache key.
Make the code format a command rather than two options of a test run, and fold in the black check that CI has been running as a step of its own. It gets the files to format from git rather than from the repository copy, so it no longer depends on a test run having made that copy and is told nothing about where either path is.
Wire up --log-level-test, which the driver has been collecting and discarding. Remove --psql-bin, --gen-check, --no-coverage-report, --build-max, --log-level-test-file, --no-optimize, --no-debug, and --debug-test-trace, none of which have done anything for some time.
Allow --vm=all only for --vm-build. Building every container at once works but running the tests on every vm never has, since a test needs the binary and the build for the vm it runs on and there is one of each.
The C harness in test/src/command/test is replaced by test/test.py with its library in test/lib, and test.pl runs that per test rather than building and running a C binary. Python is already required by meson so the only new dependency is PyYAML, and the harness no longer needs to be compiled before any test can run.
The harness is now tested the same way the rest of the code is. Test modules live in test/src/module/test and are declared in define.yaml with lang: python, so they run through the existing test.pl --module and --test interface. A python test is a plain function rather than a run inside testBegin(), since these tests are fast enough that selecting a single run is not useful, and --test-name selects one function when debugging.
Coverage from both languages merges into a single report, so a run that covered C and python produces one result. Full line and branch coverage is required and the uncovered and uncoverable annotations work as they do for C. Python reports the lines and arcs it knows about rather than everything a module contains, and which arcs a run reports depends on which ones it took, so python coverage merges by line number and by where an arc goes rather than by position.
A python test may import only the library modules it declares, which is what the C link step enforced by compiling in only what was declared. An import that is already in sys.modules never reaches an import hook, so the test module is loaded in its own interpreter with the hook installed ahead of the library.
Fix coverage for a shimmed .c.inc module, which has been missing from the report rather than reported as uncovered. The build writes the shimmed copy to the include path so it resolves ahead of the repository copy, which puts it outside src, and the filter only matched the path such a module is normally compiled from. In practice this is command/backup/process.inc, the only shimmed .c.inc.
Install PyYAML wherever the harness runs and a coverage module wherever the python tests run. rh8 installs python3.12 since PyYAML is not packaged for the platform python there, u24 and rh10 package a coverage new enough to report branch detail per line, and rh9 installs one with pip since the packaged version predates it. The jobs that run outside a container install PyYAML as each platform allows, with pip on macOS since homebrew has no formula for it.
Check the format of the harness and its tests with black, which previously covered only the smoke test.
YAML mappings are unordered, but shim key order determined the order the modules were included in the harness, e.g. the fd harness includes common/io/fd before common/io/fdWrite. A sequence makes that order explicit instead of relying on parser behavior.
A shim entry is now either a bare module name or a map with a name and a function list, matching how harness entries are already written, and the null placeholder the map form needed is gone.
The harness integration flag is also applied to its shims after the harness has been parsed, so it no longer depends on integration appearing before shim.
The restore file module is only used internally by the restore protocol handler so move it into a file.c.inc that is included directly into protocol.c, matching the approach used for the other restore modules. This makes restoreFile() static and avoids maintaining an extra header and build entry.
Move the RestoreResult enum to protocol.h since it is still needed by process.c.inc and the restore tests but file.h no longer exists to provide it.
backupFile() is only called by the backup protocol handler so include file.c into protocol.c directly, matching how the rest of the backup module is included into backup.c. This makes backupFile() static and removes the file.h header and build entry. The BackupFile/BackupFileResult structs are only used by backupFile() and the protocol handler so they become file-private as well.
The BackupCopyResult enum is still needed by the backup process to interpret protocol job results so move it to protocol.h.
The backup module has grown large so split it into .c.inc files that are included directly into backup.c, matching the approach used for the restore module. These functions are only used internally by backup so this avoids maintaining extra header and build files and keeps them static.
Since backupProcess() is shimmed for testing and now lives in an included module, the shim uses the inc option to locate and shim it there.
During restore, files bundled together in the repository are read back one file or range at a time, and each new read from an object store pays the latency of establishing a request. The restore copy loop worked around this by manually coalescing adjacent bundled files into a single read, but it could neither hide request latency nor skip over unneeded data within a bundle.
Add StorageReadMulti, which is given the full list of files and ranges up front and reads them back in order. Requests are submitted ahead of time (prefetch) so the latency of the next request overlaps reading the current one, and ranges separated by a small gap (e.g. a file that already exists on the host) are merged into a single request by skipping over the gap ("read over") rather than closing and reopening at a new offset, which is faster and cheaper than a new request for most storage classes.
To support prefetch, the S3, Azure, and GCS read drivers gain an async open. The HTTP request is submitted when the read is opened and the response is deferred until the first read via a new optional openAsync driver function, so multiple requests can be in flight at once. storageReadNew() gains an async flag; the async open is completed lazily on first read and is never combined with ignore missing.
The default StorageReadMulti driver queues reads directly on the storage. When the repository is remote a dedicated remote driver is used instead so the entire request list is forwarded to the remote in a single open and prefetch and read over run there, closer to the storage. The remote read and remote read-multi share stream state and chunk handling.
Two options control the behavior: repo-storage-prefetch (default 3, range 0-1024) sets how many requests to queue ahead, and repo-storage-read-over (default 64KiB, range 0B-1MiB) sets the largest gap to read over rather than issue a new request. The number of open requests per read is one greater than prefetch since the initial read cannot be prefetched.
Rewrite the restore copy loop to add each file that will be copied to a StorageReadMulti and read through it, removing the manual offset and limit tracking across bundled files.
The harness modules were the only files in test/src/common and each was named with a harness prefix, e.g. common/harnessLog.c. Move them to test/src/harness and drop the prefix so the file name matches the module it supports, e.g. harness/log.c.
Renaming in place would leave harness headers such as log.h, lock.h, and time.h at the same include path (common/) as their real src/common counterparts. Because test/src precedes src on the include path, the harness header would shadow the real one, so the harness must live under a distinct harness/ path.
Adjust the harness path construction in command/test/build.c, normalize the include guards to a TEST_HARNESS_ prefix (fixing blockIncr.h, which incorrectly used BLOCK_DELTA), and update the harness file references in the contributing documentation.
Several test modules defined the harness key more than once in a single test, either a bare name alongside a map or multiple maps, which relies on the YAML parser accepting duplicate mapping keys. Combine these into a single YAML list per test so a test can declare multiple harnesses without duplicate keys, which a stricter parser rejects. Modules with a single bare name or map are left unchanged.
Update the define parser to accept a harness sequence in addition to the existing bare name and single map forms; the parsed data structures are unchanged. Only the C parser consumes the harness and shim keys, so no other parsing is affected. Update the harness test fixture to the list form to cover the new sequence branch.
Running as the root user for commands other than restore can create files (e.g. backup.info in the repository) owned by root that the PostgreSQL user cannot access, causing later commands to fail.
Add the allow-root option to gate this. It defaults to false for all commands except restore, which is designed to run as root and manages file ownership carefully. The check applies to all command roles so a non-root command cannot run as root on a remote, and is skipped for help/version.
Replace the verbose, hand-built HrnLibSsh2 script entries in the SFTP unit tests with a set of per-function HRN_LIBSSH2_* response macros (one per libssh2 shim function). Each macro names its function, bakes in the fixed parameters the production code always passes, and defaults the common result, so a test supplies only the values that vary as trailing designated initializers.
The harness now defaults omitted values rather than requiring every field to be spelled out: libssh2_session_hostkey() defaults length/type/value; libssh2_sftp_stat_ex() defaults an omitted .attr to a regular file with mode 0640, defaults .flags to the standard attribute set (adding a size for regular files), and defaults .uid/.gid to the test user/group. Add HRN_LIBSSH2_ATTR_EXISTENCE (path exists but reports no attributes) and HRN_LIBSSH2_OWNER_ROOT (report ownership by root) sentinels, plus HRN_LIBSSH2_DIR/FILE/LINK/FIFO() helpers that OR a file type with an octal mode.
Replace the NULL-terminated script array and hrnLibSsh2ScriptSet(array) with HRN_LIBSSH2_SCRIPT_SET(...), which computes the script length so no terminator entry is needed; hrnLibSsh2ScriptSet() now takes an explicit size. Rebuild HRNLIBSSH2_MACRO_STARTUP/SHUTDOWN() and HOSTKEY_HASH_ENTRY() on top of the new macros, and stat_ex now verifies the requested stat_type (via .follow) instead of scripting it as a parameter.
Reorganize the tests themselves: split bundled comment groups into TEST_TITLE sections, split scripts per section, and drop redundant connect/disconnect setup. Remove tests that duplicate Posix tests for common code. Remove duplicative SFTP tests. Rename HrnLibSsh2 fields for clarity (attrPerms -> attr/mode, symlinkExTarget -> target).
These changes cut sftpTest.c roughly in half.
The restore module has been large and unwieldy for some time but I have been loathe to split it into separate compilation units because it means maintaining additional header files, updating build files, and losing optimizations from static functions. These functions are only used internally by restore and it seems wasteful to extern them.
We already have a number of cases where C files are included directly into other C files, especially .vendor.c.inc and .auto.c.inc files. We also include C files to add functionality needed for build/doc/test to core objects without having to add that functionality to core. See src/build/common/string.c for an example.
The test/coverage code already supports C includes but I had to update it to recognize the new extension.
Configuration of specific cipher suites may be required for compliance or to use preferred ciphers for security.
Cipher suites are applied to the entire process and cannot be configured on a per-connection basis, except that for object store clients (e.g. S3) verification can be disabled.
Allow users to specify HTTP in the endpoint but default to using HTTPS in all other scenarios to preserve the existing behavior.
Extend HttpUrl with a `defaultType` parameter to support either:
- Explicitly specifying a protocol via `.type` and enforcing that protocol is used in the URL.
- Allowing protocol to be parsed from URL, but providing default via `.defaultType` if no protocol is found in the URL.
Add partial write handling in fdWrite() to support non-blocking socket operations. The write loop now handles EAGAIN errors by waiting for the file descriptor to become writable, and continues writing the remaining bytes when write() returns fewer bytes than requested. This is required for HTTP, which may use non-blocking sockets, but doesn't have built in handling like the TLS client we are using for HTTPS. Also wrap the write() call and add a shim and additional logging for easier unit testing.
Previously it was possible to achieve a deadlock in a signal handler, for example when SIGTERM (i.e. sent by `pgbackrest stop --force`) arrives when a lock used in `gmtime_r` is taken. Then the next time logging is done, it will deadlock on `gmtime_r`.
In general, most stdlib functions are not safe to call in signal handlers, only so called async-signal safe functions are. In particular, `snprintf` isn't safe since it is allowed to internally call `malloc`. The `exitSafe` function isn't safe due to extensive use of allocations. Because of this, we need to use a simpler logging format in signal handlers, one that only uses async-signal safe functions.
Prior to this commit it was difficult to expire just the oldest full backup while ignoring current retention settings. The user had to manually update retention or script something to automate it. Expiring the oldest full backup is useful when disk space is running low.
Add the --oldest option to allow expiration of the oldest full backup and any dependent backups regardless of the current retention settings. Archive retention is also adjusted to expire WAL before the oldest retained full backup.
The prior code allocated the entire chunk buffer when the file was opened. However, in practice many files are smaller than the chunk buffer, especially in the main process.
Instead grow the chunk buffer as data comes in to save memory when smaller files are being processed. This adds some overhead for reallocations but modern processors do this very efficiently so it should not be significant compared to the cost of compressing, encrypting, and transferring files. Even so, the growth is fairly aggressive when the input buffers are full so only one or two reallocation are required to get to the default chunk size.
The Perl processing of config.yaml put hard limits on the format of that file. To allow flexibility in the file format remove all Perl processing on config.yaml.
This is just the beginning of migrating the preprocessor to C but even this small bit allows the removal of a lot of Perl code.
The prior implementation of the restore command did not provide any progress information. Implement it similarly to the backup command and also update the info command to display restore progress alongside backup progress.
Restore is a read operation and should not block other commands. The only exception is that multiple restores with the same lock and repository path are not allowed, as each restore must write progress to a separate file. Therefore, no lock is needed for restores with a remote role and the restore command should not be terminated when the stop command is executed.
There is currently a retry if the initial get request fails (depending on the error code) but if the read fails later on while fetching blocks of data it is fatal. In most cases there is a higher level retry (e.g. restore) but restarting the restore job might be expensive depending on how many files are being restored.
Add a retry that will catch read errors and retry from where the last data was successfully read.
A bit of history -- this patch was first started three years ago but the memory context model at that time would not allow the interface (StorageRead) to own the driver (e.g. StorageReadS3). Subsequent improvements in memory contexts have allowed this ownership model and in fact it is now the default so no ownership changes are required in this patch except in StorageReadRemote which was not updated in f6e3073.
Some C libraries (e.g.musl) render floating point numbers differently when using printf(). This does not cause any problems for the core code but the unit tests require more predictability to function smoothly without a lot of exceptions.
To accomplish this, wrap the various floating point operations in functions that mostly continue doing math using double but format the output string using integers. This leads to more predictable output at the cost of some complexity.
Rounding could also be accomplished using nearbyint() but this would require linking the math library, which does not seem worth it for a fairly simple operation.
Add support for the verify command --set option. This (internal) option was already accepted without errors but was not implemented.
The default behavior for verify is to check all the backups present. With the --set option only the specified backup will be verified. If the specified backup label is not valid an error is added to the result and verification is skipped. In addition, only WAL required to make the specified backup consistent will be verified.
If the user picks an invalid timeline (or the default is invalid) they will not discover it until after the restore is complete and recovery starts. In that case they'll receive a message like this:
FATAL: requested timeline 2 is not a child of this server's history
DETAIL: Latest checkpoint is at 0/7000028 on timeline 1, but in the history of the requested timeline, the server forked off from that timeline at 0/600AA20.
This message generally causes confusion unless one is familiar with it. In this case 1) a standby was promoted creating a new timeline 2) a new backup was made from the primary 3) the new backup was restored but could not follow the new timeline because the backup was made after the new timeline forked off. Since PostgreSQL 12 following the latest timeline has been the default so this error has become common in split brain situations.
Improve pgBackRest to read the history files and provide better error messages. Now this error is thrown before the restore starts:
ERROR: [058]: target timeline 2 forked from backup timeline 1 at 0/600aa20 which is before backup lsn of 0/7000028
HINT: was the target timeline created by accidentally promoting a standby?
HINT: was the target timeline created by testing a restore without --archive-mode=off?
HINT: was the backup made after the target timeline was created?
This saves time since it happens before the restore and gives more information about what has gone wrong.
If the backup timeline is not an ancestor of the target timeline the error message is:
ERROR: [058]: backup timeline 6, lsn 0/4ffffff is not in the history of target timeline B
HINT: was the target timeline created by promoting from a timeline < latest?
This situation should be rare but can happen during complex recovery scenarios where the user is explicitly setting the target time.
Verify that all StringIds in the project have been generated correctly.
This also makes it easy to generate new StringIds by copying an existing StringId and modifying the string. The error message will provide the required value.
This code was duplicated in each driver so this means less duplication.
In addition, some drivers were not creating a parameter list for decompression which meant they could not be used remotely. This is not a currently a bug since none of them were being used remotely, but it was a blocker for using lz4 for protocol compression.
This feature allows the archive-get, info, repo-get, repo-ls, restore, and verify commands to operate at a point-in-time on versioned buckets in Azure, GCS, and S3. This allows recovery even if a repository has been accidentally or maliciously deleted or corrupted.
This function will be needed elsewhere going forward and in any case it makes sense to move this out of the restore module.
Refactor to avoid dependency on a regular expression.
This command was used by the Perl integration tests to create buckets for storage types that required it. Now that the integration tests are written in C they can simply use the same code to create buckets.
The command was also used in the documentation but there it seems more appropriate to use the corresponding vendor CLI.
Previously it was not possible to read or write two files at the same time on the same remote because the protocol was entirely taken over by the read or write command. Multiple reads are required to make restores efficient when a list of bundled files is being read but blocks need to be retrieved from a separate file or a different part of the same file.
Improve that situation with sessions that allow related commands to be run with shared state. Also break read/write into separate requests (rather than pushing all data at once) so they can be multiplexed.
The disadvantage for read/write is that they now require more back and forth to transfer a file. This is mitigated by sending asynchronous read/write requests to keep both server and client as busy as possible. Reads that can fit into a single buffer are optimized to transfer in a single command. Reads that transfer the entire file can also skip the close command since it is implicit on end-of-file.
These changes allow the protocol to be simplified to provide one response per request, which makes the data end message obsolete. Any data sent for the request is now added to the parameters so no data needs to be sent separately to the server outside the request parameters.
Also update the Db protocol to use the new sessions. Previously this code had tracked its own sessions.
Refactor the lock module to split command-specific logic from the basic file locking functionality. Command specific logic is now in command/lock.c. This will make it easier to implement new features such as repository locking and updating lock file contents on remotes.
This implementation is essentially a drop-in replacement but there are a few differences. First, the lock names no longer require a path (the path is added in the lock module). Second, the timeout functionality has been removed since it was not being used.
lcov does not seem to be very well maintained and is often not compatible with the version of gcc it ships with until a few months after a new distro is released. In any case, lcov is that not useful for us because it generates reports on all coverage while we are mainly interested in missing coverage during development.
Instead use the JSON output generated by gcov to generate our minimal coverage report and metrics for the documentation.
There are some slight differences in the metrics. The difference in the common module was due to a bug in the old code -- build/common was being added into common as well as being reported separately. The source of the two additional branches in the backup module is unknown but almost certainly down to how exclusions are processed with regular expressions. Since there is additional coverage rather than coverage missing this seems fine.
Since this was pretty much a rewrite it was also a good time to migrate to C.
The GCS driver sent a single file delete request for each file while deleting a path. Depending on latency this could lead to rather long delete times, especially noticeable during expiration.
Improve GCS delete to use batches, which require multipart HTTP, so also add multipart HTTP infrastructure.
Connections are established using the "happy eyeballs" approach from RFC 8305, i.e. new addresses (if available) are tried if the prior address has already had a reasonable time to connect. This prevents waiting too long on a failed connection but does not try all the addresses at once. Prior connections that are still waiting are rechecked periodically if no subsequent connection is successful.
This improves substantially on 39bb8a0, which failed to take into account connection attempts that do not fail (but never connect) and use up all the available time.
The Perl integration tests were migrated as faithfully as possible, but there was some cruft and a few unit tests that it did not make sense to migrate.
Also remove all Perl code made obsolete by this migration.
All unit, performance, and integration tests are now written in C but significant parts of the test harness remain to be migrated.
The core Exec object is efficient but geared toward the specific needs of core and not ease-of-use as required for build, documentation, and testing.
execOne() works similarly to system() except that it automatically redirects stderr to stdout and captures the output.
These tests have not been maintained for several years, i.e. no tests for new features have been added. They are highly duplicative of the unit tests but do have the advantage of mixing in different storage drivers. They were allowed to remain because they were not doing any harm even if they were probably not doing any good.
However, the real integration tests (that run directly against PostgreSQL) also test storage drivers and have been updated with new features over time. The real integration tests are now being migrated to C and as part of that effort the mock integration tests need to be removed or migrated, and they do not provide enough value to migrate.
Remove all mock integration tests and a leftover Perl performance test.
During restore it is possible to read all the blocks out of a compressed super block without reading all the input. This is because the compression format may have some trailing bytes that are not required for decompression but are required to indicate that data has ended. If a buffer aligned with the compressed data in a certain way, these last bytes might not be read.
Explicitly read out any final bytes at the end of each super block to handle this case. This should always result in no additional data out and we check for that, but it does move the read position to the beginning of the next compressed super block so decompression can begin without error.
The backupFile() tests were written before the bulk of the backup command had been migrated to C. Some of them have been migrated to the complete backup tests, but others were left because there was no way to make changes to files during a backup.
Now that we have the backup script harness introduced in 337da35a it is now possible to migrate all the tests. The new tests are better because they not only test backupFile() but all the functions upstream and downstream of it.
This behavior violates an assertion but is completely possible with the current implementation. This behavior will be fixed in a future commit, but for now at least test how it works correctly and remove the assertion so the test runs without error.
Also add a new harness that allows changes during the backup to be scripted.
Migrate generation of these files from help.xml to the intermediate documentation format. This allows us to share a lot of code that is already in C and remove duplicated code in Perl. More duplicate code can be removed in Perl once man generation is migrated.
Also update the unit test harness to allow testing of modules in the doc directory.
