IMPORTANT NOTE: The restore command will now error and stop if any restored file contains an invalid page checksum, which leaves the cluster incomplete and unable to start, so the restore must be rerun. The expire command will now refuse to expire if the oldest retained backup would contain checksum errors, which also prevents archive from being expired in that repository. Set checksum-page-error=n to warn instead of erroring.
Features:
* Add checksum-page-error option to error on invalid page checksums. (Contributed by Andrew Pogrebnoi. Reviewed by Douglas J Hunley, David Steele, Greg Sabino Mullane, Stefan Fercot. Suggested by Greg Sabino Mullane.)
* Support for GCS Workload Identity Federation. (Contributed by Adol Rodriguez. Reviewed by Douglas J Hunley, David Steele.)
* Prefetch multi-file/range reads from object stores. (Reviewed by Andrew Pogrebnoi, Douglas J Hunley.)
Improvements:
* Reduce keep-alive requests to remote processes. (Reviewed by Douglas J Hunley.)
* Retry Azure batch sub-requests missing from the response. (Reviewed by Douglas J Hunley.)
* Reduce the manifest checks made when reconstructing backup.info. (Reviewed by Douglas J Hunley. Suggested by todeb.)
* Use original size to determine if a file should be bundled. (Reviewed by Douglas J Hunley.)
* Add context to block map read errors during backup. (Reviewed by Stefan Fercot, Douglas J Hunley.)
* Cast current_timestamp to text in the backup start label. (Contributed by Tiago Corcelli de Oliveira. Reviewed by David Steele.)
* Support building on illumos. (Contributed by Kalevi Kolttonen. Reviewed by Douglas J Hunley, David Steele.)
* Support OpenSSL 4. (Reviewed by Douglas J Hunley.)
* Fix clang builds. (Reviewed by Douglas J Hunley.)
* Migrate code generation from C to Python.
Documentation Improvements:
* Modernize the website. Special thanks to Jan for the insightful (and hilarious) logo. (Contributed by David Steele, Jan Wieremjewicz. Reviewed by Steve Atkins, Kai Wagner, Douglas J Hunley, Nikolaos Stamatopoulos, Matt Hudson, Stefan Fercot.)
* Document the release schedule.
* Migrate the documentation tools from C and Perl to Python.
Test Suite Improvements:
* Migrate the test harness from C and Perl to Python.
* Add a Fil-C job to CI.
* Add fuzz testing for the JSON, Ini, and Pack readers.
* Allow test user names that adduser on Debian rejects. (Contributed by Stefan Fercot. Reviewed by David Steele.)
The release notes are checked against git history that ends at the newest commit of HEAD that is also in integration, which leaves out commits made only on a release branch. CI has no local integration branch so the history ended at HEAD and included release branch commits. A release branch commit with the same subject as an integration commit then made the subject ambiguous.
CI now fetches integration from the project repository into refs/remotes/upstream/integration. The history ends at the merge-base with the local integration branch or else with the upstream branch. The origin/integration branch of a fork is not used since it may be stale.
The check that every commit is claimed still runs only when the local integration branch is related to HEAD.
Inline keywords such as options, paths, and ids are now .9em rather than .8125em. In light mode their box uses a lighter background and border, which brings it closer to the contrast of dark mode.
Use weight 500 for names and commands in light mode since 600 looked too heavy. Dark mode stays at 600.
Every item in v2.33 through v2.60.0 other than development items now has a summary, and every commit of each release is claimed by an item. Commits that were not mentioned are grouped into development items. Missing and incorrect pull request and issue references were fixed by comparing against GitHub.
When the local integration branch is related to HEAD, a release with details must claim every commit. A commit whose subject is not unique in a release is named with <commit id="..." subject="...">. Commits back-patched to a patch release are listed in <release-backpatch-list> and shown only on a development build. The commits of a patch release are read from its release/x.y.z tag.
A development item can specify the release note of a later release with <release-note version="..." subject="...">. The release note lists the development commits in its details, and on a development build the development item links to the release note. A release in development can be specified by its final version, e.g. 2.61.0 for 2.61.0dev.
Add the tester contributor role, which replaces the comments that recorded testers as reviewers.
Disable scroll anchoring on the page so opening details does not scroll the page in Firefox.
Release items from 2.60 on put the paragraphs after the first and the links to their issues, pull requests, and commits in a details element that is closed by default. The toggle is drawn in place of the bullet: a plus when the details have text and a chevron when they have only links. An item with more than one commit has a line for each commit with its subject. A button on each release opens or closes all of its details. Older releases still join extra paragraphs into one line.
The version pattern for bug fix releases also matched x.y.0, so the commits of a release reached back to v2.53, and 2.54.0 through 2.59.0 were only checked while each was the most recent release. A bug fix release is now x.y.z with z > 0, so a release keeps its commit links after a newer release is added. The 2.55.0 PostgreSQL 18 item now has a commit element because the note was reworded after the commit.
The GCS Workload Identity Federation item now has a commit element because its text does not match the commit subject.
The doc build now writes doc/resource/git-history.cache from git log when HEAD is not the newest commit in it. The file is ignored rather than committed, and release.py no longer updates it.
The committed cache had subject fixes that release items matched against. Those items now have a commit element with the subject git reports, and the rendered release notes are unchanged.
The doc and dist workflows fetch the full history since the release notes are checked against it.
Add libFuzzer targets for the JSON, ini, and pack readers in test/fuzz. Each target runs an input through the reader and discards the errors thrown for invalid input, while a failed assertion or a sanitizer report aborts. The targets are linked with the objects of the pgbackrest binary when the fuzz-engine meson option is set.
test/fuzz/build.sh builds the targets and seed corpora in the environment that OSS-Fuzz defines, so an OSS-Fuzz project can call it if pgBackRest is accepted. Until then the fuzz workflow builds the targets with clang, AddressSanitizer, and UndefinedBehaviorSanitizer and runs each one for a fixed number of runs. The seed is derived from the commit hash and ASLR and corpus reloading are disabled, so a failed run can be reproduced.
Fuzzing found or led to the bugs fixed in these commits:
81704daaf Check the mem context stack bounds in production builds.
4b8e3f438 Limit the depth of nested containers when reading JSON.
b623071a2 Fix reads past the end of JSON strings ending in an escape.
2b919843a Reject unexpected commas when reading JSON.
1808295cc Reject \u0000 when reading JSON strings.
b63357ee8 Accept hex digits in JSON \u escapes when skipping.
6963016c7 Error on invalid field ids when reading packs.
0966ebbb3 Check pack field sizes against the bytes remaining in a buffer.
950c1e149 Error on zero-size pack fields.
A pack written by pckWrite always ends with an end tag, so a nested pack is at least one byte. A zero-size pack field from corrupt data was read into an empty buffer with no allocation, which failed the buffer != NULL assertion in pckReadNewC() in debug builds. Production builds failed with unexpected EOF on the first read from the empty pack.
Error with invalid pack size in pckReadPack() and pckReadPackReadConst() when the size of a pack field is zero.
pckReadBin() and pckReadPack() allocate a buffer of the size stored for a field before reading the field. A pack read from a buffer with a corrupt size could allocate far more memory than the pack holds before failing with unexpected EOF.
Error when the size of a field is larger than the bytes remaining in the buffer. A pack read through an IoRead still allocates the size stored for the field because the total size is not known.
pckReadTagNext() cut the high-order bits of the field id delta down to an unsigned int and added it to the last id without checking the result. A large delta could produce UINT_MAX, the id that marks the end of a container, or wrap to an id at or below the last id. Either one left the read out of step with the tags, which asserted in debug builds and misread fields in production builds.
Error when the id is not greater than the last id or is UINT_MAX.
jsonReadSkipStr() required four decimal digits after \u, so a valid escape with hex digits such as \u003c failed validation even though jsonReadStr() decodes it. Accept hex digits so skipping agrees with reading.
pgBackRest does not write \u escapes, so this only affects JSON from other sources. jsonReadStr() still decodes only \u00XX, as a single byte.
jsonReadStr() decoded a \u0000 escape and appended a null character to the string. This asserted in debug builds and created a string with an embedded null in production builds, which is cut short anywhere the string is used as a zero-terminated string.
Error on \u0000. A string cannot store a null character.
jsonReadTypeNextIgnoreComma() looks past a comma to find the type of the next value, but jsonReadPush() only consumed a comma between values in a container. A comma before a value outside a container, before the first value in a container, or before an object value was left in place. The string readers asserted on it in debug builds and read it as the opening quote of a string in production builds.
Error on a comma in these positions.
jsonReadSkipStr() stepped past the terminating null when a backslash was the last character. It also advanced one character too far after a \uXXXX escape, which skipped the closing quote when the escape was the last character of the string and stepped past the terminating null when the escape was at the end of the input. jsonReadStr() did not check that both digits of a \u00XX escape were present, so a truncated escape at the end of the input was read past the terminating null.
The skip path validates the JSON values in info files and manifests. The read path reads object store responses.
Each level of a nested array switches to a mem context in jsonReadVar() and each level of any container recurses, so deeply nested JSON could fill the mem context stack or exhaust the C stack. JSON is read from repository info files, manifests, lock files, and object store responses, which pgBackRest does not always control.
Error when the depth exceeds 32. The JSON pgBackRest reads does not nest more than a few levels.
memContextNew() added to the mem context stack without checking for space. memContextSwitch() asserted on the current stack index, which can be lower than the index being written, and the assert only existed in debug builds. Filling the stack overwrote the memory after it. Nested JSON arrays in a repository info file or an object store response can fill the stack because each level of an array switches to a mem context.
Check the max stack index in both functions with CHECK() so the error is also reported in production builds.
Add UNLIKELY(), based on __builtin_expect() when the compiler supports it, and use it in CHECK(), ASSERT(), and the THROW*_ON_SYS_ERROR*() macros. The condition in these macros is an error, so the compiler can lay out the code for the path where the condition is false.
The ASSERT() redefinition in the test harness uses UNLIKELY() as well so modules that include the same code compile it with the same branches.
A user-defined implicit cast from timestamptz to text makes the || operator in the backup label ambiguous and the backup fails with "operator is not unique: unknown || timestamp with time zone". Casting current_timestamp to text resolves the operator to text || text. The label text is unchanged.
The illumos headers hide POSIX and extension functions when compiling with -std=c11 unless a feature test macro is defined, which left kill(), sigaction(), gmtime_r(), localtime_r(), nanosleep(), and flock() undeclared.
Define _POSIX_C_SOURCE=200809L and __EXTENSIONS__ in meson when the host system is sunos, as is done for Linux and MacOS. _POSIX_C_SOURCE alone does not expose flock(). Also link libsocket, which is separate from libc on illumos. Meson reports sunos for Solaris as well, but Solaris has not been tested.
Add an OmniOS job to CI that runs selected unit tests and the smoke test against the oldest and newest available PostgreSQL versions.
Archive key rotation moved the archive.info save from backup to expire, which changed when the archive.info timestamps are refreshed for object store lifecycle rules. With expire-auto disabled, or when expire errors before the save, the timestamps were no longer refreshed after a backup.
Backup now saves archive.info as before. Expire loads archive.info only when repo-cipher-rotate is set and saves it only after a rotation.
A deploy previously replaced every page in the site and in the prior version path, so pages with no changes other than the date in the footer showed as changed. Pages that differ from the copy on the site only in the release date and year are now left as they are. This applies to the site and to the prior version path, so a text-only change deployed after a release updates only the affected pages in both.
Repository format 6 and archive key rotation are not complete enough to release. Internal options are left out of the documentation and help but are still accepted, which allows format 6 to be tested with this release.
repo-gcs-key-type=web-id authenticates using an external account credential file (https://google.aip.dev/auth/4117) as generated by gcloud iam workload-identity-pools create-cred-config. The file is specified by repo-gcs-key and defines the audience, the STS token url, and the file that contains the web identity token. The web identity token is exchanged at the STS endpoint for an access token that authenticates the federated principal directly, without a service account key.
The web identity token file is reloaded on each exchange to pick up rotated tokens, e.g. Kubernetes projected service account tokens.
Service account impersonation (service_account_impersonation_url) is not supported. Only the file credential source is supported, and when credential_source.format is specified the type must be text.
Authentication token responses that are not JSON (e.g. an HTML error page) or that contain a nested error object are now reported with httpRequestError() rather than failing while parsing the response. This applies to all GCS key types that request a token.
The result of RAND_bytes() was ignored in cryptoRandomBytes(). When OpenSSL is unable to get random data from the operating system, RAND_bytes() fails and leaves the buffer unchanged, so a subkey was generated from uninitialized stack memory and a salt from data already in the output buffer. No error was reported and the repository was written with weak subkeys and salts.
Throw a CryptoError when RAND_bytes() fails, as other OpenSSL failures do. Test the error with a RAND_bytes() shim in the test module.
The results were ignored, so a failure would go unnoticed and leave the key and initialization vector or the HMAC uninitialized. Throw a CryptoError instead, as other OpenSSL failures do.
A read that does not block stops as soon as it has output, and end of file is set on a later read that gets no output. When a filter needed the same input again the read did not stop, so end of file could be set while output remained in the internal buffer used by the line and small readers. A read that loops until end of file then lost whatever remained in that buffer.
The block cipher filter needs the same input again on flush when the buffer has less room than the final block. A large encrypted info or manifest file could lose its last lines on load, which fails the checksum. This requires the lost lines to fit in the final block, so it is unlikely in practice. A filter that holds its output until the end of its input, such as the authenticated cipher proposed in #2866, would hit this on any file larger than the buffer.
Stop a read that does not block as soon as it has output, whether or not a filter needs the same input again.
At format 6 the repository can hold more than one encryption key. This is used to rotate the archive key, which is currently the only bulk data key that never changes.
Keys are stored by id in the new CipherSpecMap, with one key marked current for encrypting new files. At format 6 the cipher section of an info file stores every key by id along with the digest it derives with, e.g. cipher-pass={"1":{"digest":"sha256","key":"..."}}, and archive.info also records the current key id in cipher-pass-current. Format 5 files keep their single key, which is loaded as the key with no id.
archive-push writes the id of the archive key it used into the header of each WAL file. The readers use that id to select the key: archive-get, verify, the restore timeline read, the backup archive copy, and repo-get. A pusher that loaded archive.info before a rotation keeps working since each file records the key it was encrypted with. The filter that writes the header now also works through a repository host.
stanza-create for format 6 creates the first archive key with id 1. stanza-upgrade from format 5 keeps the existing archive key as id 0 so WAL already archived stays readable, and adds a key at id 1 for new WAL.
The new repo-cipher-rotate option sets how old the current archive key may be before expire replaces it, with a default of 90 days and a minimum of 15. Expire adds the new key, makes it current, and records the time of the rotation in archive.info as cipher-pass-rotate. At format 5 the option has no effect and warns only when set explicitly.
Expire now saves archive.info on every run. Backup did this only to refresh the timestamps for object store lifecycle rules and no longer writes archive.info, so with expire-auto disabled the timestamps are refreshed only when expire runs.
CipherSpec defaulted to SHA-256 when no digest was provided, the default for repository format 6. To work around this, the repository passphrase explicitly set SHA-1, and the format filter ignored the digest on the spec entirely in favor of the digest for the format of the file. That made it hard to tell when the digest on a spec was authoritative.
Now a spec with no digest determines it from the format of the file being read or written, and a digest set on the spec is always used. The block cipher and Info both assert that a digest has been set, so a missing digest is caught right away instead of quietly producing the wrong key.
There is no change in behavior. Tests that use the block cipher directly now set the digest explicitly.
The linter now reports C source in src and python modules in the tool libraries that are not in a coverage list in test/define.yaml. Generated .auto.c.inc modules are exempt, and modules that no test fully covers are on a skip list: src/main.c, the prototype manifest command, and vendored xxHash.
The mapping from a coverage entry to a file moves from the coverage command to define.py for use by the linter.
The C coverage filter no longer matches a harness .c.inc under test/src/harness as the module with the same path in src.
The info/manifest .c.inc modules, command/backup/resume.c.inc, and common/exec.c were not in any coverage list in test/define.yaml, so their coverage was never checked. The .c.inc modules were already fully covered.
The exec harness includes common/exec.c but execOne() had no tests, so add tests for output, an error exit, and a signal.
X509_get_subject_name() returns const X509_NAME * in OpenSSL 4. Make the subject name const and cast away the const for X509_NAME_get_index_by_NID(), which does not accept a const name before OpenSSL 3.
OpenSSL 4 also renamed the SSLv3 alert reason strings, e.g. "sslv3 alert handshake failure" is now "tls alert handshake failure". Add the new message to the TLS tests.
The coding standards in CODING.md and the Coding section of CONTRIBUTING.md covered the same ground, e.g. objects were described in one while memory contexts and logging were described in the other. Move the standards into the Coding section of CONTRIBUTING.md, move the uncoverable/uncovered code markers into Writing a Unit Test, and remove CODING.md.
Section depth is limited to three, so Naming is now a peer of Standards and Braces and Hints, Warnings, and Errors are now part of Standards.
When a command is given, argparse parses the options of the command into a new namespace and copies every value, including defaults, over the values the main parser set from options before the command. As a result, `test.py --vm=u24 vm-build` ran vm-build with --vm=none.
Move the command to the front of the argument list before parsing so the command parser parses every option. An option before the command that the command does not define is now an error instead of being ignored.
Time option values are stored in milliseconds but the table of time values in the generated parse rules was declared unsigned int, which limits a value to about 49.7 days. A default or allow-range bound larger than that overflows and fails to compile. The parser already reads time values, range bounds, and option values as int64_t, so declare the table as int64_t to match the table of size values.
All existing time option values fit in either type, so none of them change.
The test container is built with the same user name and id as the user on the host so files created inside it are owned by that user. Debian's adduser only accepts a name that matches NAME_REGEX, which excludes a dot, so vm-build failed on the Debian-based images for a host user named e.g. first.last.
Pass the flag that skips the check, since the name comes from the account running the build rather than from outside. The flag was renamed from --force-badname to --allow-bad-names and Ubuntu 22.04 ships a version that only knows the old spelling, so the spelling is taken from adduser --help when the container is built rather than hardcoded.
The archive-push-batch-size option converts the batch size to a segment count using the size of the first file in the ready list, but that list also contains backup and timeline history files. PostgreSQL does not sync backup history files, so a cluster copied from a storage snapshot may contain one with zero size, which caused every asynchronous run to error and prevented any WAL from being archived.
Use the size of the first ready WAL segment instead and skip the batch size when no segment is ready, since the remaining files are small. A zero-size WAL segment is still an error.
The new checksum-page-error option determines whether invalid page checksums are reported as a warning or an error. It is available for the backup, restore, and expire commands.
For backup the option defaults to false, which preserves the prior behavior of emitting a warning for each file that contains invalid page checksums. A warning with the total number of checksum warnings is now emitted at the end of the backup so they are less likely to be missed in the log. When the option is enabled the backup terminates with an error as soon as an invalid page checksum is found. Page misalignment is handled the same way since it prevents checksum validation.
For restore the option defaults to true. When the option is enabled the restore terminates with an error as soon as a file with invalid page checksums is restored. Since this happens before pg_control is renamed, the cluster cannot be started. When the option is disabled a warning is emitted for each file that contains invalid page checksums and the total is reported at the end of the restore. Previously the restore command did not report invalid page checksums at all.
For expire the option defaults to true. If the backup that would become the oldest retained backup contains invalid page checksums then expire terminates with an error before removing anything. When the option is disabled expire proceeds and emits a warning if the oldest retained backup contains invalid page checksums. An adhoc expire proceeds in either case but still emits the warning.
Listing with a repository target time uses the ListObjectVersions API but pagination read NextContinuationToken from the response, which is only returned by ListObjectsV2. A version listing truncated at the maximum key count failed with "unable to find child 'NextContinuationToken':0 in node 'ListVersionsResult'", so a repository with more than a thousand versions under a prefix could not be read at a target time.
Paginate the version listing with NextKeyMarker and NextVersionIdMarker instead. NextVersionIdMarker is missing or empty when the listing is truncated at a common prefix, in which case the next page starts at the key after the key marker, which is what is needed.
Build the query for each page from the base query so a marker set for one page is not reused for the next.
The mock response used to test version listing pagination returned a continuation token rather than markers, so the error was not caught.
The parsed key JSON, the file read buffer, and intermediate strings were allocated in the object context rather than a temp context, so they stayed allocated for the life of the storage object.
Format 6 derives encryption keys from the passphrase with SHA-256 rather than SHA-1. Older repositories keep SHA-1, since the migration does not rewrite the files they already encrypted. The digest is picked when a passphrase is generated rather than at each use, so it is stored with the passphrase in CipherSpec and defaults to SHA-256.
A passphrase outlives the format of the file it is stored in, so the format cannot decide how the key is derived. Info files at format 6 store cipher-digest beside cipher-pass, and a passphrase stored without one derives with SHA-1. stanza-upgrade writes back the digest the passphrase already had, so archives and backups from before the migration still load. Only a new passphrase gets SHA-256: the one for a stanza created at format 6, or for the manifest of a full backup written at it.
The info file has the same problem in reverse. Its format sits inside the content that the passphrase decrypts, so a reader would have to decrypt the file to find out how to decrypt it. Format 6 puts a plaintext header in front instead: PGBR, the three-digit format, and a spare byte for later. It goes where the OpenSSL Salted__ magic used to go and is the same eight bytes, so the salt still starts in the same place. A file that starts with the magic came from format 5, the only format older than the header. The header format is checked before anything is decrypted, then compared against the format found in the content, so a file stitched together from two others is rejected rather than half read.
The format constants move out of version.h into a new common/format, along with repoFormatValidate(), repoFormatDigest(), and CipherBlockFormat. That filter reads the header, builds a block cipher with the digest the format calls for, and streams the rest of the file through it. The block cipher no longer knows anything about repositories: it takes the digest from the spec and takes what sits in front of the content as a parameter.
Verify checksums info files as stored rather than as decrypted, since the header has to come off before decryption can start. A file and its copy are written from the same bytes, so comparing the two still works. In an encrypted repository created by a Perl version the file and its copy are separate encryptions, so their checksums differ and verify reports it at detail level. A log line is better than a separate code path for repositories with seven year old backups.
Response parts were mapped to sub-requests by iterating the parts, so when the service returned fewer parts than were requested the remaining sub-requests were silently treated as complete. Azure fails the entire batch when it is unable to parse the request and reports the failure as a single response part, so only one file per batch was removed while the command reported success.
Iterate the sub-requests instead and retry any that have no response part, just as parts that report an error are retried. More response parts than sub-requests is now an error.
The version header was not set for SAS since a SAS specifies a version in the sv parameter. That is sufficient for normal requests but batch requires the version to be set explicitly and rejects the entire batch with 400 InvalidInput when it is missing.
Azure reports the rejection as a single response part with no content-id, so it appeared that one sub-request had failed rather than that none had run. That part was retried individually and the remaining files in the batch were silently not deleted, leaving expired backups and WAL in the repository while expire reported success.
When a read is retried the driver is freed and recreated but the filter group is not set on the new driver. For drivers that require the filter group (i.e. remote) this causes a segfault when the read is opened because the filter group is NULL.
This was introduced in a953aa74b, which changed retry to recreate the driver rather than reopen it, added the filter group callback that is only set when the read is created, and replaced the driver retry flag with the storageFeatureReadRetry feature, which remote storage inherits from the repository host. Before that commit retry was never enabled for remote reads.
Even with the filter group set the retry could not work because the filters are passed to the remote when the read is opened so they cannot be run again, and bytes read are counted after the filters have been applied so the offset to restart the read cannot be determined.
Read retry is only enabled for object stores so this happened when the repository was on an object store accessed via a repository host. The repository host still retries the object store read, where the filters are applied above the driver and the retry works correctly, so the only retry lost is a redundant second attempt by the client.
