Archive key rotation moved the archive.info save from backup to expire, which changed when the archive.info timestamps are refreshed for object store lifecycle rules. With expire-auto disabled, or when expire errors before the save, the timestamps were no longer refreshed after a backup.
Backup now saves archive.info as before. Expire loads archive.info only when repo-cipher-rotate is set and saves it only after a rotation.
A deploy previously replaced every page in the site and in the prior version path, so pages with no changes other than the date in the footer showed as changed. Pages that differ from the copy on the site only in the release date and year are now left as they are. This applies to the site and to the prior version path, so a text-only change deployed after a release updates only the affected pages in both.
Repository format 6 and archive key rotation are not complete enough to release. Internal options are left out of the documentation and help but are still accepted, which allows format 6 to be tested with this release.
repo-gcs-key-type=web-id authenticates using an external account credential file (https://google.aip.dev/auth/4117) as generated by gcloud iam workload-identity-pools create-cred-config. The file is specified by repo-gcs-key and defines the audience, the STS token url, and the file that contains the web identity token. The web identity token is exchanged at the STS endpoint for an access token that authenticates the federated principal directly, without a service account key.
The web identity token file is reloaded on each exchange to pick up rotated tokens, e.g. Kubernetes projected service account tokens.
Service account impersonation (service_account_impersonation_url) is not supported. Only the file credential source is supported, and when credential_source.format is specified the type must be text.
Authentication token responses that are not JSON (e.g. an HTML error page) or that contain a nested error object are now reported with httpRequestError() rather than failing while parsing the response. This applies to all GCS key types that request a token.
The result of RAND_bytes() was ignored in cryptoRandomBytes(). When OpenSSL is unable to get random data from the operating system, RAND_bytes() fails and leaves the buffer unchanged, so a subkey was generated from uninitialized stack memory and a salt from data already in the output buffer. No error was reported and the repository was written with weak subkeys and salts.
Throw a CryptoError when RAND_bytes() fails, as other OpenSSL failures do. Test the error with a RAND_bytes() shim in the test module.
The results were ignored, so a failure would go unnoticed and leave the key and initialization vector or the HMAC uninitialized. Throw a CryptoError instead, as other OpenSSL failures do.
A read that does not block stops as soon as it has output, and end of file is set on a later read that gets no output. When a filter needed the same input again the read did not stop, so end of file could be set while output remained in the internal buffer used by the line and small readers. A read that loops until end of file then lost whatever remained in that buffer.
The block cipher filter needs the same input again on flush when the buffer has less room than the final block. A large encrypted info or manifest file could lose its last lines on load, which fails the checksum. This requires the lost lines to fit in the final block, so it is unlikely in practice. A filter that holds its output until the end of its input, such as the authenticated cipher proposed in #2866, would hit this on any file larger than the buffer.
Stop a read that does not block as soon as it has output, whether or not a filter needs the same input again.
At format 6 the repository can hold more than one encryption key. This is used to rotate the archive key, which is currently the only bulk data key that never changes.
Keys are stored by id in the new CipherSpecMap, with one key marked current for encrypting new files. At format 6 the cipher section of an info file stores every key by id along with the digest it derives with, e.g. cipher-pass={"1":{"digest":"sha256","key":"..."}}, and archive.info also records the current key id in cipher-pass-current. Format 5 files keep their single key, which is loaded as the key with no id.
archive-push writes the id of the archive key it used into the header of each WAL file. The readers use that id to select the key: archive-get, verify, the restore timeline read, the backup archive copy, and repo-get. A pusher that loaded archive.info before a rotation keeps working since each file records the key it was encrypted with. The filter that writes the header now also works through a repository host.
stanza-create for format 6 creates the first archive key with id 1. stanza-upgrade from format 5 keeps the existing archive key as id 0 so WAL already archived stays readable, and adds a key at id 1 for new WAL.
The new repo-cipher-rotate option sets how old the current archive key may be before expire replaces it, with a default of 90 days and a minimum of 15. Expire adds the new key, makes it current, and records the time of the rotation in archive.info as cipher-pass-rotate. At format 5 the option has no effect and warns only when set explicitly.
Expire now saves archive.info on every run. Backup did this only to refresh the timestamps for object store lifecycle rules and no longer writes archive.info, so with expire-auto disabled the timestamps are refreshed only when expire runs.
CipherSpec defaulted to SHA-256 when no digest was provided, the default for repository format 6. To work around this, the repository passphrase explicitly set SHA-1, and the format filter ignored the digest on the spec entirely in favor of the digest for the format of the file. That made it hard to tell when the digest on a spec was authoritative.
Now a spec with no digest determines it from the format of the file being read or written, and a digest set on the spec is always used. The block cipher and Info both assert that a digest has been set, so a missing digest is caught right away instead of quietly producing the wrong key.
There is no change in behavior. Tests that use the block cipher directly now set the digest explicitly.
The linter now reports C source in src and python modules in the tool libraries that are not in a coverage list in test/define.yaml. Generated .auto.c.inc modules are exempt, and modules that no test fully covers are on a skip list: src/main.c, the prototype manifest command, and vendored xxHash.
The mapping from a coverage entry to a file moves from the coverage command to define.py for use by the linter.
The C coverage filter no longer matches a harness .c.inc under test/src/harness as the module with the same path in src.
The info/manifest .c.inc modules, command/backup/resume.c.inc, and common/exec.c were not in any coverage list in test/define.yaml, so their coverage was never checked. The .c.inc modules were already fully covered.
The exec harness includes common/exec.c but execOne() had no tests, so add tests for output, an error exit, and a signal.
X509_get_subject_name() returns const X509_NAME * in OpenSSL 4. Make the subject name const and cast away the const for X509_NAME_get_index_by_NID(), which does not accept a const name before OpenSSL 3.
OpenSSL 4 also renamed the SSLv3 alert reason strings, e.g. "sslv3 alert handshake failure" is now "tls alert handshake failure". Add the new message to the TLS tests.
The coding standards in CODING.md and the Coding section of CONTRIBUTING.md covered the same ground, e.g. objects were described in one while memory contexts and logging were described in the other. Move the standards into the Coding section of CONTRIBUTING.md, move the uncoverable/uncovered code markers into Writing a Unit Test, and remove CODING.md.
Section depth is limited to three, so Naming is now a peer of Standards and Braces and Hints, Warnings, and Errors are now part of Standards.
When a command is given, argparse parses the options of the command into a new namespace and copies every value, including defaults, over the values the main parser set from options before the command. As a result, `test.py --vm=u24 vm-build` ran vm-build with --vm=none.
Move the command to the front of the argument list before parsing so the command parser parses every option. An option before the command that the command does not define is now an error instead of being ignored.
Time option values are stored in milliseconds but the table of time values in the generated parse rules was declared unsigned int, which limits a value to about 49.7 days. A default or allow-range bound larger than that overflows and fails to compile. The parser already reads time values, range bounds, and option values as int64_t, so declare the table as int64_t to match the table of size values.
All existing time option values fit in either type, so none of them change.
The test container is built with the same user name and id as the user on the host so files created inside it are owned by that user. Debian's adduser only accepts a name that matches NAME_REGEX, which excludes a dot, so vm-build failed on the Debian-based images for a host user named e.g. first.last.
Pass the flag that skips the check, since the name comes from the account running the build rather than from outside. The flag was renamed from --force-badname to --allow-bad-names and Ubuntu 22.04 ships a version that only knows the old spelling, so the spelling is taken from adduser --help when the container is built rather than hardcoded.
The archive-push-batch-size option converts the batch size to a segment count using the size of the first file in the ready list, but that list also contains backup and timeline history files. PostgreSQL does not sync backup history files, so a cluster copied from a storage snapshot may contain one with zero size, which caused every asynchronous run to error and prevented any WAL from being archived.
Use the size of the first ready WAL segment instead and skip the batch size when no segment is ready, since the remaining files are small. A zero-size WAL segment is still an error.
The new checksum-page-error option determines whether invalid page checksums are reported as a warning or an error. It is available for the backup, restore, and expire commands.
For backup the option defaults to false, which preserves the prior behavior of emitting a warning for each file that contains invalid page checksums. A warning with the total number of checksum warnings is now emitted at the end of the backup so they are less likely to be missed in the log. When the option is enabled the backup terminates with an error as soon as an invalid page checksum is found. Page misalignment is handled the same way since it prevents checksum validation.
For restore the option defaults to true. When the option is enabled the restore terminates with an error as soon as a file with invalid page checksums is restored. Since this happens before pg_control is renamed, the cluster cannot be started. When the option is disabled a warning is emitted for each file that contains invalid page checksums and the total is reported at the end of the restore. Previously the restore command did not report invalid page checksums at all.
For expire the option defaults to true. If the backup that would become the oldest retained backup contains invalid page checksums then expire terminates with an error before removing anything. When the option is disabled expire proceeds and emits a warning if the oldest retained backup contains invalid page checksums. An adhoc expire proceeds in either case but still emits the warning.
Listing with a repository target time uses the ListObjectVersions API but pagination read NextContinuationToken from the response, which is only returned by ListObjectsV2. A version listing truncated at the maximum key count failed with "unable to find child 'NextContinuationToken':0 in node 'ListVersionsResult'", so a repository with more than a thousand versions under a prefix could not be read at a target time.
Paginate the version listing with NextKeyMarker and NextVersionIdMarker instead. NextVersionIdMarker is missing or empty when the listing is truncated at a common prefix, in which case the next page starts at the key after the key marker, which is what is needed.
Build the query for each page from the base query so a marker set for one page is not reused for the next.
The mock response used to test version listing pagination returned a continuation token rather than markers, so the error was not caught.
The parsed key JSON, the file read buffer, and intermediate strings were allocated in the object context rather than a temp context, so they stayed allocated for the life of the storage object.
Format 6 derives encryption keys from the passphrase with SHA-256 rather than SHA-1. Older repositories keep SHA-1, since the migration does not rewrite the files they already encrypted. The digest is picked when a passphrase is generated rather than at each use, so it is stored with the passphrase in CipherSpec and defaults to SHA-256.
A passphrase outlives the format of the file it is stored in, so the format cannot decide how the key is derived. Info files at format 6 store cipher-digest beside cipher-pass, and a passphrase stored without one derives with SHA-1. stanza-upgrade writes back the digest the passphrase already had, so archives and backups from before the migration still load. Only a new passphrase gets SHA-256: the one for a stanza created at format 6, or for the manifest of a full backup written at it.
The info file has the same problem in reverse. Its format sits inside the content that the passphrase decrypts, so a reader would have to decrypt the file to find out how to decrypt it. Format 6 puts a plaintext header in front instead: PGBR, the three-digit format, and a spare byte for later. It goes where the OpenSSL Salted__ magic used to go and is the same eight bytes, so the salt still starts in the same place. A file that starts with the magic came from format 5, the only format older than the header. The header format is checked before anything is decrypted, then compared against the format found in the content, so a file stitched together from two others is rejected rather than half read.
The format constants move out of version.h into a new common/format, along with repoFormatValidate(), repoFormatDigest(), and CipherBlockFormat. That filter reads the header, builds a block cipher with the digest the format calls for, and streams the rest of the file through it. The block cipher no longer knows anything about repositories: it takes the digest from the spec and takes what sits in front of the content as a parameter.
Verify checksums info files as stored rather than as decrypted, since the header has to come off before decryption can start. A file and its copy are written from the same bytes, so comparing the two still works. In an encrypted repository created by a Perl version the file and its copy are separate encryptions, so their checksums differ and verify reports it at detail level. A log line is better than a separate code path for repositories with seven year old backups.
Response parts were mapped to sub-requests by iterating the parts, so when the service returned fewer parts than were requested the remaining sub-requests were silently treated as complete. Azure fails the entire batch when it is unable to parse the request and reports the failure as a single response part, so only one file per batch was removed while the command reported success.
Iterate the sub-requests instead and retry any that have no response part, just as parts that report an error are retried. More response parts than sub-requests is now an error.
The version header was not set for SAS since a SAS specifies a version in the sv parameter. That is sufficient for normal requests but batch requires the version to be set explicitly and rejects the entire batch with 400 InvalidInput when it is missing.
Azure reports the rejection as a single response part with no content-id, so it appeared that one sub-request had failed rather than that none had run. That part was retried individually and the remaining files in the batch were silently not deleted, leaving expired backups and WAL in the repository while expire reported success.
When a read is retried the driver is freed and recreated but the filter group is not set on the new driver. For drivers that require the filter group (i.e. remote) this causes a segfault when the read is opened because the filter group is NULL.
This was introduced in a953aa74b, which changed retry to recreate the driver rather than reopen it, added the filter group callback that is only set when the read is created, and replaced the driver retry flag with the storageFeatureReadRetry feature, which remote storage inherits from the repository host. Before that commit retry was never enabled for remote reads.
Even with the filter group set the retry could not work because the filters are passed to the remote when the read is opened so they cannot be run again, and bytes read are counted after the filters have been applied so the offset to restart the read cannot be determined.
Read retry is only enabled for object stores so this happened when the repository was on an object store accessed via a repository host. The repository host still retries the object store read, where the filters are applied above the driver and the retry works correctly, so the only retry lost is a redundant second attempt by the client.
The result was zero padded to the width of the precision, which left no room for the integer digit or for the extra digit added when rounding. A quotient less than one lost its leading zero, e.g. .87 rather than 0.87, and when the digits did not fill the precision the result was an empty string. With no precision at all every quotient less than one was empty, so 9/10 rendered as nothing rather than 1.
Pad to the precision plus a digit for the integer part, and one more when the rounding digit was added. Output for a quotient of one or more is unchanged, so strSizeFormat() is not affected.
The bundling decision and the process queue ordering used ManifestFile.size, which manifestBuildIncr sets to the prior file size when block incremental values are preserved. A file that grew since the prior backup was measured by its prior size, so a file well above repo-bundle-limit could be stored in a bundle and sorted with the small files rather than ahead of them.
Use sizeOriginal instead since that is the size found during manifest build and therefore the number of bytes that will be copied. This matches the rest of backup processing, which already uses sizeOriginal for the copy limit, the bundle size accumulator, and progress.
A block incremental file truncated to zero by PostgreSQL during the backup was stored as a block map with no blocks, i.e. a single byte containing only the version flag. Every subsequent backup that referenced the file failed with "unexpected eof" while reading the map and continued to fail until the next full backup.
Zero-length files detected during the copy were already stored as truncated rather than copied, but only when the file was in a bundle. Files larger than repo-bundle-limit get their own repository file and so missed the check.
Never write a block map with no blocks and store zero-length block incremental files as truncated whether or not they are bundled.
Fil-C 0.684 adds libxml2 and libpq to the /opt/fil distribution, so neither has to be built here, and it fixes the bug where a volatile local declared without an initializer lost its value across longjmp, so the test source no longer has to be patched.
That leaves nothing to build, only the distribution to download, which is what the container registry cache was holding anyway. Caching it saved the libxml2 and libpq builds rather than the transfer, so the cache, the registry login, and the write permission on packages all go with them.
The job is no longer informational since it passes.
The repository format has been fixed at 5 since 1.00. It was a compile-time constant written into every info file and manifest, and any mismatch at load was a hard error. A version could only read the format it wrote, so introducing a format meant a flag day where every repository had to be recreated. 1.00 was that flag day and this branch is so the next format does not need one.
A file is readable when its format is between REPOSITORY_FORMAT_MIN and REPOSITORY_FORMAT_MAX. A version can read a format it does not write and a repository can hold both while older backups and archives expire.
The format is requested with repo-format, which is repo indexed and valid only for stanza-create and stanza-upgrade. It is command line only so a format left in the configuration cannot migrate a stanza as a side effect of an upgrade run for another reason, such as a PostgreSQL version upgrade. The default stays 5.
stanza-upgrade --repo1-format=6 rewrites the two info files and nothing else, so a repository of any size migrates in the time it takes to write them. A downgrade is refused, since the info files are what stop a version from reading a format it does not support. The two files are saved separately, so an upgrade interrupted between them leaves a mismatch that running the upgrade again repairs and that every command reading both files now reports.
A backup set adopts a new format only at a full backup. A prior backup is a candidate for diff or incr only if it is at the format new backups are written with, and a resumable backup at another format is discarded, so a backup set is never mixed and its format can be read once rather than per file. Migrating therefore costs one full backup per stanza.
Once the info files are at format 6 a version that supports only format 5 cannot read the stanza at all, including the backups still at format 5. Such a version fails at info load with "expected format 5 but found 6" before touching any data.
The ports assigned to the tests started at 34343, which is inside the ephemeral port range on Linux (32768-60999). A test that binds a port in that range can fail when the port is already in use as the source port of an outgoing connection, since SO_REUSEADDR only allows the bind when the conflicting socket is in TIME_WAIT.
Assign the test ports from below the ephemeral port range instead. No module requests more than eight ports, so reducing the maximum per test from 768 to 64 leaves ample headroom and allows the range for all 32 tests to fit in 30720-32767, i.e. ending exactly where the ephemeral port range begins.
The bogus port was 34342, which is inside the ephemeral port range on Linux (32768-60999). A connection to that port can be assigned the same port as its source, which results in a TCP simultaneous open where the socket connects to itself and the connection succeeds rather than being refused. Since the socket client retries for the duration of the connect timeout each open makes many attempts, so the tests that expect a connection to be refused failed regularly.
Move the bogus port below the ephemeral port range where the kernel will never assign it, and build the expected error messages from the macro so the port only needs to be defined in one place.
4fb8a0ec passed arg_unity as c_args to the build-code and doc executables so a unity build of those would not fail on unused functions. 59bd2cd9 rewrote both in Python and removed the targets, leaving the variable assigned and never read.
The lint job passes -Wno-unused-function on the command line where it is needed, so nothing has to take its place.
Unity is how two of the checks are built rather than what any of them look for, and the job had already grown a check that does not need a unity build at all. lint describes the whole set and pairs with the code-format job below it.
Drop debug from the unused parameter build. Asserts are compiled out of a release build, so a parameter that only an assert reads is reported along with a parameter that nothing reads at all. Both are marked the same way, with a void cast that shows ignoring the parameter is intended.
Add libssh2 so the SFTP driver is built. Without it HAVE_LIBSSH2 is undefined and the driver is preprocessed away, so none of these checks were looking at it.
The assert is the only thing that touches these parameters, so once it is compiled out of a release build nothing references them at all. Replace the log entry with a void cast, which keeps the assert doing the real work and drops a log entry that never varies.
storageSftpRemove() param: ASSERT(!param.errorOnMissing) rejects an option the driver does not implement.
dbOpenProtocol(), storageRemoteFeatureProtocol(), storageRemoteReadProtocol(), storageRemoteReadMultiProtocol(), and storageRemoteWriteCloseProtocol() param: ASSERT(param == NULL) verifies the client sent no parameters for a command that takes none, so logging param: null adds nothing.
ioSinkProcess() this, input, and output: the sink discards everything and reads none of them. ioFilterProcessInOut() already checks this and output before the call, but ASSERT(input != NULL) records that the sink is never invoked with a NULL input, since ioFilterDone() reports done as soon as flushing begins for a filter with no done or inputSame function.
storagePosixInfo(), storagePosixLinkCreate(), storagePosixPathSync(), and storagePosixRemove() this: read only by ASSERT(this != NULL). objNameToLog() renders the object as a constant {StoragePosix}, and the function name in the same trace line already says which driver is running, so the entry carried nothing.
Follow up to 07f1cd6c, which did not find these because an assert counts as a use in a debug build.
pgControlCrcUpdate() pgVersion: selected the CRC algorithm. 48f511da removed the pgVersion > PG_VERSION_94 test along with support for 9.4, leaving only ASSERT(pgVersion != 0).
backupBuildIncr() infoBackup: supplied the cipher passphrase via infoPgCipherPass(infoBackupPg(infoBackup)). b9b58a17 replaced that when it started passing a cipher spec, leaving only ASSERT(infoBackup != NULL).
The FUNCTION_LOG_PARAM*() macros reference each parameter they log, so the compiler counts that as a use and cannot report parameters the function never touches. Append overrides to debug.h that compile the macros out and build again. Nothing is lost by dropping the parameter logging since this build is never run.
Enable debug so asserts are not compiled out, which would report parameters used only by an assert, and disable unused function warnings since the *ToLog() functions have nothing left to call them.
c4fc1b5c and 07f1cd6c cleaned up everything this reports.
Each of these was left behind when the code that used it was changed or removed. The compiler does not report them because the FUNCTION_LOG_PARAM*() macros count as a use.
* backupFilePut() backupData: never used, present since 1f2ce45e added the function.
* backupJobResult() bundle: decided whether the job key held a bundle id. efc09db7 replaced that with a check on the key variant type.
* restoreRecoveryWriteConf() manifest: found the pgdata path to get the owner for chown(). 3ff88ffb removed the chown().
* restoreRecoveryWriteAutoConf() manifest: read backupOptionOnline to decide whether to write recovery.signal. 9cceed6a replaced that condition.
* exitSafe() signalType: built the "terminated on signal [SIGxxx]" message for command end. e13c4629 moved that logging into the signal handler.
* cmdEnd() code: selected the "completed successfully" message at command end. 6361a061 switched the test to errorMessage == NULL.
* infoPgSet() type: set the catalog/control versions and validated the type. 0a96764c removed both. 77c721eb restored the catalog version logic using this->type, which the object already stores from infoPgNew(), so the parameter stayed unused.
* walSegmentNext() pgVersion: skipped the FF segment for PostgreSQL < 9.3. 77c721eb removed that along with support for 9.0/9.1/9.2. walSegmentRange() and queueNeed() passed pgVersion only to reach walSegmentNext(), so remove it from those as well.
* tlsSessionResultProcess() errorSys: passed errno to THROW_SYS_ERROR_CODE() for syscall errors. 08242ee6 removed that throw. The errno read there was not informative (usually 0), and the capture in tlsSessionResult() goes with it.
* storageWriteSftpNew() user, group, timeModified, truncate: passed to the write interface initializer. a953aa74 replaced the per-driver interface with a shared static, so the driver stopped passing them. storageSftpNewWrite() is the only caller and asserts that user and group are NULL, timeModified is 0, and truncate is true, so the values were fixed.
Also remove the common/user.h include from the SFTP write driver, unused since 87087fac added it.
archiveAsyncErrorClear() ignored archiveMode and always removed status files from the out queue, so the call from archive-get named files in the push queue rather than its own.
There is no user visible change. archive_mode=always is rejected by checkDbConfig() so archive-push and archive-get do not run on the same host, and the get queue is cleared by queueNeed() just before the call in any case.
Use archiveAsyncSpoolQueue() as archiveAsyncStatus() and archiveAsyncStatusErrorWrite() do.
The read drivers ignore block because ioReadInternal() loops until the output buffer is full and stops early itself when block is false. Object store path removal ignores recurse because there are no paths to recurse into.
Logging these parameters implies they have an effect on the function. Cast them to void instead, which is the pattern already used for the unused param structs in the storage drivers, and which allows a build with FUNCTION_LOG_PARAM*() compiled out to report parameters that no code touches.
The error for an invalid value in a block incremental map named the option with cfgParseOptionKeyIdxName(), which expects a key index, but the callers passed a group index. A repository configured as anything but repo1 was named as repo1. Use cfgOptionIdxName() instead, which takes the group index and maps it to the key.
The errors for the checksum size map also named the size map option since that option id was passed by mistake.
These options are internal so neither error was user-facing.
cfgOptionIdxName() read the option group from the config, which only stores it for options that are valid for the current command. An option that was not valid got the name without an index, and an index above the first failed an assertion.
Read the group from the parse rules instead since they are not specific to a command.
cfgExecParam() needs this because it loops through the options valid for the command being executed, which is not always the command being run. It also read the index total from the config, so for an option that was not valid only the first index was searched and a replacement for any other index was missed.
This is not a live issue since every indexed option of the commands passed to cfgExecParam() is also valid for the command being run.
The runner mirrorlist puts azure.archive.ubuntu.com ahead of archive.ubuntu.com and it has been serving at a crawl. apt abandons a mirror that has stopped sending but not one that is merely slow, so apt-get update either takes tens of minutes or hangs until the workflow timeout.
Add an action that drops that mirror, bounds the connection level waits, and refreshes the index under a total time bound with retries, which is the part apt has no setting for. The settings are written to /etc/apt/apt.conf.d so the installs that follow pick them up as well.
Use it in every job that installs packages on the runner, including the ones that reach apt through ci.py. The refresh moves out of ci.py since rewriting the mirror list the runner image ships is about the runner rather than about the run. It was best effort there and is now fatal after three bounded attempts, since an index that cannot be refreshed means the mirror the install downloads from is unusable as well.
The jobs that do their work in containers use the package sources of the image and are not affected.
A block incremental file stores its block map at the end of its data and the location of the map is recorded in the manifest, so when the data at that location is not a complete map the error comes from the map parser, which knows nothing about where the map came from. In #2829 that leaves the user with "unexpected eof" and no indication of which file failed, where the map was expected to be, or whether the data is short or simply not a map. The same signature in #2489 could not be diagnosed for the same reason.
Rethrow errors from the read and the parse with the manifest file, repository path, offset and map size. Also measure the bytes read from the repository, before decryption, and error when they are fewer than the manifest expects, since a short read would otherwise surface as a parse error that gives no hint that data is missing. This does not address whatever causes the mismatch, but it makes the next report actionable.
Move the read into backupFileBlockIncr() to keep the copy loop readable. The map is read in its own memory context with the filter created in the prior context, so the map buffer is freed once the filter has parsed it rather than living until the file is done.
Fil-C is a memory-safe implementation of C where pointers are checked on every access, so an out-of-bounds access, a use-after-free, or a type confusion panics instead of doing damage. It found two out-of-bounds reads that gcc, clang and valgrind missed.
Linking to code that was not built with Fil-C is severely restricted, so the /opt/fil distribution is used for the libraries it ships and libxml2 and libpq are built here with the same compiler. That takes minutes and produces the same thing every time, so the prefix is pushed to a container registry rather than actions/cache, which is not branch-scoped and so is shared across the short-lived branches this project works in, as the ccache already is. It is saved before the tests run so a failing test does not stop it being cached and a failed build is never cached.
Two things have to be turned off for libpq. Its exports list names the plain symbols but Fil-C emits pizlonated_-prefixed ones, so the version script hides every entry point, and -fvisibility=hidden hides the prefixed symbols as well because they do not inherit the default visibility PGDLLEXPORT puts on the declaration.
--gcc-toolchain keeps the host gcc out of the search path so a library that was not built with Fil-C cannot be picked up, and PKG_CONFIG_LIBDIR does the same for pkg-config. It also keeps meson from finding the libbacktrace gcc ships, which matters because the Fil-C runtime does not implement _Unwind_Backtrace. TZDIR is set because the glibc in /opt/fil looks for the timezone database under its own prefix, where there is none, and falls back to UTC without saying so.
The too large allocation test is skipped since Fil-C either satisfies an allocation or treats it as a hard error, so malloc() never returns NULL. A volatile local declared without an initializer loses its value across longjmp, which is a Fil-C bug reported upstream, so that declaration is patched in the job rather than in the test.
The packed file stored user and group by casting the interned String pointer to an integer and casting it back on unpack, spending five or six varint bytes per file to name one of a handful of values. The reference field in the same function already stores an index into referenceList with strLstFindIdxP() and reads it back with strLstGet(), so owner now does the same.
That requires the owner list to only ever grow at the end, which reference already depends on. Nothing sorts it or removes from it, so an owner seen for the first time cannot disturb files already packed.
The performance test gave every file the same owner, which made it the default so no owner was ever encoded and this code was never reached. It now spreads a few owners with one repeated, leaving that one as the default. Timings are unchanged.
The container jobs cover what pgBackRest is normally built with, so what is tested here is whether it builds and runs somewhere else. gcc is the baseline and clang is a second opinion from a different compiler, and each runs with the optional libraries present and absent, which catches code that only compiles when a feature is enabled.
The optional libraries are removed rather than just left out of the install, and what pkg-config reports is checked afterwards, since the runner image is not a fixed set of packages and one of them reappearing would quietly turn the minimal variation into a second full one.
