Repository format 6 and archive key rotation are not complete enough to release. Internal options are left out of the documentation and help but are still accepted, which allows format 6 to be tested with this release.
repo-gcs-key-type=web-id authenticates using an external account credential file (https://google.aip.dev/auth/4117) as generated by gcloud iam workload-identity-pools create-cred-config. The file is specified by repo-gcs-key and defines the audience, the STS token url, and the file that contains the web identity token. The web identity token is exchanged at the STS endpoint for an access token that authenticates the federated principal directly, without a service account key.
The web identity token file is reloaded on each exchange to pick up rotated tokens, e.g. Kubernetes projected service account tokens.
Service account impersonation (service_account_impersonation_url) is not supported. Only the file credential source is supported, and when credential_source.format is specified the type must be text.
Authentication token responses that are not JSON (e.g. an HTML error page) or that contain a nested error object are now reported with httpRequestError() rather than failing while parsing the response. This applies to all GCS key types that request a token.
At format 6 the repository can hold more than one encryption key. This is used to rotate the archive key, which is currently the only bulk data key that never changes.
Keys are stored by id in the new CipherSpecMap, with one key marked current for encrypting new files. At format 6 the cipher section of an info file stores every key by id along with the digest it derives with, e.g. cipher-pass={"1":{"digest":"sha256","key":"..."}}, and archive.info also records the current key id in cipher-pass-current. Format 5 files keep their single key, which is loaded as the key with no id.
archive-push writes the id of the archive key it used into the header of each WAL file. The readers use that id to select the key: archive-get, verify, the restore timeline read, the backup archive copy, and repo-get. A pusher that loaded archive.info before a rotation keeps working since each file records the key it was encrypted with. The filter that writes the header now also works through a repository host.
stanza-create for format 6 creates the first archive key with id 1. stanza-upgrade from format 5 keeps the existing archive key as id 0 so WAL already archived stays readable, and adds a key at id 1 for new WAL.
The new repo-cipher-rotate option sets how old the current archive key may be before expire replaces it, with a default of 90 days and a minimum of 15. Expire adds the new key, makes it current, and records the time of the rotation in archive.info as cipher-pass-rotate. At format 5 the option has no effect and warns only when set explicitly.
Expire now saves archive.info on every run. Backup did this only to refresh the timestamps for object store lifecycle rules and no longer writes archive.info, so with expire-auto disabled the timestamps are refreshed only when expire runs.
Time option values are stored in milliseconds but the table of time values in the generated parse rules was declared unsigned int, which limits a value to about 49.7 days. A default or allow-range bound larger than that overflows and fails to compile. The parser already reads time values, range bounds, and option values as int64_t, so declare the table as int64_t to match the table of size values.
All existing time option values fit in either type, so none of them change.
The new checksum-page-error option determines whether invalid page checksums are reported as a warning or an error. It is available for the backup, restore, and expire commands.
For backup the option defaults to false, which preserves the prior behavior of emitting a warning for each file that contains invalid page checksums. A warning with the total number of checksum warnings is now emitted at the end of the backup so they are less likely to be missed in the log. When the option is enabled the backup terminates with an error as soon as an invalid page checksum is found. Page misalignment is handled the same way since it prevents checksum validation.
For restore the option defaults to true. When the option is enabled the restore terminates with an error as soon as a file with invalid page checksums is restored. Since this happens before pg_control is renamed, the cluster cannot be started. When the option is disabled a warning is emitted for each file that contains invalid page checksums and the total is reported at the end of the restore. Previously the restore command did not report invalid page checksums at all.
For expire the option defaults to true. If the backup that would become the oldest retained backup contains invalid page checksums then expire terminates with an error before removing anything. When the option is disabled expire proceeds and emits a warning if the oldest retained backup contains invalid page checksums. An adhoc expire proceeds in either case but still emits the warning.
Format 6 derives encryption keys from the passphrase with SHA-256 rather than SHA-1. Older repositories keep SHA-1, since the migration does not rewrite the files they already encrypted. The digest is picked when a passphrase is generated rather than at each use, so it is stored with the passphrase in CipherSpec and defaults to SHA-256.
A passphrase outlives the format of the file it is stored in, so the format cannot decide how the key is derived. Info files at format 6 store cipher-digest beside cipher-pass, and a passphrase stored without one derives with SHA-1. stanza-upgrade writes back the digest the passphrase already had, so archives and backups from before the migration still load. Only a new passphrase gets SHA-256: the one for a stanza created at format 6, or for the manifest of a full backup written at it.
The info file has the same problem in reverse. Its format sits inside the content that the passphrase decrypts, so a reader would have to decrypt the file to find out how to decrypt it. Format 6 puts a plaintext header in front instead: PGBR, the three-digit format, and a spare byte for later. It goes where the OpenSSL Salted__ magic used to go and is the same eight bytes, so the salt still starts in the same place. A file that starts with the magic came from format 5, the only format older than the header. The header format is checked before anything is decrypted, then compared against the format found in the content, so a file stitched together from two others is rejected rather than half read.
The format constants move out of version.h into a new common/format, along with repoFormatValidate(), repoFormatDigest(), and CipherBlockFormat. That filter reads the header, builds a block cipher with the digest the format calls for, and streams the rest of the file through it. The block cipher no longer knows anything about repositories: it takes the digest from the spec and takes what sits in front of the content as a parameter.
Verify checksums info files as stored rather than as decrypted, since the header has to come off before decryption can start. A file and its copy are written from the same bytes, so comparing the two still works. In an encrypted repository created by a Perl version the file and its copy are separate encryptions, so their checksums differ and verify reports it at detail level. A log line is better than a separate code path for repositories with seven year old backups.
The repository format has been fixed at 5 since 1.00. It was a compile-time constant written into every info file and manifest, and any mismatch at load was a hard error. A version could only read the format it wrote, so introducing a format meant a flag day where every repository had to be recreated. 1.00 was that flag day and this branch is so the next format does not need one.
A file is readable when its format is between REPOSITORY_FORMAT_MIN and REPOSITORY_FORMAT_MAX. A version can read a format it does not write and a repository can hold both while older backups and archives expire.
The format is requested with repo-format, which is repo indexed and valid only for stanza-create and stanza-upgrade. It is command line only so a format left in the configuration cannot migrate a stanza as a side effect of an upgrade run for another reason, such as a PostgreSQL version upgrade. The default stays 5.
stanza-upgrade --repo1-format=6 rewrites the two info files and nothing else, so a repository of any size migrates in the time it takes to write them. A downgrade is refused, since the info files are what stop a version from reading a format it does not support. The two files are saved separately, so an upgrade interrupted between them leaves a mismatch that running the upgrade again repairs and that every command reading both files now reports.
A backup set adopts a new format only at a full backup. A prior backup is a candidate for diff or incr only if it is at the format new backups are written with, and a resumable backup at another format is discarded, so a backup set is never mixed and its format can be read once rather than per file. Migrating therefore costs one full backup per stanza.
Once the info files are at format 6 a version that supports only format 5 cannot read the stanza at all, including the backups still at format 5. Such a version fails at info load with "expected format 5 but found 6" before touching any data.
The help describes the values an option allows with an allow list. An option that allows the same values as another option can inherit that list instead of describing the values again.
This only worked for a list documented by an option on its own. A list documented under a command could not be inherited, so an option that only the commands document has its values written out once per command. The output option describes the same two values under manifest, repo-ls, info, verify, and version.
Record the list each command documents as well, and let inherit name the command to take it from. The command has to be named because more than one command can document the same option and the lists need not match. The type option allows full, diff, and incr for backup but nine values for restore.
A list is still checked against the values the inheriting option or command allows, so it can only be inherited where both allow the same values.
In the test declaration set is now taken by backup and restore so that backup can describe the values and restore can inherit them.
A value that varies by version was read inside a function, so two versions got a rendering of it that differed only by that value. pgInterfaceControlVersion() returned PG_CONTROL_VERSION, pgInterfaceWalIs() compared XLOG_PAGE_MAGIC, and pgInterfaceControlIs() compared both PG_CONTROL_VERSION and CATALOG_VERSION_NO, so none of them could ever be shared no matter how little of the version they actually depended on.
Include them in the interface struct instead. A version block captures each as a constant named for it, since a value only has a value inside the block and the struct is built after every block has undefined what it defined. A define cannot do this because it is expanded where it is used rather than where it is made, by which time the value it was given is gone. The constant also names the member it is assigned to, so adding a value is a change to the header and to nothing else.
Each value is captured by a macro of its own, so the analysis that decides which versions can share a function decides the same for a value. One that does not vary between two versions is captured once for both and named for the range it covers, e.g. pgInterfaceControlVersion13_16, which makes what a version shares readable where it is assigned rather than something to work out from the vendored header. The control version is the same from 13 through 16 and the crc offset for 15 and 16.
The functions that only compared a value go with them. Matching pg_control against an interface is now a comparison of the fields that are at the same offset in every version, which pgControlCommon and pgWalCommon already read to report a version that is not supported, and the WAL magic is compared the same way. The comparison is rendered by the generator in the form that fits the versions rendered, so the range that an unreleased version needs is only rendered while there is one and there is never a comparison that no supported version can fail.
Whether a version has been released is set from the declaration, which is where it is stated, rather than derived in the header.
The harness shims the interface so it writes with the types that interface declares rather than declaring them again, and takes the defaults a test does not ask for from the interface the code matches on rather than from a second copy of them.
This brings the binary interface to 22 rendered functions from 62 and the harness to 12 from 32.
Every supported version got a rendering of every interface function, which is 77 functions for the seven macros the binary interface declares and 33 for the three the harness declares. Many of them compile to identical code, since most of the vendored types do not change in a given release. The functions are trivial enough today that the duplication costs little, but more ambitious ones would multiply by the number of versions.
Read the vendored header as a chain of version branches per type and define, resolve each of them for each version, and take the transitive closure of what an interface macro reaches. Two versions share a rendering when every entity in that closure resolves to the same branch for both, since that is what makes the code the compiler sees the same. Following the closure rather than the names a macro uses directly is what makes the comparison sound: ControlFileData is unchanged from 13 to 14, but CheckPoint is embedded in it and does change, so the layout differs.
The oldest version of a group renders the function and the struct points the rest at it, so adding a version later never renames what is already there. A rendering more than one version shares is named for the range it covers, e.g. pgInterfaceWal14_19, and the block it is rendered in says which versions share it. This brings the binary interface to 62 functions and the harness to 32.
A macro that does not take a version is now read as a helper the function macros are written with rather than as a function of its own. The dependency scan follows a helper to the types it names, so declaring one is now possible, where before it would have been rendered as a function.
The harness had a file per version that did nothing but expand a macro for it, a block of declarations for the functions those files defined, and the struct that dispatched to them, all maintained by hand, so adding a PostgreSQL version meant adding a file and three more entries to keep in step. The interfaces the binary uses are generated from the same versions, so the harness generates its own the same way with 'build.py postgres-harness'.
What differs between the two interfaces is a declaration: the header the macros are read from, how the generated code includes it, the prefix of the functions the macros define, and where the interface is rendered. The versions, the vendored types that are renamed for each of them, and the defines that are undefined after each one are the same, so one generator renders both.
The harness interface is one translation unit with the vendored header included again for each version, the way the binary's is, rather than a translation unit per version. The functions are static, so the declarations and the pragma that silenced the missing prototypes go away with the files, and they are named for the version the way the binary's are, e.g. 19 rather than 190. The harness macros are renamed to say which function each one defines, since the generator names a function after the macro that defines it, and the macro that expanded all of them for a version is gone because the generated file expands each one itself. The header they are declared in loses its include guard and everything that does not vary by version, since it is now included once per version and the module that expands the macros includes the rest.
The system id constants are generated into the harness header rather than into a file of their own, so the header still holds everything the harness declares. The defines are replaced where they are, which is what the version in src/version.h is already generated by. Every version gets the offset form that only two of them had, since a block that reads the same for every version is easier to follow than one that does not.
The interface is generated rather than committed, the same as the interface the binary uses, so a test run generates it and the repository copy duplicates it for a unit build to compile.
main.c included the generated help data and passed it to cmdHelp() as a buffer, so no unit build ever compiled it and the help test had to run the generator and load the packed help from a file to have anything to render. Including it in help.c puts the data in the module that reads it and lets the test render exactly what the binary carries.
The generated file is added to the files the repository copy needs, since git does not list it, and the test command generates it along with the PostgreSQL interfaces. Those are the two files that are generated rather than committed, and a unit build needs both from the copy because it does not run the code generation the build does. The file is regenerated at the start of every run, so the test still runs against the current declarations without generating them itself.
cmdHelp() and helpRender() no longer take the data as a parameter and help.h no longer needs the buffer type. The help-data generator command is removed along with its only caller.
The export-ignore rules applied to every archive git produced, not just the one meson dist builds, so the source archives GitHub generates for a tag were missing the code generation tooling, the declarations it reads, and the documentation source. That left them unbuildable even for someone willing to run the tooling, which was never the intent -- the rules existed to keep the distribution tarball to what a packager needs.
Remove the rules and have dist.sh trim the distribution root instead, keeping the same set: src, the root meson files, LICENSE, and the smoke test that packagers use to verify a build. The trimming runs before the generated files are added so the documentation source is replaced by the rendered man page and HTML rather than shipped alongside them.
Trimming in a script rather than with export-ignore means git no longer guarantees the result, so check the full tarball path list in CI rather than just the generated files. Everything under src collapses to the directory since it changes too often to list, except the generated code, which stays listed so a failed generation is still caught. This catches a keep list that drifts as well as a missing dist marker, and the zero-size check widens to every file since none that ships is empty.
The comment and the error message in build/dist.sh still tell the user to run doc/release.pl, which has not existed since the documentation tooling moved to Python. Point both at doc/release.py, which is what the dist job runs to render the man page and HTML that meson dist copies into the tarball.
The values an option allows are declared in config.yaml and were written out again by hand in reference.xml as an ordinary list, so nothing held the two together. A value added, removed, or renamed in the declaration left the documentation describing something else and only a reader would find it.
An allow list is now marked as what it is and its values are checked against the declaration. Where each value needs an explanation allow-item gives it and caption says what the list is a list of, which renders the sentence that was written above each list by hand. Where the values speak for themselves the tag is left empty and the values are rendered from the declaration into the sentence around it, so they are not written out at all. Where an option allows what another option allows the list is inherited, the way the declaration inherits the values themselves, which is what log-level-file, log-level-stderr, and pg-host-type do.
Documenting the list is required rather than optional, since a value a user cannot read about is a value they cannot use. That turned up four options that never listed their values. buffer-size and repo-sftp-host-key-hash-type render the sentence they already had from the declaration, and repo-retention-archive-type and repo-retention-full-type described a value per paragraph, which is now the list it always was.
The markup is resolved where the help is parsed, so the help in the binary and the documentation are both rendered from it and no renderer knows it exists. Everything else in the documentation renders exactly as it did, except for four introductions reworded to fit the caption.
Most of these are prose in a python docstring or comment wrapped a few characters wide, which black does not reflow, and macro continuation backslashes left stranded past the margin. The rest are long calls broken after the first parenthesis, section separator comments with too many dashes, and long expectation strings split along what they hold. A few long shell commands in the workflows are wrapped with continuations.
hrnTestResultDiff() had a continuation backslash left over from when it was a macro, which spliced the closing brace onto the line above it. It is removed.
The linter that enforces the limit follows, so this commit is only the reformatting and can be ignored by blame.
Both are internal and in the past the integration tests used them. They were left in place for possible administration, investigation, and testing but in fact those uses have not materialized. The repo format has moved on so repo-put is now actively dangerous (it always was, really).
Without cipher-pass, repo-get always determines the passphrase from the repository: the repo passphrase for the info files, and the passphrase each of those carries for the files below it. That was already what happened whenever the option was not given, so what is left is a single path rather than two.
The tests now write the repository directly with the storage harness, which is all the get tests needed from repo-put.
The three items the index lists were written on it by hand, so every news item was written twice: once as the news and again as a date, a title, and a link on the index. Nothing kept the two in step, so an item added to one and forgotten in the other would go unnoticed by anything that builds the documentation.
The index asks for the items now and the build takes them from the news. The title on the index is the title of the item as the news writes it, so a title that changes on the news changes on the index as well, and the project name in one is shown there the way it is shown everywhere else.
A news item says when it was posted as an attribute written the way a date sorts rather than as the paragraph the item begins with, so the date is written once and rendered both where the item shows it and where the index lists it. An item that is more recent than the item before it is an error, since the index lists the items the news begins with.
The release notes render their date the same way, so the months are named in one place rather than in three.
The command reference described every option a command takes in full, under every command that takes it. Since most options are valid for most commands the same description was written out again and again: --config was described eighteen times, and the backup command alone ran to a hundred and thirty options, of which fifteen are actually about backup. The page came to 1.2 MB and reaching a command meant scrolling past everything before it.
An option is now described once. The options a command documents itself, which are the ones whose meaning depends on the command, stay with the command as they were. The rest are a list of links, each naming the option the way the heading at the other end names it, e.g. Repository Host (--repo-host), and pointing at the configuration reference where it is described. The page is 350 KB now.
An option that can only be given on the command line, e.g. --config or --stanza, is not in the configuration reference because it cannot be set in the configuration file, so there was nowhere for it to link to. These are described in a General Options section before the commands and each command links there. They are left out of the contents, since the contents list the commands rather than the options each of them takes.
A link now renders the markup it holds rather than its text alone, so an option named in a list keeps the monospace it has in a heading. Every link written in doc/xml holds text alone and every page but the command reference is byte for byte what it was. A link may also hold text and markup at once when a document is normalized, since otherwise a hand-written document could not say what the generated one says.
The configuration reference is unchanged.
A ctrl-c raised KeyboardInterrupt out of main() and python printed the traceback, which is the whole path from main() down to the select() the harness happened to be waiting in and says nothing about a run that was stopped on purpose. Catch it and log "terminated on signal SIGINT" as the C does, exiting with the status the C exits with for TermError.
An unexpected exception is a bug in the tool so the trace is what gets it fixed, but the trace ends with the message that says what went wrong, which is where the message gets lost. Show the message first and keep only the innermost frames, since the frames above them are the path from main() into the tool and are much the same for every error. The frame count says how many were dropped and how to get them, and the complete trace, which is what python printed before, is shown at debug level the same way the C shows a stack trace only at debug level.
The rendering is error_trace() in common/error.py, beside ToolError, so every tool reports a bug in itself the same way. The CI wrapper asks for the complete trace since a run there cannot be repeated at debug level to get it, and the code builder handles the interrupt but not the trace since it has no log level to raise.
The definitions move out of common/vm.py into a vm section at the top of test/container.yaml, next to the container revisions that go with them, and are read and checked when the module is imported as they were before. The revisions move under a revision key to make room.
The declaration is read with the loader the other declarations use, so a version keeps the text it was written as rather than arriving as a float. A new yaml_map_dict() reads a mapping as a dict where a key is looked up by name, which also reports a key that was written twice instead of silently keeping the last one.
The constants that named each vm are gone. A vm is named by the string that --vm accepts, which is how the CI workflows and the documentation build already name them. A name used as a lookup is validated by vm_get(), and a name used in a comparison is protected by branch coverage since a comparison that can never match leaves a branch unhittable. VM_NONE, VM_ALL, and the os base and architecture constants remain because they name kinds rather than vms. VM_LIST is now the vms declared as default, so it is in declaration order.
The container build no longer tests a vm by name. What a vm installs and configures is declared instead: valgrind, epel, powertools, python, coverage-python, pg-repo-key, pg-repo-release, dnf-module, and ssh-rsa. The three near identical blocks that added the PostgreSQL repo collapse into one, which removes the uncoverable branch annotation on the fedora branch. Every generated script is unchanged, so no image is rebuilt.
The supported PostgreSQL versions are generated from build/postgres.yaml rather than declared a second time. A new bld_pg_version_list() reads just the versions, since a tool that only needs to know which ones are supported has no use for the interface that bld_pg_parse() scans the vendored headers for. The release flag comes with them, so the beta repository is added for whichever installed version has not been released rather than for a hardcoded vm and version.
A vm with no container declares only default and coverage-c, since every other key says how a container is built. The os base declared for none was never read, and the os the tests run on with --vm=none is whatever the host is rather than the debian it claimed. The PostgreSQL versions declared for it were never read either, since only an integration test runs against a version and those need a container.
A PostgreSQL version installed on a vm must now be one that build/postgres.yaml declares, and a version tested on a vm must be one installed there, both of which were only true by inspection before.
doc.pl and release.pl become doc.py and release.py and the Perl library in doc/lib/pgBackRestDoc is gone, which is 32 files and 11,059 lines replaced by about 3,850 lines of Python. Nothing in the project runs on Perl now.
doc.pl and doc.py merge into one tool rather than one running the other. doc.py built the reference documents and the preprocessed user guide, wrote them to doc/output/xml, and doc.pl read them back; they are now passed in memory and those files are not written at all. That also removes a duplication the split had hidden: the preprocessor added a document's variables to its own store and left the declarations in the file, so the Perl parsed them again into a store of its own and the same variable existed twice with different values.
Most of Doc.pm does not come across. Its parse step rebuilt the parsed tree into a structure of its own and its build step invented a second way to address that structure, so a document was reached through three accessors where the tree has one. What survives is about ninety lines in common/xml.py: a field is the text of a child that holds nothing else, the text of a node is the node itself for the tags that are text and a text child for the rest, and asking for a child rather than the list of them now reports a second one as the mistake in the document that it is.
External entities are expanded before the parser sees a document, since the standard library does not fetch them. The release list is assembled from one file per release this way, 122 of them, and a declaration that is never referenced is reported rather than leaving the file it names silently unread.
Nothing validated against a document type, so the document type declarations and the two dtd files are removed. A declaration that is not enforced does not get maintained. The two documents that assemble themselves from parts keep the declaration because the parts are declared inside it.
The release notes become a document the tool builds, like the command and configuration references, rather than something the renderer reaches into a custom module to produce.
A variable may still be evaluated rather than written out, but it is a Python expression against a short list of names rather than a Perl program. The three that ask for a fact about the environment keep doing so. The four that read the project version, the current stable version, and the release date are facts the tool computes, since working them out involved constructing library objects and reading the release list a second time from inside a variable body.
exec and user move down from test/lib to doc/lib because the documentation needs both, the project version splits out into config/project, and eval and var_store move to common now that the manifest uses them as much as the preprocessor. The version is built from the parts in src/version.h rather than read from the string generated from them, which is what the Perl did and what a release needs, since a release clears the development suffix and the string is only regenerated by a test run.
libxml-checker-perl is no longer installed on CI runners, in the development container, or in the documentation images, and the RHEL image no longer installs cpanminus and a toolchain to build XML::Checker::Parser from CPAN. The user guide never ran Perl, and the Debian image never had a parser and built the same guide.
Every module in doc/lib is declared in define.yaml and has full line and branch coverage.
Ubuntu 24.04 ships a user and a group at id 1000, so a build by a user with that id could not create the test user. The test image removes whatever holds the id before creating it, as it already did for the group, and the development container image removes both because adduser gives a new group the id of the user.
The coverage summary listed the python under build/lib as modules with mangled names, since it stripped a fixed src/ prefix from paths that no longer start with one. It now summarizes what is under src, which is what the metric page says it reports.
Deploying the website no longer removes the links at the top of the site or CNAME. The Perl used find -type f, which does not match a symbolic link, and the port used a test that follows one, which took out the guides kept under retired platform names. CNAME is a regular file and the Perl removed it every deploy, which was never intended.
The rendered documentation is byte identical to what the Perl produced, for every page, the manual page, and the generated markdown. The user guide was also built end to end under Docker on both documented platforms, and rendering it again from the cache that build wrote produces the same pages, which exercises the whole path rather than only what can be checked without containers. A release was then simulated from beginning to end.
The build tool in src/build becomes build/build.py with its library in build/lib, and the C part of the documentation tool in doc/src becomes doc/doc.py with its library in doc/lib. They migrate together because doc/src was built on src/build, linking its config and help parsing along with its extensions to the core types. doc.pl runs doc.py rather than building and running a C binary, and meson runs build.py for the code it generates as it builds.
The three libraries form the hierarchy the C include path did: build/lib is the base, doc/lib sits on it, and test/lib sits on both. They are implicit namespace packages so a package spans them, and a module lives in exactly one library and holds the union of what the tools need from it, since Python resolves a module name to a single file and cannot extend one the way the C included a core module and added to it. define.yaml declares which libraries each tool may import and the test runner puts only those on the path, so a test of the build library cannot reach the doc or test library any more than the build tool can.
The declarations move with the code. config.yaml, error.yaml, and postgres.yaml are now at the root of build, and help.xml becomes doc/xml/reference.xml since it is documentation that the help happens to be generated from, so the generator is told where it is rather than knowing. dist.sh moves to build as well. The documentation tool's own option declarations are gone because they existed only to generate the option parser for the C binary and doc.py parses its command line directly.
Three modules in src/build/common were never build tool code. exec, regExp, and string extend the core types for the test harness and the unit tests, which were their only users, so they move to test/src/harness. The json and xml extensions had no users outside the two tools and are removed with them, and the tests for both are trimmed to the core functions that remain.
The help unit test built its fixture by calling the generator's C functions directly, which was the last thing outside the tools that depended on them. It now runs build.py help-data to write the packed help and loads that, so it still runs against the current declarations without needing the generated C, which may not have been written when the test runs.
The manual page wrapping is a port of the wrapper the binary uses for its own help, so the manual page and pgbackrest help break lines the same way, including where a word longer than a line is left unbroken.
libyaml is no longer needed to build pgBackRest since the only thing that read yaml was the C generator. It is dropped from the development container and from every CI platform, and meson no longer looks for it. Code generation needs nothing compiled first either, so a test run no longer builds a generator before it can select the tests.
The common source list is merged into the pgbackrest source list since it existed only so the generator could be built from the same files, and the entries that were out of alphabetical order are put in order while the list is being rewritten anyway.
Every file the generator writes is byte identical to what the C wrote, for the project declarations and for the documentation's own. The documentation is checked the same way: the HTML rendered on this branch is byte identical to the HTML rendered on integration, every file of it, which exercises the whole pipeline on the real content rather than only the four files the tool writes.
Every module in both libraries is declared in define.yaml and has full line and branch coverage. The coverage list is keyed by file rather than by module name, since the same name in two languages is two files and one of them would have silently lost its requirement while both existed.
TestError becomes ToolError, since it is the error type for three tools now rather than for the harness alone.
Command-line help is now generated at build time so it does not need to be committed. This reduces churn on commits that add configuration and/or update the help.
Since churn is no longer an issue, help.auto.c is bzip2 compressed to save space in the binary.
The Perl config parser (Data.pm) has been moved to doc/lib since the Perl build path is no longer required.
Likewise doc/xml/reference.xml has been moved to src/build/help/help.xml since it is required at build time.
The prior limitations were based on using getopt_long() to parse command-line options, which required a static list of allowed options. Setting index max too high bloated the binary unacceptably. 45a4e80 replaced the functionality of getopt_long() but the static list remained.
Improve cfgParseOption() to use available option data and remove the need for a static list. This also allows the option deprecations to be represented more compactly.
Index max is still capped at 256 because a large enough index could cause parseOptionIdxValue() to run out of memory since it allocates a static list based on the highest index found. If that function were improved with a map of found index values then index max could be set to UINT64_MAX.
Note that deprecations no longer set an index max or define whether reset is valid. These were space-saving measures which are no longer required. This means that indexed deprecated options will also be valid up to 256 and always allow reset, but it doesn't seem worth additional code to limit this behavior.
cfgParseOptionId() is no longer needed because calling cfgParseOption() with .ignoreMissingIndex = true duplicates the functionality of cfgParseOptionId(). This leads to some simplification in the help code.
This was started in c5ae047e but did not include generation of parse.auto.c.
The parser has also been improved with better errors and multiple passes to reduce dependency on ordering and produce and cleaner output.
Option order resolution now includes cycle detection.
These defaults were not getting the multiplier so the timeouts were much lower than expected.
Since PostgreSQL retries get/push this was probably not a big deal, but it could be critical in the future for a different time value.
Parse enough of config.yaml to auto-generate config.auto.h and config.auto.c.
This commit implements most of the infrastructure needed to migrate the rest of the build code to C, but each set of auto-generated files will present its own challenges.
The build is now dependent on libyaml. At this point there is no need for a hard requirement, but that will come soon so it seems better to add the dependency now.
Since the pack type was stored in 4 bits, only 15 values were allowed (0 was reserved).
Allow virtually unlimited types by storing type info in a base-128 encoded integer following the tag when the type bits in the tag are set to 0xF.
Also separate the type IDs used in the pack (PackTypeMap) from those presented to the user (PackType). The prior PackType enum exposed implementation details to the user, e.g. pckTypeUnknown.
At one time there was a lot more in this header but it got merged with the enums and constants in config.auto.c, leaving only the ConfigOptionType enum.
Auto-generating the ConfigOptionType enum is not very useful since new option types require code in parse.c.
Convert most of the remaining options that benefit from being StringIds. Since all the command modules can include config.h directly it makes sense to auto-generate these values instead of manually creating an enum for each one.
For the time being StringIds are not being auto-generated because the StringId code does not exist in Perl. However, the *_Z zero-terminated constants for each allowed option value are now auto-generated.
This fix was not used since backports worked for older Debians and the old affected Ubuntus are EOL.
Still, seems worth preserving just in case it comes up elsewhere.
These are rarely used and end up using more space than they save since most of them are never referenced.
Replace VARSTR() with VARSTRDEF() where these constants are being used.
Moving to YAML allows the configuration data to be read by C programs.
Also go back to using YAML::XS since it is the only implementation that has proper boolean support.
Up to four repositories may be configured. A potential benefit is the ability to have a local repository for fast restores and a remote repository for redundancy.
Some commands, e.g. stanza-create/stanza-update, will automatically work with all configured repositories while others, e.g. stanza-delete, will require a repository to be specified using the repo option. See the command reference for details on which commands require the repository to be specified.
Note that the repo option is not required when only repo1 is configured in order to maintain backward compatibility. However, the repo option is required when a single repo is configured as, e.g. repo2. This is to prevent command breakage if a new repository is added later.
The archive-push command will always push WAL to the archive in all configured repositories but backups will need to be scheduled individually for each repository. In many cases this is desirable since backup types and retention will vary by repository. Likewise, restores must specify a repository. It is generally better to specify a repository for restores that has low latency/cost even if that means more recovery time. Only restore testing can determine which repository will be most efficient.
For single repository configurations there should be no change in behavior.
The following options are renamed as specified:
repo1-azure-ca-file -> repo1-storage-ca-file
repo1-azure-ca-path -> repo1-storage-ca-path
repo1-azure-host -> repo1-storage-host
repo1-azure-port -> repo1-storage-port
repo1-azure-verify-tls -> repo1-storage-verify-tls
repo1-s3-ca-file -> repo1-storage-ca-file
repo1-s3-ca-path -> repo1-storage-ca-path
repo1-s3-host -> repo1-storage-host
repo1-s3-port -> repo1-storage-port
repo1-s3-verify-tls -> repo1-storage-verify-tls
The old option names (e.g. repo1-s3-port) will continue to work for repo1, but repo2, etc. will require the new names.
Repositories will be searched in order for the requested archive file.
Errors will be reported as warnings as long as a valid copy of the archive file is found.
Errors are logged to the log file rather than thrown. If, after processing all repos, one or more errors occurred, then a single error error will be thrown to indicate there were errors and the log file should be inspected.
Also update log messages to be more consistent with new patterns.
This allows the config path to be modified with a parameter to the configure script, though this commit does not do that.
Update the Perl code generator to allow literals so that defaults can be C defines rather than static strings.
YAML::XS requires libyaml so it not as portable as pure Perl versions of YAML.
Instead of using YAML:PP just use the general YAML::Any module which uses whatever is installed. We are not concerned about performance for YAML so whatever works is fine.
