Time option values are stored in milliseconds but the table of time values in the generated parse rules was declared unsigned int, which limits a value to about 49.7 days. A default or allow-range bound larger than that overflows and fails to compile. The parser already reads time values, range bounds, and option values as int64_t, so declare the table as int64_t to match the table of size values.
All existing time option values fit in either type, so none of them change.
The help describes the values an option allows with an allow list. An option that allows the same values as another option can inherit that list instead of describing the values again.
This only worked for a list documented by an option on its own. A list documented under a command could not be inherited, so an option that only the commands document has its values written out once per command. The output option describes the same two values under manifest, repo-ls, info, verify, and version.
Record the list each command documents as well, and let inherit name the command to take it from. The command has to be named because more than one command can document the same option and the lists need not match. The type option allows full, diff, and incr for backup but nine values for restore.
A list is still checked against the values the inheriting option or command allows, so it can only be inherited where both allow the same values.
In the test declaration set is now taken by backup and restore so that backup can describe the values and restore can inherit them.
A value that varies by version was read inside a function, so two versions got a rendering of it that differed only by that value. pgInterfaceControlVersion() returned PG_CONTROL_VERSION, pgInterfaceWalIs() compared XLOG_PAGE_MAGIC, and pgInterfaceControlIs() compared both PG_CONTROL_VERSION and CATALOG_VERSION_NO, so none of them could ever be shared no matter how little of the version they actually depended on.
Include them in the interface struct instead. A version block captures each as a constant named for it, since a value only has a value inside the block and the struct is built after every block has undefined what it defined. A define cannot do this because it is expanded where it is used rather than where it is made, by which time the value it was given is gone. The constant also names the member it is assigned to, so adding a value is a change to the header and to nothing else.
Each value is captured by a macro of its own, so the analysis that decides which versions can share a function decides the same for a value. One that does not vary between two versions is captured once for both and named for the range it covers, e.g. pgInterfaceControlVersion13_16, which makes what a version shares readable where it is assigned rather than something to work out from the vendored header. The control version is the same from 13 through 16 and the crc offset for 15 and 16.
The functions that only compared a value go with them. Matching pg_control against an interface is now a comparison of the fields that are at the same offset in every version, which pgControlCommon and pgWalCommon already read to report a version that is not supported, and the WAL magic is compared the same way. The comparison is rendered by the generator in the form that fits the versions rendered, so the range that an unreleased version needs is only rendered while there is one and there is never a comparison that no supported version can fail.
Whether a version has been released is set from the declaration, which is where it is stated, rather than derived in the header.
The harness shims the interface so it writes with the types that interface declares rather than declaring them again, and takes the defaults a test does not ask for from the interface the code matches on rather than from a second copy of them.
This brings the binary interface to 22 rendered functions from 62 and the harness to 12 from 32.
Every supported version got a rendering of every interface function, which is 77 functions for the seven macros the binary interface declares and 33 for the three the harness declares. Many of them compile to identical code, since most of the vendored types do not change in a given release. The functions are trivial enough today that the duplication costs little, but more ambitious ones would multiply by the number of versions.
Read the vendored header as a chain of version branches per type and define, resolve each of them for each version, and take the transitive closure of what an interface macro reaches. Two versions share a rendering when every entity in that closure resolves to the same branch for both, since that is what makes the code the compiler sees the same. Following the closure rather than the names a macro uses directly is what makes the comparison sound: ControlFileData is unchanged from 13 to 14, but CheckPoint is embedded in it and does change, so the layout differs.
The oldest version of a group renders the function and the struct points the rest at it, so adding a version later never renames what is already there. A rendering more than one version shares is named for the range it covers, e.g. pgInterfaceWal14_19, and the block it is rendered in says which versions share it. This brings the binary interface to 62 functions and the harness to 32.
A macro that does not take a version is now read as a helper the function macros are written with rather than as a function of its own. The dependency scan follows a helper to the types it names, so declaring one is now possible, where before it would have been rendered as a function.
The harness had a file per version that did nothing but expand a macro for it, a block of declarations for the functions those files defined, and the struct that dispatched to them, all maintained by hand, so adding a PostgreSQL version meant adding a file and three more entries to keep in step. The interfaces the binary uses are generated from the same versions, so the harness generates its own the same way with 'build.py postgres-harness'.
What differs between the two interfaces is a declaration: the header the macros are read from, how the generated code includes it, the prefix of the functions the macros define, and where the interface is rendered. The versions, the vendored types that are renamed for each of them, and the defines that are undefined after each one are the same, so one generator renders both.
The harness interface is one translation unit with the vendored header included again for each version, the way the binary's is, rather than a translation unit per version. The functions are static, so the declarations and the pragma that silenced the missing prototypes go away with the files, and they are named for the version the way the binary's are, e.g. 19 rather than 190. The harness macros are renamed to say which function each one defines, since the generator names a function after the macro that defines it, and the macro that expanded all of them for a version is gone because the generated file expands each one itself. The header they are declared in loses its include guard and everything that does not vary by version, since it is now included once per version and the module that expands the macros includes the rest.
The system id constants are generated into the harness header rather than into a file of their own, so the header still holds everything the harness declares. The defines are replaced where they are, which is what the version in src/version.h is already generated by. Every version gets the offset form that only two of them had, since a block that reads the same for every version is easier to follow than one that does not.
The interface is generated rather than committed, the same as the interface the binary uses, so a test run generates it and the repository copy duplicates it for a unit build to compile.
main.c included the generated help data and passed it to cmdHelp() as a buffer, so no unit build ever compiled it and the help test had to run the generator and load the packed help from a file to have anything to render. Including it in help.c puts the data in the module that reads it and lets the test render exactly what the binary carries.
The generated file is added to the files the repository copy needs, since git does not list it, and the test command generates it along with the PostgreSQL interfaces. Those are the two files that are generated rather than committed, and a unit build needs both from the copy because it does not run the code generation the build does. The file is regenerated at the start of every run, so the test still runs against the current declarations without generating them itself.
cmdHelp() and helpRender() no longer take the data as a parameter and help.h no longer needs the buffer type. The help-data generator command is removed along with its only caller.
The values an option allows are declared in config.yaml and were written out again by hand in reference.xml as an ordinary list, so nothing held the two together. A value added, removed, or renamed in the declaration left the documentation describing something else and only a reader would find it.
An allow list is now marked as what it is and its values are checked against the declaration. Where each value needs an explanation allow-item gives it and caption says what the list is a list of, which renders the sentence that was written above each list by hand. Where the values speak for themselves the tag is left empty and the values are rendered from the declaration into the sentence around it, so they are not written out at all. Where an option allows what another option allows the list is inherited, the way the declaration inherits the values themselves, which is what log-level-file, log-level-stderr, and pg-host-type do.
Documenting the list is required rather than optional, since a value a user cannot read about is a value they cannot use. That turned up four options that never listed their values. buffer-size and repo-sftp-host-key-hash-type render the sentence they already had from the declaration, and repo-retention-archive-type and repo-retention-full-type described a value per paragraph, which is now the list it always was.
The markup is resolved where the help is parsed, so the help in the binary and the documentation are both rendered from it and no renderer knows it exists. Everything else in the documentation renders exactly as it did, except for four introductions reworded to fit the caption.
Most of these are prose in a python docstring or comment wrapped a few characters wide, which black does not reflow, and macro continuation backslashes left stranded past the margin. The rest are long calls broken after the first parenthesis, section separator comments with too many dashes, and long expectation strings split along what they hold. A few long shell commands in the workflows are wrapped with continuations.
hrnTestResultDiff() had a continuation backslash left over from when it was a macro, which spliced the closing brace onto the line above it. It is removed.
The linter that enforces the limit follows, so this commit is only the reformatting and can be ignored by blame.
The three items the index lists were written on it by hand, so every news item was written twice: once as the news and again as a date, a title, and a link on the index. Nothing kept the two in step, so an item added to one and forgotten in the other would go unnoticed by anything that builds the documentation.
The index asks for the items now and the build takes them from the news. The title on the index is the title of the item as the news writes it, so a title that changes on the news changes on the index as well, and the project name in one is shown there the way it is shown everywhere else.
A news item says when it was posted as an attribute written the way a date sorts rather than as the paragraph the item begins with, so the date is written once and rendered both where the item shows it and where the index lists it. An item that is more recent than the item before it is an error, since the index lists the items the news begins with.
The release notes render their date the same way, so the months are named in one place rather than in three.
The command reference described every option a command takes in full, under every command that takes it. Since most options are valid for most commands the same description was written out again and again: --config was described eighteen times, and the backup command alone ran to a hundred and thirty options, of which fifteen are actually about backup. The page came to 1.2 MB and reaching a command meant scrolling past everything before it.
An option is now described once. The options a command documents itself, which are the ones whose meaning depends on the command, stay with the command as they were. The rest are a list of links, each naming the option the way the heading at the other end names it, e.g. Repository Host (--repo-host), and pointing at the configuration reference where it is described. The page is 350 KB now.
An option that can only be given on the command line, e.g. --config or --stanza, is not in the configuration reference because it cannot be set in the configuration file, so there was nowhere for it to link to. These are described in a General Options section before the commands and each command links there. They are left out of the contents, since the contents list the commands rather than the options each of them takes.
A link now renders the markup it holds rather than its text alone, so an option named in a list keeps the monospace it has in a heading. Every link written in doc/xml holds text alone and every page but the command reference is byte for byte what it was. A link may also hold text and markup at once when a document is normalized, since otherwise a hand-written document could not say what the generated one says.
The configuration reference is unchanged.
A ctrl-c raised KeyboardInterrupt out of main() and python printed the traceback, which is the whole path from main() down to the select() the harness happened to be waiting in and says nothing about a run that was stopped on purpose. Catch it and log "terminated on signal SIGINT" as the C does, exiting with the status the C exits with for TermError.
An unexpected exception is a bug in the tool so the trace is what gets it fixed, but the trace ends with the message that says what went wrong, which is where the message gets lost. Show the message first and keep only the innermost frames, since the frames above them are the path from main() into the tool and are much the same for every error. The frame count says how many were dropped and how to get them, and the complete trace, which is what python printed before, is shown at debug level the same way the C shows a stack trace only at debug level.
The rendering is error_trace() in common/error.py, beside ToolError, so every tool reports a bug in itself the same way. The CI wrapper asks for the complete trace since a run there cannot be repeated at debug level to get it, and the code builder handles the interrupt but not the trace since it has no log level to raise.
The definitions move out of common/vm.py into a vm section at the top of test/container.yaml, next to the container revisions that go with them, and are read and checked when the module is imported as they were before. The revisions move under a revision key to make room.
The declaration is read with the loader the other declarations use, so a version keeps the text it was written as rather than arriving as a float. A new yaml_map_dict() reads a mapping as a dict where a key is looked up by name, which also reports a key that was written twice instead of silently keeping the last one.
The constants that named each vm are gone. A vm is named by the string that --vm accepts, which is how the CI workflows and the documentation build already name them. A name used as a lookup is validated by vm_get(), and a name used in a comparison is protected by branch coverage since a comparison that can never match leaves a branch unhittable. VM_NONE, VM_ALL, and the os base and architecture constants remain because they name kinds rather than vms. VM_LIST is now the vms declared as default, so it is in declaration order.
The container build no longer tests a vm by name. What a vm installs and configures is declared instead: valgrind, epel, powertools, python, coverage-python, pg-repo-key, pg-repo-release, dnf-module, and ssh-rsa. The three near identical blocks that added the PostgreSQL repo collapse into one, which removes the uncoverable branch annotation on the fedora branch. Every generated script is unchanged, so no image is rebuilt.
The supported PostgreSQL versions are generated from build/postgres.yaml rather than declared a second time. A new bld_pg_version_list() reads just the versions, since a tool that only needs to know which ones are supported has no use for the interface that bld_pg_parse() scans the vendored headers for. The release flag comes with them, so the beta repository is added for whichever installed version has not been released rather than for a hardcoded vm and version.
A vm with no container declares only default and coverage-c, since every other key says how a container is built. The os base declared for none was never read, and the os the tests run on with --vm=none is whatever the host is rather than the debian it claimed. The PostgreSQL versions declared for it were never read either, since only an integration test runs against a version and those need a container.
A PostgreSQL version installed on a vm must now be one that build/postgres.yaml declares, and a version tested on a vm must be one installed there, both of which were only true by inspection before.
doc.pl and release.pl become doc.py and release.py and the Perl library in doc/lib/pgBackRestDoc is gone, which is 32 files and 11,059 lines replaced by about 3,850 lines of Python. Nothing in the project runs on Perl now.
doc.pl and doc.py merge into one tool rather than one running the other. doc.py built the reference documents and the preprocessed user guide, wrote them to doc/output/xml, and doc.pl read them back; they are now passed in memory and those files are not written at all. That also removes a duplication the split had hidden: the preprocessor added a document's variables to its own store and left the declarations in the file, so the Perl parsed them again into a store of its own and the same variable existed twice with different values.
Most of Doc.pm does not come across. Its parse step rebuilt the parsed tree into a structure of its own and its build step invented a second way to address that structure, so a document was reached through three accessors where the tree has one. What survives is about ninety lines in common/xml.py: a field is the text of a child that holds nothing else, the text of a node is the node itself for the tags that are text and a text child for the rest, and asking for a child rather than the list of them now reports a second one as the mistake in the document that it is.
External entities are expanded before the parser sees a document, since the standard library does not fetch them. The release list is assembled from one file per release this way, 122 of them, and a declaration that is never referenced is reported rather than leaving the file it names silently unread.
Nothing validated against a document type, so the document type declarations and the two dtd files are removed. A declaration that is not enforced does not get maintained. The two documents that assemble themselves from parts keep the declaration because the parts are declared inside it.
The release notes become a document the tool builds, like the command and configuration references, rather than something the renderer reaches into a custom module to produce.
A variable may still be evaluated rather than written out, but it is a Python expression against a short list of names rather than a Perl program. The three that ask for a fact about the environment keep doing so. The four that read the project version, the current stable version, and the release date are facts the tool computes, since working them out involved constructing library objects and reading the release list a second time from inside a variable body.
exec and user move down from test/lib to doc/lib because the documentation needs both, the project version splits out into config/project, and eval and var_store move to common now that the manifest uses them as much as the preprocessor. The version is built from the parts in src/version.h rather than read from the string generated from them, which is what the Perl did and what a release needs, since a release clears the development suffix and the string is only regenerated by a test run.
libxml-checker-perl is no longer installed on CI runners, in the development container, or in the documentation images, and the RHEL image no longer installs cpanminus and a toolchain to build XML::Checker::Parser from CPAN. The user guide never ran Perl, and the Debian image never had a parser and built the same guide.
Every module in doc/lib is declared in define.yaml and has full line and branch coverage.
Ubuntu 24.04 ships a user and a group at id 1000, so a build by a user with that id could not create the test user. The test image removes whatever holds the id before creating it, as it already did for the group, and the development container image removes both because adduser gives a new group the id of the user.
The coverage summary listed the python under build/lib as modules with mangled names, since it stripped a fixed src/ prefix from paths that no longer start with one. It now summarizes what is under src, which is what the metric page says it reports.
Deploying the website no longer removes the links at the top of the site or CNAME. The Perl used find -type f, which does not match a symbolic link, and the port used a test that follows one, which took out the guides kept under retired platform names. CNAME is a regular file and the Perl removed it every deploy, which was never intended.
The rendered documentation is byte identical to what the Perl produced, for every page, the manual page, and the generated markdown. The user guide was also built end to end under Docker on both documented platforms, and rendering it again from the cache that build wrote produces the same pages, which exercises the whole path rather than only what can be checked without containers. A release was then simulated from beginning to end.
The build tool in src/build becomes build/build.py with its library in build/lib, and the C part of the documentation tool in doc/src becomes doc/doc.py with its library in doc/lib. They migrate together because doc/src was built on src/build, linking its config and help parsing along with its extensions to the core types. doc.pl runs doc.py rather than building and running a C binary, and meson runs build.py for the code it generates as it builds.
The three libraries form the hierarchy the C include path did: build/lib is the base, doc/lib sits on it, and test/lib sits on both. They are implicit namespace packages so a package spans them, and a module lives in exactly one library and holds the union of what the tools need from it, since Python resolves a module name to a single file and cannot extend one the way the C included a core module and added to it. define.yaml declares which libraries each tool may import and the test runner puts only those on the path, so a test of the build library cannot reach the doc or test library any more than the build tool can.
The declarations move with the code. config.yaml, error.yaml, and postgres.yaml are now at the root of build, and help.xml becomes doc/xml/reference.xml since it is documentation that the help happens to be generated from, so the generator is told where it is rather than knowing. dist.sh moves to build as well. The documentation tool's own option declarations are gone because they existed only to generate the option parser for the C binary and doc.py parses its command line directly.
Three modules in src/build/common were never build tool code. exec, regExp, and string extend the core types for the test harness and the unit tests, which were their only users, so they move to test/src/harness. The json and xml extensions had no users outside the two tools and are removed with them, and the tests for both are trimmed to the core functions that remain.
The help unit test built its fixture by calling the generator's C functions directly, which was the last thing outside the tools that depended on them. It now runs build.py help-data to write the packed help and loads that, so it still runs against the current declarations without needing the generated C, which may not have been written when the test runs.
The manual page wrapping is a port of the wrapper the binary uses for its own help, so the manual page and pgbackrest help break lines the same way, including where a word longer than a line is left unbroken.
libyaml is no longer needed to build pgBackRest since the only thing that read yaml was the C generator. It is dropped from the development container and from every CI platform, and meson no longer looks for it. Code generation needs nothing compiled first either, so a test run no longer builds a generator before it can select the tests.
The common source list is merged into the pgbackrest source list since it existed only so the generator could be built from the same files, and the entries that were out of alphabetical order are put in order while the list is being rewritten anyway.
Every file the generator writes is byte identical to what the C wrote, for the project declarations and for the documentation's own. The documentation is checked the same way: the HTML rendered on this branch is byte identical to the HTML rendered on integration, every file of it, which exercises the whole pipeline on the real content rather than only the four files the tool writes.
Every module in both libraries is declared in define.yaml and has full line and branch coverage. The coverage list is keyed by file rather than by module name, since the same name in two languages is two files and one of them would have silently lost its requirement while both existed.
TestError becomes ToolError, since it is the error type for three tools now rather than for the harness alone.
Command-line help is now generated at build time so it does not need to be committed. This reduces churn on commits that add configuration and/or update the help.
Since churn is no longer an issue, help.auto.c is bzip2 compressed to save space in the binary.
The Perl config parser (Data.pm) has been moved to doc/lib since the Perl build path is no longer required.
Likewise doc/xml/reference.xml has been moved to src/build/help/help.xml since it is required at build time.
The prior limitations were based on using getopt_long() to parse command-line options, which required a static list of allowed options. Setting index max too high bloated the binary unacceptably. 45a4e80 replaced the functionality of getopt_long() but the static list remained.
Improve cfgParseOption() to use available option data and remove the need for a static list. This also allows the option deprecations to be represented more compactly.
Index max is still capped at 256 because a large enough index could cause parseOptionIdxValue() to run out of memory since it allocates a static list based on the highest index found. If that function were improved with a map of found index values then index max could be set to UINT64_MAX.
Note that deprecations no longer set an index max or define whether reset is valid. These were space-saving measures which are no longer required. This means that indexed deprecated options will also be valid up to 256 and always allow reset, but it doesn't seem worth additional code to limit this behavior.
cfgParseOptionId() is no longer needed because calling cfgParseOption() with .ignoreMissingIndex = true duplicates the functionality of cfgParseOptionId(). This leads to some simplification in the help code.
This was started in c5ae047e but did not include generation of parse.auto.c.
The parser has also been improved with better errors and multiple passes to reduce dependency on ordering and produce and cleaner output.
Option order resolution now includes cycle detection.
These defaults were not getting the multiplier so the timeouts were much lower than expected.
Since PostgreSQL retries get/push this was probably not a big deal, but it could be critical in the future for a different time value.
Parse enough of config.yaml to auto-generate config.auto.h and config.auto.c.
This commit implements most of the infrastructure needed to migrate the rest of the build code to C, but each set of auto-generated files will present its own challenges.
The build is now dependent on libyaml. At this point there is no need for a hard requirement, but that will come soon so it seems better to add the dependency now.
Since the pack type was stored in 4 bits, only 15 values were allowed (0 was reserved).
Allow virtually unlimited types by storing type info in a base-128 encoded integer following the tag when the type bits in the tag are set to 0xF.
Also separate the type IDs used in the pack (PackTypeMap) from those presented to the user (PackType). The prior PackType enum exposed implementation details to the user, e.g. pckTypeUnknown.
At one time there was a lot more in this header but it got merged with the enums and constants in config.auto.c, leaving only the ConfigOptionType enum.
Auto-generating the ConfigOptionType enum is not very useful since new option types require code in parse.c.
Convert most of the remaining options that benefit from being StringIds. Since all the command modules can include config.h directly it makes sense to auto-generate these values instead of manually creating an enum for each one.
For the time being StringIds are not being auto-generated because the StringId code does not exist in Perl. However, the *_Z zero-terminated constants for each allowed option value are now auto-generated.
This fix was not used since backports worked for older Debians and the old affected Ubuntus are EOL.
Still, seems worth preserving just in case it comes up elsewhere.
These are rarely used and end up using more space than they save since most of them are never referenced.
Replace VARSTR() with VARSTRDEF() where these constants are being used.
Moving to YAML allows the configuration data to be read by C programs.
Also go back to using YAML::XS since it is the only implementation that has proper boolean support.
Up to four repositories may be configured. A potential benefit is the ability to have a local repository for fast restores and a remote repository for redundancy.
Some commands, e.g. stanza-create/stanza-update, will automatically work with all configured repositories while others, e.g. stanza-delete, will require a repository to be specified using the repo option. See the command reference for details on which commands require the repository to be specified.
Note that the repo option is not required when only repo1 is configured in order to maintain backward compatibility. However, the repo option is required when a single repo is configured as, e.g. repo2. This is to prevent command breakage if a new repository is added later.
The archive-push command will always push WAL to the archive in all configured repositories but backups will need to be scheduled individually for each repository. In many cases this is desirable since backup types and retention will vary by repository. Likewise, restores must specify a repository. It is generally better to specify a repository for restores that has low latency/cost even if that means more recovery time. Only restore testing can determine which repository will be most efficient.
For single repository configurations there should be no change in behavior.
The following options are renamed as specified:
repo1-azure-ca-file -> repo1-storage-ca-file
repo1-azure-ca-path -> repo1-storage-ca-path
repo1-azure-host -> repo1-storage-host
repo1-azure-port -> repo1-storage-port
repo1-azure-verify-tls -> repo1-storage-verify-tls
repo1-s3-ca-file -> repo1-storage-ca-file
repo1-s3-ca-path -> repo1-storage-ca-path
repo1-s3-host -> repo1-storage-host
repo1-s3-port -> repo1-storage-port
repo1-s3-verify-tls -> repo1-storage-verify-tls
The old option names (e.g. repo1-s3-port) will continue to work for repo1, but repo2, etc. will require the new names.
This allows the config path to be modified with a parameter to the configure script, though this commit does not do that.
Update the Perl code generator to allow literals so that defaults can be C defines rather than static strings.
YAML::XS requires libyaml so it not as portable as pure Perl versions of YAML.
Instead of using YAML:PP just use the general YAML::Any module which uses whatever is installed. We are not concerned about performance for YAML so whatever works is fine.
Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands.
Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations.
Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
These options specify the number of local worker job retries and the retry interval after one immediate retry.
There is some value in allowing retries to be specified by the user but for the most part these options are for suppressing retries during testing, which can save a lot of time. The bug introduced in d1d25c7 and fixed in 8b86d5e also suggests it is better not to use retries in tests.
Remove the default delayed retries for archive-get/archive-push, leaving only the immediate retry. These commands are retried by PostgreSQL so it doesn't make sense to do too many retries internally.
These options are currently internal.
The pg option only has one current usage, to let the backup local know which pg index it should copy files from.
There are other possible uses for this option, but they need thought, tests, and documentation.
This option was added in advance of the multi-repo functionality but it has no purpose and it is not clear what the validity rules should be.
The option will be added back when multi-repo functionality is committed.
This was a hack to prevent the remote from loading host settings, which is now handled by option validity for command roles.
These options are still useful so don't remove them, but do leave them internal for now.
Building on 23f5712, limit option validity by role. This is mostly for options that weren't needed for certain roles but were harmless. However, the upcoming multi repository functionality requires the granularity implemented here.
The remote role benefits since host options can automatically excluded when building the options. Also, many options that are only required for the default role (e.g. repo-retention-full) no longer need to be passed in tests for other roles.
Validity by command was not granular enough so numerous options needed be marked internal so users would not stumble across them. Options were also needlessly being passed to roles that had no use for them.
Introduce per-role validity lists that depend on what roles are valid per command. Also add a check to ensure that only valid roles are used with a command.
This commit adds the functionality but does not introduce any new behavior, i.e. all options are valid for all roles that the command is valid for. A subsequent commit will introduce the new role restrictions to make the changes easier to audit.
Data required for parsing was spread between the config and defined modules, mostly for historical reasons because the same data was used by Perl.
Requiring all the parse rules to be accessed with function interfaces makes the code more complicated and new rules harder to implement.
Instead, move the data to the parse module so in the most complex cases no interface functions are needed. This reduces the total amount of code and paves the way for more complex parse rules.
