Follow up to 07f1cd6c, which did not find these because an assert counts as a use in a debug build.
pgControlCrcUpdate() pgVersion: selected the CRC algorithm. 48f511da removed the pgVersion > PG_VERSION_94 test along with support for 9.4, leaving only ASSERT(pgVersion != 0).
backupBuildIncr() infoBackup: supplied the cipher passphrase via infoPgCipherPass(infoBackupPg(infoBackup)). b9b58a17 replaced that when it started passing a cipher spec, leaving only ASSERT(infoBackup != NULL).
A value that varies by version was read inside a function, so two versions got a rendering of it that differed only by that value. pgInterfaceControlVersion() returned PG_CONTROL_VERSION, pgInterfaceWalIs() compared XLOG_PAGE_MAGIC, and pgInterfaceControlIs() compared both PG_CONTROL_VERSION and CATALOG_VERSION_NO, so none of them could ever be shared no matter how little of the version they actually depended on.
Include them in the interface struct instead. A version block captures each as a constant named for it, since a value only has a value inside the block and the struct is built after every block has undefined what it defined. A define cannot do this because it is expanded where it is used rather than where it is made, by which time the value it was given is gone. The constant also names the member it is assigned to, so adding a value is a change to the header and to nothing else.
Each value is captured by a macro of its own, so the analysis that decides which versions can share a function decides the same for a value. One that does not vary between two versions is captured once for both and named for the range it covers, e.g. pgInterfaceControlVersion13_16, which makes what a version shares readable where it is assigned rather than something to work out from the vendored header. The control version is the same from 13 through 16 and the crc offset for 15 and 16.
The functions that only compared a value go with them. Matching pg_control against an interface is now a comparison of the fields that are at the same offset in every version, which pgControlCommon and pgWalCommon already read to report a version that is not supported, and the WAL magic is compared the same way. The comparison is rendered by the generator in the form that fits the versions rendered, so the range that an unreleased version needs is only rendered while there is one and there is never a comparison that no supported version can fail.
Whether a version has been released is set from the declaration, which is where it is stated, rather than derived in the header.
The harness shims the interface so it writes with the types that interface declares rather than declaring them again, and takes the defaults a test does not ask for from the interface the code matches on rather than from a second copy of them.
This brings the binary interface to 22 rendered functions from 62 and the harness to 12 from 32.
When pgBackRest encounters a PostgreSQL version it does not recognize — looking up the version interface, reading pg_control, or reading a WAL header — it reports the version as unsupported with a single hint asking whether the version is supported. A frequent cause is simply that pgBackRest is out of date on a host, e.g. an old binary on the restore host reading a backup from a newer cluster, but the message gave no indication of this.
Add a second hint suggesting that pgBackRest may need to be updated on all hosts. Also report the version from pgInterfaceVersion() in user-facing form (e.g. 13 rather than 130000) and use the PG_NAME/PROJECT_NAME macros in place of the hard-coded names.
Add PostgreSQL 19 as an unreleased version (release: false) and vendor its control/checkpoint structures, catalog and control versions, and XLOG_PAGE_MAGIC.
PostgreSQL 19 stores data_checksum_version in pg_control as a four-state ChecksumStateType enum (OFF, VERSION, INPROGRESS_OFF, INPROGRESS_ON) rather than the prior 0/1 value. Validate the field against the version-appropriate maximum and clear the in-progress states to 0, since page checksums cannot be relied on while checksums are being enabled or disabled. Consumers of pageChecksumVersion therefore continue to see only 0 or 1.
Add the PG19 test harness and the supporting Perl/CI plumbing (DbVersion, VmTest, container build) and adjust the integration test matrix.
Putting these defines in meson.build is a bit annoying because of the added syntax and double commenting. Since the defines are static there is no good reason to have them there.
This also creates a place for test builds to add global defines -- again rather than in meson.build.
These data types are equivalent so no code changes required -- just renaming.
The advantage is that uint8_t makes it clear that this is byte data. It also has the advantage of being shorter.
Warn whenever a pointer is cast so as to remove a type qualifier from the target type. For example, warn if a const char * is cast to an ordinary char *.
Most of the changes for this are fairly rote: just add a const qualifier where needed. In some cases functions needed to be reworked to return non-const where before they had returned const and then cast it back to non-const. None of these patterns appeared to be bugs, but they were certainly misleading.
Some cases (especially excvp() and calls to bz2) could not be fixed because of how functions out of our control are defined. In those cases the warnings have been suppressed and a comment added to detail the exception. This was also done a few places in the tests.
Per our policy to support five EOL versions of PostgreSQL, 9.4 is no longer supported by pgBackRest. Remove all logic associated with 9.4 and update the tests.
This includes a small fix in infoPg.c to allow backup.info files with old versions to be saved. This allows expire to function when old versions are present. Even though those older versions cannot be used, they can be expired.
Tests for 9.4 are left in the expire/info tests to demonstrate that these commands work with old versions present.
Alternative WAL segment sizes can be configured in PostgreSQL <= 10 with compile-time options. We have not allowed these before since it was not a well-tested feature of PostgreSQL.
However, forks such as Greenplum allow alternative WAL segment sizes at initdb time (which are presumably well-tested) so it makes sense to allow it.
Since the PostgreSQL versions in question are all EOL it is not important to have this restriction in place anymore.
If backup_label is removed from a restored backup then PostgreSQL will instead use checkpoint information from pg_control to attempt (what is thinks is) crash recovery. This will nearly always result in a corrupt cluster because the checkpoint will not be from the beginning of the backup, and even if it is, the end point will not be specified, which could lead to recovery stopping too early.
To prevent this, invalidate the checkpoint LSN in pg_control on restore. If backup_label is removed then recovery will still fail because PostgreSQL will not be able to find the invalid checkpoint. The LSN of the checkpoint is not logged but it will be visible in pg_controldata output as 0/DEAD. This value is invalid because PostgreSQL always skips the first WAL segment when initializing a cluster.
This serves as an additional sanity check to be sure the pg_control format is as expected. The field is useful for being near the end and containing a limited number of discrete values.
This serves as an additional sanity check to be sure the pg_control format is as expected. The field is useful for being all the way at the end and being four bytes that can only have one of two values. Something more distinctive than 0 and 1 would be better, but this is what we have to work with.
Convert PgControl.pageChecksum to unsigned int and rename to PgControl.pageChecksumVersion and make all downstream changes required for the new datatype.
Alternate pages sizes can be selected at compile-time, .e.g. 4096. While compile-time settings are generally not well tested by core, some established forks such as Greenplum use them.
On certain file systems (e.g. ext4) pg_control may appear torn if there is a concurrent write while reading the file. To prevent an invalid read, retry until the checksum matches the control data.
Special handling is required for the pg-version-force feature since the offset of the checksum is not known. In this case, scan from the default position to the end of the data looking for a checksum match. This is a bit imprecise, but better than nothing, and the chance of a random collision in the control data seems very remote considering the ratio of data size (< 512 bytes) to checksum size (4 bytes).
This was discovered and a possible solution proposed for PostgreSQL in [1]. The proposed solution may work for backup, but pgBackRest needs to be able to read pg_control reliably outside of backup. So no matter what fix is adopted for PostgreSQL, pgBackRest need retries. Further adjustment may be required as the PostgreSQL fix evolves.
[1] https://www.postgresql.org/message-id/20221123014224.xisi44byq3cf5psi%40awork3.anarazel.de
Forks may update pg_control version or WAL magic without affecting the structures that pgBackRest depends on.
This option forces pgBackRest to treat a cluster as the specified version when it cannot be automatically identified.
uncrustify has been configured to be as close to the current format as possible but the following changes were required:
* Break long struct initializiers out of function calls.
* Bit fields get extra spacing.
* Strings that continue from the previous line no longer indented.
* Ternary operators that do not fit on a single line moved to the next line first.
* Align under parens for multi-line if statements.
* Macros in header #if blocks are no longer indented.
* Purposeful lack of function indentation in tests has been removed.
Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward.
Support code for uncrustify will be in a followup commit.
Allocating memory made these functions simpler but it meant that memory was leaking into the calling context when logging was enabled. It is not clear that this was an issue but it seems that trace level logging could result it a lot of memory usage depending on the use case.
This also makes it possible to audit allocations returned to the calling context, which will be done in a followup commit.
Also rename objToLog() to objNameToLog() since it seemed logical to name the new function objToLog().
This should make it a little clearer what the variable (VR) macros are doing since the declaration/definition cannot both be set to extern (but functions can).
Splitting the variable macros out also allows them to be changed in the future with little churn, while changing the function macro creates a large amount of churn.
This is immediately useful because it will detect any extern'd functions or variables that are not being used. It also detects functions or variables that are declared but not defined.
If a FV/VR_EXTERN macro is missing it will be detected either because of a mismatch in the declaration/definition or because a new defined symbol will appear in the nm test.
Eventually the unity build will be used to create a more optimized pgbackrest binary but that will need to wait.
Our new policy is to support ten versions of PostgreSQL, the five supported releases and the last five EOL releases. As of PostgreSQL 15, that means 9.0/9.1/9.2 are no longer supported by pgBackRest.
Remove all logic associated with 9.0/9.1/9.2 and update the tests.
Document the new support policy.
Update InfoPg to read/write control versions for the history in backup.info, since we can no longer rely on the mappings being available. In theory this could have been an issue after removing 8.3/8.4 if anybody was using a version that old.
Maintaining the version interfaces was complicated by the fact that each interface needed to be in separate compilation unit to avoid type conflicts. This also meant that various build/test files needed to be updated to add the new interfaces.
Solve these problems by auto-generating all the interfaces into a single file. This is made possible by parsing defines and types out of the header files and creating macros to rename the types. At the end of the version interface everything is undef'd. Another benefit is that the auto-generated interfaces can be static and included directly into postgres/interface.c.
Since some code generation is now always required for tests, change --no-gen to --min-gen in test.pl.
It would also make sense to auto-generate the version defines in postgres/version.h, but that will be left for a future commit.
PostgreSQL 15 drops support for exclusive backup and renames the start/stop backup commands.
This is based on the pgdg-testing repo since beta1 has not been released yet, but it seems unlikely that breaking changes will be made at this point. beta1 should be tagged just before our next release so we'll retest before the release.
This column has been removed in PostgreSQL 15. Rather than add a lot of special handling, it seems better just to update all versions to not depend on this column.
Add centralized functions to identify the type of database (i.e. system or user) by name and use FirstNormalObjectId when a name is not available.
The new query in the db module will still return the prior result for PostgreSQL <= 15, which will be stored in the manifest. This is important to preserve behavior when downgrading pgBackRest. There are no concerns here for PostgreSQL 15 since older versions of pgBackRest won't be able to restore backups for PostgreSQL 15 anyway.
These functions allow conversion from substrings without needing to create a String or a temporary buffer.
httpDateToTime() no longer requires a temp mem context. Also improve handling of month search to avoid an allocation.
httpUriDecode() no longer requires a temp mem context.
jsonReadStr() no longer requires a temp mem context.
pgLsnFromWalSegment() no longer requires a temp mem context.
pgVersionFromStr() no longer requires a temp mem context. Also do a bit of refactoring.
storageGcsCvtTime() no longer leaks six Strings per call.
storageS3CvtTime() no longer leaks six Strings per call.
pgLsnFromWalSegment() leaked two Strings.
Refactor pgLsnRangeToWalSegmentList() to create the StringList in the calling context rather than moving it later.
This allows code to run after the return type has been generated in the case where it is an expression.
No new functionality here yet, but this will be used by a future commit that audits memory usage.
Determining the length of arrays that could be calculated at compile time was a bit piecemeal, with special macros used sometimes and with the math done directly other times.
This macro makes the task easier, uses less space, and automatically adjusts when the type changes.
There is no evidence that users need 8.3/8.4 anymore but it does cost us in terms of development and testing, especially now that we have a number of new backup/restore features planned.
It seems to make sense to remove this support now. If there are users who need to use/migrate from these versions they can use an older version of pgBackRest.
Add the following checks:
* Checkpoint is updated in pg_control after pg_start_backup(). This helps ensure that PostgreSQL and pgBackRest have a consistent view of the storage and that PGDATA paths match.
* Timeline of backup start WAL file matches pg_control. Hard to see how this one could get hit, but we have the power...
* Standby is on the same timeline as the primary. If not, this standby is not following the primary.
* Last standby checkpoint is not greater than the backup checkpoint. If so, this standby is not following the primary.
This also requires some additional plumbing to read/write timeline/checkpoint from pg_control and parse timelines from WAL filenames. There were some changes in the backup tests caused by the fact that pg_control now has different contents for each backup.
The check to ensure that the required checkpoint was reached on the standby should also be updated to use pg_control (it currently uses pg_control_checkpoint()), but that requires non-trivial changes to the test harness and will need to wait.
A CHECK() worked exactly like ASSERT() except that it was compiled into production code. However, over time many checks have been added that should not throw AssertError, which should be reserved for probable coding errors.
Allow the error code to be specified so other error types can be thrown. Also add a human-readable message since many of these could be seen by users even when there is no coding error.
Update coverage exceptions for CHECK() to match ASSERT() since all conditions will never be covered.
There are no code changes from PostgreSQL 13 so simply add the new version.
Add CATALOG_VERSION_NO_MAX to allow the catalog version to "float" during the PostgreSQL beta/rc period so new pgBackRest versions are not required when the catalog version changes.
Update the integration tests to handle new PostgreSQL startup messages.
Some version interface test functions were integrated into the core code because they relied on the PostgreSQL versioned interface. Even though they were compiled out for production builds they cluttered the core code and made it harder to determine what was required by core.
Create a PostgreSQL version interface in a test harness to contain these functions. This does require some duplication but the cleaner core code seems a good tradeoff. It is possible for some of this code to be auto-generated but since it is only updated once per year the matter is not pressing.
There does not seem to be a good reason to have an explicit cast for such a small number, so remove it.
Leave PG_CONTROL_SIZE alone since it will be removed in an upcoming commit.
Previously, catalog versions were fixed for all versions which made maintaining the catalog versions during PostgreSQL beta and release candidate cycles very painful. A version of pgBackRest which was functionally compatible was rendered useless by a catalog version bump in PostgreSQL.
Instead use only the control version to identify a PostgreSQL version when possible. Some older versions require a catalog version to positively identify a PostgreSQL version, so include them when required.
Since the catalog number is required to work with tablespaces it will need to be stored. There's already a copy of it in backup.info so use that (even though we have been ignoring it in the C versions).
We use the Z suffix in many functions to indicate that we are expecting a zero-terminated string so make this function conform to the pattern.
As a bonus the new name is a bit shorter, which is a good quality in a commonly-used function.
