The label is written as one line of the backup_label file. A label
with a newline added extra lines to the file, and recovery could read
those as other fields, failing on them. Characters could also be
injected to make recovery behave in inconsistent ways.
Trying to use such characters in label names is now rejected, for both
pg_backup_start() and BASE_BACKUP.
As this is arguably a behavior change, no backpatch is done. I also
seriously doubt that anybody is relying on the behavior of pushing some
arbitrary data to backup_label files, and even if they do, it would be a
very bad idea to contradict what the backend decides to generate.
Reported-by: Shallow <theshallow27@gmail.com>
Author: Shihao Zhong <zhong950419@gmail.com>
Discussion: https://postgr.es/m/19730-85a6044c9e72774a@postgresql.org
Clearing the tuplestore rather than ending it avoids the need for
various memory allocations, so is slightly more efficient. However, the
main reason to do this is to correctly track the maximum storage used by
the Table Function Scan node. 40708acd65 added additional EXPLAIN
output for Table Function Scan nodes and that output does claim to be
showing "Maximum Storage", which is not true, as the maximums could be
lost after tuplestore_end() is called during a rescan.
This causes misreporting when the final rescan of a Table Function Scan
node uses less storage than some previous rescan. One example is:
EXPLAIN ANALYZE
SELECT count(*)
FROM (VALUES ((SELECT jsonb_agg(g) FROM generate_series(1, 100000) g)),
('[1]'::jsonb)) v(j),
LATERAL JSON_TABLE(v.j, '$[*]' COLUMNS (x int PATH '$')) jt;
Without this fix, the reported Table Function Scan storage is for the
'[1]'::jsonb case, whereas it should consider how much was used with the
larger document and show the maximum of all rescans.
TableFuncNext() used a NULL tupstore to tell that the rows still had to
be fetched, so add a tupstore_filled flag for that, similar to what
908a96861 did for WindowAgg.
Technically, this issue does exist in v18, but no backpatch as this is a
fairly minor issue which was only noticed during a review of a similar
feature being proposed for Function Scans.
Author: Tatsuya Kawata <kawatatatsuya0913@gmail.com>
Reviewed-by: David Rowley <dgrowleyml@gmail.com>
Discussion: https://postgr.es/m/CAHza6qdXjcqXv2aGU7OCakG0OmEsy9-jfP0Gpe2EL+k-AidOhw@mail.gmail.com
On Windows, a backend exiting resets its connection, and the client
discards any data not yet read, including the FATAL message of a
terminated backend. isolationtester polls one connection at a time, so
when it waits on a different session the message can get lost, failing
the permutation and the tests that run after it.
wait_cleanup in injection_points (down to v17) and slot_creation_error
in test_decoding (down to v15) have this pattern, and have been failing
randomly in the buildfarm. Let's skip them on Windows.
The cause of the instabilities is proving to be tricky to deal with,
pointing for example at 29992a6a50. This will be tackled later,
perhaps with a backpatch (or perhaps with a HEAD-only solution), but it
is difficult to be sure at this stage of the work. In theory,
a8458f508a should make a revert of 29992a6a50 (revert of a revert!)
safe on WIN32.
Note that 278cdea6b9, that has introduced the isolation test
slot_creation_error in test_decoding, has mentioned the potential output
instability.
Author: Zsolt Parragi <zsolt.parragi@percona.com>
Author: Michael Paquier <michael@paquier.xyz>
Discussion: https://postgr.es/m/CAN4CZFOoiKa9E4Gnh_OzLk3S5Z6RFyg_oY1e_WddCVYhkpTVow@mail.gmail.com
Backpatch-through: 15
Clearing the tuplestore rather than ending it avoids the need for
various memory allocations, so is slightly more efficient. However, the
main reason to do this is to correctly track the maximum storage used by
the Material node. 1eff8279d added additional EXPLAIN output for Material
nodes and that output does claim to be showing "Maximum Storage", which is
not true, as the maximums could be lost after tuplestore_end() is called
during a rescan.
This causes misreporting when the final rescan of a Material node uses
less storage than some previous rescan. One example is:
EXPLAIN ANALYZE
SELECT count(*) FROM (VALUES(10000),(1)) v1(r1)
LEFT JOIN LATERAL (
SELECT * FROM generate_series(1, 2) gs0
LEFT JOIN LATERAL (
SELECT * FROM generate_series(1, v1.r1) gs1
FULL JOIN generate_series(1, v1.r1) gs4 ON false
) ss on true
) q on true;
Without this fix, the reported Material storage is for the v1.r1 = 1 case,
whereas it should consider how much was used with the v1.r1 = 10000 case
and show the maximum of each.
Technically this issue does exist in v18, but no backpatch here as
parameterized Material nodes are not that common and there have been no
bug reports about the incorrect memory reporting, so it's probably not
worth risking backpatching the change.
Author: David Rowley <dgrowleyml@gmail.com>
Reviewed-by: Tatsuo Ishii <ishii@postgresql.org>
Reviewed-by: Tatsuya Kawata <kawatatatsuya0913@gmail.com>
Discussion: https://postgr.es/m/CAApHDvoa55vcRth05Ozu5be4FawgTH-aCsZ5=Z+_UXTUzUxdQg@mail.gmail.com
In clang, -fsanitize=undefined includes -fsanitize=function, which
reports every call made through a function pointer whose type does not
exactly match the called function, so it fires all over the place on
expression tree walkers and mutators, as well as a few other places.
So -fsanitize=undefined hasn't been working cleanly under clang for a
while. (Before clang 17, it only applied to C++.)
This is the same issue that caused us to use
-Wno-cast-function-type-strict with clang. That warning applies at
the place where the mismatching function pointer is passed, so there
are potentially hundreds of sites. Therefore, a global disabling is
appropriate. The sanitizer, on the other hand, triggers where the
function is called, which are only about two dozen places, so it seems
possible to silence these checks individually and still main the check
for accidental violations elsewhere.
Add pg_attribute_no_sanitize_function() and place it on the functions
that make such calls.
Reviewed-by: Chao Li <li.evan.chao@gmail.com>
Discussion: https://www.postgresql.org/message-id/flat/2db401d9-ef16-430e-a42c-34477006a49f%40eisentraut.org
preprocess_expression skips flatten_join_alias_vars for TABLESAMPLE
clauses, on the assumption that they cannot contain Vars of the
current query level. That's not true once a LATERAL subquery
containing a sampled relation has been pulled up: the relation's
TABLESAMPLE clause then becomes a lateral reference of this level, and
if it referred to a join alias Var, that Var is left unexpanded.
extract_lateral_references then pulls the join alias Var out as a
lateral reference and fails with "no relation entry for relid N".
To fix, treat such clauses like lateral function and VALUES RTEs:
introduce EXPRKIND_TABLESAMPLE_LATERAL, used for the TABLESAMPLE
clause of a lateral relation RTE, and keep skipping the alias
expansion only for the non-lateral kind.
Back-patch to all supported branches, as this dates back to the
introduction of TABLESAMPLE.
Author: Richard Guo <guofenglinux@gmail.com>
Reviewed-by: Ewan Young <kdbase.hack@gmail.com>
Reviewed-by: Ayush Tiwari <ayushtiwari.slg01@gmail.com>
Discussion: https://postgr.es/m/CAMbWs4834TwOtyyycyvja3MuOBkfpXy3dFAL0zt_vcM1KtqV-Q@mail.gmail.com
Backpatch-through: 14
If a subquery has been flattened into its parent, the joinaliasvars
entries of a join above it can be arbitrary expressions rather than
plain Vars, so expanding a reference to such a join alias may insert a
SubLink into a lower-level subquery. flatten_join_alias_vars_mutator
detects that and sets the subquery's hasSubLinks flag, but only in the
single-column code path; the whole-row path just asserted in a comment
that its recursive call would handle this, which is true only when the
alias entry is itself a Var referencing another join.
Hence a whole-row reference to such a join appearing in a sub-select
left that sub-select's hasSubLinks false, so preprocess_expression
skipped SS_process_sublinks for it, and the unprocessed SubLink
reached code that is not prepared for one: this produced "cannot
handle unplanned sub-select" from cost_qual_eval, an assertion failure
in preprocess_aggrefs, or "unrecognized node type" at execution,
depending on where in the sub-select the SubLink ended up.
To fix, make the same check in the whole-row expansion path.
Back-patch to all supported branches.
Author: Richard Guo <guofenglinux@gmail.com>
Reviewed-by: Ayush Tiwari <ayushtiwari.slg01@gmail.com>
Discussion: https://postgr.es/m/CAMbWs49PgeNFhQTZ2gSzwatF0_LXyegMPGt++jSfxUN7NZuQ5Q@mail.gmail.com
Backpatch-through: 14
MSVC has no counted_by attribute, but its source annotation
language (SAL) has _Field_size_(), which states the same property: the
field is a buffer whose writable size in elements is given by the
named expression.
MSVC's static analyzer (/analyze) will make use of the information.
Author: Tristan Partin <tristan@partin.io>
Discussion: https://www.postgresql.org/message-id/flat/DKBEBF32QLUA.3L2DJRPUYNKHZ%40partin.io
In an in-tree build, "make coverage-html" invoked lcov with two
"-d ." options, because $(srcdir) is just ".". With lcov 2.0
(and perhaps earlier versions?) that results in scanning every
foo.gcda file twice and producing counts twice what they should
be. With later lcov versions (at least 2.3), it results in
"duplicate file" errors.
In VPATH builds we do need two -d options to cover both regular
and generated .c files. But in in-tree builds it suffices to
specify "-d .", so suppress the second switch.
Author: Narek Galstyan <narek.galstyan@enterprisedb.com>
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/CAHsn6_xCDQWe8_vVFhtFk27_xTdyVV+Dr0yWzaooBZ6+-VH-5w@mail.gmail.com
Backpatch-through: 14
When repacking a table concurrently, some border conditions can cause
a concurrent update to propagate the original old tuple to the transient
table (such as a trigger that returns OLD). That's essentially just
wasted space, which REPACK is there to remove. Prevent that by setting
the isnull flag to true for any column cataloged as dropped in tuples
being "replayed" as concurrent updates.
Reported-by: Radim Marek <radim@boringsql.com>
Reviewed-by: Antonin Houska <ah@cybertec.at>
Backpatch: 19
Discussion: https://postgr.es/m/CAJgoLkK2UBzB1J9buCsSUbjf7bOqz-o_0=CeiTruU789BhBw-Q@mail.gmail.com
This removes the buffering of FK rows for batched PK index probes,
along with the after-trigger callback infrastructure used only by
batching. The per-row fast path, which probes the PK index directly
instead of going through SPI, remains. This is a cherry-pick of
commit 25649d6e79, which made the same change in REL_19_STABLE,
adjusted for some drift between the branches.
I (amitlan) initially thought we could keep improving the batching
path on master by fixing its remaining bugs and other issues as
they were found. However, we are still finding cases where delaying
checks to accumulate a batch changes their results, such as when other
AFTER ROW triggers modify the referenced table. Individual cases can
be addressed, but new ones keep turning up, which makes it hard to be
confident that fixing the known ones will be enough. It seems better
to rip batching out and reconsider how to introduce it, or at least
its useful pieces, while preserving the semantics of per-row checks.
Discussion: https://postgr.es/m/CA+HiwqFx4cjFtC3akr1DW-cPYw+B-EkCSffZPrkpTef8PNS6qA@mail.gmail.com
This is technically the more correct position according to C23,
although in practice compilers are relaxed about this. It also opens
us up to supporting MSVC's _Field_size_() in the future, which is a
similar construct.
This requires adding pg_attribute_counted_by to the list of
declaration attributes in pgindent, so that pgindent doesn't mess it
up.
Author: Tristan Partin <tristan@partin.io>
Discussion: https://www.postgresql.org/message-id/flat/DKBEBF32QLUA.3L2DJRPUYNKHZ%40partin.io
When the index cannot be used to find a recently deleted row,
RelationFindDeletedTupleInfoSeq() scans the table. It looked up the
relation's replica identity or primary key again to determine which
columns to compare, which could differ from the index selected when the
relation was opened. A concurrent DROP INDEX CONCURRENTLY could remove the
replica identity in between, so the search compared the whole row and
reported the conflict as update_deleted instead of update_missing.
Use the index recorded in the relation map instead. The relation map
already records the replica identity index, or the primary key if there is
no replica identity. It also avoids using a deferrable primary key, which
cannot serve as a replica identity; using its columns could incorrectly
report a deleted row with different values as update_deleted instead of
update_missing.
Author: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Author: Nisha Moond <nisha.moond412@gmail.com>
Reviewed-by: Vignesh C <vignesh21@gmail.com>
Reviewed-by: Zhijie Hou <houzhijie22@gmail.com>
Reviewed-by: Amit Kapila <amit.kapila16@gmail.com>
Discussion: https://postgr.es/m/CABdArM5ydwdRrpaZyK1q2p3-vY_+pnBtTmkvg_pcM=gHwmH7Kg@mail.gmail.com
Discussion: https://postgr.es/m/OS7PR01MB1831779ED93F5470CA1D57605F58C2@OS7PR01MB18317.jpnprd01.prod.outlook.com
Backpatch-through: 19
pre_indent() hides each alignas(...) call from pg_bsd_indent behind
a plain identifier, because pg_bsd_indent has a fixed table of
declaration keywords and lexes anything else followed by "(" as the
start of an expression.
The placeholder was not the same width as the text it stood in for,
though, and pg_bsd_indent decides where to put a trailing comment from
the width of the code preceding it. A member declared with alignas()
and carrying a trailing comment therefore got the comment separated by
a tab where the real width calls for a single space, or the other way
round. Pad the placeholder out to the original width so that the
decision is made on the real width.
No member currently declared with alignas() has a trailing comment, so
this changes nothing in the tree today; it is a correctness fix for the
helper.
Author: Tristan Partin <tristan@partin.io>
Co-authored-by: Peter Eisentraut <peter@eisentraut.org>
Discussion: https://www.postgresql.org/message-id/flat/DKBEBF32QLUA.3L2DJRPUYNKHZ%40partin.io
dumpIndex() emits ALTER INDEX .. SET STATISTICS for per-column
statistics targets, but dumpConstraint() never did. Targets set on the
expression columns of an exclusion constraint's index were silently
lost on dump/restore and pg_upgrade.
The same code can be used in the index and constraint dump code to
generate these ALTER INDEX commands. The logic has been refactored so
that both share a new subroutine in charge of generating all these
custom SET STATISTICS, if any.
Oversight in e4fca461ab.
Author: Zsolt Parragi <zsolt.parragi@percona.com>
Reviewed-by: Chao Li <li.evan.chao@gmail.com>
Reviewed-by: Andrew Krylosov <krylosov.andrew@gmail.com>
Discussion: https://postgr.es/m/CAN4CZFMd9riOuV5LkM+uV=uF-HYp49C8Fh22xWyMxptGLBCZ3A@mail.gmail.com
Backpatch-through: 14
FetchStatementTargetList() dereferenced the portal named by a
FetchStmt without checking that it still exists. With the extended
query protocol, a client can sent a Parse "FETCH n FROM c" while cursor
"c" is open (which caches a result's TupleDesc), issue a CLOSE on cursor
"c", and then Describe the prepared statement or a portal bound to it.
Describe would re-resolve the target list through the named cursor, and
crash. In a non-assert builds, this causes a NULL pointer dereference.
An assertion was triggered in assert builds slightly before the pointer
dereference.
This commit changes the describe of a FetchStmt to match with the
ExecuteStmt case: if the named cursor's portal is invalid, fail rather
than assume that the portal should always exist.
A test case is added to libpq_pipeline, backpatchable all the way down.
Author: Dirkjan Bussink <d.bussink@gmail.com>
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Discussion: https://postgr.es/m/7DE248EB-7509-4F8A-9F37-4694A40BBAB7@gmail.com
Backpatch-through: 14
pg_stat_progress_vacuum gains three fields, reflecting some activity
related to index cleanups during VACUUM, which is a phase that could
take a long time depending on the quantity of data to treat and the
number of indexes:
- current_index_relid, the index being processed.
- index_blks_total, total number of blocks to process.
- index_blks_done, number of blocks already processed.
There fields are new, and do not reuse the existing fields related to
heap as index cleanups may need multiple passes, where knowing both the
heap activity and the index activity could be relevant. This is
unlikely since tids are accumulated in the heap scanning phase using an
optimized radix tree, still possible. VACUUM can spawn parallel workers
to clean indexes (each worker cleans a different index); these should up
in the progress view in the shape of one extra row for each worker.
Bump catalog version.
Author: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Reviewed-by: Sami Imseih <samimseih@gmail.com>
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
Reviewed-by: Shihao Zhong <zhong950419@gmail.com>
Discussion: https://postgr.es/m/CALj2ACUgwSchK6jQ2CdKLBWUADTOE_zKdTff2Zg3E6hOuXKv-w@mail.gmail.com
Commit 1ff3180ca0 made a parallel autovacuum leader propagate changes
to the cost-based delay parameters to its workers only from
vacuum_delay_point(), which it no longer reaches once it is waiting in
WaitForParallelWorkersToFinish(). While waiting, the leader neither
processed a config reload nor noticed a change in the number of
autovacuum workers sharing the cost limit, so its workers kept running
with stale parameters until they finished.
Fix by refreshing and propagating the parameters on every wakeup of
that wait loop, and by setting the latches of the balanced autovacuum
workers when their count changes so that a waiting leader wakes up.
A regression test will be added in a separate commit.
Backpatch to v19, where parallel autovacuum was introduced.
Reported-by: Nikolay Samokhvalov <nik@postgres.ai>
Author: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Reviewed-by: Manu <manuelreyesbravo@gmail.com>
Reviewed-by: Zsolt Parragi <zsolt.parragi@percona.com>
Reviewed-by: Daniel Gustafsson <daniel@yesql.se>
Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
Discussion: https://postgr.es/m/CAM527d-GL=Jp2EJXBSnVBGPK-4XEZwWof5Cv8P0hghS_og6oAg@mail.gmail.com
Backpatch-through: 19
Commit 7164dcb665 added the repack_decode isolation test, whose
s1_decode step counts every change decoded from the slot. A concurrent
autovacuum transaction that modifies only system catalogs is decoded
as an empty BEGIN/COMMIT pair, so the count could come out larger
than expected.
Fix by passing skip-empty-xacts to test_decoding. We pass it, along
with include-xids=0, to s1_decode_updates as well so that both steps
use the same options for consistency, though that step shows only
UPDATE lines and wasn't affected.
Per buildfarm animal skink.
Reported-by: Alexander Lakhin <exclusion@gmail.com>
Discussion: https://postgr.es/m/d704cde4-b1f2-48dc-8c5e-659d1128d286@gmail.com
Backpatch-through: 19
There have been several recent complaints that << and >> don't behave
the same way on all supported platforms. There is a risk that <<
overflows the signed integer type, which is undefined behavior, and there
are platform-dependent behaviors from shifting by numbers of bits >= the
bitwidth of the type, and also from shifting by negative values.
Document this to warn unsuspecting users of these facts.
Author: David Rowley <dgrowleyml@gmail.com>
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/CAApHDvr0O6ZF8NO=Rq9Q+Sz=FHT=NZ1kkNTXZDcNph-OWuk7TQ@mail.gmail.com
Backpatch-through: 14
Canada's Northwest Territories will move to permanent UTC-06 on
2026-11-01. For now, assume their TZ abbreviation will be CST from
that time forward. This affects only America/Inuvik as the rest of
the territory is covered by America/Edmonton, for which the
equivalent change was released in 2026c.
Manitoba (America/Winnipeg) will move to permanent UTC-05 on
2026-10-31. For now, assume their TZ abbreviation will be EST
from that time forward.
(Upstream notes that more changes may be needed soon if smaller
jurisdictions choose to break away from what these have done,
but for now the plan is wait-and-see.)
Historical corrections for Colombia, Iran, and Ireland.
The backward-compatibility zone names EST5EDT, CST6CDT, MST7MDT, and
PST8PDT now conform better to POSIX. For example, EST5EDT now always
uses the abbreviation "EST" for standard time (now always 5 hours
behind UTC) and "EDT" for daylight saving time, whereas it formerly
had different UT offsets before standard time was introduced and
sometimes used abbreviations like "LMT", "EWT" and "EPT", all contrary
to POSIX. Also, though not specified by POSIX these names now use US
federal rules rather than rules of specific places like New York,
reverting to 2024a behavior. This change affects only timestamps
before 1966-10-30 at 01:00 standard time.
Backpatch-through: 14
Commit 0785d1b8b wrote
jsonapi_shlib.o: override CPPFLAGS += -DJSONAPI_USE_PQEXPBUFFER
forgetting that GNU make treats this change to CPPFLAGS as affecting
not only jsonapi_shlib.o, but anything that gets built in pursuit of
building jsonapi_shlib.o. Because of an even more ancient hack
(cf 7143b3e82), that will include jsonapi.o if it's not been built
already. If we do apply this -D switch to jsonapi.o, it gets built
with the wrong expansible-string infrastructure, which manifests
immediately as a link failure in pg_combinebackup and would misbehave
at runtime even if the build somehow completed.
Perhaps remarkably, we'd not seen field reports of build failures
till now, but it's pretty easy to reproduce a mis-build by trying
to build src/common/ with --shuffle and a high -j factor.
To fix, don't modify CPPFLAGS this way; instead make a dedicated
build rule for jsonapi_shlib.o that injects the additional switches.
There might be other places with similar problems, but some desultory
attempts with -j32 --shuffle on a 32-thread CPU didn't find any more
build failures. Also, it looks like the meson build infrastructure
lacks this problem (but I'm no meson expert).
Bug: #19727
Reported-by: Ross Burton <ross@burtonini.com>
Author: Tom Lane <tgl@sss.pgh.pa.us>
Reviewed-by: Ross Burton <ross@burtonini.com>
Discussion: https://postgr.es/m/19727-32d67d9dea410f38@postgresql.org
Backpatch-through: 18
The explanation of aggressive scans claimed that if the multixact
members grows to 2 billion entries, we launch autovacuum even if it's
nominally disabled. That's no longer true; anti-wraparound vacuum is
only triggered if we're in danger of running out of multixids, we no
longer launch it just to shrink the 'members' SLRU. There's no
wraparound danger with 'members' anymore, so it's no worse than if a
user table becomes very bloated.
We do still adjust the freeze threshold though, so that if autovacuum
runs, whether on schedule or as an emergency autovacuum to avoid
wraparound, it will freeze more aggressively if the members SLRU is
large.
I removed the mention of the 4 billion member threshold, even though
that still exists, because the explanation felt misleading. The
aggressiveness is scaled proportionally between 2 and 4 billion
members. At 4 billion it's as aggressive as it gets, but there's no
sudden cliff.
In the passing, fix some comments. Remove obsolete mention of safe
threshold for member storage triggering autovacuum. Secondly, the
freezing cutoff is adjusted to avoid the *members* SLRU from growing
too large.
Reported-by: Noah Misch <noah@leadboat.com>
Discussion: https://www.postgresql.org/message-id/20260827231757.78.noahmisch@microsoft.com
Backpatch-through: 19
ShmemRequestInternal() rejected SHMEM_ATTACH_UNKNOWN_SIZE whenever
IsUnderPostmaster was false. That also rejected an after-startup
attachment in a standalone backend, even if the area exists.
To fix, check whether the shared memory has been initialized
(ShmemIndex == NULL) rather than IsUnderPostmaster.
Refactor the tests, and add more test cases for after-startup
attachment, with SHMEM_ATTACH_UNKNOWN_SIZE and a mismatching size, and
for this single-user case.
Author: Ayush Tiwari <ayushtiwari.slg01@gmail.com>
Reviewed-by: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>
Discusssion: https://www.postgresql.org/message-id/CAJTYsWXREGXCnWdZJsXt8pT2UY_Y2GSx+HOUu2X3s8YDCLj0_A@mail.gmail.com
Backpatch-through: 19
ReplicationSlotRelease() copied MyProc->statusFlags unconditionally
when resetting PROC_IN_LOGICAL_DECODING, using its pgxactoff to decide
which area to update.
Slot invalidations can be done by the checkpointer or the startup
process, and ReplicationSlotRelease() ignored the fact that pgxactoff is
only meaningful for processes in the proc array, that auxiliary
processes never join. pgxactoff would remain at 0 for these, meaning
that an incorrect set of status flags would be updated.
The consequences of this issue could be various, depending on the
statsFlags lost in the proc slot incorrectly overwritten. Losing
PROC_AFFECTS_ALL_HORIZONS has been mentioned as the worse case, causing
VACUUM to remove rows a standby may needs even if standby feedback is
enabled. Assertions failures are another possibility.
Let's skip the update of statusFlags unless PROC_IN_LOGICAL_DECODING is
set. Nothing but StartupDecodingContext() sets it, and no auxiliary
processes call it.
Author: Vlad Lesin <vladlesin@gmail.com>
Reviewed-by: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Discussion: https://postgr.es/m/889a06dd-45fc-423b-9dd2-87d5b5dc60c6@gmail.com
Backpatch-through: 14
SJE didn't consider the possibility that two RTEs referencing the
same table have different securityQuals, and might choose to merge
them anyway, thereby possibly losing quals that need to be enforced.
This is a regression introduced by 2ebf25e7d, since before that we
didn't re-generate baserestrictinfo lists from the RTEs' securityQuals
after performing SJE.
To defend against this, only consider SJE between RTEs with the same
checkAsUser values. That solves the problem because the set of
applicable RLS policies depends only on the role that is considered
to be accessing the table, so that the securityQuals must be equal
if the checkAsUser values are. It might also keep us from creating
similar bugs if we ever invent other features that depend on the
accessing role. And it's a lot cheaper than comparing securityQuals
trees themselves would be, not least because we can't sort them so
the preliminary sort step wouldn't help.
(The first proposed solution was to not perform SJE at all on RTEs
with nonempty securityQuals, but that seems rather sad, especially
since such cases worked before 2ebf25e7d. Making the restriction
depend on checkAsUser seems much less likely to interfere with SJE
unnecessarily, since in most cases that'll be the same for all
potentially-mergeable RTEs.)
The added test cases show that SJE is rejected when necessary, and
also demonstrate two quirks of this implementation. One is that if
a removed security qual includes an InitPlan, we'll still attach the
now-unused InitPlan to the plan. That's because SS_process_sublinks
runs before self-join elimination, so the extra InitPlan has already
been made. The other quirk is that we won't merge RTEs having zero
and nonzero checkAsUser fields, even if the calling user matches the
nonzero checkAsUser value so that there is no difference for this
user. That's intentional so that SJE doesn't require having to mark
the plan as caller-dependent.
Reported-by: Yonghwa Lee <underdog@theori.io>
Author: Tom Lane <tgl@sss.pgh.pa.us>
Reviewed-by: Ayush Tiwari <ayushtiwari.slg01@gmail.com>
Discussion: https://postgr.es/m/20260926174457.13.noahmisch@microsoft.com
Backpatch-through: 18
The help text for -c option documents that zero for either transaction
ID means to leave the corresponding value unchanged. Commit
978cf02bb8 replaced the separate checks for the two values with a
single flag indicating that the option was specified, causing zero to
be written to the control file instead.
Also parse the second transaction ID with strtouint32_strict(), like
the first one. Using strtoul() there was an oversight in commit
636c1914b4, which intended to reject negative and out-of-range
arguments.
Author: Chao Li <lic@highgo.com>
Discussion: https://www.postgresql.org/message-id/F61E486B-746B-42F3-A532-B8D4F1519583@gmail.com
Backpatch-through: 19
calcSumX2 is used to inform the aggregate transition function whether it
should include the square of the transition value in the aggregate
state. The deserialize functions for various numeric aggregates that do
use calcSumX2 weren't properly setting the property to true. This does
not matter in core as we never perform any aggregate transitions after a
deserialize operation.
This fix is only required for external code which plumbs aggregate paths
together in such a way that could result in aggregate transitions being
performed after states have been deserialized.
Author: Maxime Schoemans <maxime.schoemans@enterprisedb.com>
Reviewed-by: Andrey Borodin <x4mmm@yandex-team.ru>
Discussion: https://postgr.es/m/FEBF5C3D-CC53-455F-AA9D-4D377A2E957C@enterprisedb.com
Backpatch-through: 14
If no DSM segment can be created, a ParallelContext can still be
created with zero workers, but its seg pointer is NULL. Parallel
autovacuum registered a segment detach callback on that NULL pointer,
crashing the autovacuum worker.
Fix by setting up the shared cost parameters and the detach callback
only when the parallel context has workers, as there are no workers to
propagate the parameters to otherwise.
Oversight in commit 1ff3180ca0.
Found by Bharath using AI assisted review with Claude.
Reported-by: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Author: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Discussion: https://postgr.es/m/CALj2ACVjBN03Fq4XWgKH1zOfkapK6-P3ufNLHUHnZcW1zPm+Mg@mail.gmail.com
Backpatch-through: 19
Presently, s_lock.h insists that every platform provide TAS(), and
its header comment doesn't say which of the macros a new platform
actually needs to supply. The real contract is that a platform
must provide either S_LOCK() or a TAS() for the default S_LOCK() to
be built on.
To better express that, this commit moves the "no spinlock support"
error to where the default S_LOCK() is defined, compiles s_lock()
only when the default S_LOCK() is in use, and provides a default
TAS_SPIN() only when there is a TAS() to base it on. The direct
callers of s_lock() in the spinlock tests are adjusted to match.
The header comment is reworked accordingly, dropping the note about
pre-9.5 volatile requirements and the caution about TAS()
spuriously failing along the way.
Co-authored-by: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/afkUeI7UhacZ5ZFm%40nathan
create_partial_grouping_paths() called the FDW callback
GetForeignUpperPaths for UPPERREL_PARTIAL_GROUP_AGG, but never the
general-purpose create_upper_paths_hook that non-FDW extensions rely
on. create_partial_distinct_paths() already fires both for
UPPERREL_PARTIAL_DISTINCT.
Add the missing call right next to GetForeignUpperPaths, so an
extension can add partial-aggregate paths here too, before the caller
gathers them and builds a Finalize Aggregate on top.
Reviewed-by: Bryan Green <dbryan.green@gmail.com>
Reviewed-by: Himanshu Upadhyaya <upadhyaya.himanshu@gmail.com>
Reviewed-by: Shihao Zhong <zhong950419@gmail.com>
Discussion: https://postgr.es/m/6fc72a7d-9f52-4a72-9adf-1912da255605@dunslane.net
Switch to wait_for_replay_catchup in checksum tests to explicitly
wait for the upstream flush LSN and not the write LSN. The write
LSN can include unrelated unflushed WAL which isn't available for
streaming on an idle primary.
In the test for using a rewound node as standby of a new primary in
the 013_rewind suite; also switch WAL on the source before starting
the target. This flushes through the minimum recovery point chosen
to avoid a startup delay while an unflushed full_page_writes change
waits for the background writer to log its next snapshot.
This reduces test times significantly without changing the semantics
of the tests.
Author: Nazir Bilal Yavuz <byavuz81@gmail.com>
Reviewed-by: Zsolt Parragi <zsolt.parragi@percona.com>
Reviewed-by: Daniel Gustafsson <daniel@yesql.se>
Discussion: https://postgr.es/m/CAN55FZ1Yak_xBqMaDQsD7atpBkGLEkF-DKXcs3nLHM1Uq4YRew%40mail.gmail.com
The decoding worker records the relfilenumber of the TOAST table when it
starts, and skips changes stored under any other one, but the TOAST
table was not locked until the data copy began. If VACUUM FULL rewrote
it while the worker was starting, concurrent updates of TOASTed columns
were silently lost.
Fix by locking the TOAST table in cluster_rel() in the concurrent case,
before the worker starts.
Author: Shihao Zhong <zhong950419@gmail.com>
Reported-by: Thom Brown <thom@linux.com>
Backpatch-through: 19
Discussion: https://postgr.es/m/CAA-aLv5MF6BLL+BWvix2Yw+CBardtH43AofPReQunhDZPNBtuA@mail.gmail.com
When searching the local relation by index, the apply worker can use the
first match only if the index is the relation's replica identity or
primary key. Otherwise, every match must be compared with the search slot.
Only the index OID was saved in the relation map, so this was determined
again during the search. Concurrent DROP INDEX or REINDEX can change the
index status between these two checks, causing the tuple search to fail
and the change to be silently skipped as an update_missing conflict.
Fix this by recording the answer as idxisreplident in the relation map
entry and avoid getting it second time in FindReplTupleInLocalRel() and
RelationFindReplTupleByIndex().
Oversight in 89e46da5e5.
Author: Mikhail Nikalayeu <mihailnikalayeu@gmail.com>
Author: vignesh C <vignesh21@gmail.com>
Reviewed-by: Amit Kapila <amit.kapila16@gmail.com>
Reviewed-by: Zhijie Hou <houzj.fnst@fujitsu.com>
Discussion: https://postgr.es/m/CADzfLwUJovFcnknCC9wjZKECX9xecgnGzC2r2TMV8h4QDD_jwQ@mail.gmail.com
Backpatch-through: 16, where it was introduced
Since 112faf1378, NOTICE, WARNING, and similar messages received through
dblink, postgres_fdw, and replication connections have been logged with
ereport(LOG). With the default log_min_error_statement setting, each
such log entry can also include the current local SQL statement. An active
error-context callback can add local CONTEXT information as well. Repeating
these fields for every remote message increases log volume and processing
overhead.
Use errhidestmt() and errhidecontext() when logging these messages to
prevent the local STATEMENT and CONTEXT fields, when present, from being
repeated for each remote message in the server log.
Reported-by: Merlin Moncure <mmoncure@gmail.com>
Author: Vignesh C <vignesh21@gmail.com
Reviewed-by: Merlin Moncure <mmoncure@gmail.com>
Reviewed-by: Fujii Masao <masao.fujii@gmail.com>
Discussion: https://postgr.es/m/CAHyXU0yDwOYkWpRi=gtpDwjBxVqeaezgkgZLRWm2p1_FgaiPFQ@mail.gmail.com
Backpatch-though: 19
The counted_by attribute allows specifying that a flexible array
member of a struct is "counted by" another member of the same struct.
The attribute makes arrays a little more self-documenting, but it is
also a hint to the compiler such that it can improve detection of
object size information and provide better results in compile-time
diagnostics and run-time features, such as the array bounds sanitizer.
Both GCC and clang warn when the attribute is used in C++, where it is
ignored, so define it away there.
In its current form, it is best used for flexible array members only.
Pointer array support may come later on.
This commit only contains a single use of the attribute for testing.
More will be added afterwards.
Author: Tristan Partin <tristan@partin.io>
Reviewed-by: Peter Eisentraut <peter@eisentraut.org>
Reviewed-by: Japin Li <japinli@hotmail.com>
Reviewed-by: Greg Burd <greg@burd.me>
Discussion: https://www.postgresql.org/message-id/flat/DKBEBF32QLUA.3L2DJRPUYNKHZ%40partin.io
Previously, pg_locks.waitstart could report the start time of a previous
lock wait for a new wait. For a regular backend, this could happen briefly
before the new start time was stored. On a standby, the startup process could
show the old time throughout the next wait, making it appear to have started
earlier than it did.
This happened because PGPROC->waitStart could remain set after a previous
wait in two cases. First, RemoveFromWaitQueue() did not clear it when a wait
was canceled due to a lock timeout, cancellation, or deadlock. Second, even
after a successful lock grant, ProcWakeup() could clear it before the waiter
stored its new start time.
Fix this by clearing waitStart in RemoveFromWaitQueue() when a failed wait
is removed, at the end of ProcSleep() to handle writes made after
ProcWakeup() cleared it, and in LockErrorCleanup() when an error bypasses
the ProcSleep() reset. This prevents pg_locks.waitstart from showing the
previous wait's time for a new wait.
Backpatch to all supported versions.
Author: Shihao Zhong <zhong950419@gmail.com>
Reported-by: Alex Shapalov <shapalov@gmail.com>
Reviewed-by: Chao Li <li.evan.chao@gmail.com>
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Reviewed-by: Fujii Masao <masao.fujii@gmail.com>
Reviewed-by: Andrew Krylosov <krylosov.andrew@gmail.com>
Discussion: https://postgr.es/m/CAPrb+Q+XN=sNusXiUeWmMo2H7Qgq3Y4uPekSSLkHcnCyf7GhXg@mail.gmail.com
Discussion: https://postgr.es/m/CAGRkXqQLxZBr-ouVrtaX2utMggi4+TiVMbgH04b_0JLgKryxbA@mail.gmail.com
Backpatch-through: 14
This marker could break translated strings if used, so let's remove it
and encourage instead the use of PRIu64, something that xgettext is able
to digest.
This marker has been introduced in b139bd3b6e, down to v19 that has
not been officially released yet, so a removal is still possible without
breaking the ABI requirements.
Discussion: https://postgr.es/m/aqeO1OU7uZjhioAq@alvherre.pgsql
Backpatch-through: 19
The OID8_FORMAT marker could break translated messages, as proven by
ceae3099da, so let's remove it while we can instead of encouraging its
use.
An alternative solution that would allow OID8_FORMAT is to save the
result into an intermediate string, itself used in the translated
strings, but per discussion this is not an appealing long-term approach.
Note that this commit partially reworks 072d4c4240.
Per discussion with Peter Eisentraut, Álvaro Herrera and Tom Lane.
Discussion: https://postgr.es/m/aqeO1OU7uZjhioAq@alvherre.pgsql
toast_save_datum() ended with two copies of the same three steps, one
each external on-disk TOAST pointer: allocate a datum, set its tag, then
copy the contents from a varatt_external_oid or a varatt_external_oid8
into it.
These steps are moved into a new helper function, toast_pointer_build(),
which takes the tag and a pointer to the struct and derives everything
else from the tag. For now, only two varatt_external_* exist for
external on-disk TOAST pointers. Some more are under discussion, where
this refactoring shows some benefits in the shape of less duplication.
There should be no behavior change.
Author: Nikhil Kumar Veldanda <veldanda.nikhilkumar17@gmail.com>
Discussion: https://postgr.es/m/CAFAfj_FeJQZrKyVn7ur0aJx8iz8SR14NheVJNyc8JNoqtMX+og@mail.gmail.com
_reconnectToDB() failed to consider the possibility that it could
be called without any restrict_key set. This is reachable while
generating the "restore.sql" file in tar dump mode, which nobody
noticed because that file is mostly unused (and nothing really
bad happens anyway unless you specified --inserts mode). But
I wonder if there are other code paths that reach the issue.
Bug: #19718
Reported-by: Pavel Kopylov <pkopylov@cloudlinux.com>
Author: Pavel Kopylov <pkopylov@cloudlinux.com>
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/19718-7e945d0ff9b5a589@postgresql.org
Backpatch-through: 14
When ANALYZE imports statistics from a remote server, it fetched/stored
the statistics while still running as the user executing ANALYZE, not as
the foreign table's owner. This is inconsistent with the sampling path,
and it causes a security issue: a role that owns a foreign table but has
no access to the underlying remote data (eg, no access privileges on the
remote server) could obtain that data by having a privileged user run
ANALYZE on the table; the imported statistics like most_common_vals then
expose sampled values from the remote data, as shown in the reproducer
on the discussion thread.
To fix, switch to the foreign table owner's userid in analyze_rel()
before calling the ImportForeignStatistics() routine, mirroring the
identity used by the sampling path. This not only makes the privilege
handling consistent between the sampling/import paths, but also prevents
statistics import from disclosing data the table owner couldn't
otherwise obtain.
Oversight in commit 28972b6fc.
Reported-by: Fujii Masao <masao.fujii@gmail.com>
Reported-by: Osama Abdul Qader <osamaabdulqader.cs@gmail.com>
Reported-by: Noah Misch <noah@leadboat.com>
Author: Noah Misch <noah@leadboat.com>
Reviewed-by: Matheus Alcantara <matheusssilv97@gmail.com>
Reviewed-by: Etsuro Fujita <etsuro.fujita@gmail.com>
Discussion: https://postgr.es/m/CAPmGK16jVk+i2KMkkgR9ajoPdaUAinvcspkk5Bc6urbo2xYTMQ@mail.gmail.com
Backpatch-through: 19
ri_FastPathCheck() invokes two functions on the FK values: the
equality operator's function, via the ScanKey the index AM evaluates,
and, when the FK column's type differs from the PK column's, the
implicit cast function. The SPI path gets EXECUTE on both checked by
ExecInitFunc() when it initializes the generated query, as the
referenced table's owner, but the fast path performed neither check.
Add ri_CheckFunctionPermissions() and call it once the fast-path
metadata is populated, under the switch to the referenced table's
owner, so the checks happen at the same point as the existing relation
permission check. Add a regression test for each function, reaching
the fast path through per-row validation of a new constraint, and
repeat each against a partitioned referenced table, which takes the
SPI path, to show that the two behave identically.
Reported-by: Nikolay Samokhvalov <nik@postgres.ai> (offlist)
Reviewed-by: Matheus Alcantara <matheusssilv97@gmail.com>
Discussion: https://postgr.es/m/CA+HiwqGH+b7sXmsH8sT18diujbheSObWs0gWXqLZqRccMKvZAA@mail.gmail.com
Backpatch-through: 19
