Tied arrays might report different sizes each time they are inspected.
To avoid generating a corrupt result array, fix plperl_array_to_datum()
to read av_len() of each input array only once. If the input does
appear to get shorter, we'll fill nulls for the now-missing entries,
which seems fine. Conversely, if it gets longer, we'll ignore the new
entries.
plperl_to_hstore() assumed that Perl's hv_iterinit() returns the
number of entries in the given Perl hash. Usually that's true,
but per the Perl docs, "the return value is currently only meaningful
for hashes without tie magic". That could potentially end in a memory
stomp. We don't depend on that result value anywhere else, so don't
do so here either.
Reported-by: Hcamael <baiyjrh@gmail.com>
Author: Tom Lane <tgl@sss.pgh.pa.us>
Reviewed-by: Andrew Dunstan <andrew@dunslane.net>
Backpatch-through: 14
Security: CVE-2026-14670
setup_regexp_matches() sizes the buffer used to convert matched
substrings back from pg_wchar form at the smaller of maxlen*eml and
the original string's byte length, on the assumption that such a
conversion cannot produce more bytes than the string it came
from. That assumption holds only for validly encoded input. But
pg_mb2wchar_with_len() silently accepts bytes that are invalid in the
database encoding, turning each such byte into one pg_wchar, and
converting that back can take more bytes than the input did. A string
made of such bytes therefore overruns the conversion buffer by up to
its own length, corrupting the following memory. regexp_match(),
regexp_matches(), regexp_split_to_table() and regexp_split_to_array()
are all affected.
Fix by dropping the tighter bound and always allocating maxlen*eml + 1
bytes.
Reported-by: Francesco Verardi <frevadiscor89@gmail.com>
Author: Masahiko Sawada <sawada.mshk@gmail.com>
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Backpatch-through: 14
Security: CVE-2026-14664
The only overflow hazards I could find in tsquery construction
are in QTN2QT(), which builds a flat tsquery datum from the
QTNode tree representation used by tsquery_or, tsquery_rewrite,
and allied functions. There are two:
1. It seems theoretically possible for the outputs of cntsize() to
overflow an int, so I widened them to size_t. There's no hazard
certainly in tsquery_or and friends, but tsquery_rewrite could expand
the query tree by large multiples (by replacing many identical
subtrees with a large replacement tree), so in a 64-bit machine
with plenty of available memory it should be possible to build a
QTNode tree large enough to cause that. If these counters did
overflow then we'd under-allocate the output tsquery and have a heap
overwrite problem. size_t is sufficient, since it's counting the size
of a subset of an in-memory data structure. We also have to fix the
TSQUERY_TOO_BIG() macro to not get confused if sumlen exceeds
MaxAllocSize.
2. fillQT() neglects to check that the new "distance" value for a
QI_VAL item fits into the available 20-bit field. It's quite easy
to reach this, for example by tsquery_or'ing two near-megabyte-sized
tsquerys. However, the result is only a corrupt tsquery that does
not represent the expected query, so perhaps this doesn't rise to
the level of a security bug. Nonetheless it should be fixed.
Note: I followed the practice used in other tsquery code of checking
each distance value as it's assigned, which means that the last
operand string could extend past the MAXSTRPOS boundary. This is a
bit different from the pattern used for tsvectors, which insist that
the total data length not exceed MAXSTRPOS and thereby avoid making
per-item checks. Perhaps that should be harmonized sometime, but for
now it's okay for the two types to do this differently as long as
each one is self-consistent.
Author: Tom Lane <tgl@sss.pgh.pa.us>
Reviewed-by: Amit Langote <amitlangote09@gmail.com>
Backpatch-through: 14
Security: CVE-2026-14662
The core of this patch is to prevent array_to_tsvector() from
generating invalid tsvectors. It did not check for overly-long
lexemes (so that WordEntry.len fields could overflow), nor did it
check that the total "datalen" fits within MAXSTRPOS (so that
WordEntry.pos fields could overflow, and the number of entries
in the tsvector could be much more than the normal limit).
While the field overflows couldn't do anything much worse than
produce a corrupted tsvector value, a sufficiently large number
of tsvector entries could cause integer overflows in later
processing, such as tsvectorout.
Another important fix is to prevent tsvectorrecv() from accepting
invalid tsvectors. The main problem there is that it did not
reject empty-string lexemes. Hence, even though it did (mostly)
enforce the MAXSTRPOS limit, it could still produce a result
with an unreasonable number of tsvector entries, if they were
primarily empty strings.
Also, fix tsvectorout's calculation of its required output
buffer size: it was multiplying the string lengths by
pg_database_encoding_max_length() for no reason. That contributed
to the risk of integer overflow there. With valid tsvector input,
there's no risk, but there's still no reason to make the output
buffer several times bigger than needed.
I also tried to make a couple of related routines more robust,
and spent some effort on improving the comments in ts_type.h.
Also, standardize on a single spelling of the "string is too long
for tsvector" message, using %zu instead of an assortment of formats.
These changes aren't security per se but came out of inspecting the
code for problems.
Reported-by: Yuhang Wu <yuhang@depthfirst.com> and Zhenpeng Lin
Reported-by: Zheng Yu <zheng@depthfirst.com>
Reported-by: Hcamael <baiyjrh@gmail.com>
Author: Tom Lane <tgl@sss.pgh.pa.us>
Reviewed-by: Amit Langote <amitlangote09@gmail.com>
Backpatch-through: 14
Security: CVE-2026-14662
The logic for asynchronous Append assumes that pending requests made for
subplans of an Append are drained during ExecReScanAppend. To ensure
that, commit 9e283fc85 modified postgresReScanForeignScan to drain such
a request if any, but failed to take into account that if such a request
was made for a subplan that is re-scanned with parameter changes or
pruned in the next round by runtime pruning, the postgres_fdw callback
function is called after ExecReScanAppend or never called, respectively.
This would cause such a request to remain even after ExecReScanAppend,
leading to incorrect results, an infinite loop, or an assertion failure.
To fix, modify ExecReScanAppend to, for each of the pending requests,
give the FDW a chance to drain that request using the existing
ForeignAsyncConfigureWait/ForeignAsyncNotify callback functions. This
makes the change made to postgresReScanForeignScan useless, so remove it
as well.
Back-patch to v14 where asynchronous Append was added.
Reported-by: Alexander Korotkov <aekorotkov@gmail.com>
Co-authored-by: Alexander Korotkov <aekorotkov@gmail.com>
Co-authored-by: Gleb Kashkin <g.kashkin@postgrespro.ru>
Co-authored-by: Etsuro Fujita <etsuro.fujita@gmail.com>
Reviewed-by: Alexander Pyhalov <a.pyhalov@postgrespro.ru>
Reviewed-by: Gleb Kashkin <g.kashkin@postgrespro.ru>
Discussion: https://postgr.es/m/CAPpHfduMOTnV5Zj2KGJ7zanL_10QvccZHtPUaDfJvBhsh9axnQ%40mail.gmail.com
Backpatch-through: 14
When pgstat_init_entry() fails due to an OOM in the DSA allocation,
pgstat_get_entry_ref() cleaned up the shared hashtable but forgot to
remove the local reference that pgstat_get_entry_ref_cached() had
already inserted into pgStatEntryRefHash.
Missing this cleanup would leave a backend with a stale local cache
entry whose entry_ref points to a NULL shared_stats. If
pgstat_gc_entry_refs() runs with this reference still around, it would
crash due to a pointer dereference.
The local reference is now removed before removing the shared entry,
the order being sensitive to pending interrupts.
Oversight in 8191e0c16a.
Author: Niall Newman <nn@turacolabs.com>
Discussion: https://postgr.es/m/2FDAA194-9CF3-4FD7-A450-F1A4BEB125F6@turacolabs.com
Backpatch-through: 15
Buildfarm member olingo intermittently failed this test, timing out while
waiting for the subscriber log to report an ERROR because
max_prepared_transactions is zero there. The test captured the log offset
only after issuing the publisher's
BEGIN/INSERT/PREPARE TRANSACTION/COMMIT PREPARED sequence.
Since streaming is enabled, the subscriber can receive and apply the
transaction, and log the expected ERROR, before that publisher SQL command
even returns, i.e. before the test captures the offset. The subsequent
wait_for_log() calls then searched only from a point after the message had
already been written, and timed out waiting for it.
Fix by moving the offset capture to before the publisher's transaction is
issued, ensuring it always precedes the point where the ERROR can appear
in the subscriber log.
Reported-by: Alexander Lakhin <exclusion@gmail.com>
Author: Zhijie Hou <houzj.fnst@fujitsu.com>
Reviewed-by: Amit Kapila <amit.kapila16@gmail.com>
Backpatch-through: 16, where test was introduced
Discussion: https://postgr.es/m/c43753d8-5265-4f77-83ff-9b1167276ec5@gmail.com
seq_search_localized() returns the length of the matching prefix in
*len, but because it internally case-folds the inputs, it gets
confused on the length. The caller expects to get the length of the
prefix in the original string, but what it actually returns is the
length of the prefix after case-folding, which can be different if the
case-folded characters have different byte-length than the original,
or with ICU, if the case-folding changes the number of characters
(e.g. "ß", the German double s).
To fix, once we have determined that we have a match, work harder to
find the match's length in the original string. This adds some
overhead, but the strings are expected to be short.
The function does "case-folding" by converting a string to upper-case,
then to lower-case, which is a little ugly given that we have
dedicated functions for case-folding nowadays. But switching to that
doesn't seem appropriate to backpatch in a security fix, and that's
not available in older stable versions, anyway.
Author: Heikki Linnakangas <heikki.linnakangas@iki.fi>
Reported-by: Xint Code
Reviewed-by: Jeff Davis <pgsql@j-davis.com>
Per commit 8bf6ec3ba3, a column can be GENERATED only if it is such in
the whole inheritance tree.
For this reason, ATPrepDropExpression refuses to be called with ONLY on
a partitioned table. To detect this, the current implementation checks
whether recurse is set to false and the rel has direct children.
Recursion is implemented with ATSimpleRecursion, which calls ATPrepCmd
with recurse = false for every node in the tree. Inner nodes (for
example a partition which itself has subpartitions) then fail the check,
accidentally preventing the command from working on inheritance trees of
depth > 2.
This commit fixes it by also checking that we're at the top level of the
recursive calls using the recursing parameter, which is always true when
called through ATSimpleRecursion, always false when invoked on the root
rel.
Also, remove a comment claiming that DROP EXPRESSION could be
implemented with some effort. It cannot, as the commit message for
8bf6ec3ba3 explains.
Author: Alberto Piai <alberto.piai@gmail.com>
Backpatch-through: 14
Discussion: https://postgr.es/m/DHMT78XOD8BK.341V3H87KZ7NO@gmail.com
Fix missing NULL check in MemoryContextAllocAligned(). The underlying
call to MemoryContextAllocExtended() could return NULL when
flags contains MCXT_ALLOC_NO_OOM and the underlying malloc fails.
There are no current callers using MemoryContextAllocAligned() that pass
the MCXT_ALLOC_NO_OOM in core, so no live bug fix in core here. However,
an extension might use this pattern, so we'd better fix.
Fix this so we correctly pass the NULL to the caller rather than trying
to write to a NULL memory address.
This also fixes the same bug in AlignedAllocRealloc(), which is also
unused in core.
Backpatch to v16, where these functions first appeared.
Author: Chao Li <li.evan.chao@gmail.com>
Discussion: https://postgr.es/m/07DAC4C3-120D-4F3C-8FEE-BA236F7E9C1D@gmail.com
Backpatch-through: 16
Commit 6566133c5f added a case for AuthMemRelationId in
AcquireDeletionLock(), but not ReleaseDeletionLock(). The fall-through
case would go to UnlockDatabaseObject(), which would raise a WARNING;
and the lock would be retained until the end of the transaction.
Add the missing branch.
Discussion: https://postgr.es/m/2487ddcd737d4fc8e408e87aa9ad4365eed3bbb3.camel@j-davis.com
Backpatch-through: 16
Logging connection information, even at DEBUG1, creates unnecessary
risks. Remove the entire log message because it had no other useful
content.
Addresses finding 14 in report from linked discussion.
Reported-by: Noah Misch <noah@leadboat.com>
Discussion: https://postgr.es/m/20260710195902.4f.noahmisch@microsoft.com
Backpatch-through: 14
In the not-too-likely scenario where we successfully created a hash
table entry for a TS dictionary, but then failed to make a small
memory context for it, we left the hash entry in existence but with
a garbage value for dictCtx. This confused the code the next time
through, leading to a crash. Rearrange things so that we leave
the hash entry in a well-defined state with dictCtx == NULL, and
then the next try knows it still needs to make a memory context.
Reported-by: Alexander Lakhin <exclusion@gmail.com>
Author: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/0f3ddeb5-0dbd-479c-9d0e-ae254758e624@gmail.com
Backpatch-through: 14
Allocate CompoundAffix with room for its terminator, initialize the
old-format flag buffer before NIAddAffix(), and reject incomplete or
missing Hunspell AF aliases. None of these errors would be likely to
trigger on real dictionary files, accounting for the lack of previous
reports; but they're certainly bugs.
Bug: #19595
Reported-by: Michael Malis <michaelmalis2@gmail.com>
Author: Andrey Rachitskiy <pl0h0yp1@gmail.com>
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/19595-7dc18b4e212c4757@postgresql.org
Backpatch-through: 14
Alberta (America/Edmonton) moved to permanent UTC-06 on
2026-06-18, which will affect their clocks beginning on 2026-11-01.
For lack of any clarity on the point, assume their TZ abbreviation
will be CST from that time forward.
Morocco (Africa/Casablanca) will move to permanent UTC+00,
without daylight saving time transitions, on 2026-09-20.
Backpatch-through: 14
Partition pruning for RANGE-partitioned tables could mistakenly prune
the DEFAULT partition in some cases when it was not valid to do so,
which could lead to rows missing from query results.
The only known cases where this could happen is when combining pruning
steps from an IS NOT NULL clause with other steps that matched to the
DEFAULT partition. This could occur due to RANGE partitioned tables
having two distinct internal representations for marking if the DEFAULT
partition should be scanned. The IS NOT NULL steps would mark the
"scan_default" boolean, but other steps created for different purposes
could mark a bound_offset Bitmapset, which would ultimately translate into
also scanning the default partition. This could all fail after multiple
steps were combined with a combine intersect operator, as that will
intersect the bound_offset bits and only set scan_default if all pruning
steps have that flag set. When both input steps to the intersect operator
had different representations of whether to scan the DEFAULT partition,
the resulting intersect step result would contain neither representation.
Here, we fix this by having the IS NOT NULL pruning result mark the
bound_offsets so that it uses both representations to mark that the
DEFAULT partition must be scanned.
Reported-by: Jacob Brazeal <jacob.brazeal@gmail.com>
Diagnosed-by: Jacob Brazeal <jacob.brazeal@gmail.com>
Author: David Rowley <dgrowleyml@gmail.com>
Discussion: https://postgr.es/m/CA+COZaDXrfTaBjLE=Z79MTaH6Xun1V4PeKxLvCNv8mXS8wn0rw@mail.gmail.com
Backpatch-through: 14
A cascading standby could fail to reconnect to its upstream standby with
"requested starting point ... is ahead of the WAL flush position" after
falling back to archive recovery. This happened because archive
recovery processes whole segment files, so after replaying a segment the
cascade's next read position lands at the start of the following
segment, which is ahead of the upstream's flush position reported by
GetStandbyFlushRecPtr() (still inside the just-replayed segment).
Fix by having the walreceiver check the upstream's current WAL flush
position via IDENTIFY_SYSTEM before issuing START_REPLICATION.
IDENTIFY_SYSTEM already returns this position (as xlogpos), but
walrcv_identify_system() previously discarded it; now we have a use for
it. If the requested start point exceeds the upstream's flush position
on the same timeline, the walreceiver waits for
wal_retrieve_retry_interval and retries.
The wait is limited to gaps of at most one WAL segment, which is the
expected case from the segment-granularity of archive recovery. Larger
gaps indicate the upstream is genuinely behind, so START_REPLICATION is
allowed to proceed (and fail) normally, letting the startup process fall
back to other WAL sources. The first wait is logged at LOG level;
subsequent waits are demoted to DEBUG1 to avoid log noise. The
walreceiver honors wal_receiver_timeout during the wait, so it will exit
if the upstream doesn't catch up in time.
To preserve ABI compatibility on back branches, the flush position from
IDENTIFY_SYSTEM is communicated via a new global variable
(WalRcvIdentifySystemLsn) rather than changing the signature of
walrcv_identify_system().
The bug was introduced in Postgres 9.3 by commit abfd192b1b, which
added a flush-position check in StartReplication() that rejects requests
ahead of the upstream server's WAL flush position.
Author: Marco Nenciarini <marco.nenciarini@enterprisedb.com>
Reviewed-by: Xuneng Zhou <xunengzhou@gmail.com>
Backpatch-through: 14
Discussion: https://postgr.es/m/CA+nrD2cTuTkkX5WXVZengTYYZbAO6zV8K+Tri-R0fbLFuoyMBA@mail.gmail.com
find_nonnullable_rels and find_nonnullable_vars mistakenly treated a
ScalarArrayOpExpr that could return FALSE as strict, but that's okay
only at top level of a qual expression; further down, we've got to
insist on a guaranteed-NULL result. The result was that we could draw
mistaken conclusions about whether outer joins can be simplified, if
the decision hinged on a non-top-level ScalarArrayOpExpr with a
potentially-empty array argument.
I believe this error dates to commit 72a070a36, which taught
find_nonnullable_rels to descend into non-top-level parts of qual
expressions. is_strict_saop (added earlier by 72153c058) already had
enough intelligence to do the case correctly, but it wasn't passed the
proper flag, ie "top_level" needs to be passed for "falseOK".
e006a24ad copied that mistake into find_nonnullable_vars.
Later, over-eager refactoring in commit 2f153ddfd broke
contain_nonstrict_functions' handling of ScalarArrayOpExpr by treating
it as though it were no different from an OpExpr. It is, because
we must also prove the array is non-empty before concluding that the
expression is strict. This could result in misclassifying an
expression as strict when it is not, leading to assorted planning
mistakes such as inlining a SQL function that shouldn't be inlined.
We can almost fix this by just re-adding the previous handling of
ScalarArrayOpExpr in that function, but doing only that would lead to
also calling check_functions_in_node() and thus redundantly checking
the operator's strictness. Avoid that by turning the if-series into
an else-if chain, as it arguably should have been all along.
The reason these errors have escaped detection for decades is that
they are exposed only in arcane corner cases. ScalarArrayOpExpr with
an empty array isn't typical usage, and even when that's possible
several other conditions apply before the planner can reach a mistaken
conclusion. While it's possible to build test cases demonstrating
these mistakes, I (tgl) judged them too indirect and special-purpose
to justify consuming regression test cycles forevermore.
Author: Ayush Tiwari <ayushtiwari.slg01@gmail.com>
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/CAJTYsWV3vqRJmST-gv1NsXEef-zOnjVJpYS910aBaiuMij4nFg@mail.gmail.com
Discussion: https://postgr.es/m/CAJTYsWWcLGmz0f8_QPP_Liq-fc7-geiFSCdqoq3XGeRHPPsWeA@mail.gmail.com
Backpatch-through: 14
A two-phase transaction that is assigned an XID but produces no change
to be decoded -- for example, one that only acquires row locks via
SELECT ... FOR SHARE -- has no base snapshot in the reorder
buffer. ReorderBufferReplay() already skips such a transaction at
PREPARE time and never invokes the begin_prepare/change/prepare
callbacks for it, but ReorderBufferFinishPrepared() still called the
commit_prepared (or rollback_prepared) callback. As a result a
spurious COMMIT/ROLLBACK PREPARED was sent to the output plugin with
no preceding PREPARE. For the built-in subscriber this breaks
replication (the apply worker fails to find the prepared transaction),
and test_decoding could even crash.
Fix this by detecting an empty transaction (base_snapshot == NULL) in
ReorderBufferFinishPrepared() and cleaning it up without invoking the
commit/rollback prepared callbacks, mirroring the existing empty
transaction handling in ReorderBufferReplay().
On v18 and newer versions, commit 072ee847ad changed
ReorderBufferPrepare() to send the prepare whenever it had not already
been sent, which also fires for empty transactions and emits a
spurious PREPARE. On those branches ReorderBufferPrepare() is
therefore additionally guarded with base_snapshot != NULL. This guard
and the Assert(!rbtxn_sent_prepare()) added in
ReorderBufferFinishPrepared(), are not necessary on v17 and older
versions: there ReorderBufferPrepare() only sends a prepare for
concurrently-aborted transactions (which never applies to an empty
transaction) and the RBTXN_SENT_PREPARE flag does not exist.
Back-patch to v14, where decoding of two-phase transactions was
introduced.
Bug: #19556
Reported-by: Alexander Kozhemyakin <a.kozhemyakin@postgrespro.ru>
Reviewed-by: Amit Kapila <amit.kapila16@gmail.com>
Discussion: https://postgr.es/m/19556-daa6d7ea65054d48@postgresql.org
Backpatch-through: 14
pg_get_publication_tables() collects the OIDs of the published tables
on its first call, without locking them, and then reopens each table
later, once per result row, to compute its column list and fetch its
row filter. The reopen used table_open(), which errors out with "could
not open relation with OID" if the table has been dropped in the
meantime. This could happen for any published table without an
explicit column list, which is every table in FOR ALL TABLES and FOR
TABLES IN SCHEMA publications, but also FOR TABLE entries without a
column list. The failure is common in environments where many tables
are created and dropped while publication tables are being queried,
e.g. by table synchronization on a subscriber.
Fix by opening every table with try_table_open(), which returns NULL
if the relation no longer exists, and skipping the table in that
case. Concurrently dropped tables are thus simply absent from the
result set, which is the expected point-in-time behavior.
As a side effect, tables with an explicit column list, which were
previously returned without being opened, are now also locked with
AccessShareLock, so the function can block behind concurrent DDL on
such tables where it previously did not.
Backpatch to v16, where we added the table_open() call in
pg_get_publication_tables().
Author: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Reviewed-by: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Reviewed-by: shveta malik <shveta.malik@gmail.com>
Reviewed-by: Ajin Cherian <itsajin@gmail.com>
Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
Reviewed-by: Chao Li <li.evan.chao@gmail.com>
Discussion: https://www.postgresql.org/message-id/CALj2ACVYYooWH-5tJ6cPKkU%2BmutVxwb_z4S%2BqAi-zdrFqxXE2Q%40mail.gmail.com
Backpatch-through: 16
Commit fd83c83d09 turned the recursive posting-tree cleanup in
ginVacuumPostingTreeLeaves() into an iterative sweep that follows the
tree's leaf pages via their rightlinks. The recursive version called
vacuum_delay_point() while processing the tree, but that call was removed
and never re-added to the new loop. As that commit only set out to fix a
deadlock, the removal appears to have been unintentional.
Consequently the leaf-page sweep of a single posting tree runs with no
vacuum_delay_point(), and therefore no CHECK_FOR_INTERRUPTS(). A posting
tree stores all the TIDs for one indexed key, so for a frequently
occurring key it can span a large number of leaf pages. While such a
tree is being vacuumed the operation ignores vacuum_cost_delay and does
not respond to query cancellation or statement_timeout; an autovacuum
worker likewise cannot be interrupted mid-sweep when another backend
requests a conflicting lock.
Restore the call, placed after the current page has been unlocked and
released so that no buffer content lock is held across a potential delay
(cf. 21c27af65f). The sibling loops in ginbulkdelete() and
ginvacuumcleanup() already call vacuum_delay_point() once per page.
Author: Paul Kim <mok03127@gmail.com>
Co-authored-by: Alexander Korotkov <aekorotkov@gmail.com>
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Reviewed-by: Andrey Borodin <x4mmm@yandex-team.ru>
Reviewed-by: solai v <solai.cdac@gmail.com>
Discussion: https://postgr.es/m/178447127453.110.12276981925360691905%40mail.gmail.com
Backpatch-through: 14
index_create_copy is used to create copy definitions of existing indexes.
Currently, it passes 0 as constr_flags to index_create(), which results
in the copied index to always be created as immediate (indimmediate set
to true). For deferrable unique constraints, it means that the
transient index used during the phase 2 of REINDEX CONCURRENTLY forces
immediate constraint checks on concurrent inserts, which can cause
unexpected constraint violations based on the definition of the parent
table, inconsistently set in the copied index.
To fix this without violating the contract of constr_flags (which should
only be used when creating constraints) and without relaxing the strict
assertion in index_create(), this introduces a new index creation flag:
INDEX_CREATE_DEFERRABLE. If set, a copied index's indimmediate is set
to false, meaning that unique constraints are not enforced immediately
on insertion, but at transaction commit time.
An isolation test for REINDEX CONCURRENTLY is added, based on an
injection point waiting after phase 1 of the operation, where an index
copy has been built and is able to accept DMLs for its validation in
phase 2. The test is tentatively backpatched down to v17.
INJECTION_POINT() is outside a transaction context, which should be fine
on HEAD since 8daeaa9b64 but I suspect may cause issues in v19 and
older branches due to the wait facility depending on condition variables
and a DSM setup, but let's see what the buildfarm tells.
Author: Nitin Motiani <nitinmotiani@google.com>
Discussion: https://postgr.es/m/CAH5HC97JmjPpgiQOqW9xm8qXhNiu7zZ1Qh+FfhEESJuDv69kuQ@mail.gmail.com
Backpatch-through: 14
Commit f9b7fc65 fixed a race when predicate-locking completely empty
btrees: without a buffer lock held, a matching key could be inserted
between _bt_search and the PredicateLockRelation call, so the scan would
miss concurrently inserted tuples while the writer wouldn't see the
reader's predicate lock. That commit only fixed _bt_first's _bt_search
path, though. Scans without useful insertion scan keys return early
from _bt_first via _bt_endpoint, which still didn't recheck if the
relation was empty.
To fix, add handling to _bt_endpoint that is analogous to the handling
added to _bt_search by commit f9b7fc65.
Author: Peter Geoghegan <pg@bowt.ie>
Discussion: https://postgr.es/m/CAH2-WzkNoTn3yXY0iGkSuavJ+sL8EROf+kitW+_2v2tJVWuKmA@mail.gmail.com
Backpatch-through: 14
pg_database_size() allows access to users who have either CONNECT
privilege on the target database or privileges of the pg_read_all_stats
role. However, previously, psql's \l+ checked only for CONNECT,
so users with privileges of pg_read_all_stats still saw "No Access" for
databases they could not connect to.
Fix this by making \l+ also check
pg_has_role('pg_read_all_stats', 'USAGE'), matching
pg_database_size()'s permission rules.
For back branches, emit the pg_read_all_stats check only when
connected to PostgreSQL 10 or later, since earlier releases do not have
that predefined role.
Backpatch to all supported versions.
Author: Christoph Berg <myon@debian.org>
Reviewed-by: Álvaro Herrera <alvherre@kurilemu.de>
Reviewed-by: Fujii Masao <masao.fujii@gmail.com>
Discussion: https://postgr.es/m/amCo6qRmnfPVk4-V@msg.df7cb.de
Backpatch-through: 14
Commit 1df9e8d96 made remove_useless_result_rtes() compute the set of
baserels in the jointree, to pass down to the find_dependent_phvs()
checks. But those checks are no-ops when the query contains no PHVs,
since find_dependent_phvs() and find_dependent_phvs_in_jointree() both
return early in that case. So we can avoid the
get_relids_in_jointree() scan altogether when root->glob->lastPHId is
zero, leaving baserels as NULL.
Author: Richard Guo <guofenglinux@gmail.com>
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/CAMbWs49H275KzgZr3Cd1Hy+6Lmwp35bZ+5PrVc62k3HDLj6hNQ@mail.gmail.com
Backpatch-through: 16
find_dependent_phvs() and find_dependent_phvs_in_jointree() decide
whether a PlaceHolderVar depends on the RTE_RESULT rel we're
considering removing by comparing the PHV's phrels to a singleton set
containing that rel's RT index, reasoning that if phrels contains any
other relid bits then those define an appropriate place where we can
evaluate the PHV. But since this code was originally written, we've
redefined phrels to include outer-join relids, and that breaks this
logic, potentially allowing us to remove an RTE_RESULT that leaves no
valid place to evaluate the PHV. The planner doesn't throw an error
when that happens, but it does produce an incorrect plan that will not
replace the PHV's value with NULL when needed.
In the known test case for this bug, the "extra" OJ relid is one that
we've actually decided to remove but haven't yet cleaned out of the
query's PHVs. It's not entirely clear though that that would always
be the case. Let's restore this code to the way it was designed to
work, by considering only base relids within the PHV's phrels.
Bug: #19553
Reported-by: Viktor Leis <leis@in.tum.de>
Author: Matheus Alcantara <matheusssilv97@gmail.com>
Co-authored-by: Richard Guo <guofenglinux@gmail.com>
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/19553-4561747f93f368a7@postgresql.org
Backpatch-through: 16
An index-only scan filled its result slot from the HeapTuple an index AM
returns in scan->xs_hitup by deforming it with the virtual slot's own
tuple descriptor (during GiST and SP-GiST index-only scans). But index
AMs form that heap tuple using their own descriptor, scan->xs_hitupdesc.
The AM's descriptor may disagree with the IoS virtual slot's descriptor
about each column's precise alignment, leading to "can't happen" errors
in certain rare edge cases. Hard crashes were possible but much less
likely.
To fix, deform the tuple with the descriptor it was formed with. This
is simpler, and makes xs_hitup handling (used by GiST and SP-GiST)
uniform with the nearby existing xs_itup handling (used by nbtree).
In practice this issue was very unlikely to be hit (it was found during
testing of a patch that will change the table AM API used during index
scans). The only currently affected core opclass is GiST's range_ops.
It was only possible for the datum to be accessed at an incorrectly
aligned offset when reading the second or subsequent column from a
multicolumn GiST index. This couldn't happen in the common case where
the datum used an unaligned short varlena header. Moreover, an earlier
column had to leave the range datum at an offset where the two
alignments actually disagree (e.g., an odd-length varlena datum).
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-by: Tomas Vondra <tomas@vondra.me>
Discussion: https://postgr.es/m/CAH2-WzkGXa2SKnebdW29RT1hCcQBo_p03v3iqif2u9bjzLB-aQ@mail.gmail.com
Backpatch-through: 14
Commit c7aeb775 rewrote the HOT-chain offset sanity checks in three
places, but in heap_get_root_tuples it accidentally tested offnum -- the
outer loop variable, which is already bounded by the loop condition --
instead of nextoffnum, the offset actually passed to PageGetItemId. The
pre-c7aeb775 check tested nextoffnum.
With the check ineffective, a stale t_ctid could make PageGetItemId read
past the end of the line pointer array (which is data corruption that we
expect to be able to catch here).
Author: Peter Geoghegan <pg@bowt.ie>
Reported-by: Konstantin Knizhnik <knizhnik@garret.ru>
Discussion: https://postgr.es/m/87c7d8a4-3a82-4334-bee6-e8c2ad3f3293@garret.ru
Backpatch-through: 15
Previously, libpqrcv_create_slot() checked only that
CREATE_REPLICATION_SLOT returned PGRES_TUPLES_OK before reading
values from the first row. If the server unexpectedly returned an
invalid result, such as zero rows, PQgetvalue() could return NULL,
leading to a crash while parsing the LSN.
Other replication commands, such as IDENTIFY_SYSTEM, already validate
the response shape before accessing result values, but
CREATE_REPLICATION_SLOT did not.
Fix this by verifying that CREATE_REPLICATION_SLOT response contains
exactly one row with four fields, and report a protocol violation otherwise.
Backpatch to all supported versions.
Bug: #19547
Reported-by: Yuelin Wang <1217816127@qq.com>
Author: Kenny Chen <kennychen851228@gmail.com>
Reviewed-by: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Reviewed-by: Fujii Masao <masao.fujii@gmail.com>
Discussion: https://postgr.es/m/19547-f7986f668f71e788@postgresql.org
Discussion: https://postgr.es/m/CAPXstDtW2iqe+DJAOTQTX+rRziJp2UhZSo1+HRj1COAtbu+nKw@mail.gmail.com
Backpatch-through: 14
When bulk-extending a relation, we add the newly-added blocks that we
won't immediately use to the free space map and then call
FreeSpaceMapVacuumRange() to propagate that free space up the FSM tree,
so other backends can find and reuse it.
However, the end block argument to FreeSpaceMapVacuumRange() is
exclusive, and we passed the number of the last added block (since
00d1e02be2). If that block was the first one covered by a new FSM page,
its free space wasn't propagated up the tree and was therefore invisible
to FSM searches until the next FSM vacuum.
Fix by passing the block number one past the last added block, so the
full range is vacuumed.
Author: Jingtang Zhang <mrdrivingduck@gmail.com>
Reviewed-by: Melanie Plageman <melanieplageman@gmail.com>
Discussion: https://postgr.es/m/flat/CAPsk3_Bx_vdybN%3D-DZu8HLStf%2BXnuFUBkLwxouONSMkWuO9oug%40mail.gmail.com
Backpatch-through: 16
equalPolicy() is used in the relation cache to check if two policy
definitions are equivalent, but missed to check for polpermissive.
ALTER POLICY cannot switch a policy to be PERMISSIVE or RESTRICTIVE, so
this would need a dropped and then re-created policy, which would
trigger a relcache invalidation. Anyway, there is no harm in being
consistent in the check, and if one decides to add an ALTER POLICY to
switch PERMISSIVE or RESTRICTIVE, we would be silently in trouble.
Author: Andreas Lind <andreaslindpetersen@gmail.com>
Reviewed-by: Laurenz Albe <laurenz.albe@cybertec.at>
Discussion: https://postgr.es/m/CAMxA3rv1CS6R7JR5ojz-3CmCEnZEFrqu+XXTnGbLRWrjJRH7sA@mail.gmail.com
Backpatch-through: 14
The pg_attribute_always_inline macro name is so long it forces pgindent
to format the code in strange ways. Which may incentivize patch authors
to either structure the code in strange ways (e.g. reorder prototypes),
use shorter names, etc. Neither is very desirable for code readability.
This shortens the name by removing the _attribute_ part. It also makes
it more consistent with pg_noinline, which does not have the _attribute_
part either.
Backpatched to all supported branches, to prevent conflicts when
backpatching other fixes. The backbranches however keep both the old and
new macro name, so that existing code keeps working.
Author: Andres Freund <andres@anarazel.de>
Reviewed-by: Peter Geoghegan <pg@bowt.ie>
Reviewed-by: Tomas Vondra <tomas@vondra.me>
Discussion: https://postgr.es/m/bqqdehahpoa36igpictuqyn2s2mexk3t3ehidh2ffd2slb35e5@rzgksuiszgbg
Backpatch-through: 14
Check explicitly for pqsecure_read() returning an error. It shouldn't
fail, and we would've caught it in the check for a short read, but
better to be explicit so that the error message is more informative.
We also shouldn't update 'inEnd' when the read fails, although that
too is just pro forma as we will bail out and close the connection on
error.
Reported-by: Peter Eisentraut <peter@eisentraut.org>
Discussion: https://www.postgresql.org/message-id/34844e8c-267c-4daf-b1e0-f26059a4a7d3@eisentraut.org
Backpatch-through: 14
If an expression containing an aggregate is evaluated above the plan
node that computes the aggregate, as happens with window functions or
with expressions postponed to above the final sort, setrefs.c replaces
the Aggref or WindowFunc with a Var referencing the lower node's
output. For SQL/JSON aggregates such as JSON_ARRAYAGG and
JSON_OBJECTAGG, deparsing the containing JsonConstructorExpr then
failed with "invalid JsonConstructorExpr underlying node type", since
get_json_agg_constructor() did not expect a Var there.
Fix by resolving the Var back to the underlying Aggref or WindowFunc
and deparsing the constructor as if the aggregate were computed at the
current node. The JsonConstructorExpr retains the RETURNING clause
and the ABSENT/NULL ON NULL and WITH UNIQUE options, and the arguments
come from the resolved aggregate, so the original JSON aggregate
syntax is reproduced in full. This mirrors how get_agg_expr() already
looks through such a Var when deparsing a combining aggregate.
Reported-by: Thom Brown <thom@linux.com>
Author: Richard Guo <guofenglinux@gmail.com>
Discussion: https://postgr.es/m/CAA-aLv5QYTaMOk=Qhv6cgwceeHETZV8YJvWZ_rH+yVZCuchATA@mail.gmail.com
Backpatch-through: 16
The previous commit strengthened a workaround for a hang when large
messages are split across TLS records/GSS tokens. Because that
workaround is implemented in libpq internals, it can only help us when
libpq itself is polling on the socket. In nonblocking situations,
where the client above libpq is expected to poll, the same bugs can
show up.
As a contrived example, consider a large protocol-2.0 error coming
back from a server during PQconnectPoll(), split in an odd way across
two records:
-- TLS record (8192-byte payload) --
EEEE[...repeated a total of 8192 times]
-- TLS record (8193-byte payload) --
EEEE[...repeated a total of 8192 times]\0
The first record will fill the first half of the libpq receive buffer,
which is 16k long by default. The second record completely fills the
last half with its first 8192 bytes, leaving the terminating NULL in
the OpenSSL buffer. Since we still haven't seen the terminator at our
level, PQconnectPoll() will return PGRES_POLLING_READING, expecting to
come back when the server has sent "the rest" of the data. But there
is nothing left to read from the socket; OpenSSL had to pull all of
the data in the 8193-byte record off of the wire to decrypt it.
A real server would probably not split up the records this way, nor
keep the connection open after sending a fatal connection error. But
servers that regularly use larger TLS records can get the libpq
receive buffer into the same state if DataRows are big enough, as
reported on the list. While the PostgreSQL server doesn't use larger
TLS records like that, other non-PostgreSQL servers that implement the
wire protocol are known to do that, as well as proxies that sit
between the server and the client
This is a layering violation. libpq makes decisions based on data in
the application buffer, above the transport buffer (whether SSL or
GSS), but clients are polling the socket below the transport buffer.
One way to fix this in a backportable way, without changing APIs too
much, is to ensure data never stays in the transport buffer. Then
pqReadData's postconditions will look similar for both raw sockets and
SSL/GSS: any available data is either in the application buffer, or
still on the socket.
Building on the prior commit, make pqReadData() to drain all pending
data from the transport layer into conn->inBuffer, expanding the
buffer as necessary. This is not particularly efficient from an
architectural perspective (the pqsecure_read() implementations take
care to fit their packets into the current buffer, and that effort is
now completely discarded), but it's hopefully easier to reason about
than a full rewrite would be for the back branches.
Author: Jacob Champion <jacob.champion@enterprisedb.com>
Reviewed-by: Mark Dilger <mark.dilger@enterprisedb.com>
Reviewed-by: solai v <solai.cdac@gmail.com>
Reported-by: Lars Kanis <lars@greiz-reinsdorf.de>
Discussion: https://postgr.es/m/2039ac58-d3e0-434b-ac1a-2a987f3b4cb1%40greiz-reinsdorf.de
Backpatch-through: 14
An extra check for pending bytes in the SSL layer has been part of
pqReadReady() for a very long time (79ff2e96d). But when GSS transport
encryption was added, it didn't receive the same treatment. (As
79ff2e96d notes, "The bug that I fixed in this patch is exceptionally
hard to reproduce reliably.")
Without that check, it's possible to hit a hang in gssencmode, if the
server splits a large libpq message such that the final message in a
streamed response is part of the same wrapped token as the split
message:
DataRowDataRowDataRowDataRowDataRowData
-- token boundary --
RowDataRowCommandCompleteReadyForQuery
If the split message takes up enough memory to nearly fill libpq's
receive buffer, libpq may return from pqReadData() before the later
messages are pulled out of the PqGSSRecvBuffer. Without additional
socket activity from the server, pqReadReady() (via pqSocketCheck())
will never again return true, hanging the connection.
Pull the pending-bytes check into the pqsecure API layer, where both
SSL and GSS now implement it.
Note that this does not fix the root problem! Third party clients of
libpq have no way to call pqsecure_read_is_pending() in their own
polling. This just brings the GSS implementation up to par with the
existing SSL workaround; a broader fix is left to a subsequent commit.
In preparation for the broader fix, this patch already changes the
*_read_pending() functions to return the number of bytes in the buffer
rather than just a boolean. The current callers don't need that, but
the subsequent fix will.
Author: Jacob Champion <jacob.champion@enterprisedb.com>
Discussion: https://postgr.es/m/CAOYmi%2BmpymrgZ76Jre2dx_PwRniS9YZojwH0rZnTuiGHCsj0rA%40mail.gmail.com
Backpatch-through: 14
ParallelBackupStart() stored _beginthreadex()'s return value as the
worker's thread handle without checking it. On failure that value is 0,
which would later reach WaitForMultipleObjects() as a null handle, caught
only by an Assert. The fork() path already calls pg_fatal() when it
fails; do the same for _beginthreadex(), as pgbench does.
Author: Bryan Green <dbryan.green@gmail.com>
Discussion: https://www.postgresql.org/message-id/8c712d76-ecf7-4749-a6d8-dddc01f298ec@gmail.com
Backpatch-through: 14
9d9c02ccd added code to allow the executor to stop early when processing
WindowAgg nodes where a monotonic window function starts producing
values that result in a pushed-down qual no longer matching, and will
never match again due to the window function's monotonic properties.
That commit requires a SupportRequestWFuncMonotonic to exist on the
window function and for it to detect when the function is monotonic. For
COUNT(ANY) and COUNT(*), the support function failed to consider some
cases where the WindowClause used EXCLUDE to exclude certain rows from
being aggregated. Some WindowClause definitions mean we aggregate rows
that come after the current row, and when processing those rows later,
if we EXCLUDE certain rows, the monotonic property can be broken.
Wrongly treating the COUNT(*) or COUNT(ANY) aggregate as monotonic could
lead to rows being filtered that should not be filtered from the result
set.
Another issue was that the support function for the COUNT aggregate
mistakenly thought that a WindowClause without an ORDER BY meant that
the results would be both monotonically increasing and decreasing, but
that's only true when in RANGE mode, where all rows are peers.
It is possible to support various cases that do have an EXCLUDE clause,
but getting the logic correct for the exact set of cases that are valid
is quite complex and would likely better be left for a future project.
Here, we mostly disable run condition pushdown when there is an EXCLUDE
clause unless the clause is for EXCLUDE CURRENT ROW, uses COUNT(*)
(rather than COUNT(ANY)), and the window aggregate has no FILTER clause.
Bug: #19533
Reported-by: Qifan Liu <imchifan@163.com>
Author: Chengpeng Yan <chengpeng_yan@outlook.com>
Author: David Rowley <dgrowleyml@gmail.com>
Reviewed-by: Richard Guo <guofenglinux@gmail.com>
Reviewed-by: John Naylor <johncnaylorls@gmail.com>
Discussion: https://postgr.es/m/19533-413a1014e5d0e766@postgresql.org
Backpatch-through: 15
Commit e7564ee8cd, which fixed base backup progress reporting on
backup failure, removed the external function basebackup_progress_done()
because it was no longer used in core. When that change was backpatched
to v15, it introduced an ABI break, which was reported by buildfarm
member crake.
This commit restores basebackup_progress_done() to preserve ABI
compatibility, even though it is no longer used in core, rather than
updating the .abi-compliance-history file. Because external backup
tools may still call this function.
Per buildfarm member crake.
Reported-by: Andrew Dunstan <andrew@dunslane.net>
Discussion: https://postgr.es/m/CAD5tBcJ+ktrEp=PT8Gq-f=8mA2cDtZMB-hDMV4mMJ+9V46qBeQ@mail.gmail.com
Backpatch-through: 15-18
ON SELECT rules must be named "_RETURN", while other kinds of rules
must not be; this ancient restriction is depended on by various client
code. We successfully enforced this convention in most places, but
ALTER RULE allowed renaming a non-SELECT rule to "_RETURN". Notably,
that would break dump/restore, since the eventual CREATE RULE command
would reject the name.
While at it, remove DefineQueryRewrite's hack to substitute "_RETURN"
for the convention that was used before 7.3. We dropped other
server-side code that supported restoring pre-7.3 dumps some time ago
(notably in e58a59975 and nearby commits), but this bit was missed.
Bug: #19543
Reported-by: Adam Pickering <adamkpickering@gmail.com>
Author: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/19543-461228e77f3b32fc@postgresql.org
Backpatch-through: 14
Commit ce146621 ensures that ERROR is raised if a session tries to read
pages of another session's temp table. But there is a corner case where
the other session's temp table is empty -- in this case the INSERT
command bypasses our checks and executes without any errors.
Such behavior is inconsistent and erroneous: it leaves an invalid buffer
in the temp buffers pool. Since the buffer was created for another
session's temp table, we get an error "no such file or directory" when
trying to flush it.
This commit fixes it by adding a RELATION_IS_OTHER_TEMP check in the
relation-extension path.
Backpatch to 16, because it is the first release after 31966b151e, which
introduced a separate local relation extension function
ExtendBufferedRelLocal(), which lacks of RELATION_IS_OTHER_TEMP() check.
As this fix introduces more checks to 013_temp_obj_multisession.pl, backpatch
the whole test script to 16.
Discussion: https://postgr.es/m/CAJDiXgiX2XZBHDNo%2BzBbvku%2BtchrUurvPRaN1_40mEQ1_sG90g%40mail.gmail.com
Author: Daniil Davydov <3danissimo@gmail.com>
Reviewed-by: Jim Jones <jim.jones@uni-muenster.de>
Reviewed-by: Imran Zaheer <imran.zhir@gmail.com>
Reviewed-by: ZizhuanLiu X-MAN <44973863@qq.com>
Backpatch-through: 16
Previously, MultiXactId wraparound hints suggested dropping stale
replication slots. While that advice is appropriate for transaction ID
wraparound, where replication slots can hold back XID horizons,
it was misleading for MultiXactId wraparound. Following it could lead
users to drop replication slots unnecessarily without helping resolve
the MultiXactId wraparound condition.
MultiXact cleanup is not directly delayed by replication slots.
Instead, it depends on whether old MultiXactIds can still be seen
as live by running transactions.
This commit removes the replication slot advice from MultiXactId
wraparound hints, and documents that stale replication slots are
normally not relevant to resolving MultiXactId wraparound problems.
Backpatch to all supported branches.
BUG #18876
Reported-by: Haruka Takatsuka <harukat@sraoss.co.jp>
Author: Fujii Masao <masao.fujii@gmail.com>
Discussion: https://postgr.es/m/18876-0d0b53bad5a1f4c1@postgresql.org
Backpatch-through: 14
Previously, if a base backup failed after it had started streaming
files, pg_stat_progress_basebackup could continue to show a stale
progress entry even though the backup was no longer running. This could
be observed when the client kept the replication connection open after
the error. It is normally not observable when using pg_basebackup,
because the client disconnects after the error.
The problem was that progress reporting was cleared only after
successful completion.
This commit moves the progress reporting cleanup into the progress
sink's cleanup callback so that it is cleared after both successful
and failed backups.
Backpatch to v15. v14 has the same issue, but the fix does not apply
cleanly because it lacks the base backup sink infrastructure. Since
the bug does not affect the backup itself and is normally not
observable when using pg_basebackup, skip the v14 backpatch.
Author: Chao Li <lic@highgo.com>
Reviewed-by: Fujii Masao <masao.fujii@gmail.com>
Discussion: https://postgr.es/m/EA1A6CD2-EFA6-462B-9A02-03003555AB4A@gmail.com
Backpatch-through: 15
