Commit e28cf96e9 preprocessed the copies of a query level's
PlaceHolderVars that were pushed down into its subqueries in a single
walk at the start of subquery_planner. That was wrong in two ways.
First, such copies can also be inserted later, when a join alias Var
of the outer level is expanded within a LATERAL subquery or a
SubLink's subselect and the alias expression contains a PHV. Those
copies were never preprocessed, so a SubLink within them survived into
the subquery's lateral references or into a SubPlan's argument list,
tripping an assertion in identify_current_nestloop_params or failing
in cost_qual_eval.
Second, when a pushed-down copy directly contains another PHV of the
same level, the walk preprocessed the inner one first and then the
outer one, so the outer one's preprocessing ran into the SubPlans just
created in the inner one and tripped an assertion in
flatten_join_alias_vars.
To fix, preprocess the copies right after join alias expansion
instead: within preprocess_expression, just before SubLinks are turned
into SubPlans, and for LATERAL subquery RTEs, right after their
aliases are flattened. Preprocessing a copy handles everything within
it, including any further copies nested inside its SubLinks, so the
walker no longer looks inside a copy.
Back-patch to v16, as with commit e28cf96e9.
Reported-by: Fujii Masao <masao.fujii@gmail.com>
Author: Richard Guo <guofenglinux@gmail.com>
Discussion: https://postgr.es/m/CAHGQGwH-acRnAk1=Ki4EiAJEt67+Rz=_jyg7pbP_c6BK4oHZ4Q@mail.gmail.com
Backpatch-through: 16
The PASSING arguments of JSON_EXISTS(), JSON_QUERY(), JSON_VALUE() and
JSON_TABLE() are evaluated by plain expression steps and given to the
jsonpath executor as-is.
When dealing with a varlena, JsonItemFromDatum() was building a
jbvString directly the contents of a varlena. For a varatt_external
(varatt_external[oid/oid8] on HEAD), this ignored that the contents of
the varlena need to be detoasted, leading to incorrect contents
returned.
Let's detoast the value, like the jsonb and json cases. b0feda79fd
has fixed the same issue for subscript values in jsonb.
Note that the results of DatumGetTextPP() are allocated in the memory
context used for the evaluation of the expression, which is reset once
per row.
Oversight in 6185c9737c.
Author: Chaitanya Choudhary <chaitanyyachoudhary@gmail.com>
Author: Shihao Zhong <zhong950419@gmail.com>
Discussion: https://postgr.es/m/19693-2ecd2b52c838b3e5@postgresql.org
Backpatch-through: 17
The RI fast path checks a foreign key by probing the referenced
unique index directly, using the equality operator recorded for
the constraint. Whether the fast path can be used is decided once
and cached in RI_ConstraintInfo, and that cache is invalidated on
pg_constraint changes but not on pg_amop. So after an ALTER
OPERATOR FAMILY drops the recorded operator and adds another in its
place, the cached decision is stale, and the next fast-path check
probes the index with an operator no longer in the opfamily and
errors out with "operator XXX is not a member of opfamily XXX".
The SPI path is unaffected, because the planner just stops matching
the index.
To fix, register an AMOPOPID syscache callback to flush the RI
cache on pg_amop changes, and have ri_check_fastpath_index()
recheck that the recorded operator is still the equality member of
the index's opfamily, falling back to SPI when it is not.
Add regression test coverage.
Reported-by: Nikolay Samokhvalov <nik@postgres.ai>
Author: Nikolay Samokhvalov <nik@postgres.ai>
Discussion: https://www.postgr.es/m/CAM527d9PzFzagr67N0%3DEx2ng1p5HzrcAszy3j5OoZKHXQMARXA%40mail.gmail.com
Backpatch-through: 19
The fast path required table-level SELECT on the referenced table,
rejecting checks that the SPI path allows with column-level grants.
It also omitted the UPDATE privilege required by FOR KEY SHARE.
When table privileges do not suffice, use ExecCheckOneRelPerms() with
the referenced key columns as selectedCols and an empty updatedCols.
This accepts SELECT on all referenced columns and UPDATE on any column,
the same privileges the executor would require for the SELECT ... FOR
KEY SHARE the SPI path runs. Keep the table-privilege check as a
shortcut that avoids constructing a column bitmap in the usual case.
Add missing regression test coverage for the fixed cases.
Reported-by: Nikolay Samokhvalov <nik@postgres.ai>
Author: Nikolay Samokhvalov <nik@postgres.ai>
Co-authored-by: Amit Langote <amitlangote09@gmail.com>
Discussion: https://www.postgr.es/m/CAM527d9BgPjeOOYmbCBTd57R145qHCk-dzw9qNq%2BnOrDq1j__A%40mail.gmail.com
Backpatch-through: 19
Now that pg_upgrade will rewrite multixid members, starting from
oldestMulti, it's important that oldestMulti is valid. Add a sanity
check that oldestMulti is not newer than the oldest datminmxid value
in pg_database.
One case where this could happen is if the cluster was previously
upgraded to version 9.3 with a buggy pg_upgrade version that didn't
have commit a61daa14d5. This new pg_upgrade check is similar to the
defence that was added in commit 78db307bb2 to VACUUM to avoid
truncating away multixids if oldestMulti is too new. This pg_upgrade
check differs in that we don't try to soldier on with the upgrade if
the oldestMultiXID is inconsistent, but rather just abort the upgrade.
Reported-by: Noah Misch <noah@leadboat.com>
Discussion: https://www.postgresql.org/message-id/20260827231757.78.noahmisch@microsoft.com
Backpatch-through: 19
pgoutput records the toplevel XID in the relation's streamed_txns
list, allocated in CacheMemoryContext, when it sends a relation's
schema during a streamed transaction. We clean up that list at stream
commit and stream abort, but missed stream prepare, so the XIDs stay
around until a relcache invalidation for the relation or until the
walsender exits.
Fix by cleaning up the cache after writing the stream prepare message,
marking the schema as sent as we do at stream commit. The subscriber
applies the spooled relation messages at PREPARE time and its relation
map isn't rolled back by ROLLBACK PREPARED, so we don't need to send
the schema again even if the prepared transaction eventually aborts.
Oversight in commit 63cf61cdeb.
Author: Ayush Tiwari <ayushtiwari.slg01@gmail.com>
Reviewed-by: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
Discussion: https://postgr.es/m/CAJTYsWWt5aaKHCnV=jNOwABzspd-WbnhWLndDqtESDQRQ-BkKQ@mail.gmail.com
Backpatch-through: 15
We unintentionally copied the original query's limitOption field while
building the modified subquery. If it was LIMIT_OPTION_WITH_TIES,
that behavior would propagate through to execution, potentially
causing a "more than one row returned by a subquery used as an
expression" error. Fix by resetting to the default LIMIT_OPTION_COUNT.
This bug dates back to the introduction of WITH TIES.
Bug: #19694
Reported-by: Chaitanya Choudhary <chaitanyyachoudhary@gmail.com>
Author: Chaitanya Choudhary <chaitanyyachoudhary@gmail.com>
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/19694-4e61bd0475398eef@postgresql.org
Backpatch-through: 14
Commits 2bc7e886fc and 90eae926ab taught ON CONFLICT to include
indexes matching an already selected arbiter, so that an index left
behind by REINDEX CONCURRENTLY continues to arbitrate together with its
replacement. Both checks were too permissive:
a) infer_arbiter_indexes() compared a candidate with a named
constraint's index using only attributes, expressions and predicate, but
ignored collation, NULLS NOT DISTINCT setting or deferrability. As a
result, an index with a difference in these settings could be accepted
even though it did not identify the same conflicts. Also, a deferrable
index that otherwise matches an arbiter index would also cause ON
CONFLICT to fail with "ON CONFLICT does not support deferrable unique
constraints/exclusion constraints as arbiters".
b) ExecInitPartitionInfo() also failed due to the failure to compare
deferrability, so partition-local deferrable indexes would be considered
and break inserts routed to that partition with the error mentioned
above.
Fix by making IsIndexCompatibleAsArbiter() compare those properties, and
changing infer_arbiter_indexes() to use that routine instead of open
coding equivalent logic. Also handle the named-constraint case in
infer_arbiter_indexes() separately instead of passing the constraint
index through the regular clause-inference matching. That function is
not static anymore, so move it to index.c, and also reimplement it to
use the Relation from the indexes only, no longer receiving the
IndexInfo (which wasn't really necessary); also, use equal() to compare
the expression and predicate lists rather than list_difference().
Add tests for ON CONFLICT ON CONSTRAINT with deferrable, NULLS NOT
DISTINCT, and different-collation sibling indexes, preserving the
behavior of released pre-19 versions. Also test routed inserts with a
partition-local deferrable unique constraint.
Author: Zsolt Parragi <zsolt.parragi@percona.com>
Author: Mihail Nikalayeu <mihailnikalayeu@gmail.com>
Reported-by: Zsolt Parragi <zsolt.parragi@percona.com>
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Backpatch-through: 19
Discussion: https://postgr.es/m/CAN4CZFPEYXeYFTxHpoPujfVFb+1Tx1jnXVboDMBg-ZhpgpQ-_g@mail.gmail.com
SHMEM_ATTACH_UNKNOWN_SIZE can be passed as argument to
ShmemRequestStruct() when the caller wants to attach to an existing
shared memory structure whose size it doesn't know. If the shared
memory structure doesn't exist, the request should fail, but instead,
ProcessShmemRequestsAfterStartup() tried to create the structure with
size = -1. That led to integer overflow in ShmemAllocRaw() and memory
corruption.
Fix by rejecting requests with SHMEM_ATTACH_UNKNOWN_SIZE when the
structure doesn't exist. Also add an integer overflow check in
ShmemAllocRaw(), to protect from this kind of confusion or simply too
large requests. Also document SHMEM_ATTACH_UNKNOWN_SIZE.
Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>
Discussion: https://www.postgresql.org/message-id/CAExHW5u_fTsOAS85kG981Vu6eR1GV-344rup6zYew7xMjEDREw@mail.gmail.com
Backpatch-through: 19
The query to get attribute statistics from a remote server applied the
"C" collation to a column of type name without checking the server's
version. Since the name type was made collatable in Postgres 12, this
could break attribute statistics import if the server was older. Fix by
casting the column value to text before collating it if the server is
older.
Also, fix a typo in the if-test in the same function.
Oversights in commit 28972b6fc; the former was reported by Fujii Masao.
Reported-by: Fujii Masao <masao.fujii@gmail.com>
Author: Etsuro Fujita <etsuro.fujita@gmail.com>
Reviewed-by: Corey Huinker <corey.huinker@gmail.com>
Discussion: https://postgr.es/m/CAHGQGwGvA_3q7Wg7GXQkuugGZj3%3DkxDRjDsKgVvK1uwAxHTopQ%40mail.gmail.com
Backpatch-through: 19
The reference page said WAIT "cannot be used in contexts where such a
snapshot must remain active, including transactions running at isolation
levels higher than READ COMMITTED", which reads as a blanket ban at those
levels. It is not one. The rule the code enforces is that the current
transaction must hold no snapshot, and a transaction-snapshot-mode
transaction has no snapshot until its first query: WaitStmt is exempt in
PlannedStmtRequiresSnapshot(), so the command itself does not take one.
WAIT therefore works before that first query, which is the order an
application wants anyway, and it is what 447aae13b0 set out to allow by
keeping the wait snapshot-free.
State the rule in terms of the snapshot rather than the isolation level,
say that WAIT acquires none of its own and name the other ways to hold
one, and give the consequence of holding one: it can delay replay, which
standby_replay waits for and which the other standby modes ultimately
depend on as well.
The error for a held snapshot named the internal snapshot states instead
of saying what the user did, and attached the isolation-level
explanation as DETAIL unconditionally, so a READ COMMITTED session with a
cursor was told its isolation level was too high. Report what is held
rather than how it is held, and emit the DETAIL only when the isolation
level is in fact the reason.
Reported-by: Sami Imseih <samimseih.pg@gmail.com>
Reported-by: Xuneng Zhou <xunengzhou@gmail.com>
Reviewed-by: Sami Imseih <samimseih.pg@gmail.com>
Reviewed-by: Xuneng Zhou <xunengzhou@gmail.com>
Discussion: https://postgr.es/m/CAN12%2BYJddgAZzA36JZFX%2BYrnNR7Uubp9qWFz6ASs3v0kL30AMA%40mail.gmail.com
Backpatch-through: 19
Commit a49b9cfd72 began reporting the EXCEPT clause relation with a
schema-qualified name that quotes identifiers when needed, inside a
message that already puts the name in quotes. So a schema or relation name
that needs quoting came out quoted twice.
Fix by building the qualified name as "%s.%s" without identifier quoting,
so it renders like the qualified names in other backend error messages.
The message text, and so its translations, is unchanged.
RelationGetQualifiedRelationName() has no callers left, so remove it.
Author: Manuel Reyes Bravo <manuelreyesbravo@gmail.com>
Reviewed-by: shveta malik <shveta.malik@gmail.com>
Reviewed-by: Vignesh C <vignesh21@gmail.com>
Reviewed-by: Peter Smith <smithpb2250@gmail.com>
Reviewed-by: Zhijie Hou <houzj.fnst@fujitsu.com>
Reviewed-by: Amit Kapila <amit.kapila16@gmail.com>
Discussion: https://postgr.es/m/CAHGQGwHfESBexa7fq99EvFCf31av=O9h9udnw22ymxmm6LMZzw@mail.gmail.com
Backpatch-through: 19, where it was introduced
Some code added in 489247b0e tried to prune the DEFAULT partition when
the next partition had a MINVALUE clause and likewise when the final
partition to scan had a MAXVALUE clause for the final partition key in
the pruning step. This code was incorrect as it could prune the default
partition incorrectly in cases such as:
p: PARTITION BY RANGE (a, b)
p1: FOR VALUES FROM (13, 0) TO (19, MAXVALUE)
pd: DEFAULT
SELECT * FROM t WHERE a = 32 AND b >= -7;
Here the pruning step for a = 32 and b >= -7 would see that only the
default partition needs to be scan, but it would then see that the
partition prior to the default had a MAXVALUE bound then prune away the
default thinking that it needn't be scanned. This could result in
incorrect results.
Fix this by moving the code that looks for the MAXVALUE bound into the
code handling BTEqualStrategyNumber so that when we're pruning with a
prefix of the partition keys, we check if the bound for the offset we've
calculated lands on a partition where the next partition key is bounded
with MAXVALUE. If so we don't include the default partition.
This leaves only the case of the first partition key. We handle that by
modifying the existing code that was checking the last key covered by
the given values.
Author: Ewan Young <kdbase.hack@gmail.com>
Reviewed-by: Tender Wang <tndrwang@gmail.com>
Reviewed-by: David Rowley <dgrowleyml@gmail.com>
Discussion: https://postgr.es/m/CAON2xHO=sqdqp=z8zWkybnWp0AuvefnAi2ez2vOYWrhXB6hHWQ@mail.gmail.com
Backpatch-through: 14
Building the partition pruning steps for excessively complex queries
can take some time. Add CHECK_FOR_INTERRUPTS() to
get_steps_using_prefix_recurse() to allow more timely processing of
interruptions.
Bug: #19692
Reported-by: Qifan Liu <imchifan@163.com>
Discussion: https://postgr.es/m/19692-dacfc57cda5035fc@postgresql.org
Backpatch-through: 14
When reverting I had missed updating the documentation for tracking data
checksum failures and the checksum version in the REDO record. The test-
code for enabling and disabling checksums offline was originally kept to
minimize backpatching conflicts, but the amount of testcode backpatching
doesn't really warrant keeping dead code.
Since no version has been shipped since bumping XLOG_PAGE_MAGIC it is
left at the previous bump version.
Reported-by: Fujii Masao <masao.fujii@gmail.com>
Reported-by: Manuel Reyes Bravo <manuelreyesbravo@gmail.com>
Discussion: https://postgr.es/m/CAHGQGwFLNCA9nVHSsC_=G_mvsK_fAFb+QbJAL7=R9cK2PL9ZBQ@mail.gmail.com
This commit fixes several related problems. All of these problems
escaped testing for the same reason: they do not occur when only
generated advice is supplied, and therefore test_plan_advice was
incapable of detecting these shortcomings.
First, prior to this commit, the advice feedback code never regarded
an unordered sublist of a JOIN_ORDER specification as matching an
unrolled join. As a result, when JOIN_ORDER() advice contained
unordered sublists, the advice feedback tended to be "matched, failed"
even when the advice worked exactly as intended. Only the case where
an unordered sublist was implemented by a plan shape not subject to
unrolling, such as a partitionwise join, worked properly.
Second, prior to this commit, a join order sublist in the initial
position wasn't properly handled. Note that this overlaps with the
problem described in the previous paragraph; JOIN_ORDER({a b} c) was
broken both because of the unordered sublist and because of the
sublist being in the initial position. However, there's more to this
case: JOIN_ORDER((a b) c) means the same as JOIN_ORDER(a b c), but the
advice feedback code didn't know that, and would generate "matched,
failed" for the former case even when everything was working.
Finally, prior to this commit, advice feedback didn't correctly handle
single-element sublists. Those are pretty nonsensical, since by
definition a join involves at least 2 tables, so we could just ban
that case. Instead, at least for now, I've chosen to make advice
feedback handle such cases in the same way that advice enforcement
already does: the extra grouping levels are simply disregarded, so
that JOIN_ORDER({a} ((b))) is enforced in the same way as, and also
gets the same advice feedback as, JOIN_ORDER(a b).
Reported-by: Noah Misch <noah@leadboat.com>
Backpatch-through: 19
Discussion: https://postgr.es/m/CA+TgmoYmXy-jiP5qDhqNEiYFEBzQsArO6O2d9E8szNZqi1bePQ@mail.gmail.com
Reviewed-by: Ayush Tiwari <ayushtiwari.slg01@gmail.com>
Prior to v18, the total cost of a plan was just one number, a
floating-point value. Commit e222534679
introduced a second component, the number of disabled_nodes at or
below that level of the plan tree. Unfortunately, I (rhaas) failed
to realize that GEQO extracts the cost from each candidate plan and
compares it directly as a measure of fitness.
This means that, at least in principle, it's possible for GEQO to
prefer a plan with more disabled_nodes and lower cost over one with
fewer disabled_nodes and higher cost, which is incorrect. It may be
possible for that to become a practical issue in v18, but it's much
more likely to become an issue in v19 due to pg_plan_advice. Hence,
since the straightforward fix breaks ABI compatibility, back-patch
only to v19.
Backpatch-through: 19
Discussion: https://postgr.es/m/CA+TgmoYmXy-jiP5qDhqNEiYFEBzQsArO6O2d9E8szNZqi1bePQ@mail.gmail.com
The check_oauth_validator check for an empty validator list didn't test
for a string with only whitespace, which would cause the postmaster to
crash. Instead of testing for the empty string cases, pass the config
value to SplitDirectoriesString unconditionally. If an empty list is
returned then the input string was empty. Since pstrdup cannot handle
NULL, assert that the string ie set. While users cannot set the string
to NULL, it is initialized to NULL so guard against programmer error.
Check the parsed list instead. Assert that the GUC string is non-NULL
before pstrdup(); users cannot set it to NULL, but the C variable is
initialized that way.
This also adds a TAP test that reloads a whitespace-only setting after
pg_hba_file_rules and waits until the existing backend sees the restored
GUC.
Author: Grigorev Jurij <ju.grigorev@ftdata.ru>
Reviewed-by: Jacob Champion <jacob.champion@enterprisedb.com>
Reviewed-by: Daniel Gustafsson <daniel@yesql.se>
Discussion: https://postgr.es/m/04fa84f6ebbe400f940e179ebe1070e9@localhost.localdomain
Backpatch-through: 18
The worker connects as the table owner in a new session, so a role- or
database-level lock_timeout and transaction_timeout unhelpfully apply to
it, possibly canceling the wait for older transactions. This cannot be
overridden by the REPACK calling session. Fix by passing the timeout
values from the steering backend via the DecodingWorkerShared struct and
adopting these values in the worker; this way, values from session-local
SET reach the worker too.
In passing, remove a pointless member from the same struct.
Author: Shihao Zhong <zhong950419@gmail.com>
Backpatch-through: 19
Discussion: https://postgr.es/m/CAGRkXqTYaBjFvjtjPb1+=sZWvt93=2c472bmM+xdG7w9ZvAydA@mail.gmail.com
pgstat_get_entry_ref_cached() inserts a local entry_ref with
shmem-related fields set to NULL, expecting pgstat_get_entry_ref() (its
sole caller) to fill them up before returning. If an ERROR happens
while pgstat_get_entry_ref() runs, it could be possible to finish with a
local pgstats entry partially filled.
This could lead to a crash of pgstat_gc_entry_refs(), which tolerates a
NULL shared_stats in an assertion but unconditionally dereferenced its
"dropped" and "generation" fields.
This extends 4069df21be, being a cheap insurance against NULL pointer
dereference, if some code paths of pgstat_get_entry_ref() are not able
to perform any cleanup actions (for example after a dsm_create()
throwing an ERROR).
Reviewed-by: Grigorev Jurij <ju.grigorev@ftdata.ru>
Discussion: https://postgr.es/m/aqtiKTvl519bu8-V@paquier.xyz
Backpatch-through: 15
A REPACK background worker can fail to start for various reasons, but
the REPACK steering process was not handling them correctly. For
instance, the steering process waits on a condition variable, but it
would simply hang if the worker doesn't send the signal. (A signal
would be sent by postmaster, but it would wake up the process only to be
ignored and continue sleeping.) Fix this by splitting the wait in two:
first wait for the worker to attach to the error queue, and then wait on
the condition variable as before.
The shutdown sequence can also get stuck, if the message queue gets full
while trying to shut down (this can apparently happen if the log level
is set to very noisy). Previously the worker would send a signal and
then detach; but the steering process can receive that signal, act on
it, then fail to wake up to handle the detach, sleeping indefinitely.
Fix this by having the worker send a Terminate message, which the
steering process can react to.
Author: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Author: Shihao Zhong <zhong950419@gmail.com>
Author: Álvaro Herrera <alvherre@kurilemu.de>
Reviewed-by: Antonin Houska <ah@cybertec.at>
Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
Reviewed-by: Shihao Zhong <zhong950419@gmail.com>
Reported-by: Nathan Bossart <nathandbossart@gmail.com>
Reported-by: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Reported-by: Nikolay Samokhvalov <nik@postgres.ai>
Backpatch-through: 19
Discussion: https://postgr.es/m/CALj2ACVAxA9HxvFe8HSspTJ-UO4Aoz%3DkuQdZBeLrod0gqUxH3g%40mail.gmail.com
Discussion: https://postgr.es/m/apBpOVZOyqrakEr_@nathan
The reference page and the read-your-writes section both presented
WAIT in standby_replay mode as enough for read-your-writes, without
saying which LSN to wait for. Read as written, an LSN taken after the
last statement but before COMMIT would do, and it would not: the standby
position WAIT compares against is the end of the last replayed record, so
a target that falls inside the COMMIT record is satisfied while that
record is still unapplied.
Say that the target must be at or after the end of the write
transaction's COMMIT record, and note in the example that autocommit is
what puts the UPDATE's commit before the call that reads the position.
Author: Sami Imseih <samimseih.pg@gmail.com>
Reviewed-by: Xuneng Zhou <xunengzhou@gmail.com>
Reviewed-by: Alexander Korotkov <aekorotkov@gmail.com>
Discussion: https://postgr.es/m/CAN12%2BYKN7gXb1o9bDWgfA%3D6wjDrqNEgt%2BrXJiAWXgPCXQ5TqpA%40mail.gmail.com
Backpatch-through: 19
The description of NO_THROW covered what the option does but not when an
application would want it. Explain that it suits callers that treat
timeout or not in recovery as expected results to be handled as part of
ordinary control flow, note the obligation to check the returned status,
and give the reason to omit the option.
Be precise about the option's scope: it changes only how those two wait
outcomes are reported. Soften the description's claim that the command
"returns a status string instead of throwing errors" accordingly, and say
that invalid input still raises an error, as does any condition checked
before the wait begins. Name two of those conditions as examples rather
than listing them, since the set grows.
Author: Rithvika Devisetti <devisettirithvika@gmail.com>
Reported-by: Peter Eisentraut <peter@eisentraut.org>
Reviewed-by: Kiran Kaki <itskkpg@gmail.com>
Reviewed-by: Xuneng Zhou <xunengzhou@gmail.com>
Reviewed-by: Satyanarayana Narlapuram <satyanarlapuram@gmail.com>
Reviewed-by: Alexander Korotkov <aekorotkov@gmail.com>
Discussion: https://postgr.es/m/14415b05-16d0-446f-ba7d-c891db8b3406%40eisentraut.org
Backpatch-through: 19
Previously, aborting an internal subtransaction during COMMIT or
PREPARE TRANSACTION could cause the following assertion failure.
This could happen, for example, when a deferred constraint trigger fired
at COMMIT and its PL/pgSQL exception block caught an error raised
while executing the trigger function.
TRAP: failed Assert("s->blockState == TBLOCK_SUBINPROGRESS || s->blockState
== TBLOCK_INPROGRESS || s->blockState == TBLOCK_IMPLICIT_INPROGRESS ||
s->blockState == TBLOCK_PARALLEL_INPROGRESS || s->blockState ==
TBLOCK_STARTED"), File: "xact.c", Line: 4851, PID: 73455
An internal subtransaction should be able to be aborted while the parent
transaction is in the COMMIT or PREPARE TRANSACTION phase. However,
RollbackAndReleaseCurrentSubTransaction()'s assertion check previously
did not allow TBLOCK_END and TBLOCK_PREPARE as parent transaction
states, causing the assertion failure.
This commit fixes the assertion check by allowing those two parent
transaction states.
Backpatch to all supported versions.
Reported-by: Fabrízio de Royes Mello <fabrizio@planetscale.com>
Author: Patrick Reynolds <piki@planetscale.com>
Reviewed-by: Fujii Masao <masao.fujii@gmail.com>
Reviewed-by: Chao Li <li.evan.chao@gmail.com>
Reviewed-by: Fabrízio de Royes Mello <fabrizio@planetscale.com>
Discussion: https://postgr.es/m/CABo-N97AeMbWuYTWg-3%3D2DkTR3EkvS%2BFt%3DyEaWB181STsR1mBg%40mail.gmail.com
Backpatch-through: 14
A pg_publication_rel entry can represent either an explicitly published
relation or a table excluded from a FOR ALL TABLES publication, but the
object address code treated every entry as a published relation without
checking prexcept. EXCEPT entries thus appeared as ordinary
published-table mappings in dependency messages, pg_identify_object(),
and pg_identify_object_as_address().
Report such entries as "publication excluded relation" instead, with
object identities indicating that the table is excluded from the
publication. pg_get_object_address() also accepts this new object type,
so that the output of pg_identify_object_as_address() can still be
passed back to it to look up the same object. "publication relation"
now resolves only non-EXCEPT entries, and "publication excluded
relation" only EXCEPT entries.
Oversight in commit fd366065e0.
Author: Fujii Masao <masao.fujii@gmail.com>
Author: Vignesh C <vignesh21@gmail.com>
Reviewed-by: shveta malik <shveta.malik@gmail.com>
Reviewed-by: Peter Smith <smithpb2250@gmail.com>
Reviewed-by: Zhijie Hou <houzj.fnst@fujitsu.com>
Reviewed-by: Nisha Moond <nisha.moond412@gmail.com>
Reviewed-by: Amit Kapila <amit.kapila16@gmail.com>
Reviewed-by: Chao Li <li.evan.chao@gmail.com>
Reviewed-by: Manuel Reyes Bravo <manuelreyesbravo@gmail.com>
Discussion: https://postgr.es/m/CAHGQGwHfESBexa7fq99EvFCf31av=O9h9udnw22ymxmm6LMZzw@mail.gmail.com
Backpatch-through: 19, where it was introduced
Up until now, pg_plan_advice has had a feature that allows an advice
target to mention a partition name but omit the partition schema;
that is, something like SEQ_SCAN(foo/bar) forces a sequential scan on
every child of table foo whose partition name is bar, regardless of
the schema in which bar appears. However, that feature turns out to
have a nasty design flaw: while advice enforcement handles this case
just fine, the advice feedback code doesn't know about it and will
mark such advice as "matched, failed" even when everything worked
perfectly. Unfortunately, there seems to be no simple code fix for
this problem.
Since the release of PostgreSQL 19 is imminent, take the conservative
course and revert this feature of pg_plan_advice. In other words,
require the partition schema whenever the partition name is present.
You must now write e.g. SEQ_SCAN(foo/public.bar) rather than just
SEQ_SCAN(foo/bar). This doesn't affect any cases where automatically
generated advice is supplied, since generated advice has always
included the partition schema anyway. Manually written advice will
have to conform to the new, stricter rule. For a later release, we
can consider whether to again relax this restriction in some way,
but now is not the time to design new things.
Reported-by: Noah Misch <noah@leadboat.com>
Backpatch-through: 19
Discussion: https://postgr.es/m/CA+TgmoYmXy-jiP5qDhqNEiYFEBzQsArO6O2d9E8szNZqi1bePQ@mail.gmail.com
Like in PerformOffsetsTruncation() and e.g. in TruncateSUBTRANS(),
need to step back to previous page at page boundary, to avoid passing
a cutoff page to SimpleLruTruncate() that hasn't been created
yet. Otherwise you can get a bogus 'LOG: could not truncate directory
"pg_multixact/members": apparent wraparound' message in the log.
Reported-by: Noah Misch <noah@leadboat.com>
Discussion: https://www.postgresql.org/message-id/20260827231757.78.noahmisch@microsoft.com
Backpatch-through: 19
A semijoin whose RHS is unique-ified groups the RHS on the expressions in
SpecialJoinInfo.semi_rhs_exprs. Those were recorded with whatever collation
the RHS expression itself exposes, which need not be the collation the join
compares with. Neither SortGroupClause nor the pathkey machinery carries a
collation of its own, so both Unique-over-Sort and HashAggregate then grouped
by the wrong equality: values the join considers equal survived, and the
following inner join emitted the outer row once per survivor.
With a non-deterministic collation on one side, "SELECT count(*) FROM t WHERE
c IN (SELECT c0 FROM t2)" therefore counted more rows than the same predicate
reports for the rows of t.
Label each RHS expression with the operator's input collation, the same
treatment process_equivalence() gives to equivalence class members. Every
consumer of semi_rhs_exprs reads the collation off the expression, so this
fixes the sort-based and hash-based paths together; in the branches where
create_unique_path() also passes these expressions to
relation_has_unique_index_for(), it likewise stops a unique index built with a
different collation from being taken as proof that unique-ification can be
skipped. The same goes for a subquery's DISTINCT computed under a different
collation, since translate_sub_tlist() punts on the relabeled expressions.
Reported-by: Suyang Zhong <syzhong16@gmail.com>
Author: Andrey Rachitskiy <pl0h0yp1@gmail.com>
Reviewed-by: Tender Wang <tndrwang@gmail.com>
Reviewed-by: Richard Guo <guofenglinux@gmail.com>
Reviewed-by: Alexander Korotkov <aekorotkov@gmail.com>
Discussion: https://postgr.es/m/19633-647cd4c73a84b085%40postgresql.org
Backpatch-through: 14
Previously, during REPACK (CONCURRENTLY), index_rebuild_count in
pg_stat_progress_repack and pg_stat_progress_cluster did not advance
as indexes were rebuilt. This made it impossible for users monitoring the
operation to tell how many indexes had been completed.
Fix this by incrementing the count after each index is built on the new
heap, so that both views report the number of completed index builds.
Author: Fujii Masao <masao.fujii@gmail.com>
Reviewed-by: Álvaro Herrera <alvherre@kurilemu.de>
Discussion: https://postgr.es/m/CAHGQGwFUsrBvTurkSU8TEc=DZTunqteUXrasseqNMT7AMBZKRw@mail.gmail.com
Backpatch-through: 19
Previously, when REPACK, CLUSTER, or VACUUM FULL created a transient TOAST
table, index_rebuild_count in pg_stat_progress_repack and
pg_stat_progress_cluster reported 2 before any indexes on the main table
had been rebuilt. This could mislead users monitoring the operation.
This happened because creating the TOAST index writes the CREATE INDEX phase
to the same progress slot used for the index rebuild count of the enclosing
command.
Fix this by passing INDEX_CREATE_SUPPRESS_PROGRESS when creating a TOAST
index, preserving the progress information of the enclosing command and
keeping the count at zero until the main-table indexes are rebuilt.
Author: Adam Lee <adam8157@gmail.com>
Reviewed-by: Fujii Masao <masao.fujii@gmail.com>
Reviewed-by: Álvaro Herrera <alvherre@kurilemu.de>
Discussion: https://postgr.es/m/aj4gJQMba0kClQmj@Mac
Backpatch-through: 19
Commit 5f3a49abf7 changed the primary status update message to report the
end of the last inserted WAL record instead of the WAL write position.
However, the protocol documentation still described the WAL position field as
the latest WAL write position, which no longer matches the message sent by
walsender.
Update the description of the WAL position field to indicate the end of the
last inserted WAL record, matching the implementation.
Author: Fujii Masao <masao.fujii@gmail.com>
Reviewed-by: Ayush Tiwari <ayushtiwari.slg01@gmail.com>
Discussion: https://postgr.es/m/CAHGQGwFn2QfpHDR6cWxd98tqNb5vyguEis8rtqnqidFR36p4QQ@mail.gmail.com
Backpatch-through: 19
Commit 0c5d626961 was confused about which indexes are potentially
problematic to rebuild: we thought only !indisready indexes could fail
to rebuild, and thus would attempt to rebuild !indisvalid ones that are
"ready" (that is, being maintained by DML but not yet completed).
However, that's wrong, because it could be that the validation phase of
such an index failed or was interrupted partway through. So, have
REPACK throw an error with all !indisvalid indexes, not just those
!indisready.
(!indisready implies !indisvalid, so this is an expansion of what
happened with the previous commit.)
Reported-by: shihao zhong <zhong950419@gmail.com>
Backpatch-through: 19
Discussion: https://postgr.es/m/CAGRkXqRA4YetT8Nu3+1-45+ffTwi2bd1d1eDFME39Y9OPRMytQ@mail.gmail.com
INSERT ... ON CONFLICT decides what to do based on the conflicting row
found by the arbiter index probe, but SSI never saw that read: the
probe runs with a dirty snapshot, which predicate locking ignores, and
the later fetch of the row uses SnapshotAny. When the statement then
writes nothing, as with DO NOTHING, DO UPDATE with a WHERE clause
rejecting the row, or DO SELECT, nothing records the read at all. A
concurrent writer of that row went unnoticed and write skew could
commit at SERIALIZABLE, even though the same schedule with a plain
SELECT of the row fails with a serialization error.
To fix, read the conflicting tuple again with the query snapshot,
right where the probe finds it. The table AM takes the SIREAD lock
and checks for a concurrent writer of the tuple as part of that read,
both under the buffer lock, so a writer either sees the lock or is
seen. A predicate lock by itself acquired separately after the probe
could not offer that: a writer passing its conflict check in between
would be missed. Doing this in the probe covers every conflict
action, including rows that the WHERE clause of DO UPDATE or DO SELECT
then rejects.
The DO NOTHING and DO UPDATE cases have been broken since ON CONFLICT
was added in 9.5; DO SELECT is new in v19. Backpatch to all supported
branches.
Author: Zsolt Parragi <zsolt.parragi@percona.com>
Author: Andrey Borodin <x4mmm@yandex-team.ru>
Reported-by: Andrey Borodin <x4mmm@yandex-team.ru>
Reported-by: Zsolt Parragi <zsolt.parragi@percona.com>
Discussion: https://postgr.es/m/787936C5-4155-4CF9-939D-39DC0EC1C892@yandex-team.ru
Discussion: https://postgr.es/m/CAN4CZFM1GkHJkpMeo4G5rxtacVsfeKCJYiik9E9AKX1E9VYQ1w@mail.gmail.com
Backpatch-through: 14
List of commits reverted:
8e72d914c5 Add UPDATE/DELETE FOR PORTION OF
b6ccd30d8f Add isolation tests for UPDATE/DELETE FOR PORTION OF
33db6c4baf Fix DELETE/UPDATE FOR PORTION OF with rules
7b22f15a01 Add psql tab completion for FOR PORTION OF clause
7ca8c94296 Fix FOR PORTION OF with non-updatable view columns
993a7aa0e4 Fix cross-leftover pollution in FOR PORTION OF insert triggers
7ac030d5b1 Require UPDATE permission on FOR PORTION OF column
f0aedc7cb0 Small terminology fixes in comments
7d13b03a2e Fix FOR PORTION OF for inheritance children
a272a58b94 Move FOR PORTION OF volatile check into planner
a40fdf6588 Reject child partition FDWs in FOR PORTION OF
bc3ae886a7 Forbid FOR PORTION OF with WHERE CURRENT OF
c8d49ffb00 Forbid generated columns in FOR PORTION OF
5b5e99047a Forbid FOR PORTION OF on views with INSTEAD OF triggers
c58c83ce50 Fix RLS checks for FOR PORTION OF leftover rows
5454fe4a6e Test what BEFORE UPDATE triggers do to FOR PORTION OF
2a9541ddfd Deparse FOR PORTION OF using the range column's current name.
fd92f74b91 Avoid RETURNING side effects for FOR PORTION OF leftovers.
f3a116a26f Enforce WITH CHECK OPTION on DELETE FOR PORTION OF leftovers
c7e34c31e8 Fix error code for null FOR PORTION OF target
938f4f8f34 Fix assertion failures in DELETE FOR PORTION OF tuple routing
f194032168 Resolve untyped parameters in FOR PORTION OF FROM/TO bounds
98f6984e5c Don't evaluate the FOR PORTION OF target under EXPLAIN
plus release notes entries.
Discussion: https://www.postgresql.org/message-id/62bf70cc-dc33-4188-8070-a8c177bb61de%40eisentraut.org
d96f87332b folded COPY FREEZE visibility map updates into
XLOG_HEAP2_MULTI_INSERT records. Unlike the former XLOG_HEAP2_VISIBLE
redo path, multi-insert redo only updated the FSM when the record did
not contain a heap page FPI and the heap page had less than 20% free
space.
This can leave all-frozen pages with missing or stale FSM entries on a
standby. After promotion, vacuum will skip those pages, preventing
their free space from being discovered.
When a multi-insert record sets the page all-frozen, record the heap
page's free space unconditionally.
Reported-by: Melanie Plageman <melanieplageman@gmail.com>
Discussion: https://postgr.es/m/CAAKRu_bS3nfMJCF2fjE%2Bceb5-t%3D2p_VPMk2PWuVomrOFxg5DGQ%40mail.gmail.com
Backpatch-through: 19
add323da40 consolidated visibility map updates into XLOG_HEAP2_PRUNE
records. It assumed that a record setting the visibility map would
always modify the VM page during replay. That is not true when the
requested bits are already set on the standby.
Primary and standby visibility maps can diverge for several reasons.
This issue was exposed by a CREATE DATABASE ... STRATEGY WAL_LOG bug.
After a standby with an out-of-date VM is promoted, VACUUM may set bits
that were already present on the former primary. When the former primary
later replays that record as a standby, visibilitymap_set() is a no-op.
Restore visibilitymap_set()'s former API, which returns the state of the
VM bits before setting the requested flags. Use that result to set the
VM page LSN only when the visibilitymap_set() updates the page.
Reported-by: Rogers Wang <rogers.ww@qq.com>
Discussion: https://postgr.es/m/tencent_2E870046716FD94285045E96505A2D4E2908%40qq.com
Backpatch-through: 19
During a parallel GIN build, each key is serialized into a GinTuple, a
transient representation used only while sorting. _gin_build_tuple()
lays out the whole tuple (the palloc size, the key memcpy, and the
offset of the posting list) from a local variable holding the real key
length, but then stored that length in GinTuple.keylen, which was
uint16. For a key longer than 65535 bytes, the stored length was thus
silently truncated.
On read-back, GinTupleGetFirst() and _gin_parse_tuple_items()
recompute the posting-list offset from the truncated keylen and land
inside the key data, so ginPostingListDecodeAllSegments() decodes
garbage: an assertion failure with assertions enabled, and a read past
the end of the allocation without, which could in turn lead to a crash
or garbage being written into the index. Only parallel builds are
affected, because only they materialize a GinTuple; a serial build of
the same data succeeds, as index_form_tuple() compresses large keys
before the GinMaxItemSize check.
To fix, widen GinTuple.keylen to Size, which is what VARSIZE_ANY()
returns and what GinBuffer.keylen already uses, and use Size for the
local in _gin_build_tuple() too, which was an int that truncated
VARSIZE_ANY() the same way.
Widening keylen moves GinTuple.data. The key value is accessed in
place in data, so data must be MAXALIGN'ed; before, that was only true
by accident of the field layout. Make that explicit with
alignas(MAXIMUM_ALIGNOF) on data, so that the layout stays correct on
32-bit platforms, where Size is 4 bytes and would otherwise leave data
under-aligned when MAXIMUM_ALIGNOF is 8.
Bug: #19545
Author: Ewan Young <kdbase.hack@gmail.com>
Reported-by: Yuelin Wang <1217816127@qq.com>
Reviewed-by: Peter Eisentraut <peter@eisentraut.org>
Discussion: https://www.postgresql.org/message-id/flat/19545-0f25b7e47351e8fc%40postgresql.org
378a216187 set pd_prune_xid on inserts so that on-access pruning could
later set the pages all-visible. heap_multi_insert() skipped this only
when setting the page all-frozen, but it can insert frozen tuples even
when not setting the page all-frozen. Inserting frozen tuples should not
set pd_prune_xid as they introduce no work for on-access pruning. Fix by
only setting pd_prune_xid when inserting non-frozen tuples.
Reported-by: Melanie Plageman <melanieplageman@gmail.com>
Reviewed-by: Chao Li <li.evan.chao@gmail.com>
Discussion: https://postgr.es/m/CAAKRu_a02pu7U2S0qPZ5L%2BoWb2m2Zp965sOKCCLnKUAKNo-hfg%40mail.gmail.com
Backpatch-through: 19
Commit 977d865c36 established some rules for mentioning special
values (e.g., -1, 0, an empty string) in GUC descriptions. A few
parameters either neglect to mention their special values or
describe them in a way that doesn't follow those rules. Most of
these parameters were added after the aforementioned commit. This
commit fixes them.
Reviewed-by: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Reviewed-by: Fujii Masao <masao.fujii@gmail.com>
Reviewed-by: Chao Li <li.evan.chao@gmail.com>
Reviewed-by: Álvaro Herrera <alvherre@kurilemu.de>
Discussion: https://postgr.es/m/aqMkIpRr62cVqtCZ%40nathan
Backpatch-through: 19
The command tag is WAIT, but the reference page called the command
"WAIT FOR" in its title, index entry, psql help name and throughout its
prose, and so did the error messages. FOR is a noise word that makes the
grammar read like English, exactly as INTO does for INSERT and FROM does
for DELETE; neither of those is part of the command's name. Settle on
WAIT everywhere the command is named, and keep WAIT FOR LSN wherever the
syntax itself is being shown.
Rename the reference page's id and file name to match, following
cb2c696b1d: v19 is not released, so the URL is not yet baked into a
supported version, and every other page's id matches its title.
Reported-by: Masahiko Sawada sawada.mshk@gmail.com
Author: Xuneng Zhou xunengzhou@gmail.com
Reviewed-by: Alexander Korotkov aekorotkov@gmail.com
Discussion: https://postgr.es/m/CAD21AoBdtiPTbm7T_aNeDON9JpFG9g%3DJDUP4WU-Y8t8_xqvR5Q%40mail.gmail.com
Backpatch-through: 19
