The replica identity determination was "flawed": it fell back to the
primary key if the marked index was dropped, which is a defensible
choice, but logical decoding has a differing opinion and fails to
provide usable tuple identity data in this case. "Fix" REPACK by
refusing to use the primary key in that case.
This is arguably broken behavior in RelationGetIndexList, but I'll
refrain from changing that at this stage.
Reported-by: Nathan Bossart <nathandbossart@gmail.com>
Author: Matthias van de Meent <boekewurm+postgres@gmail.com>
Author: Ewan Young <kdbase.hack@gmail.com>
Backpatch-through: 19
Discussion: https://postgr.es/m/apCBRKCH8jwKiaSY@nathan
ALTER TABLE ... DETACH PARTITION ... CONCURRENTLY leaves the partition
marked as detach-pending if its wait is interrupted, and only DETACH
PARTITION ... FINALIZE clears that mark. In that state pg_class still says
relispartition while get_partition_ancestors() already reports nothing,
which RelationBuildPublicationDesc() was not ready for. It took
relispartition to mean that the ancestor list is not empty and asked for
its last element, which is an assertion failure, or a NULL pointer
dereference without assertions. CheckCmdReplicaIdentity() needs the
descriptor for every UPDATE and DELETE of a publishable relation and
builds it on first use, so no publication has to exist for this, and a
plain UPDATE crashes the backend.
Treat such a partition as a standalone table, as after the detach is
finalized. It is then published by FOR ALL TABLES publications, but cannot
be added to an EXCEPT clause until the detach completes, which is now
reported with a hint to run DETACH PARTITION ... FINALIZE.
Oversight in fd366065e0, which added the exclusion.
Author: Mikhail Nikalayeu <mihailnikalayeu@gmail.com>
Author: shveta malik <shveta.malik@gmail.com>
Author: Nisha Moond <nisha.moond412@gmail.com>
Reviewed-by: shveta malik <shveta.malik@gmail.com>
Reviewed-by: Amit Kapila <amit.kapila16@gmail.com>
Reviewed-by: Zhijie Hou <houzj.fnst@fujitsu.com>
Reviewed-by: Álvaro Herrera <alvherre@kurilemu.de>
Reviewed-by: Nisha Moond <nisha.moond412@gmail.com>
Discussion: https://postgr.es/m/CADzfLwWoFPT%2Ba73%3DA%3DbsNWRMZQ98NpBEMgE%3Dt1FS4O4_%3DQVLfA%40mail.gmail.com
Backpatch-through: 19, where it was introduced
The tab-completion query used for ALTER DATABASE ... RESET referenced
pg_db_role_setting and pg_database without schema qualification, so a
same-named relation earlier in search_path could be used instead of the
system catalogs. This could make psql offer misleading RESET candidates.
Commit dbf5a83d46 schema-qualified the unnest() call in this same query,
but the catalog references were still left unqualified. Fix that by
referring to pg_catalog.pg_db_role_setting and pg_catalog.pg_database
explicitly.
Backpatch to 18, where the tab-completion for ALTER DATABASE ... RESET
was introduced.
Bug: #19523
Reported-by: Zhou Digoal <digoal@126.com>
Author: Vismay Tiwari <vismay.t@gmail.com>
Reviewed-by: Paul Kim <mok03127@gmail.com>
Reviewed-by: Kirill Reshke <reshkekirill@gmail.com>
Reviewed-by: Fujii Masao <masao.fujii@gmail.com>
Discussion: https://postgr.es/m/19523-424457118202f570@postgresql.org
Discussion: https://postgr.es/m/CALHMmB84qkCgv3QAR78YawgfqQZCxSPkRhppxzU=e6fg8RA+AA@mail.gmail.com
Backpatch-through: 18
The fast path probes using the referenced index's collation. When that
collation differs from the referenced column's, it can reject valid
references or accept invalid ones. Changing the scan key's collation
is not sufficient, since btree navigation must use the ordering under
which the index was built.
Cache index eligibility lazily after locking the referenced table,
reloading the constraint information, and opening the index. Compare
index and column collations using the held relation descriptors, and
fall back to SPI on a mismatch. Also move the btree eligibility check
here, avoiding a lookup of an index that REINDEX CONCURRENTLY could
drop before the referenced-table lock is acquired.
Cache both acceptance and rejection until the constraint information
is reloaded, so subsequent rows need not repeat the comparisons. Check
before probing or buffering any row. If batch entry initialization
rejects the index, remove that entry while preserving any other pending
checks.
Add ICU regression coverage for per-row validation, ordinary inserts,
reordered index columns, both directions of collation mismatch, and
fallback while another foreign key check is pending.
Reported-by: Amit Kapila <amit.kapila16@gmail.com>
Discussion: https://postgr.es/m/CAA4eK1Lk1DfgiFPL-HccZm-Nm+exg+FcsODH8KPw6BOBuzfHtA@mail.gmail.com
Backpatch-through: 19
AlterPublicationOptions() can use stale publication state when the
publication is modified concurrently while waiting for the publication
lock. In particular, a concurrent ALTER PUBLICATION ... SET ALL TABLES
can change puballtables, causing the validation to make decisions based
on the old value and leads to assertion failure. Likewise,
CheckAlterPublication() could miss a table added concurrently, letting
SET ALL TABLES EXCEPT (...) leave pg_publication_rel with a mixture of
inclusion and exclusion rows.
Fix by acquiring the lock and re-reading the publication tuple in
AlterPublication() before either path runs, and by calling
CheckAlterPublication() only after that.
Also fix 037_except.pl, which left one computed result untested and
silently reused a subscription created by an earlier test.
Author: Vignesh C <vignesh21@gmail.com>
Reviewed-by: Amit Kapila <amit.kapila16@gmail.com>
Reviewed-by: Chao Li <li.evan.chao@gmail.com>
Reviewed-by: shveta malik <shveta.malik@gmail.com>
Reviewed-by: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Discussion: https://postgr.es/m/CALDaNm1r2MkGu6h8zgU1Kj1sX-FcMQ7wGTeLSnhx-5joiyXEvg@mail.gmail.com
Backpatch-through: 19, where it was introduced
This relation option gives the possibility to define the attribute type
that can be used for chunk_id in a TOAST table when it is initially
created. An update of this parameter has no effect if a TOAST table
already exists under the relation, even on rewrites. It cannot be set
for TOAST relations.
This new option can be set only to "oid" as of this commit. This
feature is extracted as a patch of its own to ease the introduction of a
second mode in a follow-up change, related to the introduction of 8-byte
OIDs in external TOAST pointers.
Bump catalog version, as StdRdOptions has changed.
Author: Michael Paquier <michael@paquier.xyz>
Reviewed-by: Greg Burd <greg@burd.me>
Reviewed-by: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Reviewed-by: Yugo Nagata <nagata@sraoss.co.jp>
Discussion: https://postgr.es/m/af19kUjwjhaoUTLn@paquier.xyz
varatt_external_oid (varatt_external in v19 and older branches) having
no padding is a rule that has been implied for years, as pointers of
this structure may be compared using memcpy().
I doubt that anybody is going to break this rule as varatt_external is
old enough that it could vote in most countries, but this makes varatt.h
more consistent for a follow-up change that introduces a second on-disk
external TOAST pointer, and it's free to have a compile-time rule.
Suggested-by: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Discussion: https://postgr.es/m/CALj2ACVG9cjxr-0jb8PP_Vb7vJvVGBVM_fQ2srt4CpyURpfYhQ@mail.gmail.com
This commit adds a new SQL function able to set the type of a chunk_id
attribute for a TOAST table across binary upgrades. The value is set
when the creation of a TOAST table is required across upgrades, in the
same fashion as the TOAST relation's oid and relfilenode.
Note that this piece currently works only with a TOAST table's chunk_id
set at OIDOID, because it is currently the only type of on-disk external
TOAST pointer supported. This change is required for a follow-up
feature that aims to introduce support for 8-byte OIDs in external TOAST
pointers.
Taking care of this issue separately is less code churn for the next
changes, and perhaps some of the surrounding discussions with support
for different types of external on-disk TOAST pointers will find this
new facility useful.
Bump catalog version.
Author: Michael Paquier <michael@paquier.xyz>
Reviewed-by: Greg Burd <greg@burd.me>
Reviewed-by: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Reviewed-by: Yugo Nagata <nagata@sraoss.co.jp>
Discussion: https://postgr.es/m/af19kUjwjhaoUTLn@paquier.xyz
Those changes were intended to avoid dependencies on the global libc
locale settings by basing identifier casefolding on the default
locale. They introduced an unintentional behavior difference for the
builtin provider with locale C and a single-byte encoding.
Instead of fixing that edge case, just revert the changes. The
identifier casefolding behavior has always been based on the global
libc locale, and clients make an effort to use the same rules. Until
we have a plan to change that, provider-specific identifier
casefolding doesn't make sense.
Reported-by: Noah Misch <noah@leadboat.com>
Discussion: https://postgr.es/m/20260826225510.10.noahmisch@microsoft.com
Backpatch-through: 19
The general nature of this interaction has been understood (at least
by me) since before the original commit, but I failed to explicitly
document it. I also left behind an XXX in the README which I intended
to remove before commit. Fix those things.
Reported-by: Noah Misch <noah@leadboat.com>
Backpatch-through: 19
Discussion: https://postgr.es/m/CA+TgmoYmXy-jiP5qDhqNEiYFEBzQsArO6O2d9E8szNZqi1bePQ@mail.gmail.com
Commit ac58465e06 moved the discussion on clustering from CLUSTER's
refpage to REPACK's, but somehow introduced a mistake regarding when
exactly can the seqscan-and-sort strategy be used with respect to btrees
(spoiler: it was 100% backwards). Fix that, but also restructure and
reword the whole subsection more fully to try and make it more coherent.
Author: Chao Li <li.evan.chao@gmail.com>
Author: Álvaro Herrera <alvherre@kurilemu.de>
Reported-by: Antonin Houska <ah@cybertec.at>
Backpatch-through: 19
Discussion: https://postgr.es/m/69CA7636-0359-42B6-B971-16A036698187@gmail.com
Explain that returning false for TM_SelfModified follows ExecLockRows(),
which skips such tuples. This differs from TM_Updated, which is
unexpected in READ COMMITTED with FIND_LAST_VERSION.
Also remove "fall through to error" from the TM_Updated comment, since
the code calls elog(ERROR) directly.
Reported-by: Amit Kapila <amit.kapila16@gmail.com>
Discussion: https://postgr.es/m/CAA4eK1Lk1DfgiFPL-HccZm-Nm+exg+FcsODH8KPw6BOBuzfHtA@mail.gmail.com
Backpatch-through: 19
At backend level, PgStat_TableCounts currently stores all the per-table
pending counters in one structure. This is now split into two parts,
with the transactional part of the data moved out of PgStat_TableCounts
into its own structure labelled with an "xact" in PgStat_RelationStatus.
While on it, move truncdropped out of the counters and into
PgStat_RelationStatus so as both structs (existing PgStat_TableCounts
and new PgStat_TableCountsXact) contain only counters. The previous
state was slightly confusing: this flag is a state tracking if a
relation has been truncated at backend-level, and not a counter.
Because truncdropped no longer lives inside PgStat_TableCounts, this
commit teaches the flush callback to treat a pending truncate as
sufficient reason to flush, even when both counter structs are otherwise
all zeros.
This refactoring work is in preparation of a different patch that wants
to introduce the concept of transactional stats data flush. Compared to
all the other stats kinds, relations are an exception as they mix
transactional and non-transactional data.
Author: Sami Imseih <samimseih.pg@gmail.com>
Reviewed-by: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Discussion: https://postgr.es/m/aoU32VTkUtpdUewB@paquier.xyz
ri_FastPathCheck() acquired its scan snapshot before opening the
referenced relation. If it then waited for the relation lock in READ
COMMITTED mode, a referenced row committed during the wait would not be
visible to the old snapshot. The check could consequently report a
foreign key violation even though the referenced row existed.
The batched path acquires its scan snapshot after opening the relations,
so it does not have the lock-wait problem. The per-row path remains
reachable during constraint validation and checks re-entered from a
batch flush.
Take the snapshot after opening the referenced relation and reloading
the constraint information. This also agrees with the SPI path, which
acquires the referenced-relation lock before selecting the snapshot
used for the check.
Also make the scan snapshot active for the duration of the check, so
STABLE cast and equality functions use the same snapshot as the index
scan. Otherwise, a STABLE cast can miss rows inserted by earlier AFTER
triggers and cause a valid foreign key to be rejected. Add a
regression test comparing this case with the SPI path.
Finally, also advance the command counter after acquiring the
referenced-relation lock and before reloading the constraint
information, so local invalidations are processed under the lock, as in
the SPI path.
Add an injection-point test that pauses a per-row check before locking
the referenced relation, commits the missing referenced row in another
session, and verifies that validation succeeds. Force per-row
validation by using a role without SELECT privilege on the referenced
table, so the test exercises this path even with batching enabled.
Add isolation-test coverage for the visibility of a referenced row
committed after the referencing transaction has executed an earlier
command. A later command can see such a row in READ COMMITTED, but not
in REPEATABLE READ or SERIALIZABLE.
Reviewed-by: Chao Li <li.evan.chao@gmail.com>
Reviewed-by: Amit Kapila <amit.kapila16@gmail.com>
Discussion: https://postgr.es/m/CA+HiwqEhm+_=bs=2wavAJz-UqC+1KebD31++mapJQQGweE8iQQ@mail.gmail.com
Backpatch-through: 19
pg_stat_force_next_flush() has primarily been used in in-core tests to
force cumulative statistics to be flushed at the end of a transaction
without waiting for the PGSTAT_MIN_INTERVAL throttle, to speed up
things.
It is also useful in tests outside the core tree. Spoiler: I have used
it myself for some out-of-core work for the same reason as the in-core
tests, and would have unlikely known about it without my work on the
core code.
Author: Sami Imseih <samimseih.pg@gmail.com>
Discussion: https://postgr.es/m/aqEONVuLhgBiqHW6@paquier.xyz
Previously, readCommandResponse() stored every column with PQgetvalue(),
which returns an empty string for the SQL NULL value. An empty string
is not a valid value in the pgbench expression language, and when
returned by the server it was also indistinguishable from a NULL.
When receiving a value, pgbench now checks for PQgetisnull() and sets a
NULL value with setNullValue() before saving it. This behavior is more
consistent with \set, making the result saved testable with IS NULL.
This is a quality-of-life improvement, so no backpatch is done.
Author: Shinya Kato <shinya11.kato@gmail.com>
Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Reviewed-by: Shihao Zhong <zhong950419@gmail.com>
Discussion: https://postgr.es/m/CAOzEurT9J9QMmpJt_8pWZCziNYCUHOD=c-hoccOVYJLc_VxVRg@mail.gmail.com
Commits 630706ced0 and 9c8de15969 lost some encoding-specific nuances:
in libc with EUC encodings, or in ICU with any non-UTF8 encoding, we
cannot easily classify characters, so we must assume they are
case-varying. Restore these behaviors.
Previously, ILIKE using an index scan could cause wrong results.
Reported-by: Noah Misch <noah@leadboat.com>
Discussion: https://postgr.es/m/20260826221526.89.noahmisch@microsoft.com
Backpatch-through: 19
pg_stat_progress_cluster translates REPACK into VACUUM FULL when
pg_stat_progress_repack.repack_index_relid is zero, and into CLUSTER
otherwise.
Previously, repack_index_relid was set only when an index scan was used.
When REPACK used a sequential scan and sort to order the table by an index,
the field remained zero even though index ordering had been requested. As a
result, pg_stat_progress_cluster incorrectly reported the command as
VACUUM FULL instead of CLUSTER.
Fix this by setting repack_index_relid to the OID of the requested ordering
index when progress reporting starts, regardless of the scan method. This
allows pg_stat_progress_cluster to correctly report REPACK as CLUSTER when
index ordering is requested.
Also update the documentation to clarify the meaning of the index columns
and how REPACK is translated into CLUSTER or VACUUM FULL.
Suggested-by: Fujii Masao <masao.fujii@gmail.com>
Author: Chao Li <lic@highgo.com>
Reviewed-by: Fujii Masao <masao.fujii@gmail.com>
Reviewed-by: Zsolt Parragi <zsolt.parragi@percona.com>
Discussion: https://postgr.es/m/CAN4CZFMzy2V_wvRCBW5K8=wVCk1-C7nq=8otXO=u+s_1KZHqyA@mail.gmail.com
Backpatch-through: 19
Previously, when another session ran REPACK, CLUSTER, or VACUUM FULL and
the user lacked permission to view its progress details,
pg_stat_progress_cluster incorrectly reported the command as CLUSTER
instead of NULL.
pg_stat_progress_cluster is based on pg_stat_progress_repack and
translates REPACK into CLUSTER or VACUUM FULL depending on whether index
ordering is requested. However, this translation defaulted to CLUSTER,
incorrectly converting NULL commands returned for insufficient privileges
into CLUSTER.
Fix this translation to preserve a NULL command when the user lacks
permission to view the progress details.
Bump catalog version.
Author: Zsolt Parragi <zsolt.parragi@percona.com>
Reviewed-by: Fujii Masao <masao.fujii@gmail.com>
Reviewed-by: Chao Li <lic@highgo.com>
Discussion: https://postgr.es/m/CAN4CZFMzy2V_wvRCBW5K8=wVCk1-C7nq=8otXO=u+s_1KZHqyA@mail.gmail.com
Backpatch-through: 19
The SQL wrappers of the module relied on stringToNode(), that trusts
its input. When specifying node strings that to not match with the
expected ExtensibleNode "TestExtNode", the module would crash.
This adds a simple set of sanity checks to check the input, ensuring
that only an ExtensibleNode "TestExtNode" goes through. Though far
from critical, it's always nice to make these test modules not crash
when used.
Oversight in 0e944fe357.
Author: Zsolt Parragi <zsolt.parragi@percona.com>
Reviewed-by: Greg Burd <greg@burd.me>
Discussion: https://postgr.es/m/CAN4CZFOTQXMZvuPxweO4Fm7+sjrPJUqU-JvYKETLBxuhqgbPjw@mail.gmail.com
A clause connecting the same two EC members can be requested from the
EC machinery in different contexts. When building parameterized index
paths for a relation, generate_implied_equalities_for_column derives
an indexable clause with parent_ec set, to mark it as redundant with
other join clauses derived from the same EC. Meanwhile, at a join
where the relations of a multi-relation EC member first come together,
generate_join_implied_equalities emits a clause equating the same two
members with parent_ec unset, since that clause is the sole enforcer
of its member there and must not be dropped as redundant. Because
create_join_clause treats parent_ec as part of its cache-lookup key,
such requests can yield two different RestrictInfos for the same
condition, carrying different rinfo_serials.
This breaks the assumption that one condition has exactly one serial
number, which create_nestloop_path relies on to drop join clauses that
are already enforced within a parameterized inner path. As a result,
the same condition could be enforced twice at different plan levels,
wasting execution effort and applying the clause's selectivity twice,
underestimating the join's row count.
To fix, make create_join_clause copy the rinfo_serial from an existing
clause that connects the same two members with the opposite parent_ec
marking, so that the two clauses keep their distinct markings but are
recognized as the same condition.
As a result, a duplicate join filter disappears from one of the
existing regression plans (the "cardinal_number" query in join.sql);
the other plan changes are join-order differences between plans of
essentially equal cost.
Author: Richard Guo <guofenglinux@gmail.com>
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/CAMbWs4-3yZeBMGQDpmYRPdV-sL_=6H2AN=MoeneOT+VUk28PPg@mail.gmail.com
Backpatch-through: 19
Commit f836b688f ensured that at most one clone of a join clause is
chosen for a parameterized path, so long as the clone variants are
parse-tree distinct. Two ways remained to end up enforcing the same
condition more than once, which wastes execution effort and applies
the clause's selectivity multiple times, underestimating the row
count.
If a commuting outer join nulls no Var referenced by a particular
qual, the clone variants of that qual are parse-tree identical, and
the incompatible_relids test cannot tell them apart. To fix, enforce
just the first surviving variant, identified by rinfo_serial.
The other case is in get_joinrel_parampathinfo, where the
clause-recovery pass for dropped EquivalenceClasses queries the EC
machinery with a context that overlaps the preceding loop's, so the
derived-clause cache can hand back a clause that that loop already
accepted. To fix, skip any clause that is already present in the
collected list.
Also add assertions verifying that the clauses to be enforced at a
join or within a parameterized path contain no duplicate rinfo_serial,
to catch any remaining or future violations. These assertions are
added in master only, since hard prevention of duplicate clauses is a
new goal and other unfixed cases may remain.
Back-patch to v19, as with commit f836b688f.
Author: Richard Guo <guofenglinux@gmail.com>
Author: Tom Lane <tgl@sss.pgh.pa.us>
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/CAMbWs4_Ezb1Lnj7BqcTQSysC6B5UFZxfhib8xr6rqpu5uqr8dA@mail.gmail.com
Backpatch-through: 19
Commit 0ee83dd4a9 allowed postgres_fdw to push down inner joins between
foreign tables and functions in FROM clauses to the remote server.
However, planning such a join with use_remote_estimate enabled could
cause a segmentation fault. For example, the following query crashed
when ft had use_remote_estimate set to true:
EXPLAIN SELECT * FROM generate_series(1, 3) g JOIN ft ON ft.i = g;
The crash occurred because init_func_stub_fpinfo() copied the connection
information and options from the foreign side to the function-side stub,
but omitted the user mapping, leaving it NULL. When the function was the
outer side of the join, the join inherited this NULL mapping and passed it
to GetConnection() when requesting a remote estimate, causing a
segmentation fault.
Fix this by having init_func_stub_fpinfo() also copy the user mapping
from the foreign side.
Author: Fujii Masao <masao.fujii@gmail.com>
Reviewed-by: Alexander Pyhalov <a.pyhalov@postgrespro.ru>
Discussion: https://postgr.es/m/CAHGQGwGE9xPzCke0C0r4omTkh-hU08aBd2j9oSA0FYmJLCf9jA@mail.gmail.com
This commit adds a new routine called GetNewObjectId8() in varsup.c,
which is able to retrieve a 8-byte OID, based on the next OID retrieved
from the control file. GetNewObjectId() is kept compatible with its
origin, where we still check that the lower 32 bits of the counter do
not wraparound, while handling the FirstNormalObjectId case.
This change is combined with an update of the control file, with
CheckPoint.nextOid enlarged from Oid to Oid8. pg_control_checkpoint(),
pg_resetwal and pg_upgrade are updated to adapt to this change, to make
sure that the value is not truncated due to 32-bit limitations when
used, making an upgrade to a newer version where 8-byte is supported a
transparent operation.
A couple of comments in the xlog redo logic referred to wraparound
issues, which is not the case anymore with this change in place, so
refresh these.
This change is required to be able to support external TOAST pointers
based on Oid8, under discussion, and can be used for other purposes.
Tests are added for pg_upgrade (2^32 value carried across upgrades),
pg_resetwal and pg_control_checkpoint(), hence across all the tools
that need to be covered by this change.
Bump catalog version.
Bump PG_CONTROL_VERSION.
Bump XLOG_PAGE_MAGIC, due to XLOG_NEXTOID.
Author: Michael Paquier <michael@paquier.xyz>
Reviewed-by: Greg Burd <greg@burd.me>
Reviewed-by: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Reviewed-by: Yugo Nagata <nagata@sraoss.co.jp>
Discussion: https://postgr.es/m/af19kUjwjhaoUTLn@paquier.xyz
Without CHECK_FOR_INTERRUPTS(), backends might not respond to query
cancellations in a timely fashion, especially for JOIN_ORDER()
specifications, where matching against each join can take O(n^2) time
in the length of the join order list in degenerate cases.
Recursive functions should check_stack_depth().
Reported-by: Noah Misch <noah@leadboat.com>
Discussion: https://postgr.es/m/CA+TgmoYmXy-jiP5qDhqNEiYFEBzQsArO6O2d9E8szNZqi1bePQ@mail.gmail.com
Backpatch-through: 19
The SIMD path for this command reads ahead via CopyLoadInputBuf()
whenever fewer than sizeof(Vector8) bytes remain in the input
buffer, even if those bytes hold a complete end-of-copy marker.
If the input is a pipe whose writer has sent the marker but not
closed the pipe, that read blocks, and COPY waits for data it will
never use. The scalar loop asks only for the bytes it needs, so
it stops at the marker without reading any further.
To fix, teach CopyLoadInputBuf() to decline a speculative load
when the caller's remaining bytes contain a backslash, which in
text mode might begin such a marker. (CSV mode doesn't treat
\. as special, so it is unaffected.) The SIMD path then hands
those bytes to the scalar loop, as it already does when a load
comes up short. Checking for the marker in the SIMD helper itself
would be more direct, but a call there costs the compiler
registers on every line, which measurably slowed COPY of short
lines in testing.
Oversight in commit e0a3a3fd53.
Author: Nazir Bilal Yavuz <byavuz81@gmail.com>
Reviewed-by: Greg Burd <greg@burd.me>
Discussion: https://postgr.es/m/CAN55FZ1qFb4Yo3-MWeQfGQnu1_Ksx4xcFC2m3hyw8a--%2BeHQYQ%40mail.gmail.com
Backpatch-through: 19
Commit 86f7c82cf1 reported the user catalog table restriction as a
detail, but the other checks in check_concurrent_repack_requirements()
were still a mix: some were hints, and the details repeated
"REPACK (CONCURRENTLY)", which the primary message already names.
Report them as details with the same "This operation ..." wording. The
one actionable hint, ALTER TABLE ... REPLICA IDENTITY USING INDEX,
stays.
Backpatch to v19, where REPACK (CONCURRENTLY) was introduced.
Author: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Suggested-by: Alvaro Herrera <alvherre@kurilemu.de>
Discussion: https://postgr.es/m/apBIFWzHYOaG0auN%40nathan
Backpatch-through: 19
DROP TABLESPACE checked pg_shdepend for dependent objects without
first locking the tablespace. A concurrent command that recorded a
shared dependency on the tablespace right after that check could still
commit, leaving an object whose pg_shdepend entry (or, for a relation,
pg_class.reltablespace) pointed to a tablespace that no longer existed.
Close the race by having DropTableSpace() take an AccessExclusiveLock
on the tablespace before calling checkSharedDependencies(). That
conflicts with the AccessShareLock shdepLockAndCheckObject() takes when
recording a new dependency, so the loser of the race blocks and
rechecks once the winner commits.
That AccessExclusiveLock creates a new deadlock: ALTER TABLESPACE
RENAME/SET and the internal ACL/owner updates in DROP OWNED and
REASSIGN OWNED touched the catalog tuple without locking the
tablespace, risking a lock-order cycle with DROP. Fix by taking an
AccessShareLock first in all four paths, rechecking pg_shdepend after
any wait in the DROP OWNED and REASSIGN OWNED cases. GRANT, REVOKE,
and ALTER TABLESPACE ... OWNER TO already lock the object.
Add isolation tests covering both orderings of the original race
(dependency-first and drop-first) and all four previously-unlocked
update paths.
Author: Ayush Tiwari <ayushtiwari.slg01@gmail.com>
Reviewed-by: Andrew Dunstan <andrew@dunslane.net>
Discussion: https://postgr.es/m/CAJTYsWXjAQFnGKzXsht3XK8KHhyhontwFJw4JAHsUSfs_ptR4g@mail.gmail.com
Backpatch-through: 14
Previously, running REPACK (CONCURRENTLY) on a materialized view failed
with a confusing "has no identity index" error.
REPACK (CONCURRENTLY) replays changes decoded from WAL, but a
materialized view produces none, since it cannot be modified by DML and
REFRESH rewrites it through a transient heap that is not logically
decoded.
Fix this by reporting a clear error for materialized views upfront,
alongside the existing checks for system catalogs and TOAST tables, and
document the limitation.
Author: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Reviewed-by: Kiran Kaki <itskkpg@gmail.com>
Backpatch-through: 19
Discussion: https://postgr.es/m/CALj2ACXPvy4_LkQ0nHKgbZjAGK9fkDqRYUvXA3hgwxPm_X7Eng@mail.gmail.com
REPACK (CONCURRENTLY) is not an MVCC-safe operation; it doesn't
preserve the visibility information, which logical decoding needs
because it reads user catalog tables under a historic snapshot.
Disallow REPACK (CONCURRENTLY) on user catalog tables. Removing this
check requires making it MVCC-safe and logical rewrite mappings.
Backpatch to v19, where REPACK (CONCURRENTLY) was introduced.
Reported-by: Nathan Bossart <nathandbossart@gmail.com>
Author: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Reviewed-by: Antonin Houska <ah@cybertec.at>
Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
Reviewed-by: Alvaro Herrera <alvherre@kurilemu.de>
Discussion: https://postgr.es/m/apBIFWzHYOaG0auN%40nathan
Backpatch-through: 19
REPACK (CONCURRENTLY) launches a background worker to decode changes
made while the table is being rewritten. The worker connects as the
table owner but bypassed the LOGIN check only, so CONNECT was still
checked against a role that need not have it, and the command could
fail with "permission denied for database".
Pass BGWORKER_BYPASS_ALLOWCONN as well, as we do for parallel
workers. That is safe because the leader already checked the invoking
user's privileges on the table before starting the worker.
Reported-by: Nathan Bossart <nathandbossart@gmail.com>
Author: Matheus Alcantara <mths.dev@pm.me>
Reviewed-by: Nathan Bossart <nathandbossart@gmail.com>
Reviewed-by: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
Reviewed-by: Álvaro Herrera <alvherre@kurilemu.de>
Discussion: https://postgr.es/m/apBbzFd_EYAfHV45@nathan
Backpatch-through: 19
Commit 266543a made initial table synchronization use "COPY table TO"
for a partitioned table, but decided that from the relation kind
alone. A partitioned table is only accepted there since v19, so a v19
subscriber replicating from an older publisher with
publish_via_partition_root failed at table sync.
Check the publisher version, and fall back to the "COPY (SELECT ...)
TO" variant otherwise.
Backpatch to v19, where 266543a changed the initial table
synchronization.
Reported-by: Noah Misch <noah@leadboat.com>
Reviewed-by: Noah Misch <noah@leadboat.com>
Discussion: https://postgr.es/m/20260905040020.57.noahmisch@microsoft.com
Backpatch-through: 19
There's not really any good reason to store an empty advice string
in an advice stash, but it's entirely possible that some people might
do it by accident. If they do that without this fix, restarting the
system will result in failure while reloading pg_stash_advice.tsv.
The root cause is that I (rhaas) did not think clearly enough when
defining pgsa_next_tsv_field(). After returning each field on the
line, it set *cursor to the start of the next field, and after
returning all of them, it set *cursor to the trailing NUL byte. But
when the last field on the line is empty, this is ambiguous: if we're
pointing at the trailing NUL byte, it could be either because it's the
start of the zero-length field or because we're out of fields. To
remove the ambiguity, this commit decides that *cursor will instead
be set to NULL when there are no more fields.
Reported-by: Noah Misch <noah@leadboat.com>
Discussion: https://postgr.es/m/20260827171830.68.noahmisch@microsoft.com
Backpatch-through: 19
ExecInitModifyTable() evaluated ForPortionOfExpr.targetRange
unconditionally at executor start-up, with no EXEC_FLAG_EXPLAIN_ONLY
guard. This caused three problems:
- EXPLAIN (GENERIC_PLAN) failed with "no value found for parameter 1".
- Plain EXPLAIN with FOR PORTION OF (null) raised "FOR PORTION OF
target must not be null".
- Plain EXPLAIN ran user functions appearing in the target a second
time, beyond the one evaluation the planner already does for
selectivity estimation.
Now we skip the evaluation and the accompanying null check in
explain-only mode.
Reported-by: Noah Misch <noah@leadboat.com>
Author: Paul A. Jungwirth <pj@illuminatedcomputing.com>
Discussion: https://www.postgresql.org/message-id/flat/CA%2BrenyXci9khYXJ8uSwzF30DVw%2Bzc4QwGb1RdsuOu_YtrFuy1g%40mail.gmail.com
The current implementation doesn't support it correctly, leading to
mysterious warnings and some worse problems. Have the combination throw
an error.
This is intended as a temporary stopgap until we can implement it in a
future release, as it's a useful feature.
Author: Fujii Masao <masao.fujii@gmail.com>
Author: Antonin Houska <ah@cybertec.at>
Author: Osama Abdul Qader <osamaabdulqader.cs@gmail.com>
Reported-by: Nathan Bossart <nathandbossart@gmail.com>
Backpatch-through: 19
Discussion: https://postgr.es/m/apBCEl4hBHlq0axm@nathan
This reverts commit c06d1a4ba6.
The commit assumed that if the FSM code tolerates torn pages, everything else
does so. Readers that do not tolerate that include RelationCopyStorage(),
used by ALTER TABLE ... SET TABLESPACE for every fork, the read stream in
RelationCopyStorageUsingBuffer() used by CREATE DATABASE ... STRATEGY =
wal_log, and, most awkwardly, the checksum verification in base backups and
pg_checksums.
Reported-by: Noah Misch <noah@leadboat.com>
Discussion: https://postgr.es/m/20260901211837.f6.noahmisch%40microsoft.com
Backpatch-through: 14
Reviewed-by: Andrey Borodin <x4mmm@yandex-team.ru>
FOR PORTION OF valid_at FROM $1 TO $2 fails with "could not determine data
type of parameter $1", so using PREPARE requires users to say $1::date
etc. But transformForPortionOfClause() already coerces the bounds, since
it builds a constructor for the targeted range. We just need to store the
coerced expressions in targetFrom and targetTo.
This also improves pg_get_ruledef(), which now renders an untyped NULL
bound as NULL::date instead of NULL::unknown.
Reported-by: Noah Misch <noah@leadboat.com>
Author: Paul A. Jungwirth <pj@illuminatedcomputing.com>
Discussion: https://www.postgresql.org/message-id/flat/CA%2BrenyW1uJp1dw%3DiK7MsWfbbZf-dKP_4JLoQ65wj%3Dfry35FjCw%40mail.gmail.com
On slow machines, this test can take long enough to generate WAL that a
time-based checkpoint occurs before the primary is stopped. If the
checkpoint record is written to the tail WAL segment that the test later
removes, a standby initialized from the resulting backup tries to read
the missing checkpoint record and fails with a PANIC during startup.
This caused the test to fail on buildfarm member skink.
Fix this by setting checkpoint_timeout high enough to prevent unrelated
checkpoints during the test. This follows the approach used by other
recovery tests, such as 043_no_contrecord_switch.pl, that depend on a
specific WAL layout.
Backpatch to all supported versions.
Reported-by: Alexander Lakhin <exclusion@gmail.com>
Author: Fujii Masao <masao.fujii@gmail.com>
Discussion: https://postgr.es/m/CAHGQGwFajfjKhvNSTkTcE-wFn30p_A0_i1cvk-Q=FJhDr_5cXA@mail.gmail.com
Discussion: https://postgr.es/m/9ffdb19a-7a89-424e-925a-dd981c37f0ba@gmail.com
Backpatch-through: 14
This fixes issues with the same variable name being used in the same
function for two different variables. (In some cases, one of the uses
is a function argument.) The fix is in most cases to rename one or
both of them. The individual choice depends on conventions in the
surrounding code. Often, making both variable names more specific is
the best choice. In a few cases, the fix is to move one of the
variables to a lower scope, so that it no longer conflicts with the
other.
Since these are all local-variable conflicts, the changes in different
files in this patch are independent of each other.
These issues correspond to the gcc warning option -Wshadow=local,
which is not currently used but could be activated after this.
Reviewed-by: Peter Geoghegan <pg@bowt.ie>
Reviewed-by: Chao Li <li.evan.chao@gmail.com>
Discussion: https://www.postgresql.org/message-id/flat/84b6f128-91f5-480e-8a9e-7d0e8f538cea%40eisentraut.org
Commit 6168c65ddc added a test verifying that the conflict detection
slot's xmin regresses to the horizon pinned by an in-progress transaction
in the database of a newly added subscription that has retain_dead_tuples
enabled. However, it used query_until(), which can return before starting
a transaction, allowing the test to proceed without a pinned xid.
Use query_safe() so the transaction setup completes synchronously and the
xid is assigned before the xmin checks run.
Per buildfarm.
Author: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Discussion: https://postgr.es/m/OS9PR01MB12149257A97EE1BD3296F6AEBF5B12@OS9PR01MB12149.jpnprd01.prod.outlook.com
Backpatch-through: 19
pgstat_init_entry() can fail after its hash entry has been inserted, in
which case the caller has to remove the entry, as done by
pgstat_get_entry_ref() since 8191e0c16a. pgstat_read_statsfile() was
not exactly doing that, leaving an entry that could still point to an
InvalidDsaPointer.
There are no consequences here, even after 8191e0c16a. The stats are
loaded from disk by the startup process, where an ERROR gets promoted to
a FATAL, cleaning up shared memory. So this is mostly an argument of
consistency with the normal entry creation path, especially if someone
decides to downgrade this ERROR to a WARNING in pgstat_read_statsfile().
Author: Grigorev Jurij <ju.grigorev@ftdata.ru>
Discussion: https://postgr.es/m/d55ecaf911844d53bd0a931751dce582@localhost.localdomain
