Commit 6168c65ddc added a test verifying that the conflict detection
slot's xmin regresses to the horizon pinned by an in-progress transaction
in the database of a newly added subscription that has retain_dead_tuples
enabled. However, it used query_until(), which can return before starting
a transaction, allowing the test to proceed without a pinned xid.
Use query_safe() so the transaction setup completes synchronously and the
xid is assigned before the xmin checks run.
Per buildfarm.
Author: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Discussion: https://postgr.es/m/OS9PR01MB12149257A97EE1BD3296F6AEBF5B12@OS9PR01MB12149.jpnprd01.prod.outlook.com
Backpatch-through: 19
pgstat_init_entry() can fail after its hash entry has been inserted, in
which case the caller has to remove the entry, as done by
pgstat_get_entry_ref() since 8191e0c16a. pgstat_read_statsfile() was
not exactly doing that, leaving an entry that could still point to an
InvalidDsaPointer.
There are no consequences here, even after 8191e0c16a. The stats are
loaded from disk by the startup process, where an ERROR gets promoted to
a FATAL, cleaning up shared memory. So this is mostly an argument of
consistency with the normal entry creation path, especially if someone
decides to downgrade this ERROR to a WARNING in pgstat_read_statsfile().
Author: Grigorev Jurij <ju.grigorev@ftdata.ru>
Discussion: https://postgr.es/m/d55ecaf911844d53bd0a931751dce582@localhost.localdomain
This has been extracted from a larger patch set, for a scenario that I
have managed to break while working on a different change. The case of
-o with a negative value had no coverage.
Author: Michael Paquier <michael@paquier.xyz>
Discussion: https://postgr.es/m/af19kUjwjhaoUTLn@paquier.xyz
When creating a nestloop plan for a partitionwise child join, the
outer rel's relids are child relids, but PlaceHolderInfo.ph_eval_at is
always expressed in terms of the topmost parent rels. As a result,
replace_nestloop_params() and identify_current_nestloop_params()
failed to recognize that a PlaceHolderVar evaluated at the outer child
rel can be supplied as a nestloop param. Instead, the Vars within the
PHV's expression were replaced with params, but the outer child rel
emits only the PHV, not those bare Vars, leading to "variable not
found in subplan target list" errors from setrefs.c.
To fix, also include the outer rel's top parent relids in the relid
set used for these checks, so that ph_eval_at comparisons are done in
terms of parent rels while Var checks continue to work in terms of
child rels.
On v18 and later, the required-outer set passed to
identify_current_nestloop_params() has the same problem: it is in
terms of child rels once a parameterized child join path has been
reparameterized by an upper child join. With the above fix in place,
a PlaceHolderVar that depends on both the outer rel and the parameter
source becomes a single NestLoopParam, and that param was never
claimed by any nestloop node, leading to "failed to assign all
NestLoopParams to plan nodes" errors. To fix, also include the top
parents of any child rels in that set. Older branches lack this code
path, so they receive only the first change.
Back-patch to all supported branches.
Bug: #19653
Reported-by: Annie <10215501441@stu.ecnu.edu.cn>
Author: Richard Guo <guofenglinux@gmail.com>
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/19653-9352cc6ba17b662f@postgresql.org
Backpatch-through: 14
A few integer-valued GUCs representing Unix file permission masks
are customarily written and shown in octal. However, we implemented
the "shown" part via show_hook functions, which is problematic
because it only affects display of the current value. It's confusing
that, for example, the pg_settings view renders the current value in
octal but the min, max, reset, and boot values in decimal.
To improve matters, get rid of these custom show_hooks in favor of
inventing a per-GUC flag GUC_SHOW_IN_OCTAL, which can be inspected in
appropriate places. I've implemented that in places that respond to
units flags, but not in places that don't, such as GetConfigOption().
It's tempting to consider going further, in particular adjusting
input parsing so that we read the values of these variables in
octal even without a leading zero. I've refrained from doing so
here because I'm afraid that it'd break more usages than it fixes,
but perhaps there's a case to be made for that.
Bug: #19540
Reported-by: Jobin Augustine <jobinau@gmail.com>
Author: Tom Lane <tgl@sss.pgh.pa.us>
Reviewed-by: Fabrízio de Royes Mello <fabriziomello@gmail.com>
Reviewed-by: Tatsuya Kawata <kawatatatsuya0913@gmail.com>
Reviewed-by: Rui Zhao <zhaorui126@gmail.com>
Reviewed-by: Andrey Borodin <x4mmm@yandex-team.ru>
Discussion: https://postgr.es/m/3006313.1782851535@sss.pgh.pa.us
List of commits reverted:
2f094e7ac6 SQL Property Graph Queries (SQL/PGQ)
cd8844e7db Make some tests more stable by adding more explicit ordering
182cdf5aea pg_dump: Add appropriate version check
c9babbc881 Dump labels in reproducible order
040a56be4b Cleanup users and roles in graph_table_rls test
5282bf535e Fix some typos and make small stylistic improvements
720f0f89d6 Reject consecutive element patterns of same kind
288ae96872 Add a graph pattern variable only once
c5b3253b8a Property references are preferred over regular column references
a0dd0702e4 Fix cross variable references in graph pattern causing segfault
2ff289d039 Check for stack overflow when rewriting graph queries
9082680c34 Fix typos and grammar in graph table rewrite code
ac3bcc041c Fix collation of expressions in GRAPH_TABLE COLUMNS clause
9d2979dd68 pg_get_viewdef() and lateral references in COLUMNS of GRAPH_TABLE
891a57c739 Do not define type for a property graph
dc9e7c9ed9 Handle nodes that may appear in GraphPattern expression trees
f6edd8ed70 Add ORDER BY to test query to stabilize test
1190f858ea doc: Small synopsis wording change for consistency
d0eac3cafb Make spelling consistent
6827de95ee Simplify code in objectaddress.c for some property graph objects
5778fb3eaf Fix typo in error message
72498a8698 Handle element label and label property objects in object address functions
4cb2a9863d Fix LATERAL references in GRAPH_TABLE with multi-label pattern
9d8cdcbe0c Record dependencies on graph labels and properties
2a7e95b659 Readable identity strings for property graph objects
7afa11feca Prevent dropping the last label from a property graph element
96418a6da9 Fix handling of dropping a property not associated with the given label
efd7d8d7d4 Resolve unknown-type literals in GRAPH_TABLE COLUMNS
e0ff7fd9aa Make property graph object descriptions better translatable
0e4f0827f6 Fix properties orphaned by dropping a label
73dfe79fd6 Remove apparent support for SECURITY LABEL ON PROPERTY GRAPH
592de8bd21 Fix pg_dump ACL minimization for PROPERTY GRAPH.
16a4b3ef8e Fix replace_property_refs() ignoring the root of expression tree
57f93af36f Resolve unknown-type literals in property expressions
2e6578292a Prohibit locking clauses on GRAPH_TABLE
2521f9e75e doc: Fix SQL quoting in example
0316593146 Restore the ability to use | and -> as prefix operators.
780fa49746 Fix missing space before WHERE in GRAPH_TABLE deparse
fa62d4f6ea Prohibit GRANT ... ON TABLE on a property graph
63c3bbb39f doc: Add PROPERTY GRAPH to the access privilege tables
0a51a06cdb More tab-completion for DROP PROPERTY GRAPH
f585671055 Disallow aggregates, window functions, and SRFs in GRAPH_TABLE COLUMNS
9b917a9311 Report duplicate property and label names with a proper error
a82e33a83d Make generate_queries_for_path_pattern_recurse() interruptible
3c982c9bf1 Add test for view referencing labels shared by vertex and edge tables
8646d1d590 Fix inferred property graph keys with INCLUDE columns
0417bf5296 Fix pg_event_trigger_ddl_commands for GRANT ON PROPERTY GRAPH
a20d3e35f2 doc: reformat GRAPH_TABLE examples
792094a5ce Coerce GRAPH_TABLE pattern WHERE clauses to boolean
plus typedefs.list changes.
Discussion: https://www.postgresql.org/message-id/CAAKRu_bEtjWYWhYxSo0o_t3DaZYRQd5PkC0abA0g9Mp4%2BovH0w%40mail.gmail.com
Since the retain_dead_tuples feature uses a single cluster-wide
replication slot across all subscriptions, adding a new subscription on a
database with an older active transaction can require
conflict_detection_slot.xmin to move backward to avoid missing dead tuples
needed for update_deleted conflict detection.
Previously, if a newly created subscription had an older active
transaction than the slot's current xmin, the slot update occurred during
worker apply, leading to an assertion failure.
Fix this by having the launcher track the set of databases with active
tuple-retaining subscriptions. When a database joins this set (either via
a new subscription or when retention restarts), the launcher resets the
slot's xmin to the cluster-wide safe decoding horizon before starting any
workers. Because this horizon accounts for all active transactions across
the entire cluster, it serves as a safe seed without requiring retroactive
backward movement by individual apply workers.
In passing, update the retain_dead_tuples documentation to clarify two
operational boundary cases:
(a) Conflict data continues to accumulate even when a subscription is
disabled or its apply worker is stopped.
(b) track_commit_timestamp must be enabled for conflict detection to
function properly.
Reported-by: Nisha Moond <nisha.moond412@gmail.com>
Author: Zhijie Hou <houzj.fnst@fujitsu.com>
Author: Amit Kapila <amit.kapila16@gmail.com>
Reviewed-by: Amit Kapila <amit.kapila16@gmail.com>
Reviewed-by: Nisha Moond <nisha.moond412@gmail.com>
Reviewed-by: shveta malik <shveta.malik@gmail.com>
Reviewed-by: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Discussion: https://postgr.es/m/TY4PR01MB177182F547A62FC2666EC04EC94B72@TY4PR01MB17718.jpnprd01.prod.outlook.com
Backpatch-through: 19, where it was introduced
When outer-join identity 3 permits a join to commute with lower outer
joins, we generate multiple clone versions of its join clause, of
which only one should be applied in any given plan. When building a
join relation's restriction list, subbuild_joinrel_restrictlist
selects the appropriate clone by checking required_relids and
incompatible_relids, but no such selection was made for movable join
clauses pushed down into a parameterized path. As a result, a
parameterized scan or join could enforce more than one clone of the
same condition, shown by EXPLAIN as a duplicate qual. This wastes
effort evaluating the same condition repeatedly. What is worse, it
applies the clause's selectivity multiple times, underestimating the
result's row count.
To fix, in get_baserel_parampathinfo and get_joinrel_parampathinfo,
skip a clone clause if any outer join listed in its
incompatible_relids has already been computed below the point of
evaluation.
Back-patch to v19, but no further. While this bug goes back to v16,
applying the fix in released branches risks destabilizing plans that
users are currently happy with, and there have been no field
complaints.
Author: Richard Guo <guofenglinux@gmail.com>
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Reviewed-by: Tender Wang <tndrwang@gmail.com>
Discussion: https://postgr.es/m/CAMbWs4-g5TFF5k=jfe3JVWqtv_cCTOMmQRUp0_E8fApgqcqBng@mail.gmail.com
Backpatch-through: 19
When neither BKPIMAGE_HAS_HOLE nor BKPIMAGE_COMPRESSED is set,
DecodeXLogRecord() checks that bimg_len needs to be equal to BLCKSZ.
However, the associated error message reported the data length
associated to a block (DecodedBkpBlock.data_len), and not the length of
the block (DecodedBkpBlock.bimg_len).
Oversight in 57aa5b2bb1, probably due to some copy-paste from the
surroundings.
Author: Ayush Tiwari <ayushtiwari.slg01@gmail.com>
Discussion: https://postgr.es/m/CAJTYsWVz9ymE7aMt2ZiQvs1bxNvhjfTf+XaJEsdFw7MT5dvtYQ@mail.gmail.com
Backpatch-through: 14
A user-supplied or modified backup manifest could contain an invalid WAL
range LSN that was nevertheless accepted by the manifest parser. For
example, an LSN component wider than 32 bits could be silently truncated,
and trailing characters after an otherwise valid LSN could be ignored.
This could cause pg_verifybackup, pg_combinebackup, and incremental
backup validation to check WAL availability against a different location
from the one specified in the manifest. PostgreSQL-generated manifests
should not contain such values, but tools consuming manifests should
reject invalid input rather than reinterpret it.
Fix this by using the common pg_parse_lsn() helper to parse LSNs in
backup manifests. It requires the same syntax as the pg_lsn input
function.
Author: Ayush Tiwari <ayushtiwari.slg01@gmail.com>
Reviewed-by: Yuefei Shi <shiyuefei1004@gmail.com>
Reviewed-by: Fujii Masao <masao.fujii@gmail.com>
Discussion: https://postgr.es/m/CAJTYsWXieRHb-ooV2XfjHAtBsS7+P5La_-o8-Cqi15CNDhh9hQ@mail.gmail.com
When analyzing a foreign table pointing to a partitioned or inherited
table on the remote server, this method just imported relpages and
reltuples for the parent table from the remote pg_class, causing a
discrepancy or an error in results compared to the sampling method.
* In the partitioning case, while the sampling method sets relpages=0,
this method set relpages=-1. (For reltuples, the results of both
methods match.)
* In the inheritance case, while the sampling method sets
reltuples=(# of rows in the parent and child tables), this method set
reltuples=(# of rows in the parent table). (For relpages, the results
of both methods match.)
As relpages and reltuples are used for costing foreign paths, these
would lead to incorrect plans or plan changes depending on the method.
To fix, do the following:
* For the partitioning case, modify postgres_fdw to set relpages=0 to
match the sampling method.
* For the inheritance case, we could also modify it to set the correct
reltuples value by fetching reltuples for the child tables as well,
but 1) the change isn't small, and 2) table inheritance hasn't been
used that much these days, so fix by disabling this method.
While at it, adjust comments and docs slightly.
Oversight in commit 28972b6fc.
Author: Etsuro Fujita <etsuro.fujita@gmail.com>
Discussion: https://postgr.es/m/CAPmGK173b2Xvyoeq%3DY7L09H%2BajgMgxC2fYe%3D7CCunZz8tGTfFA%40mail.gmail.com
Backpatch-through: 19
A patch has been proposed to refactor the table-level statistics to
treat the non-transactional and transactional parts as two separate
entities, in the scope of a larger piece of work to introduce
transactional flushes.
This commit adds a test for a pattern that has proved to be problematic
while splitting the relation stats data, and where we had no coverage
yet: an entry marked as truncdropped resets the live/dead counters in
the flush callback, leaving any other data intact. The code path
resetting the live/dead counters was covered, but nothing in the
regression tests checked the stats views for it.
Extracted from a larger patch by the same author.
Author: Sami Imseih <samimseih.pg@gmail.com>
Reviewed-by: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Discussion: https://postgr.es/m/aoU32VTkUtpdUewB@paquier.xyz
The tests added by 88aa6ca138 could fail on Windows, due to the fact
that pg_ctl is not able to compare the PID stored in postmaster.pid with
the postmaster that has been spawned. The postmaster may not be seen as
running, and could fail the live check.
Note that pg_ctl falls back on checking the start time recorded in
postmaster.pid, with a lag of 2 seconds. So, outside the possibility of
fixing pg_ctl so as the PID file checks become more reliable, an
alternative solution would be to use the same trick as the pg_ctl test
001_start_stop.pl, with a hardcoded sleep of *3* seconds, at least. I
cannot get excited about spreading this pattern, though, so skip it is
as we do not lose any coverage.
Per report from buildfarm members fairywren.
Discussion: https://postgr.es/m/apphYnTYS1WtLkKU@paquier.xyz
If a LATERAL subquery references a PlaceHolderVar that must be
evaluated at an outer join, the subquery's lateral_relids include that
outer join's relid, since the lateral parameter is the outer join's
output. join_is_legal() verified that the rels named by a proposed
join's minimum parameterization could still be joined to from outside,
but it did not consider such outer-join relids. As a result, when
identity 3 permitted a commuted join order, we could approve a join
that includes part of that outer join's required input, even though
the outer join could then be completed only above the proposed join,
leaving the lateral parameter forever unsatisfiable.
In assert-enabled builds this tripped the backstop Assert in
try_nestloop_path(). In production builds the bogus join generated
only paths whose parameterization can never be satisfied, so they
could never appear in a complete plan. This not only wasted planning
effort, but also could mislead the clauseless-join heuristics into
thinking that legality of this join means that some other join rel
need not be formed, and that could lead to failure to find any plan at
all.
Such joins used to be rejected by the have_dangerous_phv() check in
join_is_legal(), which refused any join where a PlaceHolderVar's eval
set overlapped the other side of the join without being contained in
it. Commit a16ef313f removed that restriction, exposing this
oversight.
To fix, teach join_is_legal() to reject a proposed join if its minimum
parameterization includes an outer-join relid whose outer join cannot
be formed strictly outside the join, that is, when any rel needed to
form that outer join must become part of the proposed join's own join
tree.
Back-patch to v18, where commit a16ef313f removed the
have_dangerous_phv() restriction.
Author: Richard Guo <guofenglinux@gmail.com>
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Reviewed-by: Tender Wang <tndrwang@gmail.com>
Discussion: https://postgr.es/m/CAMbWs49ZveAYx9T5b0=gxn0pnvqq=MbN-Z4KLbLAE+kH06mWVA@mail.gmail.com
Backpatch-through: 18
This refactoring eases a little bit a follow-up change to add support
for 8-byte TOAST value IDs, as the maximum chunk size allowed for a
single chunk of TOASTed data depends on the size of varatt_external, its
value ID being part of it.
It is worth noting that TOAST_MAX_CHUNK_SIZE resolves as a uintptr_t, so
the previous calculations were in 64-bit unsigned arithmetic. These are
now done in int32, which still works the same way due to an external
TOAST value being bounded to 1GB.
Author: Michael Paquier <michael@paquier.xyz>
Reviewed-by: Yugo Nagata <nagata@sraoss.co.jp>
Reviewed-by: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Discussion: https://postgr.es/m/CALj2ACVnO6jjs5aRx3dSOBy9LV7=iakPBR1eOvoeB8rL8Q47zg@mail.gmail.com
Add test_indexscan, a new test module that provides an SQL-callable
function that performs a full index scan using whatever snapshot type
the caller asks for. Also add a set of tests that verify and document a
variety of behaviors associated with some specific snapshot type.
Preparation for an upcoming commit that will move index scan heap
fetches behind a new slot-based table AM interface. Aside from testing
non-MVCC snapshot types, these new tests can catch bugs where an index
scan incorrectly maintains xs_heap_continue during HOT chain traversal.
Such bugs are unlikely to affect MVCC scans, which makes it hard to
write tests that detect them without relying on a custom test module.
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-by: Tomas Vondra <tomas@vondra.me>
Discussion: https://postgr.es/m/CAH2-Wzn%2BC%3DmAMv9aW3Skfh80JPpHKT3yM%3DDYwkrrhYyG2f%2B_vg%40mail.gmail.com
Compile varbit.c with -ftree-vectorize where available, and adjust
the loops in bit_and(), bit_or(), bitxor(), and bitnot() so that
they are amenable to being auto-vectorized. (Mainly, that involves
reading the byte count into a local variable before the loop. As
written, the bound is recomputed from the varlena header on every
iteration, because the compiler must assume that the loop might
modify the header.)
Reviewed-by: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-by: Chao Li <li.evan.chao@gmail.com>
Discussion: https://postgr.es/m/aphxrw6NkYkLMtZN%40nathan
Add tests that verify the exact values of the idx_tup_read and
idx_tup_fetch counters in pg_stat_all_indexes. There was no existing
coverage of pg_stat_all_indexes, beyond some counter reset tests;
idx_tup_read and idx_tup_fetch had no tests at all.
This is preparation for an upcoming commit that will move index scan
heap fetches behind a new slot-based table AM interface, relocating the
code that maintains these counters.
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-by: Tomas Vondra <tomas@vondra.me>
Discussion: https://postgr.es/m/CAH2-Wz%3DV0V5WGuo85D-OF%2BTM_CxU3q%3DMMkY-ebY0K8xc-9Xykw%40mail.gmail.com
Presently, this field stores the buffer's index, i.e., the Buffer
minus one, and it uses -1 to mean "not waiting." The callers
subtract one before storing the value and add it back after reading
it. This commit changes the field to store the Buffer itself and
to use InvalidBuffer (0) as the sentinel, which gets rid of that
arithmetic.
This is preparatory work for a follow-up commit that will convert
the variable to an unsigned atomic, for which a negative sentinel
makes little sense.
Reviewed-by: Peter Eisentraut <peter@eisentraut.org>
Reviewed-by: Andres Freund <andres@anarazel.de>
Reviewed-by: Zsolt Parragi <zsolt.parragi@percona.com>
Discussion: https://postgr.es/m/alAJeRRzehDjLaF1%40nathan
nodeIndexonlyscan.c has long carried a detailed explanation of why it is
safe for index-only scans to test a visibility map bit without locking
the visibility map buffer, despite the possibility of reading a stale
value. Move that explanation into visibilitymap.c.
This is preparation for an upcoming commit that will move VM checks
needed by index-only scans out of the executor and into heapam.
Author: Peter Geoghegan <pg@bowt.ie>
Suggested-by: Andres Freund <andres@anarazel.de>
Reviewed-by: Andres Freund <andres@anarazel.de>
Reviewed-by: Tomas Vondra <tomas@vondra.me>
Discussion: https://postgr.es/m/t6mtqbv2mbfhjni4bvwdgoecppjmxvbyfwl6utovzv76xc2672%40k3o5ryevaeqv
The retain_dead_tuples feature waits for concurrent transactions to be
applied before advancing conflict_detection_slot.xmin. It gets the WAL
write position from the publisher and waits until the apply remote
position passes it. This ensures that dead tuples, commit timestamps, and
origins needed for conflict detection (update_deleted,
update_origin_differs, and delete_origin_differs) are retained while
those transactions are applied.
However, with asynchronous commit enabled on the publisher, a committed
transaction may not have updated the WAL write position yet. As a result,
the reported WAL position can lag behind the latest committed transaction.
This can cause conflict_detection_slot.xmin to advance prematurely,
allowing dead tuples needed for conflict detection to be removed before
subsequent asynchronously committed transactions are applied.
Report the end of the last inserted WAL record instead. A transaction
inserts its commit record before it is marked committed, so the insert
position cannot be behind a transaction that has already committed.
Transactions that have entered the commit phase but have not yet inserted
their commit record are still accounted for by the in-commit transaction
ID reported in the same message.
In addition to above, add missing trailing period to update_deleted
conflict detail messages which are added for the same feature.
Reported-by: Nisha Moond <nisha.moond412@gmail.com>
Author: Zhijie Hou <houzj.fnst@fujitsu.com>
Reviewed-by: Amit Kapila <amit.kapila16@gmail.com>
Discussion: https://postgr.es/m/TY4PR01MB177182F547A62FC2666EC04EC94B72@TY4PR01MB17718.jpnprd01.prod.outlook.com
Backpatch-through: 19, where it was introduced
4fff78f009 has added coverage for pg_upgrade --check, but only after
stopping the source server. This adds coverage for the case where a
source server is still running, checking for port patterns and a success
case where the old server still runs.
The chdir() required for VPATH builds needs to be moved slightly
earlier, to adapt to the generation of the logs.
Author: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Reviewed-by: Ewan Young <kdbase.hack@gmail.com>
Discussion: https://postgr.es/m/apU4/hmRv/4gv20W@bdtpg
This reverts commit cc9a8eb112, after more discussion with Etsuro
Fujita. We are still not sure about the interface that should be used
for the portion related to the statistics import, and what should be
exposed for FDW code that may want to rely on these two functions:
- import_attribute_statistics()
- import_relation_statistics()
Discussion: https://postgr.es/m/CAPmGK15HUuNvEZk4b+K_x57+sg31B9LDwB7DBgVNaCpLnafhPA@mail.gmail.com
Previously, a session that had created temporary objects could hang during
exit if its default transaction mode was SERIALIZABLE READ ONLY DEFERRABLE
and a concurrent serializable transaction was prepared. During backend
exit, temporary-relation cleanup pushed a regular transaction snapshot,
which honored the session's default settings and could wait for a safe
serializable snapshot. So, if the conflicting transaction was prepared,
this wait could last indefinitely.
The wait occurred while the backend was already exiting and interrupts
were held off, so even pg_terminate_backend() could not cancel it. The
backend therefore remained until the prepared transaction was resolved.
Temporary-relation cleanup only needs an active MVCC snapshot for fetching
TOAST data from catalog tuples while dropping temporary objects. It does
not need a transaction snapshot affected by the user's default isolation
settings. So, use a catalog snapshot instead, which provides the needed
protection without entering the deferrable safe-snapshot wait.
Backpatch to all supported versions.
Bug: #19441
Reported-by: Alexander Lakhin <exclusion@gmail.com>
Author: Andrey Rachitskiy <pl0h0yp1@gmail.com>
Reviewed-by: Andrey Borodin <x4mmm@yandex-team.ru>
Reviewed-by: Fujii Masao <masao.fujii@gmail.com>
Discussion: https://postgr.es/m/19441-ec29f3b1363b4a68@postgresql.org
Backpatch-through: 14
The buildfarm member akepa reported a failure in the
031_recovery_conflict.pl test.
The test checked pg_stat_database_conflicts immediately after detecting
a recovery conflict in the standby log. However, the conflict counter is
flushed by the canceled backend during backend exit, so WAL replay
completion and the log message did not guarantee that the updated
statistics are visible yet. So, previously, the test could see a conflict
counter of 0 even though the conflict had already occurred, triggering
the test failure.
Fix this by polling for the expected conflict counter instead of reading
it only once, handling the asynchronous pgstats update.
Per buildfarm member akepa.
Backpatch to v17, where this test is enabled and has the same race.
Author: Fujii Masao <masao.fujii@gmail.com>
Reviewed-by: Ayush Tiwari <ayushtiwari.slg01@gmail.com>
Reviewed-by: Nazir Bilal Yavuz <byavuz81@gmail.com>
Discussion: https://postgr.es/m/CAHGQGwHmiLNRfvJDAR=PmxQgf7DbzPSO1M-1RoqO8oy=t2G5KA@mail.gmail.com
Backpatch-through: 17
transformGraphElementPattern() and transformGraphPattern() ran their
WHERE clauses through transformExpr() without ever applying
coerce_to_boolean(), so a WHERE clause of any type was accepted and
its bare datum was used as the qual, which resulted in wrong results
without a diagnostic. Route both sites through
transformWhereClause(), like every other WHERE clause, and add
regression tests for the element-level and pattern-level cases.
Author: Ewan Young <kdbase.hack@gmail.com>
Reviewed-by: Chao Li <li.evan.chao@gmail.com>
Discussion: https://www.postgresql.org/message-id/flat/CAON2xHNZUS8ZwzVEJFeRirsC_-7EJvdbwABSzdGaJ_DF9DvHoQ%40mail.gmail.com
Not many buildfarm members run this test, since it's gated behind
PG_TEST_EXTRA=xid_wraparound, but of those that do, the slower ones
not infrequently fail. The reason seems to be that the test expects
an INSERT command to either succeed or fail, but there's a window
where it can succeed while issuing a warning about impending
wraparound. poll_query_until treats nonempty stderr as a failure,
so it loops an extra time until the INSERT succeeds with no warning.
There seems no reason to treat this behavior as wrong, so adjust
the test to accept it.
Reported-by: Alexander Lakhin <exclusion@gmail.com>
Diagnosed-by: Alexander Lakhin <exclusion@gmail.com>
Author: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/7a68bee2-91bd-481d-be44-160e6beebc83@gmail.com
Backpatch-through: 17
Commit f63ae72bbc converted postgresql.conf.sample to spaces so
that it lines up at any tab width, but replace_guc_value() still
pads with tabs when it re-aligns the trailing comment of a
rewritten setting. Fix that.
Oversight in commit f63ae72bbc.
Reviewed-by: Chao Li <li.evan.chao@gmail.com>
Discussion: https://postgr.es/m/apRezMywHRuR5h-6%40nathan
Backpatch-through: 19
Commit 0ee83dd4a9 detected the mixed foreign x function-RTE cases in one
if/else chain that only set a pair of bool flags, then acted on those flags in
a second chain. fpinfo_o and fpinfo_i are each assigned in one chain and read
in the other, so proving them initialized requires correlating the flags with
the assignments. gcc 13 with -Og does not manage that and reports fpinfo_i as
possibly uninitialized; clang's -Wconditional-uninitialized likewise reports
both variables.
The flags served no purpose beyond deferring the work, so merge the two chains
and read both fdw_private pointers up front, unconditionally. A function rel
never has an fdw_private of its own, so the pointer is simply NULL until the
stub replaces it, and each branch now assigns and uses the two variables in one
place.
Reported-by: Karina Litskevich <litskevichkarina@gmail.com>
Reported-by: Marina Polyakova <m.polyakova@postgrespro.ru>
Reviewed-by: Marina Polyakova <m.polyakova@postgrespro.ru>
Discussion: https://postgr.es/m/CACiT8iamoL-%3D792e6JK2uCVvwDhfCg7DXHhvqN0P33jhGg2-8A%40mail.gmail.com
pg_clear_attribute_stats() was called exactly once in the whole test
suite, only for a correct cleanup. None of its failure scenarios were
covered, in terms of:
- NULL input values
- Incorrect object names (schema, attribute, relation)
- Incorrect relkinds
- Inherited pg_statistic entry cleanup
All these cases are now covered. This would have caught the failure
fixed by 11ed011ae2, as one example.
Author: Zhong ShiHao <zhong950419@gmail.com>
Discussion: https://postgr.es/m/CAGRkXqQEFzVh_bQSoLGDqaRTq7Xta6OUoda8PDrWDPWcAuwk=w@mail.gmail.com
Commit 44fb59fc6 taught the grouping-conflict walker to treat the arg
of a simple CASE as a direct operand of each WHEN comparison, but it
only checked the collation, on the assumption that the WHEN operator
is always the type-default "=" and thus matches the grouping eqop.
That assumption fails once the arg is relabeled to another type: the
WHEN then compares under that type's "=", which need not agree with
the grouping equality. For instance, with a DISTINCT over a citext
column, a qual such as "CASE t::text WHEN 'A' THEN ..." was pushed
below the Unique, although the equivalent "t::text = 'A'" is correctly
kept above it.
Instead of special-casing the arg, have the walker bind a Var arg
while walking the WHEN conditions and resolve each CaseTestExpr to it,
so that the arg is checked exactly as each WHEN uses it: with the
opfamily and collation checks of a direct operand when the WHEN is a
comparison, and as a non-operand reference otherwise. A non-Var arg
is walked once as a non-operand, as before. The CaseTestExpr in an
ArrayCoerceExpr's elemexpr and a JsonConstructorExpr's coercion stand
for something else and are left alone.
Reported-by: Tender Wang <tndrwang@gmail.com>
Author: Richard Guo <guofenglinux@gmail.com>
Reviewed-by: Tender Wang <tndrwang@gmail.com>
Reviewed-by: Ewan Young <kdbase.hack@gmail.com>
Discussion: https://postgr.es/m/CAHewXNkvGTOgijRLjmudpg=wz0d-J8ChY2o7ksh3Be+Q_Bxwog@mail.gmail.com
Backpatch-through: 18
When recovery starts from a backup without a signal file, pg_subtrans
is not started at the beginning of recovery and remains unstarted
throughout recovery. However, previously, a restartpoint run by
the checkpointer during recovery nevertheless tried to truncate
pg_subtrans, triggering the assertion failure:
TRAP: failed Assert("TransactionIdIsValid(initial)")
This commit fixes this by tracking whether pg_subtrans has been
started during recovery, and have the checkpointer check this flag
before truncating pg_subtrans at restartpoints.
Backpatch to all supported versions.
Reported-by: Imran Zaheer <imran.zhir@gmail.com>
Author: Imran Zaheer <imran.zhir@gmail.com>
Author: Fujii Masao <masao.fujii@gmail.com>
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Discussion: https://postgr.es/m/CA+UBfamzfEReT0VOGRUW_=_AecCYQ-CNKLR_T4Q+YEmJAH3fdg@mail.gmail.com
Backpatch-through: 14
Several queries in the foreign_key test sort pg_constraint rows with
"ORDER BY oid::regclass::text". These OIDs are constraint OIDs, not
relation OIDs, so the regclass cast renders them as numeric strings,
and text comparison of numeric strings depends on digit count. If the
OID counter crosses a power of ten while the test runs, the output
order flips, causing spurious test failures. This cannot happen in a
fresh cluster, but installcheck against a long-lived cluster can hit
it.
To fix, sort by constraint name, which is unique within each of these
queries, and use COLLATE "C" so that the order does not depend on the
database's default collation either.
Author: Richard Guo <guofenglinux@gmail.com>
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Reviewed-by: jian he <jian.universality@gmail.com>
Discussion: https://postgr.es/m/CAMbWs49KaJFGs4rfBWk+YmHPTXZ+pJTMJ12iaz7ruZdSK+sC5w@mail.gmail.com
Backpatch-through: 18
When reduce_outer_joins reduces an outer join to an antijoin, any IS
NULL qual on a Var from the antijoin's nullable side is necessarily
true. Previously, such quals were discarded later in
distribute_qual_to_rels, mainly to avoid bogus selectivity estimates.
But that discard was incomplete: the qual remained in the jointree,
while its Vars were not counted in attr_needed. Since commit
2ebf25e7d, join removal edits the jointree and expects it to contain
no other references to a removed rel, so it could remove a rel that
such a discarded qual still references, and then trip an assertion on
the qual's stale Var.
To fix, move this processing to an earlier phase: such quals are now
removed from the jointree by reduce_outer_joins itself. This way
later phases see a consistent query tree, and
check_redundant_nullability_qual is no longer needed, so remove it.
Back-patch to v16, as with commit 2ebf25e7d.
Reported-by: Tender Wang <tndrwang@gmail.com>
Author: Richard Guo <guofenglinux@gmail.com>
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/CAHewXNk8b0TsSy4dL=CO7FXL2W3WBm0BcdP-zwNJePa-Qj4HzA@mail.gmail.com
Backpatch-through: 16
The statistics import and restore paths built fake FunctionCallInfos
with LOCAL_FCINFO() and InitFunctionCallInfoData(), never calling
anything through them. The only data relevantly used was the values
carried around as NullableDatum arrays, read back with PG_GETARG_*()
macros.
import_relation_statistics() and import_attribute_statistics() were
passing a set of NullablePointer pointers, leading to more round-trip
manipulations with the fake FunctionCallInfos. These are reworked so as
the values pushed to pg_class (for relation stats) and pg_statistic (for
attribute stats) are passed around in dedicated structures with all the
values assigned based on positional indexes (import code cares about
less values than the restore code), simplifying their signatures and
reducing the number of code lines required to pass the data around.
The stats_check_*() functions now take the values and the attribute
names rather than a StatsArgInfo with positional indexes, as this code
is shared between the stats restore and import. This removes the
dependency between the StatsArgInfo bits and the import code.
StatsArgInfo is used only for the restore code, for argument mapping.
No behavioral change should be introduced in this commit; this is
(should be) pure refactoring.
Reported-by: Robert Haas <robertmhaas@gmail.com>
Author: Corey Huinker <corey.huinker@gmail.com>
Author: Michael Paquier <michael@paquier.xyz>
Discussion: https://postgr.es/m/CADkLM=eo7MtuCE=YjovW+=ASw1=q39qQ3qarrsw+EKfU901ztA@mail.gmail.com
Several invocations decoded and printed the full WAL range even though
the test only needs to confirm that a command form works or that
decoding reaches a specific error. Limit those checks to one record,
and start the fall-off-the-end checks near the end of the generated WAL.
This change reduces the IPC overhead overall, particularly on Windows,
without reducing coverage.
I am usually hesitant to backpatch such changes as this is only an
improvement, but the gains are too good in terms of IO and runtime, for
both the CI and the buildfarm. Based on the numbers provided, the CI
takes 30% less time to run the test with this change on Windows (worst
case shown on the lists). These tests have been introduced in
96063e2836.
Author: Sehrope Sarkuni <sehrope@jackdb.com>
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Reviewed-by: Nazir Bilal Yavuz <byavuz81@gmail.com>
Discussion: https://postgr.es/m/CAH7T-araSEdsNpxiKaMOW8kr_ZHhekCLHx5yzm2ksq4FdgULbA@mail.gmail.com
Backpatch-through: 17
While some <refentry> instances of supported PostgreSQL versions use the
sql-alter-table style, sql-altertable is the dominant style. Change
these before they're baked into the doc URLs of a supported version.
Backpatch-through: 19
After adding pg_stats.tableid, we taught pg_dump to use it in the
attribute statistics query for servers running v19 or later.
Notably, we did not give this new form of the query a "redundant
filter clause" like we did the pre-v19 form. Testing indicates
that produces a significant performance regression, and that adding
a similar filter clause to the v19-or-later form of the query fixes
it.
Oversight in commit 4b5ba0c4ca.
Reviewed-by: Sami Imseih <samimseih@gmail.com>
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Discussion: https://postgr.es/m/apQ9nvLW_zFdarYd%40nathan
Backpatch-through: 19
Sufficiently large settings of the *_freeze_score_weight parameters
can lower the scaling thresholds below the *_freeze_max_age
settings. This can lead the scoring code to raise a value < 1.0 to
a power > 1.0, thus lowering the score, which is the opposite of
what is intended. To fix, only exponentiate the scores when doing
so won't lower their values.
Oversight in commit d7965d65fc.
Reviewed-by: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Reviewed-by: Greg Burd <greg@burd.me>
Discussion: https://postgr.es/m/apWcbq91BAfhmsKU%40nathan
Backpatch-through: 19
The SIMD path for these commands reads ahead into the input buffer
via CopyLoadInputBuf(), which reports encoding and conversion
errors. The problem is that the errant data might lie beyond the
end-of-copy marker, which we aren't copying and thus needn't error
for. To fix, add a "speculative" parameter to CopyLoadInputBuf()
that defers these errors, and use it in the SIMD path. The errors
are instead reported only if we actually reach the errant data.
Testing indicates this does not meaningfully impact performance.
Oversight in commit e0a3a3fd53.
Reviewed-by: Greg Burd <greg@burd.me>
Reviewed-by: Nazir Bilal Yavuz <byavuz81@gmail.com>
Discussion: https://postgr.es/m/apWOr1kgVSqweBnU%40nathan
Backpatch-through: 19
