An index-only scan filled its result slot from the HeapTuple an index AM
returns in scan->xs_hitup by deforming it with the virtual slot's own
tuple descriptor (during GiST and SP-GiST index-only scans). But index
AMs form that heap tuple using their own descriptor, scan->xs_hitupdesc.
The AM's descriptor may disagree with the IoS virtual slot's descriptor
about each column's precise alignment, leading to "can't happen" errors
in certain rare edge cases. Hard crashes were possible but much less
likely.
To fix, deform the tuple with the descriptor it was formed with. This
is simpler, and makes xs_hitup handling (used by GiST and SP-GiST)
uniform with the nearby existing xs_itup handling (used by nbtree).
In practice this issue was very unlikely to be hit (it was found during
testing of a patch that will change the table AM API used during index
scans). The only currently affected core opclass is GiST's range_ops.
It was only possible for the datum to be accessed at an incorrectly
aligned offset when reading the second or subsequent column from a
multicolumn GiST index. This couldn't happen in the common case where
the datum used an unaligned short varlena header. Moreover, an earlier
column had to leave the range datum at an offset where the two
alignments actually disagree (e.g., an odd-length varlena datum).
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-by: Tomas Vondra <tomas@vondra.me>
Discussion: https://postgr.es/m/CAH2-WzkGXa2SKnebdW29RT1hCcQBo_p03v3iqif2u9bjzLB-aQ@mail.gmail.com
Backpatch-through: 14
The previous text claimed that once the slot is disassociated with
ALTER SUBSCRIPTION ... SET (slot_name = NONE), DROP SUBSCRIPTION "will no
longer attempt any actions on a remote host". That is inaccurate:
DROP SUBSCRIPTION may still connect to the publisher to drop
internally-created table synchronization slots when some table
synchronization is left unfinished. Reword to describe this, and note
that if the publisher is unreachable those slots (and the main slot, if
it still exists) must be dropped manually to avoid indefinitely reserving
WAL.
Reported-by: Jeff Davis <pgsql@j-davis.com>
Author: Amit Kapila <amit.kapila16@gmail.com>
Backpatch-through: 14
Discussion: https://postgr.es/m/CAA4eK1+tyYSpPxMBy1974kjivuGeR7YY=yopwRGrK3+vCTysdg@mail.gmail.com
Discussion: https://postgr.es/m/D908370F-2695-4231-851D-17179A6A6F2A@gmail.com
Commit c7aeb775 rewrote the HOT-chain offset sanity checks in three
places, but in heap_get_root_tuples it accidentally tested offnum -- the
outer loop variable, which is already bounded by the loop condition --
instead of nextoffnum, the offset actually passed to PageGetItemId. The
pre-c7aeb775 check tested nextoffnum.
With the check ineffective, a stale t_ctid could make PageGetItemId read
past the end of the line pointer array (which is data corruption that we
expect to be able to catch here).
Author: Peter Geoghegan <pg@bowt.ie>
Reported-by: Konstantin Knizhnik <knizhnik@garret.ru>
Discussion: https://postgr.es/m/87c7d8a4-3a82-4334-bee6-e8c2ad3f3293@garret.ru
Backpatch-through: 15
uuidv7(interval) shifts the current time by the given interval before
encoding it into the 48-bit Unix-millisecond timestamp field of the
generated UUID. Two cases were mishandled:
An infinite interval ('infinity' or '-infinity') produced an infinite
timestamp, which overflowed during the conversion to Unix-epoch
microseconds and yielded a garbage UUID. Reject infinite intervals up
front, before any timestamp arithmetic.
A shift that moved the timestamp outside the range representable by
the 48-bit field was silently accepted. Timestamps before the Unix
epoch wrapped when cast to unsigned, and timestamps beyond
approximately year 10889 overflowed the field; both produced UUIDs
with bogus timestamps that break sort ordering. Reject any shifted
timestamp outside the supported range.
Also document that infinite intervals and out-of-range shifts are
rejected.
Although raising a new error changes behavior in a stable branch, this
is back-patched to 18 (where uuidv7(interval) was introduced) because
the previous behavior can silently corrupt data. Failing loudly is far
safer than silently accepting the wraparound; otherwise users may not
discover that their UUIDv7 values are no longer sortable until years
later, when recovery is painful. It also matches how PostgreSQL
already handles timestamp + interval overflow, which raises an
error. The change only affects applications passing an interval large
enough to push the result outside the representable range.
Backpatch to 18, where uuidv7(interval) was introduced.
Reported-by: Christophe Pettus <xof@thebuild.com>
Author: Baji Shaik <baji.pgdev@gmail.com>
Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
Reviewed-by: Zsolt Parragi <zsolt.parragi@percona.com>
Reviewed-by: Tristan Partin <tristan@partin.io>
Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Discussion: https://www.postgresql.org/message-id/799A70FA-6E5C-4118-99EB-2FBBE1CBAC54@thebuild.com
Backpatch-through: 18
The regression test for postgres_fdw_get_connections(true) assumed that
a terminated remote connection would still remain visible in the FDW
connection cache long enough to be reported as closed with a nonzero
remote_backend_pid.
That assumption is not always valid. postgres_fdw_get_connections()
reports only entries that are still present in ConnectionHash, while
pgfdw_inval_callback() may immediately discard an idle cached connection
(xact_depth == 0) when a relevant invalidation arrives. In CI, that can
happen between terminating the remote backend and querying
postgres_fdw_get_connections(true), causing the function to return no
rows.
Adjust the idle-connection test to accept either outcome: if the cache
entry is still present, verify that it reports the expected server name,
closed status, and nonzero remote backend PID; otherwise treat zero rows
as a legitimate result.
To preserve coverage of the terminated-backend reporting path, add a
separate check inside an explicit transaction. In that case, concurrent
invalidation may mark the connection invalid but cannot discard it
before transaction end, so postgres_fdw_get_connections(true) should
still report the terminated connection as in-use, closed, and associated
with a nonzero remote backend PID.
Backpatch to v18, where the affected postgres_fdw_get_connections(true)
test was introduced.
Reported-by: Robert Haas <robertmhaas@gmail.com>
Author: Fujii Masao <masao.fujii@gmail.com>
Reviewed-by: Robert Haas <robertmhaas@gmail.com>
Reviewed-by: Daniel Gustafsson <daniel@yesql.se>
Discussion: https://postgr.es/m/CA+Tgmoax3cHXHsm9OidN4F-xiu16y8q2W8T5dTNFic1Zoo2cOw@mail.gmail.com
Backpatch-through: 18
Commit 67846550dc removed the xreflabels for initdb options, which
turned the sentence "The second field contains the page checksum if
data checksums are enabled" into "The second field contains the page
checksum if -k are enabled", as well "Only has effect if data checksums
are enabled" into "Only has effect if -k are enabled".
Fix by setting an explicit link text, and while there also change the
link to point to the data checksum page which has more information
than just the initdb option.
The original report was for one instance, further inspection turned
up quite a few more cases. Also redirect the link in the amcheck
docs which albeit was reading right, but will be more helpful if
linking to the main page on data checksums. Backpatch to v18 where
the xreflabels were removed.
Author: Daniel Gustafsson <daniel@yesql.se>
Reported-by: y.saburov@gmail.com
Reviewed-by: Laurenz Albe <laurenz.albe@cybertec.at>
Discussion: https://postgr.es/m/178350739237.73862.4549076173872335741@wrigleys.postgresql.org
Backpatch-through: 18
Previously, libpqrcv_create_slot() checked only that
CREATE_REPLICATION_SLOT returned PGRES_TUPLES_OK before reading
values from the first row. If the server unexpectedly returned an
invalid result, such as zero rows, PQgetvalue() could return NULL,
leading to a crash while parsing the LSN.
Other replication commands, such as IDENTIFY_SYSTEM, already validate
the response shape before accessing result values, but
CREATE_REPLICATION_SLOT did not.
Fix this by verifying that CREATE_REPLICATION_SLOT response contains
exactly one row with four fields, and report a protocol violation otherwise.
Backpatch to all supported versions.
Bug: #19547
Reported-by: Yuelin Wang <1217816127@qq.com>
Author: Kenny Chen <kennychen851228@gmail.com>
Reviewed-by: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Reviewed-by: Fujii Masao <masao.fujii@gmail.com>
Discussion: https://postgr.es/m/19547-f7986f668f71e788@postgresql.org
Discussion: https://postgr.es/m/CAPXstDtW2iqe+DJAOTQTX+rRziJp2UhZSo1+HRj1COAtbu+nKw@mail.gmail.com
Backpatch-through: 14
The documentation for log_parameter_max_length said it affects messages
generated by log_duration. However, log_duration alone does not log bind
parameter values, so this is misleading.
This commit updates the documentation to reference log_min_duration_statement,
which can log bind parameters, to better reflect actual behavior.
Backpatch to all supported versions.
Author: Fujii Masao <masao.fujii@gmail.com>
Reviewed-by: Surya Poondla <suryapoondla4@gmail.com>
Discussion: https://postgr.es/m/CAHGQGwGnCVMVz8-LU9F8Sh57bkQX3jMZzx7age7M0LFEz5=Fog@mail.gmail.com
Backpatch-through: 14
Heap WAL records that clear bits on the visibility map (like inserts and
deletes) did not register the visibility map blocks they modified.
Because the WAL summarizer only records registered blocks, an
incremental backup taken over such operations would omit the changed VM
pages. On restore, the VM would retain stale all-visible/all-frozen
bits, which can cause wrong results from index-only scans and incorrect
relfrozenxid advancement due to vacuum page skipping.
Not registering the VM buffer also meant we never emitted FPIs of VM
pages when clearing bits. A torn VM page won't raise an error because
the VM is read with ZERO_ON_ERROR; with checksums on, it would be
detected and zeroed, but with checksums off, it is accepted as-is and
can lead to data corruption.
Fix this by registering the VM buffer in the WAL record when clearing VM
bits. The VM buffer must now be locked throughout the critical section
that modifies the VM and heap pages and emits the WAL record. This can
slow down operations that clear the VM, since the VM lock is held longer
and VM FPIs may be emitted, but it is required for correctness.
Note that this fix does not repair existing incremental backups.
Author: Melanie Plageman <melanieplageman@gmail.com>
Author: Andres Freund <andres@anarazel.de>
Reviewed-by: Robert Haas <robertmhaas@gmail.com>
Reviewed-by: Andrey Borodin <x4mmm@yandex-team.ru>
Discussion: https://postgr.es/m/flat/CAAKRu_bn%2Be7F4yPFBgFbnP%2BsyJRKyNK092bjD2LKvZW7O4Svag>
Backpatch-through: 17
When registering a buffer with the WAL machinery, the caller assigns it a
block reference ID, and replay must read each block back by that same ID.
Today these IDs are bare integers assigned by convention (0, 1, 2, ...),
which is easy to follow when a record registers a single block, or when the
blocks are handled during replay in their registration order.
An upcoming bug fix registers up to two visibility map blocks in addition
to the heap block(s) when clearing the VM, and these are not handled during
replay in a straightforward 1:1, in-registration-order fashion. Relying on
bare integers for the block IDs in that case is error-prone.
Introduce macros naming the block reference IDs for the heap record types
that the upcoming commit extends to register visibility map blocks, so the
registration and replay sites refer to the same block by a meaningful name.
Author: Melanie Plageman <melanieplageman@gmail.com>
Reviewed-by: Robert Haas <robertmhaas@gmail.com>
Discussion: https://postgr.es/m/66mqpfyti3qhfttcsv6r2lbvqqd32rrmpn6i47ovrsnvguts46%40gou54xc>
Backpatch through: 17
When bulk-extending a relation, we add the newly-added blocks that we
won't immediately use to the free space map and then call
FreeSpaceMapVacuumRange() to propagate that free space up the FSM tree,
so other backends can find and reuse it.
However, the end block argument to FreeSpaceMapVacuumRange() is
exclusive, and we passed the number of the last added block (since
00d1e02be2). If that block was the first one covered by a new FSM page,
its free space wasn't propagated up the tree and was therefore invisible
to FSM searches until the next FSM vacuum.
Fix by passing the block number one past the last added block, so the
full range is vacuumed.
Author: Jingtang Zhang <mrdrivingduck@gmail.com>
Reviewed-by: Melanie Plageman <melanieplageman@gmail.com>
Discussion: https://postgr.es/m/flat/CAPsk3_Bx_vdybN%3D-DZu8HLStf%2BXnuFUBkLwxouONSMkWuO9oug%40mail.gmail.com
Backpatch-through: 16
Code elsewhere in the system assumes that fork numbers and chunk sizes
are within bounds, so the code that reads those quantities from disk
should validate that they are. Without these additional checks, a
corrupted file can cause us to index off the end of fork number or chunk
entry arrays, potentially resulting in a crash.
Reported-by: oxsignal <awo@kakao.com> (chunk sizes)
Reported-by: Robert Haas <rhaas@postgresql.org> (fork numbers)
Reviewed-by: Daniel Gustafsson <daniel@yesql.se>
Discussion: http://postgr.es/m/CA+TgmoYP8RKoBGosS7C6Fdr-GNCfyz_W1zmK=Tx1Fe0ZvzGh0g@mail.gmail.com
Backpatch-through: 17
pg_clear_attribute_stats() checks its required arguments manually
because the function is not strict. Previously, when schemaname or
relname was passed as NULL, the error incorrectly reported the
argument name as "relation" in both cases:
ERROR: argument "relation" must not be null
This was misleading, especially for schemaname, and inconsistent with
the function's SQL-visible argument names.
The cause is that cleararginfo[] in attribute_stats.c used
"relation" for both the schema-name and relation-name arguments.
This commit fixes the issue by using "schemaname" and "relname" instead,
matching the function's declared argument names so that the error reports
the correct argument name.
Backpatch to v18, where pg_clear_attribute_stats() was introduced.
Author: Ilia Evdokimov <ilya.evdokimov@tantorlabs.com>
Reviewed-by: Fujii Masao <masao.fujii@gmail.com>
Discussion: https://postgr.es/m/4bf66c5e-8dd7-4ef3-8691-db67ecff6f16@tantorlabs.com
Backpatch-through: 18
equalPolicy() is used in the relation cache to check if two policy
definitions are equivalent, but missed to check for polpermissive.
ALTER POLICY cannot switch a policy to be PERMISSIVE or RESTRICTIVE, so
this would need a dropped and then re-created policy, which would
trigger a relcache invalidation. Anyway, there is no harm in being
consistent in the check, and if one decides to add an ALTER POLICY to
switch PERMISSIVE or RESTRICTIVE, we would be silently in trouble.
Author: Andreas Lind <andreaslindpetersen@gmail.com>
Reviewed-by: Laurenz Albe <laurenz.albe@cybertec.at>
Discussion: https://postgr.es/m/CAMxA3rv1CS6R7JR5ojz-3CmCEnZEFrqu+XXTnGbLRWrjJRH7sA@mail.gmail.com
Backpatch-through: 14
Commit 9a60f295b stripped the stale PlaceHolderVars left behind by
left-join removal from the surviving rels' baserestrictinfo and from
EquivalenceClass member expressions, but it overlooked join clauses.
A PlaceHolderVar embedded in a join clause can likewise retain the
removed rel and join in its phrels, since remove_rel_from_query()
fixes up the RestrictInfo's own relid sets but not the PHVs inside its
expression.
As before, this is normally harmless, because later processing
consults those relid sets rather than the embedded PHVs. However, a
restriction clause derived from such an OR join clause inherits the
stale PlaceHolderVar, and when the derived clause is translated for an
appendrel child, pull_varnos() recomputes its relids and folds the
removed relation back in. The rebuilt clause then references a
no-longer-existent relation, tripping an assertion during path
generation.
Fix by also stripping the removed relation from the PlaceHolderVars in
the surviving rels' join clauses, including the sub-clauses of any OR
clause.
Like 9a60f295b, this is only reachable on v18 and later, where
match_index_to_operand() began ignoring PlaceHolderVars.
Author: Arne Roland <arne.roland@malkut.net>
Reviewed-by: Tender Wang <tndrwang@gmail.com>
Reviewed-by: Richard Guo <guofenglinux@gmail.com>
Discussion: https://postgr.es/m/27a44087-3d65-473e-8d88-7c12228e0d7e@malkut.net
Backpatch-through: 18
The pg_attribute_always_inline macro name is so long it forces pgindent
to format the code in strange ways. Which may incentivize patch authors
to either structure the code in strange ways (e.g. reorder prototypes),
use shorter names, etc. Neither is very desirable for code readability.
This shortens the name by removing the _attribute_ part. It also makes
it more consistent with pg_noinline, which does not have the _attribute_
part either.
Backpatched to all supported branches, to prevent conflicts when
backpatching other fixes. The backbranches however keep both the old and
new macro name, so that existing code keeps working.
Author: Andres Freund <andres@anarazel.de>
Reviewed-by: Peter Geoghegan <pg@bowt.ie>
Reviewed-by: Tomas Vondra <tomas@vondra.me>
Discussion: https://postgr.es/m/bqqdehahpoa36igpictuqyn2s2mexk3t3ehidh2ffd2slb35e5@rzgksuiszgbg
Backpatch-through: 14
Check explicitly for pqsecure_read() returning an error. It shouldn't
fail, and we would've caught it in the check for a short read, but
better to be explicit so that the error message is more informative.
We also shouldn't update 'inEnd' when the read fails, although that
too is just pro forma as we will bail out and close the connection on
error.
Reported-by: Peter Eisentraut <peter@eisentraut.org>
Discussion: https://www.postgresql.org/message-id/34844e8c-267c-4daf-b1e0-f26059a4a7d3@eisentraut.org
Backpatch-through: 14
When executing an UPDATE with a RETURNING clause on a table with a
BEFORE UPDATE row trigger, the computation of the OLD values in the
RETURNING list was incorrect if the target tuple was concurrently
updated by another session, at isolation level READ COMMITTED.
The problem was that the trigger code would lock the target tuple,
waiting for the other session to commit, and then fetch the updated
target tuple, but ExecUpdate() would not realise that the target tuple
had changed, and use the outdated target tuple for computing OLD
values. Fix by having ExecUpdate() check the TM_FailureData from
trigger execution and re-fetch the target tuple if necessary.
Re-fetching the target tuple like this is a little inefficient, but
probably negligible compared to the trigger execution and update. A
better long-term fix might be to move the EPQ code out of trigger.c,
and let ExecUpdate() handle it, like ExecMergeMatched() does, but that
would likely mean changing the trigger API, which seems a bit much for
back-patching.
Backpatch to v18, where support for RETURNING OLD/NEW was added.
Bug: #19536
Reported-by: Jonas Boberg <bobergj@gmail.com>
Diagnosed-by: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Author: Dean Rasheed <dean.a.rasheed@gmail.com>
Reviewed-by: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Discussion: https://postgr.es/m/19536-73ce5847e6c0e7b1@postgresql.org
Backpatch-through: 18
Commit aa606b9316 disallowed generated columns in COPY FROM WHERE
expressions, and commit 21c69dc73f disallowed system columns.
However, the COPY reference page still mentions only the restriction
on subqueries.
Update the documentation to also list generated columns and system
columns as unsupported in COPY FROM WHERE expressions.
Backpatch the generated-column documentation change to all supported
versions. Backpatch the system-column documentation change to v19,
where that restriction was introduced.
Author: Fujii Masao <masao.fujii@gmail.com>
Reviewed-by: Ayush Tiwari <ayushtiwari.slg01@gmail.com>
Discussion: https://postgr.es/m/CAHGQGwEgxErc54yVOAVWCsr1O=8pgw4oKRPuEQ9mfhkoYGR_XA@mail.gmail.com
Backpatch-through: 14
Commit dcb0049523 accidentally changed the final expanded query's
condition to > 2 while rewriting the example into SQL operator notation.
The original query and the preceding rewritten forms all use >= 2,
and view expansion should preserve that qualification. This commit
changes the final condition from > 2 to >= 2.
Backpatch to all supported versions.
Reported-by: Yaroslav Saburov <y.saburov@gmail.com>
Author: Fujii Masao <masao.fujii@gmail.com>
Reviewed-by: Daniel Gustafsson <daniel@yesql.se>
Discussion: https://postgr.es/m/178248467618.108999.9966122434342474006@wrigleys.postgresql.org
Backpatch-through: 14
If an expression containing an aggregate is evaluated above the plan
node that computes the aggregate, as happens with window functions or
with expressions postponed to above the final sort, setrefs.c replaces
the Aggref or WindowFunc with a Var referencing the lower node's
output. For SQL/JSON aggregates such as JSON_ARRAYAGG and
JSON_OBJECTAGG, deparsing the containing JsonConstructorExpr then
failed with "invalid JsonConstructorExpr underlying node type", since
get_json_agg_constructor() did not expect a Var there.
Fix by resolving the Var back to the underlying Aggref or WindowFunc
and deparsing the constructor as if the aggregate were computed at the
current node. The JsonConstructorExpr retains the RETURNING clause
and the ABSENT/NULL ON NULL and WITH UNIQUE options, and the arguments
come from the resolved aggregate, so the original JSON aggregate
syntax is reproduced in full. This mirrors how get_agg_expr() already
looks through such a Var when deparsing a combining aggregate.
Reported-by: Thom Brown <thom@linux.com>
Author: Richard Guo <guofenglinux@gmail.com>
Discussion: https://postgr.es/m/CAA-aLv5QYTaMOk=Qhv6cgwceeHETZV8YJvWZ_rH+yVZCuchATA@mail.gmail.com
Backpatch-through: 16
If the string is preceded only by Case Ignorable characters, don't
consider it to be a final sigma.
In the process, refactor so that the preceding and following
characters are found first, and then the rule is applied, to improve
clarity.
Discussion: https://postgr.es/m/c355354e6c3f4a7aafb047361b73db247260fca0.camel@j-davis.com
Backpatch-through: 18
The previous commit strengthened a workaround for a hang when large
messages are split across TLS records/GSS tokens. Because that
workaround is implemented in libpq internals, it can only help us when
libpq itself is polling on the socket. In nonblocking situations,
where the client above libpq is expected to poll, the same bugs can
show up.
As a contrived example, consider a large protocol-2.0 error coming
back from a server during PQconnectPoll(), split in an odd way across
two records:
-- TLS record (8192-byte payload) --
EEEE[...repeated a total of 8192 times]
-- TLS record (8193-byte payload) --
EEEE[...repeated a total of 8192 times]\0
The first record will fill the first half of the libpq receive buffer,
which is 16k long by default. The second record completely fills the
last half with its first 8192 bytes, leaving the terminating NULL in
the OpenSSL buffer. Since we still haven't seen the terminator at our
level, PQconnectPoll() will return PGRES_POLLING_READING, expecting to
come back when the server has sent "the rest" of the data. But there
is nothing left to read from the socket; OpenSSL had to pull all of
the data in the 8193-byte record off of the wire to decrypt it.
A real server would probably not split up the records this way, nor
keep the connection open after sending a fatal connection error. But
servers that regularly use larger TLS records can get the libpq
receive buffer into the same state if DataRows are big enough, as
reported on the list. While the PostgreSQL server doesn't use larger
TLS records like that, other non-PostgreSQL servers that implement the
wire protocol are known to do that, as well as proxies that sit
between the server and the client
This is a layering violation. libpq makes decisions based on data in
the application buffer, above the transport buffer (whether SSL or
GSS), but clients are polling the socket below the transport buffer.
One way to fix this in a backportable way, without changing APIs too
much, is to ensure data never stays in the transport buffer. Then
pqReadData's postconditions will look similar for both raw sockets and
SSL/GSS: any available data is either in the application buffer, or
still on the socket.
Building on the prior commit, make pqReadData() to drain all pending
data from the transport layer into conn->inBuffer, expanding the
buffer as necessary. This is not particularly efficient from an
architectural perspective (the pqsecure_read() implementations take
care to fit their packets into the current buffer, and that effort is
now completely discarded), but it's hopefully easier to reason about
than a full rewrite would be for the back branches.
Author: Jacob Champion <jacob.champion@enterprisedb.com>
Reviewed-by: Mark Dilger <mark.dilger@enterprisedb.com>
Reviewed-by: solai v <solai.cdac@gmail.com>
Reported-by: Lars Kanis <lars@greiz-reinsdorf.de>
Discussion: https://postgr.es/m/2039ac58-d3e0-434b-ac1a-2a987f3b4cb1%40greiz-reinsdorf.de
Backpatch-through: 14
An extra check for pending bytes in the SSL layer has been part of
pqReadReady() for a very long time (79ff2e96d). But when GSS transport
encryption was added, it didn't receive the same treatment. (As
79ff2e96d notes, "The bug that I fixed in this patch is exceptionally
hard to reproduce reliably.")
Without that check, it's possible to hit a hang in gssencmode, if the
server splits a large libpq message such that the final message in a
streamed response is part of the same wrapped token as the split
message:
DataRowDataRowDataRowDataRowDataRowData
-- token boundary --
RowDataRowCommandCompleteReadyForQuery
If the split message takes up enough memory to nearly fill libpq's
receive buffer, libpq may return from pqReadData() before the later
messages are pulled out of the PqGSSRecvBuffer. Without additional
socket activity from the server, pqReadReady() (via pqSocketCheck())
will never again return true, hanging the connection.
Pull the pending-bytes check into the pqsecure API layer, where both
SSL and GSS now implement it.
Note that this does not fix the root problem! Third party clients of
libpq have no way to call pqsecure_read_is_pending() in their own
polling. This just brings the GSS implementation up to par with the
existing SSL workaround; a broader fix is left to a subsequent commit.
In preparation for the broader fix, this patch already changes the
*_read_pending() functions to return the number of bytes in the buffer
rather than just a boolean. The current callers don't need that, but
the subsequent fix will.
Author: Jacob Champion <jacob.champion@enterprisedb.com>
Discussion: https://postgr.es/m/CAOYmi%2BmpymrgZ76Jre2dx_PwRniS9YZojwH0rZnTuiGHCsj0rA%40mail.gmail.com
Backpatch-through: 14
ParallelBackupStart() stored _beginthreadex()'s return value as the
worker's thread handle without checking it. On failure that value is 0,
which would later reach WaitForMultipleObjects() as a null handle, caught
only by an Assert. The fork() path already calls pg_fatal() when it
fails; do the same for _beginthreadex(), as pgbench does.
Author: Bryan Green <dbryan.green@gmail.com>
Discussion: https://www.postgresql.org/message-id/8c712d76-ecf7-4749-a6d8-dddc01f298ec@gmail.com
Backpatch-through: 14
9d9c02ccd added code to allow the executor to stop early when processing
WindowAgg nodes where a monotonic window function starts producing
values that result in a pushed-down qual no longer matching, and will
never match again due to the window function's monotonic properties.
That commit requires a SupportRequestWFuncMonotonic to exist on the
window function and for it to detect when the function is monotonic. For
COUNT(ANY) and COUNT(*), the support function failed to consider some
cases where the WindowClause used EXCLUDE to exclude certain rows from
being aggregated. Some WindowClause definitions mean we aggregate rows
that come after the current row, and when processing those rows later,
if we EXCLUDE certain rows, the monotonic property can be broken.
Wrongly treating the COUNT(*) or COUNT(ANY) aggregate as monotonic could
lead to rows being filtered that should not be filtered from the result
set.
Another issue was that the support function for the COUNT aggregate
mistakenly thought that a WindowClause without an ORDER BY meant that
the results would be both monotonically increasing and decreasing, but
that's only true when in RANGE mode, where all rows are peers.
It is possible to support various cases that do have an EXCLUDE clause,
but getting the logic correct for the exact set of cases that are valid
is quite complex and would likely better be left for a future project.
Here, we mostly disable run condition pushdown when there is an EXCLUDE
clause unless the clause is for EXCLUDE CURRENT ROW, uses COUNT(*)
(rather than COUNT(ANY)), and the window aggregate has no FILTER clause.
Bug: #19533
Reported-by: Qifan Liu <imchifan@163.com>
Author: Chengpeng Yan <chengpeng_yan@outlook.com>
Author: David Rowley <dgrowleyml@gmail.com>
Reviewed-by: Richard Guo <guofenglinux@gmail.com>
Reviewed-by: John Naylor <johncnaylorls@gmail.com>
Discussion: https://postgr.es/m/19533-413a1014e5d0e766@postgresql.org
Backpatch-through: 15
transformJsonBehavior() coerced an ON EMPTY / ON ERROR DEFAULT
expression only when its type differed from the RETURNING type's OID.
When the base type matched but the RETURNING type carried a type
modifier (e.g. numeric(4,1) or varchar(3)), the coercion that enforces
the typmod was skipped, so the DEFAULT value could violate the
declared type:
SELECT JSON_VALUE(jsonb '{}', '$.a'
RETURNING numeric(4,1) DEFAULT 99999.999 ON EMPTY);
returned 99999.999, which 99999.999::numeric(4,1) would reject; the
value could even be stored into a numeric(4,1) column, as later
coercions trust its already-correct type label.
Fix by also coercing when the RETURNING type has a typmod, except for
a NULL constant. coerce_to_target_type() is a no-op when the typmod
already matches. The matching-OID short-circuit dates to 74c96699be.
Reported-by: Ewan Young <kdbase.hack@gmail.com>
Author: Ewan Young <kdbase.hack@gmail.com>
Discussion: https://postgr.es/m/CAON2xHPO9f4cAmyGn1mQ=VqoS7wN5rz4yOiqudxX78zninZpCw@mail.gmail.com
Backpatch-through: 17
The loop in MatchText() processed a leading '\' without regard to
nondeterministic locales, which is problematic if what the '\'
precedes is an ordinary character that should be subject to
nondeterministic matching. We'd insist on a literal match for it,
which is not right and is not like what happens with a '\' that
follows some ordinary characters. Worse, we'd then advance the text
and pattern pointers by one byte, so that if the escaped character
is multibyte the next loop iteration would take the nondeterministic
code path starting at a point within the character. That could very
possibly cause pg_strncoll() to misbehave.
The fix is quite simple: move the stanza that handles '\' down past
the one that handles nondeterminism. The stanzas for '%' and '_'
are fine where they are, but the '\' stanza is only correct for
deterministic matching. The logic for nondeterministic cases is
already prepared to do the right things with a '\'.
While here, I replaced tests of "locale && !locale->deterministic"
with a boolean local variable, reasoning that those are in the hot
loop paths so saving a branch and indirect fetch is worth the
trouble. I also improved a number of related comments.
Author: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/391592.1783187986@sss.pgh.pa.us
Backpatch-through: 18
Commit 85b7efa1c introduced support for LIKE with non-deterministic
collations. By moving some conditionals around, it accidentally broke
the optimization for converting a LIKE or regex exact-match pattern
to an equality indexqual when the index collation doesn't match the
expression collation. That should be allowed if the expression
collation is deterministic. This patch re-introduces the optimization
for that common case.
One important beneficiary of this optimization is the "\d tablename"
command in psql. Without this fix that will do a seqscan on pg_class
instead of an index point lookup.
Reported-by: Andres Freund <andres@anarazel.de>
Author: Jelte Fennema-Nio <postgres@jeltef.nl>
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/DHBQIZX8SZVI.ZX614ZMFL645@jeltef.nl
Backpatch-through: 18
The planner has two optimizations that move a qual clause across a
grouping boundary: subquery_planner transfers HAVING clauses to WHERE
so they can be evaluated before aggregation, and qual_is_pushdown_safe
pushes outer restriction clauses into a subquery past its DISTINCT,
DISTINCT ON, window PARTITION BY, or set-operation grouping layer.
Both produce wrong results when the moved clause's equivalence
relation disagrees with the grouping's, since the clause then filters
rows the grouping would have merged.
The disagreement has two forms. A type may belong to multiple btree
opfamilies whose equality operators disagree (e.g. record_ops vs
record_image_ops); or the grouping may use a nondeterministic
collation, where comparing the column under a different collation, or
wrapping it in a function or operator, can distinguish values the
collation considers equal. Because we cannot prove an arbitrary
expression preserves that equality, a grouping column with a
nondeterministic collation is safe to push only as a direct operand of
a comparison under its own collation.
Fix both call sites through a shared walker parameterized by a
callback that maps each Var to the grouping equality operator for its
column (or InvalidOid for non-grouping Vars). For HAVING, the
callback recovers the SortGroupClause's eqop via the GROUP Var's
varattno, which requires running before flatten_group_exprs while
havingQual still contains GROUP Vars. For subquery pushdown, the
callback recovers the eqop from subquery->distinctClause, a window's
partitionClause, or any grouping node in the SetOperationStmt tree.
The walker fires only when there is an equivalence boundary to cross,
gated by either the existing UNSAFE_NOTIN_DISTINCTON_CLAUSE and
UNSAFE_NOTIN_PARTITIONBY_CLAUSE flags or by a recursive check for any
grouping node in the set-op tree.
Back-patch to v18 only. The HAVING half relies on the RTE_GROUP
mechanism introduced in v18 (commit 247dea89f), which is what lets us
identify grouping expressions via GROUP Vars on pre-flatten
havingQual. Pre-v18 branches lack that machinery, so a back-patch
there would need a different approach. Given the absence of field
reports of these bugs on back branches, the risk of carrying a
different fix on stable branches is not justified.
Author: Richard Guo <guofenglinux@gmail.com>
Reviewed-by: Thom Brown <thom@linux.com>
Reviewed-by: Florin Irion <irionr@gmail.com>
Reviewed-by: Zsolt Parragi <zsolt.parragi@percona.com>
Reviewed-by: Tender Wang <tndrwang@gmail.com>
Reviewed-by: Chengpeng Yan <chengpeng_yan@outlook.com>
Discussion: https://postgr.es/m/CAMbWs4-QLZpn3UVOpeG2fOxxhdnkDNMZ_3Zcm3dqJwRAphz68g@mail.gmail.com
Backpatch-through: 18
Commit e7564ee8cd, which fixed base backup progress reporting on
backup failure, removed the external function basebackup_progress_done()
because it was no longer used in core. When that change was backpatched
to v15, it introduced an ABI break, which was reported by buildfarm
member crake.
This commit restores basebackup_progress_done() to preserve ABI
compatibility, even though it is no longer used in core, rather than
updating the .abi-compliance-history file. Because external backup
tools may still call this function.
Per buildfarm member crake.
Reported-by: Andrew Dunstan <andrew@dunslane.net>
Discussion: https://postgr.es/m/CAD5tBcJ+ktrEp=PT8Gq-f=8mA2cDtZMB-hDMV4mMJ+9V46qBeQ@mail.gmail.com
Backpatch-through: 15-18
ON SELECT rules must be named "_RETURN", while other kinds of rules
must not be; this ancient restriction is depended on by various client
code. We successfully enforced this convention in most places, but
ALTER RULE allowed renaming a non-SELECT rule to "_RETURN". Notably,
that would break dump/restore, since the eventual CREATE RULE command
would reject the name.
While at it, remove DefineQueryRewrite's hack to substitute "_RETURN"
for the convention that was used before 7.3. We dropped other
server-side code that supported restoring pre-7.3 dumps some time ago
(notably in e58a59975 and nearby commits), but this bit was missed.
Bug: #19543
Reported-by: Adam Pickering <adamkpickering@gmail.com>
Author: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/19543-461228e77f3b32fc@postgresql.org
Backpatch-through: 14
gbt_var_consistent() handled the <> (BtreeGistNotEqual) strategy without
distinguishing leaf from internal pages, unlike every other strategy.
In particular, it tried to apply the datatype-specific f_eq method,
which is completely wrong since internal keys might not have the same
representation as leaf keys. This led to OOB reads and potentially
crashes, and most likely to wrong query results as well.
On leaf pages we can apply the inverse of what the Equal strategy does.
On internal pages, use a correct implementation of what the previous
code intended: we can descend if the query value equals both bounds,
*so long as the bounds aren't truncated*. With truncated bounds we
don't quite know the range of what's below, so we must always descend.
Adjust the code in gbt_num_consistent() to look similar, too. This
fixes a performance buglet in that there's no need to do two comparisons
on a leaf entry, but the main point is just to keep code consistency.
Reported-by: 王跃林 <violin0613@tju.edu.cn>
Author: Ayush Tiwari <ayushtiwari.slg01@gmail.com>
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/AH*AvQCYKhQGVvPWi1GiU4oY.8.1781609375063.Hmail.3020001251@tju.edu.cn
Backpatch-through: 14
If we're dealing with leaf entries, the function to call is bitcmp
not byteacmp. Using byteacmp didn't lead to any obvious failure,
but it did result in sorting the entries in a way not matching the
datatype's actual sort order. Hence the constructed index would be
less efficient than one would expect, and in particular worse than
what you got before this code was added in v18 (by commit e4309f73f).
We might want to recommend that users reindex btree_gist indexes
on bit/varbit columns.
Author: Tom Lane <tgl@sss.pgh.pa.us>
Reviewed-by: Ayush Tiwari <ayushtiwari.slg01@gmail.com>
Discussion: https://postgr.es/m/AH*AvQCYKhQGVvPWi1GiU4oY.8.1781609375063.Hmail.3020001251@tju.edu.cn
Backpatch-through: 18
Commit ce146621 ensures that ERROR is raised if a session tries to read
pages of another session's temp table. But there is a corner case where
the other session's temp table is empty -- in this case the INSERT
command bypasses our checks and executes without any errors.
Such behavior is inconsistent and erroneous: it leaves an invalid buffer
in the temp buffers pool. Since the buffer was created for another
session's temp table, we get an error "no such file or directory" when
trying to flush it.
This commit fixes it by adding a RELATION_IS_OTHER_TEMP check in the
relation-extension path.
Backpatch to 16, because it is the first release after 31966b151e, which
introduced a separate local relation extension function
ExtendBufferedRelLocal(), which lacks of RELATION_IS_OTHER_TEMP() check.
As this fix introduces more checks to 013_temp_obj_multisession.pl, backpatch
the whole test script to 16.
Discussion: https://postgr.es/m/CAJDiXgiX2XZBHDNo%2BzBbvku%2BtchrUurvPRaN1_40mEQ1_sG90g%40mail.gmail.com
Author: Daniil Davydov <3danissimo@gmail.com>
Reviewed-by: Jim Jones <jim.jones@uni-muenster.de>
Reviewed-by: Imran Zaheer <imran.zhir@gmail.com>
Reviewed-by: ZizhuanLiu X-MAN <44973863@qq.com>
Backpatch-through: 16
Previously, MultiXactId wraparound hints suggested dropping stale
replication slots. While that advice is appropriate for transaction ID
wraparound, where replication slots can hold back XID horizons,
it was misleading for MultiXactId wraparound. Following it could lead
users to drop replication slots unnecessarily without helping resolve
the MultiXactId wraparound condition.
MultiXact cleanup is not directly delayed by replication slots.
Instead, it depends on whether old MultiXactIds can still be seen
as live by running transactions.
This commit removes the replication slot advice from MultiXactId
wraparound hints, and documents that stale replication slots are
normally not relevant to resolving MultiXactId wraparound problems.
Backpatch to all supported branches.
BUG #18876
Reported-by: Haruka Takatsuka <harukat@sraoss.co.jp>
Author: Fujii Masao <masao.fujii@gmail.com>
Discussion: https://postgr.es/m/18876-0d0b53bad5a1f4c1@postgresql.org
Backpatch-through: 14
The jsonpath .decimal(precision[, scale]) method built its numeric
typmod by calling numerictypmodin() through DirectFunctionCall1(), which
can throw a hard error for an incorrect set of precision and/or scale
vaulues. This breaks the silent mode supported by this function, that
should not fail.
Most of the jsonpath code uses the soft error reporting to bypass
errors, which is what this fix does by avoiding a direct use of
numerictypmodin(). Its code is refactored to use a new routine called
make_numeric_typmod_safe(), able to take an error context in input.
numerictypmodin() sets no context, mapping to its previous behavior.
The jsonpath code sets or not a context depending on the use of the
silent mode. This result leads to some nice simplifications:
numerictypmodin() feeds on an array, we can now pass directly values for
the scale and precision.
Oversight in 66ea94e8e6.
Author: Ewan Young <kdbase.hack@gmail.com>
Discussion: https://postgr.es/m/CAON2xHMaigKABiyPBBq3Sjd3gp7uWMJXnnMHt=s85V1ij3KP1w@mail.gmail.com
Backpatch-through: 17
The float4 and float8 btree_gist opclasses compared keys with raw C
operators (==, <, >). IEEE 754 makes every comparison involving NaN
false, so GiST disagreed with the regular float comparison operators
and with the btree opclass, which uses float[4|8]_cmp_internal()
(so that all NaNs are equal and NaN sorts after every non-NaN value).
In addition, the penalty and distance functions were not careful
about NaNs, and the penalty functions could also misbehave for IEEE
infinities. Wrong answers from the penalty functions would probably
do no more than make the index non-optimal, but the distance mistakes
were visible from SQL.
To fix, make the comparison functions rely on the same NaN-aware
comparison functions the core code uses, and rewrite the penalty
and distance functions to follow the rules that NaNs are equal
but maximally far away from non-NaNs. The penalty_num() code was
formerly shared between integral and float cases, but I chose to make
two copies so that the integral cases are not saddled with the extra
logic for NaNs and infinities/overflows. I also rewrote it as static
inline functions instead of an unreadable and uncommented macro.
The float penalty functions were previously unreached by the
regression tests, so add new test cases to exercise them.
There's no on-disk format change, but users who have NaN entries
in a btree_gist index would be well advised to reindex it.
Bug: #19501
Bug: #19524
Reported-by: Man Zeng <zengman@halodbtech.com>
Reported-by: Yuelin Wang <3020001251@tju.edu.cn>
Author: Bill Kim <billkimjh@gmail.com>
Co-authored-by: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/19501-3bff3bbc97f1e7c9@postgresql.org
Discussion: https://postgr.es/m/19524-9559d302c8455664@postgresql.org
Discussion: https://postgr.es/m/CAMQXxcgbtD2LXfX0tpgvOizxP-XxrCHV2ZDy4By_TZnJMsxXWQ@mail.gmail.com
Backpatch-through: 14
Previously, if a base backup failed after it had started streaming
files, pg_stat_progress_basebackup could continue to show a stale
progress entry even though the backup was no longer running. This could
be observed when the client kept the replication connection open after
the error. It is normally not observable when using pg_basebackup,
because the client disconnects after the error.
The problem was that progress reporting was cleared only after
successful completion.
This commit moves the progress reporting cleanup into the progress
sink's cleanup callback so that it is cleared after both successful
and failed backups.
Backpatch to v15. v14 has the same issue, but the fix does not apply
cleanly because it lacks the base backup sink infrastructure. Since
the bug does not affect the backup itself and is normally not
observable when using pg_basebackup, skip the v14 backpatch.
Author: Chao Li <lic@highgo.com>
Reviewed-by: Fujii Masao <masao.fujii@gmail.com>
Discussion: https://postgr.es/m/EA1A6CD2-EFA6-462B-9A02-03003555AB4A@gmail.com
Backpatch-through: 15
