The following commits are reverted:
* 092f3c63ef, libpq: Add "servicefile" connection option
* 6b1c4d326b, psql: Add variable SERVICEFILE
* 90354030b8, libpq: Fix "servicefile" after fallback to system
service file
After more review, PGSERVICEFILE is sufficient for the purpose of
specifying a per-session service file. Some details around this feature
will need more attention before being actually usable, if we ever get
down to it.
One independent piece that has been committed during the v19 development
cycle is df286a5b83, which is kept in the tree as that was an
unrelated change.
Reported-by: Sehrope Sarkuni <sehrope@jackdb.com>
Discussion: https://postgr.es/m/CAH7T-aq+U6GDPO8mr3Gga3KzxGpAE=BSLb38x+R8fvWjzF-H8w@mail.gmail.com
Backpatch-through: 19
Previously, REPACK and CLUSTER silently skipped foreign-table partitions
when processing a partitioned table, leaving users unaware that some
partitions had not been processed. In contrast, VACUUM FULL warns when
skipping such partitions.
Emit a warning for each skipped foreign-table partition. Inspect the
table's partition tree even when an index is specified, because foreign
tables have no indexes and therefore do not appear in the index's
partition tree. Check privileges before issuing the warning, so that
users without the required privileges receive a permission warning
instead, as with VACUUM FULL.
Note that database-wide REPACK and CLUSTER continue to skip foreign
tables silently, consistent with database-wide VACUUM FULL.
Author: Zsolt Parragi <zsolt.parragi@percona.com>
Reviewed-by: Antonin Houska <ah@cybertec.at>
Reviewed-by: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Reviewed-by: shihao zhong <zhong950419@gmail.com>
Reviewed-by: Surya Poondla <suryapoondla4@gmail.com>
Reviewed-by: solai v <solai.cdac@gmail.com>
Reviewed-by: Fujii Masao <masao.fujii@gmail.com>
Discussion: https://postgr.es/m/CAN4CZFNXFdkmOCnt8m=K_A4i3LQC+59nx59Y=5bsH4vehQHTTw@mail.gmail.com
The documentation stated that existing connections would continue to
function normally after the OOM killer terminated the postmaster.
In fact, those connections eventually terminate when server processes
detect the postmaster's death.
Clarify that the OOM killer can terminate either the postmaster or a
child process. If the postmaster is killed, the server must be restarted.
If a child process is killed, PostgreSQL typically terminates the
remaining connections and either recovers automatically or shuts down,
depending on restart_after_crash.
Backpatch to all supported versions.
Author: Takeshi Ideriha <iderihatakeshi@gmail.com>
Reviewed-by: Manuel Reyes Bravo <manuelreyesbravo@gmail.com>
Reviewed-by: Fujii Masao <masao.fujii@gmail.com>
Discussion: https://postgr.es/m/ce269321-a7f4-48f4-a9ca-a4a4983b9fd6@gmail.com
Backpatch-through: 14
The description of autovacuum_max_parallel_workers did not specify
where it can be set or how it relates to the per-table
autovacuum_parallel_workers storage parameter.
Clarify that it can be set only in postgresql.conf or on the server
command line, and that the limit can be reduced for individual tables.
Add a link to the autovacuum_parallel_workers storage parameter.
Backpatch to v19, where parallel autovacuum was introduced.
Author: Baji Shaik <baji.pgdev@gmail.com>
Reviewed-by: Sami Imseih <samimseih.pg@gmail.com>
Reviewed-by: Chao Li <li.evan.chao@gmail.com>
Reviewed-by: Fujii Masao <masao.fujii@gmail.com>
Discussion: https://postgr.es/m/CA+fm-RMmbUUPWJShxkhoB7SeiQ7XWw3YsVnHpMzp1Y2i_gtwig@mail.gmail.com
Backpatch-through: 19
Callers of WALReadFromBuffers() read data from the WAL buffers
(currently only WAL senders). This commit makes now available the
number of pages read from the WAL uffers in pg_stat_io.hits under object
"wal" and context "normal".
A read from a WAL file is counted in pg_stat_io.reads and read_bytes.
This new information can be useful for the tuning of wal_buffers, where
the number of hits matches with the number of pages found as available
in the buffers.
Note that WALReadFromBuffers() may be called for an amount of data worth
less than one WAL page, and that a partial page read is counted as one
hit.
Author: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Discussion: https://postgr.es/m/CALj2ACVfF2Uj9NoFy-5m98HNtjHpuD17EDE9twVeJng-jTAe7A@mail.gmail.com
Progress reporting in pg_basebackup and pg_stat_progress_basebackup used
the estimated size of a full backup even when taking an incremental backup.
The size estimation pass did not account for incremental files and counted
every file at its full size. Progress could therefore remain at only a few
percent even after the incremental backup completed successfully.
Fix this by making the size estimation pass account for incremental files,
counting them at the size they will occupy in the archive.
Backpatch to 17, where incremental backup was introduced.
Author: Daria Lepikhova <daria.n.lepikhova@gmail.com>
Reviewed-by: Manuel Reyes Bravo <manuelreyesbravo@gmail.com>
Reviewed-by: Zsolt Parragi <zsolt.parragi@percona.com>
Reviewed-by: Surya Poondla <suryapoondla4@gmail.com>
Reviewed-by: Fujii Masao <masao.fujii@gmail.com>
Discussion: https://postgr.es/m/CAK49S1ASwk7L-7LYUUz1MU_+uQnZ75srva2XRR4r-w0HCVZeUQ@mail.gmail.com
Backpatch-through: 17
The documentation described base backup progress in terms of data
streamed, which could be mistaken for the amount of data sent over the
connection when server-side compression is enabled. Clarify that the
counters measure archive data before server-side compression.
The progress report message was also documented as counting bytes from
the current tablespace, although it reports a cumulative count for the
whole backup. Correct the description to say that the count covers all
tablespaces since the backup began.
Backpatch to 15, where both the progress report message and server-side
compression were introduced.
Suggested-by: Surya Poondla <suryapoondla4@gmail.com>
Author: Daria Lepikhova <daria.n.lepikhova@gmail.com>
Reviewed-by: Surya Poondla <suryapoondla4@gmail.com>
Reviewed-by: Fujii Masao <masao.fujii@gmail.com>
Discussion: https://postgr.es/m/CAOVWO5pNe6XprMnt5OTT+-2Sm67z5GX4DzUNArEHubF0z9-PZQ@mail.gmail.com
Backpatch-through: 15
To avoid warnings and lines that extend beyond the right page
margin in the PDF documentation, we have zero-width spaces in
the source at strategic places. They are not visible to the
naked eye, but allow FOP to break the lines in sane places.
However, some of these zero-width spaces were in the middle of
text that people might copy and paste from the HTML documentation,
which leads to mysterious syntax errors.
Scanning the documentation for &zwsp; identified four places
that this patch improves:
- an ISO 8601 interval literal didn't actually need a line break
in either the US letter or A4 format, so simply remove that
zero-width space.
- three v18-era examples for uuid_extract_timestamp() and
uuid_extract_version() had zero-width spaces in the middle of UUID
literals. Instead, add regular spaces around the "::" cast symbols,
allowing FOP to break the lines there. This solution isn't
invisible in the HTML output, but it doesn't look bad.
Reported-by: Y. Saburov <y.saburov@gmail.com>
Author: Laurenz Albe <laurenz.albe@cybertec.at>
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/179143326813.1061054.14714458907175925072@wrigleys.postgresql.org
Backpatch-through: 18
This backwards-compatibility hack was introduced twenty years ago (in
8.2), and has surely been of no value for at least the last ten years.
However, we shouldn't just leave it sit, because as was just pointed
out it creates a dump/reload hazard: if it's set in the receiving
environment then array contents will not be restored correctly.
Other applications would likely be broken by it too, so whatever
residual value it may have seems outweighed by the possibility of
misuse.
Our precedent for retiring some other comparable parameters, such as
standard_conforming_strings, has been to leave the parameter in
existence but force it to a fixed value. I judge however that
that's not worth the trouble here, because there's no reason to
believe that a non-negligible number of applications are either
setting or reading this value. Let's just delete it in the master
branch and be done with it.
Bug: #19747
Reported-by: Ke <kehan5800@gmail.com>
Author: Tom Lane <tgl@sss.pgh.pa.us>
Reviewed-by: Laurenz Albe <laurenz.albe@cybertec.at>
Discussion: https://postgr.es/m/19747-69b01e7fc190cd58@postgresql.org
The page a bit of a mess, with notes interpersed in the list of
parameters, and two "notes" subsections that weren't in the Notes
section.
Limit the commentary under "Description" and "Parameters" to a minimum;
move the existing text to appear in the "Notes" section. Both the
"Notes on Clustering" and "Notes on Resources" subsections, which were
under "Description", are moved to be under "Notes" instead. Add a new
"Notes on Concurrent Operation" subsection there, which now carries some
text that was in "Parameters", and gets some additional text to
(hopefully) explain resource consumption more clearly -- notably, the
hard limit of 104 million tuples that can be concurrently updated/deleted
before running into the palloc size limit.
Add appropriate cross-links.
Author: Radim Marek <radim@boringsql.com>
Author: Álvaro Herrera <alvherre@kurilemu.de>
Author: Shihao Zhong <zhong950419@gmail.com>
Backpatch-through: 19
Discussion: https://postgr.es/m/CAJgoLk+dodrwuwCuXERGYwgjQzSwrKd+xgitYrL32oWDt39zvA@mail.gmail.com
bpchar equality ignores trailing spaces, so equal values in a column
with no length specifier can have different images. Similarly,
oidvector equality intentionally pays no attention to the array lower
bound field. We nevertheless allowed nbtree deduplication to be used
with indexes built with bpchar_ops, bpchar_pattern_ops, or
oidvector_ops. A posting list split could therefore place a TID in an
index tuple whose key was equal to, but not bytewise identical to, the
corresponding table value. This allowed index-only scans to return
subtly wrong results.
To fix, drop support function 4 from the affected opclasses (we can't
change catalog contents on branches 14 through 18, so we just disable
deduplication by adding hard-coding to the relevant support functions).
Deliberately leave the oidvector opclass alone on branches 14 through
18, though. Disallowing oidvector deduplication on these branches would
make pg_amcheck report that pg_proc_proname_args_nsp_index has a corrupt
metapage, needlessly alarming users.
System catalog oidvector values always have a lower bound of 0, so the
issue can't cause harm there; a user can only hit it by indexing an
oidvector column that stores the output of a function like trim_array().
It seems very unlikely that any user would ever do that.
Users should REINDEX any B-Tree index on a bpchar column.
Author: Shihao Zhong <zhong950419@gmail.com>
Co-authored-by: Peter Geoghegan <pg@bowt.ie>
Reported-by: Ke <kehan5800@gmail.com>
Reported-by: Tom Lane <tgl@sss.pgh.pa.us>
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Bug: #19749
Discussion: https://postgr.es/m/19749-d46adf7c7d910b07@postgresql.org
Discussion: https://postgr.es/m/CAGRkXqQQ311Lg=62x5UdZtPuRFgpT6XpPMw4X9E3tgbTu162Mw@mail.gmail.com
Backpatch-through: 14
seg.sgml claims that a seg can store 7 significant digits, but this
is nonsense. The underlying float4 data type is not (quite) that
precise, and seg_out() intentionally rounds off to at most FLT_DIG
decimal places. With IEEE float arithmetic, FLT_DIG will be defined
as 6, or at least that's the case on all platforms I know of.
AFAICT this thinko goes clear back to the introduction of seg
(commit a3694b420), but it wasn't questioned until now.
Bug: #19744
Reported-by: Ke <kehan5800@gmail.com>
Author: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/19744-8c700582bbb1d075@postgresql.org
Backpatch-through: 14
pg_blocking_pids() assumes that members of the same lock group
never block each other, but that hasn't been true for relation
extension locks since v13. This commit teaches the function to
report lock group members as blockers for relation extension locks.
Since parallel workers are reported with their leader's PID, the
result may now include the blocked session's own PID.
Oversight in commit 85f6b49c2c.
Author: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Reviewed-by: Chao Li <li.evan.chao@gmail.com>
Reviewed-by: Kirill Reshke <reshkekirill@gmail.com>
Reviewed-by: Shihao Zhong <zhong950419@gmail.com>
Discussion: https://postgr.es/m/apTpV8h%2BCYHf%2B1PJ%40bdtpg
Backpatch-through: 14
This commit addresses the following issues in commit de28140de:
* When the local transaction was DEFERRABLE, begin_remote_xact() started
a remote transaction in the same mode without checking the remote
server version. Since DEFERRABLE is supported only in v9.1 and later,
this would cause a syntax error on older remote servers. Fix this by
propagating the mode only to remote servers that support it.
* pgfdw_xact_callback() committed remote transactions without
synchronizing the read-only mode. If the local transaction had become
READ ONLY since the last remote operation, this would cause deferred
triggers on remote servers to still run in READ WRITE mode. Fix this
by synchronizing the mode at commit time.
* The documentation did not match the actual implementation. Update the
documentation to reflect the implementation.
Reported-by: Fujii Masao <masao.fujii@gmail.com>
Author: Etsuro Fujita <etsuro.fujita@gmail.com>
Reviewed-by: Matheus Alcantara <matheusssilv97@gmail.com>
Discussion: https://postgr.es/m/CAHGQGwE4UMaWGOZq1Z7YUTUSg-CgL4yAzO5YaGwTZe1-6Pjg2g%40mail.gmail.com
Backpatch-through: 19
pg_stat_progress_vacuum gains three fields, reflecting some activity
related to index cleanups during VACUUM, which is a phase that could
take a long time depending on the quantity of data to treat and the
number of indexes:
- current_index_relid, the index being processed.
- index_blks_total, total number of blocks to process.
- index_blks_done, number of blocks already processed.
There fields are new, and do not reuse the existing fields related to
heap as index cleanups may need multiple passes, where knowing both the
heap activity and the index activity could be relevant. This is
unlikely since tids are accumulated in the heap scanning phase using an
optimized radix tree, still possible. VACUUM can spawn parallel workers
to clean indexes (each worker cleans a different index); these should up
in the progress view in the shape of one extra row for each worker.
Bump catalog version.
Author: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Reviewed-by: Sami Imseih <samimseih@gmail.com>
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
Reviewed-by: Shihao Zhong <zhong950419@gmail.com>
Discussion: https://postgr.es/m/CALj2ACUgwSchK6jQ2CdKLBWUADTOE_zKdTff2Zg3E6hOuXKv-w@mail.gmail.com
There have been several recent complaints that << and >> don't behave
the same way on all supported platforms. There is a risk that <<
overflows the signed integer type, which is undefined behavior, and there
are platform-dependent behaviors from shifting by numbers of bits >= the
bitwidth of the type, and also from shifting by negative values.
Document this to warn unsuspecting users of these facts.
Author: David Rowley <dgrowleyml@gmail.com>
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/CAApHDvr0O6ZF8NO=Rq9Q+Sz=FHT=NZ1kkNTXZDcNph-OWuk7TQ@mail.gmail.com
Backpatch-through: 14
The explanation of aggressive scans claimed that if the multixact
members grows to 2 billion entries, we launch autovacuum even if it's
nominally disabled. That's no longer true; anti-wraparound vacuum is
only triggered if we're in danger of running out of multixids, we no
longer launch it just to shrink the 'members' SLRU. There's no
wraparound danger with 'members' anymore, so it's no worse than if a
user table becomes very bloated.
We do still adjust the freeze threshold though, so that if autovacuum
runs, whether on schedule or as an emergency autovacuum to avoid
wraparound, it will freeze more aggressively if the members SLRU is
large.
I removed the mention of the 4 billion member threshold, even though
that still exists, because the explanation felt misleading. The
aggressiveness is scaled proportionally between 2 and 4 billion
members. At 4 billion it's as aggressive as it gets, but there's no
sudden cliff.
In the passing, fix some comments. Remove obsolete mention of safe
threshold for member storage triggering autovacuum. Secondly, the
freezing cutoff is adjusted to avoid the *members* SLRU from growing
too large.
Reported-by: Noah Misch <noah@leadboat.com>
Discussion: https://www.postgresql.org/message-id/20260827231757.78.noahmisch@microsoft.com
Backpatch-through: 19
The "initializing" phase was described as expected to be very brief,
but for REPACK (CONCURRENTLY), it also covers enabling and
initializing logical decoding, either of which can take a long
time. Also, the "catch-up" phase includes waiting for the ACCESS
EXCLUSIVE lock needed to swap the files, which was not mentioned.
Adding separate progress phases for these steps is left for v20; for
v19 we only clarify the description.
Author: Sami Imseih <samimseih.pg@gmail.com>
Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
Reviewed-by: Manuel Reyes Bravo <manuelreyesbravo@gmail.com>
Discussion: https://postgr.es/m/CAN12%2BYJOeOyPYB5JS28VE_%2Brzg%3DRbzQXfj58TGfMYwv-TX9mJA%40mail.gmail.com
Backpatch-through: 19
Per ISO/IEC 9075-2:2023, 7.11 <JSON table>, Syntax Rules 1)e)iv) and
1)f)xi), a regular or formatted JSON_TABLE column that does not specify
its own ON ERROR clause takes its default error behavior from the
table-level ON ERROR clause: with ERROR ON ERROR on the table, the
column behaves as ERROR ON ERROR; otherwise the column defaults to
NULL ON ERROR.
PostgreSQL instead always defaulted such columns to NULL ON ERROR, so
SELECT * FROM JSON_TABLE(jsonb '"err"', '$'
COLUMNS (a int PATH '$') ERROR ON ERROR) jt;
returned a NULL row where the standard requires an error. The
documentation stated the divergence as if it were a rule, saying that
the table-level clause "does not affect the errors that occur when
evaluating columns".
Implement the standard behavior in the JSON_TABLE syntactic
transformation: when a column lacks its own ON ERROR clause and the
table-level behavior is ERROR ON ERROR, synthesize an implicit ERROR ON
ERROR for the column before it is transformed into a JsonExpr. A column
with its own ON ERROR clause is unaffected. EXISTS columns are not
covered by those syntax rules, so they keep their FALSE ON ERROR
default. A JSON_TABLE stored in a view is now deparsed with the
previously implicit ERROR ON ERROR shown explicitly on the affected
columns, which is a semantically equivalent, round-trip-stable form.
86ab7f4c72 introduced this cascade as part of the PLAN clause and
af6fad879f reverted it, on the grounds that it should be a deliberate
and separately documented change rather than a side effect of an
unrelated feature. This is that change.
Do not back-patch. A query that specifies ERROR ON ERROR at the table
level and relies on its columns still yielding NULL now raises an error
instead of returning rows. Nothing is silently wrong in the released
branches: the table-level clause is ignored for columns consistently,
and that is what the documentation has promised since PostgreSQL 17, so
users could reasonably have depended on it.
Discussion: https://postgr.es/m/CAPpHfdtDXseGhrL14a2asOkwGnFTVrDk8SQc3iWZgWEEhXXMGw%40mail.gmail.com
Reviewed-by: Nikita Malakhov <hukutoc@gmail.com>
SHMEM_ATTACH_UNKNOWN_SIZE can be passed as argument to
ShmemRequestStruct() when the caller wants to attach to an existing
shared memory structure whose size it doesn't know. If the shared
memory structure doesn't exist, the request should fail, but instead,
ProcessShmemRequestsAfterStartup() tried to create the structure with
size = -1. That led to integer overflow in ShmemAllocRaw() and memory
corruption.
Fix by rejecting requests with SHMEM_ATTACH_UNKNOWN_SIZE when the
structure doesn't exist. Also add an integer overflow check in
ShmemAllocRaw(), to protect from this kind of confusion or simply too
large requests. Also document SHMEM_ATTACH_UNKNOWN_SIZE.
Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>
Discussion: https://www.postgresql.org/message-id/CAExHW5u_fTsOAS85kG981Vu6eR1GV-344rup6zYew7xMjEDREw@mail.gmail.com
Backpatch-through: 19
The reference page said WAIT "cannot be used in contexts where such a
snapshot must remain active, including transactions running at isolation
levels higher than READ COMMITTED", which reads as a blanket ban at those
levels. It is not one. The rule the code enforces is that the current
transaction must hold no snapshot, and a transaction-snapshot-mode
transaction has no snapshot until its first query: WaitStmt is exempt in
PlannedStmtRequiresSnapshot(), so the command itself does not take one.
WAIT therefore works before that first query, which is the order an
application wants anyway, and it is what 447aae13b0 set out to allow by
keeping the wait snapshot-free.
State the rule in terms of the snapshot rather than the isolation level,
say that WAIT acquires none of its own and name the other ways to hold
one, and give the consequence of holding one: it can delay replay, which
standby_replay waits for and which the other standby modes ultimately
depend on as well.
The error for a held snapshot named the internal snapshot states instead
of saying what the user did, and attached the isolation-level
explanation as DETAIL unconditionally, so a READ COMMITTED session with a
cursor was told its isolation level was too high. Report what is held
rather than how it is held, and emit the DETAIL only when the isolation
level is in fact the reason.
Reported-by: Sami Imseih <samimseih.pg@gmail.com>
Reported-by: Xuneng Zhou <xunengzhou@gmail.com>
Reviewed-by: Sami Imseih <samimseih.pg@gmail.com>
Reviewed-by: Xuneng Zhou <xunengzhou@gmail.com>
Discussion: https://postgr.es/m/CAN12%2BYJddgAZzA36JZFX%2BYrnNR7Uubp9qWFz6ASs3v0kL30AMA%40mail.gmail.com
Backpatch-through: 19
The reference page and the read-your-writes section both presented
WAIT in standby_replay mode as enough for read-your-writes, without
saying which LSN to wait for. Read as written, an LSN taken after the
last statement but before COMMIT would do, and it would not: the standby
position WAIT compares against is the end of the last replayed record, so
a target that falls inside the COMMIT record is satisfied while that
record is still unapplied.
Say that the target must be at or after the end of the write
transaction's COMMIT record, and note in the example that autocommit is
what puts the UPDATE's commit before the call that reads the position.
Author: Sami Imseih <samimseih.pg@gmail.com>
Reviewed-by: Xuneng Zhou <xunengzhou@gmail.com>
Reviewed-by: Alexander Korotkov <aekorotkov@gmail.com>
Discussion: https://postgr.es/m/CAN12%2BYKN7gXb1o9bDWgfA%3D6wjDrqNEgt%2BrXJiAWXgPCXQ5TqpA%40mail.gmail.com
Backpatch-through: 19
The description of NO_THROW covered what the option does but not when an
application would want it. Explain that it suits callers that treat
timeout or not in recovery as expected results to be handled as part of
ordinary control flow, note the obligation to check the returned status,
and give the reason to omit the option.
Be precise about the option's scope: it changes only how those two wait
outcomes are reported. Soften the description's claim that the command
"returns a status string instead of throwing errors" accordingly, and say
that invalid input still raises an error, as does any condition checked
before the wait begins. Name two of those conditions as examples rather
than listing them, since the set grows.
Author: Rithvika Devisetti <devisettirithvika@gmail.com>
Reported-by: Peter Eisentraut <peter@eisentraut.org>
Reviewed-by: Kiran Kaki <itskkpg@gmail.com>
Reviewed-by: Xuneng Zhou <xunengzhou@gmail.com>
Reviewed-by: Satyanarayana Narlapuram <satyanarlapuram@gmail.com>
Reviewed-by: Alexander Korotkov <aekorotkov@gmail.com>
Discussion: https://postgr.es/m/14415b05-16d0-446f-ba7d-c891db8b3406%40eisentraut.org
Backpatch-through: 19
Up until now, pg_plan_advice has had a feature that allows an advice
target to mention a partition name but omit the partition schema;
that is, something like SEQ_SCAN(foo/bar) forces a sequential scan on
every child of table foo whose partition name is bar, regardless of
the schema in which bar appears. However, that feature turns out to
have a nasty design flaw: while advice enforcement handles this case
just fine, the advice feedback code doesn't know about it and will
mark such advice as "matched, failed" even when everything worked
perfectly. Unfortunately, there seems to be no simple code fix for
this problem.
Since the release of PostgreSQL 19 is imminent, take the conservative
course and revert this feature of pg_plan_advice. In other words,
require the partition schema whenever the partition name is present.
You must now write e.g. SEQ_SCAN(foo/public.bar) rather than just
SEQ_SCAN(foo/bar). This doesn't affect any cases where automatically
generated advice is supplied, since generated advice has always
included the partition schema anyway. Manually written advice will
have to conform to the new, stricter rule. For a later release, we
can consider whether to again relax this restriction in some way,
but now is not the time to design new things.
Reported-by: Noah Misch <noah@leadboat.com>
Backpatch-through: 19
Discussion: https://postgr.es/m/CA+TgmoYmXy-jiP5qDhqNEiYFEBzQsArO6O2d9E8szNZqi1bePQ@mail.gmail.com
Commit 5f3a49abf7 changed the primary status update message to report the
end of the last inserted WAL record instead of the WAL write position.
However, the protocol documentation still described the WAL position field as
the latest WAL write position, which no longer matches the message sent by
walsender.
Update the description of the WAL position field to indicate the end of the
last inserted WAL record, matching the implementation.
Author: Fujii Masao <masao.fujii@gmail.com>
Reviewed-by: Ayush Tiwari <ayushtiwari.slg01@gmail.com>
Discussion: https://postgr.es/m/CAHGQGwFn2QfpHDR6cWxd98tqNb5vyguEis8rtqnqidFR36p4QQ@mail.gmail.com
Backpatch-through: 19
Commit 0c5d626961 was confused about which indexes are potentially
problematic to rebuild: we thought only !indisready indexes could fail
to rebuild, and thus would attempt to rebuild !indisvalid ones that are
"ready" (that is, being maintained by DML but not yet completed).
However, that's wrong, because it could be that the validation phase of
such an index failed or was interrupted partway through. So, have
REPACK throw an error with all !indisvalid indexes, not just those
!indisready.
(!indisready implies !indisvalid, so this is an expansion of what
happened with the previous commit.)
Reported-by: shihao zhong <zhong950419@gmail.com>
Backpatch-through: 19
Discussion: https://postgr.es/m/CAGRkXqRA4YetT8Nu3+1-45+ffTwi2bd1d1eDFME39Y9OPRMytQ@mail.gmail.com
List of commits reverted:
8e72d914c5 Add UPDATE/DELETE FOR PORTION OF
b6ccd30d8f Add isolation tests for UPDATE/DELETE FOR PORTION OF
33db6c4baf Fix DELETE/UPDATE FOR PORTION OF with rules
7b22f15a01 Add psql tab completion for FOR PORTION OF clause
7ca8c94296 Fix FOR PORTION OF with non-updatable view columns
993a7aa0e4 Fix cross-leftover pollution in FOR PORTION OF insert triggers
7ac030d5b1 Require UPDATE permission on FOR PORTION OF column
f0aedc7cb0 Small terminology fixes in comments
7d13b03a2e Fix FOR PORTION OF for inheritance children
a272a58b94 Move FOR PORTION OF volatile check into planner
a40fdf6588 Reject child partition FDWs in FOR PORTION OF
bc3ae886a7 Forbid FOR PORTION OF with WHERE CURRENT OF
e994f956e4 Forbid generated columns in FOR PORTION OF
dfce19c230 Forbid FOR PORTION OF on views with INSTEAD OF triggers
20fe90bbb9 Fix RLS checks for FOR PORTION OF leftover rows
9170c8b716 Test what BEFORE UPDATE triggers do to FOR PORTION OF
7090c696cc Deparse FOR PORTION OF using the range column's current name.
62d1a5f8be Avoid RETURNING side effects for FOR PORTION OF leftovers.
9924e2d900 Enforce WITH CHECK OPTION on DELETE FOR PORTION OF leftovers
c16a1b4f2c Fix error code for null FOR PORTION OF target
610976420a Fix assertion failures in DELETE FOR PORTION OF tuple routing
438f027238 Resolve untyped parameters in FOR PORTION OF FROM/TO bounds
140fdfcdf1 Don't evaluate the FOR PORTION OF target under EXPLAIN
Discussion: https://www.postgresql.org/message-id/62bf70cc-dc33-4188-8070-a8c177bb61de%40eisentraut.org
A TOAST table whose chunk_id column is oid8 is now fed a 64-bit object
ID thanks to the introduction of a new vartag_external called
VARTAG_ONDISK_OID8, coupled with a new varatt_external_oid8. The only
difference between the existing varatt_external_oid and the new
varatt_external_oid8 is an extra 4 bytes to store 8 bytes worth an
object ID, spread across two uint32 to force alignement as memcmp may be
used for TOAST pointer comparison (line varatt_external_oid).
There is a one-one mapping between the vartag_external and the type of a
chunk_id:
- An oid8 chunk_id implies the use of VARTAG_ONDISK_OID8.
- An oid chunk_id implies the use of VARTAG_ONDISK_OID.
There are a couple of changes worth noting:
- In toast_tuple_init(), for the comparison of old and new external
values with a memcmp over VARSIZE_EXTERNAL(old_value). Now that
multiple types of external on-disk TOAST pointers are possible, we
need to add a check based on the vartag used by an old or new value.
One of the tests added by c68cba09dd cover this case.
- In toast_tuple_find_biggest_attribute(), the initial "biggest_size"
depends on the type of chunk_id. This needs a change in RelationData to
track the chunk_id type; this cached data is filled on demand.
- In detoast.h, toast_external_info_get() and toast_external_data act as
a translation layer to retrieve all the data associated to an Oid or an
Oid8 TOAST pointer, without the need to worry about the vartag dealt
with. The introduction of this function and structure has reduced a lot
of code churn that existed in the initial versions of this patch, as per
a suggestion from Bharath Rupireddy.
- amcheck is updated to check the vartag <-> chunk_id type mapping
requirements, handling both vartags depending on the underlying type of
chunk_id on the TOAST table.
Bump catalog version, due to at least the change of VARTAG_SIZE().
(This counts for the bump forgotten in a625fc570c22..)
Author: Michael Paquier <michael@paquier.xyz>
Reviewed-by: Greg Burd <greg@burd.me>
Reviewed-by: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Reviewed-by: Yugo Nagata <nagata@sraoss.co.jp>
Discussion: https://postgr.es/m/af19kUjwjhaoUTLn@paquier.xyz
This commit adds the possibility to define TOAST tables with oid8 as
their chunk_id, based on the reloption toast_value_type. All the
external TOAST pointers still rely on varatt_external_oid and a single
vartag. The values inserted in the oid8 TOAST tables are fed from the
OID8 value generator, casted to OID for now, as we do not have a
vartag_external able to store Oid8 values yet.
An upcoming commit will add support for a new vartag_external and its
associated structures, with the code being able to use a different
on-disk external TOAST pointer depending on the attribute type of
chunk_id defined in TOAST relations.
All the changes done here are mechanical, mostly around the TOAST code
that needs to be able to do chunk_id lookups based on the two types now
supported.
Bump catalog version.
Author: Michael Paquier <michael@paquier.xyz>
Reviewed-by: Greg Burd <greg@burd.me>
Reviewed-by: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Reviewed-by: Yugo Nagata <nagata@sraoss.co.jp>
Discussion: https://postgr.es/m/af19kUjwjhaoUTLn@paquier.xy
A set of documentation updates for the online checksums work, found via
manual as well as LLM-guided review.
* The state diagram was re-done to be improve readability and be
more in line with the look and feel of other diagrams.
* Paragraph about mismatched states moved to its own sect2.
* The documentation for the functions to enable/ disable checksums
failed to mention that they are superuser only.
* The documentation for the checksums progress reporting had
accidentally omitted the work "fork" and for the blocks_* columns.
The values are per fork but the documentation made it seem they
were per relation.
* Add the state image to the meson buildfile
Reported-by: Heikki Linnakangas <hlinnaka@iki.fi>
Reported-by: Noah Misch <noah@leadboat.com>
Discussion: https://postgr.es/m/5b08b4d1-0982-4b77-bac5-3bdffc6583f5@iki.fi
Backpatch-through: 19
check_control_files() detected a chain whose backups were taken under
different data checksum states, warned, and proceeded. The output
directory keeps the last backup's control file, so a full backup taken
with checksums off combined with an incremental taken after an offline
enable produces a cluster whose control file says "on" while most of
its blocks carry no checksums; it starts, and then every connection
dies on the first unchecksummed catalog page. An offline enable
between two backups of a chain is all it takes, since it rewrites
every page without logging anything, so the incremental backup does
not re-ship the pages.
Turn the warning into an error, matching what pg_rewind does for the
equivalent combinations. The check stays asymmetric on purpose: when
the last backup was taken with checksums off, stale checksums from an
earlier backup are never verified, and that chain remains usable.
Author: Zsolt Parragi <zsolt.parragi@percona.com>
Reviewed-by: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Reviewed-by: Daniel Gustafsson <daniel@yesql.se>
Discussion: https://postgr.es/m/anwm6UPxoVS41QA2@bdtpg
Backpatch-through: 19
pg_rewind installs the source's control file on the target, but every
block it does not copy keeps the target's content. With checksums
enabled on the target and disabled on the source, the rewound server
claims enabled checksums while the blocks copied from the source have
none, and fails checksum verification as soon as it reads them; in the
worst case every connection attempt dies on an unverifiable catalog
page.
Comparing the control files is not enough. Replay on the rewound
server resumes from the last common checkpoint and adopts the data
checksum state recorded there, so an offline disable on the source
after the divergence leaves both control files saying "off" while the
rewound server still resumes with verification enabled. Check the
state carried by the divergence checkpoint as well, which pg_rewind
already reads.
Refuse both cases, and an interrupted online transition on either
side, same as pg_checksums does. The opposite mismatch, checksums on
the source only, is allowed with a warning: recovery under the backup
label adopts the divergence state, so the rewound server keeps
checksums disabled (or converges through the replayed WAL if the
source enabled them online), and no page can fail verification.
Author: Zsolt Parragi <zsolt.parragi@percona.com>
Reviewed-by: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Reviewed-by: Daniel Gustafsson <daniel@yesql.se>
Discussion: https://postgr.es/m/anwm6UPxoVS41QA2@bdtpg
Backpatch-through: 19
An inprogress-on or inprogress-off control file means an online
transition was cut short, running the offline tool on top of it mixes
two procedures. Refuse it in every mode, with a hint pointing at the
start-stop cycle (or, on a standby, at letting replication finish)
that resets the state. Also print that the change applies to one
data directory only, with standby-aware wording, since in a
replication setup the same change must be made on every node.
On a primary, inprogress-on cannot survive a graceful stop: the
checksums launcher resolves it from its exit cleanup, and a crashed
primary is already rejected by the existing "cluster must be shut
down" check. inprogress-off can, however, it is set by the backend
running pg_disable_data_checksums(), and a fast shutdown arriving
between its two barriers leaves it behind in a cleanly shut down
control file, where the next start-stop cycle resolves it at end of
recovery. A standby can be stopped with either state, since it has
no launcher and only carries forward whatever state the last replayed
WAL record left it in, with a restartpoint persisting that as-is.
The standby is also the deterministic way to reach the new guard,
which is why the test coverage uses one; the primary window would
need an injection point between the two barriers.
The pg_checksums page said the tool still processes all relation files
regardless of an interrupted online transition; describe the refusal
instead.
Author: Zsolt Parragi <zsolt.parragi@percona.com>
Reviewed-by: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Reviewed-by: Daniel Gustafsson <daniel@yesql.se>
Discussion: https://postgr.es/m/anwm6UPxoVS41QA2@bdtpg
Backpatch-through: 19
