database/postgres
1
0
Fork 0
mirror of https://github.com/postgres/postgres.git synced 2026-10-23 13:29:09 +03:00
Code Activity

Prevent WAIT FOR LSN from deadlocking recovery on held locks

Browse Source 
A backend waiting for WAL replay can retain locks acquired by earlier
statements.  If the startup process needs one of those locks, directly or
through another backend, before reaching the target LSN, a deadlock can
arise: startup waits for the backend to release the lock, while the backend
waits for startup to advance replay.  The lock manager records the
backend's held locks, but WAIT FOR LSN does not register its dependency on
replay as a lock wait.  The deadlock detector therefore cannot see the
complete cycle.  With unlimited standby conflict delays and no other
timeout or cancellation, this deadlock can persist indefinitely.

Write and flush waits are restricted as well.  Their positions are floored
by the replay position, so without an active walreceiver the startup
process can be their only source of progress.  If a held lock blocks
replay, these waits can form the same cycle: the backend waits for replay
to advance, while replay waits for the backend to release the lock.

Streaming does advance them independently, but only while WAL keeps
arriving.  If reception stops before the target is reached, a blocked
startup process cannot restart the walreceiver.  It also cannot replay
newer checkpoint records needed to advance restartpoints and recycle WAL,
so continued reception can exhaust available space in pg_wal before the
target is reached.  An active receiver at the start of the wait therefore
does not guarantee that the wait can finish while replay remains blocked.

Reject an unsatisfied standby_replay, standby_write, or standby_flush wait
while recovery is active when the backend already holds a granted
heavyweight lock.  This conservative restriction covers direct
relation-lock cycles and indirect cycles involving advisory locks.  It
also rejects some write and flush waits that an active receiver could
satisfy.  Requests whose target is observed as already reached are exempt
from this check, as are primary_flush requests and requests issued after
recovery has ended.  Existing snapshot and recovery-state checks still
apply.

Report one of the held locks so the user can find it, using the same
description the deadlock report uses.

Add replay-mode tests for relation and advisory locks and for the
already-reached case, and document the restriction along with the
recommended usage pattern.

Author: Xuneng Zhou <xunengzhou@gmail.com>
Discussion: https://postgr.es/m/CABPTF7U0gW5%2B-4oL7-qdML-yerZxUb7ku4QXp7JxCYo0qyJ_Tw%40mail.gmail.com
Reviewed-by: Alexander Korotkov <aekorotkov@gmail.com>
Backpatch-through: 19
This commit is contained in:
Alexander Korotkov
2026-09-12 23:29:30 +03:00
parent b3609cd2db
commit f7c0929170
5 changed files with 164 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

40
doc/src/sgml/ref/wait_for.sgml
View File

@@ -225,6 +225,46 @@ WAIT FOR LSN '<replaceable class="parameter">lsn</replaceable>'
at isolation levels higher than <literal>READ COMMITTED</literal>.
</para>
<para>
While recovery is in progress, a wait in <literal>standby_replay</literal>
(the default), <literal>standby_write</literal>, or
<literal>standby_flush</literal> mode is rejected when the session already
holds a lock and the target <parameter>lsn</parameter> has not been reached
yet. Such a lock can make the startup process wait for this session, either
directly or through another session, while this session waits for the
startup process to advance recovery. That cycle involves no lock wait on
this side, so deadlock detection does not see it and nothing breaks it. A
wait whose target has already been reached returns immediately and is
therefore always allowed.
</para>
<para>
Issue <command>WAIT FOR</command> outside a transaction block, or as the
first statement of one, before running anything that takes locks. That is
also the natural order for the read-your-writes pattern shown in the
examples below: wait for the target <parameter>lsn</parameter> first, then
run the queries that have to see it. Note that a lock taken by an earlier
statement is still held at <literal>READ COMMITTED</literal>, even though
its snapshot is gone, so a wait placed after such a statement is rejected
even when the isolation level permits it.
</para>
<para>
The restriction covers <literal>standby_write</literal> and
<literal>standby_flush</literal> as well, even though streaming replication
can advance those positions without the startup process. Both positions are
at least the replay position, so without an active walreceiver replay can
be their only source of progress. If a held lock blocks replay, the session
waits for replay to advance while replay waits for the session to release
the lock. Under streaming replication the positions advance independently
only while WAL keeps arriving. If reception stops before the target is
reached, a blocked startup process cannot restart the walreceiver. It also
cannot replay newer checkpoint records needed to advance restartpoints and
recycle WAL, so <filename>pg_wal</filename> can fill up and reception can
stop before the target is reached. The restriction therefore also applies
when streaming is active at the start of the wait.
</para>
<para>
<command>WAIT FOR</command> waits until the specified
<parameter>lsn</parameter> is reached according to the specified

55
src/backend/commands/wait.c
View File

@@ -23,6 +23,8 @@
#include "commands/wait.h"
#include "executor/executor.h"
#include "parser/parse_node.h"
#include "storage/lmgr.h"
#include "storage/lock.h"
#include "storage/proc.h"
#include "utils/builtins.h"
#include "utils/guc.h"
@@ -194,6 +196,59 @@ ExecWaitStmt(ParseState *pstate, WaitStmt *stmt, bool isTopLevel,
"Use standby_flush mode on a standby server.")));
}
/*
* Conservatively reject an unsatisfied standby LSN wait while this
* backend holds a granted heavyweight lock. Recovery may need one of
* those locks, directly or through another backend, before replay can
* advance far enough to satisfy our wait. This can create a cycle: we
* wait for recovery, while recovery waits for us to release the lock.
*
* However, we do not register our dependency on WAL progress as a lock
* wait, so the deadlock detector cannot see the complete cycle. With
* unlimited recovery-conflict delays and no other timeout or
* cancellation, the cycle can persist indefinitely.
*
* Write and flush waits can also depend on startup. Without an active
* receiver, their replay floor can be their only source of progress, so
* holding a lock needed by replay can create the same cycle.
*
* Streaming can initially provide independent progress, but reception can
* stop before the target is reached. Restarting reception requires
* startup, and stalled replay prevents further advancement of
* restartpoints used to recycle old WAL, so continued reception can
* exhaust available space. An active receiver at the start of the wait
* therefore does not guarantee that the wait can finish while replay
* remains blocked.
*
* Apply the restriction to all standby modes, including some write and
* flush waits that an active receiver could satisfy while locks remain
* held. Requests whose target is observed as already reached are exempt
* from this restriction.
*/
if ((lsnType == WAIT_LSN_TYPE_STANDBY_REPLAY ||
lsnType == WAIT_LSN_TYPE_STANDBY_WRITE ||
lsnType == WAIT_LSN_TYPE_STANDBY_FLUSH) &&
RecoveryInProgress() &&
lsn > GetCurrentLSNForWaitType(lsnType))
{
LOCKTAG locktag;
if (GetAnyGrantedHeavyweightLock(&locktag))
{
StringInfoData locktagbuf;
initStringInfo(&locktagbuf);
DescribeLockTag(&locktagbuf, &locktag);
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("cannot wait for a standby LSN while holding locks"),
errdetail("This session holds a lock on %s, which could make recovery wait for this session while this session waits for recovery.",
locktagbuf.data),
errhint("Release the locks, or execute WAIT FOR before acquiring them.")));
}
}
/* Now wait for the LSN */
waitLSNResult = WaitForLSN(lsnType, lsn, timeout);

32
src/backend/storage/lmgr/lock.c
View File

@@ -673,6 +673,38 @@ LockHeldByMe(const LOCKTAG *locktag,
return false;
}
/*
* GetAnyGrantedHeavyweightLock -- find some heavyweight lock this backend owns
*
* Returns true and stores the tag of an arbitrary granted lock in *locktag if
* the local lock table represents any lock we own, false otherwise. Which
* lock is reported is unspecified when we hold several; callers use it only to
* give the user a starting point.
*
* A LOCALLOCK entry can remain after an unsuccessful lock acquisition, so only
* entries with a positive local hold count represent locks we own.
*/
bool
GetAnyGrantedHeavyweightLock(LOCKTAG *locktag)
{
HASH_SEQ_STATUS status;
LOCALLOCK *locallock;
hash_seq_init(&status, LockMethodLocalHash);
while ((locallock = (LOCALLOCK *) hash_seq_search(&status)) != NULL)
{
if (locallock->nLocks > 0)
{
*locktag = locallock->tag.lock;
hash_seq_term(&status);
return true;
}
}
return false;
}
#ifdef USE_ASSERT_CHECKING
/*
* GetLockMethodLocalHash -- return the hash of local locks, for modules that

1
src/include/storage/lock.h
View File

@@ -401,6 +401,7 @@ extern void LockReleaseCurrentOwner(LOCALLOCK **locallocks, int nlocks);
extern void LockReassignCurrentOwner(LOCALLOCK **locallocks, int nlocks);
extern bool LockHeldByMe(const LOCKTAG *locktag,
LOCKMODE lockmode, bool orstronger);
extern bool GetAnyGrantedHeavyweightLock(LOCKTAG *locktag);
#ifdef USE_ASSERT_CHECKING
extern HTAB *GetLockMethodLocalHash(void);
#endif

36
src/test/recovery/t/049_wait_for_lsn.pl
View File

@@ -431,6 +431,42 @@ $node_standby->psql(
ok( $stderr =~ /conflicting or redundant options/,
"get error for duplicate MODE parameter");
# An unsatisfied standby_replay wait must be rejected when the backend holds
# a lock. The relation lock covers the direct two-process cycle, while the
# advisory lock is the return edge in the indirect three-process cycle. The
# short timeout keeps these tests bounded if the check regresses.
$node_standby->psql(
'postgres', qq[
BEGIN;
SELECT count(*) FROM wait_test;
WAIT FOR LSN '${lsn3}' WITH (timeout '100ms');],
stderr => \$stderr);
like(
$stderr,
qr/cannot wait for a standby LSN while holding locks/,
"reject replay wait while holding a relation lock");
$node_standby->psql(
'postgres', qq[
BEGIN;
SELECT pg_advisory_xact_lock(42);
WAIT FOR LSN '${lsn3}' WITH (timeout '100ms');],
stderr => \$stderr);
like(
$stderr,
qr/cannot wait for a standby LSN while holding locks/,
"reject replay wait while holding an advisory lock");
# A wait whose target has already been reached returns without sleeping, so it
# cannot join a cycle and stays allowed regardless of the locks held.
my $reached = $node_standby->safe_psql(
'postgres', qq[
BEGIN;
LOCK TABLE wait_test IN ACCESS SHARE MODE;
WAIT FOR LSN '${lsn1}';]);
is($reached, 'success',
"allow an already-reached wait while holding a relation lock");
# 7a. Check the scenario of multiple standby_replay waiters. We make 5
# background psql sessions each waiting for a corresponding insertion. When
# waiting is finished, stored procedures logs if there are visible as many