database/postgres
1
0
Fork 0
mirror of https://github.com/postgres/postgres.git synced 2025-07-15 19:21:59 +03:00
Code Activity

Improve VACUUM and ANALYZE by avoiding early lock queue

Browse Source 
A caller of VACUUM can perform early lookup obtention which can cause
other sessions to block on the request done, causing potentially DOS
attacks as even a non-privileged user can attempt a vacuum fill of a
critical catalog table to block even all incoming connection attempts.

Contrary to TRUNCATE, a client could attempt a system-wide VACUUM after
building the list of relations to VACUUM, which can cause vacuum_rel()
or analyze_rel() to try to lock the relation but the operation would
just block.  When the client specifies a list of relations and the
relation needs to be skipped, ownership checks are done when building
the list of relations to work on, preventing a later lock attempt.

vacuum_rel() already had the sanity checks needed, except that those
were applied too late.  This commit refactors the code so as relation
skips are checked beforehand, making it safer to avoid too early locks,
for both manual VACUUM with and without a list of relations specified.

An isolation test is added emulating the fact that early locks do not
happen anymore, issuing a WARNING message earlier if the user calling
VACUUM is not a relation owner.

When a partitioned table is listed in a manual VACUUM or ANALYZE
command, its full list of partitions is fetched, all partitions get
added to the list to work on, and then each one of them is processed one
by one, with ownership checks happening at the later phase of
vacuum_rel() or analyze_rel().  Trying to do early ownership checks for
each partition is proving to be tedious as this would result in deadlock
risks with lock upgrades, and skipping all partitions if the listed
partitioned table is not owned would result in a behavior change
compared to how Postgres 10 has implemented vacuum for partitioned
tables.  The original problem reported related to early lock queue for
critical relations is fixed anyway, so priority is given to avoiding a
backward-incompatible behavior.

Reported-by: Lloyd Albin, Jeremy Schneider
Author: Michael Paquier
Reviewed by: Nathan Bossart, Kyotaro Horiguchi
Discussion: https://postgr.es/m/152512087100.19803.12733865831237526317@wrigleys.postgresql.org
Discussion: https://postgr.es/m/20180812222142.GA6097@paquier.xyz
This commit is contained in:
Michael Paquier
2018-08-27 09:11:12 +09:00
parent 18e586741b
commit a556549d7e
6 changed files with 328 additions and 60 deletions
Download Patch File Download Diff File Expand all files Collapse all files

149
src/test/isolation/expected/vacuum-conflict.out Normal file
View File

@ -0,0 +1,149 @@
Parsed test spec with 2 sessions
starting permutation: s1_begin s1_lock s2_auth s2_vacuum s1_commit s2_reset
step s1_begin: BEGIN;
step s1_lock: LOCK vacuum_tab IN SHARE UPDATE EXCLUSIVE MODE;
step s2_auth: SET ROLE regress_vacuum_conflict;
WARNING: skipping "vacuum_tab" --- only table or database owner can vacuum it
step s2_vacuum: VACUUM vacuum_tab;
step s1_commit: COMMIT;
step s2_reset: RESET ROLE;
starting permutation: s1_begin s2_auth s2_vacuum s1_lock s1_commit s2_reset
step s1_begin: BEGIN;
step s2_auth: SET ROLE regress_vacuum_conflict;
WARNING: skipping "vacuum_tab" --- only table or database owner can vacuum it
step s2_vacuum: VACUUM vacuum_tab;
step s1_lock: LOCK vacuum_tab IN SHARE UPDATE EXCLUSIVE MODE;
step s1_commit: COMMIT;
step s2_reset: RESET ROLE;
starting permutation: s1_begin s2_auth s1_lock s2_vacuum s1_commit s2_reset
step s1_begin: BEGIN;
step s2_auth: SET ROLE regress_vacuum_conflict;
step s1_lock: LOCK vacuum_tab IN SHARE UPDATE EXCLUSIVE MODE;
WARNING: skipping "vacuum_tab" --- only table or database owner can vacuum it
step s2_vacuum: VACUUM vacuum_tab;
step s1_commit: COMMIT;
step s2_reset: RESET ROLE;
starting permutation: s2_auth s2_vacuum s1_begin s1_lock s1_commit s2_reset
step s2_auth: SET ROLE regress_vacuum_conflict;
WARNING: skipping "vacuum_tab" --- only table or database owner can vacuum it
step s2_vacuum: VACUUM vacuum_tab;
step s1_begin: BEGIN;
step s1_lock: LOCK vacuum_tab IN SHARE UPDATE EXCLUSIVE MODE;
step s1_commit: COMMIT;
step s2_reset: RESET ROLE;
starting permutation: s1_begin s1_lock s2_auth s2_analyze s1_commit s2_reset
step s1_begin: BEGIN;
step s1_lock: LOCK vacuum_tab IN SHARE UPDATE EXCLUSIVE MODE;
step s2_auth: SET ROLE regress_vacuum_conflict;
WARNING: skipping "vacuum_tab" --- only table or database owner can analyze it
step s2_analyze: ANALYZE vacuum_tab;
step s1_commit: COMMIT;
step s2_reset: RESET ROLE;
starting permutation: s1_begin s2_auth s2_analyze s1_lock s1_commit s2_reset
step s1_begin: BEGIN;
step s2_auth: SET ROLE regress_vacuum_conflict;
WARNING: skipping "vacuum_tab" --- only table or database owner can analyze it
step s2_analyze: ANALYZE vacuum_tab;
step s1_lock: LOCK vacuum_tab IN SHARE UPDATE EXCLUSIVE MODE;
step s1_commit: COMMIT;
step s2_reset: RESET ROLE;
starting permutation: s1_begin s2_auth s1_lock s2_analyze s1_commit s2_reset
step s1_begin: BEGIN;
step s2_auth: SET ROLE regress_vacuum_conflict;
step s1_lock: LOCK vacuum_tab IN SHARE UPDATE EXCLUSIVE MODE;
WARNING: skipping "vacuum_tab" --- only table or database owner can analyze it
step s2_analyze: ANALYZE vacuum_tab;
step s1_commit: COMMIT;
step s2_reset: RESET ROLE;
starting permutation: s2_auth s2_analyze s1_begin s1_lock s1_commit s2_reset
step s2_auth: SET ROLE regress_vacuum_conflict;
WARNING: skipping "vacuum_tab" --- only table or database owner can analyze it
step s2_analyze: ANALYZE vacuum_tab;
step s1_begin: BEGIN;
step s1_lock: LOCK vacuum_tab IN SHARE UPDATE EXCLUSIVE MODE;
step s1_commit: COMMIT;
step s2_reset: RESET ROLE;
starting permutation: s1_begin s2_grant s1_lock s2_auth s2_vacuum s1_commit s2_reset
step s1_begin: BEGIN;
step s2_grant: ALTER TABLE vacuum_tab OWNER TO regress_vacuum_conflict;
step s1_lock: LOCK vacuum_tab IN SHARE UPDATE EXCLUSIVE MODE;
step s2_auth: SET ROLE regress_vacuum_conflict;
step s2_vacuum: VACUUM vacuum_tab; <waiting ...>
step s1_commit: COMMIT;
step s2_vacuum: <... completed>
step s2_reset: RESET ROLE;
starting permutation: s1_begin s2_grant s2_auth s2_vacuum s1_lock s1_commit s2_reset
step s1_begin: BEGIN;
step s2_grant: ALTER TABLE vacuum_tab OWNER TO regress_vacuum_conflict;
step s2_auth: SET ROLE regress_vacuum_conflict;
step s2_vacuum: VACUUM vacuum_tab;
step s1_lock: LOCK vacuum_tab IN SHARE UPDATE EXCLUSIVE MODE;
step s1_commit: COMMIT;
step s2_reset: RESET ROLE;
starting permutation: s1_begin s2_grant s2_auth s1_lock s2_vacuum s1_commit s2_reset
step s1_begin: BEGIN;
step s2_grant: ALTER TABLE vacuum_tab OWNER TO regress_vacuum_conflict;
step s2_auth: SET ROLE regress_vacuum_conflict;
step s1_lock: LOCK vacuum_tab IN SHARE UPDATE EXCLUSIVE MODE;
step s2_vacuum: VACUUM vacuum_tab; <waiting ...>
step s1_commit: COMMIT;
step s2_vacuum: <... completed>
step s2_reset: RESET ROLE;
starting permutation: s2_grant s2_auth s2_vacuum s1_begin s1_lock s1_commit s2_reset
step s2_grant: ALTER TABLE vacuum_tab OWNER TO regress_vacuum_conflict;
step s2_auth: SET ROLE regress_vacuum_conflict;
step s2_vacuum: VACUUM vacuum_tab;
step s1_begin: BEGIN;
step s1_lock: LOCK vacuum_tab IN SHARE UPDATE EXCLUSIVE MODE;
step s1_commit: COMMIT;
step s2_reset: RESET ROLE;
starting permutation: s1_begin s2_grant s1_lock s2_auth s2_analyze s1_commit s2_reset
step s1_begin: BEGIN;
step s2_grant: ALTER TABLE vacuum_tab OWNER TO regress_vacuum_conflict;
step s1_lock: LOCK vacuum_tab IN SHARE UPDATE EXCLUSIVE MODE;
step s2_auth: SET ROLE regress_vacuum_conflict;
step s2_analyze: ANALYZE vacuum_tab; <waiting ...>
step s1_commit: COMMIT;
step s2_analyze: <... completed>
step s2_reset: RESET ROLE;
starting permutation: s1_begin s2_grant s2_auth s2_analyze s1_lock s1_commit s2_reset
step s1_begin: BEGIN;
step s2_grant: ALTER TABLE vacuum_tab OWNER TO regress_vacuum_conflict;
step s2_auth: SET ROLE regress_vacuum_conflict;
step s2_analyze: ANALYZE vacuum_tab;
step s1_lock: LOCK vacuum_tab IN SHARE UPDATE EXCLUSIVE MODE;
step s1_commit: COMMIT;
step s2_reset: RESET ROLE;
starting permutation: s1_begin s2_grant s2_auth s1_lock s2_analyze s1_commit s2_reset
step s1_begin: BEGIN;
step s2_grant: ALTER TABLE vacuum_tab OWNER TO regress_vacuum_conflict;
step s2_auth: SET ROLE regress_vacuum_conflict;
step s1_lock: LOCK vacuum_tab IN SHARE UPDATE EXCLUSIVE MODE;
step s2_analyze: ANALYZE vacuum_tab; <waiting ...>
step s1_commit: COMMIT;
step s2_analyze: <... completed>
step s2_reset: RESET ROLE;
starting permutation: s2_grant s2_auth s2_analyze s1_begin s1_lock s1_commit s2_reset
step s2_grant: ALTER TABLE vacuum_tab OWNER TO regress_vacuum_conflict;
step s2_auth: SET ROLE regress_vacuum_conflict;
step s2_analyze: ANALYZE vacuum_tab;
step s1_begin: BEGIN;
step s1_lock: LOCK vacuum_tab IN SHARE UPDATE EXCLUSIVE MODE;
step s1_commit: COMMIT;
step s2_reset: RESET ROLE;

1
src/test/isolation/isolation_schedule
View File

@ -66,6 +66,7 @@ test: async-notify
test: vacuum-reltuples
test: timeouts
test: vacuum-concurrent-drop
test: vacuum-conflict
test: predicate-hash
test: predicate-gist
test: predicate-gin

51
src/test/isolation/specs/vacuum-conflict.spec Normal file
View File

@ -0,0 +1,51 @@
# Tests for locking conflicts with VACUUM and ANALYZE commands.
setup
{
CREATE ROLE regress_vacuum_conflict;
CREATE TABLE vacuum_tab (a int);
}
teardown
{
DROP TABLE vacuum_tab;
DROP ROLE regress_vacuum_conflict;
}
session "s1"
step "s1_begin" { BEGIN; }
step "s1_lock" { LOCK vacuum_tab IN SHARE UPDATE EXCLUSIVE MODE; }
step "s1_commit" { COMMIT; }
session "s2"
step "s2_grant" { ALTER TABLE vacuum_tab OWNER TO regress_vacuum_conflict; }
step "s2_auth" { SET ROLE regress_vacuum_conflict; }
step "s2_vacuum" { VACUUM vacuum_tab; }
step "s2_analyze" { ANALYZE vacuum_tab; }
step "s2_reset" { RESET ROLE; }
# The role doesn't have privileges to vacuum the table, so VACUUM should
# immediately skip the table without waiting for a lock.
permutation "s1_begin" "s1_lock" "s2_auth" "s2_vacuum" "s1_commit" "s2_reset"
permutation "s1_begin" "s2_auth" "s2_vacuum" "s1_lock" "s1_commit" "s2_reset"
permutation "s1_begin" "s2_auth" "s1_lock" "s2_vacuum" "s1_commit" "s2_reset"
permutation "s2_auth" "s2_vacuum" "s1_begin" "s1_lock" "s1_commit" "s2_reset"
# Same as previously for ANALYZE
permutation "s1_begin" "s1_lock" "s2_auth" "s2_analyze" "s1_commit" "s2_reset"
permutation "s1_begin" "s2_auth" "s2_analyze" "s1_lock" "s1_commit" "s2_reset"
permutation "s1_begin" "s2_auth" "s1_lock" "s2_analyze" "s1_commit" "s2_reset"
permutation "s2_auth" "s2_analyze" "s1_begin" "s1_lock" "s1_commit" "s2_reset"
# The role has privileges to vacuum the table, VACUUM will block if
# another session holds a lock on the table and succeed in all cases.
permutation "s1_begin" "s2_grant" "s1_lock" "s2_auth" "s2_vacuum" "s1_commit" "s2_reset"
permutation "s1_begin" "s2_grant" "s2_auth" "s2_vacuum" "s1_lock" "s1_commit" "s2_reset"
permutation "s1_begin" "s2_grant" "s2_auth" "s1_lock" "s2_vacuum" "s1_commit" "s2_reset"
permutation "s2_grant" "s2_auth" "s2_vacuum" "s1_begin" "s1_lock" "s1_commit" "s2_reset"
# Same as previously for ANALYZE
permutation "s1_begin" "s2_grant" "s1_lock" "s2_auth" "s2_analyze" "s1_commit" "s2_reset"
permutation "s1_begin" "s2_grant" "s2_auth" "s2_analyze" "s1_lock" "s1_commit" "s2_reset"
permutation "s1_begin" "s2_grant" "s2_auth" "s1_lock" "s2_analyze" "s1_commit" "s2_reset"
permutation "s2_grant" "s2_auth" "s2_analyze" "s1_begin" "s1_lock" "s1_commit" "s2_reset"