Don't overlook indexes during parallel VACUUM.

Commit b4af70cb, which simplified state managed by VACUUM, performed
refactoring of parallel VACUUM in passing.  Confusion about the exact
details of the tasks that the leader process is responsible for led to
code that made it possible for parallel VACUUM to miss a subset of the
table's indexes entirely.  Specifically, indexes that fell under the
min_parallel_index_scan_size size cutoff were missed.  These indexes are
supposed to be vacuumed by the leader (alongside any parallel unsafe
indexes), but weren't vacuumed at all.  Affected indexes could easily
end up with duplicate heap TIDs, once heap TIDs were recycled for new
heap tuples.  This had generic symptoms that might be seen with almost
any index corruption involving structural inconsistencies between an
index and its table.

To fix, make sure that the parallel VACUUM leader process performs any
required index vacuuming for indexes that happen to be below the size
cutoff.  Also document the design of parallel VACUUM with these
below-size-cutoff indexes.

It's unclear how many users might be affected by this bug.  There had to
be at least three indexes on the table to hit the bug: a smaller index,
plus at least two additional indexes that themselves exceed the size
cutoff.  Cases with just one additional index would not run into
trouble, since the parallel VACUUM cost model requires two
larger-than-cutoff indexes on the table to apply any parallel
processing.  Note also that autovacuum was not affected, since it never
uses parallel processing.

Test case based on tests from a larger patch to test parallel VACUUM by
Masahiko Sawada.

Many thanks to Kamigishi Rei for her invaluable help with tracking this
problem down.

Author: Peter Geoghegan <pg@bowt.ie>
Author: Masahiko Sawada <sawada.mshk@gmail.com>
Reported-By: Kamigishi Rei <iijima.yun@koumakan.jp>
Reported-By: Andrew Gierth <andrew@tao11.riddles.org.uk>
Diagnosed-By: Andres Freund <andres@anarazel.de>
Bug: #17245
Discussion: https://postgr.es/m/17245-ddf06aaf85735f36@postgresql.org
Discussion: https://postgr.es/m/20211030023740.qbnsl2xaoh2grq3d@alap3.anarazel.de
Backpatch: 14-, where the refactoring commit appears.

src/test/regress/expected/vacuum_parallel.out

@@ -0,0 +1,49 @@
+SET max_parallel_maintenance_workers TO 4;
+SET min_parallel_index_scan_size TO '128kB';
+-- Bug #17245: Make sure that we don't totally fail to VACUUM individual indexes that
+-- happen to be below min_parallel_index_scan_size during parallel VACUUM:
+CREATE TABLE parallel_vacuum_table (a int) WITH (autovacuum_enabled = off);
+INSERT INTO parallel_vacuum_table SELECT i from generate_series(1, 10000) i;
+-- Parallel VACUUM will never be used unless there are at least two indexes
+-- that exceed min_parallel_index_scan_size.  Create two such indexes, and
+-- a third index that is smaller than min_parallel_index_scan_size.
+CREATE INDEX regular_sized_index ON parallel_vacuum_table(a);
+CREATE INDEX typically_sized_index ON parallel_vacuum_table(a);
+-- Note: vacuum_in_leader_small_index can apply deduplication, making it ~3x
+-- smaller than the other indexes
+CREATE INDEX vacuum_in_leader_small_index ON parallel_vacuum_table((1));
+-- Verify (as best we can) that the cost model for parallel VACUUM
+-- will make our VACUUM run in parallel, while always leaving it up to the
+-- parallel leader to handle the vacuum_in_leader_small_index index:
+SELECT EXISTS (
+SELECT 1
+FROM pg_class
+WHERE oid = 'vacuum_in_leader_small_index'::regclass AND
+  pg_relation_size(oid) <
+  pg_size_bytes(current_setting('min_parallel_index_scan_size'))
+) as leader_will_handle_small_index;
+ leader_will_handle_small_index 
+--------------------------------
+ t
+(1 row)
+
+SELECT count(*) as trigger_parallel_vacuum_nindexes
+FROM pg_class
+WHERE oid in ('regular_sized_index'::regclass, 'typically_sized_index'::regclass) AND
+  pg_relation_size(oid) >=
+  pg_size_bytes(current_setting('min_parallel_index_scan_size'));
+ trigger_parallel_vacuum_nindexes 
+----------------------------------
+                                2
+(1 row)
+
+-- Parallel VACUUM with B-Tree page deletions, ambulkdelete calls:
+DELETE FROM parallel_vacuum_table;
+VACUUM (PARALLEL 4, INDEX_CLEANUP ON) parallel_vacuum_table;
+-- Since vacuum_in_leader_small_index uses deduplication, we expect an
+-- assertion failure with bug #17245 (in the absence of bugfix):
+INSERT INTO parallel_vacuum_table SELECT i FROM generate_series(1, 10000) i;
+RESET max_parallel_maintenance_workers;
+RESET min_parallel_index_scan_size;
+-- Deliberately don't drop table, to get further coverage from tools like
+-- pg_amcheck in some testing scenarios

src/test/regress/parallel_schedule

@@ -96,6 +96,7 @@ test: rules psql psql_crosstab amutils stats_ext collate.linux.utf8
 # run by itself so it can run parallel workers
 test: select_parallel
 test: write_parallel
+test: vacuum_parallel
 
 # no relation related tests can be put in this group
 test: publication subscription

src/test/regress/sql/vacuum_parallel.sql

@@ -0,0 +1,46 @@
+SET max_parallel_maintenance_workers TO 4;
+SET min_parallel_index_scan_size TO '128kB';
+
+-- Bug #17245: Make sure that we don't totally fail to VACUUM individual indexes that
+-- happen to be below min_parallel_index_scan_size during parallel VACUUM:
+CREATE TABLE parallel_vacuum_table (a int) WITH (autovacuum_enabled = off);
+INSERT INTO parallel_vacuum_table SELECT i from generate_series(1, 10000) i;
+
+-- Parallel VACUUM will never be used unless there are at least two indexes
+-- that exceed min_parallel_index_scan_size.  Create two such indexes, and
+-- a third index that is smaller than min_parallel_index_scan_size.
+CREATE INDEX regular_sized_index ON parallel_vacuum_table(a);
+CREATE INDEX typically_sized_index ON parallel_vacuum_table(a);
+-- Note: vacuum_in_leader_small_index can apply deduplication, making it ~3x
+-- smaller than the other indexes
+CREATE INDEX vacuum_in_leader_small_index ON parallel_vacuum_table((1));
+
+-- Verify (as best we can) that the cost model for parallel VACUUM
+-- will make our VACUUM run in parallel, while always leaving it up to the
+-- parallel leader to handle the vacuum_in_leader_small_index index:
+SELECT EXISTS (
+SELECT 1
+FROM pg_class
+WHERE oid = 'vacuum_in_leader_small_index'::regclass AND
+  pg_relation_size(oid) <
+  pg_size_bytes(current_setting('min_parallel_index_scan_size'))
+) as leader_will_handle_small_index;
+SELECT count(*) as trigger_parallel_vacuum_nindexes
+FROM pg_class
+WHERE oid in ('regular_sized_index'::regclass, 'typically_sized_index'::regclass) AND
+  pg_relation_size(oid) >=
+  pg_size_bytes(current_setting('min_parallel_index_scan_size'));
+
+-- Parallel VACUUM with B-Tree page deletions, ambulkdelete calls:
+DELETE FROM parallel_vacuum_table;
+VACUUM (PARALLEL 4, INDEX_CLEANUP ON) parallel_vacuum_table;
+
+-- Since vacuum_in_leader_small_index uses deduplication, we expect an
+-- assertion failure with bug #17245 (in the absence of bugfix):
+INSERT INTO parallel_vacuum_table SELECT i FROM generate_series(1, 10000) i;
+
+RESET max_parallel_maintenance_workers;
+RESET min_parallel_index_scan_size;
+
+-- Deliberately don't drop table, to get further coverage from tools like
+-- pg_amcheck in some testing scenarios