Avoid spurious deadlocks when upgrading a tuple lock

When two (or more) transactions are waiting for transaction T1 to release a
tuple-level lock, and transaction T1 upgrades its lock to a higher level, a
spurious deadlock can be reported among the waiting transactions when T1
finishes.  The simplest example case seems to be:

T1: select id from job where name = 'a' for key share;
Y: select id from job where name = 'a' for update; -- starts waiting for X
Z: select id from job where name = 'a' for key share;
T1: update job set name = 'b' where id = 1;
Z: update job set name = 'c' where id = 1; -- starts waiting for X
T1: rollback;

At this point, transaction Y is rolled back on account of a deadlock: Y
holds the heavyweight tuple lock and is waiting for the Xmax to be released,
while Z holds part of the multixact and tries to acquire the heavyweight
lock (per protocol) and goes to sleep; once X releases its part of the
multixact, Z is awakened only to be put back to sleep on the heavyweight
lock that Y is holding while sleeping.  Kaboom.

This can be avoided by having Z skip the heavyweight lock acquisition.  As
far as I can see, the biggest downside is that if there are multiple Z
transactions, the order in which they resume after X finishes is not
guaranteed.

Backpatch to 9.6.  The patch applies cleanly on 9.5, but the new tests don't
work there (because isolationtester is not smart enough), so I'm not going
to risk it.

Author: Oleksii Kliukin
Discussion: https://postgr.es/m/B9C9D7CD-EB94-4635-91B6-E558ACEC0EC3@hintbits.com

src/test/isolation/expected/tuplelock-upgrade-no-deadlock.out

@@ -0,0 +1,150 @@
+Parsed test spec with 3 sessions
+
+starting permutation: s1_share s2_for_update s3_share s3_for_update s1_rollback s3_rollback s2_rollback
+step s1_share: select id from tlu_job where id = 1 for share;
+id             
+
+1              
+step s2_for_update: select id from tlu_job where id = 1 for update; <waiting ...>
+step s3_share: select id from tlu_job where id = 1 for share;
+id             
+
+1              
+step s3_for_update: select id from tlu_job where id = 1 for update; <waiting ...>
+step s1_rollback: rollback;
+step s3_for_update: <... completed>
+id             
+
+1              
+step s3_rollback: rollback;
+step s2_for_update: <... completed>
+id             
+
+1              
+step s2_rollback: rollback;
+
+starting permutation: s1_keyshare s2_for_update s3_keyshare s1_update s3_update s1_rollback s3_rollback s2_rollback
+step s1_keyshare: select id from tlu_job where id = 1 for key share;
+id             
+
+1              
+step s2_for_update: select id from tlu_job where id = 1 for update; <waiting ...>
+step s3_keyshare: select id from tlu_job where id = 1 for key share;
+id             
+
+1              
+step s1_update: update tlu_job set name = 'b' where id = 1;
+step s3_update: update tlu_job set name = 'c' where id = 1; <waiting ...>
+step s1_rollback: rollback;
+step s3_update: <... completed>
+step s3_rollback: rollback;
+step s2_for_update: <... completed>
+id             
+
+1              
+step s2_rollback: rollback;
+
+starting permutation: s1_keyshare s2_for_update s3_keyshare s1_update s3_update s1_commit s3_rollback s2_rollback
+step s1_keyshare: select id from tlu_job where id = 1 for key share;
+id             
+
+1              
+step s2_for_update: select id from tlu_job where id = 1 for update; <waiting ...>
+step s3_keyshare: select id from tlu_job where id = 1 for key share;
+id             
+
+1              
+step s1_update: update tlu_job set name = 'b' where id = 1;
+step s3_update: update tlu_job set name = 'c' where id = 1; <waiting ...>
+step s1_commit: commit;
+step s3_update: <... completed>
+step s3_rollback: rollback;
+step s2_for_update: <... completed>
+id             
+
+1              
+step s2_rollback: rollback;
+
+starting permutation: s1_keyshare s2_for_update s3_keyshare s3_delete s1_rollback s3_rollback s2_rollback
+step s1_keyshare: select id from tlu_job where id = 1 for key share;
+id             
+
+1              
+step s2_for_update: select id from tlu_job where id = 1 for update; <waiting ...>
+step s3_keyshare: select id from tlu_job where id = 1 for key share;
+id             
+
+1              
+step s3_delete: delete from tlu_job where id = 1; <waiting ...>
+step s1_rollback: rollback;
+step s3_delete: <... completed>
+step s3_rollback: rollback;
+step s2_for_update: <... completed>
+id             
+
+1              
+step s2_rollback: rollback;
+
+starting permutation: s1_keyshare s2_for_update s3_keyshare s3_delete s1_rollback s3_commit s2_rollback
+step s1_keyshare: select id from tlu_job where id = 1 for key share;
+id             
+
+1              
+step s2_for_update: select id from tlu_job where id = 1 for update; <waiting ...>
+step s3_keyshare: select id from tlu_job where id = 1 for key share;
+id             
+
+1              
+step s3_delete: delete from tlu_job where id = 1; <waiting ...>
+step s1_rollback: rollback;
+step s3_delete: <... completed>
+step s3_commit: commit;
+step s2_for_update: <... completed>
+id             
+
+step s2_rollback: rollback;
+
+starting permutation: s1_share s2_for_update s3_for_update s1_rollback s2_rollback s3_rollback
+step s1_share: select id from tlu_job where id = 1 for share;
+id             
+
+1              
+step s2_for_update: select id from tlu_job where id = 1 for update; <waiting ...>
+step s3_for_update: select id from tlu_job where id = 1 for update; <waiting ...>
+step s1_rollback: rollback;
+step s2_for_update: <... completed>
+id             
+
+1              
+step s2_rollback: rollback;
+step s3_for_update: <... completed>
+id             
+
+1              
+step s3_rollback: rollback;
+
+starting permutation: s1_share s2_update s3_update s1_rollback s2_rollback s3_rollback
+step s1_share: select id from tlu_job where id = 1 for share;
+id             
+
+1              
+step s2_update: update tlu_job set name = 'b' where id = 1; <waiting ...>
+step s3_update: update tlu_job set name = 'c' where id = 1; <waiting ...>
+step s1_rollback: rollback;
+step s2_update: <... completed>
+step s2_rollback: rollback;
+step s3_update: <... completed>
+step s3_rollback: rollback;
+
+starting permutation: s1_share s2_delete s3_delete s1_rollback s2_rollback s3_rollback
+step s1_share: select id from tlu_job where id = 1 for share;
+id             
+
+1              
+step s2_delete: delete from tlu_job where id = 1; <waiting ...>
+step s3_delete: delete from tlu_job where id = 1; <waiting ...>
+step s1_rollback: rollback;
+step s2_delete: <... completed>
+step s2_rollback: rollback;
+step s3_delete: <... completed>
+step s3_rollback: rollback;

src/test/isolation/isolation_schedule

@@ -49,6 +49,7 @@ test: reindex-concurrently
 test: propagate-lock-delete
 test: tuplelock-conflict
 test: tuplelock-update
+test: tuplelock-upgrade-no-deadlock
 test: freeze-the-dead
 test: nowait
 test: nowait-2

src/test/isolation/specs/tuplelock-upgrade-no-deadlock.spec

@@ -0,0 +1,57 @@
+# This test checks that multiple sessions locking a single row in a table
+# do not deadlock each other when one of them upgrades its existing lock
+# while the others are waiting for it.
+
+setup
+{
+    drop table if exists tlu_job;
+    create table tlu_job (id integer primary key, name text);
+
+    insert into tlu_job values(1, 'a');
+}
+
+
+teardown
+{
+    drop table tlu_job;
+}
+
+session "s1"
+setup { begin; }
+step "s1_keyshare" { select id from tlu_job where id = 1 for key share;}
+step "s1_share" { select id from tlu_job where id = 1 for share; }
+step "s1_update" { update tlu_job set name = 'b' where id = 1;  }
+step "s1_delete" { delete from tlu_job where id = 1; }
+step "s1_rollback" { rollback; }
+step "s1_commit" { commit; }
+
+session "s2"
+setup { begin; }
+step "s2_for_update" { select id from tlu_job where id = 1 for update; }
+step "s2_update" { update tlu_job set name = 'b' where id = 1; }
+step "s2_delete" { delete from tlu_job where id = 1; }
+step "s2_rollback" { rollback; }
+step "s2_commit" { commit; }
+
+session "s3"
+setup { begin; }
+step "s3_keyshare" { select id from tlu_job where id = 1 for key share; }
+step "s3_share" { select id from tlu_job where id = 1 for share; }
+step "s3_for_update" { select id from tlu_job where id = 1 for update; }
+step "s3_update" { update tlu_job set name = 'c' where id = 1; }
+step "s3_delete" { delete from tlu_job where id = 1; }
+step "s3_rollback" { rollback; }
+step "s3_commit" { commit; }
+
+# test that s2 will not deadlock with s3 when s1 is rolled back
+permutation "s1_share" "s2_for_update" "s3_share" "s3_for_update" "s1_rollback" "s3_rollback" "s2_rollback"
+# test that update does not cause deadlocks if it can proceed
+permutation  "s1_keyshare" "s2_for_update" "s3_keyshare" "s1_update" "s3_update" "s1_rollback" "s3_rollback" "s2_rollback"
+permutation  "s1_keyshare" "s2_for_update" "s3_keyshare" "s1_update" "s3_update" "s1_commit"   "s3_rollback" "s2_rollback"
+# test that delete does not cause deadlocks if it can proceed
+permutation "s1_keyshare" "s2_for_update" "s3_keyshare"  "s3_delete" "s1_rollback" "s3_rollback" "s2_rollback"
+permutation "s1_keyshare" "s2_for_update" "s3_keyshare"  "s3_delete" "s1_rollback" "s3_commit"   "s2_rollback"
+# test that sessions that don't upgrade locks acquire them in order
+permutation "s1_share" "s2_for_update" "s3_for_update" "s1_rollback" "s2_rollback" "s3_rollback"
+permutation "s1_share" "s2_update" "s3_update" "s1_rollback" "s2_rollback" "s3_rollback"
+permutation "s1_share" "s2_delete" "s3_delete" "s1_rollback" "s2_rollback" "s3_rollback"