Implement genuine serializable isolation level.

Until now, our Serializable mode has in fact been what's called Snapshot
Isolation, which allows some anomalies that could not occur in any
serialized ordering of the transactions. This patch fixes that using a
method called Serializable Snapshot Isolation, based on research papers by
Michael J. Cahill (see README-SSI for full references). In Serializable
Snapshot Isolation, transactions run like they do in Snapshot Isolation,
but a predicate lock manager observes the reads and writes performed and
aborts transactions if it detects that an anomaly might occur. This method
produces some false positives, ie. it sometimes aborts transactions even
though there is no anomaly.

To track reads we implement predicate locking, see storage/lmgr/predicate.c.
Whenever a tuple is read, a predicate lock is acquired on the tuple. Shared
memory is finite, so when a transaction takes many tuple-level locks on a
page, the locks are promoted to a single page-level lock, and further to a
single relation level lock if necessary. To lock key values with no matching
tuple, a sequential scan always takes a relation-level lock, and an index
scan acquires a page-level lock that covers the search key, whether or not
there are any matching keys at the moment.

A predicate lock doesn't conflict with any regular locks or with another
predicate locks in the normal sense. They're only used by the predicate lock
manager to detect the danger of anomalies. Only serializable transactions
participate in predicate locking, so there should be no extra overhead for
for other transactions.

Predicate locks can't be released at commit, but must be remembered until
all the transactions that overlapped with it have completed. That means that
we need to remember an unbounded amount of predicate locks, so we apply a
lossy but conservative method of tracking locks for committed transactions.
If we run short of shared memory, we overflow to a new "pg_serial" SLRU
pool.

We don't currently allow Serializable transactions in Hot Standby mode.
That would be hard, because even read-only transactions can cause anomalies
that wouldn't otherwise occur.

Serializable isolation mode now means the new fully serializable level.
Repeatable Read gives you the old Snapshot Isolation level that we have
always had.

Kevin Grittner and Dan Ports, reviewed by Jeff Davis, Heikki Linnakangas and
Anssi Kääriäinen

src/test/isolation/specs/classroom-scheduling.spec

@@ -0,0 +1,29 @@
+# Classroom Scheduling test
+#
+# Ensure that the classroom is not scheduled more than once
+# for any moment in time.
+#
+# Any overlap between the transactions must cause a serialization failure.
+
+setup
+{
+ CREATE TABLE room_reservation (room_id text NOT NULL, start_time timestamp with time zone NOT NULL, end_time timestamp with time zone NOT NULL, description text NOT NULL, CONSTRAINT room_reservation_pkey PRIMARY KEY (room_id, start_time));
+ INSERT INTO room_reservation VALUES ('101', TIMESTAMP WITH TIME ZONE '2010-04-01 10:00', TIMESTAMP WITH TIME ZONE '2010-04-01 11:00', 'Bob');
+}
+
+teardown
+{
+ DROP TABLE room_reservation;
+}
+
+session "s1"
+setup		{ BEGIN ISOLATION LEVEL SERIALIZABLE; }
+step "rx1"	{ SELECT count(*) FROM room_reservation WHERE room_id = '101' AND start_time < TIMESTAMP WITH TIME ZONE '2010-04-01 14:00' AND end_time > TIMESTAMP WITH TIME ZONE '2010-04-01 13:00'; }
+step "wy1"	{ INSERT INTO room_reservation VALUES ('101', TIMESTAMP WITH TIME ZONE '2010-04-01 13:00', TIMESTAMP WITH TIME ZONE '2010-04-01 14:00', 'Carol'); }
+step "c1"	{ COMMIT; }
+
+session "s2"
+setup		{ BEGIN ISOLATION LEVEL SERIALIZABLE; }
+step "ry2"	{ SELECT count(*) FROM room_reservation WHERE room_id = '101' AND start_time < TIMESTAMP WITH TIME ZONE '2010-04-01 14:30' AND end_time > TIMESTAMP WITH TIME ZONE '2010-04-01 13:30'; }
+step "wx2"	{ UPDATE room_reservation SET start_time = TIMESTAMP WITH TIME ZONE '2010-04-01 13:30', end_time = TIMESTAMP WITH TIME ZONE '2010-04-01 14:30' WHERE room_id = '101' AND start_time = TIMESTAMP WITH TIME ZONE '2010-04-01 10:00'; }
+step "c2"	{ COMMIT; }

src/test/isolation/specs/multiple-row-versions.spec

@@ -0,0 +1,48 @@
+# Multiple Row Versions test
+#
+# This test is designed to ensure that predicate locks taken on one version
+# of a row are detected as conflicts when a later version of the row is
+# updated or deleted by a transaction concurrent to the reader.
+#
+# Due to long permutation setup time, we are only testing one specific
+# permutation, which should get a serialization error.
+
+setup
+{
+ CREATE TABLE t (id int NOT NULL, txt text) WITH (fillfactor=50);
+ INSERT INTO t (id)
+   SELECT x FROM (SELECT * FROM generate_series(1, 1000000)) a(x);
+ ALTER TABLE t ADD PRIMARY KEY (id);
+}
+
+teardown
+{
+ DROP TABLE t;
+}
+
+session "s1"
+setup		{ BEGIN ISOLATION LEVEL SERIALIZABLE; }
+step "rx1"	{ SELECT * FROM t WHERE id = 1000000; }
+# delay until after T3 commits
+step "wz1"	{ UPDATE t SET txt = 'a' WHERE id = 1; }
+step "c1"	{ COMMIT; }
+
+session "s2"
+setup		{ BEGIN ISOLATION LEVEL SERIALIZABLE; }
+step "wx2"	{ UPDATE t SET txt = 'b' WHERE id = 1000000; }
+step "c2"	{ COMMIT; }
+
+session "s3"
+setup		{ BEGIN ISOLATION LEVEL SERIALIZABLE; }
+step "wx3"	{ UPDATE t SET txt = 'c' WHERE id = 1000000; }
+step "ry3"	{ SELECT * FROM t WHERE id = 500000; }
+# delay until after T4 commits
+step "c3"	{ COMMIT; }
+
+session "s4"
+setup		{ BEGIN ISOLATION LEVEL SERIALIZABLE; }
+step "wy4"	{ UPDATE t SET txt = 'd' WHERE id = 500000; }
+step "rz4"	{ SELECT * FROM t WHERE id = 1; }
+step "c4"	{ COMMIT; }
+
+permutation "rx1" "wx2" "c2" "wx3" "ry3" "wy4" "rz4" "c4" "c3" "wz1" "c1"

src/test/isolation/specs/partial-index.spec

@@ -0,0 +1,32 @@
+# Partial Index test
+#
+# Make sure that an update which moves a row out of a partial index
+# is handled correctly.  In early versions, an attempt at optimization
+# broke this behavior, allowing anomalies.
+#
+# Any overlap between the transactions must cause a serialization failure.
+
+setup
+{
+ create table test_t (id integer, val1 text, val2 integer);
+ create index test_idx on test_t(id) where val2 = 1;
+ insert into test_t (select generate_series(0, 10000), 'a', 2);
+ insert into test_t (select generate_series(0, 10), 'a', 1);
+}
+
+teardown
+{
+ DROP TABLE test_t;
+}
+
+session "s1"
+setup		{ BEGIN ISOLATION LEVEL SERIALIZABLE; }
+step "rxy1"	{ select * from test_t where val2 = 1; }
+step "wx1"	{ update test_t set val2 = 2 where val2 = 1 and id = 10; }
+step "c1"	{ COMMIT; }
+
+session "s2"
+setup		{ BEGIN ISOLATION LEVEL SERIALIZABLE; }
+step "wy2"	{ update test_t set val2 = 2 where val2 = 1 and id = 9; }
+step "rxy2"	{ select * from test_t where val2 = 1; }
+step "c2"	{ COMMIT; }

src/test/isolation/specs/project-manager.spec

@@ -0,0 +1,30 @@
+# Project Manager test
+#
+# Ensure that the person who is on the project as a manager
+# is flagged as a project manager in the person table.
+#
+# Any overlap between the transactions must cause a serialization failure.
+
+setup
+{
+ CREATE TABLE person (person_id int NOT NULL PRIMARY KEY, name text NOT NULL, is_project_manager bool NOT NULL);
+ INSERT INTO person VALUES (1, 'Robert Haas', true);
+ CREATE TABLE project (project_no int NOT NULL PRIMARY KEY, description text NOT NULL, project_manager int NOT NULL);
+}
+
+teardown
+{
+ DROP TABLE person, project;
+}
+
+session "s1"
+setup		{ BEGIN ISOLATION LEVEL SERIALIZABLE; }
+step "rx1"	{ SELECT count(*) FROM person WHERE person_id = 1 AND is_project_manager; }
+step "wy1"	{ INSERT INTO project VALUES (101, 'Build Great Wall', 1); }
+step "c1"	{ COMMIT; }
+
+session "s2"
+setup		{ BEGIN ISOLATION LEVEL SERIALIZABLE; }
+step "ry2"	{ SELECT count(*) FROM project WHERE project_manager = 1; }
+step "wx2"	{ UPDATE person SET is_project_manager = false WHERE person_id = 1; }
+step "c2"	{ COMMIT; }

src/test/isolation/specs/receipt-report.spec

@@ -0,0 +1,47 @@
+# Daily Report of Receipts test.
+#
+# This test doesn't persist a bad state in the database; rather, it
+# provides a view of the data which is not consistent with any
+# order of execution of the serializable transactions.  It
+# demonstrates a situation where the deposit date for receipts could
+# be changed and a report of the closed day's receipts subsequently
+# run which will miss a receipt from the date which has been closed.
+#
+# There are only six permuations which must cause a serialization failure.
+# Failure cases are where s1 overlaps both s2 and s3, but s2 commits before
+# s3 executes its first SELECT.
+#
+# As long as s3 is declared READ ONLY there should be no false positives.
+# If s3 were changed to READ WRITE, we would currently expect 42 false
+# positives.  Further work dealing with de facto READ ONLY transactions
+# may be able to reduce or eliminate those false positives.
+
+setup
+{
+  CREATE TABLE ctl (k text NOT NULL PRIMARY KEY, deposit_date date NOT NULL);
+  INSERT INTO ctl VALUES ('receipt', DATE '2008-12-22');
+  CREATE TABLE receipt (receipt_no int NOT NULL PRIMARY KEY, deposit_date date NOT NULL, amount numeric(13,2));
+  INSERT INTO receipt VALUES (1, (SELECT deposit_date FROM ctl WHERE k = 'receipt'), 1.00);
+  INSERT INTO receipt VALUES (2, (SELECT deposit_date FROM ctl WHERE k = 'receipt'), 2.00);
+}
+
+teardown
+{
+  DROP TABLE ctl, receipt;
+}
+
+session "s1"
+setup		{ BEGIN ISOLATION LEVEL SERIALIZABLE; }
+step "rxwy1"	{ INSERT INTO receipt VALUES (3, (SELECT deposit_date FROM ctl WHERE k = 'receipt'), 4.00); }
+step "c1"	{ COMMIT; }
+
+session "s2"
+setup		{ BEGIN ISOLATION LEVEL SERIALIZABLE; }
+step "wx2"	{ UPDATE ctl SET deposit_date = DATE '2008-12-23' WHERE k = 'receipt'; }
+step "c2"	{ COMMIT; }
+
+session "s3"
+setup		{ BEGIN ISOLATION LEVEL SERIALIZABLE, READ ONLY; }
+step "rx3"	{ SELECT * FROM ctl WHERE k = 'receipt'; }
+step "ry3"	{ SELECT * FROM receipt WHERE deposit_date = DATE '2008-12-22'; }
+step "c3"	{ COMMIT; }

src/test/isolation/specs/referential-integrity.spec

@@ -0,0 +1,32 @@
+# Referential Integrity test
+#
+# The assumption here is that the application code issuing the SELECT
+# to test for the presence or absence of a related record would do the
+# right thing -- this script doesn't include that logic.
+#
+# Any overlap between the transactions must cause a serialization failure.
+
+setup
+{
+ CREATE TABLE a (i int PRIMARY KEY);
+ CREATE TABLE b (a_id int);
+ INSERT INTO a VALUES (1);
+}
+
+teardown
+{
+ DROP TABLE a, b;
+}
+
+session "s1"
+setup		{ BEGIN ISOLATION LEVEL SERIALIZABLE; }
+step "rx1"	{ SELECT i FROM a WHERE i = 1; }
+step "wy1"	{ INSERT INTO b VALUES (1); }
+step "c1"	{ COMMIT; }
+
+session "s2"
+setup		{ BEGIN ISOLATION LEVEL SERIALIZABLE; }
+step "rx2"	{ SELECT i FROM a WHERE i = 1; }
+step "ry2"	{ SELECT a_id FROM b WHERE a_id = 1; }
+step "wx2"	{ DELETE FROM a WHERE i = 1; }
+step "c2"	{ COMMIT; }

src/test/isolation/specs/ri-trigger.spec

@@ -0,0 +1,53 @@
+# RI Trigger test
+#
+# Test trigger-based referential integrity enforcement.
+#
+# Any overlap between the transactions must cause a serialization failure.
+
+setup
+{
+ CREATE TABLE parent (parent_id SERIAL NOT NULL PRIMARY KEY);
+ CREATE TABLE child (child_id SERIAL NOT NULL PRIMARY KEY, parent_id INTEGER NOT NULL);
+ CREATE FUNCTION ri_parent() RETURNS TRIGGER LANGUAGE PLPGSQL AS $body$
+ BEGIN
+  PERFORM TRUE FROM child WHERE parent_id = OLD.parent_id;
+  IF FOUND THEN
+    RAISE SQLSTATE '23503' USING MESSAGE = 'child row exists';
+  END IF;
+  IF TG_OP = 'DELETE' THEN
+    RETURN OLD;
+  END IF;
+  RETURN NEW;
+ END;
+ $body$;
+ CREATE TRIGGER ri_parent BEFORE UPDATE OR DELETE ON parent FOR EACH ROW EXECUTE PROCEDURE ri_parent();
+ CREATE FUNCTION ri_child() RETURNS TRIGGER LANGUAGE PLPGSQL AS $body$
+ BEGIN
+  PERFORM TRUE FROM parent WHERE parent_id = NEW.parent_id;
+  IF NOT FOUND THEN
+    RAISE SQLSTATE '23503' USING MESSAGE = 'parent row missing';
+  END IF;
+  RETURN NEW;
+ END;
+ $body$;
+ CREATE TRIGGER ri_child BEFORE INSERT OR UPDATE ON child FOR EACH ROW EXECUTE PROCEDURE ri_child();
+ INSERT INTO parent VALUES(0);
+}
+
+teardown
+{
+ DROP TABLE parent, child;
+ DROP FUNCTION ri_parent();
+ DROP FUNCTION ri_child();
+}
+
+session "s1"
+setup		{ BEGIN ISOLATION LEVEL SERIALIZABLE; }
+step "wxry1"	{ INSERT INTO child (parent_id) VALUES (0); }
+step "c1"	{ COMMIT; }
+
+session "s2"
+setup		{ BEGIN ISOLATION LEVEL SERIALIZABLE; }
+step "r2"	{ SELECT TRUE; }
+step "wyrx2"	{ DELETE FROM parent WHERE parent_id = 0; }
+step "c2"	{ COMMIT; }

src/test/isolation/specs/simple-write-skew.spec

@@ -0,0 +1,30 @@
+# Write skew test.
+#
+# This test has two serializable transactions: one which updates all
+# 'apple' rows to 'pear' and one which updates all 'pear' rows to
+# 'apple'.  If these were serialized (run one at a time) either
+# value could be present, but not both.  One must be rolled back to
+# prevent the write skew anomaly.
+#
+# Any overlap between the transactions must cause a serialization failure.
+
+setup
+{
+  CREATE TABLE test (i int PRIMARY KEY, t text);
+  INSERT INTO test VALUES (5, 'apple'), (7, 'pear'), (11, 'banana');
+}
+
+teardown
+{
+  DROP TABLE test;
+}
+
+session "s1"
+setup { BEGIN ISOLATION LEVEL SERIALIZABLE; }
+step "rwx1" { UPDATE test SET t = 'apple' WHERE t = 'pear'; }
+step "c1" { COMMIT; }
+
+session "s2"
+setup { BEGIN ISOLATION LEVEL SERIALIZABLE; }
+step "rwx2" { UPDATE test SET t = 'pear' WHERE t = 'apple'}
+step "c2" { COMMIT; }

src/test/isolation/specs/temporal-range-integrity.spec

@@ -0,0 +1,38 @@
+# Temporal Range Integrity test
+#
+# Snapshot integrity fails with simple referential integrity tests,
+# but those don't make for good demonstrations because people just
+# say that foreign key definitions should be used instead.  There
+# are many integrity tests which are conceptually very similar but
+# don't have built-in support which will fail when used in triggers.
+# This is intended to illustrate such cases.  It is obviously very
+# hard to exercise all these permutations when the code is actually
+# in a trigger; this test pulls what would normally be inside of
+# triggers out to the top level to control the permutations.
+#
+# Any overlap between the transactions must cause a serialization failure.
+
+
+setup
+{
+ CREATE TABLE statute (statute_cite text NOT NULL, eff_date date NOT NULL, exp_date date, CONSTRAINT statute_pkey PRIMARY KEY (statute_cite, eff_date));
+ INSERT INTO statute VALUES ('123.45(1)a', DATE '2008-01-01', NULL);
+ CREATE TABLE offense (offense_no int NOT NULL, statute_cite text NOT NULL, offense_date date NOT NULL, CONSTRAINT offense_pkey PRIMARY KEY (offense_no));
+}
+
+teardown
+{
+  DROP TABLE statute, offense;
+}
+
+session "s1"
+setup		{ BEGIN ISOLATION LEVEL SERIALIZABLE; }
+step "rx1"	{ SELECT count(*) FROM statute WHERE statute_cite = '123.45(1)a' AND eff_date <= DATE '2009-05-15' AND (exp_date IS NULL OR exp_date > DATE '2009-05-15'); }
+step "wy1"	{ INSERT INTO offense VALUES (1, '123.45(1)a', DATE '2009-05-15'); }
+step "c1"	{ COMMIT; }
+
+session "s2"
+setup		{ BEGIN ISOLATION LEVEL SERIALIZABLE; }
+step "ry2"	{ SELECT count(*) FROM offense WHERE statute_cite = '123.45(1)a' AND offense_date >= DATE '2008-01-01'; }
+step "wx2"	{ DELETE FROM statute WHERE statute_cite = '123.45(1)a' AND eff_date = DATE '2008-01-01'; }
+step "c2"	{ COMMIT; }

src/test/isolation/specs/total-cash.spec

@@ -0,0 +1,28 @@
+# Total Cash test
+#
+# Another famous test of snapshot isolation anomaly.
+#
+# Any overlap between the transactions must cause a serialization failure.
+
+setup
+{
+ CREATE TABLE accounts (accountid text NOT NULL PRIMARY KEY, balance numeric not null);
+ INSERT INTO accounts VALUES ('checking', 600),('savings',600);
+}
+
+teardown
+{
+ DROP TABLE accounts;
+}
+
+session "s1"
+setup		{ BEGIN ISOLATION LEVEL SERIALIZABLE; }
+step "wx1"	{ UPDATE accounts SET balance = balance - 200 WHERE accountid = 'checking'; }
+step "rxy1"	{ SELECT SUM(balance) FROM accounts; }
+step "c1"	{ COMMIT; }
+
+session "s2"
+setup		{ BEGIN ISOLATION LEVEL SERIALIZABLE; }
+step "wy2"	{ UPDATE accounts SET balance = balance - 200 WHERE accountid = 'savings'; }
+step "rxy2"	{ SELECT SUM(balance) FROM accounts; }
+step "c2"	{ COMMIT; }

src/test/isolation/specs/two-ids.spec

@@ -0,0 +1,40 @@
+# Two IDs test
+#
+# Small, simple test showing read-only anomalies.
+#
+# There are only four permuations which must cause a serialization failure.
+# Required failure cases are where s2 overlaps both s1 and s3, but s1
+# commits before s3 executes its first SELECT.
+#
+# If s3 were declared READ ONLY there would be no false positives.
+# With s3 defaulting to READ WRITE, we currently expect 12 false
+# positives.  Further work dealing with de facto READ ONLY transactions
+# may be able to reduce or eliminate those false positives.
+
+setup
+{
+ create table D1 (id int not null);
+ create table D2 (id int not null);
+ insert into D1 values (1);
+ insert into D2 values (1);
+}
+
+teardown
+{
+ DROP TABLE D1, D2;
+}
+
+session "s1"
+setup		{ BEGIN ISOLATION LEVEL SERIALIZABLE; }
+step "wx1"	{ update D1 set id = id + 1; }
+step "c1"	{ COMMIT; }
+
+session "s2"
+setup		{ BEGIN ISOLATION LEVEL SERIALIZABLE; }
+step "rxwy2"	{ update D2 set id = (select id+1 from D1); }
+step "c2"	{ COMMIT; }
+
+session "s3"
+setup		{ BEGIN ISOLATION LEVEL SERIALIZABLE; }
+step "ry3"	{ select id from D2; }
+step "c3"	{ COMMIT; }