database/postgres
1
0
Fork 0
mirror of https://github.com/postgres/postgres.git synced 2025-08-28 18:48:04 +03:00
Code Activity

Add support for prepared transactions to built-in logical replication.

Browse Source 
To add support for streaming transactions at prepare time into the
built-in logical replication, we need to do the following things:

* Modify the output plugin (pgoutput) to implement the new two-phase API
callbacks, by leveraging the extended replication protocol.

* Modify the replication apply worker, to properly handle two-phase
transactions by replaying them on prepare.

* Add a new SUBSCRIPTION option "two_phase" to allow users to enable
two-phase transactions. We enable the two_phase once the initial data sync
is over.

We however must explicitly disable replication of two-phase transactions
during replication slot creation, even if the plugin supports it. We
don't need to replicate the changes accumulated during this phase,
and moreover, we don't have a replication connection open so we don't know
where to send the data anyway.

The streaming option is not allowed with this new two_phase option. This
can be done as a separate patch.

We don't allow to toggle two_phase option of a subscription because it can
lead to an inconsistent replica. For the same reason, we don't allow to
refresh the publication once the two_phase is enabled for a subscription
unless copy_data option is false.

Author: Peter Smith, Ajin Cherian and Amit Kapila based on previous work by Nikhil Sontakke and Stas Kelvich
Reviewed-by: Amit Kapila, Sawada Masahiko, Vignesh C, Dilip Kumar, Takamichi Osumi, Greg Nancarrow
Tested-By: Haiying Tang
Discussion: https://postgr.es/m/02DA5F5E-CECE-4D9C-8B4B-418077E2C010@postgrespro.ru
Discussion: https://postgr.es/m/CAA4eK1+opiV4aFTmWWUF9h_32=HfPOW9vZASHarT0UA5oBrtGw@mail.gmail.com
This commit is contained in:
Amit Kapila
2021-07-14 07:33:50 +05:30
parent 6c9c283166
commit a8fd13cab0
43 changed files with 2382 additions and 191 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
doc/src/sgml/catalogs.sgml
View File

@@ -7641,6 +7641,18 @@ SCRAM-SHA-256$<replaceable>&lt;iteration count&gt;</replaceable>:<replaceable>&l
</para></entry>
</row>
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>subtwophasestate</structfield> <type>char</type>
</para>
<para>
State codes for two-phase mode:
<literal>d</literal> = disabled,
<literal>p</literal> = pending enablement,
<literal>e</literal> = enabled
</para></entry>
</row>
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>subconninfo</structfield> <type>text</type>

291
doc/src/sgml/protocol.sgml
View File

@@ -2811,11 +2811,17 @@ The commands accepted in replication mode are:
</term>
<listitem>
<para>
Protocol version. Currently versions <literal>1</literal> and
<literal>2</literal> are supported. The version <literal>2</literal>
is supported only for server version 14 and above, and it allows
streaming of large in-progress transactions.
</para>
Protocol version. Currently versions <literal>1</literal>, <literal>2</literal>,
and <literal>3</literal> are supported.
</para>
<para>
Version <literal>2</literal> is supported only for server version 14
and above, and it allows streaming of large in-progress transactions.
</para>
<para>
Version <literal>3</literal> is supported only for server version 15
and above, and it allows streaming of two-phase transactions.
</para>
</listitem>
</varlistentry>
@@ -2871,10 +2877,11 @@ The commands accepted in replication mode are:
<para>
The logical replication protocol sends individual transactions one by one.
This means that all messages between a pair of Begin and Commit messages
belong to the same transaction. It also sends changes of large in-progress
transactions between a pair of Stream Start and Stream Stop messages. The
last stream of such a transaction contains Stream Commit or Stream Abort
message.
belong to the same transaction. Similarly, all messages between a pair of
Begin Prepare and Prepare messages belong to the same transaction.
It also sends changes of large in-progress transactions between a pair of
Stream Start and Stream Stop messages. The last stream of such a transaction
contains a Stream Commit or Stream Abort message.
</para>
<para>
@@ -7390,6 +7397,272 @@ Stream Abort
</variablelist>
<para>
The following messages (Begin Prepare, Prepare, Commit Prepared, Rollback Prepared)
are available since protocol version 3.
</para>
<variablelist>
<varlistentry>
<term>Begin Prepare</term>
<listitem>
<para>
<variablelist>
<varlistentry>
<term>Byte1('b')</term>
<listitem><para>
Identifies the message as the beginning of a two-phase transaction message.
</para></listitem>
</varlistentry>
<varlistentry>
<term>Int64</term>
<listitem><para>
The LSN of the prepare.
</para></listitem>
</varlistentry>
<varlistentry>
<term>Int64</term>
<listitem><para>
The end LSN of the prepared transaction.
</para></listitem>
</varlistentry>
<varlistentry>
<term>Int64</term>
<listitem><para>
Prepare timestamp of the transaction. The value is in number
of microseconds since PostgreSQL epoch (2000-01-01).
</para></listitem>
</varlistentry>
<varlistentry>
<term>Int32</term>
<listitem><para>
Xid of the transaction.
</para></listitem>
</varlistentry>
<varlistentry>
<term>String</term>
<listitem><para>
The user defined GID of the two-phase transaction.
</para></listitem>
</varlistentry>
</variablelist>
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>Prepare</term>
<listitem>
<para>
<variablelist>
<varlistentry>
<term>Byte1('P')</term>
<listitem><para>
Identifies the message as a two-phase prepared transaction message.
</para></listitem>
</varlistentry>
<varlistentry>
<term>Int8</term>
<listitem><para>
Flags; currently unused (must be 0).
</para></listitem>
</varlistentry>
<varlistentry>
<term>Int64</term>
<listitem><para>
The LSN of the prepare.
</para></listitem>
</varlistentry>
<varlistentry>
<term>Int64</term>
<listitem><para>
The end LSN of the prepared transaction.
</para></listitem>
</varlistentry>
<varlistentry>
<term>Int64</term>
<listitem><para>
Prepare timestamp of the transaction. The value is in number
of microseconds since PostgreSQL epoch (2000-01-01).
</para></listitem>
</varlistentry>
<varlistentry>
<term>Int32</term>
<listitem><para>
Xid of the transaction.
</para></listitem>
</varlistentry>
<varlistentry>
<term>String</term>
<listitem><para>
The user defined GID of the two-phase transaction.
</para></listitem>
</varlistentry>
</variablelist>
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>Commit Prepared</term>
<listitem>
<para>
<variablelist>
<varlistentry>
<term>Byte1('K')</term>
<listitem><para>
Identifies the message as the commit of a two-phase transaction message.
</para></listitem>
</varlistentry>
<varlistentry>
<term>Int8</term>
<listitem><para>
Flags; currently unused (must be 0).
</para></listitem>
</varlistentry>
<varlistentry>
<term>Int64</term>
<listitem><para>
The LSN of the commit prepared.
</para></listitem>
</varlistentry>
<varlistentry>
<term>Int64</term>
<listitem><para>
The end LSN of the commit prepared transaction.
</para></listitem>
</varlistentry>
<varlistentry>
<term>Int64</term>
<listitem><para>
Commit timestamp of the transaction. The value is in number
of microseconds since PostgreSQL epoch (2000-01-01).
</para></listitem>
</varlistentry>
<varlistentry>
<term>Int32</term>
<listitem><para>
Xid of the transaction.
</para></listitem>
</varlistentry>
<varlistentry>
<term>String</term>
<listitem><para>
The user defined GID of the two-phase transaction.
</para></listitem>
</varlistentry>
</variablelist>
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>Rollback Prepared</term>
<listitem>
<para>
<variablelist>
<varlistentry>
<term>Byte1('r')</term>
<listitem><para>
Identifies the message as the rollback of a two-phase transaction message.
</para></listitem>
</varlistentry>
<varlistentry>
<term>Int8</term>
<listitem><para>
Flags; currently unused (must be 0).
</para></listitem>
</varlistentry>
<varlistentry>
<term>Int64</term>
<listitem><para>
The end LSN of the prepared transaction.
</para></listitem>
</varlistentry>
<varlistentry>
<term>Int64</term>
<listitem><para>
The end LSN of the rollback prepared transaction.
</para></listitem>
</varlistentry>
<varlistentry>
<term>Int64</term>
<listitem><para>
Prepare timestamp of the transaction. The value is in number
of microseconds since PostgreSQL epoch (2000-01-01).
</para></listitem>
</varlistentry>
<varlistentry>
<term>Int64</term>
<listitem><para>
Rollback timestamp of the transaction. The value is in number
of microseconds since PostgreSQL epoch (2000-01-01).
</para></listitem>
</varlistentry>
<varlistentry>
<term>Int32</term>
<listitem><para>
Xid of the transaction.
</para></listitem>
</varlistentry>
<varlistentry>
<term>String</term>
<listitem><para>
The user defined GID of the two-phase transaction.
</para></listitem>
</varlistentry>
</variablelist>
</para>
</listitem>
</varlistentry>
</variablelist>
<para>
The following message parts are shared by the above messages.

5
doc/src/sgml/ref/alter_subscription.sgml
View File

@@ -67,6 +67,11 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
Commands <command>ALTER SUBSCRIPTION ... REFRESH PUBLICATION</command> and
<command>ALTER SUBSCRIPTION ... {SET|ADD|DROP} PUBLICATION ...</command> with refresh
option as true cannot be executed inside a transaction block.
These commands also cannot be executed when the subscription has
<literal>two_phase</literal> commit enabled, unless <literal>copy_data = false</literal>.
See column <literal>subtwophasestate</literal> of
<xref linkend="catalog-pg-subscription"/> to know the actual two-phase state.
</para>
</refsect1>

37
doc/src/sgml/ref/create_subscription.sgml
View File

@@ -237,6 +237,43 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
are fully decoded on the publisher, and only then sent to the
subscriber as a whole.
</para>
<para>
The <literal>streaming</literal> option cannot be used with the
<literal>two_phase</literal> option.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><literal>two_phase</literal> (<type>boolean</type>)</term>
<listitem>
<para>
Specifies whether two-phase commit is enabled for this subscription.
The default is <literal>false</literal>.
</para>
<para>
When two-phase commit is enabled then the decoded transactions are sent
to the subscriber on the PREPARE TRANSACTION. By default, the transaction
prepared on the publisher is decoded as a normal transaction at commit.
</para>
<para>
The two-phase commit implementation requires that the replication has
successfully passed the initial table synchronization phase. This means
even when two_phase is enabled for the subscription, the internal
two-phase state remains temporarily "pending" until the initialization
phase is completed. See column
<literal>subtwophasestate</literal> of <xref linkend="catalog-pg-subscription"/>
to know the actual two-phase state.
</para>
<para>
The <literal>two_phase</literal> option cannot be used with the
<literal>streaming</literal> option.
</para>
</listitem>
</varlistentry>
</variablelist></para>

7
doc/src/sgml/ref/pg_dump.sgml
View File

@@ -1405,7 +1405,12 @@ CREATE DATABASE foo WITH TEMPLATE template0;
servers. It is then up to the user to reactivate the subscriptions in a
suitable way. If the involved hosts have changed, the connection
information might have to be changed. It might also be appropriate to
truncate the target tables before initiating a new full table copy.
truncate the target tables before initiating a new full table copy. If users
intend to copy initial data during refresh they must create the slot with
<literal>two_phase = false</literal>. After the initial sync, the
<literal>two_phase</literal> option will be automatically enabled by the
subscriber if the subscription had been originally created with
<literal>two_phase = true</literal> option.
</para>
</refsect1>