database/postgres
1
0
Fork 0
mirror of https://github.com/postgres/postgres.git synced 2025-07-30 11:03:19 +03:00
Code Activity

Logical decoding of sequences

Browse Source 
This extends the logical decoding to also decode sequence increments.
We differentiate between sequences created in the current (in-progress)
transaction, and sequences created earlier. This mixed behavior is
necessary because while sequences are not transactional (increments are
not subject to ROLLBACK), relfilenode changes are. So we do this:

* Changes for sequences created in the same top-level transaction are
  treated as transactional, i.e. just like any other change from that
  transaction, and discarded in case of a rollback.

* Changes for sequences created earlier are applied immediately, as if
  performed outside any transaction. This applies also after ALTER
  SEQUENCE, which may create a new relfilenode.

Moreover, if we ever get support for DDL replication, the sequence
won't exist until the transaction gets applied.

Sequences created in the current transaction are tracked in a simple
hash table, identified by a relfilenode. That means a sequence may
already exist, but if a transaction does ALTER SEQUENCE then the
increments for the new relfilenode will be treated as transactional.

For each relfilenode we track the XID of (sub)transaction that created
it, which is needed for cleanup at transaction end. We don't need to
check the XID to decide if an increment is transactional - if we find a
match in the hash table, it has to be the same transaction.

This requires two minor changes to WAL-logging. Firstly, we need to
ensure the sequence record has a valid XID - until now the the increment
might have XID 0 if it was the first change in a subxact. But the
sequence might have been created in the same top-level transaction. So
we ensure the XID is assigned when WAL-logging increments.

The other change is addition of "created" flag, marking increments for
newly created relfilenodes. This makes it easier to maintain the hash
table of sequences that need transactional handling.
Note: This is needed because of subxacts. A XID 0 might still have the
sequence created in a different subxact of the same top-level xact.

This does not include any changes to test_decoding and/or the built-in
replication - those will be committed in separate patches.

A patch adding decoding of sequences was originally submitted by Cary
Huang. This commit reworks various important aspects (e.g. the WAL
logging and transactional/non-transactional handling). However, the
original patch and reviews were very useful.

Author: Tomas Vondra, Cary Huang
Reviewed-by: Peter Eisentraut, Hannu Krosing, Andres Freund
Discussion: https://postgr.es/m/d045f3c2-6cfb-06d3-5540-e63c320df8bc@enterprisedb.com
Discussion: https://postgr.es/m/1710ed7e13b.cd7177461430746.3372264562543607781@highgo.ca
This commit is contained in:
Tomas Vondra
2022-02-10 18:43:28 +01:00
parent 0d4513b613
commit 0da92dc530
10 changed files with 786 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

65
doc/src/sgml/logicaldecoding.sgml
View File

@ -458,6 +458,7 @@ typedef struct OutputPluginCallbacks
LogicalDecodeTruncateCB truncate_cb;
LogicalDecodeCommitCB commit_cb;
LogicalDecodeMessageCB message_cb;
LogicalDecodeSequenceCB sequence_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
LogicalDecodeFilterPrepareCB filter_prepare_cb;
@ -472,6 +473,7 @@ typedef struct OutputPluginCallbacks
LogicalDecodeStreamCommitCB stream_commit_cb;
LogicalDecodeStreamChangeCB stream_change_cb;
LogicalDecodeStreamMessageCB stream_message_cb;
LogicalDecodeStreamSequenceCB stream_sequence_cb;
LogicalDecodeStreamTruncateCB stream_truncate_cb;
} OutputPluginCallbacks;
@ -481,9 +483,11 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
and <function>commit_cb</function> callbacks are required,
while <function>startup_cb</function>,
<function>filter_by_origin_cb</function>, <function>truncate_cb</function>,
and <function>shutdown_cb</function> are optional.
If <function>truncate_cb</function> is not set but a
<function>sequence_cb</function>, and <function>shutdown_cb</function> are
optional. If <function>truncate_cb</function> is not set but a
<command>TRUNCATE</command> is to be decoded, the action will be ignored.
Similarly, if <function>sequence_cb</function> is not set and a sequence
change is to be decoded, the action will be ignored.
</para>
<para>
@ -492,7 +496,8 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
<function>stream_stop_cb</function>, <function>stream_abort_cb</function>,
<function>stream_commit_cb</function>, <function>stream_change_cb</function>,
and <function>stream_prepare_cb</function>
are required, while <function>stream_message_cb</function> and
are required, while <function>stream_message_cb</function>,
<function>stream_sequence_cb</function>, and
<function>stream_truncate_cb</function> are optional.
</para>
@ -808,6 +813,35 @@ typedef void (*LogicalDecodeMessageCB) (struct LogicalDecodingContext *ctx,
</para>
</sect3>
<sect3 id="logicaldecoding-output-plugin-sequence">
<title>Sequence Callback</title>
<para>
The optional <function>sequence_cb</function> callback is called for
actions that update a sequence value.
<programlisting>
typedef void (*LogicalDecodeSequenceCB) (struct LogicalDecodingContext *ctx,
ReorderBufferTXN *txn,
XLogRecPtr sequence_lsn,
Relation rel,
bool transactional,
int64 last_value,
int64 log_cnt,
bool is_called);
</programlisting>
The <parameter>txn</parameter> parameter contains meta information about
the transaction the sequence change is part of. Note however that for
non-transactional increments, the transaction may be either NULL or not
NULL, depending on if the transaction already has XID assigned.
The <parameter>sequence_lsn</parameter> has WAL location of the sequence
update. The <parameter>transactional</parameter> says if the sequence has
to be replayed as part of the transaction or directly.
The <parameter>last_value</parameter>, <parameter>log_cnt</parameter> and
<parameter>is_called</parameter> parameters describe the sequence change.
</para>
</sect3>
<sect3 id="logicaldecoding-output-plugin-filter-prepare">
<title>Prepare Filter Callback</title>
@ -1017,6 +1051,26 @@ typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx
</para>
</sect3>
<sect3 id="logicaldecoding-output-plugin-stream-sequence">
<title>Stream Sequence Callback</title>
<para>
The optional <function>stream_sequence_cb</function> callback is called
for actions that change a sequence in a block of streamed changes
(demarcated by <function>stream_start_cb</function> and
<function>stream_stop_cb</function> calls).
<programlisting>
typedef void (*LogicalDecodeStreamSequenceCB) (struct LogicalDecodingContext *ctx,
ReorderBufferTXN *txn,
XLogRecPtr sequence_lsn,
Relation rel,
bool transactional,
int64 last_value,
int64 log_cnt,
bool is_called);
</programlisting>
</para>
</sect3>
<sect3 id="logicaldecoding-output-plugin-stream-truncate">
<title>Stream Truncate Callback</title>
<para>
@ -1197,8 +1251,9 @@ OutputPluginWrite(ctx, true);
in-progress transactions. There are multiple required streaming callbacks
(<function>stream_start_cb</function>, <function>stream_stop_cb</function>,
<function>stream_abort_cb</function>, <function>stream_commit_cb</function>
and <function>stream_change_cb</function>) and two optional callbacks
(<function>stream_message_cb</function> and <function>stream_truncate_cb</function>).
and <function>stream_change_cb</function>) and multiple optional callbacks
(<function>stream_message_cb</function>, <function>stream_sequence_cb</function>,
and <function>stream_truncate_cb</function>).
Also, if streaming of two-phase commands is to be supported, then additional
callbacks must be provided. (See <xref linkend="logicaldecoding-two-phase-commits"/>
for details).