database/postgres
1
0
Fork 0
mirror of https://github.com/postgres/postgres.git synced 2025-07-30 11:03:19 +03:00
Code Activity

Custom WAL Resource Managers.

Browse Source 
Allow extensions to specify a new custom resource manager (rmgr),
which allows specialized WAL. This is meant to be used by a Table
Access Method or Index Access Method.

Prior to this commit, only Generic WAL was available, which offers
support for recovery and physical replication but not logical
replication.

Reviewed-by: Julien Rouhaud, Bharath Rupireddy, Andres Freund
Discussion: https://postgr.es/m/ed1fb2e22d15d3563ae0eb610f7b61bb15999c0a.camel%40j-davis.com
This commit is contained in:
Jeff Davis
2022-04-06 22:26:43 -07:00
parent a8cfb0c1a9
commit 5c279a6d35
24 changed files with 526 additions and 70 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
doc/src/sgml/config.sgml
View File

@ -11189,8 +11189,8 @@ LOG: CleanUpLock: deleting: lock(0xb7acd844) id(24688,24696,0,0,0,1)
<literal>heap2</literal>, <literal>btree</literal>, <literal>hash</literal>,
<literal>gin</literal>, <literal>gist</literal>, <literal>sequence</literal>,
<literal>spgist</literal>, <literal>brin</literal>, and <literal>generic</literal>.
Only superusers and users with the appropriate <literal>SET</literal>
privilege can change this setting.
Extensions may define additional resource managers. Only superusers and users with
the appropriate <literal>SET</literal> privilege can change this setting.
</para>
</listitem>
</varlistentry>

98
doc/src/sgml/custom-rmgr.sgml Normal file
View File

@ -0,0 +1,98 @@
<!-- doc/src/sgml/custom-rmgr.sgml -->
<chapter id="custom-rmgr">
<title>Custom WAL Resource Managers</title>
<para>
This chapter explains the interface between the core
<productname>PostgreSQL</productname> system and custom WAL resource
managers, which enable extensions to integrate directly with the <link
linkend="wal"><acronym>WAL</acronym></link>.
</para>
<para>
An extension, especially a <link linkend="tableam">Table Access
Method</link> or <link linkend="indexam">Index Access Method</link>, may
need to use WAL for recovery, replication, and/or <link
linkend="logicaldecoding">Logical Decoding</link>. Custom resource managers
are a more flexible alternative to <link linkend="generic-wal">Generic
WAL</link> (which does not support logical decoding), but more complex for
an extension to implement.
</para>
<para>
To create a new custom WAL resouce manager, first define an
<structname>RmgrData</structname> structure with implementations for the
resource manager methods. Refer to
<filename>src/backend/access/transam/README</filename> and
<filename>src/include/access/xlog_internal.h</filename> in the
<productname>PostgreSQL</productname> source.
<programlisting>
/*
* Method table for resource managers.
*
* This struct must be kept in sync with the PG_RMGR definition in
* rmgr.c.
*
* rm_identify must return a name for the record based on xl_info (without
* reference to the rmid). For example, XLOG_BTREE_VACUUM would be named
* "VACUUM". rm_desc can then be called to obtain additional detail for the
* record, if available (e.g. the last block).
*
* rm_mask takes as input a page modified by the resource manager and masks
* out bits that shouldn't be flagged by wal_consistency_checking.
*
* RmgrTable[] is indexed by RmgrId values (see rmgrlist.h). If rm_name is
* NULL, the corresponding RmgrTable entry is considered invalid.
*/
typedef struct RmgrData
{
const char *rm_name;
void (*rm_redo) (XLogReaderState *record);
void (*rm_desc) (StringInfo buf, XLogReaderState *record);
const char *(*rm_identify) (uint8 info);
void (*rm_startup) (void);
void (*rm_cleanup) (void);
void (*rm_mask) (char *pagedata, BlockNumber blkno);
void (*rm_decode) (struct LogicalDecodingContext *ctx,
struct XLogRecordBuffer *buf);
} RmgrData;
</programlisting>
</para>
<para>
Then, register your new resource
manager.
<programlisting>
/*
* Register a new custom WAL resource manager.
*
* Resource manager IDs must be globally unique across all extensions. Refer
* to https://wiki.postgresql.org/wiki/CustomWALResourceManager to reserve a
* unique RmgrId for your extension, to avoid conflicts with other extension
* developers. During development, use RM_EXPERIMENTAL_ID to avoid needlessly
* reserving a new ID.
*/
extern void RegisterCustomRmgr(RmgrId rmid, RmgrData *rmgr);
</programlisting>
<function>RegisterCustomRmgr</function> must be called from the
extension module's <link linkend="xfunc-c-dynload">_PG_init</link> function.
While developing a new extension, use <literal>RM_EXPERIMENTAL_ID</literal>
for <parameter>rmid</parameter>. When you ready to release the extension to
users, reserve a new resource manager ID at the <ulink
url="https://wiki.postgresql.org/wiki/CustomWALResourceManagers">Custom WAL
Resource Manager</ulink> page.
</para>
<para>
Place the extension module implementing the custom resource manager in <xref
linkend="guc-shared-preload-libraries"/> so that it will be loaded early
during <productname>PostgreSQL</productname> startup.
</para>
<note>
<para>
The extension must remain in shared_preload_libraries as long as any
custom WAL records may exist in the system. Otherwise
<productname>PostgreSQL</productname> will not be able to apply or decode
the custom WAL records, which may prevent the server from starting.
</para>
</note>
</chapter>

1
doc/src/sgml/filelist.sgml
View File

@ -105,6 +105,7 @@
<!ENTITY storage SYSTEM "storage.sgml">
<!ENTITY tablesample-method SYSTEM "tablesample-method.sgml">
<!ENTITY generic-wal SYSTEM "generic-wal.sgml">
<!ENTITY custom-rmgr SYSTEM "custom-rmgr.sgml">
<!ENTITY backup-manifest SYSTEM "backup-manifest.sgml">
<!-- contrib information -->

19
doc/src/sgml/func.sgml
View File

@ -26001,6 +26001,25 @@ postgres=# SELECT * FROM pg_walfile_name_offset((pg_backup_stop()).lsn);
without recovery, the function returns <literal>NULL</literal>.
</para></entry>
</row>
<row>
<entry role="func_table_entry"><para role="func_signature">
<indexterm>
<primary>pg_get_wal_resource_managers</primary>
</indexterm>
<function>pg_get_wal_resource_managers</function> ()
<returnvalue>setof record</returnvalue>
( <parameter>rm_id</parameter> <type>integer</type>,
<parameter>rm_name</parameter> <type>text</type>,
<parameter>rm_builtin</parameter> <type>boolean</type> )
</para>
<para>
Returns the currently-loaded WAL resource managers in the system. The
column <parameter>rm_builtin</parameter> indicates whether it's a
built-in resource manager, or a custom resource manager loaded by an
extension.
</para></entry>
</row>
</tbody>
</tgroup>
</table>

20
doc/src/sgml/generic-wal.sgml
View File

@ -6,11 +6,25 @@
<para>
Although all built-in WAL-logged modules have their own types of WAL
records, there is also a generic WAL record type, which describes changes
to pages in a generic way. This is useful for extensions that provide
custom access methods, because they cannot register their own WAL redo
routines.
to pages in a generic way. This is useful for extensions that provide
custom access methods.
</para>
<para>
In comparison with <link linkend="custom-rmgr">Custom WAL Resource
Managers</link>, Generic WAL is simpler for an extension to implement and
does not require the extension library to be loaded in order to apply the
records.
</para>
<note>
<para>
Generic WAL records are ignored during <link
linkend="logicaldecoding">Logical Decoding</link>. If logical decoding is
required for your extension, consider a Custom WAL Resource Manager.
</para>
</note>
<para>
The API for constructing generic WAL records is defined in
<filename>access/generic_xlog.h</filename> and implemented

1
doc/src/sgml/postgres.sgml
View File

@ -262,6 +262,7 @@ break is not needed in a wider output rendering.
&tableam;
&indexam;
&generic-wal;
&custom-rmgr;
&btree;
&gist;
&spgist;

8
doc/src/sgml/ref/pg_waldump.sgml
View File

@ -173,6 +173,14 @@ PostgreSQL documentation
If <literal>list</literal> is passed as name, print a list of valid resource manager
names, and exit.
</para>
<para>
Extensions may define custom resource managers, but pg_waldump does
not load the extension module and therefore does not recognize custom
resource managers by name. Instead, you can specify the custom
resource managers as <literal>custom###</literal> where
"<literal>###</literal>" is the three-digit resource manager ID. Names
of this form will always be considered valid.
</para>
</listitem>
</varlistentry>