database/postgres
1
0
Fork 0
mirror of https://github.com/postgres/postgres.git synced 2026-10-23 13:29:09 +03:00
Code Activity

Revert online data checksum transitions

Browse Source 
The feature to enable, or disable, data checksums in an online
cluster saw a number of postcommit fixes during the beta period.
Suspicions were raised about the risk of more issues surfacing
after GA.  To avoid shipping code which may have bugs, this
reverts in full, or in part, the following commits:

aaf8b9989f, cd857dec0e, 5fee7cab1b, 25b922ec58, 8fb8ded889,
b3a37ffbc5, abac86c7a2, 3a18526e8d, 01805b7d16, 343d98c360,
e469e4784e, 8d22f52324, e5e1f6dc79, 5ab239c9a9, 0ca1b30105,
9a39056c41, 2018bd6167, 1df361e3d8, bf25e5571b, 381d19da15,
b120358c61, 07009121c2, d771b0a907, f19c0eccae, 0907112d38,
397f0fd06e, 602f19c84c, e3a27cad46, 4ae3e98c02, b364828f82

A few bits remain as they have merit on their own:
* The fix for offline checksums not preserving initial state in
  pg_control_init.  The test case for this fix has been moved to
  src/bin/pg_checksums/t/002_actions.pl to survive the revert
  which removed the previous location.
* When a page verification is logged, the additional note which
  informs if the buffer will be zeroed out was kept to aid any
  debugging around checksum failures.
* The Data Checksum section in the docs was kept, but rewritten
  to reflect the current state.
* The enum containing checksum states is kept and some hardcoded
  references to the off state (0) are replaced with the label
  PG_DATA_CHECKSUM_OFF instead.

Discussion: https://postgr.es/m/E15AC050-C4B5-488D-BB2D-3C7AC9F89EA8@yesql.se
Discussion: https://postgr.es/m/anwm6UPxoVS41QA2@bdtpg
Discussion: https://postgr.es/m/CA+Tgmob9NY6m0YNFTQ4nFH2d0iC9SQRruDYxfndGKKzh8OC80w@mail.gmail.com
Discussion: https://postgr.es/m/E07A611B-9CF3-4FDB-8CE8-A221E39040EC%40yesql.se
This commit is contained in:
Daniel Gustafsson
2026-09-16 11:01:27 +02:00
parent 0cf920d058
commit c05d5ce123
82 changed files with 79 additions and 6673 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
doc/src/sgml/config.sgml
View File

@@ -7498,7 +7498,6 @@ local0.* /var/log/postgresql
<member><literal>bgworker</literal></member>
<member><literal>bgwriter</literal></member>
<member><literal>checkpointer</literal></member>
<member><literal>checksums</literal></member>
<member><literal>ioworker</literal></member>
<member><literal>postmaster</literal></member>
<member><literal>slotsyncworker</literal></member>
@@ -12305,7 +12304,7 @@ dynamic_library_path = '/usr/local/lib/postgresql:$libdir'
</varlistentry>
<varlistentry id="guc-data-checksums" xreflabel="data_checksums">
<term><varname>data_checksums</varname> (<type>enum</type>)
<term><varname>data_checksums</varname> (<type>bool</type>)
<indexterm>
<primary><varname>data_checksums</varname> configuration parameter</primary>
</indexterm>
@@ -12313,8 +12312,6 @@ dynamic_library_path = '/usr/local/lib/postgresql:$libdir'
<listitem>
<para>
Reports the state of data checksums for this cluster.
Possible values are <literal>on</literal>, <literal>off</literal>,
<literal>inprogress-on</literal> and <literal>inprogress-off</literal>.
See <xref linkend="checksums"/> for more information.
</para>
</listitem>

79
doc/src/sgml/func/func-admin.sgml
View File

@@ -3126,83 +3126,4 @@ SELECT convert_from(pg_read_binary_file('file_in_utf8.txt'), 'UTF8');
</table>
</sect2>
<sect2 id="functions-admin-checksum">
<title>Data Checksum Functions</title>
<para>
The functions shown in <xref linkend="functions-checksums-table" /> can
be used to enable or disable data checksums in a running cluster.
</para>
<para>
Changing data checksums can be done in a cluster with concurrent activity
without blocking queries, but overall system performance will be affected.
See <xref linkend="checksums" /> for further details on how changing the
data checksums state can affect a system and possible mitigations for how
to reduce the impact.
</para>
<table id="functions-checksums-table">
<title>Data Checksum Functions</title>
<tgroup cols="1">
<thead>
<row>
<entry role="func_table_entry"><para role="func_signature">
Function
</para>
<para>
Description
</para></entry>
</row>
</thead>
<tbody>
<row>
<entry role="func_table_entry"><para role="func_signature">
<indexterm>
<primary>pg_enable_data_checksums</primary>
</indexterm>
<function>pg_enable_data_checksums</function> ( <optional><parameter>cost_delay</parameter> <type>int</type>, <parameter>cost_limit</parameter> <type>int</type></optional> )
<returnvalue>void</returnvalue>
</para>
<para>
Initiates the process of enabling data checksums for the cluster. This
will set the data checksums state to <literal>inprogress-on</literal>
as well as start a background worker that will process all pages in all
databases and enable data checksums on them. When all pages have
been processed, the cluster will automatically set data checksums state
to <literal>on</literal>. This operation is WAL logged and replicated
to all standby nodes.
</para>
<para>
If <parameter>cost_delay</parameter> and <parameter>cost_limit</parameter> are
specified, the process is throttled using the same principles as
<link linkend="runtime-config-resource-vacuum-cost">Cost-based Vacuum Delay</link>.
</para>
</entry>
</row>
<row>
<entry role="func_table_entry"><para role="func_signature">
<indexterm>
<primary>pg_disable_data_checksums</primary>
</indexterm>
<function>pg_disable_data_checksums</function> ()
<returnvalue>void</returnvalue>
</para>
<para>
Disables data checksum calculation and validation for the cluster. This
will set the data checksum state to <literal>inprogress-off</literal>
while data checksums are being disabled. When all active backends have
stopped validating data checksums, the data checksum state will be
set to <literal>off</literal>.
</para>
</entry>
</row>
</tbody>
</tgroup>
</table>
</sect2>
</sect1>

5
doc/src/sgml/func/func-info.sgml
View File

@@ -3481,11 +3481,6 @@ acl | {postgres=arwdDxtm/postgres,foo=r/postgres}
<entry><type>xid</type></entry>
</row>
<row>
<entry><structfield>data_page_checksum_version</structfield></entry>
<entry><type>integer</type></entry>
</row>
<row>
<entry><structfield>checkpoint_time</structfield></entry>
<entry><type>timestamp with time zone</type></entry>

20
doc/src/sgml/glossary.sgml
View File

@@ -234,9 +234,8 @@
which runs system- or user-supplied code.
Serves as infrastructure for several features in
<productname>PostgreSQL</productname>, such as
<glossterm linkend="glossary-replication">logical replication</glossterm>,
<glossterm linkend="glossary-parallel-query">parallel queries</glossterm> and
<glossterm linkend="glossary-data-checksums-process">data checksums processing</glossterm>.
<glossterm linkend="glossary-replication">logical replication</glossterm> and
<glossterm linkend="glossary-parallel-query">parallel queries</glossterm>.
In addition, <glossterm linkend="glossary-extension">Extensions</glossterm> can add
custom background worker processes.
</para>
@@ -575,21 +574,6 @@
<glosssee otherterm="glossary-data-directory" />
</glossentry>
<glossentry id="glossary-data-checksums-process">
<glossterm>Data Checksums (process)</glossterm>
<glossdef>
<para>
A set of
<glossterm linkend="glossary-background-worker">background worker</glossterm>
processes which can enable or disable data checksums in a running cluster.
The process which coordinates the work is known as the
<firstterm>data checksums launcher</firstterm> and the process which
operates on the individual databases is known as the
<firstterm>data checksums worker</firstterm>.
</para>
</glossdef>
</glossentry>
<glossentry id="glossary-database">
<glossterm>Database</glossterm>
<glossdef>

1
doc/src/sgml/images/Makefile
View File

@@ -3,7 +3,6 @@
# see README in this directory about image handling
ALL_IMAGES = \
datachecksums.svg \
genetic-algorithm.svg \
gin.svg \
pagelayout.svg \

14
doc/src/sgml/images/datachecksums.gv
View File

@@ -1,14 +0,0 @@
digraph G {
A -> B [label="SELECT pg_enable_data_checksums()"];
B -> C;
D -> A;
C -> D [label="SELECT pg_disable_data_checksums()"];
E -> A [label=" --no-data-checksums"];
E -> C [label=" --data-checksums"];
A [label="off"];
B [label="inprogress-on"];
C [label="on"];
D [label="inprogress-off"];
E [label="initdb"];
}

81
doc/src/sgml/images/datachecksums.svg
View File

@@ -1,81 +0,0 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!-- Generated by graphviz version 14.0.5 (20251129.0259)
-->
<!-- Title: G Pages: 1 -->
<svg width="409pt" height="383pt"
viewBox="0.00 0.00 409.00 383.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 378.5)">
<title>G</title>
<polygon fill="white" stroke="none" points="-4,4 -4,-378.5 404.74,-378.5 404.74,4 -4,4"/>
<!-- A -->
<g id="node1" class="node">
<title>A</title>
<ellipse fill="none" stroke="black" cx="80.12" cy="-268" rx="27" ry="18"/>
<text xml:space="preserve" text-anchor="middle" x="80.12" y="-262.95" font-family="Times,serif" font-size="14.00">off</text>
</g>
<!-- B -->
<g id="node2" class="node">
<title>B</title>
<ellipse fill="none" stroke="black" cx="137.12" cy="-179.5" rx="61.59" ry="18"/>
<text xml:space="preserve" text-anchor="middle" x="137.12" y="-174.45" font-family="Times,serif" font-size="14.00">inprogress&#45;on</text>
</g>
<!-- A&#45;&gt;B -->
<g id="edge1" class="edge">
<title>A&#45;&gt;B</title>
<path fill="none" stroke="black" d="M76.5,-249.68C75.22,-239.14 75.3,-225.77 81.12,-215.5 84.2,-210.08 88.49,-205.38 93.35,-201.34"/>
<polygon fill="black" stroke="black" points="95.22,-204.31 101.33,-195.66 91.16,-198.61 95.22,-204.31"/>
<text xml:space="preserve" text-anchor="middle" x="187.62" y="-218.7" font-family="Times,serif" font-size="14.00">SELECT pg_enable_data_checksums()</text>
</g>
<!-- C -->
<g id="node3" class="node">
<title>C</title>
<ellipse fill="none" stroke="black" cx="137.12" cy="-106.5" rx="27" ry="18"/>
<text xml:space="preserve" text-anchor="middle" x="137.12" y="-101.45" font-family="Times,serif" font-size="14.00">on</text>
</g>
<!-- B&#45;&gt;C -->
<g id="edge2" class="edge">
<title>B&#45;&gt;C</title>
<path fill="none" stroke="black" d="M137.12,-161.31C137.12,-153.73 137.12,-144.6 137.12,-136.04"/>
<polygon fill="black" stroke="black" points="140.62,-136.04 137.12,-126.04 133.62,-136.04 140.62,-136.04"/>
</g>
<!-- D -->
<g id="node4" class="node">
<title>D</title>
<ellipse fill="none" stroke="black" cx="63.12" cy="-18" rx="63.12" ry="18"/>
<text xml:space="preserve" text-anchor="middle" x="63.12" y="-12.95" font-family="Times,serif" font-size="14.00">inprogress&#45;off</text>
</g>
<!-- C&#45;&gt;D -->
<g id="edge4" class="edge">
<title>C&#45;&gt;D</title>
<path fill="none" stroke="black" d="M124.23,-90.43C113.36,-77.73 97.58,-59.28 84.77,-44.31"/>
<polygon fill="black" stroke="black" points="87.78,-42.44 78.62,-37.12 82.46,-46.99 87.78,-42.44"/>
<text xml:space="preserve" text-anchor="middle" x="214.75" y="-57.2" font-family="Times,serif" font-size="14.00">SELECT pg_disable_data_checksums()</text>
</g>
<!-- D&#45;&gt;A -->
<g id="edge3" class="edge">
<title>D&#45;&gt;A</title>
<path fill="none" stroke="black" d="M62.52,-36.28C61.62,-68.21 60.54,-138.57 66.12,-197.5 67.43,-211.24 70.27,-226.28 73.06,-238.85"/>
<polygon fill="black" stroke="black" points="69.64,-239.59 75.32,-248.54 76.46,-238 69.64,-239.59"/>
</g>
<!-- E -->
<g id="node5" class="node">
<title>E</title>
<ellipse fill="none" stroke="black" cx="198.12" cy="-356.5" rx="32.41" ry="18"/>
<text xml:space="preserve" text-anchor="middle" x="198.12" y="-351.45" font-family="Times,serif" font-size="14.00">initdb</text>
</g>
<!-- E&#45;&gt;A -->
<g id="edge5" class="edge">
<title>E&#45;&gt;A</title>
<path fill="none" stroke="black" d="M179.16,-341.6C159.64,-327.29 129.05,-304.86 107.03,-288.72"/>
<polygon fill="black" stroke="black" points="109.23,-286 99.1,-282.91 105.09,-291.64 109.23,-286"/>
<text xml:space="preserve" text-anchor="middle" x="208.57" y="-307.2" font-family="Times,serif" font-size="14.00"> &#45;&#45;no&#45;data&#45;checksums</text>
</g>
<!-- E&#45;&gt;C -->
<g id="edge6" class="edge">
<title>E&#45;&gt;C</title>
<path fill="none" stroke="black" d="M227.13,-348.04C242.29,-342.72 259.95,-334.06 271.12,-320.5 301.5,-283.62 316.36,-257.78 294.12,-215.5 268.41,-166.6 209.42,-135.53 171.52,-119.85"/>
<polygon fill="black" stroke="black" points="172.96,-116.65 162.37,-116.21 170.37,-123.16 172.96,-116.65"/>
<text xml:space="preserve" text-anchor="middle" x="350.87" y="-218.7" font-family="Times,serif" font-size="14.00"> &#45;&#45;data&#45;checksums</text>
</g>
</g>
</svg>
Side by Side

Before

Width:  |  Height:  |  Size: 4.2 KiB

228
doc/src/sgml/monitoring.sgml
View File

@@ -436,15 +436,6 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
</entry>
</row>
<row>
<entry><structname>pg_stat_progress_data_checksums</structname><indexterm><primary>pg_stat_progress_data_checksums</primary></indexterm></entry>
<entry>One row for the data checksum launcher process while data
checksums are being enabled or disabled. When enabling data
checksums, the view also has one row for each worker process,
showing current progress.
See <xref linkend="data-checksum-progress-reporting"/>.
</entry>
</row>
</tbody>
</tgroup>
</table>
@@ -6136,10 +6127,10 @@ FROM pg_stat_get_backend_idset() AS backendid;
which support progress reporting are <command>ANALYZE</command>,
<command>COPY</command>, <command>CREATE INDEX</command>,
<command>REPACK</command> (and its obsolete spelling <command>CLUSTER</command>),
<command>VACUUM</command>,
<command>VACUUM</command> and
<xref linkend="protocol-replication-base-backup"/> (i.e., replication
command that <xref linkend="app-pgbasebackup"/> issues to take
a base backup), and online data checksum operations.
a base backup).
This may be expanded in the future.
</para>
@@ -7867,221 +7858,6 @@ FROM pg_stat_get_backend_idset() AS backendid;
</table>
</sect2>
<sect2 id="data-checksum-progress-reporting">
<title>Data Checksum Progress Reporting</title>
<indexterm>
<primary>pg_stat_progress_data_checksums</primary>
</indexterm>
<para>
When data checksums are being enabled or disabled on a running cluster, the
<structname>pg_stat_progress_data_checksums</structname> view will contain
a row for the launcher process. When enabling data checksums, the view
will also contain one row for each worker process which is currently
calculating and writing checksums for the data pages in a database. The
launcher provides an overview of the overall progress, such as how many
databases have been processed and how many remain, while the workers track
progress for currently processed databases.
</para>
<table id="pg-stat-progress-data-checksums-view" xreflabel="pg_stat_progress_data_checksums">
<title><structname>pg_stat_progress_data_checksums</structname> View</title>
<tgroup cols="1">
<thead>
<row>
<entry role="catalog_table_entry">
<para role="column_definition">
Column Type
</para>
<para>
Description
</para>
</entry>
</row>
</thead>
<tbody>
<row>
<entry role="catalog_table_entry">
<para role="column_definition">
<structfield>pid</structfield> <type>integer</type>
</para>
<para>
Process ID of the data checksum process, launcher or worker.
</para>
</entry>
</row>
<row>
<entry role="catalog_table_entry">
<para role="column_definition">
<structfield>datid</structfield> <type>oid</type>
</para>
<para>
OID of this database, or <literal>0</literal> for the launcher process.
</para>
</entry>
</row>
<row>
<entry role="catalog_table_entry">
<para role="column_definition">
<structfield>datname</structfield> <type>name</type>
</para>
<para>
Name of this database, or <literal>NULL</literal> for the
launcher process.
</para>
</entry>
</row>
<row>
<entry role="catalog_table_entry">
<para role="column_definition">
<structfield>phase</structfield> <type>text</type>
</para>
<para>
Current processing phase, see <xref linkend="datachecksum-phases"/>
for description of the phases.
</para>
</entry>
</row>
<row>
<entry role="catalog_table_entry">
<para role="column_definition">
<structfield>databases_total</structfield> <type>bigint</type>
</para>
<para>
The total number of databases which will be processed. Only the
launcher process has this value set when enabling data checksums;
otherwise this is set to <literal>NULL</literal>.
</para>
</entry>
</row>
<row>
<entry role="catalog_table_entry">
<para role="column_definition">
<structfield>databases_done</structfield> <type>bigint</type>
</para>
<para>
The number of databases which have been processed. Only the launcher
process has this value set when enabling data checksums; otherwise
this is set to <literal>NULL</literal>.
</para>
</entry>
</row>
<row>
<entry role="catalog_table_entry">
<para role="column_definition">
<structfield>relations_total</structfield> <type>bigint</type>
</para>
<para>
The total number of relations which will be processed, or
<literal>NULL</literal> if the worker process hasn't
calculated the number of relations yet. The launcher process has
this set to <literal>NULL</literal> since it isn't responsible for
processing relations, only launching worker processes.
</para>
</entry>
</row>
<row>
<entry role="catalog_table_entry">
<para role="column_definition">
<structfield>relations_done</structfield> <type>bigint</type>
</para>
<para>
The number of relations which have been processed. The launcher
process has this set to <literal>NULL</literal>.
</para>
</entry>
</row>
<row>
<entry role="catalog_table_entry">
<para role="column_definition">
<structfield>blocks_total</structfield> <type>bigint</type>
</para>
<para>
The number of blocks in the current relation which will be processed,
or <literal>NULL</literal> if the worker process hasn't
calculated the number of blocks yet. The launcher process has
this set to <literal>NULL</literal>.
</para>
</entry>
</row>
<row>
<entry role="catalog_table_entry">
<para role="column_definition">
<structfield>blocks_done</structfield> <type>bigint</type>
</para>
<para>
The number of blocks in the current relation which have been processed.
The launcher process has this set to <literal>NULL</literal>.
</para>
</entry>
</row>
</tbody>
</tgroup>
</table>
<table id="datachecksum-phases">
<title>Data Checksum Phases</title>
<tgroup cols="2">
<colspec colname="col1" colwidth="1*"/>
<colspec colname="col2" colwidth="2*"/>
<thead>
<row>
<entry>Phase</entry>
<entry>Description</entry>
</row>
</thead>
<tbody>
<row>
<entry><literal>enabling</literal></entry>
<entry>
The command is currently enabling data checksums on the cluster.
</entry>
</row>
<row>
<entry><literal>disabling</literal></entry>
<entry>
The command is currently disabling data checksums on the cluster.
</entry>
</row>
<row>
<entry><literal>done</literal></entry>
<entry>
The command is done and the data checksum state in the cluster has
changed.
</entry>
</row>
<row>
<entry><literal>waiting on barrier</literal></entry>
<entry>
The command is currently waiting for the current active backends to
acknowledge the change in data checksum state.
</entry>
</row>
<row>
<entry><literal>waiting on temporary tables</literal></entry>
<entry>
The command is currently waiting for all temporary tables which existed
at the time the command was started to be removed.
</entry>
</row>
</tbody>
</tgroup>
</table>
</sect2>
</sect1>
<sect1 id="dynamic-trace">

6
doc/src/sgml/ref/pg_basebackup.sgml
View File

@@ -788,12 +788,6 @@ PostgreSQL documentation
in the <link linkend="monitoring-pg-stat-database-view">
<structname>pg_stat_database</structname></link> view.
</para>
<para>
Checksums must be enabled on the server for the duration of the base
backup in order to be verified. If checksums are in the process of
being enabled when the base backup starts then checksum verification is
disabled for the base backup.
</para>
</listitem>
</varlistentry>
</variablelist>

12
doc/src/sgml/ref/pg_checksums.sgml
View File

@@ -39,23 +39,13 @@ PostgreSQL documentation
<application>pg_checksums</application> checks, enables or disables data
checksums in a <productname>PostgreSQL</productname> cluster. The server
must be shut down cleanly before running <application>pg_checksums</application>.
Checksums can also be enabled while the cluster is running using
<xref linkend="checksums-online-enable-disable"/> processing, see
<xref linkend="checksums"/> for further details on the different approaches.
See <xref linkend="checksums"/> for further details on data checksums.
When verifying checksums, the exit
status is zero if there are no checksum errors, and nonzero if at least one
checksum failure is detected. When enabling or disabling checksums, the
exit status is nonzero if the operation failed.
</para>
<para>
When enabling checksums with <application>pg_checksums</application>, if
checksums were in the process of being enabled using
<xref linkend="checksums-online-enable-disable"/> when the cluster was shut
down, <application>pg_checksums</application> will still process all
relation files regardless of the progress of online checksum processing.
</para>
<para>
When verifying checksums, every file in the cluster is scanned. When
enabling checksums, each relation file block with a changed checksum is

14
doc/src/sgml/regress.sgml
View File

@@ -275,20 +275,6 @@ make check-world PG_TEST_EXTRA='kerberos ldap ssl load_balance libpq_encryption'
</programlisting>
The following values are currently supported:
<variablelist>
<varlistentry>
<term><literal>checksum</literal>, <literal>checksum_extended</literal></term>
<listitem>
<para>
Runs additional tests for enabling data checksums which inject faults
to cause re-tries in the processing, as well as tests that run pgbench
concurrently and randomly restarts the cluster. Some of these test
suites require injection points enabled in the installation.
<literal>checksum_extended</literal> is an extended version with
longer runtime, injected random delays and larger datasets.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><literal>kerberos</literal></term>
<listitem>

35
doc/src/sgml/release-19.sgml
View File

@@ -47,13 +47,6 @@
</para>
</listitem>
<listitem>
<para>
Data checksums can now be
<link linkend="checksums-online-enable-disable">enabled or disabled while the database server is running</link>.
</para>
</listitem>
<listitem>
<para>
A new <link linkend="sql-wait"><command>WAIT</command></link>
@@ -1353,34 +1346,6 @@ The previous warning was 40 million. Warnings are issued to clients and in the
<title>Server Configuration</title>
<itemizedlist>
<!--
Author: Daniel Gustafsson <dgustafsson@postgresql.org>
2026-04-03 [f19c0ecca] Online enabling and disabling of data checksums
Author: Daniel Gustafsson <dgustafsson@postgresql.org>
2026-04-08 [b364828f8] doc: Fix data_checksums data type
Author: Daniel Gustafsson <dgustafsson@postgresql.org>
2026-08-18 [397f0fd06] Add data_page_checksum_version to pg_control_checkpoint
-->
<listitem>
<para>
Allow <link linkend="functions-admin-checksum">online enabling</link> and disabling of <link linkend="checksums">data checksums</link> (Daniel Gustafsson, Magnus Hagander, Tomas Vondra)
<ulink url="&commit_baseurl;f19c0ecca">&sect;</ulink>
<ulink url="&commit_baseurl;b364828f8">&sect;</ulink>
<ulink url="&commit_baseurl;397f0fd06">&sect;</ulink>
</para>
<para>
Previously the checksum status could only be changed while the cluster was offline using <link linkend="app-pgchecksums"><application>pg_checksums</application></link>.
</para>
<para>
Application <link
linkend="app-pgcontroldata"><application>pg_controldata</application></link> and function <link linkend="functions-pg-control-checkpoint"><function>pg_control_checkpoint()</function></link> report
the checksum status at the time of the most recent checkpoint, and function <link linkend="functions-pg-control-init"><function>pg_control_init()</function></link> reports
checksum status at the time the cluster was initialized.
</para>
</listitem>
<!--
Author: Nathan Bossart <nathan@postgresql.org>

185
doc/src/sgml/wal.sgml
View File

@@ -236,20 +236,20 @@
</indexterm>
<para>
By default, data pages are protected by checksums, but this can
optionally be disabled for a cluster. When enabled, each data page includes
a checksum that is updated when the page is written and verified each time
the page is read. Only data pages are protected by checksums; internal data
structures and temporary files are not.
By default, data pages are protected by checksums, but this can optionally
be disabled for a cluster. When enabled, each data page includes a checksum
that is updated when the page is written and verified each time the page is
read. Only data pages are protected by checksums; internal data structures
and temporary files are not.
</para>
<para>
Checksums can be disabled when the cluster is initialized using <link
linkend="app-initdb-data-checksums"><application>initdb</application></link>.
They can also be enabled or disabled at a later time either as an offline
operation or online in a running cluster allowing concurrent access. Data
checksums are enabled or disabled at the full cluster level, and cannot be
specified individually for databases, tables or replicated cluster members.
They can also be enabled or disabled at a later time as an offline
operation. Data checksums are enabled or disabled at the full cluster
level, and cannot be specified individually for databases, tables or
replicated cluster members.
</para>
<para>
@@ -257,51 +257,11 @@
value of the read-only configuration variable <xref
linkend="guc-data-checksums" /> by issuing the command <command>SHOW
data_checksums</command>. <link linkend="functions-pg-control-init">
<literal>pg_control_init</literal></link> and
<link linkend="functions-pg-control-checkpoint">
<literal>pg_control_checkpoint</literal></link> can also be used for
inspecting the data checksum state at cluster initialization and current
checkpoint. Data checksum states are often referred to as data checksum
versions using an integer representation due to how they were originally
implemented. <xref linkend="data-checksums-mapping" /> contains a mapping
between state names and versions, which are defined in
<filename>src/include/storage/checksum.h</filename>.
<literal>pg_control_init</literal></link>
can also be used for
inspecting the data checksum state at cluster initialization.
</para>
<table id="data-checksums-mapping">
<title>Data Checksums States and Version Mapping</title>
<tgroup cols="2">
<thead>
<row>
<entry>State</entry>
<entry>Version</entry>
</row>
</thead>
<tbody>
<row>
<entry>off</entry>
<entry>0</entry>
</row>
<row>
<entry>on</entry>
<entry>1</entry>
</row>
<row>
<entry>inprogress-off</entry>
<entry>2</entry>
</row>
<row>
<entry>inprogress-on</entry>
<entry>3</entry>
</row>
</tbody>
</tgroup>
</table>
<para>
When attempting to recover from page corruptions, it may be necessary to
bypass the checksum protection. To do this, temporarily set the
@@ -314,127 +274,12 @@
<para>
The <link linkend="app-pgchecksums"><application>pg_checksums</application></link>
application can be used to enable or disable data checksums, as well as
verify checksums, on an offline cluster.
verify checksums, on an offline cluster. The same operation must be
executed on every node in a replicated cluster before the cluster is
restarted.
</para>
</sect2>
<sect2 id="checksums-online-enable-disable" xreflabel="Online Enabling of Checksums">
<title>Online Enabling of Checksums</title>
<para>
Checksums can be enabled or disabled online, by calling the appropriate
<link linkend="functions-admin-checksum">functions</link>.
</para>
<para>
Both enabling and disabling data checksums happens in two phases, separated
by a checkpoint to ensure durability. The different states, and their
transitions, are illustrated in <xref linkend="data-checksums-states-figure"/>
and discussed in further detail in this section.
</para>
<para>
<figure id="data-checksums-states-figure">
<title>Data Checksums States</title>
<mediaobject>
<imageobject>
<imagedata fileref="images/datachecksums.svg" format="SVG" width="100%"/>
</imageobject>
</mediaobject>
</figure>
</para>
<para>
Enabling checksums will set the cluster checksum state to
<literal>inprogress-on</literal>. During this time, checksums will be
written but not verified. In addition to this, a background worker process
is started that enables checksums on all existing data in the cluster. Once
this worker has completed processing all databases in the cluster, the
checksum state will automatically switch to <literal>on</literal>. The
processing will consume two background worker processes, make sure that
<varname>max_worker_processes</varname> allows for at least two more
additional processes.
</para>
<para>
The process will initially wait for all open transactions to finish before
it starts, so that it can be certain that there are no tables that have been
created inside a transaction that has not committed yet and thus would not
be visible to the process enabling checksums. It will also, for each database,
wait for all pre-existing temporary tables to get removed before it finishes.
If long-lived temporary tables are used in an application it may be necessary
to terminate these application connections to allow the process to complete.
</para>
<para>
If the cluster is stopped while in <literal>inprogress-on</literal> state,
for any reason, or processing was interrupted, then the checksum enable
process must be restarted manually. To do this, re-execute the function
<function>pg_enable_data_checksums()</function> once the cluster has been
restarted. The process will start over, there is no support for resuming
work from where it was interrupted. If the cluster is stopped while in
<literal>inprogress-off</literal>, then the checksum state will be set to
<literal>off</literal> when the cluster is restarted.
</para>
<para>
Disabling data checksums will set the data checksum state to
<literal>inprogress-off</literal>. During this time, checksums will be
written but not verified. After all processes acknowledge the change,
the state will automatically be set to <literal>off</literal>.
</para>
<para>
Disabling data checksums while data checksums are actively being enabled
will abort the current processing.
</para>
<sect3 id="checksums-online-system-impact">
<title>Impact on System of Online Operations</title>
<para>
Enabling data checksums can cause significant I/O to the system, as all of the
database pages will need to be rewritten, and will be written both to the
data files and the WAL. The impact may be limited by throttling using the
<parameter>cost_delay</parameter> and <parameter>cost_limit</parameter>
parameters of the <function>pg_enable_data_checksums()</function> function.
</para>
<para>
<itemizedlist>
<listitem><para>
I/O: all pages need to have data checksums calculated and written which
will generate a lot of dirty pages that will need to be flushed to disk,
as well as WAL logged.
</para></listitem>
<listitem><para>
Replication: When the standby receives the data checksum state change
in the WAL stream it will issue a <glossterm linkend="glossary-restartpoint">
restartpoint</glossterm> in order to flush the current state into the
<filename>pg_control</filename> file. The restartpoint will flush the
current state to disk and will block redo until finished. This in turn
will induce replication lag, which on synchronous standbys also blocks
the primary. Reducing <xref linkend="guc-max-wal-size"/> before the
process is started can help with reducing the time it takes for the
restartpoint to finish.
</para></listitem>
<listitem><para>
Shutdown/Restart: If the server is shut down or restarted when data
checksums are being enabled, the process will not resume and all pages
need to be recalculated and rewritten. Enabling data checksums should
be done when there is no need for regular maintenance or during a
service window.
</para></listitem>
</itemizedlist>
</para>
<para>
No I/O is incurred when disabling data checksums, but checkpoints are
still required.
</para>
</sect3>
</sect2>
</sect1>
<sect1 id="wal-intro">

58
src/backend/access/rmgrdesc/xlogdesc.c
View File

@@ -55,40 +55,6 @@ get_wal_level_string(int wal_level)
return wal_level_str;
}
const char *
get_checksum_state_string(uint32 state)
{
switch (state)
{
case PG_DATA_CHECKSUM_VERSION:
return "on";
case PG_DATA_CHECKSUM_INPROGRESS_OFF:
return "inprogress-off";
case PG_DATA_CHECKSUM_INPROGRESS_ON:
return "inprogress-on";
case PG_DATA_CHECKSUM_OFF:
return "off";
}
Assert(false);
return "?";
}
void
xlog2_desc(StringInfo buf, XLogReaderState *record)
{
char *rec = XLogRecGetData(record);
uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
if (info == XLOG2_CHECKSUMS)
{
xl_checksum_state xlrec;
memcpy(&xlrec, rec, sizeof(xl_checksum_state));
appendStringInfoString(buf, get_checksum_state_string(xlrec.new_checksum_state));
}
}
void
xlog_desc(StringInfo buf, XLogReaderState *record)
{
@@ -104,8 +70,7 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
"tli %u; prev tli %u; fpw %s; wal_level %s; logical decoding %s; xid %u:%u; oid %u; multi %u; offset %" PRIu64 "; "
"oldest xid %u in DB %u; oldest multi %u in DB %u; "
"oldest/newest commit timestamp xid: %u/%u; "
"oldest running xid %u; "
"checksums %s; %s",
"oldest running xid %u; %s",
LSN_FORMAT_ARGS(checkpoint->redo),
checkpoint->ThisTimeLineID,
checkpoint->PrevTimeLineID,
@@ -124,7 +89,6 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
checkpoint->oldestCommitTsXid,
checkpoint->newestCommitTsXid,
checkpoint->oldestActiveXid,
get_checksum_state_string(checkpoint->dataChecksumState),
(info == XLOG_CHECKPOINT_SHUTDOWN) ? "shutdown" : "online");
}
else if (info == XLOG_NEXTOID)
@@ -203,9 +167,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
xl_checkpoint_redo xlrec;
memcpy(&xlrec, rec, sizeof(xl_checkpoint_redo));
appendStringInfo(buf, "wal_level %s; checksums %s",
get_wal_level_string(xlrec.wal_level),
get_checksum_state_string(xlrec.data_checksum_version));
appendStringInfo(buf, "wal_level %s",
get_wal_level_string(xlrec.wal_level));
}
else if (info == XLOG_LOGICAL_DECODING_STATUS_CHANGE)
{
@@ -280,21 +243,6 @@ xlog_identify(uint8 info)
return id;
}
const char *
xlog2_identify(uint8 info)
{
const char *id = NULL;
switch (info & ~XLR_INFO_MASK)
{
case XLOG2_CHECKSUMS:
id = "CHECKSUMS";
break;
}
return id;
}
/*
* Returns a string giving information about all the blocks in an
* XLogRecord.

552
src/backend/access/transam/xlog.c
View File

@@ -75,7 +75,6 @@
#include "pgstat.h"
#include "port/atomics.h"
#include "postmaster/bgwriter.h"
#include "postmaster/datachecksum_state.h"
#include "postmaster/startup.h"
#include "postmaster/walsummarizer.h"
#include "postmaster/walwriter.h"
@@ -93,7 +92,6 @@
#include "storage/predicate.h"
#include "storage/proc.h"
#include "storage/procarray.h"
#include "storage/procsignal.h"
#include "storage/reinit.h"
#include "storage/spin.h"
#include "storage/subsystems.h"
@@ -556,18 +554,7 @@ typedef struct XLogCtlData
*/
XLogRecPtr lastFpwDisableRecPtr;
/* last data_checksum_version we've seen */
uint32 data_checksum_version;
slock_t info_lck; /* locks shared variables shown above */
/*
* lastChecksumChangeRecPtr points to the end of the last XLOG2_CHECKSUMS
* record inserted or replayed, i.e. the last change of
* data_checksum_version. InvalidXLogRecPtr if the state hasn't changed
* since the server started.
*/
pg_atomic_uint64 lastChecksumChangeRecPtr;
} XLogCtlData;
/*
@@ -697,21 +684,6 @@ static TimeLineID openLogTLI = 0;
static XLogRecPtr LocalMinRecoveryPoint;
static bool updateMinRecoveryPoint = true;
/*
* Local state for ControlFile data_checksum_version. After initialization
* this is only updated when absorbing a procsignal barrier during interrupt
* processing. The reason for keeping a copy in backend-private memory is to
* avoid locking for interrogating the data checksum state. Possible values
* are the data checksum versions defined in storage/checksum.h.
*/
static ChecksumStateType LocalDataChecksumState = 0;
/*
* Variable backing the GUC, keep it in sync with LocalDataChecksumState.
* See SetLocalDataChecksumState().
*/
int data_checksums = 0;
/* For WALInsertLockAcquire/Release functions */
static int MyLockNo = 0;
static bool holdingAllLocks = false;
@@ -779,8 +751,6 @@ static void WALInsertLockAcquireExclusive(void);
static void WALInsertLockRelease(void);
static void WALInsertLockUpdateInsertingAt(XLogRecPtr insertingAt);
static void XLogChecksums(uint32 new_type);
/*
* Insert an XLOG record represented by an already-constructed chain of data
* chunks. This is a low-level routine; to construct the WAL record header
@@ -4322,13 +4292,6 @@ InitControlFile(uint64 sysidentifier, uint32 data_checksum_version)
ControlFile->wal_log_hints = wal_log_hints;
ControlFile->track_commit_timestamp = track_commit_timestamp;
ControlFile->data_checksum_version = data_checksum_version;
ControlFile->data_checksum_version_init = data_checksum_version;
/*
* Set the data_checksum_version value into XLogCtl, which is where all
* processes get the current value from.
*/
XLogCtl->data_checksum_version = data_checksum_version;
}
static void
@@ -4663,6 +4626,10 @@ ReadControlFile(void)
(SizeOfXLogLongPHD - SizeOfXLogShortPHD);
CalculateCheckpointSegments();
/* Make the initdb settings visible as GUC variables, too */
SetConfigOption("data_checksums", DataChecksumsEnabled() ? "yes" : "no",
PGC_INTERNAL, PGC_S_DYNAMIC_DEFAULT);
}
/*
@@ -4696,347 +4663,13 @@ GetMockAuthenticationNonce(void)
}
/*
* DataChecksumsNeedWrite
* Returns whether data checksums must be written or not
*
* Returns true if data checksums are enabled, or are in the process of being
* enabled. During "inprogress-on" and "inprogress-off" states checksums must
* be written even though they are not verified (see datachecksum_state.c for
* a longer discussion).
*
* This function is intended for callsites which are about to write a data page
* to storage, and need to know whether to re-calculate the checksum for the
* page header. Calling this function must be performed as close to the write
* operation as possible to keep the critical section short.
* Are checksums enabled for data pages?
*/
bool
DataChecksumsNeedWrite(void)
DataChecksumsEnabled(void)
{
return (LocalDataChecksumState == PG_DATA_CHECKSUM_VERSION ||
LocalDataChecksumState == PG_DATA_CHECKSUM_INPROGRESS_ON ||
LocalDataChecksumState == PG_DATA_CHECKSUM_INPROGRESS_OFF);
}
bool
DataChecksumsOff(void)
{
bool ret;
SpinLockAcquire(&XLogCtl->info_lck);
ret = (XLogCtl->data_checksum_version == PG_DATA_CHECKSUM_OFF);
SpinLockRelease(&XLogCtl->info_lck);
return ret;
}
bool
DataChecksumsOn(void)
{
bool ret;
SpinLockAcquire(&XLogCtl->info_lck);
ret = (XLogCtl->data_checksum_version == PG_DATA_CHECKSUM_VERSION);
SpinLockRelease(&XLogCtl->info_lck);
return ret;
}
bool
DataChecksumsInProgressOn(void)
{
bool ret;
SpinLockAcquire(&XLogCtl->info_lck);
ret = (XLogCtl->data_checksum_version == PG_DATA_CHECKSUM_INPROGRESS_ON);
SpinLockRelease(&XLogCtl->info_lck);
return ret;
}
/*
* DataChecksumsNeedVerify
* Returns whether data checksums must be verified or not
*
* Data checksums are only verified if they are fully enabled in the cluster.
* During the "inprogress-on" and "inprogress-off" states they are only
* updated, not verified (see datachecksum_state.c for a longer discussion).
*
* This function is intended for callsites which have read data and are about
* to perform checksum validation based on the result of this. Calling this
* function must be performed as close to the validation call as possible to
* keep the critical section short. This is in order to protect against time of
* check/time of use situations around data checksum validation.
*/
bool
DataChecksumsNeedVerify(void)
{
return (LocalDataChecksumState == PG_DATA_CHECKSUM_VERSION);
}
/*
* GetLastChecksumChangeRecPtr
* Returns the location of the last data checksum state change
*
* Offline state changes by pg_checksums leave no trace here; callers must
* also inspect the current state.
*
* No barrier semantics are needed: pages reach disk under a new checksum
* state only after their writer absorbed the procsignal barrier for the
* change, which is emitted after the new location became visible.
*/
XLogRecPtr
GetLastChecksumChangeRecPtr(void)
{
return pg_atomic_read_u64(&XLogCtl->lastChecksumChangeRecPtr);
}
/*
* SetDataChecksumsOnInProgress
* Sets the data checksum state to "inprogress-on" to enable checksums
*
* To start the process of enabling data checksums in a running cluster the
* data_checksum_version state must be changed to "inprogress-on". See
* SetDataChecksumsOn below for a description on how this state change works.
* This function blocks until all backends in the cluster have acknowledged the
* state transition.
*/
void
SetDataChecksumsOnInProgress(void)
{
uint64 barrier;
/*
* The state transition is performed in a critical section with
* checkpoints held off to provide crash safety.
*/
START_CRIT_SECTION();
MyProc->delayChkptFlags |= DELAY_CHKPT_START;
XLogChecksums(PG_DATA_CHECKSUM_INPROGRESS_ON);
SpinLockAcquire(&XLogCtl->info_lck);
XLogCtl->data_checksum_version = PG_DATA_CHECKSUM_INPROGRESS_ON;
SpinLockRelease(&XLogCtl->info_lck);
LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
ControlFile->data_checksum_version = PG_DATA_CHECKSUM_INPROGRESS_ON;
UpdateControlFile();
LWLockRelease(ControlFileLock);
barrier = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_ON);
MyProc->delayChkptFlags &= ~DELAY_CHKPT_START;
END_CRIT_SECTION();
WaitForProcSignalBarrier(barrier);
}
/*
* SetDataChecksumsOn
* Set data checksums state to 'on' cluster-wide
*
* Enabling data checksums is performed using two barriers, the first one to
* set the state to "inprogress-on" (done by SetDataChecksumsOnInProgress())
* and the second one to set the state to "on" (done here). Below is a short
* description of the processing, a more detailed write-up can be found in
* datachecksum_state.c.
*
* To start the process of enabling data checksums in a running cluster the
* data_checksum_version state must be changed to "inprogress-on". This state
* requires data checksums to be written but not verified. This ensures that
* all data pages can be checksummed without the risk of false negatives in
* validation during the process. When all existing pages are guaranteed to
* have checksums, and all new pages will be initiated with checksums, the
* state can be changed to "on". Once the state is "on" checksums will be both
* written and verified.
*
* This function blocks until all backends in the cluster have acknowledged the
* state transition.
*/
void
SetDataChecksumsOn(void)
{
uint64 barrier;
SpinLockAcquire(&XLogCtl->info_lck);
/*
* The only allowed state transition to "on" is from "inprogress-on" since
* that state ensures that all pages will have data checksums written. Any
* other attempted state transition is likely due to a programmer error.
*/
if (XLogCtl->data_checksum_version != PG_DATA_CHECKSUM_INPROGRESS_ON)
{
SpinLockRelease(&XLogCtl->info_lck);
elog(WARNING,
"cannot set data checksums to \"on\", current state is not \"inprogress-on\", disabling");
SetDataChecksumsOff();
return;
}
SpinLockRelease(&XLogCtl->info_lck);
INJECTION_POINT("datachecksums-enable-checksums-delay", NULL);
START_CRIT_SECTION();
MyProc->delayChkptFlags |= DELAY_CHKPT_START;
XLogChecksums(PG_DATA_CHECKSUM_VERSION);
SpinLockAcquire(&XLogCtl->info_lck);
XLogCtl->data_checksum_version = PG_DATA_CHECKSUM_VERSION;
SpinLockRelease(&XLogCtl->info_lck);
/*
* Update the controlfile before waiting since if we have an immediate
* shutdown while waiting we want to come back up with checksums enabled.
*/
LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
ControlFile->data_checksum_version = PG_DATA_CHECKSUM_VERSION;
UpdateControlFile();
LWLockRelease(ControlFileLock);
barrier = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_ON);
MyProc->delayChkptFlags &= ~DELAY_CHKPT_START;
END_CRIT_SECTION();
INJECTION_POINT("datachecksums-on-before-checkpoint", NULL);
RequestCheckpoint(CHECKPOINT_FORCE | CHECKPOINT_WAIT | CHECKPOINT_FAST);
WaitForProcSignalBarrier(barrier);
}
/*
* SetDataChecksumsOff
* Disables data checksums cluster-wide
*
* Disabling data checksums must be performed with two sets of barriers, each
* carrying a different state. The state is first set to "inprogress-off"
* during which checksums are still written but not verified. This ensures that
* backends which have yet to observe the state change from "on" won't get
* validation errors on concurrently modified pages. Once all backends have
* changed to "inprogress-off", the barrier for moving to "off" can be emitted.
* This function blocks until all backends in the cluster have acknowledged the
* state transition.
*/
void
SetDataChecksumsOff(void)
{
uint64 barrier;
SpinLockAcquire(&XLogCtl->info_lck);
/* If data checksums are already disabled there is nothing to do */
if (XLogCtl->data_checksum_version == PG_DATA_CHECKSUM_OFF)
{
SpinLockRelease(&XLogCtl->info_lck);
return;
}
/*
* If data checksums are currently enabled, or in the process of being
* enabled, we first transition to the "inprogress-off" state during which
* backends continue to write checksums without verifying them. When all
* backends are in "inprogress-off" the next transition to "off" can be
* performed, after which all data checksum processing is disabled.
*/
if (XLogCtl->data_checksum_version == PG_DATA_CHECKSUM_VERSION ||
XLogCtl->data_checksum_version == PG_DATA_CHECKSUM_INPROGRESS_ON)
{
SpinLockRelease(&XLogCtl->info_lck);
START_CRIT_SECTION();
MyProc->delayChkptFlags |= DELAY_CHKPT_START;
XLogChecksums(PG_DATA_CHECKSUM_INPROGRESS_OFF);
SpinLockAcquire(&XLogCtl->info_lck);
XLogCtl->data_checksum_version = PG_DATA_CHECKSUM_INPROGRESS_OFF;
SpinLockRelease(&XLogCtl->info_lck);
LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
ControlFile->data_checksum_version = PG_DATA_CHECKSUM_INPROGRESS_OFF;
UpdateControlFile();
LWLockRelease(ControlFileLock);
barrier = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_OFF);
MyProc->delayChkptFlags &= ~DELAY_CHKPT_START;
END_CRIT_SECTION();
RequestCheckpoint(CHECKPOINT_FORCE | CHECKPOINT_WAIT | CHECKPOINT_FAST);
WaitForProcSignalBarrier(barrier);
/*
* At this point we know that no backends are verifying data checksums
* during reading. Next, we can safely move to state "off" to also
* stop writing checksums.
*/
}
else
{
/*
* Ending up here implies that the checksums state is "inprogress-off"
* and we can transition directly to "off" from there.
*/
SpinLockRelease(&XLogCtl->info_lck);
}
START_CRIT_SECTION();
/* Ensure that we don't incur a checkpoint during disabling checksums */
MyProc->delayChkptFlags |= DELAY_CHKPT_START;
XLogChecksums(PG_DATA_CHECKSUM_OFF);
SpinLockAcquire(&XLogCtl->info_lck);
XLogCtl->data_checksum_version = PG_DATA_CHECKSUM_OFF;
SpinLockRelease(&XLogCtl->info_lck);
LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
ControlFile->data_checksum_version = PG_DATA_CHECKSUM_OFF;
UpdateControlFile();
LWLockRelease(ControlFileLock);
barrier = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_OFF);
MyProc->delayChkptFlags &= ~DELAY_CHKPT_START;
END_CRIT_SECTION();
RequestCheckpoint(CHECKPOINT_FORCE | CHECKPOINT_WAIT | CHECKPOINT_FAST);
WaitForProcSignalBarrier(barrier);
}
/*
* InitLocalDataChecksumState
*
* Set up backend local caches of controldata variables which may change at
* any point during runtime and thus require special cased locking. So far
* this only applies to data_checksum_version, but it's intended to be general
* purpose enough to handle future cases.
*/
void
InitLocalDataChecksumState(void)
{
Assert(InterruptHoldoffCount > 0);
SpinLockAcquire(&XLogCtl->info_lck);
SetLocalDataChecksumState(XLogCtl->data_checksum_version);
SpinLockRelease(&XLogCtl->info_lck);
}
void
SetLocalDataChecksumState(uint32 data_checksum_version)
{
LocalDataChecksumState = data_checksum_version;
data_checksums = data_checksum_version;
}
/* guc hook */
const char *
show_data_checksums(void)
{
return get_checksum_state_string(LocalDataChecksumState);
Assert(ControlFile != NULL);
return (ControlFile->data_checksum_version > 0);
}
/*
@@ -5331,7 +4964,6 @@ LocalProcessControlFile(bool reset)
LocalControlFile = palloc_object(ControlFileData);
ControlFile = LocalControlFile;
ReadControlFile();
SetLocalDataChecksumState(ControlFile->data_checksum_version);
}
/*
@@ -5483,17 +5115,12 @@ XLOGShmemInit(void *arg)
XLogCtl->InstallXLogFileSegmentActive = false;
XLogCtl->WalWriterSleeping = false;
/* Use the checksum info from control file */
XLogCtl->data_checksum_version = ControlFile->data_checksum_version;
SetLocalDataChecksumState(XLogCtl->data_checksum_version);
SpinLockInit(&XLogCtl->Insert.insertpos_lck);
SpinLockInit(&XLogCtl->info_lck);
pg_atomic_init_u64(&XLogCtl->logInsertResult, InvalidXLogRecPtr);
pg_atomic_init_u64(&XLogCtl->logWriteResult, InvalidXLogRecPtr);
pg_atomic_init_u64(&XLogCtl->logFlushResult, InvalidXLogRecPtr);
pg_atomic_init_u64(&XLogCtl->unloggedLSN, InvalidXLogRecPtr);
pg_atomic_init_u64(&XLogCtl->lastChecksumChangeRecPtr, InvalidXLogRecPtr);
}
/*
@@ -5571,7 +5198,6 @@ BootStrapXLOG(uint32 data_checksum_version)
checkPoint.newestCommitTsXid = InvalidTransactionId;
checkPoint.time = (pg_time_t) time(NULL);
checkPoint.oldestActiveXid = InvalidTransactionId;
checkPoint.dataChecksumState = data_checksum_version;
TransamVariables->nextXid = checkPoint.nextXid;
TransamVariables->nextOid = checkPoint.nextOid;
@@ -6649,50 +6275,6 @@ StartupXLOG(void)
pfree(endOfRecoveryInfo->recoveryStopReason);
pfree(endOfRecoveryInfo);
/*
* If we reach this point with checksums in the state inprogress-on, it
* means that data checksums were in the process of being enabled when the
* cluster shut down. Since processing didn't finish, the operation will
* have to be restarted from scratch since there is no capability to
* continue where it was when the cluster shut down. Thus, revert the
* state back to off, and inform the user with a warning message. Being
* able to restart processing is a TODO, but it wouldn't be possible to
* restart here since we cannot launch a dynamic background worker
* directly from here (it has to be from a regular backend).
*/
if (XLogCtl->data_checksum_version == PG_DATA_CHECKSUM_INPROGRESS_ON)
{
XLogChecksums(PG_DATA_CHECKSUM_OFF);
SpinLockAcquire(&XLogCtl->info_lck);
XLogCtl->data_checksum_version = PG_DATA_CHECKSUM_OFF;
SetLocalDataChecksumState(XLogCtl->data_checksum_version);
SpinLockRelease(&XLogCtl->info_lck);
EmitAndWaitDataChecksumsBarrier(PG_DATA_CHECKSUM_OFF);
ereport(WARNING,
errmsg("enabling data checksums was interrupted"),
errhint("Data checksum processing must be manually restarted for checksums to be enabled."));
}
/*
* If data checksums were being disabled when the cluster was shut down,
* we know that we have a state where all backends have stopped validating
* checksums and we can move to off instead of prompting the user to
* perform any action.
*/
else if (XLogCtl->data_checksum_version == PG_DATA_CHECKSUM_INPROGRESS_OFF)
{
XLogChecksums(PG_DATA_CHECKSUM_OFF);
SpinLockAcquire(&XLogCtl->info_lck);
XLogCtl->data_checksum_version = PG_DATA_CHECKSUM_OFF;
SetLocalDataChecksumState(XLogCtl->data_checksum_version);
SpinLockRelease(&XLogCtl->info_lck);
EmitAndWaitDataChecksumsBarrier(PG_DATA_CHECKSUM_OFF);
}
/*
* All done with end-of-recovery actions.
*
@@ -6712,7 +6294,6 @@ StartupXLOG(void)
ControlFile->state = DB_IN_PRODUCTION;
SpinLockAcquire(&XLogCtl->info_lck);
ControlFile->data_checksum_version = XLogCtl->data_checksum_version;
XLogCtl->SharedRecoveryState = RECOVERY_STATE_DONE;
SpinLockRelease(&XLogCtl->info_lck);
@@ -6998,7 +6579,7 @@ GetRedoRecPtr(void)
XLogRecPtr ptr;
/*
* The possibly not up-to-date copy in XLogCtl is enough. Even if we
* The possibly not up-to-date copy in XlogCtl is enough. Even if we
* grabbed a WAL insertion lock to read the authoritative value in
* Insert->RedoRecPtr, someone might update it just after we've released
* the lock.
@@ -7576,14 +7157,6 @@ CreateCheckPoint(int flags)
checkPoint.fullPageWrites = Insert->fullPageWrites;
checkPoint.wal_level = wal_level;
/*
* Get the current data_checksum_version value from xlogctl, valid at the
* time of the checkpoint.
*/
SpinLockAcquire(&XLogCtl->info_lck);
checkPoint.dataChecksumState = XLogCtl->data_checksum_version;
SpinLockRelease(&XLogCtl->info_lck);
if (shutdown)
{
XLogRecPtr curInsert = XLogBytePosToRecPtr(Insert->CurrBytePos);
@@ -7640,9 +7213,6 @@ CreateCheckPoint(int flags)
WALInsertLockAcquire();
redo_rec.wal_level = wal_level;
SpinLockAcquire(&XLogCtl->info_lck);
redo_rec.data_checksum_version = XLogCtl->data_checksum_version;
SpinLockRelease(&XLogCtl->info_lck);
WALInsertLockRelease();
/* Include WAL level in record for WAL summarizer's benefit. */
@@ -7995,12 +7565,6 @@ CreateEndOfRecoveryRecord(void)
LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
ControlFile->minRecoveryPoint = recptr;
ControlFile->minRecoveryPointTLI = xlrec.ThisTimeLineID;
/* start with the latest checksum version (as of the end of recovery) */
SpinLockAcquire(&XLogCtl->info_lck);
ControlFile->data_checksum_version = XLogCtl->data_checksum_version;
SpinLockRelease(&XLogCtl->info_lck);
UpdateControlFile();
LWLockRelease(ControlFileLock);
@@ -8341,10 +7905,6 @@ CreateRestartPoint(int flags)
if (flags & CHECKPOINT_IS_SHUTDOWN)
ControlFile->state = DB_SHUTDOWNED_IN_RECOVERY;
}
/* we shall start with the latest checksum version */
ControlFile->data_checksum_version = lastCheckPoint.dataChecksumState;
UpdateControlFile();
}
LWLockRelease(ControlFileLock);
@@ -8783,25 +8343,6 @@ XLogReportParameters(void)
}
}
/*
* Log the new state of checksums
*/
static void
XLogChecksums(uint32 new_type)
{
xl_checksum_state xlrec;
XLogRecPtr recptr;
xlrec.new_checksum_state = new_type;
XLogBeginInsert();
XLogRegisterData((char *) &xlrec, sizeof(xl_checksum_state));
recptr = XLogInsert(RM_XLOG2_ID, XLOG2_CHECKSUMS);
pg_atomic_write_u64(&XLogCtl->lastChecksumChangeRecPtr, recptr);
XLogFlush(recptr);
}
/*
* Update full_page_writes in shared memory, and write an
* XLOG_FPW_CHANGE record if necessary.
@@ -8987,9 +8528,6 @@ xlog_redo(XLogReaderState *record)
/* ControlFile->checkPointCopy always tracks the latest ckpt XID */
LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
ControlFile->checkPointCopy.nextXid = checkPoint.nextXid;
ControlFile->data_checksum_version = checkPoint.dataChecksumState;
UpdateControlFile();
LWLockRelease(ControlFileLock);
/*
@@ -9225,20 +8763,7 @@ xlog_redo(XLogReaderState *record)
}
else if (info == XLOG_CHECKPOINT_REDO)
{
xl_checkpoint_redo redo_rec;
bool new_state = false;
memcpy(&redo_rec, XLogRecGetData(record), sizeof(xl_checkpoint_redo));
SpinLockAcquire(&XLogCtl->info_lck);
XLogCtl->data_checksum_version = redo_rec.data_checksum_version;
SetLocalDataChecksumState(redo_rec.data_checksum_version);
if (redo_rec.data_checksum_version != ControlFile->data_checksum_version)
new_state = true;
SpinLockRelease(&XLogCtl->info_lck);
if (new_state)
EmitAndWaitDataChecksumsBarrier(redo_rec.data_checksum_version);
/* nothing to do here, just for informational purposes */
}
else if (info == XLOG_LOGICAL_DECODING_STATUS_CHANGE)
{
@@ -9291,63 +8816,6 @@ xlog_redo(XLogReaderState *record)
}
}
void
xlog2_redo(XLogReaderState *record)
{
uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
if (info == XLOG2_CHECKSUMS)
{
xl_checksum_state state;
XLogRecPtr lsn = record->EndRecPtr;
memcpy(&state, XLogRecGetData(record), sizeof(xl_checksum_state));
/* advertise the location before the new state becomes visible */
pg_atomic_write_u64(&XLogCtl->lastChecksumChangeRecPtr, lsn);
SpinLockAcquire(&XLogCtl->info_lck);
XLogCtl->data_checksum_version = state.new_checksum_state;
SpinLockRelease(&XLogCtl->info_lck);
LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
ControlFile->data_checksum_version = state.new_checksum_state;
/*
* Update minRecoveryPoint to ensure that if recovery is aborted, we
* recover back up to this point before allowing hot standby again.
* The new state is durable in pg_control while its location is only
* tracked in shared memory; a standby becoming consistent below this
* record would let base backups resume checksum verification with the
* location unknown. The local copies cannot be updated as long as
* crash recovery is happening and we expect all the WAL to be
* replayed.
*/
if (InArchiveRecovery)
{
LocalMinRecoveryPoint = ControlFile->minRecoveryPoint;
}
if (XLogRecPtrIsValid(LocalMinRecoveryPoint) && LocalMinRecoveryPoint < lsn)
{
TimeLineID replayTLI;
(void) GetCurrentReplayRecPtr(&replayTLI);
ControlFile->minRecoveryPoint = lsn;
ControlFile->minRecoveryPointTLI = replayTLI;
}
UpdateControlFile();
LWLockRelease(ControlFileLock);
/*
* Block on a procsignalbarrier to await all processes having seen the
* change to checksum status. Once the barrier has been passed we can
* initiate the corresponding processing.
*/
EmitAndWaitDataChecksumsBarrier(state.new_checksum_state);
}
}
/*
* Return the extra open flags used for opening a file, depending on the
* value of the GUCs wal_sync_method, fsync and debug_io_direct.

90
src/backend/backup/basebackup.c
View File

@@ -45,7 +45,6 @@
#include "storage/reinit.h"
#include "utils/builtins.h"
#include "utils/guc.h"
#include "utils/injection_point.h"
#include "utils/ps_status.h"
#include "utils/relcache.h"
#include "utils/resowner.h"
@@ -108,7 +107,6 @@ static off_t read_file_data_into_buffer(bbsink *sink,
int *checksum_failures);
static void push_to_sink(bbsink *sink, pg_checksum_context *checksum_ctx,
size_t *bytes_done, void *data, size_t length);
static bool backup_checksums_verifiable(XLogRecPtr start_lsn);
static bool verify_page_checksum(Page page, XLogRecPtr start_lsn,
BlockNumber blkno,
uint16 *expected_checksum);
@@ -327,12 +325,6 @@ perform_base_backup(basebackup_options *opt, bbsink *sink,
/* notify basebackup sink about start of backup */
bbsink_begin_backup(sink, &state, SINK_BUFFER_LENGTH);
/*
* Allow tests to hold the backup after the starting checkpoint but
* before any file data is sent.
*/
INJECTION_POINT("basebackup-before-send-files", NULL);
/* Send off our tablespaces one by one */
foreach(lc, state.tablespaces)
{
@@ -1616,14 +1608,12 @@ sendFile(bbsink *sink, const char *readfilename, const char *tarfilename,
Assert((sink->bbs_buffer_length % BLCKSZ) == 0);
/*
* Verify checksums unless the client requested otherwise, but only for
* relation files, and only while checksums have been continuously enabled
* since the checkpoint this backup started from. Checksums can still be
* disabled while the backup runs, thus we check at each point where we
* could be validating a checksum.
* If we weren't told not to verify checksums, and if checksums are
* enabled for this cluster, and if this is a relation file, then verify
* the checksum.
*/
if (!noverify_checksums && RelFileNumberIsValid(relfilenumber) &&
backup_checksums_verifiable(sink->bbs_state->startptr))
if (!noverify_checksums && DataChecksumsEnabled() &&
RelFileNumberIsValid(relfilenumber))
verify_checksum = true;
/*
@@ -1756,9 +1746,7 @@ sendFile(bbsink *sink, const char *readfilename, const char *tarfilename,
* If the amount of data we were able to read was not a multiple of
* BLCKSZ, we cannot verify checksums, which are block-level.
*/
if (verify_checksum &&
backup_checksums_verifiable(sink->bbs_state->startptr) &&
(cnt % BLCKSZ != 0))
if (verify_checksum && (cnt % BLCKSZ != 0))
{
ereport(WARNING,
(errmsg("could not verify checksum in file \"%s\", block "
@@ -1853,10 +1841,9 @@ sendFile(bbsink *sink, const char *readfilename, const char *tarfilename,
* 'blkno' is the block number of the first page in the bbsink's buffer
* relative to the start of the relation.
*
* 'verify_checksum' determines if the user has asked to verify checksums, but
* since data checksums can be disabled, or become disabled, we need to check
* state before verifying individual pages. If we do this, we'll update
* *checksum_failures and issue warnings as appropriate.
* 'verify_checksum' indicates whether we should try to verify checksums
* for the blocks we read. If we do this, we'll update *checksum_failures
* and issue warnings as appropriate.
*/
static off_t
read_file_data_into_buffer(bbsink *sink, const char *readfilename, int fd,
@@ -1882,13 +1869,6 @@ read_file_data_into_buffer(bbsink *sink, const char *readfilename, int fd,
int reread_cnt;
uint16 expected_checksum;
/*
* The data checksum state can change at any point, so we need to
* re-check before each page.
*/
if (!backup_checksums_verifiable(sink->bbs_state->startptr))
return cnt;
page = sink->bbs_buffer + BLCKSZ * i;
/* If the page is OK, go on to the next one. */
@@ -1911,12 +1891,7 @@ read_file_data_into_buffer(bbsink *sink, const char *readfilename, int fd,
* allows us to wait until we can be certain that no write to the
* block is in progress. Since we don't have any such thing right now,
* we just do this and hope for the best.
*
* The data checksum state may also have changed concurrently so check
* again.
*/
if (!backup_checksums_verifiable(sink->bbs_state->startptr))
return cnt;
reread_cnt =
basebackup_read_file(fd, sink->bbs_buffer + BLCKSZ * i,
BLCKSZ, offset + BLCKSZ * i,
@@ -2006,27 +1981,6 @@ push_to_sink(bbsink *sink, pg_checksum_context *checksum_ctx,
}
}
/*
* Check whether data checksums can be verified for a backup started at
* start_lsn.
*
* Checksums are verified only while they have been continuously enabled
* since the checkpoint the backup started from: the state must be "on" and
* the last state change must predate the backup start. Such a checkpoint
* guarantees that every page flushed before it has a checksum written. Any
* later state change ends verification for the rest of the backup: pages
* written while checksums were off can lack checksums yet keep LSNs older
* than the backup start, and re-enabling completes before the rewritten
* pages are flushed, so observing the "on" state again is not enough to
* resume.
*/
static bool
backup_checksums_verifiable(XLogRecPtr start_lsn)
{
return DataChecksumsNeedVerify() &&
GetLastChecksumChangeRecPtr() <= start_lsn;
}
/*
* Try to verify the checksum for the provided page, if it seems appropriate
* to do so.
@@ -2052,35 +2006,9 @@ verify_page_checksum(Page page, XLogRecPtr start_lsn, BlockNumber blkno,
if (PageIsNew(page) || PageGetLSN(page) >= start_lsn)
return true;
if (!backup_checksums_verifiable(start_lsn))
return true;
/* Perform the actual checksum calculation. */
checksum = pg_checksum_page(page, blkno);
#ifdef USE_INJECTION_POINTS
{
/*
* Make it possible to test checksum verification failure without
* having to destroy data on disk. There is cap on how many times we
* want to cause verification failure to make tests more interesting
* and less log intensive. This makes it easy to test pg_basebackup
* with the command_checks_all test function.
*/
static int hit = 0;
if (IS_INJECTION_POINT_ATTACHED("basebackup-fail-checksum-verification"))
{
if (hit++ < 5)
{
checksum = 0;
INJECTION_POINT_CACHED("basebackup-fail-checksum-verification",
NULL);
}
}
}
#endif
/* See whether it matches the value from the page. */
phdr = (PageHeader) page;
if (phdr->pd_checksum == checksum)

19
src/backend/catalog/system_views.sql
View File

@@ -1473,25 +1473,6 @@ CREATE VIEW pg_stat_progress_copy AS
FROM pg_stat_get_progress_info('COPY') AS S
LEFT JOIN pg_database D ON S.datid = D.oid;
CREATE VIEW pg_stat_progress_data_checksums AS
SELECT
S.pid AS pid, S.datid, D.datname AS datname,
CASE S.param1 WHEN 0 THEN 'enabling'
WHEN 1 THEN 'disabling'
WHEN 2 THEN 'waiting on temporary tables'
WHEN 3 THEN 'waiting on barrier'
WHEN 4 THEN 'done'
END AS phase,
CASE S.param2 WHEN -1 THEN NULL ELSE S.param2 END AS databases_total,
CASE S.param3 WHEN -1 THEN NULL ELSE S.param3 END AS databases_done,
CASE S.param4 WHEN -1 THEN NULL ELSE S.param4 END AS relations_total,
CASE S.param5 WHEN -1 THEN NULL ELSE S.param5 END AS relations_done,
CASE S.param6 WHEN -1 THEN NULL ELSE S.param6 END AS blocks_total,
CASE S.param7 WHEN -1 THEN NULL ELSE S.param7 END AS blocks_done
FROM pg_stat_get_progress_info('DATACHECKSUMS') AS S
LEFT JOIN pg_database D ON S.datid = D.oid
ORDER BY S.datid; -- return the launcher process first
CREATE VIEW pg_user_mappings AS
SELECT
U.oid AS umid,

34
src/backend/commands/dbcommands.c
View File

@@ -559,23 +559,6 @@ CreateDatabaseUsingFileCopy(Oid src_dboid, Oid dst_dboid, Oid src_tsid,
Relation rel;
HeapTuple tuple;
/*
* The strategy check in createdb() runs before our transaction has an XID
* and before the pg_database row exists, so the datachecksumsworker
* launcher can start in that window and miss both the new database and
* our transaction, leaving the raw-copied files without checksums.
*/
if (DataChecksumsInProgressOn())
ereport(ERROR,
errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("create database strategy \"%s\" not allowed when data checksums are being enabled",
"file_copy"));
/*
* The XID is assigned by now, so a datachecksumsworker launcher starting
* after this point will wait for us and find the new database.
*/
/*
* Force a checkpoint before starting the copy. This will force all dirty
* buffers, including those of unlogged tables, out to disk, to ensure
@@ -1062,22 +1045,7 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt)
if (pg_strcasecmp(strategy, "wal_log") == 0)
dbstrategy = CREATEDB_WAL_LOG;
else if (pg_strcasecmp(strategy, "file_copy") == 0)
{
/*
* If data checksums are being enabled we must not use file_copy
* since it might copy source database which hasn't yet had data
* checksums enabled, and the destination database will be skipped
* as it's expected to have data checksums enabled. Once we have
* an XID assigned this needs to be rechecked, but if can error
* out already we can save a lot of work.
*/
if (DataChecksumsInProgressOn())
ereport(ERROR,
errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("create database strategy \"%s\" not allowed when data checksums are being enabled",
strategy));
dbstrategy = CREATEDB_FILE_COPY;
}
else
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
@@ -1883,8 +1851,6 @@ dropdb(const char *dbname, bool missing_ok, bool force)
systable_inplace_update_finish(inplace_state, tup);
XLogFlush(XactLastRecEnd);
INJECTION_POINT("dropdb-after-invalid-marker", NULL);
/*
* Also delete the tuple - transactionally. If this transaction commits,
* the row will be gone, but if we fail, dropdb() can be invoked again.

1
src/backend/postmaster/Makefile
View File

@@ -18,7 +18,6 @@ OBJS = \
bgworker.o \
bgwriter.o \
checkpointer.o \
datachecksum_state.o \
fork_process.o \
interrupt.o \
launch_backend.o \

28
src/backend/postmaster/auxprocess.c
View File

@@ -68,36 +68,8 @@ AuxiliaryProcessMainCommon(void)
BaseInit();
/*
* Prevent consuming interrupts between setting ProcSignalInit and setting
* the initial local data checksum value. If a barrier is emitted, and
* absorbed, before local cached state is initialized the state transition
* can be invalid.
*/
HOLD_INTERRUPTS();
ProcSignalInit(NULL, 0);
/*
* Initialize a local cache of the data_checksum_version, to be updated by
* the procsignal-based barriers.
*
* This intentionally happens after initializing the procsignal, otherwise
* we might miss a state change. This means we can get a barrier for the
* state we've just initialized - but it can happen only once.
*
* The postmaster (which is what gets forked into the new child process)
* does not handle barriers, therefore it may not have the current value
* of LocalDataChecksumState value (it'll have the value read from the
* control file, which may be arbitrarily old).
*
* NB: Even if the postmaster handled barriers, the value might still be
* stale, as it might have changed after this process forked.
*/
InitLocalDataChecksumState();
RESUME_INTERRUPTS();
/*
* Initialize the process-local logical info WAL logging state.
*

9
src/backend/postmaster/bgworker.c
View File

@@ -19,7 +19,6 @@
#include "pgstat.h"
#include "port/atomics.h"
#include "postmaster/bgworker_internals.h"
#include "postmaster/datachecksum_state.h"
#include "postmaster/postmaster.h"
#include "replication/logicallauncher.h"
#include "replication/logicalworker.h"
@@ -158,14 +157,6 @@ static const struct
{
.fn_name = "TableSyncWorkerMain",
.fn_addr = TableSyncWorkerMain
},
{
.fn_name = "DataChecksumsWorkerLauncherMain",
.fn_addr = DataChecksumsWorkerLauncherMain
},
{
.fn_name = "DataChecksumsWorkerMain",
.fn_addr = DataChecksumsWorkerMain
}
};

1979
src/backend/postmaster/datachecksum_state.c
View File

File diff suppressed because it is too large Load Diff

1
src/backend/postmaster/meson.build
View File

@@ -6,7 +6,6 @@ backend_sources += files(
'bgworker.c',
'bgwriter.c',
'checkpointer.c',
'datachecksum_state.c',
'fork_process.c',
'interrupt.c',
'launch_backend.c',

5
src/backend/postmaster/postmaster.c
View File

@@ -3021,11 +3021,6 @@ PostmasterStateMachine(void)
B_INVALID,
B_STANDALONE_BACKEND);
/* also add data checksums processes */
remainMask = btmask_add(remainMask,
B_DATACHECKSUMSWORKER_LAUNCHER,
B_DATACHECKSUMSWORKER_WORKER);
/* All types should be included in targetMask or remainMask */
Assert((remainMask.mask | targetMask.mask) == BTYPE_MASK_ALL.mask);
}

16
src/backend/replication/logical/decode.c
View File

@@ -190,22 +190,6 @@ xlog_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
}
}
void
xlog2_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
{
uint8 info = XLogRecGetInfo(buf->record) & ~XLR_INFO_MASK;
ReorderBufferProcessXid(ctx->reorder, XLogRecGetXid(buf->record), buf->origptr);
switch (info)
{
case XLOG2_CHECKSUMS:
break;
default:
elog(ERROR, "unexpected RM_XLOG2_ID record type: %u", info);
}
}
/*
* Handle rmgr XACT_ID records for LogicalDecodingProcessRecord().
*/

8
src/backend/storage/ipc/procsignal.c
View File

@@ -23,7 +23,6 @@
#include "miscadmin.h"
#include "pgstat.h"
#include "port/pg_bitutils.h"
#include "postmaster/datachecksum_state.h"
#include "replication/logicalctl.h"
#include "replication/logicalworker.h"
#include "replication/slotsync.h"
@@ -591,13 +590,6 @@ ProcessProcSignalBarrier(void)
case PROCSIGNAL_BARRIER_UPDATE_XLOG_LOGICAL_INFO:
processed = ProcessBarrierUpdateXLogLogicalInfo();
break;
case PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_ON:
case PROCSIGNAL_BARRIER_CHECKSUM_ON:
case PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_OFF:
case PROCSIGNAL_BARRIER_CHECKSUM_OFF:
processed = AbsorbDataChecksumsBarrier(type);
break;
}
/*

4
src/backend/storage/page/README
View File

@@ -10,9 +10,7 @@ http://www.cs.toronto.edu/~bianca/papers/sigmetrics09.pdf, discussed
2010/12/22 on -hackers list.
Current implementation requires this be enabled system-wide at initdb time, or
by using the pg_checksums tool on an offline cluster. Checksums can also be
enabled at runtime using pg_enable_data_checksums(), and disabled by using
pg_disable_data_checksums().
by using the pg_checksums tool on an offline cluster.
The checksum is not valid at all times on a data page!!
The checksum is valid when the page leaves the shared pool and is checked

18
src/backend/storage/page/bufpage.c
View File

@@ -107,15 +107,7 @@ PageIsVerified(PageData *page, BlockNumber blkno, int flags, bool *checksum_fail
*/
if (!PageIsNew(page))
{
/*
* There shouldn't be any check for interrupt calls happening in this
* codepath, but just to be on the safe side we hold interrupts since
* if they did happen the data checksum state could change during
* verifying checksums, which could lead to incorrect verification
* results.
*/
HOLD_INTERRUPTS();
if (DataChecksumsNeedVerify())
if (DataChecksumsEnabled())
{
checksum = pg_checksum_page(page, blkno);
@@ -126,7 +118,6 @@ PageIsVerified(PageData *page, BlockNumber blkno, int flags, bool *checksum_fail
*checksum_failure_p = true;
}
}
RESUME_INTERRUPTS();
/*
* The following checks don't prove the header is correct, only that
@@ -1519,14 +1510,9 @@ PageIndexTupleOverwrite(Page page, OffsetNumber offnum,
void
PageSetChecksum(Page page, BlockNumber blkno)
{
HOLD_INTERRUPTS();
/* If we don't need a checksum, just return */
if (PageIsNew(page) || !DataChecksumsNeedWrite())
{
RESUME_INTERRUPTS();
if (PageIsNew(page) || !DataChecksumsEnabled())
return;
}
((PageHeader) page)->pd_checksum = pg_checksum_page(page, blkno);
RESUME_INTERRUPTS();
}

2
src/backend/utils/activity/pgstat_backend.c
View File

@@ -381,8 +381,6 @@ pgstat_tracks_backend_bktype(BackendType bktype)
case B_CHECKPOINTER:
case B_IO_WORKER:
case B_STARTUP:
case B_DATACHECKSUMSWORKER_LAUNCHER:
case B_DATACHECKSUMSWORKER_WORKER:
return false;
case B_AUTOVAC_WORKER:

35
src/backend/utils/activity/pgstat_io.c
View File

@@ -354,8 +354,6 @@ pgstat_tracks_io_bktype(BackendType bktype)
case B_LOGGER:
return false;
case B_DATACHECKSUMSWORKER_LAUNCHER:
case B_DATACHECKSUMSWORKER_WORKER:
case B_AUTOVAC_LAUNCHER:
case B_AUTOVAC_WORKER:
case B_BACKEND:
@@ -456,37 +454,6 @@ pgstat_tracks_io_object(BackendType bktype, IOObject io_object,
io_context == IOCONTEXT_BULKWRITE)
return false;
/*
* The data checksums launcher scans catalogs and emits WAL records for
* checksum state changes. Catalog scans can use a bulkread strategy.
*/
if (bktype == B_DATACHECKSUMSWORKER_LAUNCHER)
{
if (io_object == IOOBJECT_WAL ||
(io_object == IOOBJECT_RELATION &&
(io_context == IOCONTEXT_BULKREAD ||
io_context == IOCONTEXT_NORMAL)))
return true;
return false;
}
/*
* The worker also scans catalogs, then processes relations using a vacuum
* access strategy. Catalog scans can use a bulkread strategy.
*/
if (bktype == B_DATACHECKSUMSWORKER_WORKER)
{
if (io_object == IOOBJECT_WAL ||
(io_object == IOOBJECT_RELATION &&
(io_context == IOCONTEXT_BULKREAD ||
io_context == IOCONTEXT_NORMAL ||
io_context == IOCONTEXT_VACUUM)))
return true;
return false;
}
return true;
}
@@ -530,8 +497,6 @@ pgstat_tracks_io_op(BackendType bktype, IOObject io_object,
if (io_object == IOOBJECT_WAL && io_op == IOOP_READ &&
(bktype == B_WAL_RECEIVER || bktype == B_BG_WRITER ||
bktype == B_AUTOVAC_LAUNCHER || bktype == B_AUTOVAC_WORKER ||
bktype == B_DATACHECKSUMSWORKER_LAUNCHER ||
bktype == B_DATACHECKSUMSWORKER_WORKER ||
bktype == B_WAL_WRITER))
return false;

3
src/backend/utils/activity/wait_event_names.txt
View File

@@ -116,8 +116,6 @@ CHECKPOINT_DELAY_COMPLETE "Waiting for a backend that blocks a checkpoint from c
CHECKPOINT_DELAY_START "Waiting for a backend that blocks a checkpoint from starting."
CHECKPOINT_DONE "Waiting for a checkpoint to complete."
CHECKPOINT_START "Waiting for a checkpoint to start."
CHECKSUM_ENABLE_STARTCONDITION "Waiting for data checksums enabling to start."
CHECKSUM_ENABLE_TEMPTABLE_WAIT "Waiting for temporary tables to be dropped for data checksums to be enabled."
EXECUTE_GATHER "Waiting for activity from a child process while executing a <literal>Gather</literal> plan node."
HASH_BATCH_ALLOCATE "Waiting for an elected Parallel Hash participant to allocate a hash table."
HASH_BATCH_ELECT "Waiting to elect a Parallel Hash participant to allocate a hash table."
@@ -369,7 +367,6 @@ SerialControl "Waiting to read or update shared <filename>pg_serial</filename> s
AioWorkerSubmissionQueue "Waiting to access AIO worker submission queue."
WaitLSN "Waiting to read or update shared Wait-for-LSN state."
LogicalDecodingControl "Waiting to read or update logical decoding status information."
DataChecksumsWorker "Waiting for data checksums worker."
AioWorkerControl "Waiting to update AIO worker information."
#

8
src/backend/utils/adt/pgstatfuncs.c
View File

@@ -297,8 +297,6 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS)
cmdtype = PROGRESS_COMMAND_BASEBACKUP;
else if (pg_strcasecmp(cmd, "COPY") == 0)
cmdtype = PROGRESS_COMMAND_COPY;
else if (pg_strcasecmp(cmd, "DATACHECKSUMS") == 0)
cmdtype = PROGRESS_COMMAND_DATACHECKSUMS;
else
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
@@ -1184,6 +1182,9 @@ pg_stat_get_db_checksum_failures(PG_FUNCTION_ARGS)
int64 result;
PgStat_StatDBEntry *dbentry;
if (!DataChecksumsEnabled())
PG_RETURN_NULL();
if ((dbentry = pgstat_fetch_stat_dbentry(dbid)) == NULL)
result = 0;
else
@@ -1199,6 +1200,9 @@ pg_stat_get_db_checksum_last_failure(PG_FUNCTION_ARGS)
TimestampTz result;
PgStat_StatDBEntry *dbentry;
if (!DataChecksumsEnabled())
PG_RETURN_NULL();
if ((dbentry = pgstat_fetch_stat_dbentry(dbid)) == NULL)
result = 0;
else

3
src/backend/utils/init/miscinit.c
View File

@@ -845,8 +845,7 @@ InitializeSessionUserIdStandalone(void)
* workers, in slot sync worker and in background workers.
*/
Assert(!IsUnderPostmaster || AmAutoVacuumWorkerProcess() ||
AmLogicalSlotSyncWorkerProcess() || AmBackgroundWorkerProcess() ||
AmDataChecksumsWorkerProcess());
AmLogicalSlotSyncWorkerProcess() || AmBackgroundWorkerProcess());
/* call only once */
Assert(!OidIsValid(AuthenticatedUserId));

30
src/backend/utils/init/postinit.c
View File

@@ -760,36 +760,8 @@ InitPostgres(const char *in_dbname, Oid dboid,
*/
SharedInvalBackendInit(false);
/*
* Prevent consuming interrupts between setting ProcSignalInit and setting
* the initial local data checksum value. If a barrier is emitted, and
* absorbed, before local cached state is initialized the state transition
* can be invalid.
*/
HOLD_INTERRUPTS();
ProcSignalInit(MyCancelKey, MyCancelKeyLength);
/*
* Initialize a local cache of the data_checksum_version, to be updated by
* the procsignal-based barriers.
*
* This intentionally happens after initializing the procsignal, otherwise
* we might miss a state change. This means we can get a barrier for the
* state we've just initialized.
*
* The postmaster (which is what gets forked into the new child process)
* does not handle barriers, therefore it may not have the current value
* of LocalDataChecksumState value (it'll have the value read from the
* control file, which may be arbitrarily old).
*
* NB: Even if the postmaster handled barriers, the value might still be
* stale, as it might have changed after this process forked.
*/
InitLocalDataChecksumState();
RESUME_INTERRUPTS();
/*
* Also set up timeout handlers needed for backend operation. We need
* these in every case except bootstrap.
@@ -928,7 +900,7 @@ InitPostgres(const char *in_dbname, Oid dboid,
errhint("You should immediately run CREATE USER \"%s\" SUPERUSER;.",
username != NULL ? username : "postgres")));
}
else if (AmBackgroundWorkerProcess() || AmDataChecksumsWorkerProcess())
else if (AmBackgroundWorkerProcess())
{
if (username == NULL && !OidIsValid(useroid))
{

6
src/backend/utils/misc/guc_parameters.dat
View File

@@ -578,13 +578,11 @@
max => '1.0',
},
{ name => 'data_checksums', type => 'enum', context => 'PGC_INTERNAL', group => 'PRESET_OPTIONS',
{ name => 'data_checksums', type => 'bool', context => 'PGC_INTERNAL', group => 'PRESET_OPTIONS',
short_desc => 'Shows whether data checksums are turned on for this cluster.',
flags => 'GUC_NOT_IN_SAMPLE | GUC_DISALLOW_IN_FILE | GUC_RUNTIME_COMPUTED',
variable => 'data_checksums',
boot_val => 'PG_DATA_CHECKSUM_OFF',
options => 'data_checksums_options',
show_hook => 'show_data_checksums',
boot_val => 'false',
},
# Can't be set by ALTER SYSTEM as it can lead to recursive definition

9
src/backend/utils/misc/guc_tables.c
View File

@@ -512,14 +512,6 @@ static const struct config_enum_entry file_extend_method_options[] = {
{NULL, 0, false}
};
static const struct config_enum_entry data_checksums_options[] = {
{"on", PG_DATA_CHECKSUM_VERSION, true},
{"off", PG_DATA_CHECKSUM_OFF, true},
{"inprogress-on", PG_DATA_CHECKSUM_INPROGRESS_ON, true},
{"inprogress-off", PG_DATA_CHECKSUM_INPROGRESS_OFF, true},
{NULL, 0, false}
};
/*
* Options for enum values stored in other modules
*/
@@ -648,6 +640,7 @@ static int shared_memory_size_in_huge_pages;
static int wal_block_size;
static int num_os_semaphores;
static int effective_wal_level = WAL_LEVEL_REPLICA;
static bool data_checksums;
static bool integer_datetimes;
#ifdef USE_ASSERT_CHECKING

9
src/backend/utils/misc/pg_controldata.c
View File

@@ -69,8 +69,8 @@ pg_control_system(PG_FUNCTION_ARGS)
Datum
pg_control_checkpoint(PG_FUNCTION_ARGS)
{
Datum values[20];
bool nulls[20];
Datum values[19];
bool nulls[19];
TupleDesc tupdesc;
HeapTuple htup;
ControlFileData *ControlFile;
@@ -154,12 +154,9 @@ pg_control_checkpoint(PG_FUNCTION_ARGS)
values[17] = TransactionIdGetDatum(ControlFile->checkPointCopy.newestCommitTsXid);
nulls[17] = false;
values[18] = Int32GetDatum(ControlFile->checkPointCopy.dataChecksumState);
values[18] = TimestampTzGetDatum(time_t_to_timestamptz(ControlFile->checkPointCopy.time));
nulls[18] = false;
values[19] = TimestampTzGetDatum(time_t_to_timestamptz(ControlFile->checkPointCopy.time));
nulls[19] = false;
htup = heap_form_tuple(tupdesc, values, nulls);
PG_RETURN_DATUM(HeapTupleGetDatum(htup));

10
src/backend/utils/misc/postgresql.conf.sample
View File

@@ -557,11 +557,11 @@
# archiver autovacuum
# backend bgworker
# bgwriter checkpointer
# checksums ioworker
# postmaster slotsyncworker
# startup syslogger
# walreceiver walsummarizer
# walwriter walsender
# ioworker postmaster
# slotsyncworker startup
# syslogger walreceiver
# walsummarizer walwriter
# walsender
#
# Level values in order of decreasing
# detail:

5
src/bin/pg_checksums/t/002_actions.pl
View File

@@ -216,6 +216,11 @@ $node->start;
command_fails([ 'pg_checksums', '--check', '--pgdata' => $pgdata ],
"fails with online cluster");
# Make sure pg_control_init reports the initial state as disabled
my $result = $node->safe_psql('postgres',
'SELECT data_page_checksum_version FROM pg_control_init();');
is($result, '0', 'ensure pg_control_init reports disabled state');
# Check corruption of table on default tablespace.
check_relation_corruption($node, 'corrupt1', 'pg_default');

2
src/bin/pg_controldata/pg_controldata.c
View File

@@ -289,8 +289,6 @@ main(int argc, char *argv[])
ControlFile->checkPointCopy.oldestCommitTsXid);
printf(_("Latest checkpoint's newestCommitTsXid:%u\n"),
ControlFile->checkPointCopy.newestCommitTsXid);
printf(_("Latest checkpoint's data_checksum_version:%u\n"),
ControlFile->checkPointCopy.dataChecksumState);
printf(_("Time of latest checkpoint: %s\n"),
ckpttime_str);
printf(_("Fake LSN counter for unlogged rels: %X/%08X\n"),

9
src/bin/pg_upgrade/controldata.c
View File

@@ -15,7 +15,6 @@
#include "access/xlog_internal.h"
#include "common/string.h"
#include "pg_upgrade.h"
#include "storage/checksum.h"
/*
@@ -737,14 +736,6 @@ check_control_data(ControlData *oldctrl,
* check_for_isn_and_int8_passing_mismatch().
*/
/*
* If data checksums are in any in-progress state then disallow the
* upgrade. The user should either let the process finish, or turn off
* data checksums, before retrying.
*/
if (oldctrl->data_checksum_version > PG_DATA_CHECKSUM_VERSION)
pg_fatal("data checksums are being enabled in the old cluster");
/*
* We might eventually allow upgrades from checksum to no-checksum
* clusters.

3
src/bin/pg_waldump/t/001_basic.pl
View File

@@ -79,8 +79,7 @@ BRIN
CommitTs
ReplicationOrigin
Generic
LogicalMessage
XLOG2$/,
LogicalMessage$/,
'rmgr list');

1
src/include/access/rmgrlist.h
View File

@@ -47,4 +47,3 @@ PG_RMGR(RM_COMMIT_TS_ID, "CommitTs", commit_ts_redo, commit_ts_desc, commit_ts_i
PG_RMGR(RM_REPLORIGIN_ID, "ReplicationOrigin", replorigin_redo, replorigin_desc, replorigin_identify, NULL, NULL, NULL, NULL)
PG_RMGR(RM_GENERIC_ID, "Generic", generic_redo, generic_desc, generic_identify, NULL, NULL, generic_mask, NULL)
PG_RMGR(RM_LOGICALMSG_ID, "LogicalMessage", logicalmsg_redo, logicalmsg_desc, logicalmsg_identify, NULL, NULL, NULL, logicalmsg_decode)
PG_RMGR(RM_XLOG2_ID, "XLOG2", xlog2_redo, xlog2_desc, xlog2_identify, NULL, NULL, NULL, xlog2_decode)

20
src/include/access/xlog.h
View File

@@ -57,7 +57,6 @@ extern PGDLLIMPORT int CommitDelay;
extern PGDLLIMPORT int CommitSiblings;
extern PGDLLIMPORT bool track_wal_io_timing;
extern PGDLLIMPORT int wal_decode_buffer_size;
extern PGDLLIMPORT int data_checksums;
extern PGDLLIMPORT int CheckPointSegments;
@@ -133,7 +132,7 @@ extern PGDLLIMPORT bool XLogLogicalInfo;
* of the bits make it to disk, but the checksum wouldn't match. Also WAL-log
* them if forced by wal_log_hints=on.
*/
#define XLogHintBitIsNeeded() (wal_log_hints || DataChecksumsNeedWrite())
#define XLogHintBitIsNeeded() (wal_log_hints || DataChecksumsEnabled())
/* Do we need to WAL-log information required only for Hot Standby and logical replication? */
#define XLogStandbyInfoActive() (wal_level >= WAL_LEVEL_REPLICA)
@@ -243,11 +242,8 @@ extern void XLogSetReplicationSlotMinimumLSN(XLogRecPtr lsn);
extern XLogRecPtr XLogGetReplicationSlotMinimumLSN(void);
extern void xlog_redo(struct XLogReaderState *record);
extern void xlog2_redo(struct XLogReaderState *record);
extern void xlog_desc(StringInfo buf, struct XLogReaderState *record);
extern void xlog2_desc(StringInfo buf, struct XLogReaderState *record);
extern const char *xlog_identify(uint8 info);
extern const char *xlog2_identify(uint8 info);
extern void issue_xlog_fsync(int fd, XLogSegNo segno, TimeLineID tli);
@@ -260,19 +256,7 @@ extern XLogRecPtr GetXLogWriteRecPtr(void);
extern uint64 GetSystemIdentifier(void);
extern char *GetMockAuthenticationNonce(void);
extern bool DataChecksumsNeedWrite(void);
extern bool DataChecksumsNeedVerify(void);
extern XLogRecPtr GetLastChecksumChangeRecPtr(void);
extern bool DataChecksumsOn(void);
extern bool DataChecksumsOff(void);
extern bool DataChecksumsInProgressOn(void);
extern void SetDataChecksumsOnInProgress(void);
extern void SetDataChecksumsOn(void);
extern void SetDataChecksumsOff(void);
extern const char *show_data_checksums(void);
extern const char *get_checksum_state_string(uint32 state);
extern void InitLocalDataChecksumState(void);
extern void SetLocalDataChecksumState(uint32 data_checksum_version);
extern bool DataChecksumsEnabled(void);
extern bool GetDefaultCharSignedness(void);
extern XLogRecPtr GetFakeLSNForUnloggedRel(void);
extern void BootStrapXLOG(uint32 data_checksum_version);

6
src/include/access/xlog_internal.h
View File

@@ -288,12 +288,6 @@ typedef struct xl_restore_point
char rp_name[MAXFNAMELEN];
} xl_restore_point;
/* Information logged when data checksum level is changed */
typedef struct xl_checksum_state
{
ChecksumStateType new_checksum_state;
} xl_checksum_state;
/* Overwrite of prior contrecord */
typedef struct xl_overwrite_contrecord
{

2
src/include/catalog/catversion.h
View File

@@ -57,6 +57,6 @@
*/
/* yyyymmddN */
#define CATALOG_VERSION_NO 202609153
#define CATALOG_VERSION_NO 202609164
#endif

9
src/include/catalog/pg_control.h
View File

@@ -22,7 +22,7 @@
/* Version identifier for this pg_control format */
#define PG_CONTROL_VERSION 1903
#define PG_CONTROL_VERSION 1905
/* Nonce key length, see below */
#define MOCK_AUTH_NONCE_LEN 32
@@ -63,9 +63,6 @@ typedef struct CheckPoint
* set to InvalidTransactionId.
*/
TransactionId oldestActiveXid;
/* data checksums state at the time of the checkpoint */
uint32 dataChecksumState;
} CheckPoint;
/* XLOG info values for XLOG rmgr */
@@ -86,10 +83,6 @@ typedef struct CheckPoint
#define XLOG_CHECKPOINT_REDO 0xE0
#define XLOG_LOGICAL_DECODING_STATUS_CHANGE 0xF0
/* XLOG info values for XLOG2 rmgr */
#define XLOG2_CHECKSUMS 0x00
/*
* System status indicator. Note this is stored in pg_control; if you change
* it, you must bump PG_CONTROL_VERSION

12
src/include/catalog/pg_proc.dat
View File

@@ -12359,18 +12359,6 @@
proname => 'jsonb_subscript_handler', prorettype => 'internal',
proargtypes => 'internal', prosrc => 'jsonb_subscript_handler' },
# data checksum management functions
{ oid => '6507', descr => 'disable data checksums',
proname => 'pg_disable_data_checksums', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => '',
prosrc => 'disable_data_checksums', proacl => '{POSTGRES=X}' },
{ oid => '6506', descr => 'enable data checksums',
proname => 'pg_enable_data_checksums', provolatile => 'v', proparallel => 'r',
prorettype => 'void', proargtypes => 'int4 int4',
proallargtypes => '{int4,int4}', proargmodes => '{i,i}',
proargnames => '{cost_delay,cost_limit}', proargdefaults => '{0,100}',
prosrc => 'enable_data_checksums', proacl => '{POSTGRES=X}' },
# collation management functions
{ oid => '3445', descr => 'import collations from operating system',
proname => 'pg_import_system_collations', procost => '100',

16
src/include/commands/progress.h
View File

@@ -188,20 +188,4 @@
#define PROGRESS_COPY_TYPE_PIPE 3
#define PROGRESS_COPY_TYPE_CALLBACK 4
/* Progress parameters for PROGRESS_DATACHECKSUMS */
#define PROGRESS_DATACHECKSUMS_PHASE 0
#define PROGRESS_DATACHECKSUMS_DBS_TOTAL 1
#define PROGRESS_DATACHECKSUMS_DBS_DONE 2
#define PROGRESS_DATACHECKSUMS_RELS_TOTAL 3
#define PROGRESS_DATACHECKSUMS_RELS_DONE 4
#define PROGRESS_DATACHECKSUMS_BLOCKS_TOTAL 5
#define PROGRESS_DATACHECKSUMS_BLOCKS_DONE 6
/* Phases of datachecksumsworker operation */
#define PROGRESS_DATACHECKSUMS_PHASE_ENABLING 0
#define PROGRESS_DATACHECKSUMS_PHASE_DISABLING 1
#define PROGRESS_DATACHECKSUMS_PHASE_WAITING_TEMPREL 2
#define PROGRESS_DATACHECKSUMS_PHASE_WAITING_BARRIER 3
#define PROGRESS_DATACHECKSUMS_PHASE_DONE 4
#endif

10
src/include/miscadmin.h
View File

@@ -370,13 +370,6 @@ typedef enum BackendType
B_WAL_SUMMARIZER,
B_WAL_WRITER,
/*
* Data checksums processes are dynamic background workers, but they use
* dedicated backend types for pgstat I/O accounting.
*/
B_DATACHECKSUMSWORKER_LAUNCHER,
B_DATACHECKSUMSWORKER_WORKER,
/*
* Logger is not connected to shared memory and does not have a PGPROC
* entry.
@@ -402,9 +395,6 @@ extern PGDLLIMPORT BackendType MyBackendType;
#define AmWalSummarizerProcess() (MyBackendType == B_WAL_SUMMARIZER)
#define AmWalWriterProcess() (MyBackendType == B_WAL_WRITER)
#define AmIoWorkerProcess() (MyBackendType == B_IO_WORKER)
#define AmDataChecksumsWorkerProcess() \
(MyBackendType == B_DATACHECKSUMSWORKER_LAUNCHER || \
MyBackendType == B_DATACHECKSUMSWORKER_WORKER)
#define AmSpecialWorkerProcess() \
(AmAutoVacuumLauncherProcess() || \

28
src/include/postmaster/datachecksum_state.h
View File

@@ -1,28 +0,0 @@
/*-------------------------------------------------------------------------
*
* datachecksum_state.h
* header file for data checksum helper background worker and data
* checksum state manipulation
*
*
* Portions Copyright (c) 1996-2026, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
*
* src/include/postmaster/datachecksum_state.h
*
*-------------------------------------------------------------------------
*/
#ifndef DATACHECKSUM_STATE_H
#define DATACHECKSUM_STATE_H
#include "storage/procsignal.h"
/* Prototypes for data checksum state manipulation */
bool AbsorbDataChecksumsBarrier(ProcSignalBarrierType barrier);
void EmitAndWaitDataChecksumsBarrier(uint32 state);
/* Background worker entrypoints */
void DataChecksumsWorkerLauncherMain(Datum arg);
void DataChecksumsWorkerMain(Datum arg);
#endif /* DATACHECKSUM_STATE_H */

2
src/include/postmaster/proctypelist.h
View File

@@ -38,8 +38,6 @@ PG_PROCTYPE(B_BACKEND, "backend", gettext_noop("client backend"), BackendMain, t
PG_PROCTYPE(B_BG_WORKER, "bgworker", gettext_noop("background worker"), BackgroundWorkerMain, true)
PG_PROCTYPE(B_BG_WRITER, "bgwriter", gettext_noop("background writer"), BackgroundWriterMain, true)
PG_PROCTYPE(B_CHECKPOINTER, "checkpointer", gettext_noop("checkpointer"), CheckpointerMain, true)
PG_PROCTYPE(B_DATACHECKSUMSWORKER_LAUNCHER, "checksums", gettext_noop("datachecksums launcher"), NULL, false)
PG_PROCTYPE(B_DATACHECKSUMSWORKER_WORKER, "checksums", gettext_noop("datachecksums worker"), NULL, false)
PG_PROCTYPE(B_DEAD_END_BACKEND, "backend", gettext_noop("dead-end client backend"), BackendMain, true)
PG_PROCTYPE(B_INVALID, "postmaster", gettext_noop("unrecognized"), NULL, false)
PG_PROCTYPE(B_IO_WORKER, "ioworker", gettext_noop("io worker"), IoWorkerMain, true)

1
src/include/replication/decode.h
View File

@@ -22,7 +22,6 @@ typedef struct XLogRecordBuffer
} XLogRecordBuffer;
extern void xlog_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf);
extern void xlog2_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf);
extern void heap_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf);
extern void heap2_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf);
extern void xact_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf);

11
src/include/storage/checksum.h
View File

@@ -16,19 +16,14 @@
#include "storage/block.h"
/*
* Checksum state 0 is used for when data checksums are disabled (OFF).
* PG_DATA_CHECKSUM_INPROGRESS_{ON|OFF} defines that data checksums are either
* currently being enabled or disabled, and PG_DATA_CHECKSUM_VERSION defines
* that data checksums are enabled. The ChecksumStateType is stored in
* pg_control so changing requires a catversion bump, and the values cannot
* be reordered. New states must be added at the end.
* Checksum state 0 is used for when data checksums are disabled (OFF), and
* PG_DATA_CHECKSUM_VERSION defines that data checksums are enabled. New
* states must be added at the end.
*/
typedef enum ChecksumStateType
{
PG_DATA_CHECKSUM_OFF = 0,
PG_DATA_CHECKSUM_VERSION = 1,
PG_DATA_CHECKSUM_INPROGRESS_OFF = 2,
PG_DATA_CHECKSUM_INPROGRESS_ON = 3,
} ChecksumStateType;
/*

1
src/include/storage/lwlocklist.h
View File

@@ -87,7 +87,6 @@ PG_LWLOCK(52, SerialControl)
PG_LWLOCK(53, AioWorkerSubmissionQueue)
PG_LWLOCK(54, WaitLSN)
PG_LWLOCK(55, LogicalDecodingControl)
PG_LWLOCK(56, DataChecksumsWorker)
PG_LWLOCK(57, AioWorkerControl)
/*

4
src/include/storage/procsignal.h
View File

@@ -50,10 +50,6 @@ typedef enum
PROCSIGNAL_BARRIER_SMGRRELEASE, /* ask smgr to close files */
PROCSIGNAL_BARRIER_UPDATE_XLOG_LOGICAL_INFO, /* ask to update
* XLogLogicalInfo */
PROCSIGNAL_BARRIER_CHECKSUM_OFF,
PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_ON,
PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_OFF,
PROCSIGNAL_BARRIER_CHECKSUM_ON,
} ProcSignalBarrierType;
/*

1
src/include/storage/subsystemlist.h
View File

@@ -84,7 +84,6 @@ PG_SHMEM_SUBSYSTEM(InjectionPointShmemCallbacks)
#endif
PG_SHMEM_SUBSYSTEM(WaitLSNShmemCallbacks)
PG_SHMEM_SUBSYSTEM(LogicalDecodingCtlShmemCallbacks)
PG_SHMEM_SUBSYSTEM(DataChecksumsShmemCallbacks)
/* AIO subsystem. This delegates to the method-specific callbacks */
PG_SHMEM_SUBSYSTEM(AioShmemCallbacks)

1
src/include/utils/backend_progress.h
View File

@@ -28,7 +28,6 @@ typedef enum ProgressCommandType
PROGRESS_COMMAND_BASEBACKUP,
PROGRESS_COMMAND_COPY,
PROGRESS_COMMAND_REPACK,
PROGRESS_COMMAND_DATACHECKSUMS,
} ProgressCommandType;
#define PGSTAT_NUM_PROGRESS_PARAM 20

1
src/test/modules/Makefile
View File

@@ -21,7 +21,6 @@ SUBDIRS = \
test_bitmapset \
test_bloomfilter \
test_cloexec \
test_checksums \
test_copy_callbacks \
test_custom_rmgrs \
test_custom_stats \

1
src/test/modules/meson.build
View File

@@ -21,7 +21,6 @@ subdir('test_binaryheap')
subdir('test_bitmapset')
subdir('test_bloomfilter')
subdir('test_cloexec')
subdir('test_checksums')
subdir('test_copy_callbacks')
subdir('test_cplusplusext')
subdir('test_custom_rmgrs')

2
src/test/modules/test_checksums/.gitignore vendored
View File

@@ -1,2 +0,0 @@
# Generated by test suite
/tmp_check/

36
src/test/modules/test_checksums/Makefile
View File

@@ -1,36 +0,0 @@
#-------------------------------------------------------------------------
#
# Makefile for src/test/modules/test_checksums
#
# Portions Copyright (c) 1996-2026, PostgreSQL Global Development Group
# Portions Copyright (c) 1994, Regents of the University of California
#
# src/test/modules/test_checksums/Makefile
#
#-------------------------------------------------------------------------
EXTRA_INSTALL = src/test/modules/injection_points
export enable_injection_points
MODULE_big = test_checksums
OBJS = \
$(WIN32RES) \
test_checksums.o
PGFILEDESC = "test_checksums - test code for data checksums"
EXTENSION = test_checksums
DATA = test_checksums--1.0.sql
TAP_TESTS = 1
ifdef USE_PGXS
PG_CONFIG = pg_config
PGXS := $(shell $(PG_CONFIG) --pgxs)
include $(PGXS)
else
subdir = src/test/modules/test_checksums
top_builddir = ../../../..
include $(top_builddir)/src/Makefile.global
include $(top_srcdir)/contrib/contrib-global.mk
endif

30
src/test/modules/test_checksums/README
View File

@@ -1,30 +0,0 @@
src/test/modules/test_checksums/README
Regression tests for data checksums
===================================
This directory contains a test suite for enabling, and disabling, data
checksums both offline as well as in a running cluster.
Running the tests with autoconf
===============================
make check
or
make installcheck
Running the tests with meson
============================
From your build directory, issue the following command:
meson test -q --print-errorlogs --suite setup --suite test_checksums
NOTE: This creates a temporary installation (in the case of "make check" or
"--suite setup"), with multiple nodes, be they master or standby(s) for the
purpose of the tests.
NOTE: This test suite requires TAP tests to be enabled, a subset of the tests
also require injection points to function. In order to run the extended test
then "checksum_extended" must be set in the PG_TEST_EXTRA environment
variable.

40
src/test/modules/test_checksums/meson.build
View File

@@ -1,40 +0,0 @@
# Copyright (c) 2026, PostgreSQL Global Development Group
test_checksums_sources = files(
'test_checksums.c',
)
test_checksums = shared_module('test_checksums',
test_checksums_sources,
kwargs: pg_test_mod_args,
)
test_install_libs += test_checksums
test_install_data += files(
'test_checksums.control',
'test_checksums--1.0.sql',
)
tests += {
'name': 'test_checksums',
'sd': meson.current_source_dir(),
'bd': meson.current_build_dir(),
'tap': {
'env': {
'enable_injection_points': get_option('injection_points') ? 'yes' : 'no',
},
'tests': [
't/001_basic.pl',
't/002_restarts.pl',
't/003_standby_restarts.pl',
't/004_offline.pl',
't/005_injection.pl',
't/006_pgbench_single.pl',
't/007_pgbench_standby.pl',
't/008_pitr.pl',
't/009_fpi.pl',
't/010_backup_straddle.pl',
't/011_standby_straddle.pl',
],
},
}

221
src/test/modules/test_checksums/t/001_basic.pl
View File

@@ -1,221 +0,0 @@
# Copyright (c) 2026, PostgreSQL Global Development Group
# Test suite for testing enabling data checksums in an online cluster
use strict;
use warnings FATAL => 'all';
use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
use Test::More;
use FindBin;
use lib $FindBin::RealBin;
use DataChecksums::Utils;
# Initialize node with checksums enabled to test pg_control_init returning
# 1 for this cluster, the remaining tests will initialize to off to test the
# return value for a cluster initialized without checksums
my $node = PostgreSQL::Test::Cluster->new('basic_node');
$node->init;
$node->start;
# Create some content to have data in the cluster
$node->safe_psql('postgres',
"CREATE TABLE t AS SELECT generate_series(1,10000) AS a;");
# Ensure that checksums are turned on
test_checksum_state($node, 'on');
# Disable data checksums and wait for the state transition to 'off'
disable_data_checksums($node, wait => 'off');
# Make sure pg_control_init reports the initial enabled state
my $result = $node->safe_psql('postgres',
'SELECT data_page_checksum_version FROM pg_control_init();');
is($result, '1', 'ensure pg_control_init reports enabled state');
# Enable data checksums and wait for the state transition to 'on'
enable_data_checksums($node, wait => 'on');
# Run a dummy query just to make sure we can read back data
$result = $node->safe_psql('postgres', "SELECT count(*) FROM t WHERE a > 1 ");
is($result, '9999', 'ensure checksummed pages can be read back');
# Ensure the new state is registered properly in pg_control_checkpoint()
$result =
$node->safe_psql('postgres',
'SELECT data_page_checksum_version FROM pg_control_checkpoint();');
is($result, '1', 'ensure pg_control_checkpoint reports enabled state');
# Regardless of the new state, pg_control_init() should still report checksums
# as on.
$result =
$node->safe_psql('postgres',
'SELECT data_page_checksum_version FROM pg_control_init();');
is($result, '1', 'ensure pg_control_init reports enabled state');
# Enable data checksums again which should be a no-op so we explicitly don't
# wait for any state transition as none should happen here.
enable_data_checksums($node);
test_checksum_state($node, 'on');
# ..and make sure we can still read/write data
$node->safe_psql('postgres', "UPDATE t SET a = a + 1;");
$result = $node->safe_psql('postgres', "SELECT count(*) FROM t WHERE a > 1");
is($result, '10000', 'ensure checksummed pages can be read back');
# Disable checksums again and wait for the state transition
disable_data_checksums($node, wait => 1);
# Test reading data again
$result = $node->safe_psql('postgres', "SELECT count(*) FROM t WHERE a > 1");
is($result, '10000', 'ensure previously checksummed pages can be read back');
# And ensure the disabled state is shown in pg_control_checkpoint()
$result =
$node->safe_psql('postgres',
'SELECT data_page_checksum_version FROM pg_control_checkpoint();');
is($result, '0', 'ensure pg_control_checkpoint reports disabled state');
# Re-enable checksums and make sure that the underlying data has changed to
# ensure that checksums will be different.
$node->safe_psql('postgres', "UPDATE t SET a = a + 1;");
enable_data_checksums($node, wait => 'on');
# Run a dummy query just to make sure we can read back the data
$result = $node->safe_psql('postgres', "SELECT count(*) FROM t WHERE a > 1");
is($result, '10000', 'ensure checksummed pages can be read back');
# Enabling checksums in a cluster which contains an invalid database left
# behind by an interrupted DROP DATABASE must be refused.
disable_data_checksums($node, wait => 1);
$node->safe_psql('postgres', "CREATE DATABASE baddb;");
$node->safe_psql('baddb',
"CREATE TABLE bad_t AS SELECT generate_series(1,100) AS a;");
# Mark the database invalid, as an interrupted DROP DATABASE would.
$node->safe_psql('postgres',
"UPDATE pg_database SET datconnlimit = -2 WHERE datname = 'baddb';");
# The request must fail up front with an actionable error, rather than fail
# halfway through processing.
my ($ret, $stdout, $stderr) =
$node->psql('postgres', "SELECT pg_enable_data_checksums();");
isnt($ret, 0, 'pg_enable_data_checksums fails with an invalid database');
like(
$stderr,
qr/invalid database "baddb"/,
'error message names the invalid database');
like(
$stderr,
qr/DROP DATABASE/,
'error message hints at dropping the database');
test_checksum_state($node, 'off');
# Dropping the invalid database clears the way.
$node->safe_psql('postgres', "DROP DATABASE baddb;");
enable_data_checksums($node, wait => 'on');
# A database dropped while processing is in progress is not an error, the
# remaining databases are still processed.
disable_data_checksums($node, wait => 1);
$node->safe_psql('postgres', "CREATE DATABASE dropme;");
$node->safe_psql('dropme',
"CREATE TABLE dropme_t AS SELECT generate_series(1,10000) AS a;");
# Hold the worker in the "postgres" database by keeping a temporary table
# around, the worker waits for pre-existing temp tables to disappear before
# it reports the database as processed. "dropme" was created last, so it is
# processed after "postgres" and is still untouched while we wait.
my $bg = $node->background_psql('postgres');
$bg->query_safe('CREATE TEMP TABLE holdme (a int);');
enable_data_checksums($node);
$node->poll_query_until(
'postgres', qq[
SELECT count(*) > 0 FROM pg_catalog.pg_stat_activity
WHERE backend_type = 'datachecksums worker' AND datname = 'postgres'
AND query LIKE 'Waiting for % temp tables to be removed']
) or die "timed out waiting for worker to wait for temporary tables";
# Verify the assumption that processing has not reached "dropme" yet, without
# it the test would silently stop covering the concurrent drop.
my $log = slurp_file($node->logfile);
unlike(
$log,
qr/initiating data checksum processing in database "dropme"/,
'processing has not reached the database to drop');
# Not processed yet and nobody is connected to it, so this must succeed.
$node->safe_psql('postgres', "DROP DATABASE dropme;");
# Let the worker in "postgres" finish, the launcher then moves on to the
# database which no longer exists.
$bg->query_safe('DROP TABLE holdme;');
$bg->quit;
wait_for_checksum_state($node, 'on');
$node->poll_query_until('postgres',
"SELECT count(*) = 0 "
. "FROM pg_catalog.pg_stat_activity "
. "WHERE backend_type = 'datachecksums launcher';");
# Same thing with DROP DATABASE ... WITH (FORCE), which terminates the
# checksums worker connected to the database being dropped.
disable_data_checksums($node, wait => 1);
$node->safe_psql('postgres', "CREATE DATABASE dropmeforce;");
$node->safe_psql('dropmeforce',
"CREATE TABLE dropme_t AS SELECT generate_series(1,10000) AS a;");
# Hold the worker inside "dropmeforce" by keeping a temporary table around
# there. The session is later terminated by DROP DATABASE ... WITH (FORCE)
# and drops the temp table while exiting; commit that drop asynchronously,
# otherwise the exit has to flush WAL behind the traffic generated by the
# checksum workers, which on slow machines can exceed the time DROP DATABASE
# waits for the session to exit.
$bg = $node->background_psql('dropmeforce');
$bg->query_safe('SET synchronous_commit = off;');
$bg->query_safe('CREATE TEMP TABLE holdme (a int);');
enable_data_checksums($node);
$node->poll_query_until(
'postgres', qq[
SELECT count(*) > 0 FROM pg_catalog.pg_stat_activity
WHERE backend_type = 'datachecksums worker' AND datname = 'dropmeforce'
AND query LIKE 'Waiting for % temp tables to be removed']
) or die "timed out waiting for worker to wait for temporary tables";
# Write out the WAL backlog from the checksum processing so far, so that the
# sessions terminated below do not get stuck behind it when writing their
# final WAL records on slow machines.
$node->safe_psql('postgres', "CHECKPOINT;");
# Terminates both the session holding the temp table and the checksums
# worker connected to the database.
$node->safe_psql('postgres', "DROP DATABASE dropmeforce WITH (FORCE);");
$bg->{run}->finish;
$bg->quit;
wait_for_checksum_state($node, 'on');
$node->poll_query_until('postgres',
"SELECT count(*) = 0 "
. "FROM pg_catalog.pg_stat_activity "
. "WHERE backend_type = 'datachecksums launcher';");
$result = $node->safe_psql('postgres', "SELECT count(*) FROM t WHERE a > 1");
is($result, '10000', 'ensure checksummed pages can be read back');
$node->stop;
# The resulting cluster must also pass offline verification, proving no
# unchecksummed files were left behind.
command_ok(
[ 'pg_checksums', '--check', '-D', $node->data_dir ],
'offline checksum verification passes after enable');
done_testing();

160
src/test/modules/test_checksums/t/002_restarts.pl
View File

@@ -1,160 +0,0 @@
# Copyright (c) 2026, PostgreSQL Global Development Group
# Test suite for testing enabling data checksums in an online cluster with a
# restart which breaks processing.
use strict;
use warnings FATAL => 'all';
use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
use Test::More;
use FindBin;
use lib $FindBin::RealBin;
use DataChecksums::Utils;
# Initialize node with checksums disabled.
my $node = PostgreSQL::Test::Cluster->new('restarts_node');
$node->init(no_data_checksums => 1);
$node->start;
# Initialize result storage for queries
my $result;
# Create some content to have un-checksummed data in the cluster
$node->safe_psql('postgres',
"CREATE TABLE t AS SELECT generate_series(1,10000) AS a;");
# Ensure that checksums are disabled
test_checksum_state($node, 'off');
# Make sure pg_control_init reports the initial disabled state
$result = $node->safe_psql('postgres',
'SELECT data_page_checksum_version FROM pg_control_init();');
is($result, '0', 'ensure pg_control_init reports disabled state');
SKIP:
{
skip 'Data checksum delay tests not enabled in PG_TEST_EXTRA', 6
if (!$ENV{PG_TEST_EXTRA}
|| $ENV{PG_TEST_EXTRA} !~ /\bchecksum_extended\b/);
# Create a barrier for checksum enablement to block on, in this case a pre-
# existing temporary table which is kept open while processing is started.
# We can accomplish this by setting up an interactive psql process which
# keeps the temporary table created as we enable checksums in another psql
# process.
#
# This is a similar test to the synthetic variant in 005_injection.pl
# which fakes this scenario.
my $bsession = $node->background_psql('postgres');
$bsession->query_safe('CREATE TEMPORARY TABLE tt (a integer);');
# In another session, make sure we can see the blocking temp table but
# start processing anyways and check that we are blocked with a proper
# wait event.
$result = $node->safe_psql('postgres',
"SELECT relpersistence FROM pg_catalog.pg_class WHERE relname = 'tt';"
);
is($result, 't', 'ensure we can see the temporary table');
# Enabling data checksums shouldn't work as the process is blocked on the
# temporary table held open by $bsession. Ensure that we reach inprogress-
# on before we do more tests.
enable_data_checksums($node, wait => 'inprogress-on');
# Wait for processing to finish and the worker waiting for leftover temp
# relations to be able to actually finish
$result = $node->poll_query_until(
'postgres',
"SELECT wait_event FROM pg_catalog.pg_stat_activity "
. "WHERE backend_type = 'datachecksums worker';",
'ChecksumEnableTemptableWait');
# The datachecksumsworker waits for temporary tables to disappear for 3
# seconds before retrying, so sleep for 4 seconds to be guaranteed to see
# a retry cycle
sleep(4);
# Re-check the wait event to ensure we are blocked on the right thing.
$result = $node->safe_psql('postgres',
"SELECT wait_event FROM pg_catalog.pg_stat_activity "
. "WHERE backend_type = 'datachecksums worker';");
is($result, 'ChecksumEnableTemptableWait',
'ensure the correct wait condition is set');
test_checksum_state($node, 'inprogress-on');
# Stop the cluster while bsession is still attached. We can't close the
# session first since the brief period between closing and stopping might
# be enough for checksums to get enabled.
$node->stop;
$bsession->quit;
$node->start;
# Ensure the checksums aren't enabled across the restart. This leaves the
# cluster in the same state as before we entered the SKIP block.
test_checksum_state($node, 'off');
}
# Test interrupting the processing with SIGINT to make sure the launcher and
# worker are cancelled. Create a barrier for checksum enablement to block on
# using the same technique as earlier with a temporary table.
my $block_session = $node->background_psql('postgres');
$block_session->query_safe('CREATE TEMPORARY TABLE tt (a integer);');
# In another session, make sure we can see the blocking temp table but
# start processing anyways and check that we are blocked with a proper
# wait event.
$result = $node->safe_psql('postgres',
"SELECT relpersistence FROM pg_catalog.pg_class WHERE relname = 'tt';");
is($result, 't', 'ensure we can see the temporary table');
# Ensure that we reach inprogress-on and the worker is launched and waiting for
# the temporary table to disappear before we try to interrupt the processing.
enable_data_checksums($node, wait => 'inprogress-on');
$result = $node->poll_query_until(
'postgres',
"SELECT wait_event FROM pg_catalog.pg_stat_activity "
. "WHERE backend_type = 'datachecksums worker';",
'ChecksumEnableTemptableWait');
is($result, '1', 'ensure the correct wait condition is set');
# Terminate launcher and worker with SIGINT
my $pid = $node->safe_psql('postgres',
"SELECT pid FROM pg_catalog.pg_stat_activity WHERE backend_type = 'datachecksums launcher';"
);
is(PostgreSQL::Test::Utils::system_log('pg_ctl', 'kill', 'INT', $pid),
0, "datachecksums launcher process signalled with INT");
# Wait for all processes to exit and make sure that the data_checksum state
# was reverted back to off since we didn't finish
$result = $node->poll_query_until(
'postgres',
"SELECT count(*) FROM pg_catalog.pg_stat_activity WHERE backend_type LIKE 'datachecksums%';",
'0');
is($result, 1, 'await datachecksums worker/launcher termination');
wait_for_checksum_state($node, "off");
$block_session->quit;
# Finish test suite by enabling checksums and make sure all data can be read
# back, no processes are left over and the initial state is still correctly
# reported
enable_data_checksums($node, wait => 'on');
$result = $node->safe_psql('postgres',
'SELECT data_page_checksum_version FROM pg_control_init();');
is($result, '0', 'ensure pg_control_init still reports disabled state');
$result = $node->safe_psql('postgres', "SELECT count(*) FROM t WHERE a > 1");
is($result, '9999', 'ensure checksummed pages can be read back');
$result = $node->poll_query_until(
'postgres',
"SELECT count(*) FROM pg_catalog.pg_stat_activity WHERE backend_type LIKE 'datachecksums%';",
'0');
is($result, 1, 'await datachecksums worker/launcher termination');
$node->stop;
done_testing();

288
src/test/modules/test_checksums/t/003_standby_restarts.pl
View File

@@ -1,288 +0,0 @@
# Copyright (c) 2026, PostgreSQL Global Development Group
# Test suite for testing enabling data checksums in an online cluster with
# streaming replication
use strict;
use warnings FATAL => 'all';
use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
use Test::More;
use FindBin;
use lib $FindBin::RealBin;
use DataChecksums::Utils;
# Initialize primary node
my $node_primary = PostgreSQL::Test::Cluster->new('standby_restarts_primary');
$node_primary->init(allows_streaming => 1, no_data_checksums => 1);
$node_primary->start;
my $slotname = 'physical_slot';
$node_primary->safe_psql('postgres',
"SELECT pg_create_physical_replication_slot('$slotname')");
# Take backup
my $backup_name = 'my_backup';
$node_primary->backup($backup_name);
# Create streaming standby linking to primary
my $node_standby = PostgreSQL::Test::Cluster->new('standby_restarts_standby');
$node_standby->init_from_backup($node_primary, $backup_name,
has_streaming => 1);
$node_standby->append_conf(
'postgresql.conf', qq[
primary_slot_name = '$slotname'
]);
$node_standby->start;
# Create some content on the primary to have un-checksummed data in the cluster
$node_primary->safe_psql('postgres',
"CREATE TABLE t AS SELECT generate_series(1,10000) AS a;");
# Wait for standby to catch up
$node_primary->wait_for_catchup($node_standby, 'replay',
$node_primary->lsn('insert'));
# Check that checksums are turned off on all nodes
test_checksum_state($node_primary, 'off');
test_checksum_state($node_standby, 'off');
# ---------------------------------------------------------------------------
# Enable checksums for the cluster, and make sure that both the primary and
# standby change state.
#
# Initiate enabling of checksums and ensure that the primary switches to
# either "inprogress-on" or "on"
enable_data_checksums($node_primary);
my $result = $node_primary->poll_query_until(
'postgres',
"SELECT setting = 'off' FROM pg_catalog.pg_settings WHERE name = 'data_checksums';",
'f');
is($result, 1, 'ensure primary has transitioned from off');
# Wait for checksum enable to be replayed
$node_primary->wait_for_catchup($node_standby, 'replay');
# Ensure that the standby has switched to "inprogress-on" or "on". Normally it
# would be "inprogress-on", but it is theoretically possible for the primary to
# complete the checksum enabling *and* have the standby replay that record
# before we reach the check below.
$result = $node_standby->poll_query_until(
'postgres',
"SELECT setting = 'off' FROM pg_catalog.pg_settings WHERE name = 'data_checksums';",
'f');
is($result, 1, 'ensure standby has absorbed the inprogress-on barrier');
$result = $node_standby->safe_psql('postgres',
"SELECT setting FROM pg_catalog.pg_settings WHERE name = 'data_checksums';"
);
is(($result eq 'inprogress-on' || $result eq 'on'),
1, 'ensure checksums are on, or in progress, on standby_1');
# Insert some more data which should be checksummed on INSERT
$node_primary->safe_psql('postgres',
"INSERT INTO t VALUES (generate_series(1, 10000));");
# Wait for checksums enabled on the primary and standby
wait_for_checksum_state($node_primary, 'on');
wait_for_checksum_state($node_standby, 'on');
$result =
$node_primary->safe_psql('postgres', "SELECT count(a) FROM t WHERE a > 1");
is($result, '19998', 'ensure we can safely read all data with checksums');
$result = $node_primary->poll_query_until(
'postgres',
"SELECT count(*) FROM pg_catalog.pg_stat_activity WHERE backend_type LIKE 'datachecksums%';",
'0');
is($result, 1, 'await datachecksums worker/launcher termination');
#
# Disable checksums and ensure it's propagated to standby and that we can
# still read all data
#
# Disable checksums and wait for the operation to be replayed
disable_data_checksums($node_primary);
$node_primary->wait_for_catchup($node_standby, 'replay');
# Ensure that the primary and standby has switched to off
wait_for_checksum_state($node_primary, 'off');
wait_for_checksum_state($node_standby, 'off');
# Double-check reading data without errors
$result =
$node_primary->safe_psql('postgres', "SELECT count(a) FROM t WHERE a > 1");
is($result, "19998", 'ensure we can safely read all data without checksums');
# ---------------------------------------------------------------------------
# Test that enabling checksums does not emit WAL for unlogged relations.
# Unlogged relations are wiped on recovery, so FPIs for them would be
# pointless and waste WAL traffic / standby I/O.
#
# Additionally, exercise standby promotion to ensure the init fork of an
# unlogged relation is still WAL-logged during checksum enable -- otherwise
# the standby keeps a stale init fork and the post-promotion main fork
# fails verification on every read (see ResetUnloggedRelations()). Both
# tables must exist BEFORE enable_data_checksums() so that their init
# forks get re-checksummed during the enable sweep.
#
$node_primary->safe_psql('postgres',
'CREATE UNLOGGED TABLE unlogged_tbl AS SELECT generate_series(1,1000) AS a;'
);
# Use a btree index so the init fork is non-trivial (one metapage).
$node_primary->safe_psql(
'postgres', q[
CREATE UNLOGGED TABLE unlogged_promo (id int PRIMARY KEY,
payload text);
INSERT INTO unlogged_promo
SELECT g, repeat('x', 100) FROM generate_series(1, 1000) g;
CREATE INDEX unlogged_promo_payload_idx ON unlogged_promo (payload);
]);
$node_primary->wait_for_catchup($node_standby, 'replay',
$node_primary->lsn('insert'));
# Get the relfilenode and database OID so we can inspect the filesystem
my $unlogged_rfn = $node_primary->safe_psql('postgres',
"SELECT relfilenode FROM pg_catalog.pg_class WHERE relname = 'unlogged_tbl';"
);
my $db_oid = $node_primary->safe_psql('postgres',
"SELECT oid FROM pg_catalog.pg_database WHERE datname = 'postgres';");
# Verify the standby only has the init fork (no main fork)
my $standby_datadir = $node_standby->data_dir;
ok( !-f "$standby_datadir/base/$db_oid/$unlogged_rfn",
'standby has no main fork for unlogged table before enable');
# Re-enable data checksums
enable_data_checksums($node_primary, wait => 'on');
wait_for_checksum_state($node_standby, 'on');
# After standby replays, the unlogged main file must still not exist.
# If the bug were present, FPI replay would materialize the full table.
$node_primary->wait_for_catchup($node_standby, 'replay',
$node_primary->lsn('insert'));
ok( !-f "$standby_datadir/base/$db_oid/$unlogged_rfn",
'standby has no main fork for unlogged table after enable');
# Verify unlogged relation size is 0 on the standby (main fork missing)
my $standby_size = $node_standby->safe_psql('postgres',
"SELECT pg_relation_size('unlogged_tbl', 'main');");
is($standby_size, '0',
'unlogged table has zero size on standby after checksum enable');
# Unlogged table should still be readable on primary
$result =
$node_primary->safe_psql('postgres', 'SELECT count(*) FROM unlogged_tbl;');
is($result, '1000',
'unlogged table readable on primary after checksum enable');
# Alter persistence to logged, and make sure we can read it on both the primary
# and standby without any page verification errors in the logfiles.
$node_primary->safe_psql('postgres', 'ALTER TABLE unlogged_tbl SET logged;');
$node_primary->wait_for_catchup($node_standby, 'replay',
$node_primary->lsn('insert'));
$result =
$node_primary->safe_psql('postgres', 'SELECT sum(a) FROM unlogged_tbl;');
is($result, '500500', 'previously unlogged table can be read on primary');
$result =
$node_standby->safe_psql('postgres', 'SELECT sum(a) FROM unlogged_tbl;');
is($result, '500500', 'previously unlogged table can be read on standby');
# ---------------------------------------------------------------------------
# Promote the standby and verify the unlogged_promo relation (created above
# before the enable sweep) is still usable. Without the init-fork WAL fix,
# every read of the index would fail with
# "page verification failed, calculated checksum X but expected 0".
#
$node_primary->stop;
$node_standby->promote;
$result =
$node_standby->safe_psql('postgres',
'SELECT count(*) FROM unlogged_promo;');
is($result, '0',
'unlogged table readable on promoted standby (truncated as expected)');
$node_standby->safe_psql('postgres',
"INSERT INTO unlogged_promo SELECT g, repeat('y',100) FROM generate_series(1,100) g;"
);
$result = $node_standby->safe_psql('postgres',
'SET enable_seqscan = off; SELECT id FROM unlogged_promo WHERE id = 50;');
is($result, '50', 'indexed lookup on promoted standby returns expected row');
$node_standby->stop;
# Perform one final pass over the logs and hunt for unexpected errors
my $log = PostgreSQL::Test::Utils::slurp_file($node_primary->logfile, 0);
unlike(
$log,
qr/page verification failed,.+\d$/m,
"no checksum validation errors in primary log");
$log = PostgreSQL::Test::Utils::slurp_file($node_standby->logfile, 0);
unlike(
$log,
qr/page verification failed,.+\d$/m,
"no checksum validation errors in standby log");
# ---------------------------------------------------------------------------
# Test that enforced state transitions during promotion (via StartupXLOG) are
# performed as expected. When the primary crashes during inprogress-on the
# standby should revert to off at promotion. In order to check the transition
# the test will keep an open psql session with the standby during promotion.
# The cluster is currently broken down from the previous test. Start up the
# primary as primary, disable checksums and create a new standby from that
# state.
$node_standby->clean_node();
$node_primary->start();
disable_data_checksums($node_primary, wait => 'off');
# Re-create a new streaming standby linking to primary. The replication slot
# name is reused from earlier but a fresh backup is taken
$backup_name = 'my_new_backup';
$node_primary->backup($backup_name);
$node_standby = PostgreSQL::Test::Cluster->new('standby_restarts_standby');
$node_standby->init_from_backup($node_primary, $backup_name,
has_streaming => 1);
$node_standby->append_conf(
'postgresql.conf', qq[
primary_slot_name = '$slotname'
]);
$node_standby->start;
$node_primary->wait_for_catchup($node_standby, 'replay');
# Open a background psql connection on the primary and inject a barrier to
# block progress on to keep the state from advancing past inprogress-on
my $node_primary_bpsql = $node_primary->background_psql('postgres');
$node_primary_bpsql->query_safe('CREATE TEMPORARY TABLE tt (a integer);');
# Also open a background psql connection to the standby to make sure we have
# an active backend during promotion.
my $node_standby_bpsql = $node_standby->background_psql('postgres');
# Start to enable checksums and wait until both primary and standby have moved
# to the inprogress-on state. Processing will block here as the temporary rel
# barrier will block the primary from finishing.
enable_data_checksums($node_primary, wait => 'inprogress-on');
$node_primary->wait_for_catchup($node_standby, 'replay');
test_checksum_state($node_standby, 'inprogress-on');
# Crash the primary before checksums are enabled and promote the standby. The
# new primary node will now revert the state of 'off' since checksums weren't
# fully enabled during the crash.
$node_primary->teardown_node();
$node_standby->promote;
wait_for_checksum_state($node_standby, 'off');
# Ensure that the any backend which was active before, and during, promotion
# sees the new state.
$result = $node_standby_bpsql->query_safe("SHOW data_checksums;");
is($result, 'off',
'ensure checksums are set to off after promotion during inprogress-on');
$node_standby_bpsql->quit;
$node_standby->stop;
done_testing();

108
src/test/modules/test_checksums/t/004_offline.pl
View File

@@ -1,108 +0,0 @@
# Copyright (c) 2026, PostgreSQL Global Development Group
# Test suite for testing enabling data checksums offline from various states
# of checksum processing
use strict;
use warnings FATAL => 'all';
use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
use Test::More;
use FindBin;
use lib $FindBin::RealBin;
use DataChecksums::Utils;
# Initialize node with checksums disabled.
my $node = PostgreSQL::Test::Cluster->new('offline_node');
$node->init(no_data_checksums => 1);
$node->start;
# Make sure pg_control_init reports the initial state as disabled
my $result = $node->safe_psql('postgres',
'SELECT data_page_checksum_version FROM pg_control_init();');
is($result, '0', 'ensure pg_control_init reports disabled state');
# Create some content to have un-checksummed data in the cluster
$node->safe_psql('postgres',
"CREATE TABLE t AS SELECT generate_series(1,10000) AS a;");
# Ensure that checksums are disabled
test_checksum_state($node, 'off');
# Enable checksums offline using pg_checksums
$node->stop;
$node->checksum_enable_offline;
$node->start;
# Ensure that checksums are enabled
test_checksum_state($node, 'on');
# Since offline checksums don't issue a checkpoint like online checksums, the
# first call to pg_control_checkpoint will show the state as off even though
# checksums are enabled. After a CHECKPOINT, pg_control_checkpoint shall
# return 1.
$result = $node->safe_psql('postgres',
'SELECT data_page_checksum_version FROM pg_control_checkpoint();');
is($result, '0', 'latest checkpoint will still see off state');
$node->safe_psql('postgres', 'CHECKPOINT;');
$result = $node->safe_psql('postgres',
'SELECT data_page_checksum_version FROM pg_control_checkpoint();');
is($result, '1', 'latest checkpoint will now see on state');
# Make sure pg_control_init still reports the initial state as disabled even
# though the current state has changed.
$result = $node->safe_psql('postgres',
'SELECT data_page_checksum_version FROM pg_control_init();');
is($result, '0', 'ensure pg_control_init still reports disabled state');
# Run a dummy query just to make sure we can read back some data
$result = $node->safe_psql('postgres', "SELECT count(*) FROM t WHERE a > 1");
is($result, '9999', 'ensure checksummed pages can be read back');
# Disable checksums offline again using pg_checksums
$node->stop;
$node->checksum_disable_offline;
$node->start;
# Ensure that checksums are disabled
test_checksum_state($node, 'off');
# Create a barrier for checksum enablement to block on, in this case a pre-
# existing temporary table which is kept open while processing is started. We
# can accomplish this by setting up an interactive psql process which keeps the
# temporary table created as we enable checksums in another psql process.
my $bsession = $node->background_psql('postgres');
$bsession->query_safe('CREATE TEMPORARY TABLE tt (a integer);');
# In another session, make sure we can see the blocking temp table but start
# processing anyways and check that we are blocked with a proper wait event.
$result = $node->safe_psql('postgres',
"SELECT relpersistence FROM pg_catalog.pg_class WHERE relname = 'tt';");
is($result, 't', 'ensure we can see the temporary table');
# Enable, but stop waiting at inprogress-on since it will sit there until the
# above temporary table is removed.
enable_data_checksums($node, wait => 'inprogress-on');
# Turn the cluster off and enable checksums offline, then start back up.
# Stop the cluster before exiting the background session since otherwise
# checksums might have time to get enabled before shutting down the cluster.
$node->stop('fast');
$bsession->quit;
$node->checksum_enable_offline;
$node->start;
# Ensure that checksums are now enabled even though processing wasn't
# restarted
test_checksum_state($node, 'on');
# Run a dummy query just to make sure we can read back some data
$result = $node->safe_psql('postgres', "SELECT count(*) FROM t WHERE a > 1");
is($result, '9999', 'ensure checksummed pages can be read back');
$node->stop;
done_testing();

215
src/test/modules/test_checksums/t/005_injection.pl
View File

@@ -1,215 +0,0 @@
# Copyright (c) 2026, PostgreSQL Global Development Group
# Test suite for testing enabling data checksums in an online cluster with
# injection point tests injecting failures into the processing
use strict;
use warnings FATAL => 'all';
use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
use Test::More;
use FindBin;
use lib $FindBin::RealBin;
use DataChecksums::Utils;
if ($ENV{enable_injection_points} ne 'yes')
{
plan skip_all => 'Injection points not supported by this build';
}
# ---------------------------------------------------------------------------
# Test cluster setup
#
# Initiate test cluster
my $node = PostgreSQL::Test::Cluster->new('injection_node');
$node->init(no_data_checksums => 1);
$node->start;
# Set up test environment
$node->safe_psql('postgres', 'CREATE EXTENSION test_checksums;');
$node->safe_psql('postgres', 'CREATE EXTENSION injection_points;');
# ---------------------------------------------------------------------------
# Inducing failures and crashes in processing
# Force enabling checksums to fail by marking one of the databases as having
# failed in processing.
disable_data_checksums($node, wait => 1);
$node->safe_psql('postgres',
"SELECT injection_points_attach('datachecksumsworker-fail-db-result','notice');"
);
enable_data_checksums($node, wait => 'off');
$node->safe_psql('postgres',
"SELECT injection_points_detach('datachecksumsworker-fail-db-result');");
# Make sure that disabling after a failure works
disable_data_checksums($node, wait => 1);
test_checksum_state($node, 'off');
# ---------------------------------------------------------------------------
# Timing and retry related tests
#
SKIP:
{
skip 'Data checksum delay tests not enabled in PG_TEST_EXTRA', 4
if (!$ENV{PG_TEST_EXTRA}
|| $ENV{PG_TEST_EXTRA} !~ /\bchecksum_extended\b/);
# Inject a delay in the barrier for enabling checksums
disable_data_checksums($node, wait => 1);
$node->safe_psql('postgres', 'SELECT dcw_inject_delay_barrier();');
enable_data_checksums($node, wait => 'on');
# Fake the existence of a temporary table at the start of processing, which
# will force the processing to wait and retry in order to wait for it to
# disappear.
disable_data_checksums($node, wait => 1);
$node->safe_psql('postgres',
"SELECT injection_points_attach('datachecksumsworker-fake-temptable-wait', 'notice');"
);
enable_data_checksums($node, wait => 'on');
$node->safe_psql('postgres',
"SELECT injection_points_detach('datachecksumsworker-fake-temptable-wait');"
);
}
# ---------------------------------------------------------------------------
# Test an interrupted DROP DATABASE while checksum enabling is in progress
#
disable_data_checksums($node, wait => 1);
$node->safe_psql('postgres', 'CREATE DATABASE invalid_dropdb;');
$node->safe_psql('invalid_dropdb',
"CREATE TABLE bad_t AS SELECT generate_series(1,1000) AS a;");
# Hold the worker in "postgres", so invalid_dropdb is still waiting in the
# launcher's database list when DROP DATABASE is interrupted.
my $hold = $node->background_psql('postgres');
$hold->query_safe('CREATE TEMP TABLE holdme (a int);');
my $dropdb_log_offset = -s $node->logfile;
enable_data_checksums($node);
$node->poll_query_until(
'postgres', qq[
SELECT count(*) > 0 FROM pg_catalog.pg_stat_activity
WHERE backend_type = 'datachecksums worker' AND datname = 'postgres'
AND query LIKE 'Waiting for % temp tables to be removed']
) or die "timed out waiting for worker to wait for temporary tables";
my $dropdb_log =
PostgreSQL::Test::Utils::slurp_file($node->logfile, $dropdb_log_offset);
unlike(
$dropdb_log,
qr/initiating data checksum processing in database "invalid_dropdb"/,
'invalid database has not been processed yet');
# Leave the database durably marked invalid, but abort DROP DATABASE before
# its catalog row and files are removed.
$node->safe_psql('postgres',
"SELECT injection_points_attach('dropdb-after-invalid-marker','error');");
my ($drop_ret, $drop_stdout, $drop_stderr) =
$node->psql('postgres', 'DROP DATABASE invalid_dropdb;');
isnt($drop_ret, 0, 'DROP DATABASE was interrupted after invalidation');
like(
$drop_stderr,
qr/dropdb-after-invalid-marker/,
'DROP DATABASE reached the invalid-marker injection point');
$node->safe_psql('postgres',
"SELECT injection_points_detach('dropdb-after-invalid-marker');");
my $invalid_state = $node->safe_psql('postgres',
"SELECT datconnlimit FROM pg_catalog.pg_database WHERE datname = 'invalid_dropdb';"
);
is($invalid_state, '-2', 'interrupted DROP left an invalid database row');
# Let checksum processing continue. The invalid database must be treated as
# a processing failure, not as a successfully dropped database.
$hold->query_safe('DROP TABLE holdme;');
$hold->quit;
$node->poll_query_until('postgres',
"SELECT count(*) = 0 "
. "FROM pg_catalog.pg_stat_activity "
. "WHERE backend_type = 'datachecksums launcher';")
or die "timed out waiting for datachecksums launcher to exit";
test_checksum_state($node, 'off');
# Remove the invalid database and continue with the remaining tests.
$node->safe_psql('postgres', 'DROP DATABASE invalid_dropdb;');
# ---------------------------------------------------------------------------
# Test concurrent CREATE DATABASE which use the file_copy strategy
#
disable_data_checksums($node, wait => 1);
my $node_loglocation = -s $node->logfile;
$node->safe_psql('postgres',
"CREATE TABLE t AS SELECT generate_series(1,10000) AS a;");
$node->safe_psql('postgres',
"SELECT injection_points_attach('createdb-before-catalog-insert','wait');"
);
$node->safe_psql('postgres',
"SELECT injection_points_attach('datachecksumsworker-fake-temptable-wait','wait');"
);
# Hold CREATE DATABASE after the strategy check, before its xact is visible.
my $bg = $node->background_psql('postgres');
$bg->query_until(
qr/starting_create/, q(
\echo starting_create
CREATE DATABASE fcdb TEMPLATE template0 STRATEGY file_copy;
));
$node->wait_for_event('client backend', 'createdb-before-catalog-insert');
# Enable checksums, worker holds before processing template0.
enable_data_checksums($node);
$node->wait_for_event('datachecksums worker',
'datachecksumsworker-fake-temptable-wait');
# Release CREATE DATABASE, must fail on the recheck instead of raw-copying.
$node->safe_psql('postgres',
"SELECT injection_points_wakeup('createdb-before-catalog-insert');");
$node->safe_psql('postgres',
"SELECT injection_points_detach('createdb-before-catalog-insert');");
# Wait for the CREATE DATABASE xact to finish before releasing the worker.
$node->poll_query_until('postgres',
"SELECT count(*) = 0 FROM pg_catalog.pg_stat_activity "
. "WHERE query LIKE 'CREATE DATABASE%' AND state != 'idle';");
$node->safe_psql('postgres',
"SELECT injection_points_wakeup('datachecksumsworker-fake-temptable-wait');"
);
$node->safe_psql('postgres',
"SELECT injection_points_detach('datachecksumsworker-fake-temptable-wait');"
);
wait_for_checksum_state($node, 'on');
my $result = $node->safe_psql('postgres',
"SELECT count(*) FROM pg_catalog.pg_database WHERE datname = 'fcdb';");
is($result, '0', 'file_copy database creation was refused');
my $log =
PostgreSQL::Test::Utils::slurp_file($node->logfile, $node_loglocation);
like(
$log,
qr/create database strategy "file_copy" not allowed/m,
'file_copy error message in log');
# ---------------------------------------------------------------------------
# Test teardown
#
$bg->{run}->finish;
$bg->quit;
$node->stop;
done_testing();

285
src/test/modules/test_checksums/t/006_pgbench_single.pl
View File

@@ -1,285 +0,0 @@
# Copyright (c) 2026, PostgreSQL Global Development Group
# Test suite for testing enabling data checksums in an online cluster with
# concurrent activity via pgbench runs
use strict;
use warnings FATAL => 'all';
use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
use Test::More;
use FindBin;
use lib $FindBin::RealBin;
use DataChecksums::Utils;
# This test suite is expensive, or very expensive, to execute. There are two
# PG_TEST_EXTRA options for running it, "checksum" for a pared-down test suite
# an "checksum_extended" for the full suite. The full suite can run for hours
# on slow or constrained systems.
my $extended = undef;
if ($ENV{PG_TEST_EXTRA})
{
$extended = 1 if ($ENV{PG_TEST_EXTRA} =~ /\bchecksum_extended\b/);
plan skip_all => 'Expensive data checksums test disabled'
unless ($ENV{PG_TEST_EXTRA} =~ /\bchecksum(_extended)?\b/);
}
else
{
plan skip_all => 'Expensive data checksums test disabled';
}
if ($ENV{enable_injection_points} ne 'yes')
{
plan skip_all => 'Injection points not supported by this build';
}
my $node;
my $node_loglocation = 0;
# The number of full test iterations which will be performed. The exact number
# of tests performed and the wall time taken is non-deterministic as the test
# performs a lot of randomized actions, but 10 iterations will be a long test
# run regardless.
my $TEST_ITERATIONS = 1;
$TEST_ITERATIONS = 10 if ($extended);
# Variables which record the current state of the cluster
my $data_checksum_state = 'off';
my $pgbench = undef;
# Start a pgbench run in the background against the server specified via the
# port passed as parameter.
sub background_rw_pgbench
{
my $port = shift;
# If a previous pgbench is still running, start by shutting it down.
$pgbench->finish if $pgbench;
my $clients = 1;
my $runtime = 2;
if ($extended)
{
# Randomize the number of pgbench clients a bit (range 1-16)
$clients = 1 + int(rand(15));
$runtime = 600;
}
my @cmd = ('pgbench', '-p', $port, '-T', $runtime, '-c', $clients);
# Randomize whether we spawn connections or not
push(@cmd, '-C') if ($extended && cointoss);
# Finally add the database name to use
push(@cmd, 'postgres');
$pgbench = IPC::Run::start(
\@cmd,
'<' => '/dev/null',
'>' => '/dev/null',
'2>' => '/dev/null',
IPC::Run::timer($PostgreSQL::Test::Utils::timeout_default));
}
# Invert the state of data checksums in the cluster, if data checksums are on
# then disable them and vice versa. Also performs proper validation of the
# before and after state.
sub flip_data_checksums
{
my $temptablewait = 0;
# First, make sure the cluster is in the state we expect it to be
test_checksum_state($node, $data_checksum_state);
if ($data_checksum_state eq 'off')
{
# Coin-toss to see if we are injecting a retry due to a temptable
if (cointoss())
{
$node->safe_psql('postgres',
"SELECT injection_points_attach('datachecksumsworker-fake-temptable-wait', 'notice');"
);
$temptablewait = 1;
}
# log LSN right before we start changing checksums
my $result =
$node->safe_psql('postgres', "SELECT pg_current_wal_lsn()");
note("LSN before enabling: " . $result . "\n");
# Ensure that the primary switches to "inprogress-on"
enable_data_checksums($node, wait => 'inprogress-on');
random_sleep() if ($extended);
# Wait for checksums enabled on the primary
wait_for_checksum_state($node, 'on');
# log LSN right after the primary flips checksums to "on"
$result = $node->safe_psql('postgres', "SELECT pg_current_wal_lsn()");
note("LSN after enabling: " . $result . "\n");
random_sleep() if ($extended);
$node->safe_psql('postgres',
"SELECT injection_points_detach('datachecksumsworker-fake-temptable-wait');"
) if ($temptablewait);
$data_checksum_state = 'on';
}
elsif ($data_checksum_state eq 'on')
{
random_sleep() if ($extended);
# log LSN right before we start changing checksums
my $result =
$node->safe_psql('postgres', "SELECT pg_current_wal_lsn()");
note("LSN before disabling: " . $result . "\n");
disable_data_checksums($node);
# Wait for checksums disabled on the primary
wait_for_checksum_state($node, 'off');
# log LSN right after the primary flips checksums to "off"
$result = $node->safe_psql('postgres', "SELECT pg_current_wal_lsn()");
note("LSN after disabling: " . $result . "\n");
random_sleep() if ($extended);
$data_checksum_state = 'off';
}
else
{
# This should only happen due to programmer error when hacking on the
# test code, but since that might pass subtly we error out.
BAIL_OUT('data_checksum_state variable has invalid state:'
. $data_checksum_state);
}
}
# Create and start a cluster with one node
$node = PostgreSQL::Test::Cluster->new('pgbench_single_main');
$node->init(allows_streaming => 1, no_data_checksums => 1);
# max_connections need to be bumped in order to accommodate for pgbench clients
# and log_statement is dialled down since it otherwise will generate enormous
# amounts of logging. Page verification failures are still logged.
$node->append_conf(
'postgresql.conf',
qq[
max_connections = 100
log_statement = none
]);
$node->start;
$node->safe_psql('postgres', 'CREATE EXTENSION test_checksums;');
$node->safe_psql('postgres', 'CREATE EXTENSION injection_points;');
# Create some content to have un-checksummed data in the cluster
$node->safe_psql('postgres',
"CREATE TABLE t AS SELECT generate_series(1, 100000) AS a;");
# Initialize pgbench
my $scalefactor = ($extended ? 10 : 1);
$node->command_ok(
[
'pgbench', '-p', $node->port, '-i',
'-s', $scalefactor, '-q', 'postgres'
]);
# Start the test suite with pgbench running.
background_rw_pgbench($node->port);
# Main test suite. This loop will start a pgbench run on the cluster and while
# that's running flip the state of data checksums concurrently. It will then
# randomly restart the cluster and then check for
# the desired state. The idea behind doing things randomly is to stress out
# any timing related issues by subjecting the cluster for varied workloads.
# A TODO is to generate a trace such that any test failure can be traced to
# its order of operations for debugging.
for (my $i = 0; $i < $TEST_ITERATIONS; $i++)
{
note("iteration ", ($i + 1), " of ", $TEST_ITERATIONS);
if (!$node->is_alive)
{
# Start, to do recovery, and stop
$node->start;
$node->stop('fast');
# Since the log isn't being written to now, parse the log and check
# for instances of checksum verification failures.
my $log = PostgreSQL::Test::Utils::slurp_file($node->logfile,
$node_loglocation);
unlike(
$log,
qr/page verification failed,.+\d$/m,
"no checksum validation errors in primary log (during WAL recovery)"
);
$node_loglocation = -s $node->logfile;
# Randomize the WAL size, to trigger checkpoints less/more often
my $sb = 64 + int(rand(1024));
$node->append_conf('postgresql.conf', qq[max_wal_size = $sb]);
note("changing max_wal_size to " . $sb);
$node->start;
# Start a pgbench in the background against the primary
background_rw_pgbench($node->port);
}
$node->safe_psql('postgres', "UPDATE t SET a = a + 1;");
flip_data_checksums();
random_sleep() if ($extended);
my $result =
$node->safe_psql('postgres', "SELECT count(*) FROM t WHERE a > 1");
is($result, '100000', 'ensure data pages can be read back on primary');
random_sleep() if ($extended);
# Potentially powercycle the node
if (cointoss())
{
$node->stop(stopmode());
PostgreSQL::Test::Utils::system_log("pg_controldata",
$node->data_dir);
my $log = PostgreSQL::Test::Utils::slurp_file($node->logfile,
$node_loglocation);
unlike(
$log,
qr/page verification failed,.+\d$/m,
"no checksum validation errors in primary log (outside WAL recovery)"
);
$node_loglocation = -s $node->logfile;
}
random_sleep() if ($extended);
}
# Make sure the node is running
if (!$node->is_alive)
{
$node->start;
}
# Testrun is over, ensure that data reads back as expected and perform a final
# verification of the data checksum state.
my $result =
$node->safe_psql('postgres', "SELECT count(*) FROM t WHERE a > 1");
is($result, '100000', 'ensure data pages can be read back on primary');
test_checksum_state($node, $data_checksum_state);
# Perform one final pass over the logs and hunt for unexpected errors
my $log =
PostgreSQL::Test::Utils::slurp_file($node->logfile, $node_loglocation);
unlike(
$log,
qr/page verification failed,.+\d$/m,
"no checksum validation errors in primary log");
$node_loglocation = -s $node->logfile;
$node->teardown_node;
done_testing();

411
src/test/modules/test_checksums/t/007_pgbench_standby.pl
View File

@@ -1,411 +0,0 @@
# Copyright (c) 2026, PostgreSQL Global Development Group
# Test suite for testing enabling data checksums in an online cluster,
# comprising of a primary and a replicated standby, with concurrent activity
# via pgbench runs
use strict;
use warnings FATAL => 'all';
use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
use Test::More;
use FindBin;
use lib $FindBin::RealBin;
use DataChecksums::Utils;
# This test suite is expensive, or very expensive, to execute. There are two
# PG_TEST_EXTRA options for running it, "checksum" for a pared-down test suite
# an "checksum_extended" for the full suite. The full suite can run for hours
# on slow or constrained systems.
my $extended = undef;
if ($ENV{PG_TEST_EXTRA})
{
$extended = 1 if ($ENV{PG_TEST_EXTRA} =~ /\bchecksum_extended\b/);
plan skip_all => 'Expensive data checksums test disabled'
unless ($ENV{PG_TEST_EXTRA} =~ /\bchecksum(_extended)?\b/);
}
else
{
plan skip_all => 'Expensive data checksums test disabled';
}
if ($ENV{enable_injection_points} ne 'yes')
{
plan skip_all => 'Injection points not supported by this build';
}
my $node_primary_slot = 'physical_slot';
my $node_primary_backup = 'primary_backup';
my $node_primary;
my $node_primary_loglocation = 0;
my $node_standby;
my $node_standby_loglocation = 0;
# The number of full test iterations which will be performed. The exact number
# of tests performed and the wall time taken is non-deterministic as the test
# performs a lot of randomized actions, but 5 iterations will be a long test
# run regardless.
my $TEST_ITERATIONS = 1;
$TEST_ITERATIONS = 5 if ($extended);
# Variables which record the current state of the cluster
my $data_checksum_state = 'off';
my $pgbench_primary = undef;
my $pgbench_standby = undef;
# Start a pgbench run in the background against the server specified via the
# port passed as parameter
sub background_pgbench
{
my ($port, $standby) = @_;
my $pgbench = ($standby ? \$pgbench_standby : \$pgbench_primary);
# Terminate any currently running pgbench process before continuing
$$pgbench->finish if $$pgbench;
my $clients = 1;
my $runtime = 5;
if ($extended)
{
# Randomize the number of pgbench clients a bit (range 1-16)
$clients = 1 + int(rand(15));
$runtime = 600;
}
my @cmd = ('pgbench', '-p', $port, '-T', $runtime, '-c', $clients);
# Randomize whether we spawn connections or not
push(@cmd, '-C') if ($extended && cointoss());
# If we run on a standby it needs to be a read-only benchmark
push(@cmd, '-S') if ($standby);
push(@cmd, '-n') if ($standby);
# Finally add the database name to use
push(@cmd, 'postgres');
$$pgbench = IPC::Run::start(
\@cmd,
'<' => '/dev/null',
'>' => '/dev/null',
'2>' => '/dev/null',
IPC::Run::timer($PostgreSQL::Test::Utils::timeout_default));
}
# Invert the state of data checksums in the cluster, if data checksums are on
# then disable them and vice versa. Also performs proper validation of the
# before and after state.
sub flip_data_checksums
{
my $temptablewait = 0;
# First, make sure the cluster is in the state we expect it to be
test_checksum_state($node_primary, $data_checksum_state);
test_checksum_state($node_standby, $data_checksum_state);
if ($data_checksum_state eq 'off')
{
# Coin-toss to see if we are injecting a retry due to a temptable
if (cointoss())
{
$node_primary->safe_psql('postgres',
"SELECT injection_points_attach('datachecksumsworker-fake-temptable-wait', 'notice');"
);
$temptablewait = 1;
}
# log LSN right before we start changing checksums
my $result =
$node_primary->safe_psql('postgres', "SELECT pg_current_wal_lsn()");
note("LSN before enabling: " . $result . "\n");
# Ensure that the primary switches to "inprogress-on"
enable_data_checksums($node_primary, wait => 'inprogress-on');
random_sleep() if ($extended);
# Wait for checksum enable to be replayed
$node_primary->wait_for_catchup($node_standby, 'replay');
# Ensure that the standby has switched to "inprogress-on" or "on".
# Normally it would be "inprogress-on", but it is theoretically
# possible for the primary to complete the checksum enabling *and* have
# the standby replay that record before we reach the check below.
$result = $node_standby->poll_query_until(
'postgres',
"SELECT setting = 'off' "
. "FROM pg_catalog.pg_settings "
. "WHERE name = 'data_checksums';",
'f');
is($result, 1,
'ensure standby has absorbed the inprogress-on barrier');
$result = $node_standby->safe_psql('postgres',
"SELECT setting "
. "FROM pg_catalog.pg_settings "
. "WHERE name = 'data_checksums';");
is( ($result eq 'inprogress-on' || $result eq 'on'),
1,
'ensure checksums are on, or in progress, on standby_1, got: '
. $result);
# Wait for checksums enabled on the primary and standby
wait_for_checksum_state($node_primary, 'on');
# log LSN right after the primary flips checksums to "on"
$result =
$node_primary->safe_psql('postgres', "SELECT pg_current_wal_lsn()");
note("LSN after enabling: " . $result . "\n");
random_sleep() if ($extended);
wait_for_checksum_state($node_standby, 'on');
$node_primary->safe_psql('postgres',
"SELECT injection_points_detach('datachecksumsworker-fake-temptable-wait');"
) if ($temptablewait);
$data_checksum_state = 'on';
}
elsif ($data_checksum_state eq 'on')
{
random_sleep() if ($extended);
# log LSN right before we start changing checksums
my $result =
$node_primary->safe_psql('postgres', "SELECT pg_current_wal_lsn()");
note("LSN before disabling: " . $result . "\n");
disable_data_checksums($node_primary);
$node_primary->wait_for_catchup($node_standby, 'replay');
# Wait for checksums disabled on the primary and standby
random_sleep() if ($extended);
wait_for_checksum_state($node_primary, 'off');
wait_for_checksum_state($node_standby, 'off');
# log LSN right after the primary flips checksums to "off"
$result =
$node_primary->safe_psql('postgres', "SELECT pg_current_wal_lsn()");
note("LSN after disabling: " . $result . "\n");
$data_checksum_state = 'off';
}
else
{
# This should only happen due to programmer error when hacking on the
# test code, but since that might pass subtly we error out.
BAIL_OUT('data_checksum_state variable has invalid state:'
. $data_checksum_state);
}
}
# Create and start a cluster with one primary and one standby node, and ensure
# they are caught up and in sync.
$node_primary = PostgreSQL::Test::Cluster->new('pgbench_standby_main');
$node_primary->init(allows_streaming => 1, no_data_checksums => 1);
# max_connections need to be bumped in order to accommodate for pgbench clients
# and log_statement is dialled down since it otherwise will generate enormous
# amounts of logging. Page verification failures are still logged.
$node_primary->append_conf(
'postgresql.conf',
qq[
max_connections = 30
log_statement = none
hot_standby_feedback = on
]);
$node_primary->start;
$node_primary->safe_psql('postgres', 'CREATE EXTENSION test_checksums;');
$node_primary->safe_psql('postgres', 'CREATE EXTENSION injection_points;');
# Create some content to have un-checksummed data in the cluster
$node_primary->safe_psql('postgres',
"CREATE TABLE t AS SELECT generate_series(1, 100000) AS a;");
$node_primary->safe_psql('postgres',
"SELECT pg_create_physical_replication_slot('$node_primary_slot');");
$node_primary->backup($node_primary_backup);
$node_standby = PostgreSQL::Test::Cluster->new('pgbench_standby_standby');
$node_standby->init_from_backup($node_primary, $node_primary_backup,
has_streaming => 1);
$node_standby->append_conf(
'postgresql.conf', qq[
primary_slot_name = '$node_primary_slot'
]);
$node_standby->start;
# Initialize pgbench and wait for the objects to be created on the standby
my $scalefactor = ($extended ? 10 : 1);
$node_primary->command_ok(
[
'pgbench', '-p', $node_primary->port, '-i', '-s', $scalefactor, '-q',
'postgres'
]);
$node_primary->wait_for_catchup($node_standby, 'replay');
# Start the test suite with pgbench running on all nodes
background_pgbench($node_standby->port, 1);
background_pgbench($node_primary->port, 0);
# Main test suite. This loop will start a pgbench run on the cluster and while
# that's running flip the state of data checksums concurrently. It will then
# randomly restart the cluster and then check for
# the desired state. The idea behind doing things randomly is to stress out
# any timing related issues by subjecting the cluster for varied workloads.
# A TODO is to generate a trace such that any test failure can be traced to
# its order of operations for debugging.
for (my $i = 0; $i < $TEST_ITERATIONS; $i++)
{
note("iteration ", ($i + 1), " of ", $TEST_ITERATIONS);
if (!$node_primary->is_alive)
{
# start, to do recovery, and stop
$node_primary->start;
$node_primary->stop('fast');
# Since the log isn't being written to now, parse the log and check
# for instances of checksum verification failures.
my $log = PostgreSQL::Test::Utils::slurp_file($node_primary->logfile,
$node_primary_loglocation);
unlike(
$log,
qr/page verification failed,.+\d$/m,
"no checksum validation errors in primary log (during WAL recovery)"
);
$node_primary_loglocation = -s $node_primary->logfile;
# randomize the WAL size, to trigger checkpoints less/more often
my $sb = 32 + int(rand(960));
$node_primary->append_conf('postgresql.conf', qq[max_wal_size = $sb]);
note("changing primary max_wal_size to " . $sb);
$node_primary->start;
# Start a pgbench in the background against the primary
background_pgbench($node_primary->port, 0);
}
if (!$node_standby->is_alive)
{
$node_standby->start;
$node_standby->stop('fast');
# Since the log isn't being written to now, parse the log and check
# for instances of checksum verification failures.
my $log =
PostgreSQL::Test::Utils::slurp_file($node_standby->logfile,
$node_standby_loglocation);
unlike(
$log,
qr/page verification failed,.+\d$/m,
"no checksum validation errors in standby_1 log (during WAL recovery)"
);
$node_standby_loglocation = -s $node_standby->logfile;
# randomize the WAL size, to trigger checkpoints less/more often
my $sb = 32 + int(rand(960));
$node_standby->append_conf('postgresql.conf', qq[max_wal_size = $sb]);
note("changing standby max_wal_size to " . $sb);
$node_standby->start;
# Start a read-only pgbench in the background on the standby
background_pgbench($node_standby->port, 1);
}
$node_primary->safe_psql('postgres', "UPDATE t SET a = a + 1;");
$node_primary->wait_for_catchup($node_standby, 'write');
flip_data_checksums();
random_sleep() if ($extended);
my $result = $node_primary->safe_psql('postgres',
"SELECT count(*) FROM t WHERE a > 1");
is($result, '100000', 'ensure data pages can be read back on primary');
random_sleep();
# Potentially powercycle the cluster (the nodes independently). A TODO is
# to randomly stop the nodes in the opposite order too.
if ($extended && cointoss())
{
$node_primary->stop(stopmode());
# print the contents of the control file on the primary
PostgreSQL::Test::Utils::system_log("pg_controldata",
$node_primary->data_dir);
# slurp the file after shutdown, so that it doesn't interfere with the recovery
my $log = PostgreSQL::Test::Utils::slurp_file($node_primary->logfile,
$node_primary_loglocation);
unlike(
$log,
qr/page verification failed,.+\d$/m,
"no checksum validation errors in primary log (outside WAL recovery)"
);
$node_primary_loglocation = -s $node_primary->logfile;
}
random_sleep() if ($extended);
if ($extended && cointoss())
{
$node_standby->stop(stopmode());
# print the contents of the control file on the standby
PostgreSQL::Test::Utils::system_log("pg_controldata",
$node_standby->data_dir);
# slurp the file after shutdown, so that it doesn't interfere with the recovery
my $log =
PostgreSQL::Test::Utils::slurp_file($node_standby->logfile,
$node_standby_loglocation);
unlike(
$log,
qr/page verification failed,.+\d$/m,
"no checksum validation errors in standby_1 log (outside WAL recovery)"
);
$node_standby_loglocation = -s $node_standby->logfile;
}
}
# make sure the nodes are running
if (!$node_primary->is_alive)
{
$node_primary->start;
}
if (!$node_standby->is_alive)
{
$node_standby->start;
}
# Testrun is over, ensure that data reads back as expected and perform a final
# verification of the data checksum state.
my $result =
$node_primary->safe_psql('postgres', "SELECT count(*) FROM t WHERE a > 1");
is($result, '100000', 'ensure data pages can be read back on primary');
test_checksum_state($node_primary, $data_checksum_state);
test_checksum_state($node_standby, $data_checksum_state);
# Perform one final pass over the logs and hunt for unexpected errors
my $log = PostgreSQL::Test::Utils::slurp_file($node_primary->logfile,
$node_primary_loglocation);
unlike(
$log,
qr/page verification failed,.+\d$/m,
"no checksum validation errors in primary log");
$node_primary_loglocation = -s $node_primary->logfile;
$log = PostgreSQL::Test::Utils::slurp_file($node_standby->logfile,
$node_standby_loglocation);
unlike(
$log,
qr/page verification failed,.+\d$/m,
"no checksum validation errors in standby_1 log");
$node_standby_loglocation = -s $node_standby->logfile;
$node_standby->teardown_node;
$node_primary->teardown_node;
done_testing();

192
src/test/modules/test_checksums/t/008_pitr.pl
View File

@@ -1,192 +0,0 @@
# Copyright (c) 2026, PostgreSQL Global Development Group
use strict;
use warnings FATAL => 'all';
use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
use Test::More;
use FindBin;
use lib $FindBin::RealBin;
use DataChecksums::Utils;
# This test suite is expensive, or very expensive, to execute. There are two
# PG_TEST_EXTRA options for running it, "checksum" for a pared-down test suite
# an "checksum_extended" for the full suite.
my $extended = undef;
if ($ENV{PG_TEST_EXTRA})
{
$extended = 1 if ($ENV{PG_TEST_EXTRA} =~ /\bchecksum_extended\b/);
plan skip_all => 'Expensive data checksums test disabled'
unless ($ENV{PG_TEST_EXTRA} =~ /\bchecksum(_extended)?\b/);
}
else
{
plan skip_all => 'Expensive data checksums test disabled';
}
my $pgbench = undef;
my $data_checksum_state = 'off';
my $node_primary;
# Invert the state of data checksums in the cluster, if data checksums are on
# then disable them and vice versa. Also performs proper validation of the
# before and after state.
sub flip_data_checksums
{
my $lsn_pre = undef;
my $lsn_post = undef;
# First, make sure the cluster is in the state we expect it to be
test_checksum_state($node_primary, $data_checksum_state);
if ($data_checksum_state eq 'off')
{
# log LSN right before we start changing checksums
$lsn_pre =
$node_primary->safe_psql('postgres', "SELECT pg_current_wal_lsn()");
note("LSN before enabling: " . $lsn_pre . "\n");
# Wait for checksums enabled on the primary
enable_data_checksums($node_primary, wait => 'on');
# log LSN right after the primary flips checksums to "on"
$lsn_post =
$node_primary->safe_psql('postgres', "SELECT pg_current_wal_lsn()");
note("LSN after enabling: " . $lsn_post . "\n");
$data_checksum_state = 'on';
}
elsif ($data_checksum_state eq 'on')
{
# log LSN right before we start changing checksums
$lsn_pre =
$node_primary->safe_psql('postgres', "SELECT pg_current_wal_lsn()");
note("LSN before disabling: " . $lsn_pre . "\n");
# Disable checksums on the primary and wait for completion
disable_data_checksums($node_primary, wait => 1);
# log LSN right after the primary flips checksums to "off"
$lsn_post =
$node_primary->safe_psql('postgres', "SELECT pg_current_wal_lsn()");
note("LSN after disabling: " . $lsn_post . "\n");
$data_checksum_state = 'off';
}
else
{
# This should only happen due to programmer error when hacking on the
# test code, but since that might pass subtly we error out.
BAIL_OUT('data_checksum_state variable has invalid state:'
. $data_checksum_state);
}
return ($lsn_pre, $lsn_post);
}
# Start a pgbench run in the background against the server specified via the
# port passed as parameter.
sub background_rw_pgbench
{
my $port = shift;
# If a previous pgbench is still running, start by shutting it down.
$pgbench->finish if $pgbench;
# Randomize the number of pgbench clients in extended mode, else 1 client
my $clients = ($extended ? 1 + int(rand(15)) : 1);
my $runtime = ($extended ? 600 : 5);
my @cmd = ('pgbench', '-p', $port, '-T', $runtime, '-c', $clients);
# Randomize whether we spawn connections or not
push(@cmd, '-C') if ($extended && cointoss());
# Finally add the database name to use
push(@cmd, 'postgres');
$pgbench = IPC::Run::start(
\@cmd,
'<' => '/dev/null',
'>' => '/dev/null',
'2>' => '/dev/null',
IPC::Run::timer($PostgreSQL::Test::Utils::timeout_default));
}
# Start a primary node with WAL archiving enabled and with enough connections
# available to handle pgbench clients.
$node_primary = PostgreSQL::Test::Cluster->new('pitr_main');
$node_primary->init(
has_archiving => 1,
allows_streaming => 1,
no_data_checksums => 1);
my $timeout_unit = 's';
$node_primary->append_conf(
'postgresql.conf',
qq[
max_connections = 100
log_statement = none
wal_sender_timeout = $PostgreSQL::Test::Utils::timeout_default$timeout_unit
wal_receiver_timeout = $PostgreSQL::Test::Utils::timeout_default$timeout_unit
]);
$node_primary->start;
# Prime the cluster with a bit of known data which we can read back to check
# for data consistency as well as page verification faults in the logfile.
$node_primary->safe_psql('postgres',
'CREATE TABLE t AS SELECT generate_series(1, 100000) AS a;');
# Initialize and start pgbench in read/write mode against the cluster
my $scalefactor = ($extended ? 10 : 1);
$node_primary->command_ok(
[
'pgbench', '-p', $node_primary->port, '-i', '-s', $scalefactor, '-q',
'postgres'
]);
background_rw_pgbench($node_primary->port);
# Take a backup to use for PITR
my $backup_name = 'my_backup';
$node_primary->backup($backup_name);
my ($pre_lsn, $post_lsn) = flip_data_checksums();
$node_primary->safe_psql('postgres', "UPDATE t SET a = a + 1;");
$node_primary->safe_psql('postgres', "SELECT pg_create_restore_point('a');");
$node_primary->safe_psql('postgres', "UPDATE t SET a = a + 1;");
$node_primary->stop('fast');
my $node_pitr = PostgreSQL::Test::Cluster->new('pitr_backup');
$node_pitr->init_from_backup(
$node_primary, $backup_name,
standby => 0,
has_restoring => 1);
$node_pitr->append_conf(
'postgresql.conf', qq{
recovery_target_lsn = '$post_lsn'
recovery_target_action = 'promote'
recovery_target_inclusive = on
});
$node_pitr->start;
$node_pitr->poll_query_until('postgres', "SELECT pg_is_in_recovery() = 'f';")
or die "Timed out while waiting for PITR promotion";
test_checksum_state($node_pitr, $data_checksum_state);
my $result =
$node_pitr->safe_psql('postgres', "SELECT count(*) FROM t WHERE a > 1");
is($result, '99999', 'ensure data pages can be read back on primary');
$node_pitr->stop;
my $log = PostgreSQL::Test::Utils::slurp_file($node_pitr->logfile, 0);
unlike(
$log,
qr/page verification failed,.+\d$/m,
"no checksum validation errors in pitr log");
done_testing();

64
src/test/modules/test_checksums/t/009_fpi.pl
View File

@@ -1,64 +0,0 @@
# Copyright (c) 2026, PostgreSQL Global Development Group
use strict;
use warnings FATAL => 'all';
use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
use Test::More;
use FindBin;
use lib $FindBin::RealBin;
use DataChecksums::Utils;
# Create and start a cluster with one node
my $node = PostgreSQL::Test::Cluster->new('fpi_node');
$node->init(allows_streaming => 1, no_data_checksums => 1);
# max_connections need to be bumped in order to accommodate for pgbench clients
# and log_statement is dialled down since it otherwise will generate enormous
# amounts of logging. Page verification failures are still logged.
$node->append_conf(
'postgresql.conf',
qq[
max_connections = 100
log_statement = none
]);
$node->start;
$node->safe_psql('postgres', 'CREATE EXTENSION test_checksums;');
# Create some content to have un-checksummed data in the cluster
$node->safe_psql('postgres',
"CREATE TABLE t AS SELECT generate_series(1, 1000000) AS a;");
# Enable data checksums and wait for the state transition to 'on'
enable_data_checksums($node, wait => 'on');
$node->safe_psql('postgres', 'UPDATE t SET a = a + 1;');
disable_data_checksums($node, wait => 1);
$node->append_conf('postgresql.conf', 'full_page_writes = off');
$node->restart;
test_checksum_state($node, 'off');
$node->safe_psql('postgres', 'UPDATE t SET a = a + 1;');
$node->safe_psql('postgres', 'DELETE FROM t WHERE a < 10000;');
$node->adjust_conf('postgresql.conf', 'full_page_writes', 'on');
$node->restart;
test_checksum_state($node, 'off');
enable_data_checksums($node, wait => 'on');
my $result = $node->safe_psql('postgres', 'SELECT count(*) FROM t;');
is($result, '990003', 'Reading back all data from table t');
$node->stop;
my $log = PostgreSQL::Test::Utils::slurp_file($node->logfile, 0);
unlike(
$log,
qr/page verification failed,.+\d$/m,
"no checksum validation errors in server log");
done_testing();

260
src/test/modules/test_checksums/t/010_backup_straddle.pl
View File

@@ -1,260 +0,0 @@
# Copyright (c) 2026, PostgreSQL Global Development Group
# Test base backups running while the data checksum state changes. The
# transition to "on" happens before the checkpoint which flushes the rewritten
# pages, so a backup straddling it reads on-disk pages which legitimately lack
# checksums and carry LSNs older than the backup start. The same applies to
# checksums being disabled and re-enabled while a backup runs: hint bit
# updates made while checksums were off reach disk without a checksum update
# and without moving the page LSN, so once the re-enabling completes the
# backup would resume verification and misjudge those pages until the
# rewritten versions are flushed.
#
# Both scenarios hold the two sides with injection points: the backup after
# its starting checkpoint but before it sends any file data, and the enabling
# after the state changed to "on" but before the checkpoint which flushes the
# rewritten pages. The backup is thus guaranteed to read the stale on-disk
# pages after the state change, without any timing assumptions.
use strict;
use warnings FATAL => 'all';
use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
use File::Path qw(rmtree);
use Test::More;
use IPC::Run;
use FindBin;
use lib $FindBin::RealBin;
use DataChecksums::Utils;
# This test suite is potentially expensive due to it requiring an increased
# shared_buffers setting. It requires the "checksum" PG_TEST_EXTRA setting to
# not cause false positives on slow or constrained systems.
if ($ENV{PG_TEST_EXTRA})
{
plan skip_all => 'Expensive data checksums test disabled'
unless ($ENV{PG_TEST_EXTRA} =~ /\bchecksum(_extended)?\b/);
}
else
{
plan skip_all => 'Expensive data checksums test disabled';
}
if ($ENV{enable_injection_points} ne 'yes')
{
plan skip_all => 'Injection points not supported by this build';
}
my $node = PostgreSQL::Test::Cluster->new('backup_node');
$node->init(no_data_checksums => 1, allows_streaming => 1);
# The pages rewritten while enabling must stay dirty in shared buffers until
# the final checkpoint, otherwise they reach disk with checksums on their own
# and nothing is left to misjudge. The background writer must not flush them
# behind our back, and shared_buffers must exceed four times the table size
# so that the scan below does not go through a ring buffer. Autovacuum is
# disabled so that nothing sets hint bits behind our back, and wal_log_hints
# (implied by allows_streaming) must be off so that setting them does not
# move the page LSNs past the backup start.
$node->append_conf('postgresql.conf', 'shared_buffers = 32MB');
$node->append_conf('postgresql.conf', 'bgwriter_lru_maxpages = 0');
$node->append_conf('postgresql.conf', 'autovacuum = off');
$node->append_conf('postgresql.conf', 'wal_log_hints = off');
$node->start;
$node->safe_psql('postgres', 'CREATE EXTENSION injection_points;');
# A body of relation pages for the backup to misjudge. The scan pulls the
# table into shared buffers so that enabling doesn't read it through a ring
# buffer, which would write the pages back out.
$node->safe_psql('postgres',
"CREATE TABLE t AS SELECT generate_series(1,100000) AS a;");
$node->safe_psql('postgres', "SELECT count(*) FROM t;");
test_checksum_state($node, 'off');
$node->safe_psql('postgres',
"SELECT injection_points_attach('basebackup-before-send-files','wait');");
$node->safe_psql('postgres',
"SELECT injection_points_attach('datachecksums-on-before-checkpoint','wait');"
);
my $backupdir = $node->backup_dir . '/straddle';
my ($out, $err) = ('', '');
my $backup = IPC::Run::start(
[
'pg_basebackup', '-D',
$backupdir, '--wal-method=none',
'--no-sync', '--checkpoint=fast',
'-d', $node->connstr('postgres')
],
'>',
\$out,
'2>',
\$err,
IPC::Run::timeout(180));
$node->wait_for_event('walsender', 'basebackup-before-send-files');
# Enable checksums while the backup is held, then release the backup once the
# enabling has reached the "on" state and is held before its checkpoint.
enable_data_checksums($node);
$node->wait_for_event('datachecksums launcher',
'datachecksums-on-before-checkpoint');
$node->safe_psql('postgres',
"SELECT injection_points_wakeup('basebackup-before-send-files');");
$node->safe_psql('postgres',
"SELECT injection_points_detach('basebackup-before-send-files');");
ok($backup->finish, 'backup straddling enable completion succeeds')
or diag("stderr: $err");
# The backup must not even mention checksums: it must skip verification
# entirely, including the warning-only path for short reads.
unlike($err, qr/checksum/, 'straddling backup does not verify checksums');
$node->safe_psql('postgres',
"SELECT injection_points_wakeup('datachecksums-on-before-checkpoint');");
$node->safe_psql('postgres',
"SELECT injection_points_detach('datachecksums-on-before-checkpoint');");
wait_for_checksum_state($node, 'on');
$node->poll_query_until('postgres',
"SELECT count(*) = 0 FROM pg_catalog.pg_stat_activity "
. "WHERE backend_type = 'datachecksums launcher';");
# A backup started once enabling has completed must verify, and pass
$node->command_ok(
[
'pg_basebackup', '-D',
$node->backup_dir . '/after_enable', '--wal-method=none',
'--no-sync', '--checkpoint=fast'
],
'backup after enable completion succeeds');
rmtree($node->backup_dir . '/after_enable');
# Now test a backup which straddles checksums being disabled and re-enabled.
# Recreate the table since the earlier scan set its hint bits and the rewrite
# gave the pages checksums; the new contents are not read here, leaving the
# hint bits unset until checksums are off.
$node->safe_psql('postgres', "DROP TABLE t;");
$node->safe_psql('postgres',
"CREATE TABLE t AS SELECT generate_series(1,100000) AS a;");
$node->safe_psql('postgres',
"SELECT injection_points_attach('basebackup-before-send-files','wait');");
$backupdir = $node->backup_dir . '/onoffon';
($out, $err) = ('', '');
$backup = IPC::Run::start(
[
'pg_basebackup', '-D',
$backupdir, '--wal-method=none',
'--no-sync', '--checkpoint=fast',
'-d', $node->connstr('postgres')
],
'>',
\$out,
'2>',
\$err,
IPC::Run::timeout(180));
$node->wait_for_event('walsender', 'basebackup-before-send-files');
disable_data_checksums($node, wait => 1);
# With checksums off, the scan sets hint bits without WAL logging them, and
# the checkpoint flushes the modified pages without updating their checksums.
# The on-disk pages now carry stale checksums and LSNs older than the backup
# start.
$node->safe_psql('postgres', "SELECT count(*) FROM t;");
$node->safe_psql('postgres', "CHECKPOINT;");
$node->safe_psql('postgres',
"SELECT injection_points_attach('datachecksums-on-before-checkpoint','wait');"
);
enable_data_checksums($node);
$node->wait_for_event('datachecksums launcher',
'datachecksums-on-before-checkpoint');
$node->safe_psql('postgres',
"SELECT injection_points_wakeup('basebackup-before-send-files');");
$node->safe_psql('postgres',
"SELECT injection_points_detach('basebackup-before-send-files');");
ok($backup->finish, 'backup straddling disable and re-enable succeeds')
or diag("stderr: $err");
unlike($err, qr/checksum/,
'backup straddling disable and re-enable does not verify checksums');
$node->safe_psql('postgres',
"SELECT injection_points_wakeup('datachecksums-on-before-checkpoint');");
$node->safe_psql('postgres',
"SELECT injection_points_detach('datachecksums-on-before-checkpoint');");
wait_for_checksum_state($node, 'on');
$node->poll_query_until('postgres',
"SELECT count(*) = 0 FROM pg_catalog.pg_stat_activity "
. "WHERE backend_type = 'datachecksums launcher';");
rmtree($backupdir);
# A backup started once re-enabling has completed must verify, and pass
$node->command_ok(
[
'pg_basebackup', '-D',
$node->backup_dir . '/after_onoffon', '--wal-method=none',
'--no-sync', '--checkpoint=fast'
],
'backup after re-enable completion succeeds');
rmtree($node->backup_dir . '/after_onoffon');
# Test another backup, but this time inject synthetic checksum verification
# failures into it. The regex matching the WARNING is different from the next
# test on purpose as it will have NOTICE output from the injection point as
# well.
$node->safe_psql('postgres',
"SELECT injection_points_attach('basebackup-fail-checksum-verification', 'notice');"
);
$node->command_checks_all(
[
'pg_basebackup', '-D',
$node->backup_dir . '/corrupt', '--wal-method=none',
'--no-sync', '--checkpoint=fast'
],
1,
[qr{^$}],
[qr/WARNING.*checksum verification failed/s],
'pg_basebackup reports checksum mismatch');
rmtree($node->backup_dir . '/corrupt');
$node->safe_psql('postgres',
"SELECT injection_points_detach('basebackup-fail-checksum-verification');"
);
# Now corrupt data on disk and take another backup to make sure the corruption
# is reported.
my $corrupt_table = $node->safe_psql('postgres',
q{CREATE TABLE corrupt_table AS SELECT a FROM generate_series(1,10000) AS a; ALTER TABLE corrupt_table SET (autovacuum_enabled=false); SELECT pg_relation_filepath('corrupt_table')}
);
$node->stop;
$node->corrupt_page_checksum($corrupt_table, 0);
$node->start;
$node->command_checks_all(
[
'pg_basebackup', '-D',
$node->backup_dir . '/corrupt', '--wal-method=none',
'--no-sync', '--checkpoint=fast'
],
1,
[qr{^$}],
[qr/^WARNING.*checksum verification failed/s],
'pg_basebackup reports checksum mismatch');
rmtree($node->backup_dir . '/corrupt');
$node->stop;
done_testing();

249
src/test/modules/test_checksums/t/011_standby_straddle.pl
View File

@@ -1,249 +0,0 @@
# Copyright (c) 2026, PostgreSQL Global Development Group
# Test that a standby does not become available for base backups before it
# has re-replayed the latest data checksum state change after a restart.
#
# When a standby replays XLOG2_CHECKSUMS it writes the new state to
# pg_control, but the location of the change is only tracked in shared
# memory. If the standby restarts before a restartpoint covers the record,
# and reaches consistency below it, base backups would resume checksum
# verification with the change location unknown, while the pages rewritten
# before the change may not have reached disk. The redo routine must
# therefore advance minRecoveryPoint to the record, so that consistency (and
# with it hot standby and base backups) is withheld until the change location
# is known again.
#
# The test holds the enabling on the primary between the state change and its
# final checkpoint, waits for the standby to replay the state change, and
# crashes the standby so that the rewritten pages never reach its disk. WAL
# from the enabling onwards is removed from the standby's pg_wal and
# streaming is disabled, so that replay after the restart stalls below the
# state change record. Restarted this way, the standby must refuse
# connections; once streaming is re-enabled and the record replayed again,
# base backups must succeed without spurious checksum failures.
use strict;
use warnings FATAL => 'all';
use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
use File::Path qw(rmtree);
use Test::More;
use FindBin;
use lib $FindBin::RealBin;
use DataChecksums::Utils;
# This test suite is potentially expensive due to it requiring an increased
# shared_buffers setting and use custom timeouts. It requires the
# "checksum_extended" PG_TEST_EXTRA setting to not cause false positives on
# slow or constrained systems.
if ($ENV{PG_TEST_EXTRA})
{
plan skip_all => 'Expensive data checksums test disabled'
unless ($ENV{PG_TEST_EXTRA} =~ /\bchecksum_extended\b/);
}
else
{
plan skip_all => 'Expensive data checksums test disabled';
}
if ($ENV{enable_injection_points} ne 'yes')
{
plan skip_all => 'Injection points not supported by this build';
}
my $node_primary = PostgreSQL::Test::Cluster->new('straddle_primary');
$node_primary->init(no_data_checksums => 1, allows_streaming => 1);
# The pages rewritten while enabling must stay dirty in shared buffers until
# the final checkpoint, and their standby copies must stay dirty as well, so
# that the on-disk pages legitimately lack checksums. wal_log_hints (implied
# by allows_streaming) is turned off to keep the page LSNs put during setup.
$node_primary->append_conf('postgresql.conf', 'shared_buffers = 128MB');
$node_primary->append_conf('postgresql.conf', 'autovacuum = off');
$node_primary->append_conf('postgresql.conf', 'wal_log_hints = off');
$node_primary->start;
$node_primary->safe_psql('postgres', 'CREATE EXTENSION injection_points;');
my $slotname = 'physical_slot';
$node_primary->safe_psql('postgres',
"SELECT pg_create_physical_replication_slot('$slotname');");
# A body of relation pages whose standby copies will lack checksums
$node_primary->safe_psql('postgres',
"CREATE TABLE t AS SELECT generate_series(1,100000) AS a;");
my $backup_name = 'straddle_backup';
$node_primary->backup($backup_name);
my $node_standby = PostgreSQL::Test::Cluster->new('straddle_standby');
$node_standby->init_from_backup($node_primary, $backup_name,
has_streaming => 1);
# The background writer must not flush the replayed page rewrites behind our
# back, or nothing is left to protect and minRecoveryPoint could move on its
# own.
$node_standby->append_conf(
'postgresql.conf', qq[
primary_slot_name = '$slotname'
bgwriter_lru_maxpages = 0
]);
$node_standby->start;
$node_primary->wait_for_catchup($node_standby, 'replay',
$node_primary->lsn('insert'));
test_checksum_state($node_primary, 'off');
test_checksum_state($node_standby, 'off');
# Pin the restartpoint the later base backups will start from: replay a
# primary checkpoint and force a restartpoint on it. No further checkpoint
# record reaches the standby until the enabling is released, so this remains
# the standby's backup starting checkpoint throughout.
$node_primary->safe_psql('postgres', 'CHECKPOINT;');
$node_primary->wait_for_catchup($node_standby, 'replay',
$node_primary->lsn('insert'));
$node_standby->safe_psql('postgres', 'CHECKPOINT;');
# Put everything the enabling writes into fresh WAL segments, so that the
# standby's copies of them can be removed later, and remember where the
# enabling era begins.
$node_primary->safe_psql('postgres', 'SELECT pg_switch_wal();');
# Enable checksums, holding the launcher after the state change but before
# the final checkpoint, so the rewritten pages stay dirty everywhere.
$node_primary->safe_psql('postgres',
"SELECT injection_points_attach('datachecksums-enable-checksums-delay', 'wait');"
);
$node_primary->safe_psql('postgres',
"SELECT injection_points_attach('datachecksums-on-before-checkpoint','wait');"
);
enable_data_checksums($node_primary);
$node_primary->wait_for_event('datachecksums launcher',
'datachecksums-enable-checksums-delay');
my $enable_start_lsn =
$node_primary->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn();');
my $enable_start_seg = $node_primary->safe_psql('postgres',
"SELECT pg_walfile_name('$enable_start_lsn');");
$node_primary->safe_psql('postgres',
"SELECT injection_points_wakeup('datachecksums-enable-checksums-delay');"
);
# Immediately start to wait for the next event and hold off on detaching the
# previous injection point till later to avoid delays and risk missing the
# wait event
$node_primary->wait_for_event('datachecksums launcher',
'datachecksums-on-before-checkpoint');
# Detach the injection point now that we have some more time
$node_primary->safe_psql('postgres',
"SELECT injection_points_detach('datachecksums-enable-checksums-delay');"
);
# The standby has now replayed the state change: its pg_control says "on"
# while the rewritten pages are only dirty in its shared buffers.
$node_primary->wait_for_catchup($node_standby, 'replay',
$node_primary->lsn('insert'));
wait_for_checksum_state($node_standby, 'on');
# Crash the standby, losing the dirty rewritten pages.
$node_standby->stop('immediate');
# The state change record must have dragged minRecoveryPoint along with it,
# otherwise the standby can become consistent below the record after the
# restart.
my ($stdout, $stderr) =
run_command([ 'pg_controldata', $node_standby->data_dir ]);
my ($min_recovery) =
$stdout =~ /Minimum recovery ending location:\s*([0-9A-F]+\/[0-9A-F]+)/;
die "could not parse pg_controldata output" unless defined $min_recovery;
my $result = $node_primary->safe_psql('postgres',
"SELECT '$min_recovery'::pg_lsn > '$enable_start_lsn'::pg_lsn;");
is($result, 't', 'minRecoveryPoint advanced past the checksum state change');
# Remove the enabling-era WAL from the standby and cut it off from the
# primary, so that replay after the restart stalls below the state change.
# The replication slot retains the removed segments on the primary.
my $wal_dir = $node_standby->data_dir . '/pg_wal';
opendir(my $dh, $wal_dir) or die "could not open $wal_dir: $!";
foreach my $segment (readdir($dh))
{
next unless $segment =~ /^[0-9A-F]{24}$/;
next unless $segment ge $enable_start_seg;
unlink("$wal_dir/$segment")
or die "could not unlink $wal_dir/$segment: $!";
}
closedir($dh);
$node_standby->append_conf('postgresql.conf', "primary_conninfo = ''");
# The standby must not reach consistency until it has re-replayed the state
# change, so startup must not complete within the timeout. Reaching hot
# standby below the record would make pg_ctl return success here.
my $started;
{
local $ENV{PGCTLTIMEOUT} = 10;
$started = $node_standby->start(fail_ok => 1);
}
is($started, 0,
'standby withholds consistency until the state change is replayed again');
my ($ret, $out, $err) = $node_standby->psql('postgres', 'SELECT 1;');
isnt($ret, 0, 'standby refuses connections while below the state change');
# Reconnect the standby; streaming provides the removed WAL again, replay
# passes the state change and the standby becomes consistent.
$node_standby->enable_streaming($node_primary);
$node_standby->reload;
$node_standby->poll_query_until('postgres', 'SELECT true;');
$node_primary->wait_for_catchup($node_standby, 'replay',
$node_primary->lsn('insert'));
# The rewritten pages are again only dirty in shared buffers, so the on-disk
# pages still lack checksums. A base backup must skip verification entirely
# and pass without mentioning checksums on stderr. Standby backups always
# print a NOTICE about WAL archiving, so stderr is not empty.
$node_standby->command_checks_all(
[
'pg_basebackup', '-D',
$node_standby->backup_dir . '/underway', '--wal-method=none',
'--no-sync', '--checkpoint=fast'
],
0,
[qr{^$}],
[qr{^(?!.*checksum)}s],
'backup from standby while enabling is underway succeeds');
rmtree($node_standby->backup_dir . '/underway');
# Release the enabling; its final checkpoint flushes the rewritten pages.
$node_primary->safe_psql('postgres',
"SELECT injection_points_wakeup('datachecksums-on-before-checkpoint');");
$node_primary->safe_psql('postgres',
"SELECT injection_points_detach('datachecksums-on-before-checkpoint');");
wait_for_checksum_state($node_primary, 'on');
$node_primary->poll_query_until('postgres',
"SELECT count(*) = 0 FROM pg_catalog.pg_stat_activity "
. "WHERE backend_type = 'datachecksums launcher';");
# A restartpoint on the final checkpoint lets verification resume, and a
# backup started from it must again pass.
$node_primary->wait_for_catchup($node_standby, 'replay',
$node_primary->lsn('insert'));
$node_standby->safe_psql('postgres', 'CHECKPOINT;');
$node_standby->command_checks_all(
[
'pg_basebackup', '-D',
$node_standby->backup_dir . '/after_enable', '--wal-method=none',
'--no-sync', '--checkpoint=fast'
],
0,
[qr{^$}],
[qr{^(?!.*checksum)}s],
'backup from standby after enable completion succeeds');
rmtree($node_standby->backup_dir . '/after_enable');
$node_standby->stop;
$node_primary->stop;
done_testing();

243
src/test/modules/test_checksums/t/DataChecksums/Utils.pm
View File

@@ -1,243 +0,0 @@
# Copyright (c) 2026, PostgreSQL Global Development Group
=pod
=head1 NAME
DataChecksums::Utils - Utility functions for testing data checksums in a running cluster
=head1 SYNOPSIS
use PostgreSQL::Test::Cluster;
use DataChecksums::Utils qw( .. );
# Create, and start, a new cluster
my $node = PostgreSQL::Test::Cluster->new('primary');
$node->init;
$node->start;
test_checksum_state($node, 'off');
enable_data_checksums($node);
wait_for_checksum_state($node, 'on');
=cut
package DataChecksums::Utils;
use strict;
use warnings FATAL => 'all';
use Exporter 'import';
use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
use Test::More;
our @EXPORT = qw(
cointoss
disable_data_checksums
enable_data_checksums
random_sleep
stopmode
test_checksum_state
wait_for_checksum_state
);
=pod
=head1 METHODS
=over
=item test_checksum_state(node, state)
Test that the current value of the data checksum GUC in the server running
at B<node> matches B<state>. If the values differ, a test failure is logged.
Returns True if the values match, otherwise False.
=cut
sub test_checksum_state
{
my ($postgresnode, $state) = @_;
my $result = $postgresnode->safe_psql('postgres',
"SELECT setting FROM pg_catalog.pg_settings WHERE name = 'data_checksums';"
);
is($result, $state,
'ensure checksums are set to '
. $state . ' on '
. $postgresnode->name());
return $result eq $state;
}
=item wait_for_checksum_state(node, state)
Test the value of the data checksum GUC in the server running at B<node>
repeatedly until it matches B<state> or times out. Processing will run for
$PostgreSQL::Test::Utils::timeout_default seconds before timing out. If the
values differ when the process times out, False is returned and a test failure
is logged, otherwise True.
=cut
sub wait_for_checksum_state
{
my ($postgresnode, $state) = @_;
my $res = $postgresnode->poll_query_until(
'postgres',
"SELECT setting FROM pg_catalog.pg_settings WHERE name = 'data_checksums';",
$state);
is($res, 1,
'ensure data checksums are transitioned to '
. $state . ' on '
. $postgresnode->name());
return $res == 1;
}
=item enable_data_checksums($node, %params)
Function for enabling data checksums in the cluster running at B<node>.
=over
=item cost_delay
The B<cost_delay> to use when enabling data checksums, default is 0.
=item cost_limit
The B<cost_limit> to use when enabling data checksums, default is 100.
=item wait
If defined, the function will wait for the state defined in this parameter,
waiting timing out, before returning. The function will wait for
$PostgreSQL::Test::Utils::timeout_default seconds before timing out.
=back
=cut
sub enable_data_checksums
{
my $postgresnode = shift;
my %params = @_;
# Set sane defaults for the parameters
$params{cost_delay} = 0 unless (defined($params{cost_delay}));
$params{cost_limit} = 100 unless (defined($params{cost_limit}));
my $query = <<'EOQ';
SELECT pg_enable_data_checksums(%s, %s);
EOQ
$postgresnode->safe_psql('postgres',
sprintf($query, $params{cost_delay}, $params{cost_limit}));
if (defined($params{wait}))
{
wait_for_checksum_state($postgresnode, $params{wait});
# If we are tasked with waiting for an end state, also wait for the
# launcher to exit.
if ($params{wait} eq 'on' || $params{wait} eq 'off')
{
$postgresnode->poll_query_until('postgres',
"SELECT count(*) = 0 "
. "FROM pg_catalog.pg_stat_activity "
. "WHERE backend_type = 'datachecksums launcher';");
}
}
}
=item disable_data_checksums($node, %params)
Function for disabling data checksums in the cluster running at B<node>.
=over
=item wait
If defined, the function will wait for the state to turn to B<off>, or
waiting timing out, before returning. The function will wait for
$PostgreSQL::Test::Utils::timeout_default seconds before timing out.
Unlike in C<enable_data_checksums> the value of the parameter is discarded.
=back
=cut
sub disable_data_checksums
{
my $postgresnode = shift;
my %params = @_;
$postgresnode->safe_psql('postgres',
'SELECT pg_disable_data_checksums();');
if (defined($params{wait}))
{
wait_for_checksum_state($postgresnode, 'off');
$postgresnode->poll_query_until('postgres',
"SELECT count(*) = 0 "
. "FROM pg_catalog.pg_stat_activity "
. "WHERE backend_type = 'datachecksums launcher';");
}
}
=item cointoss
Helper for retrieving a binary value with random distribution for deciding
whether to turn things off during testing.
=back
=cut
sub cointoss
{
return int(rand() < 0.5);
}
=item random_sleep(max)
Helper for injecting random sleeps here and there in the testrun. The sleep
duration will be in the range (0,B<max>), but won't be predictable in order to
avoid sleep patterns that manage to avoid race conditions and timing bugs.
The default B<max> is 3 seconds.
=back
=cut
sub random_sleep
{
my $max = shift;
return if (defined($max) && ($max == 0));
sleep(int(rand(defined($max) ? $max : 3))) if cointoss;
}
=item stopmode
Small helper function for randomly selecting a valid stopmode.
=back
=cut
sub stopmode
{
return 'immediate' if (cointoss);
return 'fast';
}
=pod
=back
=cut
1;

16
src/test/modules/test_checksums/test_checksums--1.0.sql
View File

@@ -1,16 +0,0 @@
/* src/test/modules/test_checksums/test_checksums--1.0.sql */
-- complain if script is sourced in psql, rather than via CREATE EXTENSION
\echo Use "CREATE EXTENSION test_checksums" to load this file. \quit
CREATE FUNCTION dcw_inject_delay_barrier(attach boolean DEFAULT true)
RETURNS pg_catalog.void
AS 'MODULE_PATHNAME' LANGUAGE C;
CREATE FUNCTION dcw_inject_launcher_delay(attach boolean DEFAULT true)
RETURNS pg_catalog.void
AS 'MODULE_PATHNAME' LANGUAGE C;
CREATE FUNCTION dcw_inject_startup_delay(attach boolean DEFAULT true)
RETURNS pg_catalog.void
AS 'MODULE_PATHNAME' LANGUAGE C;

105
src/test/modules/test_checksums/test_checksums.c
View File

@@ -1,105 +0,0 @@
/*--------------------------------------------------------------------------
*
* test_checksums.c
* Test data checksums
*
* Copyright (c) 2026, PostgreSQL Global Development Group
*
* IDENTIFICATION
* src/test/modules/test_checksums/test_checksums.c
*
* -------------------------------------------------------------------------
*/
#include "postgres.h"
#include "funcapi.h"
#include "miscadmin.h"
#include "postmaster/datachecksum_state.h"
#include "storage/latch.h"
#include "utils/injection_point.h"
#include "utils/wait_event.h"
PG_MODULE_MAGIC;
extern PGDLLEXPORT void dc_delay_barrier(const char *name, const void *private_data, void *arg);
/*
* Test for delaying emission of procsignalbarriers.
*/
void
dc_delay_barrier(const char *name, const void *private_data, void *arg)
{
(void) name;
(void) private_data;
(void) WaitLatch(MyLatch,
WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH,
(3 * 1000),
WAIT_EVENT_PG_SLEEP);
}
PG_FUNCTION_INFO_V1(dcw_inject_delay_barrier);
Datum
dcw_inject_delay_barrier(PG_FUNCTION_ARGS)
{
#ifdef USE_INJECTION_POINTS
bool attach = PG_GETARG_BOOL(0);
if (attach)
InjectionPointAttach("datachecksums-enable-checksums-delay",
"test_checksums",
"dc_delay_barrier",
NULL,
0);
else
InjectionPointDetach("datachecksums-enable-checksums-delay");
#else
elog(ERROR,
"test is not working as intended when injection points are disabled");
#endif
PG_RETURN_VOID();
}
PG_FUNCTION_INFO_V1(dcw_inject_launcher_delay);
Datum
dcw_inject_launcher_delay(PG_FUNCTION_ARGS)
{
#ifdef USE_INJECTION_POINTS
bool attach = PG_GETARG_BOOL(0);
if (attach)
InjectionPointAttach("datachecksumsworker-launcher-delay",
"test_checksums",
"dc_delay_barrier",
NULL,
0);
else
InjectionPointDetach("datachecksumsworker-launcher-delay");
#else
elog(ERROR,
"test is not working as intended when injection points are disabled");
#endif
PG_RETURN_VOID();
}
PG_FUNCTION_INFO_V1(dcw_inject_startup_delay);
Datum
dcw_inject_startup_delay(PG_FUNCTION_ARGS)
{
#ifdef USE_INJECTION_POINTS
bool attach = PG_GETARG_BOOL(0);
if (attach)
InjectionPointAttach("datachecksumsworker-startup-delay",
"test_checksums",
"dc_delay_barrier",
NULL,
0);
else
InjectionPointDetach("datachecksumsworker-startup-delay");
#else
elog(ERROR,
"test is not working as intended when injection points are disabled");
#endif
PG_RETURN_VOID();
}

4
src/test/modules/test_checksums/test_checksums.control
View File

@@ -1,4 +0,0 @@
comment = 'Test code for data checksums'
default_version = '1.0'
module_pathname = '$libdir/test_checksums'
relocatable = true

38
src/test/regress/expected/rules.out
View File

@@ -2101,44 +2101,6 @@ pg_stat_progress_create_index| SELECT s.pid,
s.param15 AS partitions_done
FROM (pg_stat_get_progress_info('CREATE INDEX'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
LEFT JOIN pg_database d ON ((s.datid = d.oid)));
pg_stat_progress_data_checksums| SELECT s.pid,
s.datid,
d.datname,
CASE s.param1
WHEN 0 THEN 'enabling'::text
WHEN 1 THEN 'disabling'::text
WHEN 2 THEN 'waiting on temporary tables'::text
WHEN 3 THEN 'waiting on barrier'::text
WHEN 4 THEN 'done'::text
ELSE NULL::text
END AS phase,
CASE s.param2
WHEN '-1'::integer THEN NULL::bigint
ELSE s.param2
END AS databases_total,
CASE s.param3
WHEN '-1'::integer THEN NULL::bigint
ELSE s.param3
END AS databases_done,
CASE s.param4
WHEN '-1'::integer THEN NULL::bigint
ELSE s.param4
END AS relations_total,
CASE s.param5
WHEN '-1'::integer THEN NULL::bigint
ELSE s.param5
END AS relations_done,
CASE s.param6
WHEN '-1'::integer THEN NULL::bigint
ELSE s.param6
END AS blocks_total,
CASE s.param7
WHEN '-1'::integer THEN NULL::bigint
ELSE s.param7
END AS blocks_done
FROM (pg_stat_get_progress_info('DATACHECKSUMS'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
LEFT JOIN pg_database d ON ((s.datid = d.oid)))
ORDER BY s.datid;
pg_stat_progress_repack| SELECT s.pid,
s.datid,
d.datname,

11
src/test/regress/expected/stats.out
View File

@@ -51,15 +51,6 @@ client backend|relation|vacuum
client backend|temp relation|normal
client backend|wal|init
client backend|wal|normal
datachecksums launcher|relation|bulkread
datachecksums launcher|relation|normal
datachecksums launcher|wal|init
datachecksums launcher|wal|normal
datachecksums worker|relation|bulkread
datachecksums worker|relation|normal
datachecksums worker|relation|vacuum
datachecksums worker|wal|init
datachecksums worker|wal|normal
io worker|relation|bulkread
io worker|relation|bulkwrite
io worker|relation|init
@@ -104,7 +95,7 @@ walsummarizer|wal|init
walsummarizer|wal|normal
walwriter|wal|init
walwriter|wal|normal
(88 rows)
(79 rows)
\a
-- ensure that both seqscan and indexscan plans are allowed
SET enable_seqscan TO on;

6
src/tools/pgindent/typedefs.list
View File

@@ -433,7 +433,6 @@ CheckPointStmt
CheckpointStatsData
CheckpointerRequest
CheckpointerShmemStruct
ChecksumBarrierCondition
ChecksumStateType
Chromosome
CkptSortItem
@@ -627,10 +626,6 @@ DSMRegistryCtxStruct
DSMRegistryEntry
DWORD
DWORD64
DataChecksumsStateStruct
DataChecksumsWorkerDatabase
DataChecksumsWorkerOperation
DataChecksumsWorkerResult
DataDirSyncMethod
DataDumperPtr
DataPageDeleteStack
@@ -4449,7 +4444,6 @@ xl_btree_unlink_page
xl_btree_update
xl_btree_vacuum
xl_checkpoint_redo
xl_checksum_state
xl_clog_truncate
xl_commit_ts_truncate
xl_dbase_create_file_copy_rec