database/postgres
1
0
Fork 0
mirror of https://github.com/postgres/postgres.git synced 2026-10-29 15:58:33 +03:00
Code Activity

Add non-text output formats to pg_dumpall

Browse Source 
pg_dumpall can now produce output in custom, directory, or tar formats
in addition to plain text SQL scripts. When using non-text formats,
pg_dumpall creates a directory containing:
- toc.glo: global data (roles and tablespaces) in custom format
- map.dat: mapping between database OIDs and names
- databases/: subdirectory with per-database archives named by OID

pg_restore is extended to handle these pg_dumpall archives, restoring
globals and then each database. The --globals-only option can be used
to restore only the global objects.

This enables parallel restore of pg_dumpall output and selective
restoration of individual databases from a cluster-wide backup.

Author: Mahendra Singh Thalor <mahi6run@gmail.com>
Co-Author: Andrew Dunstan <andrew@dunslane.net>
Reviewed-By: Tushar Ahuja <tushar.ahuja@enterprisedb.com>
Reviewed-By: Jian He <jian.universality@gmail.com>
Reviewed-By: Vaibhav Dalvi <vaibhav.dalvi@enterprisedb.com>
Reviewed-By: Srinath Reddy <srinath2133@gmail.com>

Discussion: https://postgr.es/m/cb103623-8ee6-4ba5-a2c9-f32e3a4933fa@dunslane.net
This commit is contained in:
Andrew Dunstan
2026-02-23 15:08:55 -05:00
parent 7bb50dd7d6
commit 763aaa06f0
14 changed files with 2357 additions and 169 deletions
Download Patch File Download Diff File Expand all files Collapse all files

113
doc/src/sgml/ref/pg_dumpall.sgml
View File

@@ -16,7 +16,10 @@ PostgreSQL documentation
<refnamediv>
<refname>pg_dumpall</refname>
<refpurpose>extract a <productname>PostgreSQL</productname> database cluster into a script file</refpurpose>
<refpurpose>
export a <productname>PostgreSQL</productname> database cluster as an SQL script or to other formats
</refpurpose>
</refnamediv>
<refsynopsisdiv>
@@ -33,7 +36,7 @@ PostgreSQL documentation
<para>
<application>pg_dumpall</application> is a utility for writing out
(<quote>dumping</quote>) all <productname>PostgreSQL</productname> databases
of a cluster into one script file. The script file contains
of a cluster into an SQL script file or an archive. The output contains
<acronym>SQL</acronym> commands that can be used as input to <xref
linkend="app-psql"/> to restore the databases. It does this by
calling <xref linkend="app-pgdump"/> for each database in the cluster.
@@ -52,11 +55,16 @@ PostgreSQL documentation
</para>
<para>
The SQL script will be written to the standard output. Use the
Plain text SQL scripts will be written to the standard output. Use the
<option>-f</option>/<option>--file</option> option or shell operators to
redirect it into a file.
</para>
<para>
Archives in other formats will be placed in a directory named using the
<option>-f</option>/<option>--file</option>, which is required in this case.
</para>
<para>
<application>pg_dumpall</application> needs to connect several
times to the <productname>PostgreSQL</productname> server (once per
@@ -131,16 +139,93 @@ PostgreSQL documentation
<para>
Send output to the specified file. If this is omitted, the
standard output is used.
This option can only be omitted when <option>--format</option> is plain.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>-F <replaceable class="parameter">format</replaceable></option></term>
<term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
<listitem>
<para>
Specify the format of dump files. In plain format, all the dump data is
sent in a single text stream. This is the default.
In all other modes, <application>pg_dumpall</application> first creates two files,
<filename>toc.glo</filename> and <filename>map.dat</filename>, in the directory
specified by <option>--file</option>.
The first file contains global data (roles and tablespaces) in custom format. The second
contains a mapping between database OIDs and names. These files are used by
<application>pg_restore</application>. Data for individual databases is placed in
the <filename>databases</filename> subdirectory, named using the database's OID.
<variablelist>
<varlistentry>
<term><literal>d</literal></term>
<term><literal>directory</literal></term>
<listitem>
<para>
Output directory-format archives for each database,
suitable for input into pg_restore. The directory
will have database <type>oid</type> as its name.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><literal>p</literal></term>
<term><literal>plain</literal></term>
<listitem>
<para>
Output a plain-text SQL script file (the default).
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><literal>c</literal></term>
<term><literal>custom</literal></term>
<listitem>
<para>
Output a custom-format archive for each database,
suitable for input into pg_restore. The archive
will be named <filename>dboid.dmp</filename> where <type>dboid</type> is the
<type>oid</type> of the database.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><literal>t</literal></term>
<term><literal>tar</literal></term>
<listitem>
<para>
Output a tar-format archive for each database,
suitable for input into pg_restore. The archive
will be named <filename>dboid.tar</filename> where <type>dboid</type> is the
<type>oid</type> of the database.
</para>
</listitem>
</varlistentry>
</variablelist>
See <xref linkend="app-pgdump"/> for details on how the
various non-plain-text archive formats work.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>-g</option></term>
<term><option>--globals-only</option></term>
<listitem>
<para>
Dump only global objects (roles and tablespaces), no databases.
Note: <option>--globals-only</option> cannot be used with
<option>--clean</option> with non-text dump format.
</para>
</listitem>
</varlistentry>
@@ -936,13 +1021,21 @@ exclude database <replaceable class="parameter">PATTERN</replaceable>
<refsect1 id="app-pg-dumpall-ex">
<title>Examples</title>
<para>
To dump all databases:
To dump all databases in plain text format (the default):
<screen>
<prompt>$</prompt> <userinput>pg_dumpall &gt; db.out</userinput>
</screen>
</para>
<para>
To dump all databases using other formats:
<screen>
<prompt>$</prompt> <userinput>pg_dumpall --format=directory -f db.out</userinput>
<prompt>$</prompt> <userinput>pg_dumpall --format=custom -f db.out</userinput>
<prompt>$</prompt> <userinput>pg_dumpall --format=tar -f db.out</userinput>
</screen>
</para>
<para>
To restore database(s) from this file, you can use:
<screen>
@@ -956,6 +1049,16 @@ exclude database <replaceable class="parameter">PATTERN</replaceable>
the script will attempt to drop other databases immediately, and that
will fail for the database you are connected to.
</para>
<para>
If the dump was taken in a non-plain-text format, use
<application>pg_restore</application> to restore the databases:
<screen>
<prompt>$</prompt> <userinput>pg_restore db.out -d postgres -C</userinput>
</screen>
This will restore all databases. To restore only some databases, use
the <option>--exclude-database</option> option to skip those not wanted.
</para>
</refsect1>
<refsect1>

119
doc/src/sgml/ref/pg_restore.sgml
View File

@@ -18,8 +18,9 @@ PostgreSQL documentation
<refname>pg_restore</refname>
<refpurpose>
restore a <productname>PostgreSQL</productname> database from an
archive file created by <application>pg_dump</application>
restore <productname>PostgreSQL</productname> databases from archives
created by <application>pg_dump</application> or
<application>pg_dumpall</application>
</refpurpose>
</refnamediv>
@@ -38,13 +39,14 @@ PostgreSQL documentation
<para>
<application>pg_restore</application> is a utility for restoring a
<productname>PostgreSQL</productname> database from an archive
created by <xref linkend="app-pgdump"/> in one of the non-plain-text
<productname>PostgreSQL</productname> database or cluster from an archive
created by <xref linkend="app-pgdump"/> or
<xref linkend="app-pg-dumpall"/> in one of the non-plain-text
formats. It will issue the commands necessary to reconstruct the
database to the state it was in at the time it was saved. The
archive files also allow <application>pg_restore</application> to
database or cluster to the state it was in at the time it was saved. The
archives also allow <application>pg_restore</application> to
be selective about what is restored, or even to reorder the items
prior to being restored. The archive files are designed to be
prior to being restored. The archive formats are designed to be
portable across architectures.
</para>
@@ -52,14 +54,34 @@ PostgreSQL documentation
<application>pg_restore</application> can operate in two modes.
If a database name is specified, <application>pg_restore</application>
connects to that database and restores archive contents directly into
the database. Otherwise, a script containing the SQL
commands necessary to rebuild the database is created and written
the database.
When restoring from a dump made by <application>pg_dumpall</application>,
each database will be created and then the restoration will be run in that
database.
Otherwise, when a database name is not specified, a script containing the SQL
commands necessary to rebuild the database or cluster is created and written
to a file or standard output. This script output is equivalent to
the plain text output format of <application>pg_dump</application>.
the plain text output format of <application>pg_dump</application> or
<application>pg_dumpall</application>.
Some of the options controlling the output are therefore analogous to
<application>pg_dump</application> options.
</para>
<para>
A non-plain-text archive made using <application>pg_dumpall</application>
is a directory containing a <filename>toc.glo</filename> file with global
objects (roles and tablespaces), a <filename>map.dat</filename> file
listing the databases, and a subdirectory for each database containing
its archive. When restoring such an archive,
<application>pg_restore</application> first restores global objects from
<filename>toc.glo</filename>, then processes each database listed in
<filename>map.dat</filename>. Lines in <filename>map.dat</filename> can
be commented out with <literal>#</literal> to skip restoring specific
databases.
</para>
<para>
Obviously, <application>pg_restore</application> cannot restore information
that is not present in the archive file. For instance, if the
@@ -130,6 +152,12 @@ PostgreSQL documentation
ignorable error messages will be reported,
unless <option>--if-exists</option> is also specified.
</para>
<para>
When restoring a <application>pg_dumpall</application> archive,
<option>--if-exists</option> is implied by <option>--clean</option>,
since global objects such as roles and tablespaces may not exist
in the target cluster.
</para>
</listitem>
</varlistentry>
@@ -152,6 +180,8 @@ PostgreSQL documentation
commands that mention this database.
Access privileges for the database itself are also restored,
unless <option>--no-acl</option> is specified.
<option>--create</option> is required when restoring multiple databases
from a non-plain-text archive made using <application>pg_dumpall</application>.
</para>
<para>
@@ -247,6 +277,28 @@ PostgreSQL documentation
</listitem>
</varlistentry>
<varlistentry>
<term><option>-g</option></term>
<term><option>--globals-only</option></term>
<listitem>
<para>
Restore only global objects (roles and tablespaces), no databases.
</para>
<para>
This option is only relevant when restoring from a non-plain-text archive made using <application>pg_dumpall</application>.
Note: <option>--globals-only</option> cannot be used with
<option>--data-only</option>,
<option>--schema-only</option>,
<option>--statistics-only</option>,
<option>--statistics</option>,
<option>--exit-on-error</option>,
<option>--single-transaction</option>,
<option>--clean</option>, or
<option>--transaction-size</option>.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>-I <replaceable class="parameter">index</replaceable></option></term>
<term><option>--index=<replaceable class="parameter">index</replaceable></option></term>
@@ -581,6 +633,28 @@ PostgreSQL documentation
</listitem>
</varlistentry>
<varlistentry>
<term><option>--exclude-database=<replaceable class="parameter">pattern</replaceable></option></term>
<listitem>
<para>
Do not restore databases whose name matches
<replaceable class="parameter">pattern</replaceable>.
Multiple patterns can be excluded by writing multiple
<option>--exclude-database</option> switches. The
<replaceable class="parameter">pattern</replaceable> parameter is
interpreted as a pattern according to the same rules used by
<application>psql</application>'s <literal>\d</literal>
commands (see <xref linkend="app-psql-patterns"/>),
so multiple databases can also be excluded by writing wildcard
characters in the pattern. When using wildcards, be careful to
quote the pattern if needed to prevent shell wildcard expansion.
</para>
<para>
This option is only relevant when restoring from a non-plain-text archive made using <application>pg_dumpall</application>.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
<listitem>
@@ -669,7 +743,9 @@ PostgreSQL documentation
in <option>--clean</option> mode. This suppresses <quote>does not
exist</quote> errors that might otherwise be reported. This
option is not valid unless <option>--clean</option> is also
specified.
specified. This option is implied when restoring a
<application>pg_dumpall</application> archive with
<option>--clean</option>.
</para>
</listitem>
</varlistentry>
@@ -1125,6 +1201,27 @@ CREATE DATABASE foo WITH TEMPLATE template0;
</para>
</listitem>
<listitem>
<para>
When restoring from a non-plain-text archive made using
<application>pg_dumpall</application>, the <option>--section</option>
option may be used, but must include <option>pre-data</option>.
</para>
</listitem>
<listitem>
<para>
The following options cannot be used when restoring from a non-plain-text
archive made using <application>pg_dumpall</application>:
<option>-a/--data-only</option>,
<option>-l/--list</option>,
<option>-L/--use-list</option>,
<option>--no-schema</option>,
<option>--statistics-only</option>, and
<option>--strict-names</option>.
</para>
</listitem>
</itemizedlist>
</para>