database/postgres
1
0
Fork 0
mirror of https://github.com/postgres/postgres.git synced 2025-07-27 12:41:57 +03:00
Code Activity

Introduce framework for parallelizing various pg_upgrade tasks.

Browse Source 
A number of pg_upgrade steps require connecting to every database
in the cluster and running the same query in each one.  When there
are many databases, these steps are particularly time-consuming,
especially since they are performed sequentially, i.e., we connect
to a database, run the query, and process the results before moving
on to the next database.

This commit introduces a new framework that makes it easy to
parallelize most of these once-in-each-database tasks by processing
multiple databases concurrently.  This framework manages a set of
slots that follow a simple state machine, and it uses libpq's
asynchronous APIs to establish the connections and run the queries.
The --jobs option is used to determine the number of slots to use.
To use this new task framework, callers simply need to provide the
query and a callback function to process its results, and the
framework takes care of the rest.  A more complete description is
provided at the top of the new task.c file.

None of the eligible once-in-each-database tasks are converted to
use this new framework in this commit.  That will be done via
several follow-up commits.

Reviewed-by: Jeff Davis, Robert Haas, Daniel Gustafsson, Ilya Gladyshev, Corey Huinker
Discussion: https://postgr.es/m/20240516211638.GA1688936%40nathanxps13
This commit is contained in:
Nathan Bossart
2024-09-16 16:10:33 -05:00
parent d891c49286
commit 40e2e5e92b
6 changed files with 474 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
doc/src/sgml/ref/pgupgrade.sgml
View File

@ -118,7 +118,7 @@ PostgreSQL documentation
<varlistentry>
<term><option>-j <replaceable class="parameter">njobs</replaceable></option></term>
<term><option>--jobs=<replaceable class="parameter">njobs</replaceable></option></term>
<listitem><para>number of simultaneous processes or threads to use
<listitem><para>number of simultaneous connections and processes/threads to use
</para></listitem>
</varlistentry>
@ -587,8 +587,8 @@ NET STOP postgresql-&majorversion;
<para>
The <option>--jobs</option> option allows multiple CPU cores to be used
for copying/linking of files and to dump and restore database schemas
in parallel; a good place to start is the maximum of the number of
for copying/linking of files, dumping and restoring database schemas
in parallel, etc.; a good place to start is the maximum of the number of
CPU cores and tablespaces. This option can dramatically reduce the
time to upgrade a multi-database server running on a multiprocessor
machine.