database/postgres
1
0
Fork 0
mirror of https://github.com/postgres/postgres.git synced 2025-07-28 23:42:10 +03:00
Code Activity

Add support for asynchronous execution.

Browse Source 
This implements asynchronous execution, which runs multiple parts of a
non-parallel-aware Append concurrently rather than serially to improve
performance when possible.  Currently, the only node type that can be
run concurrently is a ForeignScan that is an immediate child of such an
Append.  In the case where such ForeignScans access data on different
remote servers, this would run those ForeignScans concurrently, and
overlap the remote operations to be performed simultaneously, so it'll
improve the performance especially when the operations involve
time-consuming ones such as remote join and remote aggregation.

We may extend this to other node types such as joins or aggregates over
ForeignScans in the future.

This also adds the support for postgres_fdw, which is enabled by the
table-level/server-level option "async_capable".  The default is false.

Robert Haas, Kyotaro Horiguchi, Thomas Munro, and myself.  This commit
is mostly based on the patch proposed by Robert Haas, but also uses
stuff from the patch proposed by Kyotaro Horiguchi and from the patch
proposed by Thomas Munro.  Reviewed by Kyotaro Horiguchi, Konstantin
Knizhnik, Andrey Lepikhov, Movead Li, Thomas Munro, Justin Pryzby, and
others.

Discussion: https://postgr.es/m/CA%2BTgmoaXQEt4tZ03FtQhnzeDEMzBck%2BLrni0UWHVVgOTnA6C1w%40mail.gmail.com
Discussion: https://postgr.es/m/CA%2BhUKGLBRyu0rHrDCMC4%3DRn3252gogyp1SjOgG8SEKKZv%3DFwfQ%40mail.gmail.com
Discussion: https://postgr.es/m/20200228.170650.667613673625155850.horikyota.ntt%40gmail.com
This commit is contained in:
Etsuro Fujita
2021-03-31 18:45:00 +09:00
parent 66392d3965
commit 27e1f14563
39 changed files with 2068 additions and 57 deletions
Download Patch File Download Diff File Expand all files Collapse all files

14
doc/src/sgml/config.sgml
View File

@ -4787,6 +4787,20 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class="
</para>
<variablelist>
<varlistentry id="guc-enable-async-append" xreflabel="enable_async_append">
<term><varname>enable_async_append</varname> (<type>boolean</type>)
<indexterm>
<primary><varname>enable_async_append</varname> configuration parameter</primary>
</indexterm>
</term>
<listitem>
<para>
Enables or disables the query planner's use of async-aware
append plan types. The default is <literal>on</literal>.
</para>
</listitem>
</varlistentry>
<varlistentry id="guc-enable-bitmapscan" xreflabel="enable_bitmapscan">
<term><varname>enable_bitmapscan</varname> (<type>boolean</type>)
<indexterm>

90
doc/src/sgml/fdwhandler.sgml
View File

@ -1483,6 +1483,96 @@ ShutdownForeignScan(ForeignScanState *node);
</para>
</sect2>
<sect2 id="fdw-callbacks-async">
<title>FDW Routines for Asynchronous Execution</title>
<para>
A <structname>ForeignScan</structname> node can, optionally, support
asynchronous execution as described in
<filename>src/backend/executor/README</filename>. The following
functions are all optional, but are all required if asynchronous
execution is to be supported.
</para>
<para>
<programlisting>
bool
IsForeignPathAsyncCapable(ForeignPath *path);
</programlisting>
Test whether a given <structname>ForeignPath</structname> path can scan
the underlying foreign relation asynchronously.
This function will only be called at the end of query planning when the
given path is a direct child of an <structname>AppendPath</structname>
path and when the planner believes that asynchronous execution improves
performance, and should return true if the given path is able to scan the
foreign relation asynchronously.
</para>
<para>
If this function is not defined, it is assumed that the given path scans
the foreign relation using <function>IterateForeignScan</function>.
(This implies that the callback functions described below will never be
called, so they need not be provided either.)
</para>
<para>
<programlisting>
void
ForeignAsyncRequest(AsyncRequest *areq);
</programlisting>
Produce one tuple asynchronously from the
<structname>ForeignScan</structname> node. <literal>areq</literal> is
the <structname>AsyncRequest</structname> struct describing the
<structname>ForeignScan</structname> node and the parent
<structname>Append</structname> node that requested the tuple from it.
This function should store the tuple into the slot specified by
<literal>areq-&gt;result</literal>, and set
<literal>areq-&gt;request_complete</literal> to <literal>true</literal>;
or if it needs to wait on an event external to the core server such as
network I/O, and cannot produce any tuple immediately, set the flag to
<literal>false</literal>, and set
<literal>areq-&gt;callback_pending</literal> to <literal>true</literal>
for the <structname>ForeignScan</structname> node to get a callback from
the callback functions described below. If no more tuples are available,
set the slot to NULL, and the
<literal>areq-&gt;request_complete</literal> flag to
<literal>true</literal>. It's recommended to use
<function>ExecAsyncRequestDone</function> or
<function>ExecAsyncRequestPending</function> to set the output parameters
in the <literal>areq</literal>.
</para>
<para>
<programlisting>
void
ForeignAsyncConfigureWait(AsyncRequest *areq);
</programlisting>
Configure a file descriptor event for which the
<structname>ForeignScan</structname> node wishes to wait.
This function will only be called when the
<structname>ForeignScan</structname> node has the
<literal>areq-&gt;callback_pending</literal> flag set, and should add
the event to the <structfield>as_eventset</structfield> of the parent
<structname>Append</structname> node described by the
<literal>areq</literal>. See the comments for
<function>ExecAsyncConfigureWait</function> in
<filename>src/backend/executor/execAsync.c</filename> for additional
information. When the file descriptor event occurs,
<function>ForeignAsyncNotify</function> will be called.
</para>
<para>
<programlisting>
void
ForeignAsyncNotify(AsyncRequest *areq);
</programlisting>
Process a relevant event that has occurred, then produce one tuple
asynchronously from the <structname>ForeignScan</structname> node.
This function should set the output parameters in the
<literal>areq</literal> in the same way as
<function>ForeignAsyncRequest</function>.
</para>
</sect2>
<sect2 id="fdw-callbacks-reparameterize-paths">
<title>FDW Routines for Reparameterization of Paths</title>

5
doc/src/sgml/monitoring.sgml
View File

@ -1564,6 +1564,11 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
</thead>
<tbody>
<row>
<entry><literal>AppendReady</literal></entry>
<entry>Waiting for subplan nodes of an <literal>Append</literal> plan
node to be ready.</entry>
</row>
<row>
<entry><literal>BackupWaitWalArchive</literal></entry>
<entry>Waiting for WAL files required for a backup to be successfully

28
doc/src/sgml/postgres-fdw.sgml
View File

@ -371,6 +371,34 @@ OPTIONS (ADD password_required 'false');
</sect3>
<sect3>
<title>Asynchronous Execution Options</title>
<para>
<filename>postgres_fdw</filename> supports asynchronous execution, which
runs multiple parts of an <structname>Append</structname> node
concurrently rather than serially to improve performance.
This execution can be controled using the following option:
</para>
<variablelist>
<varlistentry>
<term><literal>async_capable</literal></term>
<listitem>
<para>
This option controls whether <filename>postgres_fdw</filename> allows
foreign tables to be scanned concurrently for asynchronous execution.
It can be specified for a foreign table or a foreign server.
A table-level option overrides a server-level option.
The default is <literal>false</literal>.
</para>
</listitem>
</varlistentry>
</variablelist>
</sect3>
<sect3>
<title>Updatability Options</title>