database/postgres
1
0
Fork 0
mirror of https://github.com/postgres/postgres.git synced 2025-07-30 11:03:19 +03:00
Code Activity

Adjust batch size in postgres_fdw to not use too many parameters

Browse Source 
The FE/BE protocol identifies parameters with an Int16 index, which
limits the maximum number of parameters per query to 65535. With
batching added to postges_fdw this limit is much easier to hit, as
the whole batch is essentially a single query, making this error much
easier to hit.

The failures are a bit unpredictable, because it also depends on the
number of columns in the query. So instead of just failing, this patch
tweaks the batch_size to not exceed the maximum number of parameters.

Reported-by: Hou Zhijie <houzj.fnst@cn.fujitsu.com>
Reviewed-by: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Discussion: https://postgr.es/m/OS0PR01MB571603973C0AC2874AD6BF2594299%40OS0PR01MB5716.jpnprd01.prod.outlook.com
This commit is contained in:
Tomas Vondra
2021-06-08 20:22:18 +02:00
parent d1f0aa7696
commit cb92703384
6 changed files with 52 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files

11
doc/src/sgml/postgres-fdw.sgml
View File

@ -372,6 +372,17 @@ OPTIONS (ADD password_required 'false');
overrides an option specified for the server.
The default is <literal>1</literal>.
</para>
<para>
Note the actual number of rows <filename>postgres_fdw</filename> inserts at
once depends on the number of columns and the provided
<literal>batch_size</literal> value. The batch is executed as a single
query, and the libpq protocol (which <filename>postgres_fdw</filename>
uses to connect to a remote server) limits the number of parameters in a
single query to 65535. When the number of columns * <literal>batch_size</literal>
exceeds the limit, the <literal>batch_size</literal> will be adjusted to
avoid an error.
</para>
</listitem>
</varlistentry>