1
0
mirror of https://github.com/postgres/postgres.git synced 2025-07-27 12:41:57 +03:00

Fix serious performance problems in json(b) to_tsvector().

In an off-list followup to bug #14745, Bob Jones complained that
to_tsvector() on a 2MB jsonb value took an unreasonable amount of
time and space --- enough to draw the wrath of the OOM killer on
his machine.  On my machine, his example proved to require upwards
of 18 seconds and 4GB, which seemed pretty bogus considering that
to_tsvector() on the same data treated as text took just a couple
hundred msec and 10 or so MB.

On investigation, the problem is that the implementation scans each
string element of the json(b) and converts it to tsvector separately,
then applies tsvector_concat() to join those separate tsvectors.
The unreasonable memory usage came from leaking every single one of
the transient tsvectors --- but even without that mistake, this is an
O(N^2) or worse algorithm, because tsvector_concat() has to repeatedly
process the words coming from earlier elements.

We can fix it by accumulating all the lexeme data and applying
make_tsvector() just once.  As a side benefit, that also makes the
desired adjustment of lexeme positions far cheaper, because we can
just tweak the running "pos" counter between JSON elements.

In passing, try to make the explanation of that tweak more intelligible.
(I didn't think that a barely-readable comment far removed from the
actual code was helpful.)  And do some minor other code beautification.
This commit is contained in:
Tom Lane
2017-07-18 12:45:51 -04:00
parent fb9bd4b046
commit b4c6d31c0b
2 changed files with 61 additions and 74 deletions

View File

@ -86,15 +86,6 @@ typedef struct
#define MAXNUMPOS (256)
#define LIMITPOS(x) ( ( (x) >= MAXENTRYPOS ) ? (MAXENTRYPOS-1) : (x) )
/*
* In case if a TSVector contains several parts and we want to treat them as
* separate, it's necessary to add an artificial increment to position of each
* lexeme from every next part. It's required to avoid the situation when
* tsquery can find a phrase consisting of lexemes from two of such parts.
* TS_JUMP defined a value of this increment.
*/
#define TS_JUMP 1
/* This struct represents a complete tsvector datum */
typedef struct
{