mirror of
https://github.com/postgres/postgres.git
synced 2025-12-19 17:02:53 +03:00
87b8db3774
method to pass extra data to the consistent() and comparePartial() methods. This is the core infrastructure needed to support the soon-to-appear contrib/btree_gin module. The APIs are still upward compatible with the definitions used in 8.3 and before, although *not* with the previous 8.4devel function definitions. catversion bump for changes in pg_proc entries (although these are just cosmetic, since GIN doesn't actually look at the function signature before calling it...) Teodor Sigaev and Oleg Bartunov
435 lines
17 KiB
Plaintext
435 lines
17 KiB
Plaintext
<!-- $PostgreSQL: pgsql/doc/src/sgml/gin.sgml,v 2.18 2009/03/25 22:19:01 tgl Exp $ -->
|
|
|
|
<chapter id="GIN">
|
|
<title>GIN Indexes</title>
|
|
|
|
<indexterm>
|
|
<primary>index</primary>
|
|
<secondary>GIN</secondary>
|
|
</indexterm>
|
|
|
|
<sect1 id="gin-intro">
|
|
<title>Introduction</title>
|
|
|
|
<para>
|
|
<acronym>GIN</acronym> stands for Generalized Inverted Index. It is
|
|
an index structure storing a set of (key, posting list) pairs, where
|
|
a <quote>posting list</> is a set of rows in which the key occurs. Each
|
|
indexed value can contain many keys, so the same row ID can appear in
|
|
multiple posting lists.
|
|
</para>
|
|
|
|
<para>
|
|
It is generalized in the sense that a <acronym>GIN</acronym> index
|
|
does not need to be aware of the operation that it accelerates.
|
|
Instead, it uses custom strategies defined for particular data types.
|
|
</para>
|
|
|
|
<para>
|
|
One advantage of <acronym>GIN</acronym> is that it allows the development
|
|
of custom data types with the appropriate access methods, by
|
|
an expert in the domain of the data type, rather than a database expert.
|
|
This is much the same advantage as using <acronym>GiST</acronym>.
|
|
</para>
|
|
|
|
<para>
|
|
The <acronym>GIN</acronym>
|
|
implementation in <productname>PostgreSQL</productname> is primarily
|
|
maintained by Teodor Sigaev and Oleg Bartunov. There is more
|
|
information about <acronym>GIN</acronym> on their
|
|
<ulink url="http://www.sai.msu.su/~megera/wiki/Gin">website</ulink>.
|
|
</para>
|
|
</sect1>
|
|
|
|
<sect1 id="gin-extensibility">
|
|
<title>Extensibility</title>
|
|
|
|
<para>
|
|
The <acronym>GIN</acronym> interface has a high level of abstraction,
|
|
requiring the access method implementer only to implement the semantics of
|
|
the data type being accessed. The <acronym>GIN</acronym> layer itself
|
|
takes care of concurrency, logging and searching the tree structure.
|
|
</para>
|
|
|
|
<para>
|
|
All it takes to get a <acronym>GIN</acronym> access method working is to
|
|
implement four (or five) user-defined methods, which define the behavior of
|
|
keys in the tree and the relationships between keys, indexed values,
|
|
and indexable queries. In short, <acronym>GIN</acronym> combines
|
|
extensibility with generality, code reuse, and a clean interface.
|
|
</para>
|
|
|
|
<para>
|
|
The four methods that an operator class for
|
|
<acronym>GIN</acronym> must provide are:
|
|
</para>
|
|
|
|
<variablelist>
|
|
<varlistentry>
|
|
<term>int compare(Datum a, Datum b)</term>
|
|
<listitem>
|
|
<para>
|
|
Compares keys (not indexed values!) and returns an integer less than
|
|
zero, zero, or greater than zero, indicating whether the first key is
|
|
less than, equal to, or greater than the second.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term>Datum *extractValue(Datum inputValue, int32 *nkeys)</term>
|
|
<listitem>
|
|
<para>
|
|
Returns an array of keys given a value to be indexed. The
|
|
number of returned keys must be stored into <literal>*nkeys</>.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term>Datum *extractQuery(Datum query, int32 *nkeys,
|
|
StrategyNumber n, bool **pmatch, Pointer **extra_data)</term>
|
|
<listitem>
|
|
<para>
|
|
Returns an array of keys given a value to be queried; that is,
|
|
<literal>query</> is the value on the right-hand side of an
|
|
indexable operator whose left-hand side is the indexed column.
|
|
<literal>n</> is the strategy number of the operator within the
|
|
operator class (see <xref linkend="xindex-strategies">).
|
|
Often, <function>extractQuery</> will need
|
|
to consult <literal>n</> to determine the data type of
|
|
<literal>query</> and the key values that need to be extracted.
|
|
The number of returned keys must be stored into <literal>*nkeys</>.
|
|
If the query contains no keys then <function>extractQuery</>
|
|
should store 0 or -1 into <literal>*nkeys</>, depending on the
|
|
semantics of the operator. 0 means that every
|
|
value matches the <literal>query</> and a sequential scan should be
|
|
performed. -1 means nothing can match the <literal>query</>.
|
|
<literal>pmatch</> is an output argument for use when partial match
|
|
is supported. To use it, <function>extractQuery</> must allocate
|
|
an array of <literal>*nkeys</> booleans and store its address at
|
|
<literal>*pmatch</>. Each element of the array should be set to TRUE
|
|
if the corresponding key requires partial match, FALSE if not.
|
|
If <literal>*pmatch</> is set to NULL then GIN assumes partial match
|
|
is not required. The variable is initialized to NULL before call,
|
|
so this argument can simply be ignored by operator classes that do
|
|
not support partial match.
|
|
<literal>extra_data</> is an output argument that allows
|
|
<function>extractQuery</> to pass additional data to the
|
|
<function>consistent</> and <function>comparePartial</> methods.
|
|
To use it, <function>extractQuery</> must allocate
|
|
an array of <literal>*nkeys</> Pointers and store its address at
|
|
<literal>*extra_data</>, then store whatever it wants to into the
|
|
individual pointers. The variable is initialized to NULL before
|
|
call, so this argument can simply be ignored by operator classes that
|
|
do not require extra data. If <literal>*extra_data</> is set, the
|
|
whole array is passed to the <function>consistent</> method, and
|
|
the appropriate element to the <function>comparePartial</> method.
|
|
</para>
|
|
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term>bool consistent(bool check[], StrategyNumber n, Datum query,
|
|
int32 nkeys, Pointer extra_data[], bool *recheck)</term>
|
|
<listitem>
|
|
<para>
|
|
Returns TRUE if the indexed value satisfies the query operator with
|
|
strategy number <literal>n</> (or might satisfy, if the recheck
|
|
indication is returned). The <literal>check</> array has length
|
|
<literal>nkeys</>, which is the same as the number of keys previously
|
|
returned by <function>extractQuery</> for this <literal>query</> datum.
|
|
Each element of the
|
|
<literal>check</> array is TRUE if the indexed value contains the
|
|
corresponding query key, ie, if (check[i] == TRUE) the i-th key of the
|
|
<function>extractQuery</> result array is present in the indexed value.
|
|
The original <literal>query</> datum (not the extracted key array!) is
|
|
passed in case the <function>consistent</> method needs to consult it.
|
|
<literal>extra_data</> is the extra-data array returned by
|
|
<function>extractQuery</>, or NULL if none.
|
|
On success, <literal>*recheck</> should be set to TRUE if the heap
|
|
tuple needs to be rechecked against the query operator, or FALSE if
|
|
the index test is exact.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
</variablelist>
|
|
|
|
<para>
|
|
Optionally, an operator class for
|
|
<acronym>GIN</acronym> can supply a fifth method:
|
|
</para>
|
|
|
|
<variablelist>
|
|
|
|
<varlistentry>
|
|
<term>int comparePartial(Datum partial_key, Datum key, StrategyNumber n,
|
|
Pointer extra_data)</term>
|
|
<listitem>
|
|
<para>
|
|
Compare a partial-match query to an index key. Returns an integer
|
|
whose sign indicates the result: less than zero means the index key
|
|
does not match the query, but the index scan should continue; zero
|
|
means that the index key does match the query; greater than zero
|
|
indicates that the index scan should stop because no more matches
|
|
are possible. The strategy number <literal>n</> of the operator
|
|
that generated the partial match query is provided, in case its
|
|
semantics are needed to determine when to end the scan. Also,
|
|
<literal>extra_data</> is the corresponding element of the extra-data
|
|
array made by <function>extractQuery</>, or NULL if none.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
</variablelist>
|
|
|
|
<para>
|
|
To support <quote>partial match</> queries, an operator class must
|
|
provide the <function>comparePartial</> method, and its
|
|
<function>extractQuery</> method must set the <literal>pmatch</>
|
|
parameter when a partial-match query is encountered. See
|
|
<xref linkend="gin-partial-match"> for details.
|
|
</para>
|
|
|
|
</sect1>
|
|
|
|
<sect1 id="gin-implementation">
|
|
<title>Implementation</title>
|
|
|
|
<para>
|
|
Internally, a <acronym>GIN</acronym> index contains a B-tree index
|
|
constructed over keys, where each key is an element of the indexed value
|
|
(a member of an array, for example) and where each tuple in a leaf page is
|
|
either a pointer to a B-tree over heap pointers (PT, posting tree), or a
|
|
list of heap pointers (PL, posting list) if the list is small enough.
|
|
</para>
|
|
|
|
<sect2 id="gin-fast-update">
|
|
<title>GIN fast update technique</title>
|
|
|
|
<para>
|
|
Updating a <acronym>GIN</acronym> index tends to be slow because of the
|
|
intrinsic nature of inverted indexes: inserting or updating one heap row
|
|
can cause many inserts into the index (one for each key extracted
|
|
from the indexed value). As of <productname>PostgreSQL</productname> 8.4,
|
|
<acronym>GIN</> is capable of postponing much of this work by inserting
|
|
new tuples into a temporary, unsorted list of pending entries.
|
|
When the table is vacuumed, or if the pending list becomes too large
|
|
(larger than <xref linkend="guc-work-mem">), the entries are moved to the
|
|
main <acronym>GIN</acronym> data structure using the same bulk insert
|
|
techniques used during initial index creation. This greatly improves
|
|
<acronym>GIN</acronym> index update speed, even counting the additional
|
|
vacuum overhead. Moreover the overhead can be done by a background
|
|
process instead of in foreground query processing.
|
|
</para>
|
|
|
|
<para>
|
|
The main disadvantage of this approach is that searches must scan the list
|
|
of pending entries in addition to searching the regular index, and so
|
|
a large list of pending entries will slow searches significantly.
|
|
Another disadvantage is that, while most updates are fast, an update
|
|
that causes the pending list to become <quote>too large</> will incur an
|
|
immediate cleanup cycle and thus be much slower than other updates.
|
|
Proper use of autovacuum can minimize both of these problems.
|
|
</para>
|
|
|
|
<para>
|
|
If consistent response time is more important than update speed,
|
|
use of pending entries can be disabled by turning off the
|
|
<literal>FASTUPDATE</literal> storage parameter for a
|
|
<acronym>GIN</acronym> index. See <xref linkend="sql-createindex"
|
|
endterm="sql-createindex-title"> for details.
|
|
</para>
|
|
</sect2>
|
|
|
|
<sect2 id="gin-partial-match">
|
|
<title>Partial match algorithm</title>
|
|
|
|
<para>
|
|
GIN can support <quote>partial match</> queries, in which the query
|
|
does not determine an exact match for one or more keys, but the possible
|
|
matches fall within a reasonably narrow range of key values (within the
|
|
key sorting order determined by the <function>compare</> support method).
|
|
The <function>extractQuery</> method, instead of returning a key value
|
|
to be matched exactly, returns a key value that is the lower bound of
|
|
the range to be searched, and sets the <literal>pmatch</> flag true.
|
|
The key range is then searched using the <function>comparePartial</>
|
|
method. <function>comparePartial</> must return zero for an actual
|
|
match, less than zero for a non-match that is still within the range
|
|
to be searched, or greater than zero if the index key is past the range
|
|
that could match.
|
|
</para>
|
|
</sect2>
|
|
|
|
</sect1>
|
|
|
|
<sect1 id="gin-tips">
|
|
<title>GIN tips and tricks</title>
|
|
|
|
<variablelist>
|
|
<varlistentry>
|
|
<term>Create vs insert</term>
|
|
<listitem>
|
|
<para>
|
|
Insertion into a <acronym>GIN</acronym> index can be slow
|
|
due to the likelihood of many keys being inserted for each value.
|
|
So, for bulk insertions into a table it is advisable to drop the GIN
|
|
index and recreate it after finishing bulk insertion.
|
|
</para>
|
|
|
|
<para>
|
|
As of <productname>PostgreSQL</productname> 8.4, this advice is less
|
|
necessary since delayed indexing is used (see <xref
|
|
linkend="gin-fast-update"> for details). But for very large updates
|
|
it may still be best to drop and recreate the index.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><xref linkend="guc-maintenance-work-mem"></term>
|
|
<listitem>
|
|
<para>
|
|
Build time for a <acronym>GIN</acronym> index is very sensitive to
|
|
the <varname>maintenance_work_mem</> setting; it doesn't pay to
|
|
skimp on work memory during index creation.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><xref linkend="guc-work-mem"></term>
|
|
<listitem>
|
|
<para>
|
|
During a series of insertions into an existing <acronym>GIN</acronym>
|
|
index that has <literal>FASTUPDATE</> enabled, the system will clean up
|
|
the pending-entry list whenever it grows larger than
|
|
<varname>work_mem</>. To avoid fluctuations in observed response time,
|
|
it's desirable to have pending-list cleanup occur in the background
|
|
(i.e., via autovacuum). Foreground cleanup operations can be avoided by
|
|
increasing <varname>work_mem</> or making autovacuum more aggressive.
|
|
However, enlarging <varname>work_mem</> means that if a foreground
|
|
cleanup does occur, it will take even longer.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><xref linkend="guc-gin-fuzzy-search-limit"></term>
|
|
<listitem>
|
|
<para>
|
|
The primary goal of developing <acronym>GIN</acronym> indexes was
|
|
to create support for highly scalable, full-text search in
|
|
<productname>PostgreSQL</productname>, and there are often situations when
|
|
a full-text search returns a very large set of results. Moreover, this
|
|
often happens when the query contains very frequent words, so that the
|
|
large result set is not even useful. Since reading many
|
|
tuples from the disk and sorting them could take a lot of time, this is
|
|
unacceptable for production. (Note that the index search itself is very
|
|
fast.)
|
|
</para>
|
|
<para>
|
|
To facilitate controlled execution of such queries
|
|
<acronym>GIN</acronym> has a configurable soft upper limit on the
|
|
number of rows returned, the
|
|
<varname>gin_fuzzy_search_limit</varname> configuration parameter.
|
|
It is set to 0 (meaning no limit) by default.
|
|
If a non-zero limit is set, then the returned set is a subset of
|
|
the whole result set, chosen at random.
|
|
</para>
|
|
<para>
|
|
<quote>Soft</quote> means that the actual number of returned results
|
|
could differ slightly from the specified limit, depending on the query
|
|
and the quality of the system's random number generator.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
</variablelist>
|
|
|
|
</sect1>
|
|
|
|
<sect1 id="gin-limit">
|
|
<title>Limitations</title>
|
|
|
|
<para>
|
|
<acronym>GIN</acronym> doesn't support full index scans: because there are
|
|
often many keys per value, each heap pointer would be returned many times,
|
|
and there is no easy way to prevent this.
|
|
</para>
|
|
|
|
<para>
|
|
When <function>extractQuery</function> returns zero keys,
|
|
<acronym>GIN</acronym> will emit an error. Depending on the operator,
|
|
a void query might match all, some, or none of the indexed values (for
|
|
example, every array contains the empty array, but does not overlap the
|
|
empty array), and <acronym>GIN</acronym> cannot determine the correct
|
|
answer, nor produce a full-index-scan result if it could determine that
|
|
that was correct.
|
|
</para>
|
|
|
|
<para>
|
|
It is not an error for <function>extractValue</> to return zero keys,
|
|
but in this case the indexed value will be unrepresented in the index.
|
|
This is another reason why full index scan is not useful — it would
|
|
miss such rows.
|
|
</para>
|
|
|
|
<para>
|
|
It is possible for an operator class to circumvent the restriction against
|
|
full index scan. To do that, <function>extractValue</> must return at least
|
|
one (possibly dummy) key for every indexed value, and
|
|
<function>extractQuery</function> must convert an unrestricted search into
|
|
a partial-match query that will scan the whole index. This is inefficient
|
|
but might be necessary to avoid corner-case failures with operators such
|
|
as <literal>LIKE</>.
|
|
</para>
|
|
</sect1>
|
|
|
|
<sect1 id="gin-examples">
|
|
<title>Examples</title>
|
|
|
|
<para>
|
|
The <productname>PostgreSQL</productname> source distribution includes
|
|
<acronym>GIN</acronym> operator classes for <type>tsvector</> and
|
|
for one-dimensional arrays of all internal types. Prefix searching in
|
|
<type>tsvector</> is implemented using the <acronym>GIN</> partial match
|
|
feature.
|
|
The following <filename>contrib</> modules also contain
|
|
<acronym>GIN</acronym> operator classes:
|
|
</para>
|
|
|
|
<variablelist>
|
|
<varlistentry>
|
|
<term>btree-gin</term>
|
|
<listitem>
|
|
<para>B-Tree equivalent functionality for several data types</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term>hstore</term>
|
|
<listitem>
|
|
<para>Module for storing (key, value) pairs</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term>intarray</term>
|
|
<listitem>
|
|
<para>Enhanced support for int4[]</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term>pg_trgm</term>
|
|
<listitem>
|
|
<para>Text similarity using trigram matching</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
</variablelist>
|
|
</sect1>
|
|
|
|
</chapter>
|