database/postgres
1
0
Fork 0
mirror of https://github.com/postgres/postgres.git synced 2025-07-30 11:03:19 +03:00
Code Activity

Extended statistics on expressions

Browse Source 
Allow defining extended statistics on expressions, not just just on
simple column references.  With this commit, expressions are supported
by all existing extended statistics kinds, improving the same types of
estimates. A simple example may look like this:

  CREATE TABLE t (a int);
  CREATE STATISTICS s ON mod(a,10), mod(a,20) FROM t;
  ANALYZE t;

The collected statistics are useful e.g. to estimate queries with those
expressions in WHERE or GROUP BY clauses:

  SELECT * FROM t WHERE mod(a,10) = 0 AND mod(a,20) = 0;

  SELECT 1 FROM t GROUP BY mod(a,10), mod(a,20);

This introduces new internal statistics kind 'e' (expressions) which is
built automatically when the statistics object definition includes any
expressions. This represents single-expression statistics, as if there
was an expression index (but without the index maintenance overhead).
The statistics is stored in pg_statistics_ext_data as an array of
composite types, which is possible thanks to 79f6a942bd.

CREATE STATISTICS allows building statistics on a single expression, in
which case in which case it's not possible to specify statistics kinds.

A new system view pg_stats_ext_exprs can be used to display expression
statistics, similarly to pg_stats and pg_stats_ext views.

ALTER TABLE ... ALTER COLUMN ... TYPE now treats indexes the same way it
treats indexes, i.e. it drops and recreates the statistics. This means
all statistics are reset, and we no longer try to preserve at least the
functional dependencies. This should not be a major issue in practice,
as the functional dependencies actually rely on per-column statistics,
which were always reset anyway.

Author: Tomas Vondra
Reviewed-by: Justin Pryzby, Dean Rasheed, Zhihong Yu
Discussion: https://postgr.es/m/ad7891d2-e90c-b446-9fe2-7419143847d7%40enterprisedb.com
This commit is contained in:
Tomas Vondra
2021-03-26 23:22:01 +01:00
parent 98376c18f1
commit a4d75c86bf
43 changed files with 5838 additions and 819 deletions
Download Patch File Download Diff File Expand all files Collapse all files

295
doc/src/sgml/catalogs.sgml
View File

@ -7385,8 +7385,22 @@ SCRAM-SHA-256$<replaceable>&lt;iteration count&gt;</replaceable>:<replaceable>&l
<literal>d</literal> for n-distinct statistics,
<literal>f</literal> for functional dependency statistics, and
<literal>m</literal> for most common values (MCV) list statistics
<literal>e</literal> for expression statistics
</para></entry>
</row>
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>stxexprs</structfield> <type>pg_node_tree</type>
</para>
<para>
Expression trees (in <function>nodeToString()</function>
representation) for statistics object attributes that are not simple
column references. This is a list with one element per expression.
Null if all statistics object attributes are simple references.
</para></entry>
</row>
</tbody>
</tgroup>
</table>
@ -7452,7 +7466,7 @@ SCRAM-SHA-256$<replaceable>&lt;iteration count&gt;</replaceable>:<replaceable>&l
(references <link linkend="catalog-pg-statistic-ext"><structname>pg_statistic_ext</structname></link>.<structfield>oid</structfield>)
</para>
<para>
Extended statistic object containing the definition for this data
Extended statistics object containing the definition for this data
</para></entry>
</row>
@ -7484,6 +7498,15 @@ SCRAM-SHA-256$<replaceable>&lt;iteration count&gt;</replaceable>:<replaceable>&l
<structname>pg_mcv_list</structname> type
</para></entry>
</row>
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>stxexprs</structfield> <type>pg_node_tree</type>
</para>
<para>
A list of any expressions covered by this statistics object.
</para></entry>
</row>
</tbody>
</tgroup>
</table>
@ -7637,6 +7660,16 @@ SCRAM-SHA-256$<replaceable>&lt;iteration count&gt;</replaceable>:<replaceable>&l
see <xref linkend="logical-replication-publication"/>.
</para></entry>
</row>
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>stxdexpr</structfield> <type>pg_statistic[]</type>
</para>
<para>
Per-expression statistics, serialized as an array of
<structname>pg_statistic</structname> type
</para></entry>
</row>
</tbody>
</tgroup>
</table>
@ -9444,6 +9477,11 @@ SCRAM-SHA-256$<replaceable>&lt;iteration count&gt;</replaceable>:<replaceable>&l
<entry>extended planner statistics</entry>
</row>
<row>
<entry><link linkend="view-pg-stats-ext-exprs"><structname>pg_stats_ext_exprs</structname></link></entry>
<entry>extended planner statistics for expressions</entry>
</row>
<row>
<entry><link linkend="view-pg-tables"><structname>pg_tables</structname></link></entry>
<entry>tables</entry>
@ -12696,10 +12734,19 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
(references <link linkend="catalog-pg-attribute"><structname>pg_attribute</structname></link>.<structfield>attname</structfield>)
</para>
<para>
Name of the column described by this row
Names of the columns included in the extended statistics object
</para></entry>
</row>
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>exprs</structfield> <type>text[]</type>
</para>
<para>
Expressions included in the extended statistics object
</para></entry>
</row>
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>inherited</structfield> <type>bool</type>
@ -12851,7 +12898,8 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
<para>
The view <structname>pg_stats_ext</structname> provides access to
the information stored in the <link
information about each extended statistics object in the database,
combining information stored in the <link
linkend="catalog-pg-statistic-ext"><structname>pg_statistic_ext</structname></link>
and <link linkend="catalog-pg-statistic-ext-data"><structname>pg_statistic_ext_data</structname></link>
catalogs. This view allows access only to rows of
@ -12908,7 +12956,7 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
(references <link linkend="catalog-pg-namespace"><structname>pg_namespace</structname></link>.<structfield>nspname</structfield>)
</para>
<para>
Name of schema containing extended statistic
Name of schema containing extended statistics object
</para></entry>
</row>
@ -12918,7 +12966,7 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
(references <link linkend="catalog-pg-statistic-ext"><structname>pg_statistic_ext</structname></link>.<structfield>stxname</structfield>)
</para>
<para>
Name of extended statistics
Name of extended statistics object
</para></entry>
</row>
@ -12928,7 +12976,7 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
(references <link linkend="catalog-pg-authid"><structname>pg_authid</structname></link>.<structfield>rolname</structfield>)
</para>
<para>
Owner of the extended statistics
Owner of the extended statistics object
</para></entry>
</row>
@ -12938,7 +12986,7 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
(references <link linkend="catalog-pg-attribute"><structname>pg_attribute</structname></link>.<structfield>attname</structfield>)
</para>
<para>
Names of the columns the extended statistics is defined on
Names of the columns the extended statistics object is defined on
</para></entry>
</row>
@ -12947,7 +12995,7 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
<structfield>kinds</structfield> <type>char[]</type>
</para>
<para>
Types of extended statistics enabled for this record
Types of extended statistics object enabled for this record
</para></entry>
</row>
@ -13032,6 +13080,237 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
</sect1>
<sect1 id="view-pg-stats-ext-exprs">
<title><structname>pg_stats_ext_exprs</structname></title>
<indexterm zone="view-pg-stats-ext-exprs">
<primary>pg_stats_ext_exprs</primary>
</indexterm>
<para>
The view <structname>pg_stats_ext_exprs</structname> provides access to
information about all expressions included in extended statistics objects,
combining information stored in the <link
linkend="catalog-pg-statistic-ext"><structname>pg_statistic_ext</structname></link>
and <link linkend="catalog-pg-statistic-ext-data"><structname>pg_statistic_ext_data</structname></link>
catalogs. This view allows access only to rows of
<link linkend="catalog-pg-statistic-ext"><structname>pg_statistic_ext</structname></link> and <link linkend="catalog-pg-statistic-ext-data"><structname>pg_statistic_ext_data</structname></link>
that correspond to tables the user has permission to read, and therefore
it is safe to allow public read access to this view.
</para>
<para>
<structname>pg_stats_ext_exprs</structname> is also designed to present
the information in a more readable format than the underlying catalogs
&mdash; at the cost that its schema must be extended whenever the structure
of statistics in <link linkend="catalog-pg-statistic"><structname>pg_statistic</structname></link> changes.
</para>
<table>
<title><structname>pg_stats_ext_exprs</structname> Columns</title>
<tgroup cols="1">
<thead>
<row>
<entry role="catalog_table_entry"><para role="column_definition">
Column Type
</para>
<para>
Description
</para></entry>
</row>
</thead>
<tbody>
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>schemaname</structfield> <type>name</type>
(references <link linkend="catalog-pg-namespace"><structname>pg_namespace</structname></link>.<structfield>nspname</structfield>)
</para>
<para>
Name of schema containing table
</para></entry>
</row>
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>tablename</structfield> <type>name</type>
(references <link linkend="catalog-pg-class"><structname>pg_class</structname></link>.<structfield>relname</structfield>)
</para>
<para>
Name of table the statistics object is defined on
</para></entry>
</row>
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>statistics_schemaname</structfield> <type>name</type>
(references <link linkend="catalog-pg-namespace"><structname>pg_namespace</structname></link>.<structfield>nspname</structfield>)
</para>
<para>
Name of schema containing extended statistics object
</para></entry>
</row>
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>statistics_name</structfield> <type>name</type>
(references <link linkend="catalog-pg-statistic-ext"><structname>pg_statistic_ext</structname></link>.<structfield>stxname</structfield>)
</para>
<para>
Name of extended statistics object
</para></entry>
</row>
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>statistics_owner</structfield> <type>name</type>
(references <link linkend="catalog-pg-authid"><structname>pg_authid</structname></link>.<structfield>rolname</structfield>)
</para>
<para>
Owner of the extended statistics object
</para></entry>
</row>
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>expr</structfield> <type>text</type>
</para>
<para>
Expression included in the extended statistics object
</para></entry>
</row>
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>null_frac</structfield> <type>float4</type>
</para>
<para>
Fraction of expression entries that are null
</para></entry>
</row>
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>avg_width</structfield> <type>int4</type>
</para>
<para>
Average width in bytes of expression's entries
</para></entry>
</row>
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>n_distinct</structfield> <type>float4</type>
</para>
<para>
If greater than zero, the estimated number of distinct values in the
expression. If less than zero, the negative of the number of distinct
values divided by the number of rows. (The negated form is used when
<command>ANALYZE</command> believes that the number of distinct values is
likely to increase as the table grows; the positive form is used when
the expression seems to have a fixed number of possible values.) For
example, -1 indicates a unique expression in which the number of distinct
values is the same as the number of rows.
</para></entry>
</row>
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>most_common_vals</structfield> <type>anyarray</type>
</para>
<para>
A list of the most common values in the expression. (Null if
no values seem to be more common than any others.)
</para></entry>
</row>
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>most_common_freqs</structfield> <type>float4[]</type>
</para>
<para>
A list of the frequencies of the most common values,
i.e., number of occurrences of each divided by total number of rows.
(Null when <structfield>most_common_vals</structfield> is.)
</para></entry>
</row>
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>histogram_bounds</structfield> <type>anyarray</type>
</para>
<para>
A list of values that divide the expression's values into groups of
approximately equal population. The values in
<structfield>most_common_vals</structfield>, if present, are omitted from this
histogram calculation. (This expression is null if the expression data type
does not have a <literal>&lt;</literal> operator or if the
<structfield>most_common_vals</structfield> list accounts for the entire
population.)
</para></entry>
</row>
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>correlation</structfield> <type>float4</type>
</para>
<para>
Statistical correlation between physical row ordering and
logical ordering of the expression values. This ranges from -1 to +1.
When the value is near -1 or +1, an index scan on the expression will
be estimated to be cheaper than when it is near zero, due to reduction
of random access to the disk. (This expression is null if the expression's
data type does not have a <literal>&lt;</literal> operator.)
</para></entry>
</row>
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>most_common_elems</structfield> <type>anyarray</type>
</para>
<para>
A list of non-null element values most often appearing within values of
the expression. (Null for scalar types.)
</para></entry>
</row>
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>most_common_elem_freqs</structfield> <type>float4[]</type>
</para>
<para>
A list of the frequencies of the most common element values, i.e., the
fraction of rows containing at least one instance of the given value.
Two or three additional values follow the per-element frequencies;
these are the minimum and maximum of the preceding per-element
frequencies, and optionally the frequency of null elements.
(Null when <structfield>most_common_elems</structfield> is.)
</para></entry>
</row>
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>elem_count_histogram</structfield> <type>float4[]</type>
</para>
<para>
A histogram of the counts of distinct non-null element values within the
values of the expression, followed by the average number of distinct
non-null elements. (Null for scalar types.)
</para></entry>
</row>
</tbody>
</tgroup>
</table>
<para>
The maximum number of entries in the array fields can be controlled on a
column-by-column basis using the <link linkend="sql-altertable"><command>ALTER
TABLE SET STATISTICS</command></link> command, or globally by setting the
<xref linkend="guc-default-statistics-target"/> run-time parameter.
</para>
</sect1>
<sect1 id="view-pg-tables">
<title><structname>pg_tables</structname></title>

116
doc/src/sgml/ref/create_statistics.sgml
View File

@ -21,9 +21,13 @@ PostgreSQL documentation
<refsynopsisdiv>
<synopsis>
CREATE STATISTICS [ IF NOT EXISTS ] <replaceable class="parameter">statistics_name</replaceable>
ON ( <replaceable class="parameter">expression</replaceable> )
FROM <replaceable class="parameter">table_name</replaceable>
CREATE STATISTICS [ IF NOT EXISTS ] <replaceable class="parameter">statistics_name</replaceable>
[ ( <replaceable class="parameter">statistics_kind</replaceable> [, ... ] ) ]
ON <replaceable class="parameter">column_name</replaceable>, <replaceable class="parameter">column_name</replaceable> [, ...]
ON { <replaceable class="parameter">column_name</replaceable> | ( <replaceable class="parameter">expression</replaceable> ) }, { <replaceable class="parameter">column_name</replaceable> | ( <replaceable class="parameter">expression</replaceable> ) } [, ...]
FROM <replaceable class="parameter">table_name</replaceable>
</synopsis>
@ -39,6 +43,19 @@ CREATE STATISTICS [ IF NOT EXISTS ] <replaceable class="parameter">statistics_na
database and will be owned by the user issuing the command.
</para>
<para>
The <command>CREATE STATISTICS</command> command has two basic forms. The
first form allows univariate statistics for a single expression to be
collected, providing benefits similar to an expression index without the
overhead of index maintenance. This form does not allow the statistics
kind to be specified, since the various statistics kinds refer only to
multivariate statistics. The second form of the command allows
multivariate statistics on multiple columns and/or expressions to be
collected, optionally specifying which statistics kinds to include. This
form will also automatically cause univariate statistics to be collected on
any expressions included in the list.
</para>
<para>
If a schema name is given (for example, <literal>CREATE STATISTICS
myschema.mystat ...</literal>) then the statistics object is created in the
@ -79,14 +96,16 @@ CREATE STATISTICS [ IF NOT EXISTS ] <replaceable class="parameter">statistics_na
<term><replaceable class="parameter">statistics_kind</replaceable></term>
<listitem>
<para>
A statistics kind to be computed in this statistics object.
A multivariate statistics kind to be computed in this statistics object.
Currently supported kinds are
<literal>ndistinct</literal>, which enables n-distinct statistics,
<literal>dependencies</literal>, which enables functional
dependency statistics, and <literal>mcv</literal> which enables
most-common values lists.
If this clause is omitted, all supported statistics kinds are
included in the statistics object.
included in the statistics object. Univariate expression statistics are
built automatically if the statistics definition includes any complex
expressions rather than just simple column references.
For more information, see <xref linkend="planner-stats-extended"/>
and <xref linkend="multivariate-statistics-examples"/>.
</para>
@ -98,8 +117,22 @@ CREATE STATISTICS [ IF NOT EXISTS ] <replaceable class="parameter">statistics_na
<listitem>
<para>
The name of a table column to be covered by the computed statistics.
At least two column names must be given; the order of the column names
is insignificant.
This is only allowed when building multivariate statistics. At least
two column names or expressions must be specified, and their order is
not significant.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><replaceable class="parameter">expression</replaceable></term>
<listitem>
<para>
An expression to be covered by the computed statistics. This may be
used to build univariate statistics on a single expression, or as part
of a list of multiple column names and/or expressions to build
multivariate statistics. In the latter case, separate univariate
statistics are built automatically for each expression in the list.
</para>
</listitem>
</varlistentry>
@ -125,6 +158,13 @@ CREATE STATISTICS [ IF NOT EXISTS ] <replaceable class="parameter">statistics_na
reading it. Once created, however, the ownership of the statistics
object is independent of the underlying table(s).
</para>
<para>
Expression statistics are per-expression and are similar to creating an
index on the expression, except that they avoid the overhead of index
maintenance. Expression statistics are built automatically for each
expression in the statistics object definition.
</para>
</refsect1>
<refsect1 id="sql-createstatistics-examples">
@ -196,6 +236,72 @@ EXPLAIN ANALYZE SELECT * FROM t2 WHERE (a = 1) AND (b = 2);
in the table, allowing it to generate better estimates in both cases.
</para>
<para>
Create table <structname>t3</structname> with a single timestamp column,
and run queries using expressions on that column. Without extended
statistics, the planner has no information about the data distribution for
the expressions, and uses default estimates. The planner also does not
realize that the value of the date truncated to the month is fully
determined by the value of the date truncated to the day. Then expression
and ndistinct statistics are built on those two expressions:
<programlisting>
CREATE TABLE t3 (
a timestamp
);
INSERT INTO t3 SELECT i FROM generate_series('2020-01-01'::timestamp,
'2020-12-31'::timestamp,
'1 minute'::interval) s(i);
ANALYZE t3;
-- the number of matching rows will be drastically underestimated:
EXPLAIN ANALYZE SELECT * FROM t3
WHERE date_trunc('month', a) = '2020-01-01'::timestamp;
EXPLAIN ANALYZE SELECT * FROM t3
WHERE date_trunc('day', a) BETWEEN '2020-01-01'::timestamp
AND '2020-06-30'::timestamp;
EXPLAIN ANALYZE SELECT date_trunc('month', a), date_trunc('day', a)
FROM t3 GROUP BY 1, 2;
-- build ndistinct statistics on the pair of expressions (per-expression
-- statistics are built automatically)
CREATE STATISTICS s3 (ndistinct) ON date_trunc('month', a), date_trunc('day', a) FROM t3;
ANALYZE t3;
-- now the row count estimates are more accurate:
EXPLAIN ANALYZE SELECT * FROM t3
WHERE date_trunc('month', a) = '2020-01-01'::timestamp;
EXPLAIN ANALYZE SELECT * FROM t3
WHERE date_trunc('day', a) BETWEEN '2020-01-01'::timestamp
AND '2020-06-30'::timestamp;
EXPLAIN ANALYZE SELECT date_trunc('month', a), date_trunc('day', a)
FROM t3 GROUP BY 1, 2;
</programlisting>
Without expression and ndistinct statistics, the planner has no information
about the number of distinct values for the expressions, and has to rely
on default estimates. The equality and range conditions are assumed to have
0.5% selectivity, and the number of distinct values in the expression is
assumed to be the same as for the column (i.e. unique). This results in a
significant underestimate of the row count in the first two queries. Moreover,
the planner has no information about the relationship between the expressions,
so it assumes the two <literal>WHERE</literal> and <literal>GROUP BY</literal>
conditions are independent, and multiplies their selectivities together to
arrive at a severe overestimate of the group count in the aggregate query.
This is further exacerbated by the lack of accurate statistics for the
expressions, forcing the planner to use a default ndistinct estimate for the
expression derived from ndistinct for the column. With such statistics, the
planner recognizes that the conditions are correlated, and arrives at much
more accurate estimates.
</para>
</refsect1>
<refsect1>