database/postgres
1
0
Fork 0
mirror of https://github.com/postgres/postgres.git synced 2025-09-03 15:22:11 +03:00
Code Activity

Remove pessimistic cost penalization from Incremental Sort

Browse Source 
When incremental sorts were added in v13 a 1.5x pessimism factor was added
to the cost modal.  Seemingly this was done because the cost modal only
has an estimate of the total number of input rows and the number of
presorted groups.  It assumes that the input rows will be evenly
distributed throughout the presorted groups.  The 1.5x pessimism factor
was added to slightly reduce the likelihood of incremental sorts being
used in the hope to avoid performance regressions where an incremental
sort plan was picked and turned out slower due to a large skew in the
number of rows in the presorted groups.

An additional quirk with the path generation code meant that we could
consider both a sort and an incremental sort on paths with presorted keys.
This meant that with the pessimism factor, it was possible that we opted
to perform a sort rather than an incremental sort when the given path had
presorted keys.

Here we remove the 1.5x pessimism factor to allow incremental sorts to
have a fairer chance at being chosen against a full sort.

Previously we would generally create a sort path on the cheapest input
path (if that wasn't sorted already) and incremental sort paths on any
path which had presorted keys.  This meant that if the cheapest input path
wasn't completely sorted but happened to have presorted keys, we would
create a full sort path *and* an incremental sort path on that input path.
Here we change this logic so that if there are presorted keys, we only
create an incremental sort path, and create sort paths only when a full
sort is required.

Both the removal of the cost pessimism factor and the changes made to the
path generation make it more likely that incremental sorts will now be
chosen.  That, of course, as with teaching the planner any new tricks,
means an increased likelihood that the planner will perform an incremental
sort when it's not the best method.  Our standard escape hatch for these
cases is an enable_* GUC.  enable_incremental_sort already exists for
this.

This came out of a report by Pavel Luzanov where he mentioned that the
master branch was choosing to perform a Seq Scan -> Sort -> Group
Aggregate for his query with an ORDER BY aggregate function.  The v15 plan
for his query performed an Index Scan -> Group Aggregate, of course, the
aggregate performed the final sort internally in nodeAgg.c for the
aggregate's ORDER BY.  The ideal plan would have been to use the index,
which provided partially sorted input then use an incremental sort to
provide the aggregate with the sorted input.  This was not being chosen
due to the pessimism in the incremental sort cost modal, so here we remove
that and rationalize the path generation so that sort and incremental sort
plans don't have to needlessly compete.  We assume that it's senseless
to ever use a full sort on a given input path where an incremental sort
can be performed.

Reported-by: Pavel Luzanov
Reviewed-by: Richard Guo
Discussion: https://postgr.es/m/9f61ddbf-2989-1536-b31e-6459370a6baa%40postgrespro.ru
This commit is contained in:
David Rowley
2022-12-16 15:22:23 +13:00
parent 8b6b043cee
commit 4a29eabd1d
8 changed files with 220 additions and 419 deletions
Download Patch File Download Diff File Expand all files Collapse all files

92
src/backend/optimizer/path/allpaths.c
View File

@@ -3207,70 +3207,52 @@ generate_useful_gather_paths(PlannerInfo *root, RelOptInfo *rel, bool override_r
continue;
/*
* Consider regular sort for the cheapest partial path (for each
* useful pathkeys). We know the path is not sorted, because we'd
* not get here otherwise.
* Try at least sorting the cheapest path and also try
* incrementally sorting any path which is partially sorted
* already (no need to deal with paths which have presorted keys
* when incremental sort is disabled unless it's the cheapest
* input path).
*/
if (subpath != cheapest_partial_path &&
(presorted_keys == 0 || !enable_incremental_sort))
continue;
/*
* Consider regular sort for any path that's not presorted or if
* incremental sort is disabled. We've no need to consider both
* sort and incremental sort on the same path. We assume that
* incremental sort is always faster when there are presorted
* keys.
*
* This is not redundant with the gather paths created in
* generate_gather_paths, because that doesn't generate ordered
* output. Here we add an explicit sort to match the useful
* ordering.
*/
if (cheapest_partial_path == subpath)
if (presorted_keys == 0 || !enable_incremental_sort)
{
Path *tmp;
tmp = (Path *) create_sort_path(root,
rel,
subpath,
useful_pathkeys,
-1.0);
rows = tmp->rows * tmp->parallel_workers;
path = create_gather_merge_path(root, rel,
tmp,
rel->reltarget,
tmp->pathkeys,
NULL,
rowsp);
add_path(rel, &path->path);
/* Fall through */
subpath = (Path *) create_sort_path(root,
rel,
subpath,
useful_pathkeys,
-1.0);
rows = subpath->rows * subpath->parallel_workers;
}
else
subpath = (Path *) create_incremental_sort_path(root,
rel,
subpath,
useful_pathkeys,
presorted_keys,
-1);
path = create_gather_merge_path(root, rel,
subpath,
rel->reltarget,
subpath->pathkeys,
NULL,
rowsp);
/*
* Consider incremental sort, but only when the subpath is already
* partially sorted on a pathkey prefix.
*/
if (enable_incremental_sort && presorted_keys > 0)
{
Path *tmp;
/*
* We should have already excluded pathkeys of length 1
* because then presorted_keys > 0 would imply is_sorted was
* true.
*/
Assert(list_length(useful_pathkeys) != 1);
tmp = (Path *) create_incremental_sort_path(root,
rel,
subpath,
useful_pathkeys,
presorted_keys,
-1);
path = create_gather_merge_path(root, rel,
tmp,
rel->reltarget,
tmp->pathkeys,
NULL,
rowsp);
add_path(rel, &path->path);
}
add_path(rel, &path->path);
}
}
}

38
src/backend/optimizer/path/costsize.c
View File

@@ -1959,8 +1959,8 @@ cost_incremental_sort(Path *path,
double input_tuples, int width, Cost comparison_cost, int sort_mem,
double limit_tuples)
{
Cost startup_cost = 0,
run_cost = 0,
Cost startup_cost,
run_cost,
input_run_cost = input_total_cost - input_startup_cost;
double group_tuples,
input_groups;
@@ -1969,10 +1969,9 @@ cost_incremental_sort(Path *path,
group_input_run_cost;
List *presortedExprs = NIL;
ListCell *l;
int i = 0;
bool unknown_varno = false;
Assert(presorted_keys != 0);
Assert(presorted_keys > 0 && presorted_keys < list_length(pathkeys));
/*
* We want to be sure the cost of a sort is never estimated as zero, even
@@ -2025,12 +2024,11 @@ cost_incremental_sort(Path *path,
/* expression not containing any Vars with "varno 0" */
presortedExprs = lappend(presortedExprs, member->em_expr);
i++;
if (i >= presorted_keys)
if (foreach_current_index(l) + 1 >= presorted_keys)
break;
}
/* Estimate number of groups with equal presorted keys. */
/* Estimate the number of groups with equal presorted keys. */
if (!unknown_varno)
input_groups = estimate_num_groups(root, presortedExprs, input_tuples,
NULL, NULL);
@@ -2039,22 +2037,19 @@ cost_incremental_sort(Path *path,
group_input_run_cost = input_run_cost / input_groups;
/*
* Estimate average cost of sorting of one group where presorted keys are
* equal. Incremental sort is sensitive to distribution of tuples to the
* groups, where we're relying on quite rough assumptions. Thus, we're
* pessimistic about incremental sort performance and increase its average
* group size by half.
* Estimate the average cost of sorting of one group where presorted keys
* are equal.
*/
cost_tuplesort(&group_startup_cost, &group_run_cost,
1.5 * group_tuples, width, comparison_cost, sort_mem,
group_tuples, width, comparison_cost, sort_mem,
limit_tuples);
/*
* Startup cost of incremental sort is the startup cost of its first group
* plus the cost of its input.
*/
startup_cost += group_startup_cost
+ input_startup_cost + group_input_run_cost;
startup_cost = group_startup_cost + input_startup_cost +
group_input_run_cost;
/*
* After we started producing tuples from the first group, the cost of
@@ -2062,17 +2057,20 @@ cost_incremental_sort(Path *path,
* group, plus the total cost to process the remaining groups, plus the
* remaining cost of input.
*/
run_cost += group_run_cost
+ (group_run_cost + group_startup_cost) * (input_groups - 1)
+ group_input_run_cost * (input_groups - 1);
run_cost = group_run_cost + (group_run_cost + group_startup_cost) *
(input_groups - 1) + group_input_run_cost * (input_groups - 1);
/*
* Incremental sort adds some overhead by itself. Firstly, it has to
* detect the sort groups. This is roughly equal to one extra copy and
* comparison per tuple. Secondly, it has to reset the tuplesort context
* for every group.
* comparison per tuple.
*/
run_cost += (cpu_tuple_cost + comparison_cost) * input_tuples;
/*
* Additionally, we charge double cpu_tuple_cost for each input group to
* account for the tuplesort_reset that's performed after each group.
*/
run_cost += 2.0 * cpu_tuple_cost * input_groups;
path->rows = input_tuples;