Window functions run in a loop in
Frame_cursor::compute_values_for_current_row which can include a large
number of rows.
Adjust this function to check for the current thd being killed by only
every 256 rows, so as not to destroy any CPU pipelining or similar.
/home/buildbot/amd64-last-N-failed/build/sql/sql_window.cc: In function ‘ORDER* concat_order_lists(MEM_ROOT*, ORDER*, ORDER*)’:
/home/buildbot/amd64-last-N-failed/build/sql/sql_window.cc:2976:13: error: ‘void* memcpy(void*, const void*, size_t)’ writing to an object of type ‘ORDER’ {aka ‘struct st_order’} with no trivial copy-assignment; use copy-assignment or copy-initialization instead [-Werror=class-memaccess]
2976 | memcpy(copy, cur, sizeof(ORDER));
| ~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /home/buildbot/amd64-last-N-failed/build/sql/field.h:29,
from /home/buildbot/amd64-last-N-failed/build/sql/sql_class.h:32,
from /home/buildbot/amd64-last-N-failed/build/sql/sql_acl.h:21,
from /home/buildbot/amd64-last-N-failed/build/sql/sql_parse.h:19,
from /home/buildbot/amd64-last-N-failed/build/sql/sql_window.cc:18:
/home/buildbot/amd64-last-N-failed/build/sql/table.h:236:16: note: ‘ORDER’ {aka ‘struct st_order’} declared here
236 | typedef struct st_order {
| ^~~~~~~~
/home/buildbot/amd64-last-N-failed/build/sql/sql_window.cc: In member function ‘bool Window_funcs_sort::setup(THD*, SQL_SELECT*, List_iterator<Item_window_func>&, JOIN_TAB*)’:
/home/buildbot/amd64-last-N-failed/build/sql/sql_window.cc:3151:11: error: ‘void* memset(void*, int, size_t)’ clearing an object of type ‘ORDER’ {aka ‘struct st_order’} with no trivial copy-assignment; use assignment or value-initialization instead [-Werror=class-memaccess]
3151 | memset(order, 0, sizeof(*order));
| ~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /home/buildbot/amd64-last-N-failed/build/sql/field.h:29,
from /home/buildbot/amd64-last-N-failed/build/sql/sql_class.h:32,
from /home/buildbot/amd64-last-N-failed/build/sql/sql_acl.h:21,
from /home/buildbot/amd64-last-N-failed/build/sql/sql_parse.h:19,
from /home/buildbot/amd64-last-N-failed/build/sql/sql_window.cc:18:
/home/buildbot/amd64-last-N-failed/build/sql/table.h:236:16: note: ‘ORDER’ {aka ‘struct st_order’} declared here
236 | typedef struct st_order {
| ^~~~~~~~
cc1plus: all warnings being treated as errors
make[2]: *** [sql/CMakeFiles/sql.dir/build.make:2640: sql/CMakeFiles/sql.dir/sql_window.cc.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [CMakeFiles/Makefile2:10639: sql/CMakeFiles/sql.dir/all] Error 2
make: *** [Makefile:166: all] Error 2
Optimization in st_select_lex_unit::prepare() removes ORDER BY for
certain subqueries. That excludes ORDER BY items from being fixed, but
sl->window_funcs still contains window function items and those
related to optimized out ORDER BY are unfixed. The error about missing
window spec is thrown when the item is fixed. Hence we get redundant
processing of window function items without checking window spec
existence.
The fix removes the related window function items when ORDER BY is
optimized out. ORDER accumulates window_funcs at parser stage which
are then removed from SELECT_LEX::window_funcs. The fix also updates
similar optimization in mysql_make_view().
When the query uses several Window Functions:
SELECT
WIN_FUNC1() OVER (ORDER BY 'const', col1),
WIN_FUNC2() OVER (ORDER BY col1 RANGE BETWEEN CURRENT ROW
AND 5 FOLLOWING)
compare_window_funcs_by_window_specs() will try to get the Window Specs to
reuse the ORDER BY lists. If the lists produce the same order (like above)
Window Spec of the WIN_FUNC2 will reuse the ORDER BY list of WIN_FUNC1.
However, WIN_FUNC2 has a RANGE-type window frame. It expects to get
ORDER BY list with one element, which it will use to compute frame bounds.
Proving it with ORDER BY list from WIN_FUNC1 ('const', col1) was caused an
assertion failure
The fix is to:
Use the original ORDER BY list when constructing RANGE-type frames
Fix an apparent typo bug in compare_window_funcs_by_window_specs():
assignment
win_spec1->save_order_list= win_spec2->order_list;
Saved the order list from the wrong spec. Instead, take one from win_spec1.
add_special_frame_cursors() did not check the return
value offset_func->fix_fields(). It can return an error
if the data type does not support the operator "minus".
When there is no bounds on the upper or lower part of the window,
it doesn't matter if the type is numeric.
It also doesn't matter how many ORDER BY items there are in the
query.
Reviewers: Sergei Petrunia and Oleg Smirnov
The function setup_windows() called at the prepare phase of processing a
select builds a list of all window specifications used in the select. This list
is built on the statement memory and it must be done only once.
Approved by Oleksandr Byelkin <sanja@mariadb.com>
Before this patch, the code in Item_field::print() used
this convention (described in sql_explain.h:ExplainDataStructureLifetime):
- By default, the table that Item_field refers to is accessible.
- ANALYZE and SHOW {EXPLAIN|ANALYZE} may print Items after some
temporary tables have been dropped. They use
QT_DONT_ACCESS_TMP_TABLES flag. When it is ON, Item_field::print
will not access the table it refers to, if it is a temp.table
The bug was that EXPLAIN statement also may compute subqueries (depending
on subquery context and @@expensive_subquery_limit setting). After the
computation, the subquery calls JOIN::cleanup(true) which drops some of
its temporary tables. Calling Item_field::print() that refer to such table
will cause an access to free'd memory.
In this patch, we take into account that query optimization can compute
a subquery and discard its temporary tables. Item_field::print() now
assumes that any temporary table might have already been dropped.
This means QT_DONT_ACCESS_TMP_TABLES flag is not needed - we imply it is
always present.
But we also make one exception: derived tables are not freed in
JOIN::cleanup() call. They are freed later in close_thread_tables(),
at the same time when regular tables are closed.
Because of that, Item_field::print may assume that temp.tables
representing derived tables are available.
Initial patch by: Rex Jonston
Reviewed by: Monty <monty@mariadb.org>
The problematic query outlined a bug in window functions sorting
optimization. When multiple window functions are present in a query,
we sort the sorting key (as defined by PARTITION BY and ORDER BY) from
generic to specific.
SELECT RANK() OVER (ORDER BY const_col) as r1,
RANK() OVER (ORDER BY const_col, a) as r2,
RANK() OVER (PARTITION BY c) as r3,
RANK() OVER (PARTITION BY c ORDER BY b) as r4
FROM table;
For these functions, the sorting we need to do for window function
computations are: [(const_col), (const_col, a)] and [(c), (c, b)].
Instead of doing 4 different sort order, the sorts grouped within [] are
compatible and we can use the most *specific* sort to cover both window
functions.
The bug was caused by an incorrect flagging of which sort is most
specific for a compatible group of functions. In our specific test case,
instead of picking (const_col, a) as the most specific sort, it would
only sort by (const_col), which lead to wrong results for rank function.
By ensuring that we pick the last sort key before an "incompatible sort"
flag is met in our "ordered array of sorting specifications", we
guarantee correct results.
This patch is the result of running
run-clang-tidy -fix -header-filter=.* -checks='-*,modernize-use-equals-default' .
Code style changes have been done on top. The result of this change
leads to the following improvements:
1. Binary size reduction.
* For a -DBUILD_CONFIG=mysql_release build, the binary size is reduced by
~400kb.
* A raw -DCMAKE_BUILD_TYPE=Release reduces the binary size by ~1.4kb.
2. Compiler can better understand the intent of the code, thus it leads
to more optimization possibilities. Additionally it enabled detecting
unused variables that had an empty default constructor but not marked
so explicitly.
Particular change required following this patch in sql/opt_range.cc
result_keys, an unused template class Bitmap now correctly issues
unused variable warnings.
Setting Bitmap template class constructor to default allows the compiler
to identify that there are no side-effects when instantiating the class.
Previously the compiler could not issue the warning as it assumed Bitmap
class (being a template) would not be performing a NO-OP for its default
constructor. This prevented the "unused variable warning".
Window Functions code tries to minimize the number of times it
needs to sort the select's resultset by finding "compatible"
OVER (PARTITION BY ... ORDER BY ...) clauses.
This employs compare_order_elements(). That function assumed that
the order expressions are Item_field-derived objects (that refer
to a temp.table). But this is not always the case: one can
construct queries order expressions are arbitrary item expressions.
Add handling for such expressions: sort them according to the window
specification they appeared in.
This means we cannot detect that two compatible PARTITION BY clauses
that use expressions can share the sorting step.
But at least we won't crash.
SHOW EXPLAIN/ANALYZE FORMAT=JSON tries to access items that have already been
freed by a call to free_items() during THD::cleanup_after_query().
The solution is to disallow APC calls including SHOW EXPLAIN/ANALYZE
just before the call to free_items().
from view
A crash of the server happened when executing a stored procedure whose the
only query calculated window functions over a mergeable view specified
as a select from non-mergeable view. The crash could be reproduced if
the window specifications of the window functions were identical and both
contained PARTITION lists and ORDER BY lists. A crash also happened on
the second execution of the prepared statement created for such query.
If to use derived tables or CTE instead of views the problem still
manifests itself crashing the server.
When optimizing the window specifications of a window function the
server can substitute the partition lists and the order lists for
the corresponding lists from another window specification in the case
when the lists are identical. This substitution is not permanent and should
be rolled back before the second execution. It was not done and this
ultimately led to a crash when resolving the column names at the second
execution of SP/PS.
- Better, easier to read code (no used of 'random' constants).
- All defines are now unique, so it is easier to find bugs if
somethings goes wrong.
Other things:
- Created sub function of common code in Aggregator_distinct::setup() and
Item_func_group_concat::setup() that set item->marker
- More documentation
- Folded a few long lines.
- Allmost all changes in item.cc, sql_lex.cc and sql_window.cc are done
with 'replace'.
