This bug could cause a crash when executing queries that used mutually
recursive CTEs with system variable big_tables set to 1. It happened due
to several bugs in the code that handled recursive table references
referred mutually recursive CTEs. For each recursive table reference a
temporary table is created that contains all rows generated for the
corresponding recursive CTE table on the previous step of recursion.
This temporary table should be created in the same way as the temporary
table created for a regular materialized derived table using the
method select_union::create_result_table(). In this case when the
temporary table is created it uses the select_union::TMP_TABLE_PARAM
structure as the parameter for the table construction. However the
code created the temporary table using just the function create_tmp_table()
and passed pointers to certain fields of the TMP_TABLE_PARAM structure
used for accumulation of rows of the recursive CTE table as parameters
for update. This was a mistake because now different temporary tables
cannot share some TMP_TABLE_PARAM fields in a general case. Besides,
depending on how mutually recursive CTE tables were defined and which
of them were referred in the executed query the select_union object
allocated for a recursive table reference could be allocated again after
the the temporary table had been created. In this case the TMP_TABLE_PARAM
object associated with the temporary table created for the recursive
table reference contained unassigned fields needed for execution when
Aria engine is employed as the engine for temporary tables.
This patch ensures that
- select_union object is created only once for any recursive table
reference
- any temporary table created for recursive CTEs uses its own
TMP_TABLE_PARAM structure
The patch also fixes a problem caused by incomplete cleanup of join tables
associated with recursive table references.
Approved by Oleksandr Byelkin <sanja@mariadb.com>
When the query using a recursive CTE whose definition contained wildcard
symbols in the recursive part was processed at the prepare stage an
assertion was hit if the query was executed without any default database
set. The failure happened when the function insert_fields() tried to check
column privileges for the temporary table created for a recursive
reference to the CTE. No acl checks are needed for any CTE. That's why this
check should be blocked as well. The patch formulates a stricter condition
at which this check is to be blocked that covers the case when a query
using recursive CTEs is executed with no default database set.
Approved by Oleksandr Byelkin <sanja@mariadb.com>
Due to a premature cleanup of the unit that specified a recursive CTE
used in the second operand of union the server fell into an infinite
loop in the reported test case. In other cases this premature cleanup
could cause other problems.
The bug is the result of a not quite correct fix for MDEV-17024. The
unit that specifies a recursive CTE has to be cleaned only after the
cleanup of the last external reference to this CTE. It means that
cleanups of the unit triggered not by the cleanup of a external
reference to the CTE must be blocked.
Usage of local table chains in selects to get external references to
recursive CTEs was not correct either because of possible merges of
some selects.
Also fixed a minor bug in st_select_lex::set_explain_type() that caused
typing 'RECURSIVE UNION' instead of 'UNION' in EXPLAIN output for external
references to a recursive CTE.
This bug is the same as the bug MDEV-17024. The crashes caused by these
bugs were due to premature cleanups of the unit specifying recursive CTEs
that happened in some cases when there were several outer references the
same recursive CTE.
The problem of premature cleanups for recursive CTEs could be already
resolved by the correction in TABLE_LIST::set_as_with_table() introduced
in this patch. ALL other changes introduced by the patches for MDEV-17024
and MDEV-22748 guarantee that this clean-ups are performed as soon as
possible: when the select containing the last outer reference to a
recursive CTE is being cleaned up the specification of the recursive CTE
should be cleaned up as well.
When processing a query with a recursive CTE a temporary table is used for
each recursive reference of the CTE. As any temporary table it uses its own
mem-root for table definition structures. Due to specifics of the current
implementation of ANALYZE stmt command this mem-root can be freed only at
the very of query processing. Such deallocation of mem-root memory happens
in close_thread_tables(). The function looks through the list of the tmp
tables rec_tables attached to the THD of the query and frees corresponding
mem-roots. If the query uses a stored function then such list is created
for each query of the function. When a new rec_list has to be created the
old one has to be saved and then restored at the proper moment.
The bug occurred because only one rec_list for the query containing CTE was
created. As a result close_thread_tables() freed tmp mem-roots used for
rec_tables prematurely destroying some data needed for the output produced
by the ANALYZE command.
This allows one to run the test suite even if any of the following
options are changed:
- character-set-server
- collation-server
- join-cache-level
- log-basename
- max-allowed-packet
- optimizer-switch
- query-cache-size and query-cache-type
- skip-name-resolve
- table-definition-cache
- table-open-cache
- Some innodb options
etc
Changes:
- Don't print out the value of system variables as one can't depend on
them to being constants.
- Don't set global variables to 'default' as the default may not
be the same as the test was started with if there was an additional
option file. Instead save original value and reset it at end of test.
- Test that depends on the latin1 character set should include
default_charset.inc or set the character set to latin1
- Test that depends on the original optimizer switch, should include
default_optimizer_switch.inc
- Test that depends on the value of a specific system variable should
set it in the test (like optimizer_use_condition_selectivity)
- Split subselect3.test into subselect3.test and subselect3.inc to
make it easier to set and reset system variables.
- Added .opt files for test that required specfic options that could
be changed by external configuration files.
- Fixed result files in rockdsb & tokudb that had not been updated for
a while.
When the with clause of a query contains a recursive CTE that is not used
then processing of EXPLAIN for this query does not require optimization
of the unit specifying this CTE. In this case if 'derived' is the
TABLE_LIST object created for this CTE then derived->derived_result is NULL
and any assignment to derived->derived_result->table causes a crash.
After fixing this problem in the code of st_select_lex_unit::prepare()
EXPLAIN for such a query worked without crashes. Yet an execution
plan for the recursive CTE appeared there. The cause of this problem was
an incorrect condition used in JOIN::save_explain_data_intern() that
determined whether CTE was to be optimized or not. A similar condition was
used in select_describe() and this patch has corrected it as well.
This bug in the code of the function With_element::check_unrestricted_recursive()
could force a recursive CTE to be executed in a non-standard compliant mode
in which recursive UNION ALL could lead to an infinite execution. This
problem could occur only in the case when this CTE was used by another
recursive CTE at least twice.
The function st_select_lex_unit::exec_recursive() missed resetting of
select_limit_cnt and offset_limit_cnt before execution of union parts.
As a result recursive CTEs specified by UNIONs whose SELECTs contained
LIMIT/OFFSET could return wrong sets of records.
This problem manifested itself when a join query used two or more
materialized CTE such that each of them employed the same recursive CTE.
The bug caused a crash. The crash happened because the cleanup()
function was performed premature for recursive CTE. This clean up was
induced by the cleanup of the first CTE referenced the recusrsive CTE.
This cleanup destroyed the structures that would allow to read from the
temporary table containing the rows of the recursive CTE and an attempt to read
these rows for the second CTE referencing the recursive CTE triggered a
crash.
The clean up for a recursive CTE R should be performed after the cleanup
of the last materialized CTE that uses R.
1. The changed variant did not fail without the patch for MDEV-16629
while the original test case did fail.
2. In any case the test case should go to cte_recursive_not_embedded.test
that was not created yet.
At the end of a test, 'connection default' should be in a usable state.
This was not the case, because there was a preceding 'send' without a
'reap'. If 'reap' was added, an error would be reported because the
server was restarted after the 'send'. It is easiest to 'send' from a
separate connection and do the restart from 'connection default'.
When processing a query containing with clauses a call of the function
check_dependencies_in_with_clauses() before opening tables used in the
query is necessary if with clauses include specifications of recursive
CTEs.
This call was missing if such a query belonged to a stored function.
This caused misbehavior of the server: it could report a fake error
as in the test case for MDEV-16629 or the executed query could hang
as in the test cases for MDEV-16661 and MDEV-15151.
The current code does not support recursive CTEs whose specifications
contain a mix of ALL UNION and DISTINCT UNION operations.
This patch catches such specifications and reports errors for them.
with recursive subquery
There were two problems:
1. The code did not report that usage of global ORDER BY / LIMIT clauses
was not supported yet.
2. The code just reset fake_select_lex of the the unit specifying
a recursive CTE to NULL and that caused memory leaks in some cases.
It has been done to demonstrate that the fix of this bug is good for 10.3
as well. The previous test case is not good for this purpose because
10.2 and 10.3 use different rules for determining the types of recursive
CTEs.
Usage of aggregate/window functions in non-recursive parts of recursive CTEs
is allowed. Error messages complaining about this were reported by mistake.
This bug manifested itself when the optimizer chose an execution plan with
an access of the recursive CTE in a recursive query by key and ARIA/MYISAM
temporary tables were used to store recursive tables.
The problem appeared due to passing an incorrect parameter to the call of
instantiate_tmp_table() in the function With_element::instantiate_tmp_tables().
This bug happened due to a defect of the implementation of the handler
function ha_delete_all_rows() for the ARIA engine.
The function maria_delete_all_rows() truncated the table, but it didn't
touch the write cache, so the cache's write offset was not reset.
In the scenario like in the function st_select_lex_unit::exec_recursive
when first all records were deleted from the table and then several new
records were added some metadata became inconsistent with the state of
the cache. As a result the table scan function could not read records
at the end of the table.
The same defect could be found in the implementation of ha_delete_all_rows()
for the MYISAM engine mi_delete_all_rows().
Additionally made late instantiation for the temporary table used to store
rows that were used for each new iteration when executing a recursive CTE.
with recursive reference in subquery
If a recursive CTE uses a subquery with recursive reference then
the virtual function reset() must be called after each iteration
performed at the execution of the CTE.
During the user-defined variable defined by the recursive CTE handling procedure
check_dependencies_in_with_clauses that checks dependencies between the tables
that are defined in the CTE and find recursive definitions wasn't called.
The support of embedded CTEs was not correct in the cases when
embedded CTEs were used multiple times. The problems occurred with
both non-recursive (bug mdev-13780) and recursive (bug mdev-14184)
embedded CTEs.
The bug happened when the specification of a recursive CTE had
no recursive references at the top level of the specification.
In this case the regular processing of derived table references
of the select containing a non-recursive reference to this
recursive CTE misses handling the specification unit.
At the preparation stage any non-recursive reference to a
recursive CTE must be handled after the preparation of the
specification unit for this CTE. So we have to force this
preparation when regular handling of derived tables does not
do it.
When the rows produced on the current iteration are sent to the
temporary table T of the UNION type created for CTE the rows
that were not there simultaneously are sent to the temporary
table D that contains rows for the next iteration. The test
whether a row was in T checks the return code of writing into T.
If just a HEAP table is used for T then the return code is
HA_ERR_FOUND_DUPP_KEY, but if an ARIA table is used for T then
the return code is HA_ERR_FOUND_DUPP_UNIQUE.
The implementation of select_union_recursive::send_data()
erroneously checked only for the first return code. So if an Aria
table was used for T then all rows produced by the current iteration
went to D and and in most cases D grew with each iteration.
Whether T has reached stabilization is detected by
checking whether D is empty. So as a result, the iterations were
never stopped unless a limit for them was set.
Fixed by checking for both HA_ERR_FOUND_DUPP_KEY and
HA_ERR_FOUND_DUPP_UNIQUE as return codes returned by
the function writing a row into the temporary table T.
This patch fixed some problems that occurred with subqueries that
contained directly or indirectly recursive references to recursive CTEs.
1. A [NOT] IN predicate with a constant left operand and a non-correlated
subquery as the right operand used in the specification of a recursive CTE
was considered as a constant predicate and was evaluated only once.
Now such a predicate is re-evaluated after every iteration of the process
that produces the records of the recursive CTE.
2. The Exists-To-IN transformation could be applied to [NOT] IN predicates
with recursive references. This opened a possibility of materialization
for the subqueries used as right operands. Yet, materialization
is prohibited for the subqueries if they contain a recursive reference.
Now the Exists-To-IN transformation cannot be applied for subquery
predicates with recursive references.
The function st_select_lex::check_subqueries_with_recursive_references()
is called now only for the first execution of the SELECT.
The function st_select_lex_unit::exec_recursive() incorrectly determined
that a CTE mutually recursive with some others was stabilized in the case
when the non-recursive part of the CTE returned an empty set. As a result
the server fell into an infinite loop when executing a query using
this CTE.
Mutually recursive CTE could cause a crash of the server in the case
when they were not Standard compliant. The crash happened in
mysql_derived_prepare(), because the destructor the derived_result
object created for a CTE that was mutually recursive with some others
was called twice. Yet this destructor should not be called for resursive
references.
The method With_element::check_unrestricted_recursive() icorrectly performed
the check that no recursive reference is not encountered in inner parts of
outer joins. As a result the server reported errors for valid specifications
with outer joins.
The temporary tables created for recursive table references
should be closed in close_thread_tables(), because they might
be used in the statements like ANALYZE WITH r AS (...) SELECT * from r
where r is defined through recursion.
1. The rows of a recursive CTE at some point may overflow
the HEAP temporary table containing them. At this point
the table is converted to a MyISAM temporary table and the
new added rows are placed into this MyISAM table.
A bug in the of select_union_recursive::send_data prevented
the server from writing the row that caused the overflow
into the temporary table used for the result of the iteration
steps. This could lead, in particular,to a premature end
of the iterations.
2. The method TABLE::insert_all_rows_into() that was used
to copy all rows of one temporary table into another
did not take into account that the destination temporary
table must be converted to a MyISAM table at some point.
This patch fixed this problem. It also renamed the method
into TABLE::insert_all_rows_into_tmp_table() and added
an extra parameter needed for the conversion.
The bug was in the code of the recursive method
With_element::check_unrestricted_recursive. For recursive
calls of this method sel->get_with_element()->owner != owner.
The server missed to call check_dependencies_in_with_clauses()
when processing PREPARE ... FROM CREATE ... SELECT / INSERT ... SELECT
with WITH clause before SELECT.
When a prepared statement uses a CTE definition with a column list
renaming of columns of the CTE expression must be performed
for every execution of the prepared statement.
