MDEV-37325 unconditionally wrapped union subqueries in derived tables,
breaking name resolution of outer-scope/correlated references inside
the unions (producing ER_UNKNOWN_TABLE errors).
Delay the wrap until a following operator has a different linkage.
In Oracle mode all set operators share one priority and bind
left-to-right, so wrapping the accumulated prefix on each operator
change enforces it. This also corrects the row multiplicity of mixed
set operations toward left-to-right order.
create_priority_nest(): when the nest covers the whole prefix, point
the wrapper's first_nested at itself. Cut the prefix with
cut_next() before wrapping and re-register it on the outer unit;
Aleksey Midenkov:
In Oracle mode optimize_bag_operation() returns early, so union_distinct is
never recomputed and the stale value reaches execution.
Register the wrapper first, then run fix_distinct(): reset_distinct() now
scans the new outer chain (the wrapper alone) and correctly leaves
union_distinct NULL; a following DISTINCT operand re-establishes it.
On parsing the statement 'DECLARE var1,...,varN TYPE OF cur DEFAULT value'
an instance of the class Item_splocal that created on handling the clause
'DEFAULT value' is shared between two classes sp_instr_cursor_copy_struct and
sp_instr_set. In result, in case metadata changed and cursor has to be re-parsed,
items associated with the SP instruction sp_instr_cursor_copy_struct
are deallocated and one of deallocated items is the one representing the value
the `DEFAULT value` clause. On the other hand, the data member
sp_instr_set::m_value still references the deallocated item. It leads to server
abnormal termination on second run of a stored routine containing the statement
'DECLARE ... TYPE OF ... DEFAULT value'
To fix the issue pass owning of an instance of the class Item_splocal to
the class sp_instr_cursor_copy_struct and deallocated it explicitly on
closing the stored routine.
"NOT column" rewrites itself into "column = 0" in fix_fields().
However the rewrite is not done when the "column" is wrapped in an Item_ref.
Later, pushdown_from_having_into_where() will strip the Item_ref object
and will call
Item_func_not(Item_field())->fix_fields(thd, ref=NULL);
This call will attempt the rewrite and crash when writing into NULL pointer.
Fixed this in two places:
* Make Item_func_not::fix_fields() use real_item() and so recognize fields
wrapped in Item_ref-like objects.
* Make pushdown_from_having_into_where() pass the right pointer as
fix_fields() argument.
Function calls in INTERVAL expression of DDL have little
sense. SETVAL() fails because sequences require opened tables and
vers_set_interval() is called at earlier stage.
The fix throws ER_SUBQUERIES_NOT_SUPPORTED for sequence functions
SETVAL(), NEXTVAL(), LASTVAL() when the context does not allow
subselect (determined by clause_that_disallows_subselect).
Query optimization can merge derived tables (VIEWs being a type of derived
table) into outer queries, leaving behind stranded st_select_lex_unit objects
("stranded units") for post-query cleanup.
Previously, these were cleaned up recursively. For queries with many merged
derived tables, the deep recursion over the list of stranded units could
exhaust the stack. This change replaces the recursive cleanup with an
iterative loop to prevent stack overflows.
cleanup_stranded_units() was added at the start of
st_select_lex_unit::cleanup() by 34a8209d66. This causes a
use-after-free when nested subqueries are merged into their parent
unit. With nested subqueries like:
SELECT * FROM t1
WHERE a IN (SELECT b FROM t2
WHERE a IN (SELECT c FROM t3 WHERE FALSE HAVING c < 0));
the stranded_clean_list chains the units as: Unit1 -> Unit2 -> Unit3.
Because cleanup_stranded_units() was called first, Unit1->cleanup()
would recursively trigger Unit2->cleanup(), which in turn would
trigger Unit3->cleanup(). Unit3's cleanup frees its heap-allocated
join structures. But since Unit3 was merged into Unit2, Unit2 still
holds references to Unit3's structures (e.g., st_join_table). When
control returns to Unit2 for its own local cleanup, it accesses
already-freed memory.
Fix: move cleanup_stranded_units() to the end of cleanup(). This way,
each unit completes its own local cleanup first—clearing its
references to any child structures—before triggering cleanup of its
stranded (child) units. This enforces a parent-first cleanup order.
Affected functions:
find_matching_field_pair(Item *item, List<Field_pair> pair_list)
get_corresponding_field_pair(Item *item, List<Field_pair> pair_list)
Both only traverse the pair_list.
They use List_iterator so we can't easily switch to using const-reference.
Optimization in st_select_lex_unit::prepare() removes ORDER BY for
certain subqueries. That excludes ORDER BY items from being fixed, but
sl->window_funcs still contains window function items and those
related to optimized out ORDER BY are unfixed. The error about missing
window spec is thrown when the item is fixed. Hence we get redundant
processing of window function items without checking window spec
existence.
The fix removes the related window function items when ORDER BY is
optimized out. ORDER accumulates window_funcs at parser stage which
are then removed from SELECT_LEX::window_funcs. The fix also updates
similar optimization in mysql_make_view().
cleanup_stranded_units() was added at the start of
st_select_lex_unit::cleanup() by 34a8209d66. This causes a
use-after-free when nested subqueries are merged into their parent
unit. With nested subqueries like:
SELECT * FROM t1
WHERE a IN (SELECT b FROM t2
WHERE a IN (SELECT c FROM t3 WHERE FALSE HAVING c < 0));
the stranded_clean_list chains the units as: Unit1 -> Unit2 -> Unit3.
Because cleanup_stranded_units() was called first, Unit1->cleanup()
would recursively trigger Unit2->cleanup(), which in turn would
trigger Unit3->cleanup(). Unit3's cleanup frees its heap-allocated
join structures. But since Unit3 was merged into Unit2, Unit2 still
holds references to Unit3's structures (e.g., st_join_table). When
control returns to Unit2 for its own local cleanup, it accesses
already-freed memory.
Fix: move cleanup_stranded_units() to the end of cleanup(). This way,
each unit completes its own local cleanup first—clearing its
references to any child structures—before triggering cleanup of its
stranded (child) units. This enforces a parent-first cleanup order.
wrap_select_chain_into_derived() inserts wildcard asterisk over
wrapped SELECT. setup_wild() expands the item wildcard but it does not
update field_translation name which still contains the asterisk. Field
translation was implemented for BUG#5147 to keep view column names
intact, find_field_in_view() uses these field translation names.
Now, the fix uses full field list instead of the wildcard in
wrap_select_chain_into_derived() whenever it is possible.
Each ORDER and WHERE slot may generate split, see code like this:
if ((item->with_sum_func() && item->type() != Item::SUM_FUNC_ITEM) ||
item->with_window_func())
item->split_sum_func(thd, ref_ptrs, all_fields, SPLIT_SUM_SELECT);
Such kind of code is done in JOIN::prepare(), setup_order(),
setup_fields(), setup_group() and split_sum_func2() itself.
Since we are at the phase of ref_ptrs allocation, items are not fixed
yet and we cannot calculate precisely how much ref_ptrs is needed. We
can estimate at most how much is needed. In the worst case each window
function generates split on each ORDER BY field, GROUP BY field and
WHERE field, so the counts of these should be multiplied by window
funcs count.
As the split can be done in both setup_without_group() and
JOIN::prepare() simultaneously, the factor of window funcs should be
multiplied by 2.
The similar case may be with inner sumfunc items as of the condition
item->with_sum_func() && item->type() != Item::SUM_FUNC_ITEM
but factor of these is harder to predict at the stage of unfixed
items.
of multi-table-styled DELETE from a view
Analysis:
The item_list of builtin_select stores the fields that are there in the
RETURNING clause.
During the "EXECUTE" command, a "dummy item" is added into the item_list
of the select_lex(builtin_select) representing DELETE during
Sql_cmd_delete::precheck(). This snippet that adds a dummy item is added
because columnstore needs for temporary table. Results are put into a
temporary table and to create a temporary table we need to know what
columns are there which we get from the select_lex->item_list.
As a result, the item_list now has an item even when there is not really
RETURNING clause, resulting in execution of the setup_returning_fields()
when it should have exited already.
Fix:
Instead of checking whether builint_select's item_list is empty to
determine whether there is RETURNING clause, use a flag.
Comparison between vector and scalar is invalid (ER_OPERAND_COLUMNS)
and handled by the parser. The problem is outer_context is missing
because relink_hack() cannot recover it due to
!builtin_select.first_inner_unit() condition. This condition was set
by previous relink hack called for previous expression some(select 1).
Since there can be arbitrary number of such expressions there seems to
be no point in such a limitation. MTR test do not fail without that
condition, so the fix proposes to remove it.
Rename cloning methods of class Item and its descendants
in the following way:
(from) (to)
do_build_clone -> deep_copy
build_clone -> deep_copy_with_checks
do_get_copy -> shallow_copy
get_copy -> shallow_copy_with_checks
to better reflect their functionality.
Also make Item::deep_copy() and shallow_copy() protected.
Outside users should call deep_copy_with_checks()
and shallow_copy_with_checks().
Routine name resolution performed on a temporary memory root
during execution. So on the second execution LEX::spname members
pointed to a cleared memory.
Fixing to peform the resolution to parse time, like the CALL statement does.
During the execution time LEX::spname members now stay untouched.
Oracle mode has no priority between set-operators but the current
implementation with disable_index_if_needed() can not work with
arbitrary mix of distinct/non-distinct set-operators as the algorithm
is tied to the properly wrapped version of expression. Index can be
disabled only once and cannot be reenabled afterwards.
To adapt Oracle mode for the aforementioned implementation we
prioritize each set operator with parens in the order of
appearance. So the expression:
S1 op1 S2 op2 S3 op3 S4
will be rewritten like this:
(((S1 op1 S2) op2 S3) op3 S4).
The rewritten expression is valid for both Oracle and default mode and
renders the same result. Prioritizing is done with
create_priority_nest() but it was tweaked to work with derived to be
the first operand. Originally create_priority_nest() could wrap only
second operand, like this:
S1 op1 S2 op2 S3 -> S1 op1 (S2 op2 S3).
Now this behaviour is kept by non-NULL attach_to argument. With NULL
attach_to argument it produces this transformation:
S1 op1 S2 -> (S1 op1 S2), but to be strict it will look like this:
SELECT .. FROM (S1 op1 S2).
This expression is treated as the first one in the select list, so
nothing to "attach to".
On importance of not making fake_select_lex after
create_priority_nest() in parsed_select_expr_start(). This will fail
nastily in Protocol::valid_handler() check in store_longlong() for
EXPLAIN. This happens because select_options are inherited from newly
made fake_select_lex but without SELECT_DESCRIBE flag which is meant
to be assigned by mysql_explain_union() but union_needs_tmp_table() is
false (because now top-level unit has no set-operator, it is just
select from derived), so SELECT_DESCRIBE assignment is
skipped. Depending on that flag send_result_set_metadata() is
conditionally executed in JOIN::exec_inner(). And originally it is
done via select_describe() branch for top-level select when there is
no set operator (it is select from derived as explained above), not
via direct send_result_set_metadata().
On IS_OR_WAS_ORACLE. Sql_mode_save_for_frm_handling removes MODE_ORACLE
when view is registered or opened. We cannot return it back as this
conflicts with MDEV-12478 (tested by compat/oracle.func_concat). So we
are temporarily adding WAS_ORACLE flag instead. WAS_MODE_ORACLE is for
testing both WAS_ORACLE and MODE_ORACLE simultaneously.
Queries having the following form may cause a crash
SELECT t1.a FROM ( SELECT a AS a1 FROM t1 ) dt
JOIN t1 ON a1 LIKE EXISTS ( SELECT a + RAND () FROM t1 UNION SELECT a FROM t1);
because the table t1 has some JOIN cleanup operations performed prematurely
during the subselect.
In this particular case, the presence of RAND() makes the subquery
uncacheable, necessitating the need to execute the subquery multiple
times during join record evaluation. Each time the subquery runs, it
creates its own JOIN structure which has references to the table t1.
When the subquery completes, JOIN::cleanup and functions called by it
result in ha_end_keyread() being called on table t1. However, we are
not done with table t1 because the upper level `select t1.a from...`
query requires table t1 to be open. To solve this, we make the executor
aware of when we're in subqueries like this and delay JOIN cleanup
until the end of the query.
For prepared statements with derived tables defined by CTEs, and
during second execution, there is a dangling pointer to an instance
of a JOIN object that no longer exists.
Normally, the join member of a SELECT_LEX is freed during a call to
st_select_lex::cleanup() which recursively traverses the query tree.
But due to CTE merging during mysql_derived_merge, the unit containing
the SELECT_LEX that would've been freed is cutoff from the query tree.
We now remember all such units in a linked list so that they're
cleaned up at the end of the lifetime of the query.
- Removed duplicate words, like "the the" and "to to"
- Removed duplicate lines (one double sort line found in mysql.cc)
- Fixed some typos found while searching for duplicate words.
Command used to find duplicate words:
egrep -rI "\s([a-zA-Z]+)\s+\1\s" | grep -v param
Thanks to Artjoms Rimdjonoks for the command and pointing out the
spelling errors.
As part of optimization, we can simplify queries by merging a derived
table into it's parent instead of materializing it. When this happens
the context paths, describing how each SELECT_LEX is positioned in our
query needs updating. The best place to do this is in the call to
SELECT_LEX::exclude_level().
We follow MySQLs example here.
Approved by Sanja Byelkin (sanja@mariadb.com)
Description:
------------
When the delimiter is set to a non-default symbol and the SQL
statement contains an unquoted semicolon (;) within a MySQL-specific
comment, the SQL executes successfully in the source database.
However, the binlog record becomes incomplete, leading to a syntax
error in the replica database.
Analysis:
------------
When the delimiter is set to a non-default symbol and an SQL
statement contains an unquoted semicolon within a MySQL-specific
comment, the client transmits the entire SQL statement, including
the MySQL-specific comment, up to the delimiter to the server.
During parsing, the server interprets the semicolon as the end of the
command while processing the comment, resulting in the execution of
a partial statement. The truncated statement is then recorded in
the binary log and propagated to the replica, leading to an error.
Fix:
------------
When the delimiter is set to a non-default symbol, treat
MySQL-specific comments containing unquoted semicolons as syntax
errors and return a parser error.
Change-Id: I00d6b4ced89e79a7350c94218bf2527553054aed
Anonymous block is represented internally by the class sp_head,
so every statement inside an anonymous block is a SP instruction.
On the other hand, the anonymous block specified in the FROM clause of
the PREPARE statement is treated as a single statement. In result,
all parameter markers (represented by the character ?) are parts of
the anonymous block specified in the prepared statement and at the same
time parameter are markers, internally represented by instances of
the class Item_param and distributed among SP instructions representing
SQL statements (every SQL statement is represented by an instance of
the class sp_instr_stmt)
In case table metadata changed on running an anonymous block in prepared
statement mode, only SP instruction's statement is re-parsed. Before
re-parsing a SP's statement, all items are cleaned up including
instances of the class Item_param that represent positional parameters.
Unfortunately, this leads to presence of a dangling pointer in
Prepared_statement::param_array that references to the deleted
Item_param while invoking reset_stmt_params happening on every execution
of a prepared statement.
To fix the issue, no instances of Item_param created on re-parsings
a statement for failed SP instruction, rather instances of Item_param
left from first time parsing are re-used. As a consequence, all pointers
to instances of the class Item_param stored in the array
Prepared_statememt::param_array and possibly spread along the code base
(e.g. select_lex->limit_params.select_limit)
still point to valid Items.
Get rid of need of matherialization for usual INSERT (cache results in
Item_cache* if needed)
- subqueries in VALUE do not see new records in the table we are
inserting to
- subqueries in RETIRNING prohibited to use the table we are inserting to
failed in find_field_in_table_ref
The main crash with segfault in find_field_in_tables() was fixed by
6aa47fae30 (MDEV-35276). This fix is for debug assertion.
Since Item_default_value is also Item_field there is nothing to be
done except adding DEFAULT_VALUE_ITEM type check.
It was not possible to use a package body variable as a
fetch target:
CREATE PACKAGE BODY pkg AS
vc INT := 0;
FUNCTION f1 RETURN INT AS
CURSOR cur IS SELECT 1 AS c FROM DUAL;
BEGIN
OPEN cur;
FETCH cur INTO vc; -- this returned "Undeclared variable: vc" error.
CLOSE cur;
RETURN vc;
END;
END;
FETCH assumed that all fetch targets reside of the same sp_rcontext
instance with the cursor. This patch fixes the problem.
Now a cursor and its fetch target can reside in different sp_rcontext
instances.
Details:
- Adding a helper class sp_rcontext_addr
(a combination of Sp_rcontext_handler pointer and an offset in the rcontext)
- Adding a new class sp_fetch_target deriving from sp_rcontext_addr.
Fetch targets in "FETCH cur INTO target1, target2 ..." are now collected
into this structure instead of sp_variable.
sp_variable cannot be used any more to store fetch targets,
because it does not have a pointer to Sp_rcontext_handler
(it only has the current rcontext offset).
- Removing members sp_instr_set members m_rcontext_handler and m_offset.
Deriving sp_instr_set from sp_rcontext_addr instead.
- Renaming sp_instr_cfetch member "List<sp_variable> m_varlist"
to "List<sp_fetch_target> m_fetch_target_list".
- Fixing LEX::sp_add_cfetch() to return the pointer to the
created sp_fetch_target instance (instead of returning bool).
This helps to make the grammar in sql_yacc.c simpler
- Renaming LEX::sp_add_cfetch() to LEX::sp_add_instr_cfetch(),
as `if(sp_add_cfetch())` changed its meaning to the opposite,
to avoid automatic wrong merge from earlier versions.
- Chaning the "List<sp_variable> *vars" parameter to sp_cursor::fetch
to have the data type "List<sp_fetch_target> *".
- Changing the data type of "List<sp_variable> &vars" in
sp_cursor::Select_fetch_into_spvars::send_data_to_variable_list()
to "List<sp_fetch_target> &".
- Adding THD helper methods get_rcontext() and get_variable().
- Moving the code from sql_yacc.yy into a new LEX method
LEX::make_fetch_target().
- Simplifying the grammar in sql_yacc.yy using the new LEX method.
Changing the data type of the bison rule sp_fetch_list from "void"
to "List<sp_fetch_target> *".
