When equality propagation (build_equal_items()) merges an equality that
contains a subquery, such as "t1.a = (SELECT ...)", with an outer join's
ON equality, it can inject a reference to that subquery into the ON
expression. If the join columns have compatible types the subquery ends
up as the constant of a multiple equality (which Item::walk() skips); if
they differ (e.g. BIGINT vs INT) the field cannot be merged and the
subquery is substituted in as a plain "tbl.col = (SELECT ...)" argument.
In the latter case, if that outer join is removed by table elimination,
mark_as_eliminated() walks the ON expression and flags the shared
Item_subselect as eliminated. The subquery, however, still lives in
another part of the query and has to be executed, tripping
DBUG_ASSERT(!eliminated) in Item_subselect::exec() (and, in release
builds, disabling the subquery cache and hiding it from EXPLAIN).
The surviving reference can be:
- a WHERE/HAVING/select-list/ORDER/GROUP expression (subquery written
there and pushed down into the eliminated ON), or
- the ON expression of an outer join that was not eliminated (subquery
written in a surviving outer ON and pushed down into an eliminated
inner one).
Fix: after table elimination, walk the expressions that survive into
execution (WHERE, HAVING, select list, ORDER/GROUP BY and the ON
expressions of outer joins that were not eliminated) and clear the
"eliminated" flag on any subquery still reachable from them.
Because a subquery can also be the constant of a multiple equality, and
Item::walk() does not visit an Item_equal's constant, Item_equal gets an
unmark_as_eliminated_processor() override that descends into its constant
explicitly.
Assisted by Claude Opus
Problem:
When hybrid function `COALESCE` is used with- a `ROW` type or a stored
routine returning a `ROW` type, server fails to report error generated
during `Item_hybrid_func::fix_attributes`. This leads to-
- Infinite loop in `cmp_row_type` in the later stage, when aggregating
type handler for hybrid field type.
- SIGABRT in `Type_handler_composite::Item_update_null_value`, when
fixing `Item_row`.
Fix:
Propogate error generated by `Item_hybrid_func::fix_attributes` during
hybrid function's fix fields time and report error early.
Problem:
Queries returning JSON from hybrid functions (`NULLIF`, `IF`, `NVL2`)
crash. This is a regression from MDEV-36716, which added early calls
to `Item_hybrid_func_fix_attributes()` inside
- `Item_func_nullif::fix_length_and_dec()` and
- `Item_func_case_abbreviation2::cache_type_info()`
to block `ROW` arguments.
Standard hybrid functions populate their `m_type_handler` by calling
`aggregate_for_result()` prior to fixing attributes. However, since
`NULLIF` and `IF` deduce their return type from a single argument,
they optimize away the `aggregate_for_result()` step.
This caused `Item_hybrid_func_fix_attributes()` to execute while
`m_type_handler` was still in its uninitialized default state,
triggering a debug assertion failure in the JSON handler.
Fix:
Explicitly set the type handler from the source argument before
calling `Item_hybrid_func_fix_attributes()`.
Item_func_ifnull and Item_func_case_abbreviation2 did not propagate
collation errors when used inside DEFAULT expressions, causing a
diagnostics area assertion instead of returning ER_CANT_AGGREGATE_2COLLATIONS.
Test cases added for IFNULL, IF, and NVL2 in mysql-test/main/create.test.
Reviewed by: Alexander Barkov
Reviewed by: Joro Kodinov
(Edited by Sergei Petrunia)
Problem: when the optimizer attempts to push a condition in form
"merged_view.not_null_column IS NULL" into a derived table other
than the merged_view itself, it may crash.
The cause is this scenario:
1. Condition pushdown code uses walk() with excl_dep_on_grouping_fields()
(or excl_dep_on_table()) to determine whether an Item expression can
be pushed. Item_args::excl_dep_on_table() assumes that any item with
const_item()==1 can be pushed anywhere.
2. Item_func_isnull(not_null_expr) will report const_item()=1 as it
will always evaluate false. However it will have non-const expression
as its argument. As described in #1, this item will be considered pushable.
3. Transformation of item for pushdown is done in "transformer function"
Item_XXX::{derived|grouping}_field_transformer_for_where().
This function will be called for Item_func_isnull's argument.
If the argument is an Item_direct_view_ref, its transformer function
will call find_matching_field_pair() (or find_producing_item())
to locate the origin of the item inside the SELECT we're pushing the
condition into. It will get a NULL pointer (origin not found) and crash.
== The fix ==
Let Item_func_is_null(not_null_expr) report const_item()==true but
let its used_tables() still report those from not_null_expr.
Then, in Item_args, make excl_dep_on_grouping_fields() and
excl_dep_on_table() to check the used tables/columns for items
that report const_item()==true but have non-zero used_tables().
Rename cloning methods of class Item and its descendants
in the following way:
(from) (to)
do_build_clone -> deep_copy
build_clone -> deep_copy_with_checks
do_get_copy -> shallow_copy
get_copy -> shallow_copy_with_checks
to better reflect their functionality.
Also make Item::deep_copy() and shallow_copy() protected.
Outside users should call deep_copy_with_checks()
and shallow_copy_with_checks().
- Removed duplicate words, like "the the" and "to to"
- Removed duplicate lines (one double sort line found in mysql.cc)
- Fixed some typos found while searching for duplicate words.
Command used to find duplicate words:
egrep -rI "\s([a-zA-Z]+)\s+\1\s" | grep -v param
Thanks to Artjoms Rimdjonoks for the command and pointing out the
spelling errors.
Parser changes made by Alexander Barkov <bar@mariadb.com>.
Part of the tests made by Iqbal Hassan <iqbal@hasprime.com>.
Initially marking with ORA_JOIN flag made also by
Iqbal Hassan <iqbal@hasprime.com>.
Main idea is that parser mark fields with (+) with a flag
(ORA_JOIN).
During Prepare the flag bring to the new created items if needed.
Later after preparing (fix_firlds()) WHERE confition the
relations betweel the tables analyzed and tables reordered
so to make JOIN/LEFT JOIN operators in chain equivalent to
the query with oracle outer join operator (+).
Then the flags of (+) removed.
- Moving the definition of "class Type_handler_row" into a new file
sql_type_row.h. Also moving *some* of its methods into sql_type_row.cc.
The rest of the methods will be moved in the patch for MDEV-34319.
Moving the definition of my_var_sp_row_field into sql_type_row.cc.
- Fixing the grammar for function_call_generic to get the first
production as "ident_cli_func" rather than "ident_func".
The upcoming patch needs to know the position of the function name
within the client query.
- Adding new data types to store data types defined by "TYPE" declarations:
* sp_type_def
* sp_type_def_list
sp_pcontext now derives from sp_type_def_list
- A new virtual method in Field:
virtual Item_field *make_item_field_spvar(THD *thd,
const Spvar_definition &def);
Using it in sp_rcontext::init_var_items().
- Fixing my_var_sp to get sp_rcontext_addr in the parameter
instead of two separate parameters (rcontext_handler + offset).
- Adding new virtual methods in my_var:
virtual bool set_row(THD *thd, List<Item> &select_list);
It's used when a select_list record is assigned to a
single composite variable, such as ROW, specified in the INTO clause.
Using it in select_dumpvar::send_data().
virtual bool check_assignability(THD *thd,
const List<Item> &select_list,
bool *assign_as_row) const;
It's used to check if the select_list is compatible with
a single INTO variable, in select_dumpvar::prepare().
- Fixing LEX methods create_outvar() to get identifiers
a Lex_ident_sys_st values instead of generic LEX_CSTRING values.
- Adding virtual methods in Type_handler:
// Used in Item_func_null_predicate::check_arguments()
virtual bool has_null_predicate() const;
// Used in LEX::sp_variable_declarations_finalize()
virtual bool sp_variable_declarations_finalize(THD *thd,
LEX *lex, int nvars,
const Column_definition &def)
const;
// Handle SELECT 1 INTO spvar;
virtual my_var *make_outvar(THD *thd,
const Lex_ident_sys_st &name,
const sp_rcontext_addr &addr,
sp_head *sphead,
bool validate_only) const;
// Handle SELECT 1 INTO spvar.field;
virtual my_var *make_outvar_field(THD *thd,
const Lex_ident_sys_st &name,
const sp_rcontext_addr &addr,
const Lex_ident_sys_st &field,
sp_head *sphead,
bool validate_only) const;
// create the value in: DECLARE var rec_t DEFAULT rec_t(1,'c');
virtual Item *make_typedef_constructor_item(THD *thd,
const sp_type_def &def,
List<Item> *arg_list) const;
- A new helper method:
Row_definition_list *Row_definition_list::deep_copy(THD *thd) const;
main/statistics_json.result is updated for f8ba5ced55 (MDEV-36099)
The test uses 'delete from t1' in many places and then populates
the table again. The natural order of rows in a MyISAM table is well
defined and the test was implicitly relying on that.
before f8ba5ced55 delete was deleting rows one by one, using
ha_myisam::delete_row() because the connection was stuck in rbr mode.
This caused rows to be shown in the reverse insertion order (because of
the delete link list).
MDEV-36099 fixes this bug and the server now correctly uses
ha_myisam::delete_all_rows(). This makes rows to be shown in the
insertion order as expected.
SQL Standard specifies that a <simple case> should return the <result>
from the left-most <simple when clause> where the comparison evaluates
to TRUE.
That is, it should not stop comparing on NULLs
Nullability is decided in two stages-
1. Based on argument NULL-ness
Problem:
- COALESCE currently uses a generic logic- "Result of a function
is nullable if any of the arguments is nullable", which is wrong.
- IFNULL sets nullability using second argument alone, which incorrectly
sets the result to NULL even when first argument is not null.
Fix:
- Result of COALESCE and IFNULL is set to NULL only if all arguments are
NULL.
2. Based on type conversion safety of fallback value
Problem:
- The generic `Item_hybrid_func_fix_attributes` logic would mark the
function's result as nullable if any argument involved a type
conversion that could yield NULL.
Fix:
- For COALESCE and IFNULL, nullability is set to NOT NULL if the first
non-null argument can be safely converted to function's target return
type.
- For other functions, if any argument's conversion to target type could
result in NULL, the function is marked nullable.
Tests included in `mysql-test/main/func_hybrid_type.test`
The patch for SYS_REFCURSOR (MDEV-20034) overrode these methods:
- Item_func_case_searched::check_arguments()
- Item_func_if::check_arguments()
to validate WHEN-style arguments (e.g. args[0] in case of IF) for being
able to return a boolean result.
However, this unintentionally removed the test for the THEN-style arguments
that they are not expressions of the ROW data type.
This led to a crash inside Type_handler_hybrid_field_type::aggregate_for_result
on a DBUG_ASSERT that arguments are not of the ROW data type.
Fix:
The fix restores blocking ROW expressions in the not supported cases,
to avoid the DBUG_ASSERT and to raise an SQL error instead.
Blocking ROW_RESULT expressions is done per Item_func_case_expression
descendant individually, instead of blocking any ROW_RESULT arguments
at the Item_func_case_expression level.
The fix is done taking into account the upcoming patch for associative arrays
(MDEV-34319). It should be possible to pass associative array expressions into
some hybrid type functions, where ROW type expressions are not possible.
As a side effect, some lecagy ER_OPERAND_COLUMNS changed to
a newer ER_ILLEGAL_PARAMETER_DATA_TYPE_FOR_OPERATION
Changes in the top affected class Item_func_case_expression:
- item_func.h:
Overriding Item_func_case_expression::check_arguments() to return false,
without checking any arguments. Descendant validate arguments
in a various different ways. No needs to block all non-scalar data type at
this level, to prevent disallowing associative arrays.
Changes in descendants:
- item_cmpfunc.cc:
Adding a test in Item_func_case_simple::aggregate_switch_and_when_arguments()
preventing passing ROW_RESULT expression in predicant and WHEN in a
simple CASE:
CASE predicant WHEN when1 THEN .. WHEN when2 THEN .. END;
This is not supported yet. Should be preferrably fixed before MDEV-34319.
- item_cmpfunc.cc:
Calling args[0]->type_handler()->Item_hybrid_func_fix_attributes()
from Item_func_nullif::fix_length_and_dec().
This prevents a ROW expression to be passed to args[0] of NULLIF().
But will allow to pass associative arrays.
args[1] is still only checked to be comparable with args[0].
No needs to add additional tests for it.
- item_cmpfunc.h:
Adding a call for Item_hybrid_func_fix_attributes() in
Item_func_case_abbreviation2::cache_type_info().
This prevents calling the descendant functions with
a ROW expression in combination with an explicit NULL
in the THEN-style arguments (but will allow to pass associative arrays):
IFNULL(row_expression, NULL)
IFNULL(NULL, row_expression)
IF(switch, row_expression, NULL)
IF(switch, NULL, row_expression)
NVL2(switch, row_expression, NULL)
NVL2(switch, NULL, row_expression)
Adding a THD* argument into involved methods.
- item_cmpfunc.h:
Overriding Item_func_case_abbreviation2_switch::check_arguments() to
check that the first argument in IF() and NVL2() can return bool.
Removing Item_func_if::check_arguments(), as it become redundant.
- sql_type.cc:
Fixing sql_type.cc not to disallow items[0] with ROW_RESULT.
This makes it call Item_hybrid_func_fix_attributes() at the end,
which block ROW arguments into THEN-style arguments of hybrid functions.
But this will allow to pass Type_handler_assoc_array expressions.
- sql_type.cc:
Changing Type_handler_row::Item_hybrid_func_fix_attributes to raise the
ER_ILLEGAL_PARAMETER_DATA_TYPE_FOR_OPERATION error instead of the DBUG_ASSERT.
This patch adds support for SYS_REFCURSOR (a weakly typed cursor)
for both sql_mode=ORACLE and sql_mode=DEFAULT.
Works as a regular stored routine variable, parameter and return value:
- can be passed as an IN parameter to stored functions and procedures
- can be passed as an INOUT and OUT parameter to stored procedures
- can be returned from a stored function
Note, strongly typed REF CURSOR will be added separately.
Note, to maintain dependencies easier, some parts of sql_class.h
and item.h were moved to new header files:
- select_results.h:
class select_result_sink
class select_result
class select_result_interceptor
- sp_cursor.h:
class sp_cursor_statistics
class sp_cursor
- sp_rcontext_handler.h
class Sp_rcontext_handler and its descendants
The implementation consists of the following parts:
- A new class sp_cursor_array deriving from Dynamic_array
- A new class Statement_rcontext which contains data shared
between sub-statements of a compound statement.
It has a member m_statement_cursors of the sp_cursor_array data type,
as well as open cursor counter. THD inherits from Statement_rcontext.
- A new data type handler Type_handler_sys_refcursor in plugins/type_cursor/
It is designed to store uint16 references -
positions of the cursor in THD::m_statement_cursors.
- Type_handler_sys_refcursor suppresses some derived numeric features.
When a SYS_REFCURSOR variable is used as an integer an error is raised.
- A new abstract class sp_instr_fetch_cursor. It's needed to share
the common code between "OPEN cur" (for static cursors) and
"OPER cur FOR stmt" (for SYS_REFCURSORs).
- New sp_instr classes:
* sp_instr_copen_by_ref - OPEN sys_ref_curor FOR stmt;
* sp_instr_cfetch_by_ref - FETCH sys_ref_cursor INTO targets;
* sp_instr_cclose_by_ref - CLOSE sys_ref_cursor;
* sp_instr_destruct_variable - to destruct SYS_REFCURSOR variables when
the execution goes out of the BEGIN..END block
where SYS_REFCURSOR variables are declared.
- New methods in LEX:
* sp_open_cursor_for_stmt - handles "OPEN sys_ref_cursor FOR stmt".
* sp_add_instr_fetch_cursor - "FETCH cur INTO targets" for both
static cursors and SYS_REFCURSORs.
* sp_close - handles "CLOSE cur" both for static cursors and SYS_REFCURSORs.
- Changes in cursor functions to handle both static cursors and SYS_REFCURSORs:
* Item_func_cursor_isopen
* Item_func_cursor_found
* Item_func_cursor_notfound
* Item_func_cursor_rowcount
- A new system variable @@max_open_cursors - to limit the number
of cursors (static and SYS_REFCURSORs) opened at the same time.
Its allowed range is [0-65536], with 50 by default.
- A new virtual method Type_handler::can_return_bool() telling
if calling item->val_bool() is allowed for Items of this data type,
or if otherwise the "Illegal parameter for operation" error should be raised
at fix_fields() time.
- New methods in Sp_rcontext_handler:
* get_cursor()
* get_cursor_by_ref()
- A new class Sp_rcontext_handler_statement to handle top level statement
wide cursors which are shared by all substatements.
- A new virtual method expr_event_handler() in classes Item and Field.
It's needed to close (and make available for a new OPEN)
unused THD::m_statement_cursors elements which do not have any references
any more. It can happen in various moments in time, e.g.
* after evaluation parameters of an SQL routine
* after assigning a cursor expression into a SYS_REFCURSOR variable
* when leaving a BEGIN..END block with SYS_REFCURSOR variables
* after setting OUT/INOUT routine actual parameters from formal
parameters.
Disallow range optimization for BETWEEN when casting one of the arguments
from STRING to a numeric type would be required to construct a range for
the query.
Adds a new method on Item_func_between called can_optimize_range_const
which allows range optimization when the types of the arguments to BETWEEN
would permit it.
During optimize_cond, we incorrectly removed the NOT LIKE condition when
attempting to remove any equality conditions.
Item_func_like's override of eq_cmp_result() returns COND_TRUE when
its collation is the binary collation. Item_bool_func2's implementation
of remove_eq_conds would then attempt to detect if both arguments were
equal to one another and return a NULL condition to optimize_conds. This
removes the condition from ever being evaluated (and
Item_func_like::val_bool is never called in this case), rendering the
incorrect result.
Fix this by checking the negated condition during eq_cmp_result() to
return either COND_FALSE in the negated==true case, or COND_TRUE in the
negated==false case which has the effect of not removing the NOT LIKE/LIKE
condition for the query.
normalize_cond() translated `WHERE col` into `WHERE col<>0`
But the opetator "not equal to 0" does not necessarily exists
for all data types.
For example, the query:
SELECT * FROM t1 WHERE inet6col;
was translated to:
SELECT * FROM t1 WHERE inet6col<>0;
which further failed with this error:
ERROR : Illegal parameter data types inet6 and bigint for operation '<>'
This patch changes the translation from `col<>0` to `col IS TRUE`.
So now
SELECT * FROM t1 WHERE inet6col;
gets translated to:
SELECT * FROM t1 WHERE inet6col IS TRUE;
Details:
1. Implementing methods:
- Field_longstr::val_bool()
- Field_string::val_bool()
- Item::val_int_from_val_str()
If the input contains bad data,
these methods raise a better error message:
Truncated incorrect BOOLEAN value
Before the change, the error was:
Truncated incorrect DOUBLE value
2. Fixing normalize_cond() to generate Item_func_istrue/Item_func_isfalse
instances instead of Item_func_ne/Item_func_eq
3. Making Item_func_truth sargable, so it uses the range optimizer.
Implementing the following methods:
- get_mm_tree(), get_mm_leaf(), add_key_fields() in Item_func_truth.
- get_func_mm_tree(), for all Item_func_truth descendants.
4. Implementing the method negated_item() for all Item_func_truth
descendants, so the negated item has a chance to be sargable:
For example,
WHERE NOT col IS NOT FALSE -- this notation is not sargable
is now translated to:
WHERE col IS FALSE -- this notation is sargable
(Review input addressed)
After this patch, the optimizer can handle virtual column expressions
in WHERE/ON clauses. If the table has an indexed virtual column:
ALTER TABLE t1
ADD COLUMN vcol INT AS (col1+1),
ADD INDEX idx1(vcol);
and the query uses the exact virtual column expression:
SELECT * FROM t1 WHERE col1+1 <= 100
then the optimizer will be able use index idx1 for it.
This is achieved by walking the WHERE/ON clauses and replacing instances
of virtual column expression (like "col1+1" above) with virtual column's
Item_field (like "vcol"). The latter can be processed by the optimizer.
Replacement is considered (and done) only in items that are potentially
usable to the range optimizer.
Make Item_func_eq of the following forms sargable by updating the relevant range
analysis methods:
1. substr(col, 1, n) = str
2. str = substr(col, 1, n)
3. left(col, n) = str
4. str = left(col, n)
where col is a indexed column and str is a const and inexpensive item
of length n.
We do this by factoring out Item_func_like::get_mm_leaf() and apply it
to a string obtained from escaping str and then appending a wildcard
"%" to it.
The addition of the two Functype enums, LEFT_FUNC and SUBSTR_FUNC,
requires changes in the spider group by handler to continue handling
LEFT and SUBSTR correctly.
Co-authored-by: Yuchen Pei <ycp@mariadb.com>
Co-authored-by: Sergei Petrunia <sergey@mariadb.com>
