the non-recursive CTE defined with UNION
The problem appears as the columns of the non-recursive CTE weren't renamed.
The renaming procedure was called for recursive CTEs only.
To fix it in the procedure st_select_lex_unit::prepare
With_element::rename_columns_of_derived_unit is called now for both CTEs:
recursive and non-recursive.
is not supported
Allowed to use recursive references in derived tables.
As a result usage of recursive references in operands of
INTERSECT / EXCEPT is now supported.
This was done in, among other things:
- thd->db and thd->db_length
- TABLE_LIST tablename, db, alias and schema_name
- Audit plugin database name
- lex->db
- All db and table names in Alter_table_ctx
- st_select_lex db
Other things:
- Changed a lot of functions to take const LEX_CSTRING* as argument
for db, table_name and alias. See init_one_table() as an example.
- Changed some function arguments from LEX_CSTRING to const LEX_CSTRING
- Changed some lists from LEX_STRING to LEX_CSTRING
- threads_mysql.result changed because process list_db wasn't always
correctly updated
- New append_identifier() function that takes LEX_CSTRING* as arguments
- Added new element tmp_buff to Alter_table_ctx to separate temp name
handling from temporary space
- Ensure we store the length after my_casedn_str() of table/db names
- Removed not used version of rename_table_in_stat_tables()
- Changed Natural_join_column::table_name and db_name() to never return
NULL (used for print)
- thd->get_db() now returns db as a printable string (thd->db.str or "")
TRASH was mapped to TRASH_FREE and was supposed to be used for memory
that should not be accessed anymore, while TRASH_ALLOC() is to be
used for uninitialized but to-be-used memory.
But sometimes TRASH() was used in the latter sense.
Remove TRASH() macro, always use explicit TRASH_ALLOC() or TRASH_FREE().
with recursive reference in subquery
If a recursive CTE uses a subquery with recursive reference then
the virtual function reset() must be called after each iteration
performed at the execution of the CTE.
Most "new" failures fixed in the following files:
- sql_select.cc
- item.cc
- item_func.cc
- opt_subselect.cc
Other things:
- Allocate udf_handler strings in mem_root
- Required changes in sql_string.h
- Add mem_root as argument to some new [] calls
- Mark udf_handler strings as thread specific
- Removed some comment blocks with code
With INTERSECT/EXCEPT fact that subquery item of IN/ALL/ANY was not assigned value does not mean that temporary table used for calculating unit is empty (records could be deleted).
With INTERSECT/EXCEPT fact that subquery item of IN/ALL/ANY was not assigned value does not mean that temporary table used for calculating unit is empty (records could be deleted).
As a result of this merge the code for the following tasks appears in 10.3:
- MDEV-12172 Implement tables specified by table value constructors
- MDEV-12176 Transform [NOT] IN predicate with long list of values INTO
[NOT] IN subquery.
- Added sql/mariadb.h file that should be included first by files in sql
directory, if sql_plugin.h is not used (sql_plugin.h adds SHOW variables
that must be done before my_global.h is included)
- Removed a lot of include my_global.h from include files
- Removed include's of some files that my_global.h automatically includes
- Removed duplicated include's of my_sys.h
- Replaced include my_config.h with my_global.h
"Optimization for equi-joins of derived tables with GROUP BY"
should be considered rather as a 'proof of concept'.
The task itself is targeted at an optimization that employs re-writing
equi-joins with grouping derived tables / views into lateral
derived tables. Here's an example of such transformation:
select t1.a,t.max,t.min
from t1 [left] join
(select a, max(t2.b) max, min(t2.b) min from t2
group by t2.a) as t
on t1.a=t.a;
=>
select t1.a,tl.max,tl.min
from t1 [left] join
lateral (select a, max(t2.b) max, min(t2.b) min from t2
where t1.a=t2.a) as t
on 1=1;
The transformation pushes the equi-join condition t1.a=t.a into the
derived table making it dependent on table t1. It means that for
every row from t1 a new derived table must be filled out. However
the size of any of these derived tables is just a fraction of the
original derived table t. One could say that transformation 'splits'
the rows used for the GROUP BY operation into separate groups
performing aggregation for a group only in the case when there is
a match for the current row of t1.
Apparently the transformation may produce a query with a better
performance only in the case when
- the GROUP BY list refers only to fields returned by the derived table
- there is an index I on one of the tables T used in FROM list of
the specification of the derived table whose prefix covers the
the fields from the proper beginning of the GROUP BY list or
fields that are equal to those fields.
Whether the result of the re-writing can be executed faster depends
on many factors:
- the size of the original derived table
- the size of the table T
- whether the index I is clustering for table T
- whether the index I fully covers the GROUP BY list.
This patch only tries to improve the chosen execution plan using
this transformation. It tries to do it only when the chosen
plan reaches the derived table by a key whose prefix covers
all the fields of the derived table produced by the fields of
the table T from the GROUP BY list.
The code of the patch does not evaluates the cost of the improved
plan. If certain conditions are met the transformation is applied.
TVC can be used in UNION-statement, in view and in subquery.
Files where TVC is defined and its methods are stored added.
Methods exec and prepare for TVC added.
Tests for TVC added.
prepare of "fake_select" for union made in JOIN::prepare only if
we do not execute it then before reset, i.e it was for PS prepare
and now required for CREATE VIEW to make global ORDER BY which
belongs to "fake_select" prepared.
GROUP BY
Issue 1:
--------
This problem occurs in the following conditions:
1) A UNION is present in the subquery of select list and
handles multiple columns.
2) Query has a GROUP BY.
A temporary table is created to handle the UNION.
Item_field objects are based on the expressions of the
result of the UNION (ie. the fake_select_lex). While
checking validity of the columns in the GROUP BY list, the
columns of the temporary table are checked in
Item_ident::local_column. But the Item_field objects
created for the temporary table don't have information like
the Name_resolution_context that they belong to or whether
they are dependent on an outer query. Since these members
are null, incorrect behavior is caused.
This can happen when such Item objects are cached to apply
the IN-to-EXISTS transform for Item_row.
Solution to Issue 1:
--------------------
Context information of the first select in the UNION will
be assigned to the new Item_field objects.
Issue 2:
--------
This problem occurs in the following conditions:
1) A UNION is present in the subquery of select list.
2) A column in the UNION's first SELECT refers to a table
in the outer-query making it a dependent union.
3) GROUP BY column refers to the outer-referencing column.
While resolving the select list with an outer-reference, an
Item_outer_ref object is created to handle the
outer-query's GROUP BY list. The Item_outer_ref object
replaces the Item_field object in the item tree.
Item_outer_ref::fix_fields will be called only while fixing
the inner references of the outer query.
Before resolving the outer-query, an Item_type_holder
object needs to be created to handle the UNION. But as
explained above, the Item_outer_ref object has not been
fixed yet. Having a fixed Item object is a pre-condition
for creating an Item_type_holder.
Solution to Issue 2:
--------------------
Use the reference (real_item()) of an Item_outer_ref object
instead of the object itself while creating an
Item_type_holder.
Introducing a new class Type_holder (used internally in sql_union.cc),
to reuse exactly the same data type attribute aggregation Type_handler API
for hybrid functions and UNION.
This fixes a number of bugs in UNION:
- MDEV-9495 Wrong field type for a UNION of a signed and an unsigned INT expression
- MDEV-9497 UNION and COALESCE produce different field types for DECIMAL+INT
- MDEV-12594 UNION between fixed length double columns does not always preserve scale
- MDEV-12595 UNION converts INT to BIGINT
- MDEV-12599 UNION is not symmetric when mixing INT and CHAR
Details:
- sql_union.cc: Reusing attribute aggregation for UNION.
Adding new methods:
* st_select_lex_unit::join_union_type_handlers()
* st_select_lex_unit::join_union_type_attributes()
* st_select_lex_unit::join_union_item_types()
Removing the old join_types()-based code.
- Changing Type_handler::Item_hybrid_func_fix_attributes()
to accept "name", Type_handler_hybrid_field_type, Type_all_attributes
as three separate parameters instead of a single Item_hybrid_func parameter,
to make it possible to pass both Item_hybrid_func and Type_holder.
- Moving the former special GEOMETRY and ENUM/SET attribute aggregation code
from Item_type_holder::join_types() to
* Type_handler_typelib::Item_hybrid_func_fix_attributes().
* Type_handler_geometry::Item_hybrid_func_fix_attrubutes().
This makes GEOMETRY/ENUM/SET symmetric with all other data types
(from the UNION point of view).
Removing Item_type_holder::join_types() and Item_type_holder::get_full_info().
- Adding new methods into Type_all_attributes:
* Type_all_attributes::set_geometry_type() and
Item_hybrid_func::set_geometry_type().
* Adding Type_all_attributes::get_typelib().
* Adding Type_all_attributes::set_typelib().
- Adding Type_handler_typelib as a common parent for
Type_handler_enum and Type_handler_set, to avoid code duplication: they have
already had two common methods, and we're adding one more shared method.
- Adding Type_all_attributes::set_maybe_null(), as some type handlers
may want to set maybe_null (e.g. Type_handler_geometry) during data type
attribute aggregation.
- Changing Type_geometry_attributes() to accept Type_handler
and Type_all_attributes as two separate parameters, instead
of a single Item parameter, to make it possible to pass Type_holder.
- Adding Item_args::add_argument().
- Moving Item_args::alloc_arguments() from "protected" to "public".
- Moving Item_type_holder::Item_type_holder() from item.cc to item.h, as
now it's very simple.
Btw, this constructor should probably be eventually removed.
It's now used only in sql_show.cc, which could be modified to use
Item_return_decimal (for symmetry with Item_return_xxx created for all
other data types). Or, another option: remove all Item_return_xxx and
use Item_type_holder for all data types instead.
- storage/tokudb/mysql-test/tokudb/r/type_float.result
Recording new results (MDEV-12594).
- mysql-test/r/cte_recursive.result
Recording new results (MDEV-9497)
- mysql-test/r/subselect*.result
Recording new results (MDEV-12595)
- mysql-test/r/metadata.result
Recording new results (MDEV-9495)
- mysql-test/r/temp_table.result
Recording new results (MDEV-12594)
- mysql-test/r/type_float.result
Recording new results (MDEV-12594)
When the rows produced on the current iteration are sent to the
temporary table T of the UNION type created for CTE the rows
that were not there simultaneously are sent to the temporary
table D that contains rows for the next iteration. The test
whether a row was in T checks the return code of writing into T.
If just a HEAP table is used for T then the return code is
HA_ERR_FOUND_DUPP_KEY, but if an ARIA table is used for T then
the return code is HA_ERR_FOUND_DUPP_UNIQUE.
The implementation of select_union_recursive::send_data()
erroneously checked only for the first return code. So if an Aria
table was used for T then all rows produced by the current iteration
went to D and and in most cases D grew with each iteration.
Whether T has reached stabilization is detected by
checking whether D is empty. So as a result, the iterations were
never stopped unless a limit for them was set.
Fixed by checking for both HA_ERR_FOUND_DUPP_KEY and
HA_ERR_FOUND_DUPP_UNIQUE as return codes returned by
the function writing a row into the temporary table T.
