Virtial_tmp_table did not set the "field_index" member for its Fields.
Fixing Virtual_tmp_table::add() to set "field_index" to the Field's ordinal position
inside the table, like a normal TABLE does, for consistency.
Although, this flaw did not seem to cause any bugs, having field_index properly
set is helpful for debugging purposes.
After MDEV-14212, the Virtual_tmp_table instance that stores a ROW
variable elements is accessible from the underlying Field_row
(rather than Item_field_row).
This patch makes some further changes by moving the code from
sp_instr_xxx, sp_rcontext, Item_xxx to Virtual_tmp_table and Field_xxx.
The data type specific code (scalar vs ROW) now resides in
a new virtual method Field_xxx::sp_prepare_and_store_item().
The the code in sp_rcontext::set_variable() and sp_eval_expr()
is now symmetric for scalar and ROW values.
The code in sp_rcontext::set_variable_row_field(), sp_rcontext::set_variable_row_field(), sp_rcontext::set_variable_row()
is now symmetric for ROW elements (i.e. scalar and ROW elements inside a ROW).
Rationale:
Prepare the code to implement these tasks soon easier:
- MDEV-12252 ROW data type for stored function return values
- MDEV-12307 ROW data type for built-in function return values
- MDEV-6121 Data type: Array
- MDEV-10593 sql_mode=ORACLE: TYPE .. AS OBJECT: basic functionality
- ROW with ROW fields (no MDEV yet)
Details:
1. Moving the code in sp_eval_expr() responsible to backup/restore
thd->count_cuted_fields, thd->abort_on_warning,
thd->transaction.stmt.modified_non_trans_table
into a new helper class Sp_eval_expr_state, to reuse it easier.
Fixing sp_eval_expr() to use this new class.
2. Moving sp_eval_expr() and sp_prepare_func_item() from public functions
to methods in THD, so they can be reused in *.cc files easier without
a need to include "sp_head.h".
Splitting sp_prepare_func_item() into two parts.
Adding a new function sp_fix_func_item(), which fixes
the underlying items, but does not do check_cols() for them.
Reusing sp_fix_func_item() in Field_row::sp_prepare_and_store_item().
3. Moving the code to find ROW fields by name from Item to Virtual_tmp_table
Moving the code searching for ROW fields by their names
from Item_field_row::element_index_by_name() to a new method
Item_field_row to Virtual_tmp_table::sp_find_field_by_name().
Adding wrapper methods sp_rcontext::find_row_field_by_name() and
find_row_field_by_name_or_error(), to search for a ROW variable
fields by the variable offset and its field name.
Changing Item_splocal_row_field_by_name::fix_fields() to do
use sp_rcontext::find_row_field_by_name_or_error().
Removing virtual Item::element_index_by_name().
4. Splitting sp_rcontext::set_variable()
Adding a new virtual method Field::sp_prepare_and_store_item().
Spliting the two branches of the code in sp_rcontext::set_variable()
into two virtual implementations of Field::sp_prepare_and_store_item(),
(for Field and for Field_row).
Moving the former part of sp_rcontext::set_variable() with the loop
doing set_null() for all ROW fields into a new method
Virtual_tmp_table::set_all_fields_to_null() and using it in
Field_row::sp_prepare_and_store_item().
Moving the former part of sp_rcontext::set_variable() with the loop
doing set_variable_row_field() into a new method
Virtual_tmp_table::set_all_fields_from_item() and using it in
Field_row::sp_prepare_and_store_item().
The loop in the new method now uses sp_prepare_and_store_item()
instead of set_variable_row_field(), because saving/restoring
THD flags is now done on the upper level. No needs to save/restore
on every iteration.
5. Fixing sp_eval_expr() to simply do two things:
- backup/restore THD flags
- call result_field->sp_prepare_and_store_item()
So now sp_eval_expr() can be used for both scalar and ROW variables.
Reusing it in sp_rcontext::set_variable*().
6. Moving the loop in sp_rcontext::set_variable_row() into a
new method Virtual_tmp_table::sp_set_all_fields_from_item_list().
Changing the loop body to call field->sp_prepare_and_store_item()
instead of doing set_variable_row_field(). This removes
saving/restoring of the THD flags from every interation.
Instead, adding the code to save/restore the flags around
the entire loop in set_variable_row(), using Sp_eval_expr_state.
So now saving/restoring is done only once for the entire ROW
(a slight performance improvement).
7. Removing the code in sp_instr_set::exec_core() that sets
a variable to NULL if the value evaluation failed.
sp_rcontext::set_variable() now makes sure to reset
the variable properly by effectively calling sp_eval_expr(),
which calls virtual Field::sp_prepare_and_store_item().
Removing the similar code from sp_instr_set_row_field::exec_core()
and sp_instr_set_row_field_by_name::exec_core().
Removing the method sp_rcontext::set_variable_row_field_to_null(),
as it's not used any more.
8. Removing the call for sp_prepare_func_item() from
sp_rcontext::set_variable_row_field(), as it was duplicate:
it was done inside sp_eval_expr(). Now it's done inside
virtual Field::sp_prepare_and_store_item().
9. Moving the code from sp_instr_set_row_field_by_name::exec_core()
into sp_rcontext::set_variable_row_field_by_name(), for symmetry
with other sp_instr_set*::exec_core()/sp_rcontext::set_variable*() pairs.
Now sp_instr_set_row_field_by_name::exec_core() calls
sp_rcontext::set_variable_row_field_by_name().
10. Misc:
- Adding a helper private method sp_rcontext::virtual_tmp_table_for_row(),
reusing it in a new sp_rcontext methods.
- Removing Item_field_row::get_row_field(), as it's not used any more.
- Removing the "Item *result_item" from sp_eval_expr(),
as it's not needed any more.
The assertion failure was caused by an incorrectly set read_set for
functions in the ORDER BY clause in part of a union, when we are using
a mergeable view and the order by clause can be skipped (removed).
An order by clause can be skipped if it's part of one part of the UNION as
the result set is not meaningful when multiple SELECT queries are UNIONed. The
server is aware of this optimization and tries to remove the order by
clause before JOIN::prepare. The problem is that we need to throw an
error when the ORDER BY clause contains invalid columns. To do this, we
attempt resolving the ORDER BY expressions, then subsequently drop them
if resolution succeeded. However, ORDER BY resolution had the side
effect of adding the expressions to the all_fields list, which is used
to construct temporary tables to store the result. We may be ignoring
the ORDER BY statement, but the tmp table still tried to compute the
values for the expressions, even if the columns are never used.
The assertion only shows itself if the order by clause contains members
which were not previously in the select list, and are part of a
function.
There is an additional question as to why this only manifests when using
VIEWS and not when using a regular table. The difference lies with the
"reset" of the read_set for the temporary table during
SELECT_LEX::update_used_tables() in JOIN::optimize(). The changes
introduced in fdf789a7ea cleared the
read_set when a mergeable view is encountered in the TABLE_LIST
defintion.
Upon initial order_list resolution, the table's read_set is updated
correctly. JOIN::optimize() will only reset the read_set if it
encounters a VIEW. Since we no longer have ORDER BY clause in
JOIN::optimize() we never get to correctly update the read_set again.
Other relevant commit by Timour, which first introduced the order
resolution when we "can_skip_sort_order":
883af99e7d
Solution:
Don't add the resolved ORDER BY elements to all_fields. We only resolve
them to check if an error should be returned for the query. Ignore them
completely otherwise.
in joined table + GROUP BY + GROUP_CONCAT + HAVING + ORDER BY
[by field from HAVING] + 1 row expected
The fix is actually a port of the fix for bug #17055185 from
mysql code line (see commit f289aeeef0743508ff87211084453b3b88a6d017
by Mithun C Y into mysql-5.6). The test case for the bug #17055185
was also ported.
Changing datatypes for:
- Item_spvar_args::m_table
- sp_rcontext::m_var_table
- return value of create_virtual_tmp_table()
from TABLE* to Virtual_tmp_table*
Advantages:
- Stricter data type control
- Removing the duplicate code (a loop with free_blobs)
from destructors ~sp_rcontext() and ~Item_spvar_args(),
using "delete m_(var_)table" in both instead.
- Using Virtual_tmp_table::delete makes the code call Field::delete,
which calls TRASH() for the freed fields,
which is good for valgrind test runs.
Currently condition pushdown into materialized views / derived tables
is not implemented yet (see mdev-12387) and grouping views are
optimized early when subqueries are converted to semi-joins in
convert_join_subqueries_to_semijoins(). If a subquery that is converted
to a semi-join uses a grouping view this view is optimized in two phases.
For such a view V only the first phase of optimization is done after
the conversion of subqueries of the outer join into semi-joins.
At the same time the reference of the view V appears in the join
expression of the outer join. In fixed code there was an attempt to push
conditions into this view and to optimize it after this. This triggered
the second phase of the optimization of the view and it was done
prematurely. The second phase of the optimization for the materialized
view is supposed to be called after the splitting condition is pushed
into the view in the call of JOIN::improve_chosen_plan for the outer
join.
The fix blocks the attempt to push conditions into splittable views
if they have been already partly optimized and the following
optimization for them.
The test case of the patch shows that the code for mdev-13369
basically supported the splitting technique for materialized views /
derived tables.
The patch also replaces the name of the state JOIN::OPTIMIZATION_IN_STAGE_2
for JOIN::OPTIMIZATION_PHASE_1_DONE and fixes a bug in
TABLE_LIST::fetch_number_of_rows()
with window functions (mdev-10855).
This patch just modified the function pushdown_cond_for_derived()
to support this feature.
Some test cases demonstrating this optimization were added to
derived_cond_pushdown.test.
"Optimization for equi-joins of derived tables with GROUP BY"
should be considered rather as a 'proof of concept'.
The task itself is targeted at an optimization that employs re-writing
equi-joins with grouping derived tables / views into lateral
derived tables. Here's an example of such transformation:
select t1.a,t.max,t.min
from t1 [left] join
(select a, max(t2.b) max, min(t2.b) min from t2
group by t2.a) as t
on t1.a=t.a;
=>
select t1.a,tl.max,tl.min
from t1 [left] join
lateral (select a, max(t2.b) max, min(t2.b) min from t2
where t1.a=t2.a) as t
on 1=1;
The transformation pushes the equi-join condition t1.a=t.a into the
derived table making it dependent on table t1. It means that for
every row from t1 a new derived table must be filled out. However
the size of any of these derived tables is just a fraction of the
original derived table t. One could say that transformation 'splits'
the rows used for the GROUP BY operation into separate groups
performing aggregation for a group only in the case when there is
a match for the current row of t1.
Apparently the transformation may produce a query with a better
performance only in the case when
- the GROUP BY list refers only to fields returned by the derived table
- there is an index I on one of the tables T used in FROM list of
the specification of the derived table whose prefix covers the
the fields from the proper beginning of the GROUP BY list or
fields that are equal to those fields.
Whether the result of the re-writing can be executed faster depends
on many factors:
- the size of the original derived table
- the size of the table T
- whether the index I is clustering for table T
- whether the index I fully covers the GROUP BY list.
This patch only tries to improve the chosen execution plan using
this transformation. It tries to do it only when the chosen
plan reaches the derived table by a key whose prefix covers
all the fields of the derived table produced by the fields of
the table T from the GROUP BY list.
The code of the patch does not evaluates the cost of the improved
plan. If certain conditions are met the transformation is applied.
If the optimizer chose an execution plan where
a semi-join nest were materialized and the
result of materialization was scanned to access
other tables by ref access it could build a key
over columns of the tables from the nest that
were actually inaccessible.
The patch performs a proper check whether a key
that uses columns of the tables from a materialized
semi-join nest can be employed to access outer tables.
The usage of windows functions when all tables were optimized away
by min/max optimization were not supported. As result a result,
the queries that used window functions with min/max aggregation
over the whole table returned wrong result sets.
The patch fixed this problem.
Benefits of this patch:
- Removed a lot of calls to strlen(), especially for field_string
- Strings generated by parser are now const strings, less chance of
accidently changing a string
- Removed a lot of calls with LEX_STRING as parameter (changed to pointer)
- More uniform code
- Item::name_length was not kept up to date. Now fixed
- Several bugs found and fixed (Access to null pointers,
access of freed memory, wrong arguments to printf like functions)
- Removed a lot of casts from (const char*) to (char*)
Changes:
- This caused some ABI changes
- lex_string_set now uses LEX_CSTRING
- Some fucntions are now taking const char* instead of char*
- Create_field::change and after changed to LEX_CSTRING
- handler::connect_string, comment and engine_name() changed to LEX_CSTRING
- Checked printf() related calls to find bugs. Found and fixed several
errors in old code.
- A lot of changes from LEX_STRING to LEX_CSTRING, especially related to
parsing and events.
- Some changes from LEX_STRING and LEX_STRING & to LEX_CSTRING*
- Some changes for char* to const char*
- Added printf argument checking for my_snprintf()
- Introduced null_clex_str, star_clex_string, temp_lex_str to simplify
code
- Added item_empty_name and item_used_name to be able to distingush between
items that was given an empty name and items that was not given a name
This is used in sql_yacc.yy to know when to give an item a name.
- select table_name."*' is not anymore same as table_name.*
- removed not used function Item::rename()
- Added comparision of item->name_length before some calls to
my_strcasecmp() to speed up comparison
- Moved Item_sp_variable::make_field() from item.h to item.cc
- Some minimal code changes to avoid copying to const char *
- Fixed wrong error message in wsrep_mysql_parse()
- Fixed wrong code in find_field_in_natural_join() where real_item() was
set when it shouldn't
- ER_ERROR_ON_RENAME was used with extra arguments.
- Removed some (wrong) ER_OUTOFMEMORY, as alloc_root will already
give the error.
TODO:
- Check possible unsafe casts in plugin/auth_examples/qa_auth_interface.c
- Change code to not modify LEX_CSTRING for database name
(as part of lower_case_table_names)
Define my_thread_id as an unsigned type, to avoid mismatch with
ulonglong. Change some parameters to this type.
Use size_t in a few more places.
Declare many flag constants as unsigned to avoid sign mismatch
when shifting bits or applying the unary ~ operator.
When applying the unary ~ operator to enum constants, explictly
cast the result to an unsigned type, because enum constants can
be treated as signed.
In InnoDB, change the source code line number parameters from
ulint to unsigned type. Also, make some InnoDB functions return
a narrower type (unsigned or uint32_t instead of ulint;
bool instead of ibool).
The issue was that JOIN::rollup_write_data() used
JOIN::tmp_table_param::[start_]recinfo, which had uninitialized data.
These fields have uninitialized data, because JOIN::tmp_table_param
currently only stores some grouping-related data fields. The data about
the work (temporary) tables themselves is stored in
join->join_tab[...].tmp_table_param.
The fix is to make JOIN::rollup_write_data follow this convention
and look at the right TMP_TABLE_PARAM object
JOIN_CACHE's were initialized in check_join_cache_usage()
from make_join_readinfo(). After that make_join_readinfo() was looking
whether it's possible to use keyread. Later, after make_join_readinfo(),
optimizer decided whether to use filesort. And even later, at the
execution time, from join_read_first(), keyread was actually enabled.
The problem is, that if a query uses a vcol, base columns that it
depends on are automatically added to the read_set - because they're
needed to calculate the vcol. But if we're doing keyread, vcol is taken
from the index, not calculated, and base columns do not need to be
in the read set (even should not be - as they aren't getting values).
The bug was that JOIN_CACHE used read_set with base columns,
they were not read because of keyread, so it was caching garbage.
So read_set is only known after the keyread was decided. And after the
filesort was decided, as filesort doesn't use keyread. But
check_join_cache_usage() needs to be done in make_join_readinfo(),
as the code below depends on these checks,
Fix: keep JOIN_CACHE checks where they were, but move initialization
down to the very end of JOIN::optimize_inner. If keyread was enabled,
update the read_set to include only columns that are part of the index.
Copy the keyread logic from join_read_first() to happen at optimize time.
- Tabular EXPLAIN now prints "RECURSIVE UNION".
- There is a basic implementation of EXPLAIN FORMAT=JSON.
- it produces "recursive_union" JSON struct
- No other details or ANALYZE support, yet.
Temporary tables created for recursive CTE
were instantiated at the prepare phase. As
a result these temporary tables missed
indexes for look-ups and optimizer could not
use them.
Variant #4 of the fix.
Make ORDER BY optimization functions take into account multiple
equalities. This is done in several places:
- remove_const() checks whether we can sort the first table in the
join, or we need to put rows into temp.table and then sort.
- test_if_order_by_key() checks whether there are indexes that
can be used to produce the required ordering
- make_unireg_sortorder() constructs sort criteria for filesort.
This bug revealed a serious problem: if the same partition list
was used in two window specifications then the temporary table created
to calculate window functions contained fields for two identical
partitions. This problem was fixed as well.
