This piece of the code in Item_func_or_sum::agg_item_set_converter:
if (!conv && ((*arg)->collation.repertoire == MY_REPERTOIRE_ASCII))
conv= new (thd->mem_root) Item_func_conv_charset(thd, *arg, coll.collation, 1);
was wrong because:
1. It could change Item_cache to Item_func_conv_charset
(with the old Item_cache in args[0]).
Such Item type change is not always supported, e.g.
the code in Item_singlerow_subselect::reset() expects only Item_cache,
to be able to call Item_cache::set_null(). So it erroneously
reinterpreted Item_func_conv_charset to Item_cache and called
a non-existing method set_null(), which crashed the server.
2. The 1 in the last parameter to Item_func_conv_charset() was also
a problem. In MariaDB versions where the reported query did not crash,
it erroneously returned "empty set" instead of one row, because
the 1 made subselects execute too earlier and return NULL.
Fix:
1. Removing the above two lines from Item_func_or_sum::agg_item_set_converter()
2. Adding the repertoire test inside the constructor of Item_func_conv_charset,
so it now detects itself as "safe" in more cases than before.
This is needed to avoid new "Illegal mix of collations" after
removing the wrong code in various scenarios when character set
conversion from pure ASCII happens, including the reported scenario.
So now this sequence:
Item_cache -> Item_func_concat
is replaced to this compatible sequence (the top Item is still Item_cache):
new Item_cache -> Item_func_conv_charset -> Item_func_concat
Before the fix it was replaced to this incompatible sequence:
Item_func_conv_charset -> old Item_cache -> Item_func_concat
This corrects the patch for mdev-10006.
The current code supports only those semi-join nests that are placed at
the join top level. So such nests cannot depend on other tables or nests.
The code that chooses between materialization of a non-correlated
IN subquery and its transformation into an EXISTS correlated
subquery assumes that the execution plan for the outer select
has been already built. However it was not always so if subqueries
occurred in the expressions used for ref access to tables of
the outer select. A call of the function create_ref_for_key() in
get_best_combination() could trigger a premature execution of
the above mentioned code when the execution plan structures for
the outer select were not fully built. This could cause a crash
of the server.
The fix postpones the calls of create_ref_for_key() until the
structures for the execution plan is fully built.
Also fixed a wrong result for a test case for mdev-7691
(the alternative one).
The test cases for all these bug have materialized semi-joins used
inside dependent sub-queries.
The patch actually reverts the change inroduced by Monty in 2003.
It looks like this change is not valid anymore after the implementation
of semi-joins.
Adjusted output from EXPLAIN for many other test cases.
Also, implement MDEV-11027 a little differently from 5.5 and 10.0:
recv_apply_hashed_log_recs(): Change the return type back to void
(DB_SUCCESS was always returned).
Report progress also via systemd using sd_notifyf().
Also, implement MDEV-11027 a little differently from 5.5:
recv_sys_t::report(ib_time_t): Determine whether progress should
be reported.
recv_apply_hashed_log_recs(): Rename the parameter to last_batch.
Add a testcase and backport this fix:
Bug#14338686: MYSQL IS GENERATING DIFFERENT AND SLOWER
(IN NEWER VERSIONS) EXECUTION PLAN
PROBLEM:
While checking for an index to sort for the order by clause
in this query
"SELECT datestamp FROM contractStatusHistory WHERE
contract_id = contracts.id ORDER BY datestamp asc limit 1;"
we do not calculate the number of rows to be examined correctly.
As a result we choose index 'idx_contractStatusHistory_datestamp'
defined on the 'datestamp' field, rather than choosing index
'contract_id'. And hence the lower performance.
ANALYSIS:
While checking if an index is present to give the records in
sorted order(datestamp), we consider the selectivity of the
'ref_key'(contract_id here) using 'table->quick_condition_rows'.
'ref_key' here can be an index from 'REF_ACCESS' or from 'RANGE'.
As this is a 'REF_ACCESS', 'table->quick_condition_rows' is not
set to the actual value which is 2. Instead is set to the number
of tuples present in the table indicating that every row that
is selected would be satisfying the condition present in the query.
Hence, the selectivity becomes 1 even when we choose the index
on the order by column instead of the join_condition.
But, in reality as only 2 rows satisy the condition, we need to
examine half of the entire data set to get one tuple when we
choose index on the order by column.
Had we chosen the 'REF_ACCESS' we would have examined only 2 tuples.
Hence the delay in executing the query specified.
FIX:
While calculating the selectivity of the ref_key:
For REF_ACCESS consider quick_rows[ref_key] if range
optimizer has an estimate for this key. Else consider
'rec_per_key' statistic.
For RANGE ACCESS consider 'table->quick_condition_rows'.
- Make JOIN::const_key_parts include keyparts for which
the WHERE clause has an equality in form
"t.key_part=reference_outside_this_select"
- This allows to avoid filesort'ing in some cases (and also
avoid a difficult choice between using filesort or using an index)
includes:
* remove some remnants of "Bug#14521864: MYSQL 5.1 TO 5.5 BUGS PARTITIONING"
* introduce LOCK_share, now LOCK_ha_data is strictly for engines
* rea_create_table() always creates .par file (even in "frm-only" mode)
* fix a 5.6 bug, temp file leak on dummy ALTER TABLE
Analysis:
The reason for the inefficent plan was that Item_subselect::is_expensive()
didn't detect the special case when a subquery was optimized, but had no
join plan because it either has no table, or its tables have been optimized
away, or the optimizer detected that the result set is empty.
Solution:
Identify the special cases above in the Item_subselect::is_expensive(),
and consider such degenerate subqueries inexpensive.
The fix backports from MWL#182: Explain running statements the logic that
saves the original JOIN_TAB array of a query plan after optimization. This
array is later used during EXPLAIN to iterate over the original JOIN plan
nodes in the cases when this plan could be changed by early subquery
execution during the optimization phase of the outer query.
Analysis:
When a subquery that needs a temp table is executed during
the prepare or optimize phase of the outer query, at the end
of the subquery execution all the JOIN_TABs of the subquery
are replaced by a new JOIN_TAB that selects from the temp table.
However that temp table has no corresponding TABLE_LIST.
Once EXPLAIN execution reaches its last phase, it tries to print
the names of the subquery tables through its TABLE_LISTs, but in
the case of this bug there is no such TABLE_LIST (it is NULL),
hence a crash.
Solution:
The fix is to block subquery evaluation inside
Item_func_like::fix_fields and Item_func_like::select_optimize()
using the Item::is_expensive() test.
The patch enables back constant subquery execution during
query optimization after it was disabled during the development
of MWL#89 (cost-based choice of IN-TO-EXISTS vs MATERIALIZATION).
The main idea is that constant subqueries are allowed to be executed
during optimization if their execution is not expensive.
The approach is as follows:
- Constant subqueries are recursively optimized in the beginning of
JOIN::optimize of the outer query. This is done by the new method
JOIN::optimize_constant_subqueries(). This is done so that the cost
of executing these queries can be estimated.
- Optimization of the outer query proceeds normally. During this phase
the optimizer may request execution of non-expensive constant subqueries.
Each place where the optimizer may potentially execute an expensive
expression is guarded with the predicate Item::is_expensive().
- The implementation of Item_subselect::is_expensive has been extended
to use the number of examined rows (estimated by the optimizer) as a
way to determine whether the subquery is expensive or not.
- The new system variable "expensive_subquery_limit" controls how many
examined rows are considered to be not expensive. The default is 100.
In addition, multiple changes were needed to make this solution work
in the light of the changes made by MWL#89. These changes were needed
to fix various crashes and wrong results, and legacy bugs discovered
during development.
Bug#13639204 64111: CRASH ON SELECT SUBQUERY WITH NON UNIQUE INDEX
The crash happened due to wrong calculation
of key length during creation of reference for
sort order index. The problem is that
keyuse->used_tables can have OUTER_REF_TABLE_BIT enabled
but used_tables parameter(create_ref_for_key() func) does
not have it. So key parts which have OUTER_REF_TABLE_BIT
are ommited and it could lead to incorrect key length
calculation(zero key length).
mysql-test/r/subselect_innodb.result:
test result
mysql-test/t/subselect_innodb.test:
test case
sql/sql_select.cc:
added OUTER_REF_TABLE_BIT to the used_tables parameter
for create_ref_for_key() function.
storage/innobase/handler/ha_innodb.cc:
added assertion, request from Inno team
storage/innodb_plugin/handler/ha_innodb.cc:
added assertion, request from Inno team
The patch differs from the original MySQL patch as follows:
- All test case differences have been reviewed one by one, and
care has been taken to restore the original plan so that each
test case executes the code path it was designed for.
- A bug was found and fixed in MariaDB 5.3 in
Item_allany_subselect::cleanup().
- ORDER BY is not removed because we are unsure of all effects,
and it would prevent enabling ORDER BY ... LIMIT subqueries.
- ref_pointer_array.m_size is not adjusted because we don't do
array bounds checking, and because it looks risky.
Original comment by Jorgen Loland:
-------------------------------------------------------------
WL#5953 - Optimize away useless subquery clauses
For IN/ALL/ANY/SOME/EXISTS subqueries, the following clauses are
meaningless:
* ORDER BY (since we don't support LIMIT in these subqueries)
* DISTINCT
* GROUP BY if there is no HAVING clause and no aggregate
functions
This WL detects and optimizes away these useless parts of the
query during JOIN::prepare()
sql/sql_insert.cc:
CREATE ... IF NOT EXISTS may do nothing, but
it is still not a failure. don't forget to my_ok it.
******
CREATE ... IF NOT EXISTS may do nothing, but
it is still not a failure. don't forget to my_ok it.
sql/sql_table.cc:
small cleanup
******
small cleanup
Analysis:
The crash is a result of the same cause as all similar
bugs (lp:827416, lp:718763, lp:778413, lp:806943,
lp:611690). The general pattern is that some optimization
requires the evaluation of some condition (e.g. the WHERE
clause), and this condition contains a subquery, such that
the subquery itself requires a temporary table for its
execution. During the subquery execution the original
tables in the FROM clause are replaced by the temporary
table needed for the final GROUP or ORDER operation. All
this happens during optimization of the outer query. Later
when EXPLAIN is run for the subquery, explain attempts to
print the name of the tables in the FROM clause, but it
finds there a temporary table without a corresponding
TABLE_LIST object. The attempt to print the name of a
NULL table list results in a crash.
Solution:
This patch extends the fix to bug lp:702301, and dissalows
constant substitution of aggregate functions if the filter
condition used to check MIN/MAX keys is an expensive condition.
Analysis:
Constant table optimization of the outer query finds that
the right side of the equality is a constant that can
be used for an eq_ref access to fetch one row from t1,
and substitute t1 with a constant. Thus constant optimization
triggers evaluation of the subquery during the optimize
phase of the outer query.
The innermost subquery requires a plan with a temporary
table because with InnoDB tables the exact count of rows
is not known, and the empty tables cannot be optimzied
way. JOIN::exec for the innermost subquery substitutes
the subquery tables with a temporary table.
When EXPLAIN gets to print the tables in the innermost
subquery, EXPLAIN needs to print the name of each table
through the corresponding TABLE_LIST object. However,
the temporary table created during execution doesn't
have a corresponding TABLE_LIST, so we get a null
pointer exception.
Solution:
The solution is to forbid using expensive constant
expressions for eq_ref access for contant table
optimization. Notice that eq_ref with a subquery
providing the value is still possible during regular
execution.
- Set the default
- Adjust the testcases so that 'new' tests are run with optimizations turned on.
- Pull out relevant tests from "irrelevant" tests and run them with optimizations on.
- Run range.test and innodb.test with both mrr=on and mrr=off
This assumption in Item_cache_datetime::cache_value_int
was wrong:
- /* Assume here that the underlying item will do correct conversion.*/
- int_value= example->val_int_result();
mysql-test/r/subselect_innodb.result:
New test case.
mysql-test/t/subselect_innodb.test:
New test case.
sql/item.cc:
In Item_cache_datetime::cache_value_int()
- call get_time() or get_date() depending on desired type
- convert the returned MYSQL_TIME value to longlong depending on desired type
sql/item.h:
The cached int_value in Item_cache_datetime should not be unsigned:
- it is used mostly in signed context
- it can actually have negative value (for TIME data type)
sql/item_cmpfunc.cc:
Add comment on Bug#59685
sql/item_subselect.cc:
Add some DBUG_TRACE for easier bug-hunting.
Cleanup the sideeffects from most of the testcases with sideeffects.
mysql-test/mysql-test-run.pl:
Add option "check-testcases" to mysql-test-run.pl
Will execute "include/check-testcase.test" once before each tescase and record the output into "var/tmp/check-testcase.result"
After the teastcase it will run again and this time compare the output with previously recorded file.
mysql-test/r/analyze.result:
Drop table t1 at end of test
mysql-test/r/create_select_tmp.result:
Drop table t1 at end of test
mysql-test/r/ctype_cp932.result:
Drop table t1 at end of test
mysql-test/r/ctype_recoding.result:
Drop table t1 at end of test
mysql-test/r/grant2.result:
Drop user mysqltest_2 and mysqltest_A@'%'
mysql-test/r/join_outer.result:
Drop view v1 to cleanup
mysql-test/r/ps_1general.result:
Drop table t1 at end of test
mysql-test/r/query_cache.result:
Drop function "f1"
mysql-test/r/read_only.result:
Reset the "read_only" flag
mysql-test/r/rpl000001.result:
Remove user "blafasel2"
mysql-test/r/rpl000017.result:
Remove user "replicate"
mysql-test/r/rpl_failed_optimize.result:
Drop table t1 to cleanup
mysql-test/r/rpl_flush_tables.result:
Drop tables t3, t4, t5
mysql-test/r/rpl_ignore_revoke.result:
Delete user "user_foo"
mysql-test/r/rpl_insert_id.result:
Drop table t1 to cleanup
mysql-test/r/rpl_loaddata.result:
Drop tyable t1 to cleanup
mysql-test/r/rpl_loaddata_rule_m.result:
Drop tyable t1 to cleanup
mysql-test/r/rpl_loaddata_rule_s.result:
Drop tyable t1 to cleanup
mysql-test/r/rpl_misc_functions.result:
Drop tyable t1 to cleanup
mysql-test/r/rpl_multi_update3.result:
Drop tyable t1 and t2 to cleanup
mysql-test/r/rpl_replicate_do.result:
Drop tyable t1 to cleanup
mysql-test/r/rpl_skip_error.result:
Drop tyable t1 to cleanup
mysql-test/r/rpl_slave_status.result:
Drop tyable t1 to cleanup
mysql-test/r/sp-prelocking.result:
Drop view v1 and tables t1, t2, t3 and t4 to cleanup
mysql-test/r/sp-security.result:
Delete users to cleanup
Delete remaining traces in tables_priv and procs_priv
mysql-test/r/subselect_innodb.result:
Drop procedure p1 to cleanup
mysql-test/r/trigger-compat.result:
Drop trigger wl2818_trg1 and wl2818_trg2.
Drop table t1, t2
Drop database mysqltest_db1
And the users "mysqltest_dfn@localhost" and "mysqltest_inv@localhost"
mysql-test/r/type_bit.result:
Drop tables t1 and t2 to cleanup
mysql-test/r/variables.result:
Set GLOBAL max_join_size to 10 as it originally was in variables-master.opt
mysql-test/r/view_grant.result:
Dop user "test@localhost" to cleanup
mysql-test/t/analyze.test:
Drop table t1 to cleanup
mysql-test/t/create_select_tmp.test:
Drop table t1 to cleanup
mysql-test/t/ctype_cp932.test:
Drop table t1 to cleanup
mysql-test/t/ctype_recoding.test:
Drop table t1 to cleanup
mysql-test/t/fulltext_var.test:
Restore the original ft_boolean_syntax
mysql-test/t/grant2.test:
Drop users "mysqltest_2" and "mysqltest_A@'%'" to cleanup
mysql-test/t/innodb_cache.test:
Reset query_cache_size to original value
mysql-test/t/join_outer.test:
Drop view v1 to cleanup
mysql-test/t/ps_1general.test:
Drop table t1 to cleanup
mysql-test/t/query_cache.test:
Drop function "f1" to cleanup
mysql-test/t/read_only.test:
Reset the readonly flag
mysql-test/t/rpl000001.test:
Delete user "blafasel2" to cleanup
mysql-test/t/rpl000017.test:
Delete user "replicate" to cleanup
mysql-test/t/rpl_failed_optimize.test:
Drop table t1 to cleanup
mysql-test/t/rpl_flush_tables.test:
Droip table t3, t4 and t5 to cleanup
mysql-test/t/rpl_ignore_revoke.test:
Delet user "user_foo" to cleanup
mysql-test/t/rpl_insert_id.test:
drop table t1 to cleanup
mysql-test/t/rpl_loaddata.test:
Drop table t1 to cleanup
mysql-test/t/rpl_loaddata_rule_m.test:
Drop table t1 to cleanup
mysql-test/t/rpl_loaddata_rule_s.test:
Drop table t1 to cleanup
mysql-test/t/rpl_misc_functions.test:
Drop table t1 to cleanup
mysql-test/t/rpl_multi_update3.test:
Drop table t1 and t2 to cleanup
mysql-test/t/rpl_replicate_do.test:
Drop table t1 to cleanup
mysql-test/t/rpl_skip_error.test:
Drop table t1 to cleanup
mysql-test/t/rpl_slave_status.test:
Drop table t1 to cleanup
mysql-test/t/sp-prelocking.test:
Drop table t1, t2 t3 and t4 to cleanup
Drop view v1
mysql-test/t/sp-security.test:
Delete test users from mysql.user, mysql.db, mysql.procs_priv and mysql.tables_priv
Drop table t1 to cleanup
mysql-test/t/subselect_innodb.test:
Drop procedure p1 to cleanup
mysql-test/t/trigger-compat.test:
Drop trigger wl2818_trg1 and wl2818_trg2 to cleanup
Drop table t1, t2
Drop users
drop database mysqltest_db1
mysql-test/t/type_bit.test:
drop table t1 and t2 to cleanup
mysql-test/t/variables-master.opt:
Increase max_join_size to 100.
mysql-test/t/variables.test:
Set max_join_size to 10, which was the original value in variables-master.opt
mysql-test/t/view_grant.test:
Drop the user "test@localhost"
mysql-test/include/check-testcase.test:
New BitKeeper file ``mysql-test/include/check-testcase.test''
mysql-test/r/subselect_innodb.result:
BUG#14342 test case
mysql-test/t/subselect_innodb.test:
BUG#14342 test case
sql/opt_range.cc:
avoiding of calling Item::val_* methods family with opt_range mem_root, because its life time is too short.
The bug was in JOIN::join_free which was wrongly determining that
all joins have been already executed and therefore all used tables
can be closed.
mysql-test/r/subselect_innodb.result:
- test results fixed (Bug#12736 "Server crash during a select
mysql-test/t/subselect_innodb.test:
- a test case for Bug#12736 "Server crash during a select": test
that ha_index_or_rnd_end and mysql_unlock_tables are called
for all used tables in proper order.
sql/item_subselect.cc:
- implement subselect_union_engine::is_executed
sql/item_subselect.h:
- implement Item_subselect::is_evaluated. This function is used
to check whether we can clean up a non-correlated join of a subquery
when cleaning up the join of the outer query
sql/sql_lex.h:
- declare st_select_lex::cleanup_all_joins
sql/sql_select.cc:
- remove an argument from JOIN::join_free, it's now not used
- reimplement JOIN::join_free to not unlock tables if there
is a subquery that has not yet been evaluated. Make sure that the
new implementation calls ha_index_or_rnd_end for every table in
the join and inner joins, because all table cursors must be closed
before mysql_unlock_tables.
sql/sql_select.h:
- JOIN::join_free signature changed
sql/sql_union.cc:
- implement a helper method st_select_lex::cleanup_all_joins, which
recursively walks over a tree of joins and calls cleanup() for
each join.
mysql-test/r/subselect.result:
test depends on innodb moved from 'subselect' to 'subselect_innodb'
mysql-test/r/subselect_innodb.result:
test depends on innodb moved from 'subselect' to 'subselect_innodb'
mysql-test/t/subselect.test:
test depends on innodb moved from 'subselect' to 'subselect_innodb'
mysql-test/t/subselect_innodb.test:
test depends on innodb moved from 'subselect' to 'subselect_innodb'
mysql-test/r/subselect_innodb.result:
test of unlocking innodb tables and subquery in HAVING clause
mysql-test/t/subselect_innodb.test:
test of unlocking innodb tables and subquery in HAVING clause
sql/item_subselect.cc:
mark SELECT with subquery in HAVING clause
sql/sql_lex.cc:
mark SELECT with subquery in HAVING clause
sql/sql_lex.h:
mark SELECT with subquery in HAVING clause
sql/sql_select.cc:
do not unlock tables early if we have subquery in HAVING clause
