SELECT * FROM t1 WHERE v IN ('a','b') AND v = 'b' kept both conjuncts
when v is a utf8mb4 column, while the same query on a latin1 column was
simplified to v = 'b'. BETWEEN and the constant in a join equality
(t1.v = t2.v AND t1.v = 'b') were left alone in the same way.
The rewrite is done by the multiple equality machinery, not by
propagate_cond_constants(). check_simple_equality() turns v = 'b' into
an Item_equal holding the column and the constant. Later
substitute_for_best_equal_field() walks the remaining predicates, and
Item_func_in::propagate_equal_fields() asks each argument that is a
column of an Item_equal to replace itself with that constant whenever
the predicate compares under the Item_equal's collation. 'b' IN
('a','b') is then constant and remove_eq_conds() folds it away. The
same path serves BETWEEN, <=> and every other comparison, and
eliminate_item_equal() derives t2.v = 'b' from the join equality.
A constant was only admitted into an Item_equal when the column's
collation handler reported propagate() true. That flag was added in
2005 for Bug#9509, when a constant taken from an Item_equal was
substituted into any expression, so LENGTH(s1) saw 'oe' instead of the
o-umlaut it equals under latin1_german2_ci. Every utf8mb3 and utf8mb4
case-insensitive collation and every UCA collation report false, so
once MDEV-19123 made utf8mb4 the default, no string column got a
constant in its Item_equal. Since MDEV-7152 the substitution is
decided per site by a Context: comparison operators accept the constant
only when their comparison collation is the Item_equal's, everything
else only for binary no-pad collations, and LIKE refuses pad and
non-1-to-1 collations on its own. The flag therefore duplicated a
finer check, and propagate_cond_constants() had been substituting the
same constants into = and < under the same collations without
consulting it. Field_str::test_if_equality_guarantees_uniqueness() no
longer consults it either; the remaining tests, that the constant
compares as a string and under the column's own collation, stay.
That exposes one case the flag was hiding. Charset narrowing
(cset_narrowing) lets a utf8mb3_general_ci column and a
utf8mb4_general_ci column share an Item_equal, so the Item_equal now
also has to hold a utf8mb4 constant for its utf8mb3 member.
eliminate_item_equal() manufactures utf8mb3_col = utf8mb4_const for
that member, which fails collation aggregation when the constant has a
character outside the BMP. Field_longstr::get_equal_const_item()
converts the constant from the Item_equal's comparison collation into
the member's character set, through the mb4-to-mb3 narrowing charset
when narrowing applies, so that a character outside the BMP becomes
U+FFFD, which is what utf8mb4_general_ci compares it as. It returns
NULL, and the equality is dropped, when the constant cannot be
represented. eliminate_item_equal() calls it for every member it
builds an equality for; Item_field::propagate_equal_fields() already
did.
The tests added to func_in.test come from Dave Gosselin's patch for
this issue, which handled IN in propagate_cond_constants() instead.
Two of its expectations change with this approach: an Item_equal prints
its constant first, and v IN ('a',NULL) AND v = 'b' now folds to an
impossible WHERE, as 'b' IN ('a',NULL) is NULL. The other re-recorded
results show the same reordering, or improved plans where a constant
now reaches another table. In main.ps and main.join a self-join of
INFORMATION_SCHEMA tables on TABLE_NAME now narrows both sides to the
named table, so the InnoDB tables in the data directory are no longer
opened and their "Unknown storage engine" warnings disappear. In
spider/bugfix.mdev_27172 the CASE greeting WHEN 'Aloha!' ... = 'one'
conjunct, used there to keep the group by handler out, folds to TRUE
once greeting is substituted by its constant, so the condition Spider
pushes down to the remote is now just greeting = 'Aloha!'.
This commit was prepared with Claude Code (Fable 5.1), which traced the
missing rewrite to the propagate() check, showed by experiment that
removing it restores the rewrite and passes the func_in tests, found
the narrowing failure from a query supplied by the author, wrote
Field_longstr::get_equal_const_item() and its call in
eliminate_item_equal(), re-recorded the affected results, and ran the
main suite in normal and --ps-protocol mode before and after.
When a NATURAL JOIN operates on a derived table (or view with
derived_merge=off) having more than 64 columns, the optimizer's
generate_derived_keys_for_table() would:
1) Overflow a 32-bit shift: (key_part_map)(1 << parts) when parts >= 32
2) Create a derived key with more parts than Bitmap<64>/key_part_map can hold
3) Crash on DBUG_ASSERT(prefix_size <= width) in Bitmap<64>::is_prefix(65)
Fix: cap the number of hash-join KEYUSE entries for a materialized derived
table at the handler's max_key_parts(). Excess columns are excluded from the
key, so generate_derived_keys_for_table() never builds a key with more parts
than key_part_map (64 bits) or Bitmap<64> can represent. Also fix the shift
to cast before shifting: (key_part_map) 1 << parts.
Problem:
Executing an `UPDATE ... FOR PORTION OF` statement on a table without
a defined period results in assertion failure. The code in
`SELECT_LEX::period_setup_conds` attempts to compare a NULL
period name identifier using `streq()`, which triggers the assertion
in `strnncoll` funtion.
Fix:
Replace `streq` with `streq_safe` in `SELECT_LEX::period_setup_conds`
to handle NULL pointers safely via an early-exit.
Remove the unscoped #pragma GCC diagnostic ignored
"-Wdeprecated-declarations" from include/violite.h and replace all
sprintf/vsprintf calls with snprintf/vsnprintf across the codebase.
Where possible, pass actual buffer sizes as function parameters
instead of using hardcoded constants or recomputing strlen().
Notable API changes:
- get_date(): added to_len parameter for buffer size
- nice_time(), end_timer() in client/mysql.cc: added buff_size
- generate_new_name(): added name_size parameter
- calc_md5(): added buffer_size parameter (fixes off-by-one)
- MD5_HASH_TO_STRING macro: added _size parameter
- Added MYSQL_UDF_MAX_RESULT_LENGTH constant in mysql_com.h
Also fixed a pre-existing bug in sql/rpl_mi.cc where the ellipsis
"..." was written to the wrong buffer (dbuff instead of buff).
Vendored code is excluded: extra/readline, extra/wolfssl,
wsrep-lib, libmariadb, storage/mroonga/vendor, zlib.
When evaluating (SELECT NULL, NULL) IN (SELECT 1, 2 FROM t), the result
was incorrectly 0 instead of NULL.
The IN-to-EXISTS transformation wraps comparison predicates with
trigcond() guards:
WHERE trigcond(NULL = 1) AND trigcond(NULL = 2)
During optimization, make_join_select() evaluated this as a "constant
condition". With guards ON, the condition evaluated to FALSE (NULL
treated as FALSE), triggering "Impossible WHERE". At runtime, guards
would be turned OFF for NULL columns, but the optimizer had already
marked the subquery as returning no rows.
Fix: check can_eval_in_optimize() instead of is_expensive() in
make_join_select(). Unlike is_expensive(), can_eval_in_optimize()
also verifies const_item(), which returns FALSE for Item_func_trig_cond
(Variant 3, more comments)
When multiple indexes produce ordering that matches the
ORDER BY ... LIMIT clause, like
INDEX idx1(kp1, kp2)
INDEX idx2(kp2)
and the query with
WHERE kp1=const ORDER BY kp2 LIMIT 2
then test_if_cheaper_ordering() in pre-11.0 versions will choose
the index with the smallest KEY::user_defined_key_parts (idx2 in
this example). This can produce a much worse query plan.
The fix is to do what MariaDB 11.0 would do: use the index for which
we have the cheapest access method. The cost of access method here
takes into account that we will stop after producing #LIMIT rows.
The fix is controlled by the
@@optimizer_adjust_secondary_key_costs=fix_order_by_index_choice
flag and is OFF by default.
The bug was that thd->lex->limit_rows_examined_cnt was not reset after
queries. It is reset in lex_start() at the start of the next query
execution.
This causes general_log_write(), which is called before lex_start(), to
exceed the limit. The effect is a crash or the next query would not be
executed.
Fixed by resetting limit_rows_examined_cnt at end of query.
ft_handler isn't getting initialized for subqueries inside explain
delete/update queries. However, ft_handler is accessed inside ha_ft_read(),
and is the reason for NULL pointer exception.
This is not the case with non-explain delete/update queries, as
well as explain/non-explain select queries.
Follow the approach the SELECT statements are using in
JOIN::optimize_constant_subqueries(): remove SELECT_DESCRIBE
flag when invoking optimization of constant subqueries.
Single-table UPDATE/DELETEs have SELECT_LEX but don't have JOIN.
So, we make optimize_constant_subqueries() not to be a member
of JOIN class, and instead move it to SELECT_LEX, and then
invoke it from single-table UPDATE/DELETE as well as for SELECT queries.
Index merge and rowid filter should not be used together, however,
even if index merge is not chosen earlier in best_access_path, it may
be chosen again in make_join_select, inside ref_to_range. Therefore
this patch ensures that rowid filter is not used when index merge is
chosen there.
Rename `Item::clone_item()` to `clone_constant()`, and do
the same for any overloads in descendant items.
The function returns non-NULL only for items that represent
constant literals.
One can define a derived table with a zero-length field, for example:
(SELECT * FROM (SELECT '' FROM t) as x)
Then, the optimizer may create a zero-length index on it (this is not
allowed for regular tables). Attempt to use this index will fail an
assert in create_ref_for_key().
Fix this by marking zero-length index as an Ignored Index, so the
optimizer doesn't try to use it.
Loose Index Scan optimization can only be used when the index it is
uses covers all columns used by the query. So, it would not be
attempted for indexes that are not present in table->covering_keys.
Before commit ed0a723566 (pushed to 11.0), Clustered PK would be
present in table->covering_keys if primary key columns are covering.
However, that commit gave Clustered PKs special treatment: they are
no longer present in table->covering_keys. This caused Loose Scan not
to be considered for Clustered PKs.
Fixed this by making the check in get_best_group_min_max() to allow
Loose Scan if all fields used by the query are parts of the Clustered PK.
Also made changes:
- The logic in JOIN_TAB::save_explain_data() assumed that Loose Scan
could only be used for index-only scans. This is not the case for
clustered PKs.
- One query in group_min_max_innodb started to use Loose Scan. It did so
in 10.11, too.
With_element::prepare_unreferenced() contains query from recursive CTE
which is upper owner of unreferenced CTE itself. Such recursive
derived table is resolved into its expression and it tries to resolve
fields or any other data from subqueries which are not yet
prepared. These subqueries are preparing in upper frames of the same
stack after prepare_unreferenced() finishes.
That does not happen when recursion happens in referenced CTE because
in that case there are links between the derived tables defined by the
parser and descending into such recursion descends further into all
the dependency queries. No such link exists in unreferenced query
therefore it misses its upper dependencies.
The fix moves prepare_unreferenced() later after JOIN::prepare() so it
has the chance to access all the prepared data.
There may be another fix of failing unreferenced queries with error as
such kind of queries seem to be not usable.
Fix rownum in HAVING clause or subquery
Description
==========
When rownum used in HAVING without GROUP BY, accepted_rows should be
incremented even if the having cond's value is 0.
When rownum used in subquery
1. Forbid subquery transformed to semi join.
2. New conditions must be added into having clause when doing in2exists.
They were based on the maximum possible key tuple length, which can be
much larger than the real data size.
The return value is used by handler::keyread_time(), which is used
to estimate the cost of range access.
This could cause range access not to be picked, even if it uses
the clustered PK and reads about 8% of the table.
The fix is to add KEY::stat_storage_length (next to KEY::rec_per_key) and
have the storage engine fill it in handler::info(HA_STATUS_CONST).
Currently, only InnoDB fills this based on its internal statistics:
index->stat_index_size and ib_table->stat_n_rows.
Also changed:
- In handler::calculate_costs(), use ha_keyread_clustered_time() when
computing clustered PK read cost, not ha_keyread_time().
The fix is OFF by default and enabled by setting FIX_INDEX_LOOKUP_COST flag
in @@new_mode.
Problem:
After the re-design of `UPDATE` and `DELETE` in MDEV-28883, the call
to find select handler is missing. This prevents the server from
handring over multi-update/multi-delete queries to storage engines
capable of executng such queries on their own (e.g., ColumnStore).
MDEV-32382 introduced a check in `find_select_handler_inner` function,
that blocked pushdown of queries involving CTEs, without checking if
the storage engine is capable of handling such queries.
Fix:
Add a call to find the select handler for the engine involved in the
multi-update/multi-delete query, allowing the storage engines to
execute such queries.
Fix the `find_select_handler_inner` function by allowing the storage
engine's create_select functions to decide the pushdown of queries
involving CTEs.
The NULL-aware index statistics fix is now controlled by the
FIX_INDEX_STATS_FOR_ALL_NULLS flag and disabled by default
for preserving execution plan stability in stable versions.
To enable:
SET @@new_mode = 'FIX_INDEX_STATS_FOR_ALL_NULLS';
Or via command line:
--new-mode=FIX_INDEX_STATS_FOR_ALL_NULLS
Or in configuration file:
[mysqld]
new_mode=FIX_INDEX_STATS_FOR_ALL_NULLS
`all_nulls_key_parts` bitmap is now calculated at set_statistics_for_table()
Variant 2: "Don't call optimize_stage2 too early"
The affected query had Split-Materialized derived table in another
derived table:
select * -- select#1
from (
select * -- select#2
from t1,
(select * from t2 ... group by t2.group_col) DT -- select#3
where t1.col=t2.group_col
) TBL;
The optimization went as follows:
JOIN::optimize() calls select_lex->handle_derived(DT_OPTIMIZE) which
calls JOIN::optimize() for all (direct and indirect) children SELECTs.
select#1->handle_derived() calls JOIN::optimize() for select#2 and #3;
select#2->handle_derived() calls JOIN::optimize() for select#3 the second
time. That call would call JOIN::optimize_stage2(), assuming the query
plan choice has been finalized for select#3.
But after that, JOIN::optimize() for select#2 would continue and consider
Split-Materialized for select#3. This would attempt to pick another join
order and cause a crash.
The fix is to have JOIN::optimize() not call optimize_stage2() ever.
Finalizing the query plan choice is now done by a separate call:
select_lex->handle_derived(thd->lex, DT_OPTIMIZE_STAGE2)
which invokes JOIN::optimize_stage2() and saves the query plan.
When all values in an indexed column are NULL, EITS statistics show
avg_frequency == 0. This commit adds logic to distinguish between
"no statistics available" and "all values are NULL" scenarios.
For NULL-rejecting conditions (e.g., t1.col = t2.col), when statistics
confirm all indexed values are NULL, the optimizer can now return a
very low cardinality estimate (1.0) instead of unknown (0.0), since
NULL = NULL never matches.
For non-NULL-rejecting conditions (e.g., t1.col <=> t2.col),
normal cardinality estimation continues to apply since matches are possible.
Changes:
- Added KEY::rec_per_key_null_aware() to check nulls_ratio from column
statistics when avg_frequency is 0
- Modified best_access_path() in sql_select.cc to use the new
rec_per_key_null_aware() method for ref access cost estimation
- The optimization works with single-column and composite indexes,
checking each key part's NULL-rejecting status via notnull_part bitmap
Wrong result is produced when split-materialized optimization is used for
grouping with order by and limit.
The fix is to not let Split-Materialized optimization to happen
when the sub-query has an ORDER BY with LIMIT, by returning FALSE early
in the method opt_split.cc#check_for_splittable_materialized()
However, with just the above change, there is a side-effect of
NOT "using index for group by" in the scenario when
all the following conditions are met: -
1. The query has derived table with GROUP BY and ORDER BY LIMIT
2. joined in a way that would allow Split-Materialized
if ORDER BY LIMIT wasn't present
3. An index suitable for using "index for group-by"
4. No where clause so that, "using for group by" is applicable,
but the index is not included in "possible_keys".
The reason being, join_tab's "keys" field wasn't being set in
sql_select.cc#make_join_select(). So, made this change as well
as part of this PR.
The "FETCH FIRST n ROWS WITH TIES" was not enforced when the SELECT used
"using index for group-by".
This was caused by an optimization which removed the ORDER BY clause
when the GROUP BY clause prescribed a compatible ordering.
Other GROUP BY strategies used workarounds to still handle WITH TIES,
see comment to using_with_ties_and_group_min_max() in this patch for
details. QUICK_GROUP_MIN_MAX_SELECT didn't have a workaround.
Fix this by disabling removal of ORDER BY when
QUICK_GROUP_MIN_MAX_SELECT is used.
