This merges the patch series that fixes a lot of missing validation
checks for read_log_event(). The merge includes a few added fixes that
are needed in new code introduced in 12.3.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
This SQL script failed:
SET NAMES latin1 COLLATE latin1_swedish_ci;
CREATE OR REPLACE VIEW v1 AS SELECT 'Y' AS c1;
SET NAMES big5 COLLATE big5_chinese_ci;
SELECT * FROM v1 WHERE c1='y';
with the following error:
ERROR 1267 (HY000): Illegal mix of collations
(latin1_swedish_ci,COERCIBLE) and (big5_chinese_ci,COERCIBLE)
for operation '='
Note, latin1_swedish_ci and big5_chinese_ci are used here as examples.
The error also happened with different collation combinations.
Fix main idea:
If two collations have equal comparison rules (known as "tailoring")
on a given character repertoire,
like latin1_swedish_ci and big5_chinese_ci on ASCII letters,
then the "Illegal mix of collations" error can be avoided in
a comparison operator. We can choose any of the sides as the operation
effective collation - the result will be equal.
This optimization is not applied when at least one side has an explicit
COLLATE clause. Two explicit COLLATE clauses in one comparison are
already illegal when the character sets are the same, so for
consistency this stays illegal when the character sets differ too.
Most important details:
- Splitting enum_repertoire_t into smaller subsets,
for better repertoire granularity.
A variable holding a repertoire value can now have multiple
MY_REPERTOIRE_XXX flags set.
This patch implements detecting tailoring equality
on these repertoires:
* MY_REPERTOIRE_ASCII_DIGITS - [0..9]
* MY_REPERTOIRE_ASCII_MINUS_DOT_DIGITS - [-.0..9], e.g. '-10.5'.
It consists of the new bits MY_REPERTOIRE_ASCII_MINUS and
MY_REPERTOIRE_ASCII_DOT. They were a part of
MY_REPERTOIRE_ASCII_NOT_IDENT before. As a result, the numeric values
of MY_REPERTOIRE_ASCII and MY_REPERTOIRE_UNICODE30 changed
(223 and 255).
* MY_REPERTOIRE_ASCII_ALNUM - [A..Z,a..z,0..9].
* MY_REPERTOIRE_ASCII_IDENT - ALNUM + underscore
* MY_REPERTOIRE_ASCII - the entire range U+0000..U+007F
- Adding a new virtual function "tailoring" in my_collation_handler_st.
It returns the tailoring on the given repertoire for the given
collation.
If cs1->coll->tailoring(cs1, some_repertoire) returns {0,0},
it means the illegal mix optimization cannot be used
for this collation on the given repertoire.
If these calls:
tr1= cs1->coll->tailoring(cs1, some_repertoire);
tr2= cs2->coll->tailoring(cs2, some_repertoire);
return both non-NULL results and tr1.str==tr2.str,
then these collations are equal on the given repertoire
and are mutually replaceable for a comparison operator,
so "Illegal mix of collations" can be avoided.
- Tailoring strings are shared constants my_tailoring_str_*
(strings/strings_def.h). Tailorings are compared by the string
pointer, so tailoring() implementations return strings only from
these constants. The strings do not depend on PAD/NOPAD.
- my_string_repertoire() for ucs2, utf16, utf32 and NONASCII 8bit
character sets no longer stops at the first bad byte sequence with
the repertoire found so far, which could be MY_REPERTOIRE_NONE
(compatible with everything). It returns MY_REPERTOIRE_EXTENDED
for such strings. A unit test was added into unittest/strings.
- Adding CHARSET_INFO::is_ascii_superset(). It tells if a character set
can store all ASCII characters (it is false for swe7).
It is used in Item_func_conv_charset to calculate "safe", and in
left_is_algorithmically_simpler() to prefer other character sets
to those where ASCII strings cannot be converted safely.
- DBUG_ASSERTs were added into mysys/charset.c, to check that
the "tailoring" method is set in collation handlers when a collation
is added: compiled-in, from ctype-extra.c, or loaded from Index.xml.
- Adding a new function my_collations_equal_on_repertoire(cs1, cs2,
repertoire) in strings/. It tells if two collations are equal on
the repertoire, taking tailoring() results and PAD/NOPAD and the UCA
version into account. Callers do not compare tailoring() results
directly, so the way tailorings are compared (currently by the pointer
to the shared constant string) is private to strings/ and can be
extended later, e.g. for UCA collations on MY_REPERTOIRE_UNICODE30.
- Adding a new method DTCollation::aggregate_by_tailoring().
Collations with different MY_CS_NOPAD are never compatible.
UCA collations of different versions are not compatible on
the repertoire with ASCII punctuation, as the UCA version
affects its order (e.g. UCA-6.2.0 moved GRAVE ACCENT
and CIRCUMFLEX ACCENT after PERCENT SIGN).
- DTCollation::aggregate() now makes the repertoire of the result cover
the repertoires of both sides in all successful branches.
Before, set(dt) replaced the repertoire with the repertoire of the
winner side only, so the result could have a too narrow repertoire,
e.g. IF(1, 1.5, HEX(255)) had MY_REPERTOIRE_ASCII_ALNUM although the
value '1.5' has a punctuation character. With the more granular
repertoires this could wrongly allow mixing collations which are
equal on ALNUM but different on punctuation.
- Adding a new flag MY_COLL_ALLOW_BY_TAILORING.
It indicates to DTCollation::aggregate() that the illegal
mix optimization by repertoire can be used in the given context.
MY_COLL_CMP_CONV now includes MY_COLL_ALLOW_BY_TAILORING.
Note, only comparison operators pass this flag.
Functions returning a string result do not pass this flag,
because in operations like CONCAT(a,b) we still need to evaluate
precisely the collation of the result - we cannot just choose
a collation of one of the sides (even if they are compatible
on the given repertoire). This also applies to ExtractValue()
and UpdateXML(), which now aggregate their arguments without
this flag.
- As in my_repertoire_t the value MY_REPERTOIRE_ASCII is now a set of
bits rather than a single bit, the way to detect "is only ASCII"
repertoires has changed in the code.
For example:
// repertoire *IS* ascii
if (repertoire == MY_REPERTOIRE_ASCII)
has changed in multiple places in the code to
// repertoire *HAS* only ascii characters
if (my_repertoire_is_subset_of(repertoire, MY_REPERTOIRE_ASCII))
The new function my_repertoire_is_subset_of() in m_ctype.h
is used for this purpose in C code. In C++ code
DTCollation::repertoire_is_subset_of() is used, e.g.:
if (collation.repertoire_is_subset_of(MY_REPERTOIRE_ASCII))
- Repertoire of some expressions was adjusted to the new meaning:
* Item_null now has MY_REPERTOIRE_NONE (was ASCII).
* MY_LOCALE::repertoire() now returns MY_REPERTOIRE_UNICODE30
(was EXTENDED).
* Lex_string_with_metadata_st::repertoire(cs) now scans the string
contents to detect the actual repertoire.
* HEX() now has MY_REPERTOIRE_ASCII_ALNUM.
* DATE_FORMAT() decides on MY_REPERTOIRE_EXTENDED using the locale
which is actually used: the explicit third argument, or
@@lc_time_names. It was always @@lc_time_names before.
A non-constant locale argument is assumed to be non-ASCII.
* QUOTE, MAKE_SET, EXPORT_SET, LPAD, RPAD, GROUP_CONCAT, JSON_ARRAY,
JSON_OBJECT and JSON_OBJECTAGG add the repertoire of the extra
characters they put into the result (quotes, separators, padding,
brackets).
* LOWER() and UPPER() add the letters of both cases to the repertoire,
because the result can contain letters of the opposite case.
In Turkish collations they also add MY_REPERTOIRE_EXTENDED, as an
ASCII letter can be converted to a non-ASCII one (I -> dotless i).
Unicode collations are detected by the new member
casefold_info_st::can_convert_to_non_ascii_on_casefolding,
simple 8bit collations by to_lower['I'] and to_upper['i'].
- Lex_string_with_metadata_st::repertoire(cs) now scans a string literal
for any byte oriented character set (mbminlen==1), not only for
ASCII based ones. It was assumed to have the full repertoire for swe7,
which is not ASCII based. The scan is done by Unicode code points, so
swe7 strings with digits and letters get a narrow repertoire, while
e.g. 0x5B (LATIN CAPITAL LETTER A WITH DIAERESIS in swe7) is non-ASCII.
Therefore swe7_swedish_ci can be mixed with other collations now,
in the same way as other collations with MY_CS_IDENT_CASEUP_CI.
The test func_time was adjusted: a swe7 literal ' ull' with ASCII
characters is now safe to convert, the error is still tested for '[ull'.
- cp1250_czech_cs sorts -, . and 0..9 in the code point order, so it gets
the MINUS_DOT_DIGITS tailoring. latin2_czech_cs gets only DIGITS,
as it sorts the minus and the dot differently. Both were verified
with ctype_digits_verify.
- The tis620 collations now set CHARSET_INFO::tab_to_uni (was NULL),
so my_charset_is_ascii_based() is true for tis620.
It was false for tis620, although tis620 is ASCII compatible.
Therefore tis620 string literals were always considered to have
a non-ASCII repertoire, and the repertoire optimizations did not work
for them. Now ASCII-only tis620 literals are detected as ASCII.
This changes the result of subselect_extra_no_semijoin
from an error to success.
- tis620_thai_ci and tis620_thai_nopad_ci fold letters to lower case
and then compare by the code point on the entire ASCII range
(the Thai specific rules affect only non-ASCII characters).
A new function my_tailoring_ascii_casedn_ci() returns tailorings
for such collations on ALNUM, IDENT and ASCII. For example, the
underscore sorts before the letters, unlike in the caseup tailorings.
- New flags were added for CHARSET_INFO::state
* MY_CS_ASCII_CASEUP_CI - for simple 8bit case insensitive collations.
It means that this collation has no irregularities on the ASCII
repertoire, and converts lower case letters to upper case ones,
so the underscore sorts after the letters (with an upper to lower
case conversion it would sort before the letters).
* MY_CS_IDENT_CASEUP_CI - for simple 8bit case insensitive collations.
It means that this collation has no irregularities on the IDENT
repertoire (but can have irregularities say on punctuation).
* MY_CS_ASCII_STD_UCA - for UCA collations.
It means that a UCA collation does not reorder ASCII characters.
* MY_CS_DIGITS_STD - for any collation. It means that the ASCII digits
0..9 are sorted in the code point order and are not tailored, and,
for PAD collations, the space sorts before the digits. Strings of
digits are compared in the same way in all such collations, even if
they have irregularities on letters (latin7, cp866, czech, danish,
turkish UCA, etc.), or differ by case sensitivity. It is detected
from sort_order for 8bit collations, and from the tailoring rules
for UCA collations. Collations with MY_CS_DIGITS_STD still must have
the same PAD/NOPAD attribute.
* MY_CS_ASCII_MINUS_DOT_DIGITS - for any collation. Means
MY_CS_DIGITS_STD, and additionally the minus sorts before the dot,
which sorts before the digits (and the space sorts before the minus
in PAD collations). Strings like '-10.5' are compared in the same
way in all such collations. It is detected from sort_order for simple
8bit collations (my_coll_init_simple()), and from the tailoring rules
for UCA collations. For the multi-byte collations big5_chinese_ci,
gbk_chinese_ci, gb2312_chinese_ci and sjis_japanese_ci it is declared
by my_tailoring_ident_caseup_ci_generic(). The flags MY_CS_DIGITS_STD
and MY_CS_ASCII_MINUS_DOT_DIGITS are stored in ctype-extra.c for
compiled collations (conf_to_src.c prints them), and the detection is
skipped for a collation which already has any of the repertoire flags
(MY_CS_REPERTOIRE_FLAGS).
- strings/conf_to_src.c was modified to detect and print
MY_CS_ASCII_CASEUP_CI and MY_CS_IDENT_CASEUP_CI flags.
- strings/ctype-extra.c was regenerated with new flags.
- 8bit collations are not given MY_CS_ASCII_CASEUP_CI and
MY_CS_IDENT_CASEUP_CI if they are PAD and the space does not sort
before the digits. A shorter string is padded with spaces for
comparison, so the position of the space matters even for strings
without spaces ("ab" is compared to "abc" as "ab " to "abc").
A test collation latin1_space_after_letters_ci was added to
mysql-test/std_data/ldml/ for this. ctype-extra.c does not change.
- The initializer of my_collation_cs_handler in ctype-utf8.c
(utf8mb3_general_cs, under HAVE_UTF8_GENERAL_CS) was fixed to match
my_collation_handler_st: the missing get_id and get_collation_name
members were added, so that eq_collation was not initialized in their
place, and the new member "tailoring" is now set to my_tailoring_none.
- Results of the existing tests func_str, func_test, ps, view
and subselect_sj changed from "Illegal mix of collations" to success.
In these tests ASCII-only strings were compared using collations
with equal tailoring on ASCII, e.g. latin1_swedish_ci with
latin2_general_ci (func_str, ps), koi8r_general_ci with
latin1_swedish_ci (func_test), latin1_general_ci with
latin1_swedish_ci on letters and digits (view), cp932_japanese_ci
with latin1_swedish_ci (subselect_sj).
Such comparisons are now allowed.
Cases with collations that are still incompatible on ASCII,
e.g. latin7_general_ci, were added to the tests and still fail.
- Adding a number of MTR tests in
plugin/func_test/mysql-test/func_test/.
They display a tailoring by collation name and repertoire as
returned by:
cs->coll->tailoring(cs, some_repertoire)
A dynamically linked plugin function collation_tailoring() was added
for the purpose of these tests. Tailorings do not depend on PAD/NOPAD
and on the UCA version, so the function shows "[nopad]" for NOPAD
collations, and "[version X.Y.Z]" for UCA collations on the ASCII
repertoire. The version is formatted by a new class UCAVersion,
which uses a new method CharBuffer::append_uint8().
- Adding the test plugin/func_test/mysql-test/func_test/
ctype_caseup_ci_verify.test. It checks the order of ASCII characters
for all collations declared "caseup CI" on the ASCII or IDENT range.
- Adding the test plugin/func_test/mysql-test/func_test/
collation_tailoring_args.test. It checks the arguments of the plugin
function collation_tailoring(): the repertoire passed by name or by
number, and the NULL results for unknown names.
- Adding the test plugin/func_test/mysql-test/func_test/ctype_ldml.test.
It displays the tailorings of collations loaded at runtime from
Index.xml: simple 8bit collations (flags detected from the weights)
and Unicode collations (LDML rules which do or do not reorder ASCII).
- Adding a number of MTR tests mysql-test/main/ctype_xxx_tailoring.test
They display two-dimensional charts showing which collations
are compatible on which repertoires.
Tests for DATE_FORMAT() with an explicit locale were added to
ctype_big5_tailoring.test, ctype_cp932_tailoring.test and
ctype_latin5_tailoring.test, which also has the chart for latin5.
- Adding a number of MTR tests in the form of the originally
reported script for various collations:
SELECT Insert_priv FROM mysql.user WHERE Insert_priv='...';
The repeated blocks of queries are shared through
mysql-test/include/
ctype_tailoring_0{1,2,3}_mysql_user_Insert_priv*.inc.
- Adding tests to ctype_cp932.test checking that ExtractValue() and
UpdateXML() still raise "Illegal mix of collations".
A field of an element of an associative array of RECORDs, passed to
an OUT or INOUT parameter of a stored routine, crashed the server:
CREATE PROCEDURE p1(a OUT INT) AS BEGIN a:= 7; END;
DECLARE
TYPE rec_t IS RECORD (i INT, s VARCHAR(10));
TYPE tab_t IS TABLE OF rec_t INDEX BY INT;
t tab_t:= tab_t(1 => rec_t(0, 'x'));
BEGIN
p1(t(1).i);
END;
There were three problems:
- Item_splocal_assoc_array_element_field inherited set_value() of
Item_splocal_row_field_by_name, which treated the associative array
variable as a ROW variable (assertion failure in
sp_rcontext::virtual_tmp_table_for_row()). Added
Item_splocal_assoc_array_element_field::set_value(), which stores the
value into the field of the element.
- Before the call, sp_head::bind_input_param() destructed the whole
variable for an OUT argument, even if the argument was only a part of
the variable (a ROW field, an assoc array element or its field), so
the other elements of the array were lost: after p1(a0(2)), the array
a0(1), a0(2), a0(3) had only one element left. Added
Item_splocal::is_part_of_variable().
- sp_head::bind_output_param() evaluated the actual argument in the
context of the callee. For arguments like t(i) or t(i).field this
resolved the variables t and i in a wrong context. Now the argument
is evaluated in the context of the caller.
The second problem also affected a SYS_REFCURSOR field of a ROW variable
passed as an OUT argument: the cursors in the other fields of the ROW
variable were closed. Now they are kept.
Known limitations, not changed by this fix:
- The actual OUT argument is evaluated when the value is copied back,
not when the routine is called. If an earlier OUT argument of the same
call changes the variable used as the key, as in CALL p(i, a0(i)), the
new value of the key is used.
- For an OUT argument which is a part of a variable (a ROW field, an
assoc array element or its field), only the destruct of the whole
variable is skipped, the part itself is not destructed. A
SYS_REFCURSOR field of a ROW stays open during the call and is
released when the new value is copied back.
Tests: sp-assoc-array (OUT and INOUT elements and fields of elements,
NULL key, element deleted by the callee, ROW and RECORD fields) and
type_sys_refcursor-type_row-debug (SYS_REFCURSOR fields of a ROW
variable).
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Between two executions of a prepared statement, Item_sp::cleanup() freed
the stored function's memory root but left the arena's free list pointing
into it. That list is filled when the function call returns and the active
arena is restored.
On the next execution Item_sum_sp::clear() walked the stale list and
destroyed already freed items.
Free the items before the memory they live in. The sequence is now
in Item_sp::free_call_ctx(), so the three places that repeat it
cannot drift apart again.
setup_group no longer writes Item::marker. The ONLY_FULL_GROUP_BY
check now tests GROUP BY membership by walking the GROUP BY list.
A query that defines a WINDOW but never refers to it could crash in
add_key_field, for example
WITH cte AS (SELECT i FROM (SELECT i FROM t1 GROUP BY i) dt
WINDOW w AS (PARTITION BY i))
SELECT a.i FROM cte a JOIN cte b ON a.i=b.i WHERE a.i != 5;
A query that defines a WINDOW goes through setup_group, which set
marker to MARKER_UNDEF_POS (-1) on each GROUP BY expression so that
the ONLY_FULL_GROUP_BY check could skip it. Other code reads marker
as a set of flag bits (-1 sets all bits).
Item_direct_view_ref::grouping_field_transformer_for_where then took
the ref as flagged for substitution and followed a path that ends in
the crash.
The ONLY_FULL_GROUP_BY check was the only reader of that value, so
MARKER_UNDEF_POS is removed. The necessary check is local to the
setup_group function.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Implement shallow_copy() for Item_cache_year, same as for Item_cache_bool
to avoid typeid mismatch from inheriting this method from Item_cache_int.
Triggered when creating a clone of a condition for pushdown into a
derived table.
(Testcase amended by Sergei Petrunia)
EXCHANGE PARTITION fails with ERROR 1736 (Tables have different definitions)
when tables contain generated columns with AND/OR conditions, even when the
expressions are logically equivalent. This occurs because when expressions are
re-parsed (e.g., via CREATE TABLE ... LIKE), the order of arguments in AND/OR
conditions may change, but the comparison was order-sensitive.
The Item_cond::eq() method was implemented to perform set-based comparison
for commutative AND/OR operations. The set-based comparison algorithm ensures
that two Item_cond expressions are considered equal if they contain the same set
of equivalent arguments, regardless of order.
Added comprehensive test case covering:
- Generated columns with OR conditions
- Generated columns with AND conditions
- Multiple generated columns with different AND/OR combinations
- Nested AND/OR conditions
The fix allows EXCHANGE PARTITION to succeed when expressions are logically
equivalent but have different argument ordering, which is correct behavior
since AND/OR operations are commutative.
MDEV-38757 Limit unordered vcol condition comparison to EXCHANGE PARTITION
Review follow-up to the previous commit, which made Item_cond::eq()
compare AND/OR argument lists as sets for every caller. That changed
equality semantics globally and broke main.derived_cond_pushdown, where
conditions that eq() started reporting as equal were dropped from
attached_condition. Reordering AND/OR operands also changes evaluation
order, which is observable when operands are functions, so the relaxed
comparison must not be the default.
Item::Eq_config gains an unordered_conditions flag, defaulting to false,
next to the existing binary_cmp and omit_table_names flags.
Item_cond::eq() compares its argument lists as sets only when that flag
is set, and otherwise reports two distinct Item_cond objects as unequal,
as it did before this patch series. The flag is threaded through
Virtual_column_info::is_equal() and a new mysql_compare_tables()
parameter, and only Sql_cmd_alter_table_exchange_partition passes it as
true, so the relaxed comparison stays confined to the EXCHANGE PARTITION
metadata check.
The set comparison tests containment in both directions rather than
comparing element counts, so an expression also matches a form that
repeats one of its terms, for example
col1 > 10 and col2 < 100 or col3 > 50
col3 > 50 or col1 > 10 and col2 < 100 or col3 > 50
Test 5 of parts.partition_exchange_generated_columns covers that case.
When equality propagation (build_equal_items()) merges an equality that
contains a subquery, such as "t1.a = (SELECT ...)", with an outer join's
ON equality, it can inject a reference to that subquery into the ON
expression. If the join columns have compatible types the subquery ends
up as the constant of a multiple equality (which Item::walk() skips); if
they differ (e.g. BIGINT vs INT) the field cannot be merged and the
subquery is substituted in as a plain "tbl.col = (SELECT ...)" argument.
In the latter case, if that outer join is removed by table elimination,
mark_as_eliminated() walks the ON expression and flags the shared
Item_subselect as eliminated. The subquery, however, still lives in
another part of the query and has to be executed, tripping
DBUG_ASSERT(!eliminated) in Item_subselect::exec() (and, in release
builds, disabling the subquery cache and hiding it from EXPLAIN).
The surviving reference can be:
- a WHERE/HAVING/select-list/ORDER/GROUP expression (subquery written
there and pushed down into the eliminated ON), or
- the ON expression of an outer join that was not eliminated (subquery
written in a surviving outer ON and pushed down into an eliminated
inner one).
Fix: after table elimination, walk the expressions that survive into
execution (WHERE, HAVING, select list, ORDER/GROUP BY and the ON
expressions of outer joins that were not eliminated) and clear the
"eliminated" flag on any subquery still reachable from them.
Because a subquery can also be the constant of a multiple equality, and
Item::walk() does not visit an Item_equal's constant, Item_equal gets an
unmark_as_eliminated_processor() override that descends into its constant
explicitly.
Assisted by Claude Opus
When a grouping select referring to a derived table has an outer
reference within it's select list, that outer reference isn't
properly fixed. It isn't added to
outer_context->select_lex->inner_refs_list for later resolution during
JOIN::prepare. As it is not fixed, the Item pointer arrays associated
with the temporary table filled during the join process are now incorrect
and we end up with an incorrect join->outer_ref_cond evaluation and
potentially a wrong result.
Item_name_const forwarded type_handler() to the wrapped value but not
decimal_precision(), so a hex literal wrapped in NAME_CONST reported the
generic precision instead of its own. Forward decimal_precision() too.
A 0xHHHH / b'..' literal keeps its hex_hybrid type through COALESCE/IF/CASE/...,
but was evaluated, sized and stored as a string, so it behaved differently from
a bare literal (10+COALESCE(0x31) gave 11, not 59).
Type_handler_hex_hybrid now implements the numeric side to match the bare
literal: val_int/val_real/val_decimal read bytes as a hybrid, Item_decimal_precision
returns the integer precision, Item_save_in_field stores like the bare literal.
The traditional type merge keeps the hybrid through a typeless NULL (as for BIT),
and Item_hybrid_func_fix_attributes attributes it binary/unsigned.
In Item_func_json_array_append::fix_length_and_dec,
Item_bool::max_char_length() was being used to
calculate the length that is required to fit a boolean.
Previously it was returning 1, which doesn't fit
"false".
Change Item_bool to return strlen("false"), 5.
use_all_stored_columns() cleared read_set bits for all generated
columns, including persistent generated columns.
This caused stored generated columns to be accessed without being
marked for read, leading to a marked_for_read() assertion during
CHECKSUM TABLE EXTENDED.
Only exclude non-stored virtual generated columns from read_set.
When setup_copy_fields() needs to copy a non-aggregate function value,
it doesn't construct an Item_copy directly. Instead, it calls
Type_handler::create_item_copy, which is a kind of factory. The base
Type_handler::create_item_copy returns Item_copy_string. Some type
handlers override it, like timestamp and fixed binary. However, the
numeric type handlers (e.g., float, double, int, decimal) did not, so
they fell through to that base and got Item_copy_string.
A SELECT that aggregates will copy each non aggregate function value
into a temporary table through an Item_copy object, whose concrete
type is chosen by the create_item_copy method on the value's type
handler. For numeric types that method returned Item_copy_string,
which stores the value as text. A FLOAT keeps only FLT_DIG
significant digits as text, too few to reproduce its 24 bit mantissa,
so the copied value differed from the original. With one row per
group, CAST(c1 AS FLOAT) - MIN(CAST(c1 AS FLOAT)) returned a large
number instead of zero.
Add Item_copy_real with Item_copy_float and Item_copy_double variants
that keep the value as a double, the same way Item_cache_real does, and
let the float and double type handlers create them. This mirrors the
existing copy items for timestamp and fixed binary types.
The pushdown from HAVING into WHERE optimization cleans up and refixes
every condition to be pushed.
The virtual column (vcol) index substitution optimization replaces
vcol expressions in GROUP BY (and WHERE and ORDER BY) with vcol
fields.
The refixing requires the correct name resolution context to find the
vcol fields.
The commit 0316c6e4f2 assigns context
from the select_lex that the GROUP BY belongs to, but that may not
work when there are derived table subqueries.
In this commit we assign the correct context during substitution for
the newly constructed vcol Item_field in the substitution.
Also make the walk of vcol expressions with
intersect_field_part_of_key not descend to subquery since the
generated columns can't have subqueries.
Alternative considered:
1. Assign the context when constructing vcol_info, in
unpack_vcol_info_from_frm. This does not work because the
current_context() in parsing is not the correct context, not to
mention that unpack_vcol_info_from_frm is not always called from a
SELECT statement.
2. Get the correct context for vcol_info after its construction and
before the substitution. Debugger with watch -a vcol_info shows that
there are no common functions accessing vcol_info before the
substitution.
3. Cache the name resolution context value to a new vcol_info->context
field and assign value to it in an Item_field constructor as well as
during substitution. The problem with this is that
vcol_info->context->select_lex may be cleared at the end of the one
SELECT statement, which can cause crash in the next SELECT. It looks
like vcol_info is meant for "static"/ddl information, not
statement-specific info, so it is the wrong place to cache the
context.
(Edited by Sergei Petrunia)
Problem: when the optimizer attempts to push a condition in form
"merged_view.not_null_column IS NULL" into a derived table other
than the merged_view itself, it may crash.
The cause is this scenario:
1. Condition pushdown code uses walk() with excl_dep_on_grouping_fields()
(or excl_dep_on_table()) to determine whether an Item expression can
be pushed. Item_args::excl_dep_on_table() assumes that any item with
const_item()==1 can be pushed anywhere.
2. Item_func_isnull(not_null_expr) will report const_item()=1 as it
will always evaluate false. However it will have non-const expression
as its argument. As described in #1, this item will be considered pushable.
3. Transformation of item for pushdown is done in "transformer function"
Item_XXX::{derived|grouping}_field_transformer_for_where().
This function will be called for Item_func_isnull's argument.
If the argument is an Item_direct_view_ref, its transformer function
will call find_matching_field_pair() (or find_producing_item())
to locate the origin of the item inside the SELECT we're pushing the
condition into. It will get a NULL pointer (origin not found) and crash.
== The fix ==
Let Item_func_is_null(not_null_expr) report const_item()==true but
let its used_tables() still report those from not_null_expr.
Then, in Item_args, make excl_dep_on_grouping_fields() and
excl_dep_on_table() to check the used tables/columns for items
that report const_item()==true but have non-zero used_tables().
Add comments to
- find_producing_item()
- Item_field::derived_field_transformer_for_where()
- Item_field::grouping_field_transformer_for_where()
Also note that Item_default_value::grouping_field_transformer_for_where()
is not implemented.
During join execution of a query involving an aggregation,
ref items pointing to types that call Item_ref::val_native
(which includes TIMESTAMP fields) will return a value from
Item_field::field, rather than Item_field::result_field.
This will result in a potentially incorrect grouping comparison
and an incorrect result.
Affected statements will involve a derived grouped table with
an aggregation function TIMESTAMP field and outer having clause.
Add two functions intended for use from debugger:
bool dbug_is_mem_on_mem_root(MEM_ROOT *mem_root, void *ptr);
const char *dbug_which_mem_root(THD *thd, void *ptr);
Also, collect declarations of all other functions intended for use
from debugger in sql/sql_test.h
Problem:
A stored procedure querying a view with `CONCAT`/`GROUP_CONCAT` could
crash due to a NULL pointer dereference during query optimization.
During condition pushdown, `Item_direct_ref_to_item::deep_copy`
creates a clone where the clone's `ref` pointer incorrectly remains
pointing to the original object's `m_item`, rather than its own.
Because the clone is tethered to the original object, subsequent
transformations happen on the original item instead of the clone.
Calling `fix_fields` on malformed `Item_direct_ref_to_item` clone
resolves to `Item_field::fix_fields` instead of `Item_ref::fix_fields`
leaving the member `Item_ref::ref` uninitialized (NULL). When this is
dereferenced in `Item_ref::const_item`, server crashes.
Fix:
The `Item_direct_ref_to_item::set_item` method is made to update the
item and reference, making the clone complete. This results in correct
`fix_fields` resolution, making `Item_ref::ref` safe to access.
Adding support for cursors on prepared statements.
- SQL Standard way:
DECLARE c CURSOR FOR stmt;
PREPARE stmt FROM 'SELECT ?';
OPEN c USING 1;
- Oracle-style way with SYS_REFCURSOR variables:
DECLARE
c SYS_REFCURSOR;
BEGIN
OPEN c FOR 'SELECT ?' USING 1;
Fixing the return type of Type_handler::Item_save_in_value() from
bool to void.
Adding a new method st_value::is_null().
This makes the code less confusing.
