Problem: comparison of a DATETIME sp variable and NOW()
led to Illegal mix of collations error when
character_set_connection=utf8.
Introduced by "WL#2649 Number-to-string conversions".
Error happened in Arg_comparator::set_compare_func(),
because the first argument was errouneously converted to utf8,
while the second argument was not.
Fix: separate agg_arg_charsets_for_comparison() into two functions:
- agg_arg_charsets_for_comparison() - for pure comparison,
when we don't need to return any string result and therefore
don't need to convert arguments to @@character_set_connection:
SELECT a = b;
- agg_arg_charsets_for_string_results_with_comparison() - when
we need to return a string result, but we also need to do
comparison internally: SELECT REPLACE(a,b,c)
If all arguments are numbers:
SELECT REPLACE(123,2,3) -> 133
we convert arguments to @@character_set_connection.
@ mysql-test/include/ctype_numconv.inc
@ mysql-test/r/ctype_binary.result
@ mysql-test/r/ctype_cp1251.result
@ mysql-test/r/ctype_latin1.result
@ mysql-test/r/ctype_ucs.result
@ mysql-test/r/ctype_utf8.result
Adding tests
@ sql/item.cc
@ sql/item.h
@ sql/item_func.cc
@ sql/item_func.h
@ sql/item_strfunc.cc
Introducing and using new function
agg_item_charsets_for_string_result_with_comparison() and
its Item_func wrapper agg_arg_charsets_for_string_result_with_comparison().
Problem: comparison of a DATETIME sp variable and NOW()
led to Illegal mix of collations error when
character_set_connection=utf8.
Introduced by "WL#2649 Number-to-string conversions".
Error happened in Arg_comparator::set_compare_func(),
because the first argument was errouneously converted to utf8,
while the second argument was not.
Fix: separate agg_arg_charsets_for_comparison() into two functions:
- agg_arg_charsets_for_comparison() - for pure comparison,
when we don't need to return any string result and therefore
don't need to convert arguments to @@character_set_connection:
SELECT a = b;
- agg_arg_charsets_for_string_results_with_comparison() - when
we need to return a string result, but we also need to do
comparison internally: SELECT REPLACE(a,b,c)
If all arguments are numbers:
SELECT REPLACE(123,2,3) -> 133
we convert arguments to @@character_set_connection.
@ mysql-test/include/ctype_numconv.inc
@ mysql-test/r/ctype_binary.result
@ mysql-test/r/ctype_cp1251.result
@ mysql-test/r/ctype_latin1.result
@ mysql-test/r/ctype_ucs.result
@ mysql-test/r/ctype_utf8.result
Adding tests
@ sql/item.cc
@ sql/item.h
@ sql/item_func.cc
@ sql/item_func.h
@ sql/item_strfunc.cc
Introducing and using new function
agg_item_charsets_for_string_result_with_comparison() and
its Item_func wrapper agg_arg_charsets_for_string_result_with_comparison().
Analysis:
A query with implicit grouping is one with aggregate functions and
no GROUP BY clause. MariaDB inherits from MySQL an SQL extenstion
that allows mixing aggregate functions with non-aggregate fields.
If a query with such mixed select clause produces an empty result
set, the meaning of aggregate functions is well defined - either
NULL (MIN, MAX, etc.), or 0 (count(*)). However the non-aggregated
fields must also have some value, and the only reasonable value in
the case of empty result is NULL.
The cause of the many wrong results was that if a field is declared
as non-nullable (e.g. because it is a PK or NOT NULL), the semantic
analysis and the optimization phases treat this field as non-nullable,
and generate all related query plan elements based on this assumption.
Later during execution, these incorrectly configured/generated query
plan elements result in a wrong result because the selected fields
are not null due to the not-null assumption during optimization.
Solution:
Detect before the context analysys phase that a query uses implicit
grouping with mixed aggregates/non-aggregates, and set all fields
as nullable. The parser already walks the SELECT clause, and
already sets Item::with_sum_func for Items that reference aggreagate
functions. The patch adds a symmetric Item::with_field so that all
Items that reference an Item_field are marked during their
construction at parse time in the same way as with aggregate function
use.
Part 2. Function QUOTE() was not multi-byte safe.
@ mysql-test/r/ctype_ucs.result
@ mysql-test/t/ctype_ucs.test
Adding tests
@ sql/item_strfunc.cc
Fixing Item_func_quote::val_str to be multi-byte safe.
@ sql/item_strfunc.h
Multiple size needed for quote characters to mbmaxlen
Part 2. Function QUOTE() was not multi-byte safe.
@ mysql-test/r/ctype_ucs.result
@ mysql-test/t/ctype_ucs.test
Adding tests
@ sql/item_strfunc.cc
Fixing Item_func_quote::val_str to be multi-byte safe.
@ sql/item_strfunc.h
Multiple size needed for quote characters to mbmaxlen
Problem: wrong character set pointer was passed to my_strtoll10_mb2,
which led to DBUG_ASSERT failure in some cases.
@ mysql-test/r/func_encrypt_ucs2.result
@ mysql-test/t/func_encrypt_ucs2.test
@ mysql-test/r/ctype_ucs.result
@ mysql-test/t/ctype_ucs.test
Adding tests
@ sql/item_func.cc
"cs" initialization was wrong (res does not necessarily point to &str_value)
@ sql/item_strfunc.cc
Item_func_dec_encrypt::val_str() and Item_func_des_descrypt::val_str()
did not set character set for tmp_value (the returned value),
so the old value, which was previously copied from args[1]->val_str(),
was incorrectly returned with tmp_value.
Problem: wrong character set pointer was passed to my_strtoll10_mb2,
which led to DBUG_ASSERT failure in some cases.
@ mysql-test/r/func_encrypt_ucs2.result
@ mysql-test/t/func_encrypt_ucs2.test
@ mysql-test/r/ctype_ucs.result
@ mysql-test/t/ctype_ucs.test
Adding tests
@ sql/item_func.cc
"cs" initialization was wrong (res does not necessarily point to &str_value)
@ sql/item_strfunc.cc
Item_func_dec_encrypt::val_str() and Item_func_des_descrypt::val_str()
did not set character set for tmp_value (the returned value),
so the old value, which was previously copied from args[1]->val_str(),
was incorrectly returned with tmp_value.
- Fixed some issues with partitions and connection_string, which also fixed lp:716890 "Pre- and post-recovery crash in Aria"
- Fixed wrong assert in Aria
Now need to merge with latest xtradb before pushing
sql/ha_partition.cc:
Ensure that m_ordered_rec_buffer is not freed before close.
sql/mysqld.cc:
Changed to use opt_stack_trace instead of opt_pstack.
Removed references to pstack
sql/partition_element.h:
Ensure that connect_string is initialized
storage/maria/ma_key_recover.c:
Fixed wrong assert
Introduced by the fix for bug#44766.
Problem: it's not correct to use args[0]->str_value as a buffer,
because args[0] may need this buffer for its own purposes.
Fix: adding a new class member tmp_value to use as return value.
@ mysql-test/r/ctype_many.result
@ mysql-test/t/ctype_many.test
Adding tests
@ sql/item_strfunc.cc
Changing code into traditional style:
use "str" as a buffer for the argument and tmp_value for the result value.
@ sql/item_strfunc.h
Adding tmp_value
Introduced by the fix for bug#44766.
Problem: it's not correct to use args[0]->str_value as a buffer,
because args[0] may need this buffer for its own purposes.
Fix: adding a new class member tmp_value to use as return value.
@ mysql-test/r/ctype_many.result
@ mysql-test/t/ctype_many.test
Adding tests
@ sql/item_strfunc.cc
Changing code into traditional style:
use "str" as a buffer for the argument and tmp_value for the result value.
@ sql/item_strfunc.h
Adding tmp_value
other crashes
Some string manipulating SQL functions use a shared string object intended to
contain an immutable empty string. This object was used by the SQL function
SUBSTRING_INDEX() to return an empty string when one argument was of the wrong
datatype. If the string object was then modified by the sql function INSERT(),
undefined behavior ensued.
Fixed by instead modifying the string object representing the function's
result value whenever string manipulating SQL functions return an empty
string.
Relevant code has also been documented.
other crashes
Some string manipulating SQL functions use a shared string object intended to
contain an immutable empty string. This object was used by the SQL function
SUBSTRING_INDEX() to return an empty string when one argument was of the wrong
datatype. If the string object was then modified by the sql function INSERT(),
undefined behavior ensued.
Fixed by instead modifying the string object representing the function's
result value whenever string manipulating SQL functions return an empty
string.
Relevant code has also been documented.
Problem: DATE_ADD() is a hybrid function and can return
DATE, DATETIME or VARCHAR data type depending on arguments.
In case of VARCHAR data type, DATE_ADD() reported "binary" character set,
which was wrong.
Fix: make DATE_ADD() return @character_set_connection in VARCHAR context.
@ mysql-test/include/ctype_numconv.inc
Adding tests
@ mysql-test/r/ctype_binary.result
Adding tests
@ mysql-test/r/ctype_cp1251.result
Adding tests
@ mysql-test/r/ctype_latin1.result
Adding tests
@ mysql-test/r/ctype_ucs.result
Adding tests
@ mysql-test/r/ctype_utf8.result
Adding tests
@ sql/item_strfunc.cc
- Moving code from Item_str_ascii_func::val_str() to
Item_str_func::val_str_from_val_str_ascii(), as
this code needs to be shared by Item_date_add_interval.
- Adding str2 parameter to be used as a buffer, instead of
using private ascii_buf member.
@ sql/item_strfunc.h
- Moving code from Item_str_ascii_func::val_str() to
Item_str_func::val_str_from_val_str_ascii()
- Removing "String *val_str_convert_from_ascii(String *str, String *ascii_buf)"
prototype as it was neither used nor declared.
@ sql/item_timefunc.h
- Overwriting parent's charset_for_protocol() method,
becase we need to behave differenlty in VARCHAR and DATE/DATETYPE context.
- Adding ascii_buf for conversion.
- Adding val_str_ascii() prototype.
- Adding val_str() which uses newly added
Item_str_func::val_str_from_val_str_ascii(),
passing ascii_buf as a conversion buffer.
Problem: DATE_ADD() is a hybrid function and can return
DATE, DATETIME or VARCHAR data type depending on arguments.
In case of VARCHAR data type, DATE_ADD() reported "binary" character set,
which was wrong.
Fix: make DATE_ADD() return @character_set_connection in VARCHAR context.
@ mysql-test/include/ctype_numconv.inc
Adding tests
@ mysql-test/r/ctype_binary.result
Adding tests
@ mysql-test/r/ctype_cp1251.result
Adding tests
@ mysql-test/r/ctype_latin1.result
Adding tests
@ mysql-test/r/ctype_ucs.result
Adding tests
@ mysql-test/r/ctype_utf8.result
Adding tests
@ sql/item_strfunc.cc
- Moving code from Item_str_ascii_func::val_str() to
Item_str_func::val_str_from_val_str_ascii(), as
this code needs to be shared by Item_date_add_interval.
- Adding str2 parameter to be used as a buffer, instead of
using private ascii_buf member.
@ sql/item_strfunc.h
- Moving code from Item_str_ascii_func::val_str() to
Item_str_func::val_str_from_val_str_ascii()
- Removing "String *val_str_convert_from_ascii(String *str, String *ascii_buf)"
prototype as it was neither used nor declared.
@ sql/item_timefunc.h
- Overwriting parent's charset_for_protocol() method,
becase we need to behave differenlty in VARCHAR and DATE/DATETYPE context.
- Adding ascii_buf for conversion.
- Adding val_str_ascii() prototype.
- Adding val_str() which uses newly added
Item_str_func::val_str_from_val_str_ascii(),
passing ascii_buf as a conversion buffer.
Problem: LIKE over an indexed column optimized away good results,
because my_like_range_utf32/utf16 returned wrong ranges for contractions.
Contraction related code was missing in my_like_range_utf32/utf16,
but did exist in my_like_range_ucs2/utf8.
It was forgotten in utf32/utf16 versions (during mysql-6.0 push/revert mess).
Fix:
The patch removes individual functions my_like_range_ucs2,
my_like_range_utf16, my_like_range_utf32 and introduces a single function
my_like_range_generic() instead. The new function handles contractions
correctly. It can handle any character set with cs->min_sort_char and
cs->max_sort_char represented in Unicode code points.
added:
@ mysql-test/include/ctype_czech.inc
@ mysql-test/include/ctype_like_ignorable.inc
@ mysql-test/r/ctype_like_range.result
@ mysql-test/t/ctype_like_range.test
Adding tests
modified:
@ include/m_ctype.h
- Adding helper functions for contractions.
- Prototypes: removing ucs2,utf16,utf32 functions, adding generic function.
@ mysql-test/r/ctype_uca.result
@ mysql-test/r/ctype_utf16_uca.result
@ mysql-test/r/ctype_utf32_uca.result
@ mysql-test/t/ctype_uca.test
@ mysql-test/t/ctype_utf16_uca.test
@ mysql-test/t/ctype_utf32_uca.test
- Adding tests.
@ strings/ctype-mb.c
- Pad function did not put the last character.
- Implementing my_like_range_generic() - an universal replacement
for three separate functions
my_like_range_ucs2(), my_like_range_utf16() and my_like_range_utf32(),
with correct contraction handling.
@ strings/ctype-ucs2.c
- my_fill_mb2 did not put the high byte, as previously
it was used to put only characters in ASCII range.
Now it puts high byte as well
(needed to pupulate cs->max_sort_char correctly).
- Adding DBUG_ASSERT()
- Removing character set specific functions:
my_like_range_ucs2(), my_like_range_utf16() and my_like_range_utf32().
- Using my_like_range_generic() instead of the old functions.
@ strings/ctype-uca.c
- Using generic function instead of the old character set specific ones.
@ sql/item_create.cc
@ sql/item_strfunc.cc
@ sql/item_strfunc.h
- Adding SQL functions LIKE_RANGE_MIN and LIKE_RANGE_MAX,
available only in debug build to make sure like_range()
works correctly for all character sets and collations.
Problem: LIKE over an indexed column optimized away good results,
because my_like_range_utf32/utf16 returned wrong ranges for contractions.
Contraction related code was missing in my_like_range_utf32/utf16,
but did exist in my_like_range_ucs2/utf8.
It was forgotten in utf32/utf16 versions (during mysql-6.0 push/revert mess).
Fix:
The patch removes individual functions my_like_range_ucs2,
my_like_range_utf16, my_like_range_utf32 and introduces a single function
my_like_range_generic() instead. The new function handles contractions
correctly. It can handle any character set with cs->min_sort_char and
cs->max_sort_char represented in Unicode code points.
added:
@ mysql-test/include/ctype_czech.inc
@ mysql-test/include/ctype_like_ignorable.inc
@ mysql-test/r/ctype_like_range.result
@ mysql-test/t/ctype_like_range.test
Adding tests
modified:
@ include/m_ctype.h
- Adding helper functions for contractions.
- Prototypes: removing ucs2,utf16,utf32 functions, adding generic function.
@ mysql-test/r/ctype_uca.result
@ mysql-test/r/ctype_utf16_uca.result
@ mysql-test/r/ctype_utf32_uca.result
@ mysql-test/t/ctype_uca.test
@ mysql-test/t/ctype_utf16_uca.test
@ mysql-test/t/ctype_utf32_uca.test
- Adding tests.
@ strings/ctype-mb.c
- Pad function did not put the last character.
- Implementing my_like_range_generic() - an universal replacement
for three separate functions
my_like_range_ucs2(), my_like_range_utf16() and my_like_range_utf32(),
with correct contraction handling.
@ strings/ctype-ucs2.c
- my_fill_mb2 did not put the high byte, as previously
it was used to put only characters in ASCII range.
Now it puts high byte as well
(needed to pupulate cs->max_sort_char correctly).
- Adding DBUG_ASSERT()
- Removing character set specific functions:
my_like_range_ucs2(), my_like_range_utf16() and my_like_range_utf32().
- Using my_like_range_generic() instead of the old functions.
@ strings/ctype-uca.c
- Using generic function instead of the old character set specific ones.
@ sql/item_create.cc
@ sql/item_strfunc.cc
@ sql/item_strfunc.h
- Adding SQL functions LIKE_RANGE_MIN and LIKE_RANGE_MAX,
available only in debug build to make sure like_range()
works correctly for all character sets and collations.
Open issues:
- A better fix for #57688; Igor is working on this
- Test failure in index_merge_innodb.test ; Igor promised to look at this
- Some Innodb tests fails (need to merge with latest xtradb) ; Kristian promised to look at this.
- Failing tests: innodb_plugin.innodb_bug56143 innodb_plugin.innodb_bug56632 innodb_plugin.innodb_bug56680 innodb_plugin.innodb_bug57255
- Werror is disabled; Should be enabled after merge with xtradb.
Bug#57820 extractvalue crashes
Problem: ExtractValue and Replace crashed in some cases
due to invalid handling of empty and NULL arguments.
Per file comments:
@mysql-test/r/ctype_ujis.result
@mysql-test/r/xml.result
@mysql-test/t/ctype_ujis.test
@mysql-test/t/xml.test
Adding tests
@sql/item_strfunc.cc
Make sure Item_func_replace::val_str safely handles empty strings.
@sql/item_xmlfunc.cc
set null_value if nodeset_func returned NULL,
which is possible when the second argument is an
unset user variable.
Bug#57820 extractvalue crashes
Problem: ExtractValue and Replace crashed in some cases
due to invalid handling of empty and NULL arguments.
Per file comments:
@mysql-test/r/ctype_ujis.result
@mysql-test/r/xml.result
@mysql-test/t/ctype_ujis.test
@mysql-test/t/xml.test
Adding tests
@sql/item_strfunc.cc
Make sure Item_func_replace::val_str safely handles empty strings.
@sql/item_xmlfunc.cc
set null_value if nodeset_func returned NULL,
which is possible when the second argument is an
unset user variable.
Problem: some call of INET_NTOA() function may lead
to a crash due to missing its character set initialization.
Fix: explicitly set the character set.
mysql-test/r/func_misc.result:
Fix for bug#57283: inet_ntoa() crashes
- test result.
mysql-test/t/func_misc.test:
Fix for bug#57283: inet_ntoa() crashes
- test case.
sql/item_strfunc.cc:
Fix for bug#57283: inet_ntoa() crashes
- explicitly set buffer's character set.
Problem: if multibyte and binary string arguments passed to
RPAD(), LPAD() or INSERT() functions, they might return
wrong results or even lead to a server crash due to missed
character set convertion.
Fix: perform the convertion if necessary.
mysql-test/r/ctype_utf8.result:
Fix for bug#57272: crash in rpad() when using utf8
- test result.
mysql-test/t/ctype_utf8.test:
Fix for bug#57272: crash in rpad() when using utf8
- test case.
sql/item_strfunc.cc:
Fix for bug#57272: crash in rpad() when using utf8
- convert multibyte argument's character set to binary in case of
FUNCTION(MULTIBYTE_ARG, .., BINARY_ARG,..) for RPAD(), LPAD() and
INSERT() functions.
Problem: if multibyte and binary string arguments passed to
RPAD(), LPAD() or INSERT() functions, they might return
wrong results or even lead to a server crash due to missed
character set convertion.
Fix: perform the convertion if necessary.
Buffer overrun when trying to format DBL_MAX
mysql-test/r/func_math.result:
Add test case for Bug#57209
mysql-test/t/func_math.test:
Add test case for Bug#57209
sql/item_strfunc.cc:
Allocate a larger buffer for the result.
