This merges the patch series that fixes a lot of missing validation
checks for read_log_event(). The merge includes a few added fixes that
are needed in new code introduced in 12.3.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
This SQL script failed:
SET NAMES latin1 COLLATE latin1_swedish_ci;
CREATE OR REPLACE VIEW v1 AS SELECT 'Y' AS c1;
SET NAMES big5 COLLATE big5_chinese_ci;
SELECT * FROM v1 WHERE c1='y';
with the following error:
ERROR 1267 (HY000): Illegal mix of collations
(latin1_swedish_ci,COERCIBLE) and (big5_chinese_ci,COERCIBLE)
for operation '='
Note, latin1_swedish_ci and big5_chinese_ci are used here as examples.
The error also happened with different collation combinations.
Fix main idea:
If two collations have equal comparison rules (known as "tailoring")
on a given character repertoire,
like latin1_swedish_ci and big5_chinese_ci on ASCII letters,
then the "Illegal mix of collations" error can be avoided in
a comparison operator. We can choose any of the sides as the operation
effective collation - the result will be equal.
This optimization is not applied when at least one side has an explicit
COLLATE clause. Two explicit COLLATE clauses in one comparison are
already illegal when the character sets are the same, so for
consistency this stays illegal when the character sets differ too.
Most important details:
- Splitting enum_repertoire_t into smaller subsets,
for better repertoire granularity.
A variable holding a repertoire value can now have multiple
MY_REPERTOIRE_XXX flags set.
This patch implements detecting tailoring equality
on these repertoires:
* MY_REPERTOIRE_ASCII_DIGITS - [0..9]
* MY_REPERTOIRE_ASCII_MINUS_DOT_DIGITS - [-.0..9], e.g. '-10.5'.
It consists of the new bits MY_REPERTOIRE_ASCII_MINUS and
MY_REPERTOIRE_ASCII_DOT. They were a part of
MY_REPERTOIRE_ASCII_NOT_IDENT before. As a result, the numeric values
of MY_REPERTOIRE_ASCII and MY_REPERTOIRE_UNICODE30 changed
(223 and 255).
* MY_REPERTOIRE_ASCII_ALNUM - [A..Z,a..z,0..9].
* MY_REPERTOIRE_ASCII_IDENT - ALNUM + underscore
* MY_REPERTOIRE_ASCII - the entire range U+0000..U+007F
- Adding a new virtual function "tailoring" in my_collation_handler_st.
It returns the tailoring on the given repertoire for the given
collation.
If cs1->coll->tailoring(cs1, some_repertoire) returns {0,0},
it means the illegal mix optimization cannot be used
for this collation on the given repertoire.
If these calls:
tr1= cs1->coll->tailoring(cs1, some_repertoire);
tr2= cs2->coll->tailoring(cs2, some_repertoire);
return both non-NULL results and tr1.str==tr2.str,
then these collations are equal on the given repertoire
and are mutually replaceable for a comparison operator,
so "Illegal mix of collations" can be avoided.
- Tailoring strings are shared constants my_tailoring_str_*
(strings/strings_def.h). Tailorings are compared by the string
pointer, so tailoring() implementations return strings only from
these constants. The strings do not depend on PAD/NOPAD.
- my_string_repertoire() for ucs2, utf16, utf32 and NONASCII 8bit
character sets no longer stops at the first bad byte sequence with
the repertoire found so far, which could be MY_REPERTOIRE_NONE
(compatible with everything). It returns MY_REPERTOIRE_EXTENDED
for such strings. A unit test was added into unittest/strings.
- Adding CHARSET_INFO::is_ascii_superset(). It tells if a character set
can store all ASCII characters (it is false for swe7).
It is used in Item_func_conv_charset to calculate "safe", and in
left_is_algorithmically_simpler() to prefer other character sets
to those where ASCII strings cannot be converted safely.
- DBUG_ASSERTs were added into mysys/charset.c, to check that
the "tailoring" method is set in collation handlers when a collation
is added: compiled-in, from ctype-extra.c, or loaded from Index.xml.
- Adding a new function my_collations_equal_on_repertoire(cs1, cs2,
repertoire) in strings/. It tells if two collations are equal on
the repertoire, taking tailoring() results and PAD/NOPAD and the UCA
version into account. Callers do not compare tailoring() results
directly, so the way tailorings are compared (currently by the pointer
to the shared constant string) is private to strings/ and can be
extended later, e.g. for UCA collations on MY_REPERTOIRE_UNICODE30.
- Adding a new method DTCollation::aggregate_by_tailoring().
Collations with different MY_CS_NOPAD are never compatible.
UCA collations of different versions are not compatible on
the repertoire with ASCII punctuation, as the UCA version
affects its order (e.g. UCA-6.2.0 moved GRAVE ACCENT
and CIRCUMFLEX ACCENT after PERCENT SIGN).
- DTCollation::aggregate() now makes the repertoire of the result cover
the repertoires of both sides in all successful branches.
Before, set(dt) replaced the repertoire with the repertoire of the
winner side only, so the result could have a too narrow repertoire,
e.g. IF(1, 1.5, HEX(255)) had MY_REPERTOIRE_ASCII_ALNUM although the
value '1.5' has a punctuation character. With the more granular
repertoires this could wrongly allow mixing collations which are
equal on ALNUM but different on punctuation.
- Adding a new flag MY_COLL_ALLOW_BY_TAILORING.
It indicates to DTCollation::aggregate() that the illegal
mix optimization by repertoire can be used in the given context.
MY_COLL_CMP_CONV now includes MY_COLL_ALLOW_BY_TAILORING.
Note, only comparison operators pass this flag.
Functions returning a string result do not pass this flag,
because in operations like CONCAT(a,b) we still need to evaluate
precisely the collation of the result - we cannot just choose
a collation of one of the sides (even if they are compatible
on the given repertoire). This also applies to ExtractValue()
and UpdateXML(), which now aggregate their arguments without
this flag.
- As in my_repertoire_t the value MY_REPERTOIRE_ASCII is now a set of
bits rather than a single bit, the way to detect "is only ASCII"
repertoires has changed in the code.
For example:
// repertoire *IS* ascii
if (repertoire == MY_REPERTOIRE_ASCII)
has changed in multiple places in the code to
// repertoire *HAS* only ascii characters
if (my_repertoire_is_subset_of(repertoire, MY_REPERTOIRE_ASCII))
The new function my_repertoire_is_subset_of() in m_ctype.h
is used for this purpose in C code. In C++ code
DTCollation::repertoire_is_subset_of() is used, e.g.:
if (collation.repertoire_is_subset_of(MY_REPERTOIRE_ASCII))
- Repertoire of some expressions was adjusted to the new meaning:
* Item_null now has MY_REPERTOIRE_NONE (was ASCII).
* MY_LOCALE::repertoire() now returns MY_REPERTOIRE_UNICODE30
(was EXTENDED).
* Lex_string_with_metadata_st::repertoire(cs) now scans the string
contents to detect the actual repertoire.
* HEX() now has MY_REPERTOIRE_ASCII_ALNUM.
* DATE_FORMAT() decides on MY_REPERTOIRE_EXTENDED using the locale
which is actually used: the explicit third argument, or
@@lc_time_names. It was always @@lc_time_names before.
A non-constant locale argument is assumed to be non-ASCII.
* QUOTE, MAKE_SET, EXPORT_SET, LPAD, RPAD, GROUP_CONCAT, JSON_ARRAY,
JSON_OBJECT and JSON_OBJECTAGG add the repertoire of the extra
characters they put into the result (quotes, separators, padding,
brackets).
* LOWER() and UPPER() add the letters of both cases to the repertoire,
because the result can contain letters of the opposite case.
In Turkish collations they also add MY_REPERTOIRE_EXTENDED, as an
ASCII letter can be converted to a non-ASCII one (I -> dotless i).
Unicode collations are detected by the new member
casefold_info_st::can_convert_to_non_ascii_on_casefolding,
simple 8bit collations by to_lower['I'] and to_upper['i'].
- Lex_string_with_metadata_st::repertoire(cs) now scans a string literal
for any byte oriented character set (mbminlen==1), not only for
ASCII based ones. It was assumed to have the full repertoire for swe7,
which is not ASCII based. The scan is done by Unicode code points, so
swe7 strings with digits and letters get a narrow repertoire, while
e.g. 0x5B (LATIN CAPITAL LETTER A WITH DIAERESIS in swe7) is non-ASCII.
Therefore swe7_swedish_ci can be mixed with other collations now,
in the same way as other collations with MY_CS_IDENT_CASEUP_CI.
The test func_time was adjusted: a swe7 literal ' ull' with ASCII
characters is now safe to convert, the error is still tested for '[ull'.
- cp1250_czech_cs sorts -, . and 0..9 in the code point order, so it gets
the MINUS_DOT_DIGITS tailoring. latin2_czech_cs gets only DIGITS,
as it sorts the minus and the dot differently. Both were verified
with ctype_digits_verify.
- The tis620 collations now set CHARSET_INFO::tab_to_uni (was NULL),
so my_charset_is_ascii_based() is true for tis620.
It was false for tis620, although tis620 is ASCII compatible.
Therefore tis620 string literals were always considered to have
a non-ASCII repertoire, and the repertoire optimizations did not work
for them. Now ASCII-only tis620 literals are detected as ASCII.
This changes the result of subselect_extra_no_semijoin
from an error to success.
- tis620_thai_ci and tis620_thai_nopad_ci fold letters to lower case
and then compare by the code point on the entire ASCII range
(the Thai specific rules affect only non-ASCII characters).
A new function my_tailoring_ascii_casedn_ci() returns tailorings
for such collations on ALNUM, IDENT and ASCII. For example, the
underscore sorts before the letters, unlike in the caseup tailorings.
- New flags were added for CHARSET_INFO::state
* MY_CS_ASCII_CASEUP_CI - for simple 8bit case insensitive collations.
It means that this collation has no irregularities on the ASCII
repertoire, and converts lower case letters to upper case ones,
so the underscore sorts after the letters (with an upper to lower
case conversion it would sort before the letters).
* MY_CS_IDENT_CASEUP_CI - for simple 8bit case insensitive collations.
It means that this collation has no irregularities on the IDENT
repertoire (but can have irregularities say on punctuation).
* MY_CS_ASCII_STD_UCA - for UCA collations.
It means that a UCA collation does not reorder ASCII characters.
* MY_CS_DIGITS_STD - for any collation. It means that the ASCII digits
0..9 are sorted in the code point order and are not tailored, and,
for PAD collations, the space sorts before the digits. Strings of
digits are compared in the same way in all such collations, even if
they have irregularities on letters (latin7, cp866, czech, danish,
turkish UCA, etc.), or differ by case sensitivity. It is detected
from sort_order for 8bit collations, and from the tailoring rules
for UCA collations. Collations with MY_CS_DIGITS_STD still must have
the same PAD/NOPAD attribute.
* MY_CS_ASCII_MINUS_DOT_DIGITS - for any collation. Means
MY_CS_DIGITS_STD, and additionally the minus sorts before the dot,
which sorts before the digits (and the space sorts before the minus
in PAD collations). Strings like '-10.5' are compared in the same
way in all such collations. It is detected from sort_order for simple
8bit collations (my_coll_init_simple()), and from the tailoring rules
for UCA collations. For the multi-byte collations big5_chinese_ci,
gbk_chinese_ci, gb2312_chinese_ci and sjis_japanese_ci it is declared
by my_tailoring_ident_caseup_ci_generic(). The flags MY_CS_DIGITS_STD
and MY_CS_ASCII_MINUS_DOT_DIGITS are stored in ctype-extra.c for
compiled collations (conf_to_src.c prints them), and the detection is
skipped for a collation which already has any of the repertoire flags
(MY_CS_REPERTOIRE_FLAGS).
- strings/conf_to_src.c was modified to detect and print
MY_CS_ASCII_CASEUP_CI and MY_CS_IDENT_CASEUP_CI flags.
- strings/ctype-extra.c was regenerated with new flags.
- 8bit collations are not given MY_CS_ASCII_CASEUP_CI and
MY_CS_IDENT_CASEUP_CI if they are PAD and the space does not sort
before the digits. A shorter string is padded with spaces for
comparison, so the position of the space matters even for strings
without spaces ("ab" is compared to "abc" as "ab " to "abc").
A test collation latin1_space_after_letters_ci was added to
mysql-test/std_data/ldml/ for this. ctype-extra.c does not change.
- The initializer of my_collation_cs_handler in ctype-utf8.c
(utf8mb3_general_cs, under HAVE_UTF8_GENERAL_CS) was fixed to match
my_collation_handler_st: the missing get_id and get_collation_name
members were added, so that eq_collation was not initialized in their
place, and the new member "tailoring" is now set to my_tailoring_none.
- Results of the existing tests func_str, func_test, ps, view
and subselect_sj changed from "Illegal mix of collations" to success.
In these tests ASCII-only strings were compared using collations
with equal tailoring on ASCII, e.g. latin1_swedish_ci with
latin2_general_ci (func_str, ps), koi8r_general_ci with
latin1_swedish_ci (func_test), latin1_general_ci with
latin1_swedish_ci on letters and digits (view), cp932_japanese_ci
with latin1_swedish_ci (subselect_sj).
Such comparisons are now allowed.
Cases with collations that are still incompatible on ASCII,
e.g. latin7_general_ci, were added to the tests and still fail.
- Adding a number of MTR tests in
plugin/func_test/mysql-test/func_test/.
They display a tailoring by collation name and repertoire as
returned by:
cs->coll->tailoring(cs, some_repertoire)
A dynamically linked plugin function collation_tailoring() was added
for the purpose of these tests. Tailorings do not depend on PAD/NOPAD
and on the UCA version, so the function shows "[nopad]" for NOPAD
collations, and "[version X.Y.Z]" for UCA collations on the ASCII
repertoire. The version is formatted by a new class UCAVersion,
which uses a new method CharBuffer::append_uint8().
- Adding the test plugin/func_test/mysql-test/func_test/
ctype_caseup_ci_verify.test. It checks the order of ASCII characters
for all collations declared "caseup CI" on the ASCII or IDENT range.
- Adding the test plugin/func_test/mysql-test/func_test/
collation_tailoring_args.test. It checks the arguments of the plugin
function collation_tailoring(): the repertoire passed by name or by
number, and the NULL results for unknown names.
- Adding the test plugin/func_test/mysql-test/func_test/ctype_ldml.test.
It displays the tailorings of collations loaded at runtime from
Index.xml: simple 8bit collations (flags detected from the weights)
and Unicode collations (LDML rules which do or do not reorder ASCII).
- Adding a number of MTR tests mysql-test/main/ctype_xxx_tailoring.test
They display two-dimensional charts showing which collations
are compatible on which repertoires.
Tests for DATE_FORMAT() with an explicit locale were added to
ctype_big5_tailoring.test, ctype_cp932_tailoring.test and
ctype_latin5_tailoring.test, which also has the chart for latin5.
- Adding a number of MTR tests in the form of the originally
reported script for various collations:
SELECT Insert_priv FROM mysql.user WHERE Insert_priv='...';
The repeated blocks of queries are shared through
mysql-test/include/
ctype_tailoring_0{1,2,3}_mysql_user_Insert_priv*.inc.
- Adding tests to ctype_cp932.test checking that ExtractValue() and
UpdateXML() still raise "Illegal mix of collations".
DATE_FORMAT calculated its maximum result length solely from the format
string. Formatting a negative TIME also prepends a minus sign, so the
cursor protocol trusted an undersized result and truncated -01 to -0.
Account for the possible sign in format_length() for TIME_FORMAT() and
DATE_FORMAT() with a native TIME argument. Enable the existing cursor
protocol regression cases for binary, latin1, and utf32 result character
sets.
Problem:
The following functions-
- `Item_func_last_day::get_date`
- `Item_date_typecast::get_date`
- `Item_datetime_typecast::get_date` forwarded the TIME_INTERVAL_DAY /
TIME_INTERVAL_hhmmssff flags, set by an enclosing EXTRACT, into the
DATE/DATETIME conversion of their argument.This made, for ex:
LAST_DAY(200012010000) and CAST(200012010000 AS DATETIME) parse number
as TIME interval instead of a DATETIME. Under ZERO_DATE_TIME_CAST
that hour became an out-of-range month, overflowing days_in_month in
`check_date` and invalidating the Datetime.
Fix:
A DATE / DATETIME can never be an interval, so suppress the
TIME_INTERVAL_hhmmssff / TIME_INTERVAL_DAY flags when `LAST_DAY` and
`CAST` converts its argument. Also add a debug assert in `check_date`
to catch any future caller that violates its "month is assumed valid"
contract.
Before this fix, TO_DATE() returned NULL when the date string
contained extra non-alphanumeric separators that were not present
in the format string. For example:
TO_DATE('2002 - AD', 'YYYY AD') -- returned NULL
TO_DATE('2024---01---01', 'YYYYMMDD') -- returned NULL
Oracle handles these cases flexibly by ignoring extra separators.
Fix: skip non-alphanumeric separators in the date string before
matching each format token. A '-' or '+' sign is preserved when
the format is SYYYY, to allow signed year parsing.
Reported-by: Elena Stepanova <elenastepanova@mariadb.com>
Signed-off-by: Alexandru Diaconu <alexandrudiaconu93@gmail.com>
Remove the unscoped #pragma GCC diagnostic ignored
"-Wdeprecated-declarations" from include/violite.h and replace all
sprintf/vsprintf calls with snprintf/vsnprintf across the codebase.
Where possible, pass actual buffer sizes as function parameters
instead of using hardcoded constants or recomputing strlen().
Notable API changes:
- get_date(): added to_len parameter for buffer size
- nice_time(), end_timer() in client/mysql.cc: added buff_size
- generate_new_name(): added name_size parameter
- calc_md5(): added buffer_size parameter (fixes off-by-one)
- MD5_HASH_TO_STRING macro: added _size parameter
- Added MYSQL_UDF_MAX_RESULT_LENGTH constant in mysql_com.h
Also fixed a pre-existing bug in sql/rpl_mi.cc where the ellipsis
"..." was written to the wrong buffer (dbuff instead of buff).
Vendored code is excluded: extra/readline, extra/wolfssl,
wsrep-lib, libmariadb, storage/mroonga/vendor, zlib.
The Oracle-mode TO_DATE() function incorrectly parsed strings containing
conflicting format specifiers within the Day-name family (e.g., 'DAY DY')
without throwing an ER_STD_INVALID_ARGUMENT error.
This patch completes the existing validation framework for family-based
conflict checks in item_timefunc.cc during format string parsing.
Specifically, it adds formats_used() checks for FMT_DAY, ensuring that
specifiers within the Day-name family strictly block each other. This
aligns the Day-name handling with the already existing checks for the
Year, Month, Hour, and Meridian families.
Compatible specifiers, such as YYYY and DDD (Year and Day of Year),
remain unaffected and parse successfully. A comprehensive MTR test
suite has been added to func_to_date.test to cover standard and
edge-case conflicts.
Reported-by: Elena Stepanova <elenastepanova@github>
Signed-off-by: Alexandru Diaconu <alexandrudiaconu93@gmail.com>
Update STR_TO_DATE() to accept an optional third argument for the locale.
DATE_FORMAT() and STR_TO_DATE() return NULL when the locale argument is NULL;
issue a warning and fall back to the session @@lc_time_names for invalid locale names.
STR_TO_DATE() can now be used in indexed virtual columns while having all 3 arguments,
but not allowed when 2 as it depends then on the session @@lc_time_names
Months names were taken from https://unicode.org/cldr/charts/48/summary/root.html,
which has two different categories for full month names: "Months - wide - Formatting"
and "Months - wide - Standalone".
Formatting is normaly used when the month is a part of a date (e.g., "12 January 2026").
In many languages (like Slavic or Baltic), the month's ending changes in this context.
Standalone is used when the month name appears alone.
%B prints "formatting" months names, %M - "standalone" names
Syntax:
TO_DATE(string_expression [DEFAULT string_expression ON CONVERSION ERROR],
format_string [,NLS_FORMAT_STRING])
The format_string has the same format elements as TO_CHAR(), except a
few elements that are not supported/usable for TO_DATE().
TO_DATE() returns a datetime or date value, depending on if the format
element FF is used.
Allowed separators, same as TO_CHAR():
space, tab and any of !#%'()*+,-./:;<=>
'&' can also be used if next character is not a character a-z or A-Z
"text' indicates a text string that is verbatim in the format. One cannot
use " as a separator.
Format elements supported by TO_DATE():
AD Anno Domini ("in the year of the Lord")
AD_DOT Anno Domini ("in the year of the Lord")
AM Meridian indicator (Before midday)
AM_DOT Meridian indicator (Before midday)
DAY Name of day
DD Day (1-31)
DDD Day of year (1-336)
DY Abbreviated name of day
FF[1-6] Fractional seconds
HH Hour (1-12)
HH12 Hour (1-12)
HH24 Hour (0-23)
MI Minutes (0-59)
MM Month (1-12)
MON Abbreviated name of month
MONTH Name of Month
PM Meridian indicator (After midday)
PM_DOT Meridian indicator (After midday)
RR 20th century dates in the 21st century. 2 digits
50-99 is assumed from 2000, 0-49 is assumed from 1900.
RRRR 20th century dates in the 21st century. 4 digits
SS Seconds
SYYYY Signed 4 digit year; MariaDB only supports positive years
Y 1 digit year
YY 2 digits year
YYY 3 digits year
YYYY 4 digits year
Note that if there is a missing part of the date, the current date is used!
For example if 'MM-DD HH-MM-SS' then the current year will be used.
(Oracle behaviour)
Not supported options:
- BC, D, DL, DS, E, EE, FM, FX, RM, SSSSS, TS, TZD, TZH, TZR, X,SY
BC is not supported by MariaDB datetime.
- Most of the other are exotic formats does not make sence in MariaDB as
we return datetime or datetime with fractions, not string.
- D (day-of-week) is not supported as it is not clear exactly how it would
map to MariaDB. This element depends on the NLS territory of the session.
- RR only works with 2 digit years (In Oracle RR can also work with 4
digit years in some context but the rules are not clear).
Extensions / differences compared to Oracle;
- MariaDB supports FF (fractional seconds). If FF[#] is used,
then TO_DATE will return a datetime with # of subseconds.
If FF is not used a datetime will be returned.
There is warning (no error) if string contains more digts than what
is specified with F(#]
- Names can be shortened to it's unique prefix. For example January and Ja
works fine.
- No error if the date string is shorter format_string and the next
not used character is not a number.. This is useful to get a date
from a mixed set of strings in date or datetime format.
Oracle gives an error if date string is too short.
- MariaDB supports short locales as language names
- NLS_DATE_FORMAT can use both " and ' for quoting.
- NLS_DATE_FORMAT must be a constant string.
- This is to ensure that the server knows which locale to use
when executing the function.
New formats handled by TO_CHAR():
FF[1-6] Fractional seconds
DDD Daynumber 1-366
IW Week 1-53 according to ISO 8601
I 1 digit year according to ISO 8601
IY 2 digit year according to ISO 8601
IYY 3 digit year according to ISO 8601
IYYY 4 digit year according to ISO 8601
SYYY 4 digit year according to ISO 8601 (Oracle can do signed)
Supported NLS_FORMAT_STRING options are:
NLS_CALENDAR=GREGORIAN
NLS_DATE_LANGUAGE=language
Support languages are:
- All MariaDB short locales, like en_AU.
- The following Oracle language names:
ALBANIAN, AMERICAN, ARABIC, BASQUE, BELARUSIAN, BRAZILIAN PORTUGUESE
BULGARIAN, CANADIAN FRENCH, CATALAN, CROATIAN, CYRILLIC SERBIAN CZECH,
DANISH, DUTCH, ENGLISH, ESTONIAN, FINNISH, FRENCH, GERMAN,
GREEK, HEBREW, HINDI, HUNGARIAN, ICELANDIC, INDONESIAN ITALIAN,
JAPANESE, KANNADA, KOREAN, LATIN AMERICAN SPANISH, LATVIAN,
LITHUANIAN, MACEDONIAN, MALAY, MEXICAN SPANISH, NORWEGIAN, POLISH,
PORTUGUESE, ROMANIAN, RUSSIAN, SIMPLIFIED CHINESE, SLOVAK, SLOVENIAN,
SPANISH, SWAHILI, SWEDISH, TAMIL, THAI, TRADITIONAL CHINESE, TURKISH,
UKRAINIAN, VIETNAMESE
Development bugs fixed:
MDEV-38403 Server crashes in Item_func_to_date::fix_length_and_dec upon
using an invalid argument
MDEV-38400 compat/oracle.func_to_date fails with PS protocol and cursor
protocol (Fixed by Serg)
MDEV-38404 TO_DATE: MTR coverage omissions, round 1
MDEV-38509 TO_DATE: AD_DOT does not appear to be supported
MDEV-38513 TO_DATE: NULL value for format string causes assertion failure
MDEV-38521 TO_DATE: Date strings with non-ASCII symbols cause warnings
and wrong results
MDEV-38578 TO_DATE: Possibly unexpected results upon wrong input
MDEV-38582 TO_DATE: NLS_DATE_LANGUAGE=JAPANESE does not parse values
which work in Oracle
MDEV-38584 TO_DATE: NLS_DATE_LANGUAGE=VIETNAMESE does not parse values
which work in Oracle
MDEV-38703 TO_DATE: Quotation for multi-word NLS_DATE_LANGUAGE leads
to syntax error in view definition
MDEV-38675 TO_DATE: MSAN/Valgrind/UBSAN errors in
extract_oracle_date_time
MDEV-38635 TO_DATE: UBSAN errors in item_timefunc.h upon comparison with
a view column
MDEV-38719 TO_DATE: Assertion `&my_charset_bin != charset()' failed in
String::append_for_single_quote_using_mb_wc
MDEV-38756 TO_DATE: MSAN/Valgrind errors in
Item_func_to_date::fix_length_and_dec upon PREPARE with
parameters
Known issues:
- Format string character matches inside quotes are done
one-letter-to-one-letter, like in LIKE predicate. That means things
like expansions and contractions do not work.
For example 'ss' does not match 'ß' in collations which treat them
as equal for the comparison operator.
Match is done taking into account case and accent sensitivity
of the subject argument collation, so for example this now works:
MariaDB [test]> SELECT TO_DATE('1920á12','YYYY"a"MM') AS c;
+---------------------+
| c |
+---------------------+
| 1920-12-17 00:00:00 |
+---------------------+
Co-author and reviewer: Alexander Barkov <bar@mariadb.com>
If one compiles with EXTENDED_MONTHS_BETWEEN MariaDB would take hours,
minutes, seconds and fractional seconds into account when comparing
dates when computing the fractional months.
For example
MONTHS_BETWEEN("1995-02-15 12:00:00", "1995-01-17 12:00:01") would
return "one day less" than if time part would not have been given.
It returns the DATETIME data type (which is the closest to Oracle's DATE).
The following Oracle formats are supported:
Truncate to day: DD, DDD,J
Truncate to month: MM,MON,MONTH,RM
Truncate to Year: SYEAR,SYYYY,Y,YEAR,YY,YYY,YYYY
TRUNC(date) is same as TRUNC(date, "DD")
This patch incorporates a fix for:
MDEV-37414 SIGSEGV in Binary_string::c_ptr | Item_func_trunc::get_date
Fixing the problem that the code did not take into
account that args[1] can return SQL NULL.
- Removed duplicate words, like "the the" and "to to"
- Removed duplicate lines (one double sort line found in mysql.cc)
- Fixed some typos found while searching for duplicate words.
Command used to find duplicate words:
egrep -rI "\s([a-zA-Z]+)\s+\1\s" | grep -v param
Thanks to Artjoms Rimdjonoks for the command and pointing out the
spelling errors.
STR_TO_DATE function in MariaDB does date to string conversion for invalid
dates strings, e.g.: 1949-02-30, and 1949-06-31, but null is return by
other invalid dates such as 1949-01-32 because the checking function
returns null for any date whose day is larger than 31, which is too naive
and needs to be changed.
This commit changed the rule of check_date in extract_date_time function,
which is executed after extracting date and time and check if the result
is valid. Now the conversion will consider the number of days in a month,
including skip years.
All new code of the whole pull request, including one or several files that
are either new files or modified ones, are contributed under the BSD-new
license. I am contributing on behalf of my employer Amazon Web Services, Inc.
This patch adds support for SYS_REFCURSOR (a weakly typed cursor)
for both sql_mode=ORACLE and sql_mode=DEFAULT.
Works as a regular stored routine variable, parameter and return value:
- can be passed as an IN parameter to stored functions and procedures
- can be passed as an INOUT and OUT parameter to stored procedures
- can be returned from a stored function
Note, strongly typed REF CURSOR will be added separately.
Note, to maintain dependencies easier, some parts of sql_class.h
and item.h were moved to new header files:
- select_results.h:
class select_result_sink
class select_result
class select_result_interceptor
- sp_cursor.h:
class sp_cursor_statistics
class sp_cursor
- sp_rcontext_handler.h
class Sp_rcontext_handler and its descendants
The implementation consists of the following parts:
- A new class sp_cursor_array deriving from Dynamic_array
- A new class Statement_rcontext which contains data shared
between sub-statements of a compound statement.
It has a member m_statement_cursors of the sp_cursor_array data type,
as well as open cursor counter. THD inherits from Statement_rcontext.
- A new data type handler Type_handler_sys_refcursor in plugins/type_cursor/
It is designed to store uint16 references -
positions of the cursor in THD::m_statement_cursors.
- Type_handler_sys_refcursor suppresses some derived numeric features.
When a SYS_REFCURSOR variable is used as an integer an error is raised.
- A new abstract class sp_instr_fetch_cursor. It's needed to share
the common code between "OPEN cur" (for static cursors) and
"OPER cur FOR stmt" (for SYS_REFCURSORs).
- New sp_instr classes:
* sp_instr_copen_by_ref - OPEN sys_ref_curor FOR stmt;
* sp_instr_cfetch_by_ref - FETCH sys_ref_cursor INTO targets;
* sp_instr_cclose_by_ref - CLOSE sys_ref_cursor;
* sp_instr_destruct_variable - to destruct SYS_REFCURSOR variables when
the execution goes out of the BEGIN..END block
where SYS_REFCURSOR variables are declared.
- New methods in LEX:
* sp_open_cursor_for_stmt - handles "OPEN sys_ref_cursor FOR stmt".
* sp_add_instr_fetch_cursor - "FETCH cur INTO targets" for both
static cursors and SYS_REFCURSORs.
* sp_close - handles "CLOSE cur" both for static cursors and SYS_REFCURSORs.
- Changes in cursor functions to handle both static cursors and SYS_REFCURSORs:
* Item_func_cursor_isopen
* Item_func_cursor_found
* Item_func_cursor_notfound
* Item_func_cursor_rowcount
- A new system variable @@max_open_cursors - to limit the number
of cursors (static and SYS_REFCURSORs) opened at the same time.
Its allowed range is [0-65536], with 50 by default.
- A new virtual method Type_handler::can_return_bool() telling
if calling item->val_bool() is allowed for Items of this data type,
or if otherwise the "Illegal parameter for operation" error should be raised
at fix_fields() time.
- New methods in Sp_rcontext_handler:
* get_cursor()
* get_cursor_by_ref()
- A new class Sp_rcontext_handler_statement to handle top level statement
wide cursors which are shared by all substatements.
- A new virtual method expr_event_handler() in classes Item and Field.
It's needed to close (and make available for a new OPEN)
unused THD::m_statement_cursors elements which do not have any references
any more. It can happen in various moments in time, e.g.
* after evaluation parameters of an SQL routine
* after assigning a cursor expression into a SYS_REFCURSOR variable
* when leaving a BEGIN..END block with SYS_REFCURSOR variables
* after setting OUT/INOUT routine actual parameters from formal
parameters.
Adding support for the "FM" format in function TO_CHAR(date_time, fmt).
"FM" in the format string disables padding of all components following it.
So now TO_CHAR() works as follows:
- By default string format components DAY (weekday name) and
MONTH (month name) are right-padded with spaces to the maximum
possible DAY and MONTH name lengths respectively,
according to the current locale specified in @@lc_time_names.
So for example, with lc_time_names='en_US' all month names are
right-padded with spaces up to 9 characters ('September' is the longest).
SET lc_time_names='en_US';
SELECT TO_CHAR('0001-02-03', 'MONTH'); -> 'February ' (padded to 9 chars)
NEW: When typed after FM, DAY and MONTH names are not right-padded
with trailing spaces any more:
SET lc_time_names='en_US';
SELECT TO_CHAR('0001-02-03', 'FMMONTH'); -> 'February' (not padded)
- By default numeric components YYYY, YYY, YY, Y, DD, H12, H24, MI, SS
are left-padded with leading digits '0' up to the maximum possible
number of digits in the component (e.g. 4 for YYYY):
SELECT TO_CHAR('0001-02-03', 'YYYY'); -> '0001' (padded to 4 chars)
NEW: When typed after FM, these numeric components are not left-padded
with leading zeros any more:
SELECT TO_CHAR('0001-02-03', 'FMYYYY'); -> '1' (not padded)
- If FM is specified multiple times in a format string,
every FM negates the previous padding state:
* an odd FM disables padding
* an even FM enables padding
Implementation details:
- Adding a helper class Date_time_format_oracle.
- Adding a helper method Date_time_format_oracle::append_lex_cstring()
- Moving the function append_val() to Date_time_format_oracle as a method.
- Moving the function make_date_time_oracle() to Date_time_format_oracle
as a method format().
- Adding helper methods month_name() and day_name() in class MY_LOCALE,
to return the corresponding components as LEX_CSTRINGs.
Item_{date|datetime}_typecase::get_date() erroneously passed the
TIME_INTERVAL_DAY flag from the caller to args[0] which made
CAST('100000:00:00' AS DATETIME) parse '100000:00:00' as TIME
rather that DATETIME.
Suppressing this flag.
Changing the return type of the following functions:
- CURRENT_TIMESTAMP, CURRENT_TIMESTAMP(), NOW()
- SYSDATE()
- FROM_UNIXTIME()
from DATETIME to TIMESTAMP.
Note, the old function NOW() returning DATETIME is still available
as LOCALTIMESTAMP or LOCALTIMESTAMP(), e.g.:
SELECT
LOCALTIMESTAMP, -- DATETIME
CURRENT_TIMESTAMP; -- TIMESTAMP
The change in the functions return data type fixes some problems
that occurred near a DST change:
- Problem #1
INSERT INTO t1 (timestamp_field) VALUES (CURRENT_TIMESTAMP);
INSERT INTO t1 (timestamp_field) VALUES (COALESCE(CURRENT_TIMESTAMP));
could result into two different values inserted.
- Problem #2
INSERT INTO t1 (timestamp_field) VALUES (FROM_UNIXTIME(1288477526));
INSERT INTO t1 (timestamp_field) VALUES (FROM_UNIXTIME(1288477526+3600));
could result into two equal TIMESTAMP values near a DST change.
Additional changes:
- FROM_UNIXTIME(0) now returns SQL NULL instead of '1970-01-01 00:00:00'
(assuming time_zone='+00:00')
- UNIX_TIMESTAMP('1970-01-01 00:00:00') now returns SQL NULL instead of 0
(assuming time_zone='+00:00'
These additional changes are needed for consistency with TIMESTAMP fields,
which cannot store '1970-01-01 00:00:00 +00:00'
