This merges the patch series that fixes a lot of missing validation
checks for read_log_event(). The merge includes a few added fixes that
are needed in new code introduced in 13.1.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
This merges the patch series that fixes a lot of missing validation
checks for read_log_event(). The merge includes a few added fixes that
are needed in new code introduced in 12.3.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
This merges the patch series that fixes a lot of missing validation
checks for read_log_event(). The merge includes a few added fixes that
are needed in new code introduced in 11.8.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
This merges the patch series that fixes a lot of missing validation
checks for read_log_event(). The merge includes a few added fixes that
are needed in new code introduced in 11.4.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
Protect accesses into the m_optional_metadata buffer to not access outside
of the buffer in case of corrupt/malicious event data.
Protect against buffer overflow of the m_column_name array in case of
excessive column names in the event data.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
This patch goes through all the constructors of Log_event subclasses that
are used to implement read_log_event() and adds error checks on the size of
the event buffer to guard against out-of-buffer reads.
Adds a lot of checks, and fixes some existing checks that were wrong or in
somewhat violation of C++ standard (eg. computing a pointer outside of the
available buffer).
This removes support for reading Table_map_log_event without metadata; such
events were only written by MySQL 5.1 servers going back to 2007.
Fix some missing bits in MAX_LOG_EVENT_HEADER, and update test case
rpl.rpl_packet with a longer query to still exceed max_allowed_packet.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
Introduce struct LEX_Parser_state and make LEX inherit from it. It holds the
members that are used only while a statement is being parsed: the select and
WITH-clause stacks, FK/key/option building state, window-spec scratch fields,
parsing_options, lex_options and a few parser flags.
LEX::start() now resets them through LEX_Parser_state::reset_parser_state(),
which initializes every member (several were never reset before).
parse_sql() marks the structure with TRASH_FREE() once parsing is done, so
that any code that still reads parser-only state after parsing is caught.
reset_parser_state() makes the memory addressable again for the next
statement.
Members that look parse-only but are used after parsing (with_clauses_list,
context_stack, select_stack_top, has_returning_list, clause_winfuncs) stay in
LEX.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The member was never set or read. The create_view_select in
LEX::parsed_create_view() is a local variable that merely shadowed it.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The flag was set by the parser but never read. Item_func_like keeps its own
escape_used_in_parsing member, which is unaffected.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The only use of this member was removed back in 2013 (!), but some (now
dead) code remains that accesses the member. This patch removes the member
and the related dead code properly.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
Follow-up patch, remove a lot of Format_description_log_event * function
arguments that are no longer needed.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
The FORMAT_DESCRIPTION_EVENT contains values common_header_len and
post_header_len array for individual event types, which were intended to
describe the format of following events in the binlog/relaylog. However,
this flexibility has been problematic and never useful:
- The values have never changed in MariaDB since before 10.0, nor in MySQL.
- Having to always have the correct format description event available in
the code when reading an event causes a lot of complexity (and has
historically caused numerous bugs).
- The potential for a corrupt or malicious format description event
requires careful validation that these length values are correct, which
again has historically seen bugs.
Since the values never change anyway, this patch removed the code that reads
them when reading the format description event, just using the hard-coded
values that are the only valid numbers anyway.
Remove support for ancient row events with post_header_len=6.
Remove support for reading binlogs created by custom MySQL forks with
different common_header_len (ie. MDEV-4645).
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
cover_definer_clause() wrapped DEFINER in "*/ /*!50017 ... */" and broke
when the statement text contained a comment. Dump the server's DDL
as returned.
Also:
- drop /*!NNNNN */ gates below 50700 in mysqldump, mysqlbinlog,
mysqladmin, mysqlcheck, mysqlshow, mysqlimport, mysqlhotcopy,
innotop and mtr_warnings.sql; mysqlbinlog's skip_replication
becomes /*M! ...*/
- SHOW CREATE DATABASE no longer wraps IF NOT EXISTS and DEFAULT
CHARACTER SET in comments
- remove checks for servers older than 5.7 in mysqldump, mysqlcheck,
mysql, innotop and the client library, including dump_trigger_old()
and mysqlcheck's SHOW TABLES fallback
- mysqldump warns on stderr if SHOW CREATE DATABASE fails
Behavior changes:
- --compatible=mysql323 and mysql40 are accepted and ignored. The other
modes wrap only some SET and ALTER TABLE ... KEYS statements in
/*!40101 */.
- mysqlhotcopy --old_server is accepted, ignored, and prints a warning.
Assisted-By: Claude:claude-5.5-sonnet
INTERVAL is evaluated at parse time, when placeholders and stored
procedure variables have no value yet.
We reuse the `check_partition_func_processor` to check for this. It
also bans user and system variables as well as functions lik RAND(),
but nonetheless slightly stricter than strictly needed - for example,
`CONCAT('1','.','30') HOUR_MINUTE` is no longer allowed.
A locking read with SKIP LOCKED through a secondary index calls
row_sel_get_clust_rec_for_mysql() to lock the clustered index record.
On a lock conflict, that function returned DB_LOCK_WAIT without
setting the output record, and row_search_mvcc() then called
lock_trx_handle_wait() to cancel the wait. If the conflicting
transaction had committed in the meantime, the waiting lock was
already granted and lock_trx_handle_wait() returned DB_SUCCESS.
row_search_mvcc() then found clust_rec == NULL for a locking read:
debug builds failed an assertion, and release builds skipped a row
whose lock the transaction now held.
Row_sel_get_clust_rec_for_mysql::operator()(): Call
lock_trx_handle_wait() here instead of in row_search_mvcc(). When
the lock was granted meanwhile, return the clustered index record
and store the cursor position as for any granted lock, like
row_search_mvcc() does with "goto locks_ok" for the record that it
is positioned on.
`@@rpl_semi_sync_slave_enabled` can be changed without stopping
replication, because it takes effect when a connection is established.
But when it takes effect, it applies to *all* replication connections.
This means that starting an async connection will switch any ongoing
semi-sync connections to async as well, and vice versa, which will
cause those connections to fail to replicate in the wrong mode.
This fix resolves this oversight by changing the value application
to be specific to only the establishing connection (technically,
moving the singleton’s field `Repl_semi_sync_slave::m_slave_enabled`
to the instance variable `Master_info::semi_sync_enabled`).
This design builds toward MDEV-40941, which proposes migrating
`@@rpl_semi_sync_slave_enabled` to a proper per-connection config.
Note, `Master_info::semi_sync_reply_enabled` must remain separate,
because it only disables ACK replying in async slave fallback,
whereas the semi-sync slave setting also controls whether the IO
thread should expect additional semi-sync flags in event packets.
To maintain backward compatibility until MDEV-40941,
the status variable `Rpl_semi_sync_slave_status` is kept showing
the applied `@@rpl_semi_sync_slave_enabled` for the most recent
START SLAVE in any connection; that is, not yet made per-connection.
Reviewed-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
DuckDB has no row locks and its parser rejects locking clauses, so
SELECT ... FOR UPDATE on a DuckDB table failed with ER_GET_ERRMSG
("Parser Error: SELECT locking clause is not supported!") coming from
the embedded DuckDB.
Check for locking reads in ha_duckdb::external_lock() and raise
ER_NOT_SUPPORTED_YET from the storage engine instead. The check only
applies to SQLCOM_SELECT, so INSERT ... SELECT, UPDATE and DELETE are
not affected.
A WHERE conjunct that calls rand() but doesn't refer to a column, such
as rand() < 0, is checked on the last non-const top level base table
in the join order. A join of only const tables has no such table and
returns at most one row, so the conjunct is checked once before the
join starts, together with the conditions on columns of an outer
query.
When every top level base table is const and materialized semi-joins
follow them, the join order also has no non-const top level base
table, but only_const_tables() is false. The conjunct was never
checked, and such a query gave wrong results.
The check before the join now applies whenever the join order has no
non-const top level base table, which is the same condition under
which no table in the join order receives the conjunct. Such a join
also returns at most one row.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A join of only const tables stored the WHERE conjuncts that refer to a
column of an outer query or call rand() in pseudo_bits_cond. The
executor read pseudo_bits_cond only when the query needed no temporary
table, and read outer_ref_cond otherwise. A window function or
DISTINCT with ROLLUP needs a temporary table, so those conjuncts were
never checked, and the query returned rows that the WHERE clause
rejects.
pseudo_bits_cond came from the fix for LP bug 908269 in 2012. For a
join of only const tables, that fix also collected conjuncts that call
rand(), and it stored them in a new field without stating a reason.
The code that filled the two fields and the code that read them chose
between them by the same test. The tests have differed since
MDEV-8646 in 2016, which added the temporary table condition to the
executor's test. Later changes only display the field, in EXPLAIN
FORMAT=JSON (MDEV-19714) and in the optimizer trace (MDEV-21095). In
MDEV-19714, its author wrote that it was not clear why pseudo_bits_cond
was needed in addition to outer_ref_cond.
pseudo_bits_cond is removed, and outer_ref_cond holds these conjuncts
for every join. The executor checks it on both paths, so where the
conjuncts are stored no longer depends on the executor's test.
Conjuncts that call rand() are still collected only for a join of only
const tables. EXPLAIN FORMAT=JSON shows these conjuncts as
outer_ref_condition instead of pseudo_bits_condition.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
When a versioned table stores its history in partitions, an implicit
query (no FOR SYSTEM_TIME) needs only current rows, which all live in
the current partition. vers_setup_conds() no longer adds a row_end
condition for such a table, and prune_partitions() restricts
read_partitions to the current partition instead.
read_partitions is reset to all partitions every statement in
open_table(), so the restriction must be applied on each execution.
prune_partitions() runs on each execution, also for merged derived
tables and views, where vers_setup_conds() is not called again. Both
functions guard with !vers_conditions.was_set(). In prune_partitions()
!is_set() would not do: set_all() sets type (flips is_set()) but not
orig_type (was_set()), so is_set() would be true on prepared-statement /
stored-procedure re-executions and skip the restriction, leaving history
partitions readable - rows matched then grow on each execute/call.
A query whose time comes from @@system_versioning_asof must still read
history, so the restriction is also skipped when from_sysvar is set.
init_from_sysvar() sets this new flag rather than orig_type, so
was_set(), print() and the write lock check still see only an explicit
FOR SYSTEM_TIME. Like orig_type, the flag persists to prepared-statement
re-execution.
EXPLAIN no longer shows "Using where" for such a query: correctness now
comes from partition selection rather than a row_end filter.
With sql_safe_updates, DELETE without WHERE on such a table is now
rejected even with LIMIT, as for a non-versioned table. With an index
on row_end, UPDATE or DELETE without WHERE (UPDATE also without LIMIT)
is now rejected too: it was accepted as a range scan on row_end. A
versioned table without SYSTEM_TIME partitions keeps the old behavior
(MDEV-41393).
read_partitions is reset to all partitions (or the PARTITION() set) once
per statement in open_table() (set_partition_bitmaps()), and
prune_partitions() only intersects it with lock_partitions and the pruned
set instead of rebuilding it.
mark_all_partitions_as_used() is dropped.
Pure refactoring, results unchanged. It lets a restriction set on
read_partitions before pruning survive it - needed when read_partitions
is pre-restricted, e.g. a versioned table whose history is partitioned
reads only the current partition.
prune_partitions() built the used-partition set directly in
read_partitions: it cleared it, let find_used_partitions*() fill it, and
in the index-merge case used it as scratch too - both buffer and result.
The patch adds PART_PRUNE_PARAM::parts_bitmap and builds the return
into it instead. read_partitions is produced from it only at the end:
copied when the condition was analyzed (IMPOSSIBLE included,
parts_bitmap left empty), or set to all locked partitions via
mark_all_partitions_as_used() when nothing could be pruned.
Pure refactoring: read_partitions ends up identical on every path. It
decouples the pruning scratch from read_partitions, so a later commit can
switch the final bitmap_copy() to bitmap_intersect() - letting pruning
narrow a pre-set read_partitions (e.g. a versioned table selecting only
its current partition) instead of overwriting it.
The test checks that sys.innodb_buffer_stats_by_table reports old
pages (pages_old) after a server restart. InnoDB creates the old
sublist of the buffer pool LRU list only when the list holds at least
BUF_LRU_OLD_MIN_LEN (512) pages. After the restart, most of the pages
in the list come from the buffer pool load at startup. This load runs
asynchronously, and the server accepts connections before it
completes. If the test runs the check first, the list can be shorter
than 512 pages, no page is old, and the result shows 0 instead of 1.
The option --innodb-buffer-pool-load-at-startup=0 disables the load.
This is the extreme case of the race: the check always runs before
any page of the dump is loaded. With this option, the test fails every
time with innodb_page_size=4k and 16k.
Wait for the buffer pool load to complete before the check, as other
tests do.
Without passing through the compile and linker flags, options
like -stdlib=libc++ can create ABI differences between ducksdb
extensions and what the server links too.
The jemalloc has C files so CMAKE_C_FLAGS is passed through too.
The warnings were in RocksDB (not MyRocks) and looked like this:
rocksdb/db/db_impl/db_impl.cc: In member function ‘virtual rocksdb::Status
rocksdb::DBImpl::FlushWAL(bool)’:
rocksdb/db/db_impl/db_impl.cc:1329:23: warning: redundant move in return
statement [-Wredundant-move]
1329 | return std::move(io_s);
| ~~~~~~~~~^~~~~~
GetJVM() only looked for %JAVA_HOME%\bin\client\jvm.dll, and for the
RuntimeLib registry value of the "Java Runtime Environment" key. Since
JDK 9 there is no "client" JVM, jvm.dll is in bin\server, and the JDK
installers write JavaHome, but not RuntimeLib, under JavaSoft\JDK.
Look for jvm.dll the way Java installations are laid out. Prefer
bin\server over bin\client, in the JDK and in its jre subdirectory.
Without connect_jvm_path (which stays exclusive, as before), try
JAVA_HOME first, then the JavaSoft\{JDK|JRE} registry keys:
CurrentVersion, then the RuntimeLib or JavaHome value of that version.
Keep the first meaningful LoadLibrary or registry error (e.g. a 32/64
bit mismatch) rather than the "not found" of the last probe, reject
over-long paths instead of truncating them, accept REG_EXPAND_SZ
values, ignore an empty JAVA_HOME, and remove a stray printf to stdout.
Tested manually with Oracle JDK 23: with JAVA_HOME unset (registry),
with JAVA_HOME set, and with an invalid JAVA_HOME falling back to the
registry.
Assisted-By: Claude:claude-5.5-sonnet
SET GLOBAL event_scheduler=1 executed by a connection during shutdown
could start the scheduler thread after close_connections() called
Events::stop(). Nothing stops this thread and close_connections()
waits for it forever.
Now it hangs instead of a crash since MDEV-30260 fix: it moved
deinit() from close_connections() to clean_up(), so the scheduler is
not freed under the running thread anymore.
The fix makes Event_scheduler::start() do nothing when abort_loop is
set. abort_loop is set before close_connections() and is checked under
LOCK_scheduler_state, which stop() also takes, so the scheduler is
either started before stop() and stopped by it, or not started at all.
No error needed: the server is shutting down.
my_strntoull10rnd_8bit() did not set *error when a signed negative value
was exactly LONGLONG_MIN (-9223372036854775808): that path returned the
value and left *error untouched. Callers passing an uninitialized
"int error", such as the associative array key conversion in
Type_handler_assoc_array, then read a garbage value and raised a
spurious ER_WRONG_VALUE. Field_num::get_int() and Field_year::store()
also pass an uninitialized error, and my_strntoull10rnd_mb2_or_mb4()
calls this function too.
Fix: initialize *error to 0 at function entry, like the sibling
my_strntoll_8bit() and my_strntoll_mb2_or_mb4() do, so that no exit
path can leave it unset.
The bug was introduced by commit 031f11717d (Fix all warnings given by
UBSAN, 2021-04-18), which started skipping the *error assignment when
the value equals LONGLONG_MIN.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This SQL script failed:
SET NAMES latin1 COLLATE latin1_swedish_ci;
CREATE OR REPLACE VIEW v1 AS SELECT 'Y' AS c1;
SET NAMES big5 COLLATE big5_chinese_ci;
SELECT * FROM v1 WHERE c1='y';
with the following error:
ERROR 1267 (HY000): Illegal mix of collations
(latin1_swedish_ci,COERCIBLE) and (big5_chinese_ci,COERCIBLE)
for operation '='
Note, latin1_swedish_ci and big5_chinese_ci are used here as examples.
The error also happened with different collation combinations.
Fix main idea:
If two collations have equal comparison rules (known as "tailoring")
on a given character repertoire,
like latin1_swedish_ci and big5_chinese_ci on ASCII letters,
then the "Illegal mix of collations" error can be avoided in
a comparison operator. We can choose any of the sides as the operation
effective collation - the result will be equal.
This optimization is not applied when at least one side has an explicit
COLLATE clause. Two explicit COLLATE clauses in one comparison are
already illegal when the character sets are the same, so for
consistency this stays illegal when the character sets differ too.
Most important details:
- Splitting enum_repertoire_t into smaller subsets,
for better repertoire granularity.
A variable holding a repertoire value can now have multiple
MY_REPERTOIRE_XXX flags set.
This patch implements detecting tailoring equality
on these repertoires:
* MY_REPERTOIRE_ASCII_DIGITS - [0..9]
* MY_REPERTOIRE_ASCII_MINUS_DOT_DIGITS - [-.0..9], e.g. '-10.5'.
It consists of the new bits MY_REPERTOIRE_ASCII_MINUS and
MY_REPERTOIRE_ASCII_DOT. They were a part of
MY_REPERTOIRE_ASCII_NOT_IDENT before. As a result, the numeric values
of MY_REPERTOIRE_ASCII and MY_REPERTOIRE_UNICODE30 changed
(223 and 255).
* MY_REPERTOIRE_ASCII_ALNUM - [A..Z,a..z,0..9].
* MY_REPERTOIRE_ASCII_IDENT - ALNUM + underscore
* MY_REPERTOIRE_ASCII - the entire range U+0000..U+007F
- Adding a new virtual function "tailoring" in my_collation_handler_st.
It returns the tailoring on the given repertoire for the given
collation.
If cs1->coll->tailoring(cs1, some_repertoire) returns {0,0},
it means the illegal mix optimization cannot be used
for this collation on the given repertoire.
If these calls:
tr1= cs1->coll->tailoring(cs1, some_repertoire);
tr2= cs2->coll->tailoring(cs2, some_repertoire);
return both non-NULL results and tr1.str==tr2.str,
then these collations are equal on the given repertoire
and are mutually replaceable for a comparison operator,
so "Illegal mix of collations" can be avoided.
- Tailoring strings are shared constants my_tailoring_str_*
(strings/strings_def.h). Tailorings are compared by the string
pointer, so tailoring() implementations return strings only from
these constants. The strings do not depend on PAD/NOPAD.
- my_string_repertoire() for ucs2, utf16, utf32 and NONASCII 8bit
character sets no longer stops at the first bad byte sequence with
the repertoire found so far, which could be MY_REPERTOIRE_NONE
(compatible with everything). It returns MY_REPERTOIRE_EXTENDED
for such strings. A unit test was added into unittest/strings.
- Adding CHARSET_INFO::is_ascii_superset(). It tells if a character set
can store all ASCII characters (it is false for swe7).
It is used in Item_func_conv_charset to calculate "safe", and in
left_is_algorithmically_simpler() to prefer other character sets
to those where ASCII strings cannot be converted safely.
- DBUG_ASSERTs were added into mysys/charset.c, to check that
the "tailoring" method is set in collation handlers when a collation
is added: compiled-in, from ctype-extra.c, or loaded from Index.xml.
- Adding a new function my_collations_equal_on_repertoire(cs1, cs2,
repertoire) in strings/. It tells if two collations are equal on
the repertoire, taking tailoring() results and PAD/NOPAD and the UCA
version into account. Callers do not compare tailoring() results
directly, so the way tailorings are compared (currently by the pointer
to the shared constant string) is private to strings/ and can be
extended later, e.g. for UCA collations on MY_REPERTOIRE_UNICODE30.
- Adding a new method DTCollation::aggregate_by_tailoring().
Collations with different MY_CS_NOPAD are never compatible.
UCA collations of different versions are not compatible on
the repertoire with ASCII punctuation, as the UCA version
affects its order (e.g. UCA-6.2.0 moved GRAVE ACCENT
and CIRCUMFLEX ACCENT after PERCENT SIGN).
- DTCollation::aggregate() now makes the repertoire of the result cover
the repertoires of both sides in all successful branches.
Before, set(dt) replaced the repertoire with the repertoire of the
winner side only, so the result could have a too narrow repertoire,
e.g. IF(1, 1.5, HEX(255)) had MY_REPERTOIRE_ASCII_ALNUM although the
value '1.5' has a punctuation character. With the more granular
repertoires this could wrongly allow mixing collations which are
equal on ALNUM but different on punctuation.
- Adding a new flag MY_COLL_ALLOW_BY_TAILORING.
It indicates to DTCollation::aggregate() that the illegal
mix optimization by repertoire can be used in the given context.
MY_COLL_CMP_CONV now includes MY_COLL_ALLOW_BY_TAILORING.
Note, only comparison operators pass this flag.
Functions returning a string result do not pass this flag,
because in operations like CONCAT(a,b) we still need to evaluate
precisely the collation of the result - we cannot just choose
a collation of one of the sides (even if they are compatible
on the given repertoire). This also applies to ExtractValue()
and UpdateXML(), which now aggregate their arguments without
this flag.
- As in my_repertoire_t the value MY_REPERTOIRE_ASCII is now a set of
bits rather than a single bit, the way to detect "is only ASCII"
repertoires has changed in the code.
For example:
// repertoire *IS* ascii
if (repertoire == MY_REPERTOIRE_ASCII)
has changed in multiple places in the code to
// repertoire *HAS* only ascii characters
if (my_repertoire_is_subset_of(repertoire, MY_REPERTOIRE_ASCII))
The new function my_repertoire_is_subset_of() in m_ctype.h
is used for this purpose in C code. In C++ code
DTCollation::repertoire_is_subset_of() is used, e.g.:
if (collation.repertoire_is_subset_of(MY_REPERTOIRE_ASCII))
- Repertoire of some expressions was adjusted to the new meaning:
* Item_null now has MY_REPERTOIRE_NONE (was ASCII).
* MY_LOCALE::repertoire() now returns MY_REPERTOIRE_UNICODE30
(was EXTENDED).
* Lex_string_with_metadata_st::repertoire(cs) now scans the string
contents to detect the actual repertoire.
* HEX() now has MY_REPERTOIRE_ASCII_ALNUM.
* DATE_FORMAT() decides on MY_REPERTOIRE_EXTENDED using the locale
which is actually used: the explicit third argument, or
@@lc_time_names. It was always @@lc_time_names before.
A non-constant locale argument is assumed to be non-ASCII.
* QUOTE, MAKE_SET, EXPORT_SET, LPAD, RPAD, GROUP_CONCAT, JSON_ARRAY,
JSON_OBJECT and JSON_OBJECTAGG add the repertoire of the extra
characters they put into the result (quotes, separators, padding,
brackets).
* LOWER() and UPPER() add the letters of both cases to the repertoire,
because the result can contain letters of the opposite case.
In Turkish collations they also add MY_REPERTOIRE_EXTENDED, as an
ASCII letter can be converted to a non-ASCII one (I -> dotless i).
Unicode collations are detected by the new member
casefold_info_st::can_convert_to_non_ascii_on_casefolding,
simple 8bit collations by to_lower['I'] and to_upper['i'].
- Lex_string_with_metadata_st::repertoire(cs) now scans a string literal
for any byte oriented character set (mbminlen==1), not only for
ASCII based ones. It was assumed to have the full repertoire for swe7,
which is not ASCII based. The scan is done by Unicode code points, so
swe7 strings with digits and letters get a narrow repertoire, while
e.g. 0x5B (LATIN CAPITAL LETTER A WITH DIAERESIS in swe7) is non-ASCII.
Therefore swe7_swedish_ci can be mixed with other collations now,
in the same way as other collations with MY_CS_IDENT_CASEUP_CI.
The test func_time was adjusted: a swe7 literal ' ull' with ASCII
characters is now safe to convert, the error is still tested for '[ull'.
- cp1250_czech_cs sorts -, . and 0..9 in the code point order, so it gets
the MINUS_DOT_DIGITS tailoring. latin2_czech_cs gets only DIGITS,
as it sorts the minus and the dot differently. Both were verified
with ctype_digits_verify.
- The tis620 collations now set CHARSET_INFO::tab_to_uni (was NULL),
so my_charset_is_ascii_based() is true for tis620.
It was false for tis620, although tis620 is ASCII compatible.
Therefore tis620 string literals were always considered to have
a non-ASCII repertoire, and the repertoire optimizations did not work
for them. Now ASCII-only tis620 literals are detected as ASCII.
This changes the result of subselect_extra_no_semijoin
from an error to success.
- tis620_thai_ci and tis620_thai_nopad_ci fold letters to lower case
and then compare by the code point on the entire ASCII range
(the Thai specific rules affect only non-ASCII characters).
A new function my_tailoring_ascii_casedn_ci() returns tailorings
for such collations on ALNUM, IDENT and ASCII. For example, the
underscore sorts before the letters, unlike in the caseup tailorings.
- New flags were added for CHARSET_INFO::state
* MY_CS_ASCII_CASEUP_CI - for simple 8bit case insensitive collations.
It means that this collation has no irregularities on the ASCII
repertoire, and converts lower case letters to upper case ones,
so the underscore sorts after the letters (with an upper to lower
case conversion it would sort before the letters).
* MY_CS_IDENT_CASEUP_CI - for simple 8bit case insensitive collations.
It means that this collation has no irregularities on the IDENT
repertoire (but can have irregularities say on punctuation).
* MY_CS_ASCII_STD_UCA - for UCA collations.
It means that a UCA collation does not reorder ASCII characters.
* MY_CS_DIGITS_STD - for any collation. It means that the ASCII digits
0..9 are sorted in the code point order and are not tailored, and,
for PAD collations, the space sorts before the digits. Strings of
digits are compared in the same way in all such collations, even if
they have irregularities on letters (latin7, cp866, czech, danish,
turkish UCA, etc.), or differ by case sensitivity. It is detected
from sort_order for 8bit collations, and from the tailoring rules
for UCA collations. Collations with MY_CS_DIGITS_STD still must have
the same PAD/NOPAD attribute.
* MY_CS_ASCII_MINUS_DOT_DIGITS - for any collation. Means
MY_CS_DIGITS_STD, and additionally the minus sorts before the dot,
which sorts before the digits (and the space sorts before the minus
in PAD collations). Strings like '-10.5' are compared in the same
way in all such collations. It is detected from sort_order for simple
8bit collations (my_coll_init_simple()), and from the tailoring rules
for UCA collations. For the multi-byte collations big5_chinese_ci,
gbk_chinese_ci, gb2312_chinese_ci and sjis_japanese_ci it is declared
by my_tailoring_ident_caseup_ci_generic(). The flags MY_CS_DIGITS_STD
and MY_CS_ASCII_MINUS_DOT_DIGITS are stored in ctype-extra.c for
compiled collations (conf_to_src.c prints them), and the detection is
skipped for a collation which already has any of the repertoire flags
(MY_CS_REPERTOIRE_FLAGS).
- strings/conf_to_src.c was modified to detect and print
MY_CS_ASCII_CASEUP_CI and MY_CS_IDENT_CASEUP_CI flags.
- strings/ctype-extra.c was regenerated with new flags.
- 8bit collations are not given MY_CS_ASCII_CASEUP_CI and
MY_CS_IDENT_CASEUP_CI if they are PAD and the space does not sort
before the digits. A shorter string is padded with spaces for
comparison, so the position of the space matters even for strings
without spaces ("ab" is compared to "abc" as "ab " to "abc").
A test collation latin1_space_after_letters_ci was added to
mysql-test/std_data/ldml/ for this. ctype-extra.c does not change.
- The initializer of my_collation_cs_handler in ctype-utf8.c
(utf8mb3_general_cs, under HAVE_UTF8_GENERAL_CS) was fixed to match
my_collation_handler_st: the missing get_id and get_collation_name
members were added, so that eq_collation was not initialized in their
place, and the new member "tailoring" is now set to my_tailoring_none.
- Results of the existing tests func_str, func_test, ps, view
and subselect_sj changed from "Illegal mix of collations" to success.
In these tests ASCII-only strings were compared using collations
with equal tailoring on ASCII, e.g. latin1_swedish_ci with
latin2_general_ci (func_str, ps), koi8r_general_ci with
latin1_swedish_ci (func_test), latin1_general_ci with
latin1_swedish_ci on letters and digits (view), cp932_japanese_ci
with latin1_swedish_ci (subselect_sj).
Such comparisons are now allowed.
Cases with collations that are still incompatible on ASCII,
e.g. latin7_general_ci, were added to the tests and still fail.
- Adding a number of MTR tests in
plugin/func_test/mysql-test/func_test/.
They display a tailoring by collation name and repertoire as
returned by:
cs->coll->tailoring(cs, some_repertoire)
A dynamically linked plugin function collation_tailoring() was added
for the purpose of these tests. Tailorings do not depend on PAD/NOPAD
and on the UCA version, so the function shows "[nopad]" for NOPAD
collations, and "[version X.Y.Z]" for UCA collations on the ASCII
repertoire. The version is formatted by a new class UCAVersion,
which uses a new method CharBuffer::append_uint8().
- Adding the test plugin/func_test/mysql-test/func_test/
ctype_caseup_ci_verify.test. It checks the order of ASCII characters
for all collations declared "caseup CI" on the ASCII or IDENT range.
- Adding the test plugin/func_test/mysql-test/func_test/
collation_tailoring_args.test. It checks the arguments of the plugin
function collation_tailoring(): the repertoire passed by name or by
number, and the NULL results for unknown names.
- Adding the test plugin/func_test/mysql-test/func_test/ctype_ldml.test.
It displays the tailorings of collations loaded at runtime from
Index.xml: simple 8bit collations (flags detected from the weights)
and Unicode collations (LDML rules which do or do not reorder ASCII).
- Adding a number of MTR tests mysql-test/main/ctype_xxx_tailoring.test
They display two-dimensional charts showing which collations
are compatible on which repertoires.
Tests for DATE_FORMAT() with an explicit locale were added to
ctype_big5_tailoring.test, ctype_cp932_tailoring.test and
ctype_latin5_tailoring.test, which also has the chart for latin5.
- Adding a number of MTR tests in the form of the originally
reported script for various collations:
SELECT Insert_priv FROM mysql.user WHERE Insert_priv='...';
The repeated blocks of queries are shared through
mysql-test/include/
ctype_tailoring_0{1,2,3}_mysql_user_Insert_priv*.inc.
- Adding tests to ctype_cp932.test checking that ExtractValue() and
UpdateXML() still raise "Illegal mix of collations".
Problem: JSON_OVERLAPS treats unequal nested arrays as equal when only
one parser reaches the array end. Requiring both ends also exposes a
cleanup issue: scanning ']' moves stack_p to the parent, so skipping
the current level consumes parent elements. This misses the common [2]
in [[1],[2]] and [[],[2]], and can skip unequal recursive elements.
Fix: Require both parsers to reach JST_ARRAY_END. Remember their array
levels before comparing and skip to those saved levels after completed
comparisons and mismatches, preserving parent siblings. Explain why the
saved levels are needed. Add regression coverage for both argument
orders, unequal lengths, empty arrays, recursive comparisons, type
mismatches, array order and a later common object member.
get_clone() returns NULL on out-of-memory, but two of its callers in
prep_alter_part_table() dereferenced the result without a check, so the
server crashed:
changed INTERVAL of SYSTEM_TIME partitioning
ADD PARTITION under LOCK TABLES
get_clone() itself did not check the allocation of Vers_part_info and
dereferenced it.
handle_alter_part_error() did not check the clone either. The clone only
kept the DDL log entries after the table is closed. In the branch where
DDL log recovery fails, write_log_completed() released the entries
through lpt->part_info, which the table close may have freed, and
release_log_entries() released the same entries again through the
clone. That branch is not reachable now, as ddl_log_execute_entry()
always returns false.
The fix checks the allocation of Vers_part_info in get_clone() and the
result of get_clone() in prep_alter_part_table() like the other callers
there do. handle_alter_part_error() keeps a copy of DDL_LOG_STATE
instead of the clone and passes it to write_log_completed() and
release_log_entries(), which now take DDL_LOG_STATE. The redundant
release_log_entries() after write_log_completed() is removed.
ALTER TABLE ... PARTITION BY SYSTEM_TIME with a changed INTERVAL or
STARTS on a table already partitioned by SYSTEM_TIME took the fast
ALTER path, which only rewrites the table definition. The history rows
stayed where the old layout put them, so after the ALTER they were in
wrong partitions. An unchanged interval took the copy path instead.
MDEV-29873 replaced memcmp() of the intervals, which is true when they
differ, with interval_t::operator==, which is true when they are equal,
and lost the negation.
The copy path was also broken: it assigned the whole vers_info of the
new partitioning spec to the cloned partition info, now_part and
hist_part included. The ALTER does not list partitions, so now_part
became NULL and check_partition_info() failed with ER_VERS_WRONG_PARTS
for a table with explicit partitions (MDEV-33370 test case).
The fix negates the comparison again (now with interval_t::operator!=),
so a changed interval rebuilds the table and redistributes the history.
It copies only interval, limit and auto_hist from the new spec and keeps
now_part and hist_part of the existing partitions.
Without STARTS the start of the interval defaults to the current time
rounded down to the INTERVAL unit. So ALTER with the same INTERVAL and
no STARTS usually gets a different start and now rebuilds the table.
Vers_part_info::operator=, which copied now_part and hist_part and had
no other callers, is deleted.
Fast ALTER was also chosen when the same statement changed the table
definition, e.g. ADD COLUMN with the same INTERVAL and STARTS, or DROP
COLUMN with LIMIT. It wrote the new .frm, but the partitions kept the
old record layout. Now any change of the table definition
(alter_info->flags) takes the copy path.
PURGE BINARY LOGS could end with a warning instead of deleting files, as
the test below shows when run on the base of this commit. The warning
names a wrong reason. It follows an automatic purge that was refused by
the slave guard, slave_connections_needed_for_purge: the automatic purge
does not delete a binlog while it might still be needed by a slave, not
necessarily a connected one, that is while fewer than the variable's
number of slaves are connected.
Since MDEV-34504 the manual PURGE is meant to ignore the guard.
Yet it still protects a binlog that a slave is reading.
Technically this was possible because the automatic and the manual purge
share a code path, can_purge_log(), which cached the refusal and replayed
it, under the reason of the first arm below, to any later purge:
if (is_active(name) ||
(!is_relay_log && waiting_for_slave_to_change_binlog &&
purge_sending_new_binlog_file == sending_new_binlog_file &&
!strcmp(name, purge_binlog_name)))
reason= "it is the current active binlog";
So "active" had to be read as "active or busy". The fix removes this
duality too: the two arms are now separate and have their own reasons.
The main issue is fixed by narrowing the busy arm with an additional
`!interactive &&` conjunct, so that a manual PURGE never consults the
cached refusal.
Running the test below on an unfixed 11.4 server (source tree at commit
42038e1145) fails with "Result length mismatch" and, in part 2, this
diff against the recorded result:
PURGE BINARY LOGS TO 'master-bin.000006';
+Warnings:
+Note 1375 Binary log 'master-bin.000003' is not purged because it is the current active binlog
SHOW WARNINGS;
Level Code Message
+Note 1375 Binary log 'master-bin.000003' is not purged because it is the current active binlog
show binary logs;
Log_name File_size
+master-bin.000003 #
+master-bin.000004 #
+master-bin.000005 #
master-bin.000006 #
master-bin.000007 #
master-bin.000008 #
Test binlog.binlog_purge_stale_refusal_cache: the test sets
slave_connections_needed_for_purge=1, binlog_expire_logs_seconds=0 and
max_binlog_total_size=0 first and restores them at the end. It starts
with RESET MASTER so that the binlog names, 000003 and so on, do not depend
on the tests run before it in the same mtr environment. Part 1 is a control with
nothing cached; part 2 makes an automatic purge refuse 000003 (the files
must be older than binlog_expire_logs_seconds, hence --real_sleep 2 before
the rotation) and then runs PURGE BINARY LOGS TO 'master-bin.000006',
which must remove 000003..000005.
As a workaround, a PURGE could work if the following settings are made
before it, in this order:
SET GLOBAL binlog_expire_logs_seconds=0, slave_connections_needed_for_purge=0;
The second setting invalidates the cached refusal. The first one keeps
the purge that this setting triggers from refusing and caching again.
Note that both settings change the server's policy, not only this one
PURGE: the time-based expiry and the slave guard are off until they are
restored.
A field of an element of an associative array of RECORDs, passed to
an OUT or INOUT parameter of a stored routine, crashed the server:
CREATE PROCEDURE p1(a OUT INT) AS BEGIN a:= 7; END;
DECLARE
TYPE rec_t IS RECORD (i INT, s VARCHAR(10));
TYPE tab_t IS TABLE OF rec_t INDEX BY INT;
t tab_t:= tab_t(1 => rec_t(0, 'x'));
BEGIN
p1(t(1).i);
END;
There were three problems:
- Item_splocal_assoc_array_element_field inherited set_value() of
Item_splocal_row_field_by_name, which treated the associative array
variable as a ROW variable (assertion failure in
sp_rcontext::virtual_tmp_table_for_row()). Added
Item_splocal_assoc_array_element_field::set_value(), which stores the
value into the field of the element.
- Before the call, sp_head::bind_input_param() destructed the whole
variable for an OUT argument, even if the argument was only a part of
the variable (a ROW field, an assoc array element or its field), so
the other elements of the array were lost: after p1(a0(2)), the array
a0(1), a0(2), a0(3) had only one element left. Added
Item_splocal::is_part_of_variable().
- sp_head::bind_output_param() evaluated the actual argument in the
context of the callee. For arguments like t(i) or t(i).field this
resolved the variables t and i in a wrong context. Now the argument
is evaluated in the context of the caller.
The second problem also affected a SYS_REFCURSOR field of a ROW variable
passed as an OUT argument: the cursors in the other fields of the ROW
variable were closed. Now they are kept.
Known limitations, not changed by this fix:
- The actual OUT argument is evaluated when the value is copied back,
not when the routine is called. If an earlier OUT argument of the same
call changes the variable used as the key, as in CALL p(i, a0(i)), the
new value of the key is used.
- For an OUT argument which is a part of a variable (a ROW field, an
assoc array element or its field), only the destruct of the whole
variable is skipped, the part itself is not destructed. A
SYS_REFCURSOR field of a ROW stays open during the call and is
released when the new value is copied back.
Tests: sp-assoc-array (OUT and INOUT elements and fields of elements,
NULL key, element deleted by the callee, ROW and RECORD fields) and
type_sys_refcursor-type_row-debug (SYS_REFCURSOR fields of a ROW
variable).
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
1) take into account TMP_TABLE_ALL_COLUMNS when
we are modifying agg_item->result_field
2) remove unused now "bool materialized_subquery;"
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The test sent
ALTER TABLE t1 ADD COLUMN f3 INTEGER; INSERT INTO t1 VALUES (NULL, 10000, 10000);
from node_2 as a single --send. Nothing ordered that INSERT against the
CREATE UNIQUE INDEX issued from node_1, so the second DDL could reach node_2
while the INSERT was still executing there and BF-abort it. The automatic
autocommit retry then replicated with last_seen_seqno behind the seqno of the
DDL and failed certification against its exclusive key on test.t1. As
wsrep_retry_autocommit defaults to 1 there was no second retry and the client
got ER_LOCK_DEADLOCK.
Send only the ALTER TABLE and execute the INSERT after both DDL statements
have completed on both nodes.
Waiting for the second DDL by polling INFORMATION_SCHEMA.PROCESSLIST did not
prove that it had been ordered either: wsrep_TOI_begin() sets the
"Waiting to execute in isolation" stage before it replicates the writeset, so
that state can be observed before the DDL has reached total order. Wait on
node_2 for the new index to appear instead, which happens only once the DDL
has been ordered behind the parked ALTER TABLE, and record on node_1 that
neither DDL has been applied there while the first one is still parked.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
