This merges the patch series that fixes a lot of missing validation
checks for read_log_event(). The merge includes a few added fixes that
are needed in new code introduced in 13.1.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
This merges the patch series that fixes a lot of missing validation
checks for read_log_event(). The merge includes a few added fixes that
are needed in new code introduced in 12.3.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
A multi-byte PERIOD name at the 64-character limit (192 bytes) filled
the name buffer exactly, leaving no room for the '_N' suffix appended
when generating a unique name for the implicit CHECK constraint.
Truncate on a character boundary, but only once a suffix is actually
needed (mirroring make_unique_key_name), so a non-colliding name is
never altered.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
EXCHANGE PARTITION fails with ERROR 1736 (Tables have different definitions)
when tables contain generated columns with AND/OR conditions, even when the
expressions are logically equivalent. This occurs because when expressions are
re-parsed (e.g., via CREATE TABLE ... LIKE), the order of arguments in AND/OR
conditions may change, but the comparison was order-sensitive.
The Item_cond::eq() method was implemented to perform set-based comparison
for commutative AND/OR operations. The set-based comparison algorithm ensures
that two Item_cond expressions are considered equal if they contain the same set
of equivalent arguments, regardless of order.
Added comprehensive test case covering:
- Generated columns with OR conditions
- Generated columns with AND conditions
- Multiple generated columns with different AND/OR combinations
- Nested AND/OR conditions
The fix allows EXCHANGE PARTITION to succeed when expressions are logically
equivalent but have different argument ordering, which is correct behavior
since AND/OR operations are commutative.
MDEV-38757 Limit unordered vcol condition comparison to EXCHANGE PARTITION
Review follow-up to the previous commit, which made Item_cond::eq()
compare AND/OR argument lists as sets for every caller. That changed
equality semantics globally and broke main.derived_cond_pushdown, where
conditions that eq() started reporting as equal were dropped from
attached_condition. Reordering AND/OR operands also changes evaluation
order, which is observable when operands are functions, so the relaxed
comparison must not be the default.
Item::Eq_config gains an unordered_conditions flag, defaulting to false,
next to the existing binary_cmp and omit_table_names flags.
Item_cond::eq() compares its argument lists as sets only when that flag
is set, and otherwise reports two distinct Item_cond objects as unequal,
as it did before this patch series. The flag is threaded through
Virtual_column_info::is_equal() and a new mysql_compare_tables()
parameter, and only Sql_cmd_alter_table_exchange_partition passes it as
true, so the relaxed comparison stays confined to the EXCHANGE PARTITION
metadata check.
The set comparison tests containment in both directions rather than
comparing element counts, so an expression also matches a form that
repeats one of its terms, for example
col1 > 10 and col2 < 100 or col3 > 50
col3 > 50 or col1 > 10 and col2 < 100 or col3 > 50
Test 5 of parts.partition_exchange_generated_columns covers that case.
A failed ALTER TABLE ... RENAME INDEX ... ALGORITHM=INSTANT changed the
outcome of a subsequent, unrelated ALTER TABLE ... ADD FOREIGN KEY. A
failed statement must have no side effects, but here the second statement
wrongly succeeded where on a fresh table it correctly fails with
ER_DUP_KEYNAME.
Root cause: while rebuilding the key list, mysql_prepare_alter_table()
handled a RENAME INDEX request by clearing HA_GENERATED_KEY in place on
key_info->flags. key_info points into the (possibly cached) TABLE object
that is reused across statements. The ALGORITHM=INSTANT incompatibility is
only detected later, after mysql_prepare_alter_table() has returned, so the
statement fails with ER_ALTER_OPERATION_NOT_SUPPORTED with the cleared flag
never restored. The cached generated FK-support index (fk1) was thus left
permanently marked as user-defined.
That corrupted flag flips the de-duplication tie-break in the next ALTER:
adding FOREIGN KEY ind1 (b) creates a generated support index on column b
that prefix-matches fk1(b). Normally fk1 (generated) is dropped and the new
ind1 survives, colliding by name with the existing user index ind1(a) and
raising ER_DUP_KEYNAME. With fk1 no longer marked generated, the new ind1
is dropped instead, so no name collision is reached and the ADD FOREIGN KEY
silently succeeds.
Fix: do not mutate the cached key_info->flags. Track the "renamed => no
longer generated" decision in a per-key local variable (generated_key),
initialised from the flag, set to false on rename, and passed to the Key
constructor. This preserves the in-statement behaviour while leaving the
cached TABLE metadata untouched, so a failed ALTER has no lingering effect.
All new code of the whole pull request, including one or several files that
are either new files or modified ones, are contributed under the BSD-new
license. I am contributing on behalf of my employer Amazon Web Services, Inc.
Currently, ALTER TABLE ... AUTO_INCREMENT = N silently fails when using
values lower than the next AUTO_INCREMENT value in the column.
Add a note when attempting to set AUTO_INCREMENT to a lower value than
the next AUTO_INCREMENT value, and maintain the higher value.
Update existing test suites and add a new test to the maria suite for
the newly added note.
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.
Force REPEATABLE READ isolation for the duration of mysql_alter_table,
so HA_CHECK_UNIQUE_AFTER_WRITE never triggers during ALTER's exclusive table copy.
(cherry-pick of 46ee272a10 - online ALTER TABLE (MDEV-32100))
Field CHECK constraints always have the name of the corresponding field.
Table-level CHECK constraint can be arbitrary named and can have
the name of the field with a CHECK constraint.
Detect this and throw ER_DUP_CONSTRAINT_NAME.
Special treatment for auto-generated constraint names
(CONSTRAINT_1, etc) they can match a field name too - this is not
an error, just auto-generate a different name.
Assisted-By: Claude:claude-4.6-sonnet
1.
Only allow #mysql50# prefix on table names that
can not be decoded from a file name to a table name.
It is an error to put this prefix on a name that can be accessed
as a valid table name without a prefix. Use `t1` not `#mysql50#t1`,
use `a-b` not `#mysql50#a@002db`.
2.
Don't allow creation of new tables with the #mysql50# prefix.
Except when in SQLCOM_ALTER_DB_UPGRADE - main.upgrade test runs
ALTER DATABASE `#mysql50#mysqltest-1` UPGRADE DATA DIRECTORY NAME;
This command moves tables from the old db name to the new db name,
and some of these tables also have old #mysql50# names.
3.
Also don't allow to create such databases, views, partitions,
subpartitions, sequences, triggers.
Add new acquire_lock() overloads on MDL_context that combine MDL_request
initialization and lock acquisition into a single call, returning
MDL_ticket* directly. Convert applicable call sites across the codebase
to use the new ACQUIRE_LOCK macro.
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.
MDEV-38716 (fa36b269f1) fixes were incomplete. It still allowed a
table's default_fields to be left un-restored after an ALTER table
finished its copy_data_between_fields().
This patch actually matches the original conception of MDEV-38716. It
was later changed after finding a test failure in maria.alter, where I
thought the patch broke that test. But actually, maria.alter itself
relies on somewhat broken/inconsistent server behavior.
It is the MDEV-19055 extension of the test which broke. To summarize the
broken part of the test, first, it creates a temporary table t2, adds a
new column of type DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP, and adds
a constraint check on that new type:
CREATE TEMPORARY TABLE t2 (a TIME) ENGINE=Aria;
ALTER TABLE t2 ADD b DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP;
ALTER TABLE t2 ADD CHECK (b = 4);
The next part results in inconsistent behavior:
INSERT IGNORE INTO t2 () VALUES (),(),(),();
Prior to this patch, this would create 4 new rows, each with a zeroed
out timestamp for `b`. This is due to a the default_field not resetting
after the ALTER TABLE, thus the DEFAULT CURRENT_TIMESTAMP clause is lost
after the ALTER TABLE ADD CHECK.
With this patch, because the default_field is restored after the ALTER,
there is no affect of the INSERT IGNORE INTO t2. I.e., t2 is empty after
this insert.
This change in results further changes how an ALTER TABLE behaves later
in the test:
SET SQL_MODE= 'STRICT_ALL_TABLES';
SELECT count(a),sum(a) FROM t2;
--error ER_TRUNCATED_WRONG_VALUE
ALTER TABLE t2 CHANGE IF EXISTS d c INT;
Without this patch, there are rows in t2, and so this ALTER TABLE
results in an error. With this patch, t2 is empty, and therefore there
is no data that was truncated to warn about. It is generally bad test
practice to have a table with no rows, as it has historically masked
other bugs. So we add back a couple rows into the table to ensure it
hits additional logic. However, because the bug has been fixed, invalid
rows are not able to be inserted into the table and thereby the
ER_TRUNCATED_WRONG_VALUE error is still not thrown. So that error is
removed.
Reviewed-by: Monty <monty@mariadb.com>
Signed-off-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
Problem:
InnoDB index corruption occurs when an index is renamed to a name that
differs only in case (e.g., 'b' to 'B'). The SQL layer uses
case-insensitive comparison and fails to recognize the change.
Fix:
Use case-sensitive comparison when matching index names during
ALTER TABLE to correctly identify and handle case changes.
Problem:
UBSAN reports runtime errors in string comparision functions when
pointer arithmetic is done without checking NULL.
Fix:
- Add debug asserts in `strnncoll` function to catch NULL pointers.
- Define a new function `streq_safe` as a replacement for `streq`,
capable of handling NULL pointers. Replace `streq` with `streq_safe`
at multiple call sites, identified by the debug asserts.
- Add empty identifier checks in `is_infoschema_db` and
`is_perfschema_db` before calling the deep character-set based
comparision.
Reason:
=======
Following the changes in MDEV-38734, the server no longer marks
all indexed virtual columns during an UPDATE operation.
Consequently, ha_innobase::update() populates the upd_t vector
with old_vrow but omits the actual data for these virtual columns.
Despite this omission, trx_undo_page_report_modify() continues to
write metadata for indexed virtual columns into the undo log. Because
the actual values are missing from the update vector, the undo log
entry is recorded without the historical data for these columns.
When the purge thread processes the undo log to reconstruct a
previous record state for MVCC, it identifies an indexed virtual
column but finds no associated data.
The purge thread incorrectly interprets this missing data as a NULL
value, rather than a "missing/unrecorded" value. The historical
record is reconstructed with an incorrect NULL for the virtual column.
This causes the purge thread to incorrectly identify and purge
records that are not actually delete-marked, leading to abort
of server.
Solution:
=========
ha_innobase::column_bitmaps_signal(): Revert the column-marking
logic to the state prior to commit a4e4a56720, ensuring all
indexed virtual columns are unconditionally marked during an UPDATE.
The previous "optimization" attempted to manually detect indexed
column changes before marking virtual columns. The manual check
for indexed column modifications is redundant. InnoDB already
provides the UPD_NODE_NO_ORD_CHANGE flag within row_upd_step().
This flag is being used in trx_undo_page_report_modify() and
trx_undo_read_undo_rec() should log or read virtual column values.
Refactored column_bitmaps_signal() to accept a mark_for_update
parameter that controls when indexed virtual columns are marked.
TABLE::mark_columns_needed_for_update() is the only place that needs
mark_for_update=true because only UPDATE operations need to mark
indexed virtual columns for InnoDB's undo logging mechanism.
INSERT operation is already handled by
TABLE::mark_virtual_columns_for_write(insert_fl=true).
Even the commit a4e4a56720 changes are
going to affect TABLE::mark_virtual_column_for_write(false) and
It is called during UPDATE operation, and that's when
column_bitmaps_signal() needs to mark indexed virtual columns.
Online DDL has separate code path which is handled by
row_log_mark_virtual_cols() for all DML operations
Problem:
========
The issue occurs when ALTER TABLE contains duplicate DROP FOREIGN KEY
operations (e.g., "DROP FOREIGN KEY f1, DROP FOREIGN KEY f1").
In fk_prepare_copy_alter_table(), Removes from the list all foreign
keys which are to be dropped. But iterator continues to accessing the
removed entry when it encounters duplicate foreign key.
Solution:
=========
mysql_prepare_alter_table(): Add duplicate detection logic to handle
cases where the same foreign key is specified multiple times in a
single ALTER TABLE statement's DROP statement.
fk_prepare_copy_alter_table(): fk_parent_key_list and fk_child_key_list
contain all existing foreign keys where the table acts as parent or child
respectively, and they are populated through storage engine calls.
After this phase, the code iterates through these collections comparing
each foreign key against alter_info->drop_list using case-insensitive
string matching for constraint names and database/table names.
- When matches are found, instead of removing them immediately,
the matching FOREIGN_KEY_INFO objects are added to the
keys_to_remove collection, which acts as a temporary holding area.
Once all matching keys are collected in keys_to_remove, use it to
remove the key entries from original lists.
Problem:
=========
An assertion failure occurs in InnoDB during consecutive
ALTER TABLE operations using the COPY algorithm. The crash
happens during the table rename phase because the
source table (dict_table_t::n_ref_count) is non-zero, despite
the thread holding an exclusive Metadata Lock (MDL).
Reason:
========
When ALTER IGNORE TABLE is executed via the COPY algorithm,
it generates undo logs for every row inserted into the
intermediate table (e.g., #sql-alter-...). The background Purge
Thread, responsible for cleaning up these undo logs, attempts
to take an MDL on the table to prevent the table from
being dropped while in use.
Race condition:
==================
First ALTER: Creates #sql-alter-, copies data, and renames it to t1.
Purge Activation: The Purge thread picks up the undo logs from step 1.
It takes an MDL on the temporary name (#sql-alter-) and increments
the table's n_ref_count.
Identity Shift: InnoDB renames the physical table object to t1, but
the Purge thread still holds a reference to this object.
Second ALTER: Starts a new copy process. When it attempts to rename
the "new" t1 to a backup name, it checks if n_ref_count == 0.
Because the Purge thread is still "pinning" the object to
clean up logs from the first ALTER, the count is > 0,
triggering the assertion failure.
Solution:
========
ALTER IGNORE TABLE needs row-level undo logging to easily
roll back the last inserted row in case of duplicate key errors.
By discarding the last undo log record after inserting each row,
purge will not process any log records generated by
ALTER IGNORE TABLE, preventing unexpected access from the purge
subsystem during subsequent DDL operations.
Make skip_alter_undo (1-bit) to (2-bit enum)
to support different ALTER operation modes:
- NO_UNDO (0): Normal mode with standard undo logging
- SKIP_UNDO (1): ALTER mode that skips undo logging
- IGNORE_UNDO (2): ALTER IGNORE mode that rewrites undo blocks
trx_undo_report_row_operation(): Add ALTER IGNORE undo
rewriting logic. Store old undo record info before writing
new records for IGNORE_UNDO mode. Reset undo top_offset
to maintain only latest insert undo
row_ins_clust_index_entry_low(): Prevent ALTER IGNORE from
using bulk insert optimization. since it requires to write
undo log for each row operation that's incompatible with
existing bulk operation.
- This issue shares the same root cause of MDEV-38667. This
patch fixes the remaining caller of HA_EXTRA_END_ALTER_COPY by
temporarily setting abort_on_warning to false that prevents
the warning to error escalation.
Syntax:
TO_DATE(string_expression [DEFAULT string_expression ON CONVERSION ERROR],
format_string [,NLS_FORMAT_STRING])
The format_string has the same format elements as TO_CHAR(), except a
few elements that are not supported/usable for TO_DATE().
TO_DATE() returns a datetime or date value, depending on if the format
element FF is used.
Allowed separators, same as TO_CHAR():
space, tab and any of !#%'()*+,-./:;<=>
'&' can also be used if next character is not a character a-z or A-Z
"text' indicates a text string that is verbatim in the format. One cannot
use " as a separator.
Format elements supported by TO_DATE():
AD Anno Domini ("in the year of the Lord")
AD_DOT Anno Domini ("in the year of the Lord")
AM Meridian indicator (Before midday)
AM_DOT Meridian indicator (Before midday)
DAY Name of day
DD Day (1-31)
DDD Day of year (1-336)
DY Abbreviated name of day
FF[1-6] Fractional seconds
HH Hour (1-12)
HH12 Hour (1-12)
HH24 Hour (0-23)
MI Minutes (0-59)
MM Month (1-12)
MON Abbreviated name of month
MONTH Name of Month
PM Meridian indicator (After midday)
PM_DOT Meridian indicator (After midday)
RR 20th century dates in the 21st century. 2 digits
50-99 is assumed from 2000, 0-49 is assumed from 1900.
RRRR 20th century dates in the 21st century. 4 digits
SS Seconds
SYYYY Signed 4 digit year; MariaDB only supports positive years
Y 1 digit year
YY 2 digits year
YYY 3 digits year
YYYY 4 digits year
Note that if there is a missing part of the date, the current date is used!
For example if 'MM-DD HH-MM-SS' then the current year will be used.
(Oracle behaviour)
Not supported options:
- BC, D, DL, DS, E, EE, FM, FX, RM, SSSSS, TS, TZD, TZH, TZR, X,SY
BC is not supported by MariaDB datetime.
- Most of the other are exotic formats does not make sence in MariaDB as
we return datetime or datetime with fractions, not string.
- D (day-of-week) is not supported as it is not clear exactly how it would
map to MariaDB. This element depends on the NLS territory of the session.
- RR only works with 2 digit years (In Oracle RR can also work with 4
digit years in some context but the rules are not clear).
Extensions / differences compared to Oracle;
- MariaDB supports FF (fractional seconds). If FF[#] is used,
then TO_DATE will return a datetime with # of subseconds.
If FF is not used a datetime will be returned.
There is warning (no error) if string contains more digts than what
is specified with F(#]
- Names can be shortened to it's unique prefix. For example January and Ja
works fine.
- No error if the date string is shorter format_string and the next
not used character is not a number.. This is useful to get a date
from a mixed set of strings in date or datetime format.
Oracle gives an error if date string is too short.
- MariaDB supports short locales as language names
- NLS_DATE_FORMAT can use both " and ' for quoting.
- NLS_DATE_FORMAT must be a constant string.
- This is to ensure that the server knows which locale to use
when executing the function.
New formats handled by TO_CHAR():
FF[1-6] Fractional seconds
DDD Daynumber 1-366
IW Week 1-53 according to ISO 8601
I 1 digit year according to ISO 8601
IY 2 digit year according to ISO 8601
IYY 3 digit year according to ISO 8601
IYYY 4 digit year according to ISO 8601
SYYY 4 digit year according to ISO 8601 (Oracle can do signed)
Supported NLS_FORMAT_STRING options are:
NLS_CALENDAR=GREGORIAN
NLS_DATE_LANGUAGE=language
Support languages are:
- All MariaDB short locales, like en_AU.
- The following Oracle language names:
ALBANIAN, AMERICAN, ARABIC, BASQUE, BELARUSIAN, BRAZILIAN PORTUGUESE
BULGARIAN, CANADIAN FRENCH, CATALAN, CROATIAN, CYRILLIC SERBIAN CZECH,
DANISH, DUTCH, ENGLISH, ESTONIAN, FINNISH, FRENCH, GERMAN,
GREEK, HEBREW, HINDI, HUNGARIAN, ICELANDIC, INDONESIAN ITALIAN,
JAPANESE, KANNADA, KOREAN, LATIN AMERICAN SPANISH, LATVIAN,
LITHUANIAN, MACEDONIAN, MALAY, MEXICAN SPANISH, NORWEGIAN, POLISH,
PORTUGUESE, ROMANIAN, RUSSIAN, SIMPLIFIED CHINESE, SLOVAK, SLOVENIAN,
SPANISH, SWAHILI, SWEDISH, TAMIL, THAI, TRADITIONAL CHINESE, TURKISH,
UKRAINIAN, VIETNAMESE
Development bugs fixed:
MDEV-38403 Server crashes in Item_func_to_date::fix_length_and_dec upon
using an invalid argument
MDEV-38400 compat/oracle.func_to_date fails with PS protocol and cursor
protocol (Fixed by Serg)
MDEV-38404 TO_DATE: MTR coverage omissions, round 1
MDEV-38509 TO_DATE: AD_DOT does not appear to be supported
MDEV-38513 TO_DATE: NULL value for format string causes assertion failure
MDEV-38521 TO_DATE: Date strings with non-ASCII symbols cause warnings
and wrong results
MDEV-38578 TO_DATE: Possibly unexpected results upon wrong input
MDEV-38582 TO_DATE: NLS_DATE_LANGUAGE=JAPANESE does not parse values
which work in Oracle
MDEV-38584 TO_DATE: NLS_DATE_LANGUAGE=VIETNAMESE does not parse values
which work in Oracle
MDEV-38703 TO_DATE: Quotation for multi-word NLS_DATE_LANGUAGE leads
to syntax error in view definition
MDEV-38675 TO_DATE: MSAN/Valgrind/UBSAN errors in
extract_oracle_date_time
MDEV-38635 TO_DATE: UBSAN errors in item_timefunc.h upon comparison with
a view column
MDEV-38719 TO_DATE: Assertion `&my_charset_bin != charset()' failed in
String::append_for_single_quote_using_mb_wc
MDEV-38756 TO_DATE: MSAN/Valgrind errors in
Item_func_to_date::fix_length_and_dec upon PREPARE with
parameters
Known issues:
- Format string character matches inside quotes are done
one-letter-to-one-letter, like in LIKE predicate. That means things
like expansions and contractions do not work.
For example 'ss' does not match 'ß' in collations which treat them
as equal for the comparison operator.
Match is done taking into account case and accent sensitivity
of the subject argument collation, so for example this now works:
MariaDB [test]> SELECT TO_DATE('1920á12','YYYY"a"MM') AS c;
+---------------------+
| c |
+---------------------+
| 1920-12-17 00:00:00 |
+---------------------+
Co-author and reviewer: Alexander Barkov <bar@mariadb.com>
