Atomic CREATE OR REPLACE allows to keep an old table intact if the
command fails or during the crash. That is done by renaming the
original table to temporary name, as a backup and restoring it if the
CREATE fails. When the command is complete and logged the backup
table is deleted.
Atomic replace algorithm
Two DDL chains are used for CREATE OR REPLACE:
ddl_log_state_create (C) and ddl_log_state_rm (D).
1. (C) Log rename of ORIG to TMP table (Rename TMP to original).
2. Rename orignal to TMP.
3. (C) Log CREATE_TABLE_ACTION of ORIG (drops ORIG);
4. Do everything with ORIG (like insert data)
5. (D) Log drop of TMP
6. Write query to binlog (this marks (C) to be closed in
case of failure)
7. Execute drop of TMP through (D)
8. Close (C) and (D)
If there is a failure before 6) we revert the changes in (C)
Chain (D) is only executed if 6) succeded (C is closed on
crash recovery).
Foreign key errors will be found at the 1) stage.
Additional notes
- CREATE TABLE without REPLACE and temporary tables is not affected
by this commit.
set @@drop_before_create_or_replace=1 can be used to
get old behaviour where existing tables are dropped
in CREATE OR REPLACE.
- CREATE TABLE is reverted if binlogging the query fails.
- Engines having HTON_EXPENSIVE_RENAME flag set are not affected by
this commit. Conflicting tables marked with this flag will be
deleted with CREATE OR REPLACE.
- Replication execution is not affected by this commit.
- Replication will first drop the conflicting table and then
creating the new one.
- CREATE TABLE .. SELECT XID usage is fixed and now there is no need
to log DROP TABLE via DDL_CREATE_TABLE_PHASE_LOG (see comments in
do_postlock()). XID is now correctly updated so it disables
DDL_LOG_DROP_TABLE_ACTION. Note that binary log is flushed at the
final stage when the table is ready. So if we have XID in the
binary log we don't need to drop the table.
- Three variations of CREATE OR REPLACE handled:
1. CREATE OR REPLACE TABLE t1 (..);
2. CREATE OR REPLACE TABLE t1 LIKE t2;
3. CREATE OR REPLACE TABLE t1 SELECT ..;
- Test case uses 6 combinations for engines (aria, aria_notrans,
myisam, ib, lock_tables, expensive_rename) and 2 combinations for
binlog types (row, stmt). Combinations help to check differences
between the results. Error failures are tested for the above three
variations.
- expensive_rename tests CREATE OR REPLACE without atomic
replace. The effect should be the same as with the old behaviour
before this commit.
- Triggers mechanism is unaffected by this change. This is tested in
create_replace.test.
- LOCK TABLES is affected. Lock restoration must be done after new
table is created or TMP is renamed back to ORIG
- Moved ddl_log_complete() from send_eof() to finalize_ddl(). This
checkpoint was not executed before for normal CREATE TABLE but is
executed now.
- CREATE TABLE will now rollback also if writing to the binary
logging failed. See rpl_gtid_strict.test
backup ddl log changes
- In case of a successfull CREATE OR REPLACE we only log
the CREATE event, not the DROP TABLE event of the old table.
ddl_log.cc changes
ddl_log_execute_action() now properly return error conditions.
ddl_log_disable_entry() added to allow one to disable one entry.
The entry on disk is still reserved until ddl_log_complete() is
executed.
On XID usage
Like with all other atomic DDL operations XID is used to avoid
inconsistency between master and slave in the case of a crash after
binary log is written and before ddl_log_state_create is closed. On
recovery XIDs are taken from binary log and corresponding DDL log
events get disabled. That is done by
ddl_log_close_binlogged_events().
On linking two chains together
Chains are executed in the ascending order of entry_pos of execute
entries. But entry_pos assignment order is undefined: it may assign
bigger number for the first chain and then smaller number for the
second chain. So the execution order in that case will be reverse:
second chain will be executed first.
To avoid that we link one chain to another. While the base chain
(ddl_log_state_create) is active the secondary chain
(ddl_log_state_rm) is not executed. That is: only one chain can be
executed in two linked chains.
The interface ddl_log_link_chains() was defined in "MDEV-22166
ddl_log_write_execute_entry() extension".
Atomic info parameters in HA_CREATE_INFO
Many functions in CREATE TABLE pass the same parameters. These
parameters are part of table creation info and should be in
HA_CREATE_INFO (or whatever). Passing parameters via single
structure is much easier for adding new data and
refactoring.
Aria changes:
- Fixed issue in Aria engine with CREATE + locked tables
that data was not properly commited in some cases in
case of crashes.
InnoDB related changes (by Marko Mäkelä):
- table_name_t::is_create_or_replace(): A new predicate to check for
CREATE OR REPLACE TABLE will rename an old table to
and eventually drop after creating the replacement.
- dict_table_t::parse_name(): Do acquire MDL on #sql-create- names
for partitioned tables.
- dict_table_rename_in_cache(): On CREATE OR REPLACE TABLE ... SELECT,
forget the original dict_table_t::mdl_name so that purge will
acquire MDL on the #sql-create- name instead. In this way, the
MDL_EXCLUSIVE that the CREATE OR REPLACE TABLE holds on the
user-visible name will not unnecessarily block any purge of old history
until the very end when the #sql-create- table will be dropped.
- ha_innobase::delete_table(): Do not check FOREIGN KEY consistency
when dropping an #sql-create- table.
- row_rename_table_for_mysql(): Update SYS_FOREIGN.ID also
when renaming to #sql-create- in order to avoid any
duplicate key error when CREATE OR REPLACE TABLE is
creating some FOREIGN KEY constraints by names
that existed in the old table.
Other changes:
- Removed some auto variables in log.cc for better code readability.
- Fixed old bug that CREATE ... SELECT would not be able to auto repair
a table that is part of the SELECT.
- Marked MyISAM that it does not support ROLLBACK (not required but
done for better consistency with other engines).
- maria_create_trn_for_mysql() does not register a new transaction
handler for commits. This was needed to ensure create or replace
will not end with an active transaction.
- We do not get anymore warnings about "Engine not supporting atomic
create" when doing a legal CREATE OR REPLACE on a table with
foreign key constraints.
- Updated VIDEX engine flags to disable CREATE SEQUENCE.
- Removed mysql_mutex_unlock(&LOCK_gdl) / mysql_mutex_lock(&LOCK_gdl)
around calls to binlog as these are unsafe. The binlog code uses
global variables that needs protection from other caller.
- EITS data is preserved if create or replace fails if
drop_before_create_or_replace=OFF. If ON, then create or replace
will drop EITS before the drop of the original table (as before).
- Using CREATE OR REPLACE on a encrypted table that the user cannot
decrypt will fail instead of replacing the encrypted table.
The encrypted table will unchanged.
Known issues:
- One cannot use create or replace on an InnoDB tables that has foreign
key references point to it
- CREATE OR REPLACE TEMPORARY table is not full atomic. Any conflicting
table will always be dropped before creating a new one. (Old behaviour).
Bug fixes related to this MDEV:
MDEV-36435 Assertion failure in finalize_locked_tables()
MDEV-36439 Assertion `thd_arg->lex->sql_command != SQLCOM_CREATE_SEQUENCE...
MDEV-36498 Failed CoR in non-atomic mode no longer generates DROP in RBR...
MDEV-36508 Temporary files #sql-create-....frm occasionally stay after
crash recovery
MDEV-38479 Crash in CREATE OR REPLACE SEQUENCE when new sequence cannot
be created
MDEV-36497 Assertion failure after atomic CoR with Aria under lock in
transactional context
MDEV-36501 EITS data is lost after failed attempt to CREATE OR REPLACE
table
MDEV-36493 Atomic CREATE OR REPLACE ... SELECT blocks InnoDB purge
MDEV-39367 MSAN/valgrind errors in temp_file_size_cb_func,
main.tmp_space_usage fails
MDEV-39446 Atomic CREATE OR REPLACE fails if a table cannot be decrypted
MDEV-40776 Atomic CREATE OR REPLACE silently breaks the foreign key
MDEV-40765 Assertion `"unexpected references" == 0' failed upon failing
CREATE OR REPLACE
MDEV-40994 Atomic CoR: Transaction not registered for MariaDB 2PC, but
transaction is active
Reverted commits:
- MDEV-36685 "CREATE-SELECT may lose in binlog side-effects of
stored-routine" as it did not take into account that it safe to
clear binlogs if the created table is non transactional and there
are no other non transactional tables used.
This was done because it caused extra logging when it is not
needed (not using any non transactional tables) and it also did
not solve side effects when using statement based loggging.
This is part of MDEV-25292 Atomic CREATE OR REPLACE TABLE.
Removed default values for arguments, added flags argument to specify
filename flags (FN_TO_IS_TMP, FN_FROM_IS_TMP) and forward the flag to
build_table_name().
Original patch from: Aleksey Midenkov <midenok@gmail.com>
DENY ... TO PUBLIC denies everyone, including whoever tries to revoke
it, via the "deny wins" merge at every scope (global, db, table, column,
routine). Allow REVOKE DENY ... FROM PUBLIC when the revoker can UPDATE
mysql.global_priv (same as hand-editing).
Assisted-by: Claude:claude-5-sonnet
The "is it my own thread" check compared the login user name
(Security_context::user) and ignored the host, so u1@localhost could see
and kill threads of u1@'127.0.0.1'.
Compare the authenticated account instead:
* all comparisons are done in sctx->is_priv_user() now
* change user_matches() to priv_user_matches(), which uses is_priv_user()
* use it in KILL, KILL USER, SHOW PROCESSLIST, I_S.PROCESSLIST,
COM_PROCESS_INFO and SHOW EXPLAIN/ANALYZE FOR.
* all the remaining places use is_priv_user() directly instead
of doing strcmp: SHOW GRANTS, SHOW CREATE PROCEDURE, I_S.VIEWS,
the DEFINER clause, optimizer trace and change_security_context().
Assisted-by: Claude:claude-5-opus
In propagate_role_grants_action() (called by grant_reload() on startup
and FLUSH PRIVILEGES), role privilege merging uses a counter-based
bottom-up ordering: each role's counter tracks how many of its granted
sub-roles still need to be processed, and merge_role_privileges() merges
a role only once its counter reaches zero.
The bug: when an intermediate role such as org_role_1 (which inherits
only from a USAGE-only app_role_1) was merged with no privilege changes,
merge_role_privileges() returned 1 as an optimisation to stop upward
traversal. This prevented the traversal from ever reaching user_role_1
to decrement its counter. A second leaf traversal (from app_role_2)
could decrement user_role_1's counter only once, leaving it at 1
instead of 0, so user_role_1 was never merged and had no effective
privileges.
The "stop if nothing changed" optimisation is valid for incremental
propagate_role_grants() calls (after a single GRANT/REVOKE), where
every ancestor was already correctly merged. It is not valid for the
full-reload case, where the counters must be decremented by every leaf
traversal to guarantee correct bottom-up ordering.
Fix: add an initial_load flag to PRIVS_TO_MERGE. When set (only in
propagate_role_grants_action), the early-stop optimisation is disabled
so traversals always visit all ancestors.
Assisted-By: Claude:claude-4.6-sonnet
a90779098b changed the
interface to the json_valid to take a json_engine_t
so that it could be interupted and so that warnings
could become visible.
On merge from that 10.11 to 11.4 it was discovered
that the ColumnStore engine uses this interface.
Restored the original json_valid function exactly
how ColumnStore expects this and added renamed
the function with json_engine_t to be json_valid_engine.
This follows the same convention as in the commit
756fc6fd9a that was
for json_normalize, that ColumnStore also used.
a90779098b changed the
interface to the json_valid to take a json_engine_t
so that it could be interupted and so that warnings
could become visible.
On merge from that 10.11 to 11.4 it was discovered
that the ColumnStore engine uses this interface.
Restored the original json_valid function exactly
how ColumnStore expects this and added renamed
the function with json_engine_t to be json_valid_engine.
This follows the same convention as in the commit
756fc6fd9a that was
for json_normalize, that ColumnStore also used.
Or max_statement_time limit.
Change the json_valid to take a json_engine_t argument.
Adjust the Item_json_valid to have a json_engine_t and
to report an error.
Note this means that invalid json syntax now has a
note on any json error. Raising this to a warning can
cause check constraints of JSON_VALID() to result
in ER_JSON_SYNTAX rather than ER_CONSTRAINT_FAILED
(valided in main.type_json test).
Consequential changes in debug build to json acl
testing validation and unit test having a dummy json_engine_t.
json_normalize also uses the same engine that was
passed as its argument to test json_valid. Because
the character set may have changed, if there's an error
adjust the je->s.c_str pointer. Its only the offset that
is currently used in the error messages anyway so there
may be offset errors depending on the original character
set.
The database name in GRANT is stored in mysql.db.Db, which holds 64
characters. Escaping '_' or '%' with a backslash to match them literally
can make the stored name longer than 64 characters, and it was silently
truncated into a non-functional grant. Reject it with ER_WRONG_DB_NAME
instead, the same error CREATE DATABASE gives for an over-long name.
Add support for CLIENT_PLUGIN_AUTH_LENENC_CLIENT_DATA in COM_CHANGE_USER
packet parsing, allowing passwords >= 251 bytes.
Changes:
- Server: parse_com_change_user_packet() now handles LENENC-encoded
password length when CLIENT_PLUGIN_AUTH_LENENC_CLIENT_DATA is set
- Server: fix db pointer calculation for old protocol without
CLIENT_SECURE_CONNECTION (need +1 to skip null terminator)
- Add regression test for COM_CHANGE_USER with long passwords
Note: This fix requires a corresponding client-side fix in libmariadb's
send_change_user_packet() to send LENENC-encoded passwords.
Document that the packet buffer is null-terminated by the network layer,
ensuring strend(user) remains within bounds and passwd stays within
the packet.
Remove the unscoped #pragma GCC diagnostic ignored
"-Wdeprecated-declarations" from include/violite.h and replace all
sprintf/vsprintf calls with snprintf/vsnprintf across the codebase.
Where possible, pass actual buffer sizes as function parameters
instead of using hardcoded constants or recomputing strlen().
Notable API changes:
- get_date(): added to_len parameter for buffer size
- nice_time(), end_timer() in client/mysql.cc: added buff_size
- generate_new_name(): added name_size parameter
- calc_md5(): added buffer_size parameter (fixes off-by-one)
- MD5_HASH_TO_STRING macro: added _size parameter
- Added MYSQL_UDF_MAX_RESULT_LENGTH constant in mysql_com.h
Also fixed a pre-existing bug in sql/rpl_mi.cc where the ellipsis
"..." was written to the wrong buffer (dbuff instead of buff).
Vendored code is excluded: extra/readline, extra/wolfssl,
wsrep-lib, libmariadb, storage/mroonga/vendor, zlib.
removed:
* DES encryption
* --secure-auth
* --old
* spider table options: bfz, btt, cmd, ctp, cwg, isa, ilm, ios, smd, stc, stl
extended under old-mode:
* YEAR(2), still available when old-mode=2_DIGIT_YEAR
un-deprecated:
* keep_files_on_create, originally (MDEV-23570) the idea was to make it
TRUE and deprecate. It cannot be removed when FALSE, but TRUE breaks
mariabackup.aria_backup where a table is altered from Aria to InnoDB
during a backup, so both t.MAD/t.MAI and t.ibd gets into a backup.
* rename to follow the standard vprintf name convention
* remove `(LEX_CSTRING*) &lex_string` casts
* remove some redundant casts too
* LEX_CSTRING -> Lex_ident_db in path
* Hash_set<Sroutine_hash_entry>
* move assert out of the loop
* some comments and whitespace changes
PARTITION BY [LINEAR] KEY ALGORITHM={MYSQL51|MYSQL55|BASE31|CRC32C|XXH32|XXH3}
- The MYSQL5X algorithms are the existing algorithms, with MYSQL55
being the default
- The BASE31 algorithm uses a base-31 representation of the bytes, see
Modular hashing in https://algs4.cs.princeton.edu/34hash/. It serves
as a simple baseline that distributes better than the old mysql5x
algorithms
- CRC32C uses my_crc32c.
- XXH32 and XXH3 are xxhash algorithms - xxhash.h copied from latest
release (0.8.3) of https://github.com/Cyan4973/xxHash
For performance (esp. xxh) we use one-shot hash functions in binary
hash_sort, and streaming hash function otherwise for byte-by-byte
hashing. XXH is the only stateful hash function. The other hash
algorithms are stateless and homomorphic, so streaming and one-shot
functions are identical.
Also updated the columnstore hash due to the change of MY_HASH_ADD
signature.
The following testing commands were run at an earlier version of the
patch with the following patch that changes the default algorithm from
MYSQL55 to CRC32C, and XXH32 (changing the patch accordingly)
mtr --suite main --do-test=.*partition
mtr --suite parts
modified sql/ha_partition.cc
@@ -10336,6 +10336,8 @@ uint32 ha_partition::calculate_key_hash_value(Field **field_array)
switch ((*field_array)->table->part_info->key_algorithm)
{
case partition_info::KEY_ALGORITHM_NONE:
+ hasher.set_algorithm(HASH_ALGORITHM_CRC32C);
+ break;
case partition_info::KEY_ALGORITHM_55:
/* Hasher default to mysql55 */
break;
modified sql/partition_info.cc
@@ -2328,7 +2328,7 @@ bool partition_info::fix_parser_data(THD *thd)
if ((thd_sql_command(thd) == SQLCOM_CREATE_TABLE ||
thd_sql_command(thd) == SQLCOM_ALTER_TABLE) &&
key_algorithm == KEY_ALGORITHM_NONE)
- key_algorithm= KEY_ALGORITHM_55;
+ key_algorithm= PARTITION_INFO_DEFAULT_ALGORITHM;
}
DBUG_RETURN(FALSE);
}
@@ -2344,7 +2344,7 @@ bool partition_info::fix_parser_data(THD *thd)
if ((thd_sql_command(thd) == SQLCOM_CREATE_TABLE ||
thd_sql_command(thd) == SQLCOM_ALTER_TABLE) &&
key_algorithm == KEY_ALGORITHM_NONE)
- key_algorithm= KEY_ALGORITHM_55;
+ key_algorithm= PARTITION_INFO_DEFAULT_ALGORITHM;
}
defined_max_value= FALSE; // in case it already set (CREATE TABLE LIKE)
do
modified sql/partition_info.h
@@ -446,6 +446,8 @@ class partition_info : public DDL_LOG_STATE, public Sql_alloc
int gen_part_type(THD *thd, String *str) const;
};
+#define PARTITION_INFO_DEFAULT_ALGORITHM partition_info::KEY_ALGORITHM_CRC32C
+
void part_type_error(THD *thd, partition_info *work_part_info,
const char *part_type, partition_info *tab_part_info);
modified sql/sql_partition.cc
@@ -2471,7 +2471,7 @@ static int add_key_with_algorithm(String *str, const partition_info *part_info)
err+= str->append(STRING_WITH_LEN("KEY "));
if (part_info->key_algorithm != partition_info::KEY_ALGORITHM_NONE &&
- part_info->key_algorithm != partition_info::KEY_ALGORITHM_55)
+ part_info->key_algorithm != PARTITION_INFO_DEFAULT_ALGORITHM)
{
err+= str->append(STRING_WITH_LEN("ALGORITHM = "));
switch (part_info->key_algorithm)
@@ -2479,6 +2479,9 @@ static int add_key_with_algorithm(String *str, const partition_info *part_info)
case partition_info::KEY_ALGORITHM_51:
err+= str->append(STRING_WITH_LEN("MYSQL51"));
break;
+ case partition_info::KEY_ALGORITHM_55:
+ err+= str->append(STRING_WITH_LEN("MYSQL55"));
+ break;
case partition_info::KEY_ALGORITHM_BASE31:
err+= str->append(STRING_WITH_LEN("BASE31"));
break;
