This merges the patch series that fixes a lot of missing validation
checks for read_log_event(). The merge includes a few added fixes that
are needed in new code introduced in 12.3.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
Bug 1: put_dbopt() used strmov() to copy schema_comment into a fixed
DATABASE_COMMENT_MAXLEN+1 buffer. validate_comment_length() only
truncates comment->length in non-strict sql_mode, leaving comment->str
NUL-terminated at its original (unbounded) length. strmov() copies
until the source NUL, ignoring the truncated length, overflowing the
destination buffer for long comments.
The fix uses strmake() bounded by comment->length instead, matching
the LEX_CSTRING contract (length is authoritative, str need not be
NUL-terminated at length).
Bug 2: write_db_opt() used strxnmov() to copy the full un-truncated
comment until it ran out of buffer space mid-string with no trailing
newline. Which made load_db_opt() silently discard the whole
unterminated "comment=" line on the next restart, losing the comment
entirely instead of just truncating it.
The fix bounds the comment copy into db.opt by the already-validated
comment->length via strmake(), instead of relying on the source
string's own NUL terminator, matching the put_dbopt() fix.
Bug 3: validate_comment_length() only runs on a COMMENT clause given
in the current statement. ALTER DATABASE without one instead pulls
the existing comment off disk via load_db_opt(), which never bounded
it. That unvalidated length then reached write_db_opt()'s own
comment= copy into its stack buffer, so a legacy or hand-edited
db.opt with an overlong comment= line overflowed it on ALTER DATABASE.
The fix: load_db_opt() now clamps the parsed comment to
DATABASE_COMMENT_MAXLEN right when it reads the "comment=" line, so
every consumer (put_dbopt(), write_db_opt()'s ALTER path) always sees
an already-bounded value.
The clamp itself must truncate by bytes, not characters:
Well_formed_prefix()'s LEX_CSTRING overload takes a character count,
but DATABASE_COMMENT_MAXLEN sizes the buffers in bytes.
Bug 4: write_db_opt() still trusted validate_comment_length() to bound
a directly-given COMMENT to DATABASE_COMMENT_MAXLEN bytes, but it only
bounds it to DATABASE_COMMENT_MAXLEN *characters* -- so a multi-byte
comment could still overflow the same fixed buffers.
The fix: write_db_opt() clamps schema_comment to
DATABASE_COMMENT_MAXLEN bytes itself, right after
validate_comment_length() returns.
1.
Only allow #mysql50# prefix on table names that
can not be decoded from a file name to a table name.
It is an error to put this prefix on a name that can be accessed
as a valid table name without a prefix. Use `t1` not `#mysql50#t1`,
use `a-b` not `#mysql50#a@002db`.
2.
Don't allow creation of new tables with the #mysql50# prefix.
Except when in SQLCOM_ALTER_DB_UPGRADE - main.upgrade test runs
ALTER DATABASE `#mysql50#mysqltest-1` UPGRADE DATA DIRECTORY NAME;
This command moves tables from the old db name to the new db name,
and some of these tables also have old #mysql50# names.
3.
Also don't allow to create such databases, views, partitions,
subpartitions, sequences, triggers.
Problem:
UBSAN reports runtime errors in string comparision functions when
pointer arithmetic is done without checking NULL.
Fix:
- Add debug asserts in `strnncoll` function to catch NULL pointers.
- Define a new function `streq_safe` as a replacement for `streq`,
capable of handling NULL pointers. Replace `streq` with `streq_safe`
at multiple call sites, identified by the debug asserts.
- Add empty identifier checks in `is_infoschema_db` and
`is_perfschema_db` before calling the deep character-set based
comparision.
* rename to follow the standard vprintf name convention
* remove `(LEX_CSTRING*) &lex_string` casts
* remove some redundant casts too
* LEX_CSTRING -> Lex_ident_db in path
* Hash_set<Sroutine_hash_entry>
* move assert out of the loop
* some comments and whitespace changes
Added caching of database directories that did not have a db.opt file.
This was common for older MariaDB installaiton or if a user created
a database with 'mkdir'.
Other things:
- Give a note "no db.opt file" if one uses SHOW CREATE DATABASE one
a database without a db.opt file.
[Breaking]
Good news:
GCC now checks your `my_snprintf` (direct) calls (`-Wformat` was on);
Bad news:
The build process no longer lets your incorrect
formats/arguments sneak past (`-Werror` was also on).
As such, this commit also migrates all direct `my_snprintf` calls from
the old specifiers to MDEV-21978’s new `-Wformat`-compatible suffixes.
The next commits will cover non-direct calls to `my_snprintf`.
(I call them “`my_snprintf` descendants”.)
This commit does not update the ABI records because
there’re more ABI “changes” to come – half a dozen
`include/mysql/plugin_*.h.pp`s are now missing the new `__attribute__`s.
Partial commit of the greater MDEV-34348 scope.
MDEV-34348: MariaDB is violating clang-16 -Wcast-function-type-strict
Change the type of my_hash_get_key to:
1) Return const
2) Change the context parameter to be const void*
Also fix casting in hash adjacent areas.
Reviewed By:
============
Marko Mäkelä <marko.makela@mariadb.com>
create templates
thd->alloc<X>(n) to use instead of (X*)thd->alloc(sizeof(X)*n)
and the same for thd->calloc(). By the default the type is char,
so old usage of thd->alloc(size) works too.
The patch for MDEV-31340 fixed the following bugs:
MDEV-33084 LASTVAL(t1) and LASTVAL(T1) do not work well with lower-case-table-names=0
MDEV-33085 Tables T1 and t1 do not work well with ENGINE=CSV and lower-case-table-names=0
MDEV-33086 SHOW OPEN TABLES IN DB1 -- is case insensitive with lower-case-table-names=0
MDEV-33088 Cannot create triggers in the database `MYSQL`
MDEV-33103 LOCK TABLE t1 AS t2 -- alias is not case sensitive with lower-case-table-names=0
MDEV-33108 TABLE_STATISTICS and INDEX_STATISTICS are case insensitive with lower-case-table-names=0
MDEV-33109 DROP DATABASE MYSQL -- does not drop SP with lower-case-table-names=0
MDEV-33110 HANDLER commands are case insensitive with lower-case-table-names=0
MDEV-33119 User is case insensitive in INFORMATION_SCHEMA.VIEWS
MDEV-33120 System log table names are case insensitive with lower-cast-table-names=0
Backporting the fixes from 11.5 to 10.5
This patch also fixes:
MDEV-33050 Build-in schemas like oracle_schema are accent insensitive
MDEV-33084 LASTVAL(t1) and LASTVAL(T1) do not work well with lower-case-table-names=0
MDEV-33085 Tables T1 and t1 do not work well with ENGINE=CSV and lower-case-table-names=0
MDEV-33086 SHOW OPEN TABLES IN DB1 -- is case insensitive with lower-case-table-names=0
MDEV-33088 Cannot create triggers in the database `MYSQL`
MDEV-33103 LOCK TABLE t1 AS t2 -- alias is not case sensitive with lower-case-table-names=0
MDEV-33109 DROP DATABASE MYSQL -- does not drop SP with lower-case-table-names=0
MDEV-33110 HANDLER commands are case insensitive with lower-case-table-names=0
MDEV-33119 User is case insensitive in INFORMATION_SCHEMA.VIEWS
MDEV-33120 System log table names are case insensitive with lower-cast-table-names=0
- Removing the virtual function strnncoll() from MY_COLLATION_HANDLER
- Adding a wrapper function CHARSET_INFO::streq(), to compare
two strings for equality. For now it calls strnncoll() internally.
In the future it will turn into a virtual function.
- Adding new accent sensitive case insensitive collations:
- utf8mb4_general1400_as_ci
- utf8mb3_general1400_as_ci
They implement accent sensitive case insensitive comparison.
The weight of a character is equal to the code point of its
upper case variant. These collations use Unicode-14.0.0 casefolding data.
The result of
my_charset_utf8mb3_general1400_as_ci.strcoll()
is very close to the former
my_charset_utf8mb3_general_ci.strcasecmp()
There is only a difference in a couple dozen rare characters, because:
- the switch from "tolower" to "toupper" comparison, to make
utf8mb3_general1400_as_ci closer to utf8mb3_general_ci
- the switch from Unicode-3.0.0 to Unicode-14.0.0
This difference should be tolarable. See the list of affected
characters in the MDEV description.
Note, utf8mb4_general1400_as_ci correctly handles non-BMP characters!
Unlike utf8mb4_general_ci, it does not treat all BMP characters
as equal.
- Adding classes representing names of the file based database objects:
Lex_ident_db
Lex_ident_table
Lex_ident_trigger
Their comparison collation depends on the underlying
file system case sensitivity and on --lower-case-table-names
and can be either my_charset_bin or my_charset_utf8mb3_general1400_as_ci.
- Adding classes representing names of other database objects,
whose names have case insensitive comparison style,
using my_charset_utf8mb3_general1400_as_ci:
Lex_ident_column
Lex_ident_sys_var
Lex_ident_user_var
Lex_ident_sp_var
Lex_ident_ps
Lex_ident_i_s_table
Lex_ident_window
Lex_ident_func
Lex_ident_partition
Lex_ident_with_element
Lex_ident_rpl_filter
Lex_ident_master_info
Lex_ident_host
Lex_ident_locale
Lex_ident_plugin
Lex_ident_engine
Lex_ident_server
Lex_ident_savepoint
Lex_ident_charset
engine_option_value::Name
- All the mentioned Lex_ident_xxx classes implement a method streq():
if (ident1.streq(ident2))
do_equal();
This method works as a wrapper for CHARSET_INFO::streq().
- Changing a lot of "LEX_CSTRING name" to "Lex_ident_xxx name"
in class members and in function/method parameters.
- Replacing all calls like
system_charset_info->coll->strcasecmp(ident1, ident2)
to
ident1.streq(ident2)
- Taking advantage of the c++11 user defined literal operator
for LEX_CSTRING (see m_strings.h) and Lex_ident_xxx (see lex_ident.h)
data types. Use example:
const Lex_ident_column primary_key_name= "PRIMARY"_Lex_ident_column;
is now a shorter version of:
const Lex_ident_column primary_key_name=
Lex_ident_column({STRING_WITH_LEN("PRIMARY")});
Under terms of MDEV 27490 we'll add support for non-BMP identifiers
and upgrade casefolding information to Unicode version 14.0.0.
In Unicode-14.0.0 conversion to lower and upper cases can increase octet length
of the string, so conversion won't be possible in-place any more.
This patch removes virtual functions performing in-place casefolding:
- my_charset_handler_st::casedn_str()
- my_charset_handler_st::caseup_str()
and fixes the code to use the non-inplace functions instead:
- my_charset_handler_st::casedn()
- my_charset_handler_st::caseup()
Backporting a part of MDEV-32026 (which also fixed MDEV-32025 in 11.3)
from 11.3 to 10.4.
The reported crash happened with --lower-case-table-names=2
on statements like:
ALTER DATABASE Db1 DEFAULT CHARACTER SET utf8;
ALTER DATABASE `#mysql50#D+b1` UPGRADE DATA DIRECTORY NAME;
lock_schema_name() expects a normalized database name
and assert if a non-normalized name comes.
mysql_alter_db_internal() and mysql_upgrade_db() get
a non-normalized database name in the parameter.
Fixing them to normalize the database name before passing
it to lock_schema_name().
Also fixes MDEV-32025 Crashes in MDL_key::mdl_key_init with lower-case-table-names=2
Change overview:
- In changes made in MDEV-31948, MDEV-31982 the code path
which originaly worked only in case of lower-case-table-names==1
also started to work in case of lower-case-table-names==2 in a mistake.
Restoring the original check_db_name() compatible behavior
(but without re-using check_db_name() itself).
- MDEV-31978 erroneously added a wrong DBUG_ASSERT. Removing.
Details:
- In mysql_change_db() the database name should be lower-cased only
in case of lower_case_table_names==1. It should not be lower-cased
for lower_case_table_names==2. The problem was caused by MDEV-31948.
The new code version restored the pre-MDEV-31948 behavior, which
used check_db_name() behavior.
- Passing lower_case_table_names==1 instead of just lower_case_table_names
to the "casedn" parameter to DBNameBuffer constructor in sql_parse.cc
The database name should not be lower-cased for lower_case_table_names==2.
This restores pre-MDEV-31982 behavioir which used check_db_name() here.
- Adding a new data type Lex_ident_db_normalized, it stores database
names which are both checked and normalized to lower case
in case lower_case_table_names==1 and lower_case_table_names==2.
- Changing the data type for the "db" parameter to Lex_ident_db_normalized in
lock_schema_name(), lock_db_routines(), find_db_tables_and_rm_known_files().
This is to avoid incorrectly passing a non-normalized name in the future.
- Restoring the database name normalization in mysql_create_db_internal()
and mysql_rm_db_internal() before calling lock_schema_name().
The problem was caused MDEV-31982.
- Adding database name normalization in mysql_alter_db_internal()
and mysql_upgrade_db(). This fixes MDEV-32026.
- Removing a wrong assert in Create_sp_func::create_with_db() was incorrect:
DBUG_ASSERT(Lex_ident_fs(*db).ok_for_lower_case_names());
The database name comes to here checked, but not normalized
to lower case with lower-case-table-names=2.
The assert was erroneously added by MDEV-31978.
- Recording lowercase_tables2.results and lowercase_tables4.results
according to
MDEV-29446 Change SHOW CREATE TABLE to display default collations
These tests are skipped on buildbot on all platforms, so this change
was forgotten in the patch for MDEV-29446.
- Adding a new class Lex_ident_db, to store normalized database names:
lower-cased if lower-case-table-name says so,
and checked to be a valid database name using Lex_ident_fs::check_db_name()
- Reusing the new class in parameters to functions:
prepare_db_action()
mysql_create_db()
mysql_alter_db()
mysql_rm_db()
mysql_upgrade_db()
This change removed two old-style check_db_name() calls.
- Changing the global function ok_for_lower_case_names()
into a method in class Lex_ident_fs.
- Changing a few functions/methods to get the database name
as a "const LEX_CSTRING" instead of a "const char *".
All these functions/methods use ok_for_lower_case_names()
inside. This change helps to avoid new strlen() calls, and also
removes a few old strlen() calls.
The function normalize_db_name() fully repeated the functionality
of the class DBNameBuffer. This patch removes normalize_db_name()
and replaces it to a DBNameBuffer based code.
- Adding a class Lex_ident_fs, to store identifiers for on-disk
database objects, such as databases, tables, triggers.
- Moving the validation code from check_db_name()
to non-modifying methods in Lex_ident_fs:
Lex_ident_fs::check_body()
Lex_ident_fs::check_db_name()
Adding a new method Lex_ident_fs::check_db_name_with_error(),
which performs validation and raises an error on validation failure.
Unlike the old function check_db_name(), the new class Lex_ident_fs
does not lower-case the identifier during the validation.
Lower-casing must be done before calling Lex_ident_fs validation methods.
- Adding a low level helper template class CharBuffer which can:
* store exact or lower-cased strings with a short fixed maximum length
* return the value as a LEX_CSTRING efficiently
- Adding a helper template class DBNameBuffer (deriving from CharBuffer), to
allocate optionally lower-cased database identifiers on stack when relevant.
Useful for temporary values which don't need to be allocated on MEM_ROOT.
- Using DBNameBuffer in mysql_change_db()
- Using DBNameBuffer in show_create_db()
Specifically:
Revert "MDEV-29664 Assertion `!n_mysql_tables_in_use' failed in innobase_close_connection"
This reverts commit ba875e9396.
Revert "MDEV-29620 Assertion `next_insert_id == 0' failed in handler::ha_external_lock"
This reverts commit aa08a7442a.
Revert "MDEV-29628 Memory leak after CREATE OR REPLACE with foreign key"
This reverts commit c579d66ba6.
Revert "MDEV-29609 create_not_windows test fails with different result"
This reverts commit cb583b2f1b.
Revert "MDEV-29544 SIGSEGV in HA_CREATE_INFO::finalize_locked_tables"
This reverts commit dcd66c3814.
Revert "MDEV-28933 CREATE OR REPLACE fails to recreate same constraint name"
This reverts commit cf6c517632.
Revert "MDEV-28933 Moved RENAME_CONSTRAINT_IDS to include/sql_funcs.h"
This reverts commit f1e1c1335b.
Revert "MDEV-28956 Locking is broken if CREATE OR REPLACE fails under LOCK TABLES"
This reverts commit a228ec80e3.
Revert "MDEV-25292 gcol.gcol_bugfixes --ps fix"
This reverts commit 24fff8267d.
Revert "MDEV-25292 Disable atomic replace for slave-generated or-replace"
This reverts commit 2af15914cb.
Revert "MDEV-25292 backup_log improved"
This reverts commit 34398a20b5.
Revert "MDEV-25292 Atomic CREATE OR REPLACE TABLE"
This reverts commit 93c8252f02.
Revert "MDEV-25292 Table_name class for (db, table_name, alias)"
This reverts commit d145dda9c7.
Revert "MDEV-25292 ha_table_exists() cleanup and improvement"
This reverts commit 409b8a86de.
Revert "MDEV-25292 Cleanups"
This reverts commit 595dad83ad.
Revert "MDEV-25292 Refactoring: moved select_field_count into Alter_info."
This reverts commit f02af1d229.
Also fixes
MDEV-27782 Wrong columns when using table level `CHARACTER SET utf8mb4 COLLATE DEFAULT`
MDEV-28644 Unexpected error on ALTER TABLE t1 CONVERT TO CHARACTER SET utf8mb3, DEFAULT CHARACTER SET utf8mb4
https://jira.mariadb.org/browse/MDEV-26221
my_sys DYNAMIC_ARRAY and DYNAMIC_STRING inconsistancy
The DYNAMIC_STRING uses size_t for sizes, but DYNAMIC_ARRAY used uint.
This patch adjusts DYNAMIC_ARRAY to use size_t like DYNAMIC_STRING.
As the MY_DIR member number_of_files is copied from a DYNAMIC_ARRAY,
this is changed to be size_t.
As MY_TMPDIR members 'cur' and 'max' are copied from a DYNAMIC_ARRAY,
these are also changed to be size_t.
The lists of plugins and stored procedures use DYNAMIC_ARRAY,
but their APIs assume a size of 'uint'; these are unchanged.
This fixed the MySQL bug# 20338 about misuse of double underscore
prefix __WIN__, which was old MySQL's idea of identifying Windows
Replace it by _WIN32 standard symbol for targeting Windows OS
(both 32 and 64 bit)
Not that connect storage engine is not fixed in this patch (must be
fixed in "upstream" branch)
