When a select containing 32 ranges is made on a table containing a
compound key with 32 parts, the range optimizer can run off the end
of a stack variable, invalidly overwriting subsequent stack variables.
In the struct st_sel_arg_range_seq, we have an array
RANGE_SEQ_ENTRY stack[MAX_REF_PARTS];
MAX_REF_PARTS is 32.
check_quick_select
/ sel_arg_range_seq_init
initialises stack[0] as NOT a key part
/ sel_arg_range_seq_next
iterates through the key parts, adding key part n to stack[n+1]
key part #32 gets referenced by step_down_to(), setting seq->i off the
end of the array.
Fix: RANGE_SEQ_ENTRY stack[MAX_REF_PARTS+1];
The above change exposed an issue with key length calculation on
MS Windows. Calling make_prev_keypart_map(32) caused the resultant
bitmap to be calculated as (1UL << 32) - 1. Using the MSVC compiler
this resulted in an empty key length calculation during
handler::index_read_map, causing an assertion in ha_innobase::index_read().
As we only need 32 bits to represent our key map, we change the type thus
-typedef ulong key_part_map;
+typedef uint32 key_part_map;
We correct make_keypart_map() and make_prev_keypart_map() to call our
overflow safe my_set_bits(). We also correct bka_range_seq_next()
and bkah_range_seq_next() to use make_prev_keypart_map().
We also add some DBUG_ASSERTS in key_part_map processing elsewhere,
exposing some issues in our BNLH implementation. We cap the number of
keyuse parts here, altering the explain output of 2 of our tests.
Numerous places needed bit shift operations altered to use
make*keymap_part and various format strings needed to be corrected.
Defer the host-privileged/host-blocked check for a PROXY-header-derived
address until after the client's SSL handshake completes, instead of
sending it immediately in clear text.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
my_large_malloc() attempted large pages whenever --large-pages was
enabled, silently rounding the size up and reporting it back via an
in/out parameter. ut_malloc_dontdump() never passed that adjusted
size on to its own callers (the InnoDB redo log buffer and
recv_sys_t::tmp_buf), so freeing later used the original, smaller
size, causing the reported "faux memory leak".
Only the buffer pool and the MyISAM/Aria key caches are documented
to benefit from large pages. Everything else that ended up calling
my_large_malloc() only wanted its "do not dump to core" property and
picked up large pages as an undocumented side effect; those buffers
are also small and sequentially accessed, so they would have gained
little from large pages anyway.
Add MY_TRY_LARGE_PAGES: my_large_malloc() and my_large_virtual_alloc()
now only attempt large pages when a caller passes this flag, instead
of always trying whenever the global option is set. Only the buffer
pool and the key caches pass it. The redo log buffer, tmp_buf, and
row0log.cc's crypt buffers no longer request large pages at all,
which removes the size-rounding bug for them without touching that
code.
my_large_virtual_alloc()'s fallback (no usable large page size) must
also return read-write memory right away, like the Windows large-pages
fallback already does, since my_virtual_mem_commit() is a no-op for
MY_TRY_LARGE_PAGES. my_large_pages_flag is now set once, in
my_init_large_pages(), and never changed thereafter, on any platform.
Both my_virtual_mem_commit() and my_virtual_mem_decommit() are now
no-ops, aside from accounting, whenever large pages are requested.
Also fix a broken mtr suppression regex in main.large_pages that
would fail the test on Windows.
Across a range of JSON functions taking a path argument
there are SQL expressions that dont' have a character
set. If these expressions don't have a character set
fall back to the character set of the argument of the
json function that represent the document being operated
on. If this doesn't have a character set fall back to
my_charset_utf8mb4_bin.
This covers the 11.4 JSON_KEY_VALUE function also as it
reuses the Json_path_extractor::extract method.
Add nonnull and warn_unused_result to the json path
functions to facilitate compiler and UBSAN catching of the
problem early.
As null values of s_p are incompatible with report_path_error,
jump directly to a null return which is consistent with
the defination of the JSON sql funciton.
Protect all READ_ONLY sysvars from run-time changes.
Put them into a separate section and use my_virtual_mem_protect()
to make this section read-only before the server starts accepting
connections.
Verify that they're all protected in the sys_var constructor.
One exception: opt_noacl (--skip-grant-tables) can be changed
from 1 to 0 on FLUSH PRIVILEGES. Let's briefly drop the
protection for this 1->0 change. It can be needed only once
in a server lifetime and only if it was started with --skip-grant-tables
On shutdown the protection is removed, making variables writable
again because shutdown resets some of them during the cleanup
Assisted-By: Claude:claude-4.8-opus
* it's called ROOT_FLAG_VMEM
* it does not imply any protection, ROOT_FLAG_MPROTECT was a misnomer
* it means that the memory is allocated using my_virtual_mem_commit,
not my_malloc, so not on heap
* memory allocated this way can be protected with my_virtual_mem_protect()
* thus root using ROOT_FLAG_VMEM can be protected with protect_root()
Assisted-By: Claude:claude-5-sonnet
make my_virtual_mem_reserve() work on Windows and on Linux
(falling back to my_large_virtual_alloc() - that's what InnoDB
did instead). This makes the usage pattern portable without #ifdef:
- reserve a memory range
- commit (allocate) memory from a reserved range
- decommit memory, keeping it reserved
- release reserved memory back to OS
invalid pointer pairs are when the length/memory of one string are
intermixed with another.
For comp_err, the end null pointer was compared to soffset within
my_strtoll10. As we didn't need the end position a NULL arg option
was compatble.
For uca-dump, Address Sanitizer raises invalid pointer pair because
argv options (opt) don't have an obvious correlation of having a start
at opt_X.length, even though the lstrncmp makes this true.
The DBUG_ASSERTS of strmov (added MDEV-11752) where incompatible with
pointer-compare. Replaced strmov with static inline version in
m_string.h using memmove that allows overlaps, and being inline allows
the uneeded parts of the implementation to be optimized away.
* restore result=NULL after free in mysqlshow.c (to prevent double-free)
* remove DBUG_EVALUATE_IF again
* remove useless 'stop slave' from a test
* restore charset encoding in mysqltest.{test,result}
* correct error for unparseable BINLOG data is ER_SYNTAX_ERROR now
* restore old libfmt on rhel-7, gcc 4.8.5 cannot compile anything newer
With proxy protocol thd_set_peer_addr() runs twice (proxy host, then the
real client from the proxy header). Connect errors are accounted against
the real client, but check_connection() incorrectly uses condition on the
proxy host's count, rather than real client's address.
Fix: reset both the proxy host, and real client's connect errors on
successful connection.
Added tests for incomplete handshake, and reset behavior,
under proxy protocol, for both real client errors, and proxy host errors.
a90779098b changed the
interface to the json_valid to take a json_engine_t
so that it could be interupted and so that warnings
could become visible.
On merge from that 10.11 to 11.4 it was discovered
that the ColumnStore engine uses this interface.
Restored the original json_valid function exactly
how ColumnStore expects this and added renamed
the function with json_engine_t to be json_valid_engine.
This follows the same convention as in the commit
756fc6fd9a that was
for json_normalize, that ColumnStore also used.
14c16e02b2 changed the
interface for json_normalize to take a json_engine_t so that
there could be some error checking.
Unfortunately ColumnStore depended on this interface.
Restored original interface of json_normalize() and
added a json_normalize_engine() that take as engine as the
argument.
Or max_statement_time limit.
Change the json_valid to take a json_engine_t argument.
Adjust the Item_json_valid to have a json_engine_t and
to report an error.
Note this means that invalid json syntax now has a
note on any json error. Raising this to a warning can
cause check constraints of JSON_VALID() to result
in ER_JSON_SYNTAX rather than ER_CONSTRAINT_FAILED
(valided in main.type_json test).
Consequential changes in debug build to json acl
testing validation and unit test having a dummy json_engine_t.
json_normalize also uses the same engine that was
passed as its argument to test json_valid. Because
the character set may have changed, if there's an error
adjust the je->s.c_str pointer. Its only the offset that
is currently used in the error messages anyway so there
may be offset errors depending on the original character
set.
Or max_statement_time limit.
JSON_EQUAL/JSON_NORMALIZE also didn't report warnings
on invalid JSON inputs.
Change the json_normalize function to take a json_engine_t
as a argument and implement the sql/item_jsonfunc.cc calls
to have a json_engine_t.
To the JSON_EQUALS and JSON_NORMALIZE, add error handling
to produce an error when there was one. compare_nested_object,
part of JSON_OVERLAPS, uses a new json_engine and
copies the error/position to the returning function.
json_normalize expects the incoming je structure to have
a valid, or nullptr, killed_ptr, as json_start will reset this.
Or max_statement_time limit or MAX_DEPTH.
Correct interuptability and depth checking of:
* JSON_CONTAINS
* JSON_CONTAINS_PATH
* JSON_EXISTS
* JSON_EXTRACT
* JSON_KEYS
* ST_GeomFromGeoJSON
Populate in the json library the killed as a specific
error message. As the json library has a killed_ptr
is approprate it handles it.
As json_report_error now processes the killed there's
no need of thd->check_killed().
Verified that the path after every json_scan_next, the
location of checking the killed_ptr, reports errors
correctly.
Backport to 10.6 without JSON_OVERLAPS
The json_engine killed_ptr was of type uchar however the enum
in the server is dependant on the architecture. On big-endian
architectures like IBM Z, retreiving a uchar* retreived the
unmodifed part of the THD->kill enum location and was always 0.
As C++11 allows enums to inherit a type, used uint32_t as the base
class of killed_state type in the server and used uint32_t in the
json library.
reload_acl_and_cache required an expression change to avoid
the compile error:
sql/sql_reload.cc:472:24: error: enumerated and non-enumerated
type in conditional expression [-Werror=enum-conversion]
472 | return result || (thd ? thd->killed : 0);
Added the kill_ptr assignment the following functions can be
killed:
* Item_func_json_depth::val_int
* Item_func_json_type::val_str
* Item_func_json_length::val_int
Item_func_json_array_append::val_str, check_killed() only applied
to json_error: label and not return_null.
Item_func_json_format::val_str(), corresponding to the SQL,
json_compact, json_detailed, json_loose - add debug instrumentation
for the func_json_notembedded test.
MDEV-26726 (fcd345de48) added a json_pause_execution sync point
however the test case used debug_max_statement_time, which doesn't
exist. Fix the sync point name in the test.
Clang upstream bug https://github.com/llvm/llvm-project/issues/173210
shows that a "int x=x" construct will in the intermediate representation
have code to read from x. With this generated sanitizer=undefined and
static analyzers will all see the uninitialized read and write.
Because clang has a stronger implementation of following paths to
uninitialized variables, "#define UNINIT_VAR(x) x" is the which
is what our release binaries use is the ideal path for this compiler.
Clang based compilers with error during compilation if any
uninitialized behaviour is detected at compile time because of
0c80ddb519.
Corrects MDEV-36542 - 6fd57f478f.
scripts/fill_help_tables.sql from
https://github.com/mariadb-corporation/mariadb-docs/ as of
3fb208fcdfed06beb52ef9e305017de2387ff012
Other URL changes are taken from the HTTP redirection URL
from the old KB.
Some test cases where impacted:
* ps_missed_cmds{,_bin_prot} - used a topic_id of 0 that is
now used, used MAX_INT instead.
* range_notembedded used the help data as a test case. As its not
a stable table data, replaces with static data. Test case valid
by ensuring the ANALYZE EXPLAIN and optimizer trace had the same
structure. Also valided the MDEV-24739/MDEV-25069 bug fixes from
where the test case originaled, when reverted, exhibited the
previous behavour before the bug fixes.
The json_engine killed_ptr was of type uchar however the enum
in the server is dependant on the architecture. On big-endian
architectures like IBM Z, retreiving a uchar* retreived the
unmodifed part of the THD->kill enum location and was always 0.
As C++11 allows enums to inherit a type, used uint32_t as the base
class of killed_state type in the server and used uint32_t in the
json library.
reload_acl_and_cache required an expression change to avoid
the compile error:
sql/sql_reload.cc:472:24: error: enumerated and non-enumerated
type in conditional expression [-Werror=enum-conversion]
472 | return result || (thd ? thd->killed : 0);
Added the kill_ptr assignment the following functions can be
killed:
* Item_func_json_depth::val_int
* Item_func_json_type::val_str
* Item_func_json_length::val_int
Item_func_json_array_append::val_str, check_killed() only applied
to json_error: label and not return_null.
Item_func_json_overlaps::val_bool ensure check_killed() is called.
Item_func_json_format::val_str(), corresponding to the SQL,
json_compact, json_detailed, json_loose - add debug instrumentation
for the func_json_notembedded test.
MDEV-26726 (fcd345de48) added a json_pause_execution sync point
however the test case used debug_max_statement_time, which doesn't
exist. Fix the sync point name in the test.
..passed as argument 2, which is declared to never be null
(dynstr_append_mem).
dynstr_append_mem, uses memcpy to append to the string which in the glibc
library is declared to not take null string as the src. Defensively we
declare dynstr_append_mem to not take a null argument too.
mariadb-test, to prevent it calling dynstr_append_mem with a null
mtr variable value, one that hasn't been initialized, we return an
error if the value is null. The result of this is the $6 in the test
case is never assigned a value and would error.
With this resolved, strchr, if the string isn't found, a null pointer is
returned. UBSAN will complain both about incrementing a pointer beyond
the boundary, and also incrementing if it is null.
Remove the unscoped #pragma GCC diagnostic ignored
"-Wdeprecated-declarations" from include/violite.h and replace all
sprintf/vsprintf calls with snprintf/vsnprintf across the codebase.
Where possible, pass actual buffer sizes as function parameters
instead of using hardcoded constants or recomputing strlen().
Notable API changes:
- get_date(): added to_len parameter for buffer size
- nice_time(), end_timer() in client/mysql.cc: added buff_size
- generate_new_name(): added name_size parameter
- calc_md5(): added buffer_size parameter (fixes off-by-one)
- MD5_HASH_TO_STRING macro: added _size parameter
- Added MYSQL_UDF_MAX_RESULT_LENGTH constant in mysql_com.h
Also fixed a pre-existing bug in sql/rpl_mi.cc where the ellipsis
"..." was written to the wrong buffer (dbuff instead of buff).
Vendored code is excluded: extra/readline, extra/wolfssl,
wsrep-lib, libmariadb, storage/mroonga/vendor, zlib.
Replaced macros with optimized inline functions, that use memcpy
instead of unaligned access (translates to single instruction
on architectures we support). Used compiler byteswap intrinsics
where appropriate.
Removed byte_order_generic.h as well as Intel specific headers
byte_order_generic_{x86|x86_64}.h
Added some casts to places where intNstore writes into "const char*".
my_large_virtual_alloc(): If large page allocation fails, clear
my_use_large_pages. Outside large page allocation (and IBM AIX),
invoke mmap(2) with PROT_NONE so that no page table entries will
have to be allocated. Also, fix the error reporting.
my_virtual_mem_commit(): Do not set the Linux specific flag MAP_POPULATE
because it would introduce a performance regression on startup.
There is a better, system-wide way to disable the overcommit:
sudo sysctl vm.overcommit_memory=2
HAVE_UNACCESSIBLE_AFTER_MEM_DECOMMIT: Remove. We will always make
the virtual memory unaccessible, so that the operating system
can deallocate the page table entries.
buf_pool_t::create(): Skip the my_virtual_mem_commit() on AIX only.
buf_pool_t::page_guess(): Unconditionally check if the pointer is
within the currently allocated buffer pool.
innodb_buffer_pool_size_max: On 64-bit systems other than IBM AIX,
we will default to 8 TiB.
On 32-bit systems, we retain the default innodb_buffer_pool_size_max=0,
that is, by default, SET GLOBAL innodb_buffer_pool_size=... will be
unable to increase from the initial value.
There is no functional change to the startup logic that adjusts
innodb_buffer_pool_size_max to be at least innodb_buffer_pool_size
rounded up to the extent size (2 MiB on 32-bit, 8 MiB on 64-bit).
Problem:
=========
An assertion failure occurs in InnoDB during consecutive
ALTER TABLE operations using the COPY algorithm. The crash
happens during the table rename phase because the
source table (dict_table_t::n_ref_count) is non-zero, despite
the thread holding an exclusive Metadata Lock (MDL).
Reason:
========
When ALTER IGNORE TABLE is executed via the COPY algorithm,
it generates undo logs for every row inserted into the
intermediate table (e.g., #sql-alter-...). The background Purge
Thread, responsible for cleaning up these undo logs, attempts
to take an MDL on the table to prevent the table from
being dropped while in use.
Race condition:
==================
First ALTER: Creates #sql-alter-, copies data, and renames it to t1.
Purge Activation: The Purge thread picks up the undo logs from step 1.
It takes an MDL on the temporary name (#sql-alter-) and increments
the table's n_ref_count.
Identity Shift: InnoDB renames the physical table object to t1, but
the Purge thread still holds a reference to this object.
Second ALTER: Starts a new copy process. When it attempts to rename
the "new" t1 to a backup name, it checks if n_ref_count == 0.
Because the Purge thread is still "pinning" the object to
clean up logs from the first ALTER, the count is > 0,
triggering the assertion failure.
Solution:
========
ALTER IGNORE TABLE needs row-level undo logging to easily
roll back the last inserted row in case of duplicate key errors.
By discarding the last undo log record after inserting each row,
purge will not process any log records generated by
ALTER IGNORE TABLE, preventing unexpected access from the purge
subsystem during subsequent DDL operations.
Make skip_alter_undo (1-bit) to (2-bit enum)
to support different ALTER operation modes:
- NO_UNDO (0): Normal mode with standard undo logging
- SKIP_UNDO (1): ALTER mode that skips undo logging
- IGNORE_UNDO (2): ALTER IGNORE mode that rewrites undo blocks
trx_undo_report_row_operation(): Add ALTER IGNORE undo
rewriting logic. Store old undo record info before writing
new records for IGNORE_UNDO mode. Reset undo top_offset
to maintain only latest insert undo
row_ins_clust_index_entry_low(): Prevent ALTER IGNORE from
using bulk insert optimization. since it requires to write
undo log for each row operation that's incompatible with
existing bulk operation.
On IA-32 and AMD64 (x86-64) CPU implementations, the latency of the
PAUSE instruction (REPZ NOP that was repurposed from a no-operation
on the Intel Pentium 4) can vary a lot, and we may adjust the
my_cpu_relax_multiplier accordingly. Let us display the value of
that variable on startup if --log-warnings=3 or larger was specified.
Microsoft Windows is available for multiple instruction set architectures.
Therefore, we had better not assume that defined(_WIN32) implies the
availability of a PAUSE instruction.
Reviewed by: Sergey Vojtovich
The backup of encrypted Aria tables was not supported.
Added support for this. One complication is that the page checksum is
for the not encrypted page. To be able to verify the checksum I have to
temporarly decrypt the page.
In the backup we store the encrypted pages.
Other things:
- Fixed some (not critical) memory leaks in mariabackup
Clang upstream bug https://github.com/llvm/llvm-project/issues/173210
shows that a "int x=x" construct will in the intermediate representation
have code to read from x. With this generated sanitizer=undefined and
static analyzers will all see the uninitialized read and write.
Because clang has a stronger implementation of following paths to
uninitialized variables, "#define UNINIT_VAR(x) x" is the which
is what our release binaries use is the ideal path for this compiler.
Clang based compilers with error during compilation if any
uninitialized behaviour is detected at compile time because of
0c80ddb519.
Corrects MDEV-36542 - 6fd57f478f.
Fixed by setting server_status SERVER_STATUS_RETURNED_ROW if send_data
is called for stored procedures.
This make the definition of Empty_queries well defined:
"Empty_queries" is the number of SELECT queries that returns 0 rows.
mtr.commit_file() call in fil_space_t::drop() removes space from
fil_system.named_spaces, but then the space can be inserted in the
container again by some another thread while fil_space_t::drop() is
waiting for pending operations finishing.
The fix is to check and remove a space from fil_system.named_spaces
after all pengind operations on the space are finished. Also the ut_d()
macro is removed for space->max_lsn=0 assignments to avoid repeated
space removing from fil_system.named_spaces.
There is error in ilist::pop_back(). ilist::end() returns sentinel,
and the pop_back() removes sentinel from the list instead of the last
element. The error is fixed in this commit.
Reviewed by Marko Mäkelä
These changes was done as part of fixing
MDEV-36858 MariaDB MyISAM secondary indexes silently break for
tables > 10B rows
Changes done in myisamchk:
- Tables that are checked are opened in readonly mode if --force is not
used.
- *.MYD files will be opened in readonly mode for repair if --quick
is used.
- Added information about check progress if --verbose is used.
- Output information about repaired/checked rows every 10000 rows instead
of every 1000 rows. Note that this also affects aria_chk
- Store open file mode in share->index_mode and share->data_mode instead
of in share->mode.
- Added new option --keys-active= as a simpler version of keys-used.
- Changed output for "myisamchk -dvv" to get nicer output for tables
with 10 billion rows.
This was generally good to get done but also needed to be able to run
mariabackup test under asan.
Things freed:
- Allocated variables (mysql_tmpdir_list, opt_passwd etc)
- InnoDB variables
- Results from SQL queries (A lot of sql queries did not free their result)
- Allocated sys_vars
- Server variables (mysql_server_end())
- Memory allocated by plugins (encryption)
- Free variables allocated by my_default. (Old code had a bug that caused
these to not be freed)
Other things:
- Moved freeing of mysql_tmpdir_list to main, as the old code did not
free the last mysqltmp_dir allocation. Now we also initialize the
variable only once.
- Fixed a serious, potentially 'crashing at end' bug where we called
free_defaults() with wrong pointers.
- Fixed a bug related to update_malloc_size() where we did not take
into account the it was not changed.
- Fixed a bug in Sys_var_charptr_base where we did not allocate
default values. This could lead to trying to free not allocated values
in xtrabackup.
- Added sf_have_memory_leak() to be able to easily check if there was
a memory leak when using safemalloc()
- sf_report_leaked_memory() now returns 1 if a memory leak was found.
Clang processes the "int x=x" code from UNINIT_VAR
literally resulting in an uninitialized read and write.
This is something we want to avoid. Gcc does the same
without emitting warnings.
As the UNINIT_VAR was around avoiding compiler false detection,
and clang doesn't false detect, is default action is a
noop.
Static analysers (examined Infer and SonarQube) are
clang based and have the same detection.
Using a __clang__ instead of WITH_UBSAN would acheived
a better result, however reviewer wanted to keep WITH_UBSAN
only.
LINT_INIT_STRUCT is no longer required, even a gcc-4.8.5
doesn't warn with this construct removed which matches
the comment that it was fixed in gcc ~4.7.
mysql.cc - all paths in com_go populate buff before use.
json: Item_func_json_merge::val_str
LINT_INIT(js2) unneeded as usage in the previous statements
it is explicitly initialized to NULL.
Item_func_json_contains_path::val_bool n_found is guarded
by an uninitialized read by mode_one and from
gcc-13.3.0 in Ubuntu 24.04 this is detected. As the only
remaining use of LINIT_INIT this usage has been applied
with the expanded macro with the unused _lint define removed.
The LINT_INIT macro is removed.
_ma_ck_delete - org_key only valid under share->now_transactional
likewise with _ma_ck_write_btree_with_log
connect engine never used anything that FORCE_INIT_OF_VARS
would change.
Reviewer: Monty
