Item::shallow_copy() was meant to create a copy of the same class as the
item it is called on, but nothing enforced that. A class that does not
declare its own override inherits the one of its nearest concrete parent,
and get_item_copy<Parent>() then silently creates an object of the parent
class, losing the child's virtual behaviour (see MDEV-38801 for an earlier
instance with Item_cache_year).
Add the ITEM_SHALLOW_COPY(CLASS) macro, which declares the override, and
Item::dbug_check_shallow_copy_type(), a debug-only check called from
Query_arena::free_items() for every item freed. Once per dynamic type it
creates a shallow copy and asserts that it has the same typeid as the
original. Then, the copy is destroyed, which provides additional
coverage.
The check found these classes copied as their parent class, now fixed with
ITEM_SHALLOW_COPY():
Item_join_operator_plus, Item_null_result, Item_old_field, Item_datetime,
Item_empty_string, Item_return_int, Item_trigger_type_of_statement,
in_string::Item_string_for_in_vector, Item_is_not_null_test,
Item_interval_DDhhmmssff_typecast, Item_string_xml_non_const,
Item_func_trt_trx_sees_eq, Item_field_refcursor (type_cursor plugin),
Item_field_packable_scalar (type_assoc_array plugin).
These two classes do not support copying so the patch makes their
shallow_copy() to return NULL:
Item_cache_wrapper, Item_sum_sp
Item_field_packable_row::shallow_copy() has assert(0). The patch makes
it return NULL.
This commit was prepared with help of Claude Code (Opus 5).
Sergei Petrunia added "delete item_copy" and did other minor edits.
Do not use the fast path when the statement uses a stored routine.
Evaluating that function during fill changes THD open-table state and
trips restore_backup_open_tables_state().
Do not use the fast path when the query has a subquery. Evaluating
that predicate while get_all_tables() uses a temporary MEM_ROOT
freed the subquery JOIN before outer JOIN cleanup.
Keep the I_S.TABLES fast path only for plain client SELECTs. Fall back
when the result must go through a file, ANALYZE, SQL_BUFFER_RESULT,
PROCEDURE ANALYSE(), or the cursor protocol.
Stop the I_S.TABLES fast path after a matching row is sent, not after
the first scanned table. Honour LIMIT / @@sql_select_limit via
plan->max_rows so filtered rows do not count.
For simple single-schema SELECT queries on information_schema.tables,
bypass ha_write_tmp_row() and stream rows directly via the protocol.
Add EXPLAIN is_fast_path visibility and MTR coverage, including Windows
EXPLAIN casing normalization.
When CHANGE MASTER sets master_heartbeat_period to a value greater than
@@GLOBAL.slave_net_timeout, a ER_SLAVE_HEARTBEAT_VALUE_OUT_OF_RANGE_MAX
warning is emitted. However, the reverse scenario, lowering
slave_net_timeout below an existing master_heartbeat_period, produced no
warning.
Add an on_check callback for slave_net_timeout that iterates all
Master_info entries and warns if any heartbeat period exceeds the
requested timeout. Connections that are not configured are skipped. A
DEFAULT heartbeat period is skipped unless --master-heartbeat-period was
given at startup: without it the period follows slave_net_timeout/2 and
can never exceed the timeout, but with it the period is a fixed value
and is checked like an explicit one.
The warning is reported with a new, generalized error
ER_OPERATION_INVALIDATES_CONFIGURED_SLAVE, which names the affected
replication connection and carries a parameterized description of the
misconfiguration. Naming the connection matters for multi-source: when
several connections exceed the new timeout, the user can see exactly
which ones are problematic. One warning is emitted per affected
connection.
Add a multi_source test, run with --master-heartbeat-period=30, that
configures two named connections with explicit heartbeat periods above
the timeout and a third left at DEFAULT, and verifies a distinct warning
is emitted for each.
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.
CREATE USER u@'192.168.0.0/24'; -- same as '192.168.0.0/255.255.255.0'
CIDR is an input spelling only, normalised to the netmask form while parsing.
A malformed mask is now rejected instead of stored, and one already in
the privilege tables is skipped at load with a warning instead of being honoured.
Nothing rejected these before, and they are not merely
useless: a non-contiguous mask such as 10.0.0.0/255.0.255.0 matches the
scattered set 10.*0.*. Skipping rather than erroring keeps such a row droppable and renamable, except for a literal CIDR host stored by an older version (for example 10.0.0.0/24): the parser converts that name, so it can no longer be named and must be removed with a direct DELETE from the privilege tables.
New error ER_INVALID_HOST_NETMASK, ER_ACL_ENTRY_IGNORED_BAD_HOST_MASK
Replaces std::function / std::optional / std::unordered_map /
std::unordered_set / std::string_view and template-heavy code in the
master.info / relay-log.info persistence layer with plain MariaDB
idioms.
- On-disk file format unchanged.
- SQL syntrax unchanged
- No notable test changes, except those for which new error messages was
added and output from SHOW VARIABLES.
- master_* CLI options moved from inline globals in
rpl_master_info_file.h to Sys_var_* in sys_vars.cc (READ_ONLY,
AUTO_SET on use_gtid and heartbeat_period). master_heartbeat_period
is Sys_var_double (seconds); master_use_gtid is Sys_var_enum. Global
storage sits outside HAVE_REPLICATION so libmariadbd links.
- This adds the rpl_master_info variables to SHOW VARIABLES
- Value classes are no longer templates. Persistent renamed to Value;
Optional_* classes renamed (Uint_value_with_default, Path_value,
etc.). has_value flag in the base; is_default()
non-virtual. Multi-line method bodies moved to new rpl_info_file.cc
and rpl_master_info_file.cc.
- VALUE_LIST (Mem_fn array) and VALUE_MAP (std::unordered_map)
replaced by per-instance Value *const value_list[] / value_map[]
arrays + a static TYPELIB populated at first construction from each
Value's variable_name. seen-set dedup is now a uint32 bitmask.
- Default handling is not anymore done with using special values.
Setting a value to default is now done by calling set_default(),
which sets the has_value member to 0 and stores the default value
in the variable.
When reading a value, one does not have to check if the value is
a default value.
This makes the interface cleaner and easier to manage.
- CHANGE MASTER TO parser no longer builds std::function lambdas:
LEX_MASTER_INFO carries mi_used_options / mi_used_default bitmasks
plus plain value fields; change_master() applies them via bit-test +
direct copy or set_default(). std::optional<T> and trilean are gone
from the value class API.
- Heartbeat_period_value::load_from uses my_strtod() instead of
my_decimal for the on-disk format. from_decimal kept for CHANGE
MASTER.
- RocksDB #if __cplusplus < 201703L guard at the top of
rpl_master_info_file.h dropped; header now compiles in C++11.
cstdint dropped; uint32_t / UINT32_MAX in touched code replaced by
MariaDB's uint32 / UINT_MAX32. include/my_global.h gains
INT_BUFFER_SIZE next to LONGLONG_BUFFER_SIZE.
- Pointer-style throughout (T * not T &). Multi-line inline functions
used in more than one place moved to .cc files per style.
New features:
- SHOW VARIABLES now shows the master default values.
- If there is an wrong value for a row in a master.info file the
corresponding variable name and the value is now written to the
error log.
Bug fixes uncovered by the refactor:
- master_ssl_verify_server_cert had a pre-existing inconsistency with
security implications: the documented default and the Sys_var_mybool
default are TRUE (validate the master's certificate), but the OLD
init_ssl_config() set the per-Master_info field to FALSE on RESET
SLAVE ALL. Concretely: after RESET SLAVE ALL followed by a bare
CHANGE MASTER TO (no explicit MASTER_SSL_VERIFY_SERVER_CERT), the
slave would connect with MASTER_SSL=1 enabled but with certificate
validation silently disabled.
Fixed by not resetting master_ssl_verify_server_cert in
RESET SLAVE ALL;
- As some replication setups sets master_ssl_verify_server_cert= 0
while other does not do that, the expected value used in
check-testcase.inc "Master_SSL_Verify_Server_Cert No" does not make
sense. The value of this variable is now ignored by
check-testcase.inc
- The old code would fail to read a master info file if there was a new
variable longer than 23 characters. This is now extended to 80.
- Removed warning on RESET SLAVE that "RESET SLAVE is implicitly changing"
the value of 'Using_Gtid' from 'No' to 'Slave_Pos'" as this change did
not really happen. This error was given if the master could support
GTID but the user had specified directly or indirectly to not use GTID.
Other things
- Size of mariadb (optimize build) shrunk with 1,408,009 bytes
(Based on 'dec' from the size command)
- Size of code in sql directory did not change notable.
- Reworked Master_use_gtid_value to remember users last choice and use
this as default when connecting to slave.
Replace mi->using_gtid, with was a pointer to master_use_gtid, with
two functions:
- using_gtid() returns the current mode, for the code that needs the
value itself (SHOW SLAVE STATUS and performance schema output,
comparisons against CURRENT_POS and SLAVE_POS).
- is_gtid_supported() tells whether the connection uses gtid at all.
The many callers that compared the mode against USE_GTID_NO with
== or != now call this function instead of testing the enum value.
- Master_use_gtid_value::supported() did the same test as
is_gtid_supported() and is removed
- Removed some operators in rpl_master_info_file.* that hid
that functions was used to read and update values.
Behavior changes:
- The last used value in CHANGE MASTER for MASTER_USE_GTID will be
remembered for future CHANGE MASTER for the same connection. This is
true even if RESET SLAVE is called.
- If Using git is not used with change master but log_file_name, pos,
relay_log_name or relay_log_pos is specified this is treated as
if Using_git=NO would have been given. (Old behavior, but not
documented).
--auto-master-use-gtid does nothing. The default value of
master_use_gtid is by default SlavePos that will automatically
change to NO if the master does not support GTID. This already
handles what auto-master-use-gtid tries to do.
- CHANGE MASTER's MASTER_HEARTBEAT_PERIOD now works like other MASTER
options: When set to a value, this value is kept until value is
changed or set to DEFAULT. The DEFAULT is
@global.master_heartbeat_period if this is set as startup, if not
then @global.slave_net_timeout/2 is used.
The test for:
DBUG_ASSERT(thd->lex->query_tables != table_list);
does not make any sense:
- table_list is a pointer we just got from malloc)()
- thd->lex->query_tables can point to an existing TABLE_LIST
or is a pointer to already freed memory.
In 'main' I got the above assert multiple times triggered when
running rpl.rpl_row_img_blobs on a valgrind build. There was
nothing wrong with the allocations, just random change have
malloc return a recently freed pointer.
MariaDB TIME is a duration up to 838:59:59, but SELECT converted DuckDB
TIME through ToString(), which asserts clock-time validity in debug
builds. Convert microseconds to MYSQL_TIME directly instead.
SQL DML quoted TIME strings, so '25:00:00' failed conversion. Store
microseconds through mariadb_time() instead, matching the appender path.
Buffered_logs::flush() prints and frees the buffered early-option
messages. Call it on every early exit, so warnings and errors are
not lost and the buffer is not leaked.
Make the early-option error exit in mysqld_main() unconditional.
Without the performance schema, a failed early option parse left
"----file-marker----" in argv, and it was reported as an unknown
option.
Index.xml parse errors go through error_handler_hook, not
vprint_msg_to_log(), so --validate-config reported
"Configuration is valid." Install a hook that sets
validate_config_has_warnings.
Test cleanup.
Add a new --validate-config command-line option that validates the server
configuration (from config files and command line) and exits with 0 on
success or non-zero on failure, without actually starting the server.
This is useful for DBAs to verify configuration changes before restarting:
mariadbd --defaults-file=/etc/my.cnf --validate-config
The validation checks the following:
- Config files explicitly specified via --defaults-file or --defaults-extra-file
must exist and be readable
- There are no unknown options specified (in config files or on the command line)
- Option values are type-checked (e.g. numeric options reject non-numeric input).
Out-of-range values are adjusted to valid limits with a warning, not rejected.
Output: [Note] messages are suppressed during validation (as with
--silent-startup), so a clean configuration prints only
Configuration is valid.
If any warning was printed while checking the configuration, the verdict
is instead "Configuration is valid, but warnings were reported; see
above." The verdict is written directly to stdout rather than as a [Note],
so neither the note suppression nor --silent-startup can hide it.
Warnings and errors are still printed.
Exit codes:
0 - Configuration is valid
2 - Unknown option encountered
7 - Unknown variable encountered
Other non-zero codes are possible for other parse errors (e.g. invalid enum
value, numeric parse error). The option description now correctly states
"non-zero for failure" rather than "1 for failure".
The implementation reuses the existing --help code path (opt_abort), which
already loads plugins to validate their variables and runs a final parsing
pass with skip_unknown=0 to detect unknown options. The only difference is
that --validate-config suppresses help output.
--help and --version take precedence: when either is present alongside
--validate-config, the server prints help/version and exits cleanly without
performing config validation, also with --verbose.
Warning detection:
validate_config_has_warnings is set in one place, vprint_msg_to_log() in
log.cc, which every printed error-log message goes through
(sql_print_warning(), option_error_reporter(), buffered early-option
warnings, the charset reporter). Warnings that are not printed are not
counted: option_error_reporter() drops --loose-* option warnings under
--bootstrap before reaching vprint_msg_to_log(), so the --loose-* options
in mtr's $MYSQLD_BOOTSTRAP_CMD do not turn the verdict into the
"warnings were reported" variant.
Warnings for options parsed in handle_early_options() (e.g.
max_connections) used to be buffered and printed through the error log
only in builds with the performance schema; otherwise mysys printed them
straight to stderr, bypassing the error log. The buffering is now done in
all builds, so these warnings are printed through the error log and
counted like any other.
Bug fixes included in this patch:
- --validate-config --log-bin no longer creates a spurious <host>-bin.index
file in the datadir. The binlog index open is now guarded by !opt_abort
(was: !opt_help), skipping it for all early-exit paths including --help
--verbose.
- Simplified early-exit condition in init_common_variables() from a
four-variable expression to (opt_help || opt_version) && !opt_verbose,
which is equivalent and clearer (vuvova review).
Note: this is intentionally named --validate-config (not --validate-defaults)
for MySQL compatibility, although --validate-defaults would be more consistent
with MariaDB's existing defaults-related options (--defaults-file,
--no-defaults, my_print_defaults, etc.).
Tests: the runs that expect the clean verdict pass options that keep the
server from printing environment-dependent warnings (open files ulimit,
case-insensitive file system), since any printed warning now correctly
produces the "warnings were reported" verdict.
dbug_print_sel_arg() printed the key values of a SEL_ARG over a SPATIAL
index through Field::val_str(). The key of such a SEL_ARG holds the MBR
(Minimum Bounding Rectangle) of the value, four doubles, not the
geometry itself, and the SEL_ARG has no min/max bounds: the spatial
relation is carried in min_flag as GEOM_FLAG | HA_READ_MBR_xxx. So the
output was garbage bytes between "-inf"/"+inf" style bounds.
Detect GEOM_FLAG and print the SEL_ARG the same way print_range() does
for the optimizer trace: "col mbr_relation POLYGON(...)", using
print_mbr_range_operator() and Field_geom::print_key_part_value() with
image_type itMBR.
Ranges over geometry columns in non-SPATIAL indexes (itRAW) still go
through the old path.
Ranges built over GIS (geometry) columns could not be printed in the
optimizer trace or the recorded optimizer context: Field_geom printed
every key value as the placeholder "unprintable_geometry_value",
regardless of whether the index stored the column's raw value (or a
prefix of it) or, for a SPATIAL index, its MBR (Minimum Bounding
Rectangle).
Field::print_key_part_value() now takes an image_type argument (see
Field::image_type()) that says which of the two the key holds. For a
SPATIAL index (image_type itMBR), Field_geom::print_key_part_value()
decodes the four doubles the key stores and prints them as a WKT
POLYGON. For every other index (image_type itRAW), the key holds the
value or a prefix of it, and it now prints in binary form, the same
way Field_blob does, instead of the placeholder.
print_mbr_range_operator() prints the spatial relation a GEOM range
carries (MBRWITHIN, MBRCONTAINS, MBRINTERSECTS, MBRDISJOINT,
MBREQUALS), inverted where needed so the indexed column reads on the
left. print_range() and print_key_value() thread the new image_type
argument through to Field_geom::print_key_part_value().
Added new tests in opt_trace, opt_context_store_stats,
opt_context_replay_basic
I_S.OPTIMIZER_CONTEXT has this statement:
set @opt_context='CONTEXT'; -- denotes CONTEXT-STMT
And opt_context_save_in_var.inc just took the CONTEXT and put it
in a variable:
SET @opt_context= (select REGEXP_SUBSTR(...) from i_s.optimizer_context);
This ignored the fact that the "CONTEXT" is escaped for the SQL parser.
Fix this by extracting the entire CONTEXT-STMT and running it.
Problem:
A sequence referenced only in a column's DEFAULT expression (e.g.
"a INT DEFAULT NEXTVAL(s1)") is opened only when a statement
evaluates DEFAULT values (INSERT, LOAD DATA, etc). A plain SELECT
never opens it, so the sequence never appears in
thd->lex->query_tables, and Optimizer_context_recorder::
dump_sql_script() had no way to see it. The dependent table's
definition was then captured without the sequence it depends on,
making the captured context unusable on replay.
Fix:
TABLE::internal_tables lists these sequences. dump_sql_script() walks
it for each dumped table and calls the new dump_sequence_placeholder()
for each entry, before the table's CREATE TABLE so that replay can
recreate both in order. The sequence is not opened; only its database
and name are used, so for now it is captured as a placeholder
"CREATE SEQUENCE IF NOT EXISTS db.name" without its parameters or
current value. The name is qualified with the sequence's own database,
so a sequence in a different database than the table is recreated in
that database.
Such a placeholder is also remembered in a separate hash. If the query
uses the same sequence directly as well, the main loop of
dump_sql_script() reaches it later, finds the placeholder, emits
"DROP SEQUENCE IF EXISTS db.name", and then dumps the sequence's real
definition and current value (SETVAL) as for any sequence used in the
query, regardless of the order the tables are listed in.
To share code with the existing dump loop, the SETVAL logic of the loop
is factored out into dump_sequence_current_value().
Even for the scenarios where sequence is dropped after table creation,
a name is replaced by a view, or a user doesn't privilege on the sequence;
The name is still captured as a sequence.
Tests (opt_context_store_ddls.test): INSERT-opened sequence; plain
SELECT; LOCK TABLES; a user without privilege on the sequence; two
sequences in one table; dropped sequence; stored function in the
query; name replaced by a view; re-execution as a prepared statement;
sequence in a different database; sequence used both directly and as a
DEFAULT dependency, in both orders.
The buf_pool.LRU list needs to reasonably accurately reflect recently
accessed blocks, so that they will not be evicted prematurely.
Because the list is protected by buf_pool.mutex, it is not a good idea
to maintain the position on every page access.
Instead of maintaining the LRU position on each access, we will
decide on each access whether the block qualifies for promotion, and
record the decision in a "promote" flag in the block descriptor. The
buf_flush_page_cleaner() thread as well as some traversal of the
buf_pool.LRU list, all of which hold buf_pool.mutex anyway, will move
the flagged blocks to the "recently used" end of buf_pool.LRU.
A block qualifies only if it is accessed again at least
innodb_old_blocks_time after the reference point, which is its first
access and, after that, its most recent promotion. The age is measured
in whole seconds and checked on the access, not when a sweep reaches
the block, so that a block that a table scan accessed in one burst
will not be promoted, however long it stays in buf_pool.LRU_old. After
a promotion, the age is measured from the time the sweep moved the
block, which can be later than the access that qualified it. The rule
applies at any position in buf_pool.LRU, and the flag stays set until
a sweep moves the block. Thus, a block that qualified outside
buf_pool.LRU_old keeps the flag until a sweep reaches it, which is
usually after it has moved into buf_pool.LRU_old; before, such a
block was moved on any access when freed_page_clock showed that it was
no longer close to the "recently used" end.
buf_page_make_young_if_needed(), buf_page_make_young(),
buf_page_peek_if_too_old(), buf_page_peek_if_young(),
btr_cur_nonleaf_make_young(), buf_page_t::set_accessed(): Replaced by
buf_page_t::touch(), buf_page_t::touch_no_stamp() and
buf_page_t::make_young_if_needed().
buf_pool_t::freed_page_clock, buf_page_t::freed_page_clock: Remove.
This is no longer meaningful in the revised design.
INFORMATION_SCHEMA.INNODB_BUFFER_PAGE(_LRU).FREE_PAGE_CLOCK now
always reports 0. Before the first eviction, buf_LRU_stat_update() now
records statistics intervals, and buf_LRU_evict_from_unzip_LRU() uses
its formula instead of assuming a disk-bound workload.
page_zip_des_t::state: An atomic 16-bit field that will include
the PROMOTE and OLD flags that would more logically belong
to buf_page_t. We maintain them here (along with some
ROW_FORMAT=COMPRESSED specific state that is protected by page
latches) in order to avoid race conditions and unnecessary
memory overhead. m_end moves out to its own field, shrinking the
packed word to 16 bits; n_blobs narrows from 12 bits to 10 to make
room for PROMOTE and OLD, still comfortably above the 744-column
maximum.
The helpers that set or clear one bit of page_zip_des_t::state always
perform the atomic read-modify-write, which is a full barrier on x86.
A caller that can find the bit already in the target state checks it
first: touch_no_stamp() for PROMOTE, buf_LRU_add_block() for OLD, and
page_zip_write_rec() for NONEMPTY. page_zip_des_t::copy_from() and
set_n_blobs_and_empty() skip the fetch_add() when the delta is 0.
page_zip_decompress_clust() counts the BLOB pointers locally and
invokes add_n_blobs() once.
page_zip_des_t::copy_from(): Replaces the copy constructor that
page_zip_copy_recs() invoked. It copies m_end, NONEMPTY and n_blobs
from the source page, but preserves the PROMOTE and OLD flags of the
destination block, because they describe the position of that block
in buf_pool.LRU. Copying OLD from the source would make
buf_pool.LRU_old_len inconsistent with the list. Like
set_n_blobs_and_empty(), it updates the state with a single
fetch_add(), which preserves any concurrent change of PROMOTE or OLD.
buf_page_t::init(): Takes the ROW_FORMAT=COMPRESSED shift size
(ssize) directly and clears the zip descriptor state itself, so
callers (buf_block_t::initialise(), buf_page_init_for_read()) no
longer need a separate page_zip_des_init()/page_zip_set_size() call.
buf_page_t::invalidate(): Replaces buf_block_modify_clock_inc().
Instead of maintaining a 64-bit counter, we will maintain one
comprising 32+16=48 bits, in modify_clock_low,modify_clock_high.
Worst case there will be exactly n<<48 calls to
buf_page_t::invalidate() before some operation such as
btr_pcur_t::restore_position() is executed. Such a count should
be extremely unlikely but not completely impossible. It is worth
noting that the DB_TRX_ID is only 48 bits, and each transaction
start and commit/rollback will consume an identifier.
buf_page_t::modify_clock(): Replaces the read access of
buf_block_t::modify_clock. Assert that the caller is holding
a page latch. Note: because invalidate() and modify_clock() are
protected with buf_pool.mutex or the buf_page_t::lock, there can
be no issue with regard to the atomicity of accessing the 48-bit field.
PageBulk::m_modify_clock: Remove. It was assigned but never read.
buf_page_t::relocate_from(): Copies a descriptor that is relocated
to this block, in buf_relocate() and buf_pool_t::shrink(), but keeps
the frame and the modify_clock() of this block. The copy constructor
does not copy the clock: it belongs to the block descriptor and must
only grow during its lifetime, as buf_block_t::modify_clock did.
Otherwise, the clock of this block could return to a value that a
btr_pcur_t saved before the block was freed, and
btr_pcur_t::restore_position() could reuse a record pointer although
the page was modified in another block in the meantime.
buf_page_t::access_time: Store the 16-bit buf_pool.access_clock
rather than a 32-bit millisecond ut_time_ms(). It wraps around every
18.2 hours; ages are computed as uint16_t(now - access_time), which
stays correct across that wrap. This avoids any alignment loss: the
adjacent fields modify_clock_low, modify_clock_high, access_time of
32+16+16 bits nicely add up to 64 bits. access_time is stamped on the
first access after the block was initialized, and on each promotion
by make_young_if_needed(), so that the age of a frequently promoted
block stays exact across the wrap.
buf_pool_t::access_clock: uint16_t(my_interval_timer() / 1000000000),
never 0 (see buf_pool_t::now()), refreshed about once per second by
buf_pool_t::refresh_clock() so that page accesses need not read the
system clock. srv_master_callback() refreshes it. From
buf_pool_t::create() until srv_master_timer is started, and for good
when srv_master_timer is not started (innodb_read_only,
innodb_force_recovery>=2, mariadb-backup), a separate
buf_pool_clock_timer refreshes it.
buf_pool_t::access_clock, buf_pool_t::LRU_old_threshold: Located in a
cache line of their own, because they are read on page accesses and
the adjacent buf_pool fields are frequently written.
buf_page_t::touch(): Stamp access_time on the first access, then
invoke touch_no_stamp(). Return whether this was not the first access,
as the result of buf_page_make_young_if_needed() used to be. A
buffer-fix is sufficient, as in MVCC undo page lookups.
buf_page_t::touch_no_stamp(): Set the PROMOTE flag if
innodb_old_blocks_time is 0, or if accessed_at() is at least that old,
at any position of the block in buf_pool.LRU. Because
buf_pool.access_clock advances once per refresh_clock(), about once per
second, a difference of threshold ticks can span less than
innodb_old_blocks_time; the difference must exceed the threshold, so
that a burst of accesses that crosses a refresh is not promoted. Like
btr_cur_nonleaf_make_young(), do not stamp access_time. Once PROMOTE is
set, later accesses only load the state and return.
buf_page_t::make_young_if_needed(): If PROMOTE is set, invoke
buf_page_t::make_young(). This part is inline, so that a sweep pays no
function call for a block that stays in place. It reads the state of
the block once: buf_pool.mutex, which the caller holds, protects OLD,
and only make_young() clears PROMOTE. The block need not be in
buf_pool.LRU_old: while buf_pool.LRU is shorter than
BUF_LRU_OLD_MIN_LEN, buf_pool.LRU_old does not exist, and blocks are
evicted without ever becoming old. The template parameter only_old
leaves a block that is not old in place; buf_do_flush_list_batch()
uses it, because it visits blocks in the order of oldest_modification,
and moving a block that is not old would not save it from eviction.
The template parameter count_not_young selects whether an old block
that has an accessed_at() stamp but is not flagged is counted in
buf_pool.stat.n_pages_not_made_young. This includes a block that had
no access since its last promotion, so the count is of sweep visits,
not of rejected accesses; only the eviction sweeps
buf_LRU_free_from_common_LRU_list() and buf_flush_LRU_list_batch() do
this, and a block that a sweep leaves in the list can be counted again
by a later sweep. As before, buf_pool.stat.n_pages_made_young counts
only the moves of old blocks. Because a sweep moves the block, not the
access, and counts the blocks that it does not move,
Innodb_buffer_pool_pages_made_young and
Innodb_buffer_pool_pages_made_not_young can differ from before for the
same workload.
buf_page_t::make_young(): Clear PROMOTE, stamp access_time, and move
the block to the "recently used" end of buf_pool.LRU. The block can be
read-fixed, because buf_pool_t::unzip() copies PROMOTE and OLD from the
compressed-only descriptor and releases buf_pool.mutex during
buf_zip_decompress(). Unlike buf_page_make_young(), we do not skip such
a block: a page access no longer acquires buf_pool.mutex, and the
sweeps hold buf_pool.mutex but no buffer-fix on the block.
buf_LRU_remove_block(): When buf_pool.LRU becomes too short for
buf_pool.LRU_old, clear the OLD flags from buf_pool.LRU_old to the end
of the list, which holds all the old blocks, not the flags of every
block. While buf_pool.LRU_old does not exist, no block in buf_pool.LRU
is old, so on later removals the loop does nothing.
buf_LRU_add_block(): Write the OLD flag only if it changes. In a list
that is too short for buf_pool.LRU_old, assert that buf_pool.LRU_old
is not defined and that the block is not old, instead of writing the
flag.
innodb_old_blocks_time_update(): New sysvar update callback,
replacing a NULL one, that calls buf_pool_t::set_old_threshold_ms(),
so that SET GLOBAL innodb_old_blocks_time also updates the
LRU_old_threshold in seconds that page accesses read. The threshold
is clamped to 65534 seconds (UINT16_MAX - 1), because the age
uint16_t(now - access_time) cannot exceed 65535 and a larger threshold
would prevent any promotion. The sysvar itself still accepts up to
UINT_MAX32 milliseconds.
buf_pool_t::set_old_threshold_ms(): Round the millisecond threshold
up, not down, to the nearest second. innodb_old_blocks_time is
documented and accepted in milliseconds; flooring instead of ceiling
would make any configured value from 1 to 999 silently behave as 0
(disabled).
buf_page_t::is_accessed(): Renamed accessed_at(), to stop reading as
a boolean. It returns the access_time stamp of the first access or
of the last promotion, in seconds;
INFORMATION_SCHEMA.INNODB_BUFFER_PAGE(_LRU).ACCESS_TIME now reflects
that.
buf_read_ahead_random(): A page now qualifies once accessed_at()
holds, together with either zip.is_promote() or !zip.old().
buf_read_ahead_linear(): Compare access_time stamps as a signed
16-bit difference, not raw unsigned, so the monotonic-access check
stays correct across the access_time wrap. The resolution of the
stamps is 1 second instead of 1 millisecond.
buf_LRU_scan_and_free_block(): Declare static.
buf_flush_LRU_list_batch(): In a run of blocks that
make_young_if_needed() moves, release and reacquire buf_pool.mutex
after every 512 scanned blocks, except on the first scanned block.
A move costs much less than an eviction, so the stride is longer than
the one of the eviction path.
buf_pool_invalidate(): Define in the same compilation unit with
buf_LRU_scan_and_free_block().
buf_pool.LRU_old_time_threshold: Replaces buf_LRU_old_threshold_ms.
PageConverter::run(): Renamed from fil_iterate(). In debug builds,
initialize and acquire a dummy exclusive latch on the block, so the
assertion in buf_page_t::invalidate() is satisfied; free the latch on
every path, not only on success.
AbstractCallback::m_zip_ssize: Replaces m_zip_size.
page_zip_des_t: Add calc_ssize()/zip_size() helpers, replacing the
zip_size<->ssize conversion duplicated across buf0buf.cc, buf0rea.cc
and row0import.cc.
innodb.buf_lru_scan_resistance: A new big test that checks that pages
of a table that is accessed in one burst are evicted, even if a sweep
reaches them only after innodb_old_blocks_time, that pages accessed
again after that time are promoted, and that pages read while the
buffer pool is being filled are not promoted later.
innodb_zip.n_blobs_700: A new test that stores 700 BLOB pointers on
one ROW_FORMAT=COMPRESSED page, within the 10-bit n_blobs field. The
index_page_splits counter confirms that the clustered index consists
of its root page alone, without depending on which pages remain in the
buffer pool.
Co-Authored-By: Alessandro Vetere <iminelink@gmail.com>
log_t::checkpoint_margin(): In the target LSN computation, account
for the special case resize_log.m_file==log.m_file which signals
that a checkpoint write is in progress. Assert that the target
checkpoint LSN is sensible.
Fixes up commit 3394a9d114 (MDEV-37949).
Reviewed by: Thirunarayanan Balathandayuthapani
Blob values for empty strings should return a pointer to an empty
string and not NULL. `hp_materialize_one_blob()` returned NULL,
which its callers `hp_rec_key_cmp()` and `hp_key_cmp()` read as an
allocation failure.
`hp_test_write_dup-t.c` was extended to test key reads on `TEXT`
columns. This could not be done in MTR, as a `MEMORY` table cannot
be created with a key on a `TEXT` column.
`HA_NO_KEY_READ` marks a key whose blob segment
`heap_prepare_hp_create_info()` converted from the VARTEXT2 form, so
that `heap_rkey()` refuses an index read on it. It never worked,
because the mark was written to the wrong structure member.
`heap_rkey()` tests `HP_KEYDEF::flag`, which is where the flag
belongs: `HA_NO_KEY_READ` is declared among the key flags, not the
key-seg flags. The assignment instead targeted `HA_KEYSEG::flag`,
which no reader consults for this flag, so the guard could never
fire. That member is also `uint16`, so bit 20 was discarded on
assignment as well; `-Wall -Wextra` does not warn, only
`-Wconversion` does, and it is not enabled.
Write the flag to `keydef[key].flag` instead. `HP_KEYDEF::flag` is
`uint` and holds bit 20, and `heap_create()` copies it into the
share that `heap_rkey()` reads.
`hp_test_key_setup-t` covers the marking, the unmarked case, that
`heap_create()` does not lose the flag while folding its own bits
into `keydef->flag`, and that `heap_rkey()` refuses a marked key
while accepting an unmarked one. The last pair clears `my_assert`
so the guard reports instead of aborting, the same way the server's
`--debug-assert=0` does, and skips on builds without `DBUG_ASSERT`.
`hp_alloc_from_tail()` tests the table memory ceiling before it decides
whether the leaf it is about to use has to be allocated or is merely
being reclaimed. Reclaiming adds no memory, so on that branch the test
answers a question nobody asked.
The state it misjudges is routine. `hp_find_free_hash()` allocates an
index leaf without consulting `max_table_size`, and no leaf is smaller
than `heap_min_allocation_block`, so a table with two hash indexes is
already over a 32K ceiling once it holds its first row. That is legal:
the ceiling is only tested when the record cursor lands on a leaf
boundary, and the first row tests it while both counters are still
zero.
`hp_shrink_tail()` puts the cursor back on a leaf boundary whenever it
empties the tail, and `data_length` goes on counting the leaf, which is
still allocated. The next write re-reads that sum and reports
`HA_ERR_RECORD_FILE_FULL` for a row the table held a moment earlier.
Move the ceiling test to the arm that calls `hp_get_new_block()`. The
`max_records` row-count test stays where it is, because it caps rows
however the slot is obtained.
**Why a master and a slave disagreed about the same statement.**
`REPLACE INTO t SELECT * FROM t` feeds the row back out of the table,
so the record buffer still points into the chain that the delete
parked; the chain is adopted rather than freed and the cursor never
moves. The slave builds the row from the replication event buffer,
nothing points into the parked chain, it is freed, the tail empties and
the write is refused. Replication is one way to reach the state, not
the cause: a targeted `DELETE` and a re-`INSERT` on a single server
reach it too.
Tests: `heap.blob_delete_reinsert_ceiling` covers the single-server
route and checks that a table that genuinely needs more memory is still
refused; `heap.blob_replace_repl_ceiling` covers the reported
master/slave divergence; `hp_test_freelist` test 22 pins the branch
itself. Each fails without the change.
**Storing the row limit as a ceiling.** `heap_create()` recorded its
`max_records` argument in the share unchanged, so 0 arrived there
still meaning "no limit" and every reader had to special-case it. The
row-count test above carried that as a second condition,
`&& info->max_records`.
Write `NO_LIMIT_RECORDS` into the share instead. The share then
always holds a real ceiling and the write path tests one value. That
frees 0 to mean on the share what it says, a table that accepts no
rows, which is the natural way to create one that is never written to;
only `heap_create()`'s argument keeps using 0 for "no limit".
`ha_heap::info()` multiplies `max_records` by the record length to
report `MAX_DATA_LENGTH`. An unlimited table now reports
`~(my_off_t) 0` rather than a product that would overflow. Nothing
reaches that branch through SQL, because `ha_heap::create()` derives
`max_records` from the memory ceiling and a `MAX_ROWS` clause only
lowers it. Measured before and after, the reported `MAX_DATA_LENGTH`
is unchanged for a plain `MEMORY` table and for one created with
`MAX_ROWS`.
The unit tests wrote the old convention themselves and move with it:
`hp_test_freelist` assigned `share->max_records= 0` to lift the limit,
which now reads as a limit of zero rows, and `hp_test_block_size`
asserted the share kept the 0.
Nothing says how many times a statement asks for the result of a group,
and the answer must not depend on it. A `HAVING` clause on the alias is
the shortest statement that asks twice, and it returns a different value
than the same aggregate asked once:
SELECT GROUP_CONCAT(a ORDER BY a LIMIT 2 OFFSET 4) v FROM t1;
-> (empty)
SELECT GROUP_CONCAT(a ORDER BY a LIMIT 2 OFFSET 4) v FROM t1
HAVING v LIKE '%';
-> a,b
`val_str()` walks only while `result_finalized` is false, and
`dump_leaf_key()` raises that flag for the first row it writes. A row
that falls inside the offset is skipped by an earlier return, which
decrements the offset counter and leaves the flag alone. The row-limit
arm immediately above it does raise the flag before its own early
return, so two adjacent early returns behave differently.
A walk in which every row was skipped therefore writes nothing and
records nothing. The next caller walks again with the offset already
spent, and the rows skipped the first time are appended to a result
buffer that was handed over once already.
Once the duplicate filter has spilled to disk the second walk is worse
than wrong. `Unique::reset()` documents the contract:
Clear the tree and the file.
You must call reset() if you want to reuse Unique after walk().
The first walk flushed the tree and emptied it, so the second flushes an
empty tree, appending a chunk that holds no rows. `merge_walk()` reads
nothing back from it and fails `DBUG_ASSERT(bytes_read)`. A build
without assertions goes on to take keys from that chunk.
Set `result_finalized` where the walk block ends, so that it records
every path that has consumed the filter rather than only the paths that
wrote a row.
On the release branches only the form without `ORDER BY` reaches the
duplicate filter. Since MDEV-21879 the `DISTINCT ... ORDER BY`
combination builds its result the same way, so both forms can reach the
assertion here.
A TEXT value longer than `group_concat_max_len` is cut on its way into
`blob_storage`, in `Field_blob::handle_group_concat()`. That happens
while the group is being built, not when the answer is put together, so
whether the answer is any shorter for it depends on whether the row
carrying the value reaches the answer at all. Every such cut was
reported as `ER_CUT_VALUE_GROUP_CONCAT`, which says that the answer lost
something the user asked for, and that is true of only some of them.
`Blob_mem_storage` now writes one byte in front of every value it
stores and returns the pointer past it, so `was_cut()` answers for any
value a reader holds a pointer to. `Field_blob::store()` sends every
blob of a table that has a `Blob_mem_storage` through
`handle_group_concat()`, so no value in that storage is without the
byte, and `Field_blob::get_ptr()` on the record of a row hands back
exactly the pointer that was stored.
`dump_leaf_key()` reads the mark off each row it appends and sets
`value_cut_in_result`. `val_str()` then reports:
1. **A warning**, `ER_CUT_VALUE_GROUP_CONCAT`, when a row that reached
the answer carried a cut value. The answer is short by what was cut.
The result being cut at `gconcat_max_len()` already gives that same
warning, and a group that hits both is told once, not twice.
2. **A note**, `ER_CUT_VALUES_WHILE_PROCESSING`, when a value was cut
but no row carrying one reached the answer. The answer may well be
what a larger limit would have given. One note per aggregate is
enough for a statement however many groups had a value cut, and
`cleanup()` clears the mark so a statement run again gets its own.
Reporting the loss as a warning keeps a strict `sql_mode` aborting on
it, which it does because `THD::raise_condition()` promotes a warning
and never promotes a note.
`ST_COLLECT` is not affected. It reports `ER_CUT_VALUE_GROUP_CONCAT`
itself, against `group_collect_max_len`.
`main.gconcat_cut_note` covers the split with one group holding a short
value and a long one, where a `LIMIT` alone decides which of them the
answer is built from, over both the sort tree and the duplicate filter.
`main.func_gconcat` shows the granularity: of five groups at
`group_concat_max_len=499999`, the one holding exactly 499999 bytes is
the one that does not warn.
Note that `blob_storage` only exists when the aggregate has an
`ORDER BY` or a `DISTINCT` and a blob field, so this is the only shape
in which a value is cut this way.
Give a warning when `GROUP_CONCAT(DISTINCT x)` or
`JSON_ARRAYAGG(DISTINCT x)` returns only part of a group, or nothing at
all, because the walk of the duplicate filter failed. The result is
wrong rather than deliberately cut, and nothing was said about it.
Both build their result in `val_str()` by walking `unique_filter`, and
threw the walk's return value away. `Unique::walk()` reports its own
failures through it, from allocating the merge buffer to reading back
the chunks it merged, so a failure gave a short result, or an empty
one, in silence.
The return value cannot be used on its own. `dump_leaf_key()` also
stops the walk, for two reasons that are not failures: it cuts the
result at `group_concat_max_len`, which it already reports by setting
`result_cut`, and it stops without losing anything once the `LIMIT` is
used up. Reporting every non-zero return as a cut warns about
`GROUP_CONCAT(DISTINCT a LIMIT 5)` returning exactly the five rows
that were asked for.
`dump_leaf_key()` now records that it was the one that stopped the
walk, so `val_str()` warns only when the walk itself failed.
The warning is a new one. `ER_CUT_VALUE_GROUP_CONCAT` reports the row
the result was cut at, and there is no such row here: the walk failed
before it delivered anything, and how much was lost is not known, so
it would read `Row 0 was cut by group_concat()`.
`ER_RESULT_CUT_BY_LIMIT` says that the result was cut and names the
limit that cut it, which is the memory `Unique` was given to work in:
the smaller of `tmp_memory_table_size` and `max_heap_table_size`.
Not every failure is silent either. The merge buffer is allocated with
`MY_WME` and the spill file is opened with `MY_WME`, so running out of
memory or failing to read raises an error of its own. Only the guard at
the top of `merge_walk()`, which refuses a merge buffer too small to
hold one key per chunk, returns without saying anything. Warn only when
no error was raised: where one was, the user has been told and the
statement is failing, so describing the length of a result nobody will
see adds nothing.
The debug keyword `unique_walk_merge_fail` fails the merging walk
quietly and `unique_walk_merge_error` fails it with an error raised.
`main.gconcat_distinct_walk_fail` uses both. The `LIMIT` case needs no
debug build and is checked in `main.gconcat_distinct_spill`.
`Item_func_group_concat::add()` decided whether a row was a duplicate
by checking whether `Unique::elements_in_tree()` had grown after
`unique_add()`:
uint count= unique_filter->elements_in_tree();
unique_filter->unique_add(get_record_pointer());
if (count == unique_filter->elements_in_tree())
row_eligible= FALSE;
`Unique` flushes its whole in-memory tree to disk when it runs out of
memory, and `elements_in_tree()` only counts what is still in memory.
After the first flush the test says nothing about the rows that were
already spilled.
**MDEV-11563** made this harmless for `GROUP_CONCAT(DISTINCT x)` by
building the result in `val_str()` from `unique_filter->walk()`, which
merges the spilled parts back in. It left the `ORDER BY` case alone.
There the result comes from the sort tree, which `add()` fills gated by
`row_eligible`, so the defect is still fully live.
Both directions of the failure are reachable, depending on how often
the filter flushes relative to the insert:
1. Duplicates reach the result. 100 rows holding 50 distinct values
give all 100 values back.
2. Rows are lost. 30 distinct rows of 2000 bytes give one value back.
`JSON_ARRAYAGG(DISTINCT x ORDER BY y)` fails in the same way.
Fixed by not filling the sort tree from `add()` when `DISTINCT` is
used. `val_str()` now walks the merged `unique_filter` into the sort
tree and then walks the sort tree, so the rows are sorted after the
duplicate filtering is complete instead of during it.
`Unique::walk()` merges everything it flushed, so the sort tree can be
handed more rows than fit in memory. `insert_to_order_tree()` repacks
it on the same memory budget `add()` used, and a walk that runs out of
memory sets `result_cut`, so the user gets a cut value warning rather
than a silently short result.
**Behaviour change.** `ORDER BY` does not order rows that tie on the
ordering expression, and which of them comes first changes here. It
used to follow the order the rows were read in; it now follows the
order the duplicate filter keeps them in. Unlike the old order, the new
one depends on neither the memory available nor the physical row order.
`main.gconcat_distinct_spill` checks that, and `main.func_gconcat`
records one such tie.
Both tests exercise server facilities that an embedded build does not
have, so neither can run there.
`INSERT DELAYED` has no delayed insert thread in an embedded build.
The whole facility sits inside `#ifndef EMBEDDED_LIBRARY`, including
the check that routes a delayed statement away from the ordinary
insert path, so the statement is an ordinary insert and
`DELAYED_WRITES` stays at 0.
`ALTER TABLE ... LOCK=NONE` is never online in an embedded build.
`online` is hard-wired to `false` when `HAVE_REPLICATION` is
undefined, and `my_global.h` leaves it undefined for
`EMBEDDED_LIBRARY`. The source lock is therefore not downgraded, the
`alter_table_online_downgraded` sync point is never reached, and the
test's `WAIT_FOR downgraded` runs out its `debug_sync` timeout.
Skip both with `include/not_embedded.inc`, as `main.delayed` and
`main.alter_table_online_debug` already do.
GROUP_CONCAT() with ORDER BY collects all rows of the group in a TREE
and only cuts it down in repack_tree(). The repack was triggered by
(tree_len >> GCONCAT_REPACK_FACTOR) > thd->gconcat_max_len()
with GCONCAT_REPACK_FACTOR 10, that is when the rows in the tree had
produced 1024 * group_concat_max_len bytes, or 1G with the default
settings. On top of that tree_len only counted the length of the
strings, while the tree costs sizeof(TREE_ELEMENT) + reclength per row.
For GROUP_CONCAT(int_col ORDER BY int_col) that is about 40 bytes per
row against 6 bytes of result, so the tree had grown to several GB
before the first repack. In practice the server ran out of memory
first and the repack code was close to never used.
The tree is now limited by the memory it has really allocated,
tree->allocated, instead of by the length of the strings it holds.
The limit is MY_MAX(thd->ram_limitation(), thd->gconcat_max_len()) and
is never set so low that the tree can not hold a few rows.
repack_tree() builds a new tree while the old one is still in memory,
so the peak usage is the size we start the repack at plus the size we
copy to. To keep the sum within the limit it is split into
GCONCAT_TREE_PARTS parts; the repack starts when
GCONCAT_TREE_REPACK_PARTS of them are used and copies to the remaining
part. The part we do not copy to is also the room the tree has to grow
before the next repack, which keeps the repacks amortized.
Other changes:
- tree_len is removed. It was only read by the old trigger.
- repack_tree() decided that it had run out of memory by testing
st.len <= st.maxlen after the walk. That test was only valid because
the old trigger guaranteed that a complete copy had to overshoot
st.maxlen. A repack triggered by memory can complete the walk with
st.len far below st.maxlen, which would have failed the query with a
wrong out of memory error. There is now an explicit flag for it.
- The length that decides which rows to keep now also counts the
separator that is put between two rows, so that it matches what
val_str() will produce.
- When the memory limit stops the copy, the result becomes shorter
than group_concat_max_len. dump_leaf_key() can not detect this, as
the result never reaches the maximum length. This is now remembered
in result_cut and reported to the user.
- All cut value reporting is moved to val_str(); dump_leaf_key() only
marks that the result was cut. This removes the need to clear the
truncated flag of table->blob_storage to avoid a duplicated warning,
and gives one warning per group also when val_str() is called more
than once for the same group, which repeated the warning before.
- Added a function comment for repack_tree() that describes where the
rows are cut away and why building a copy frees memory.
Co-author: Arcadiy Ivanov <arcadiy@ivanov.biz>
- Fixed a bug in copy_tree()
tmp_memory_table_size is limiting the size of internal temporary memory
tables. max_heap_table_size is there to limiting the size of explictely
created memory tables. max_heap_table_size can be much larger than
tmp_memory_table_size as the memory used by temporary tables is in the
control of the user.
This commit changes the usage of max_heap_table_size for internal buffers
to min(max_heap_table_size, tmp_memory_table_size), like we do for
internal temporary tables.
This changes the in memory buffer allocations for:
- GROUP_CONCAT()
- Calculating the cost for scanning memory tables (the original code was
wrong here as it used the wrong size for memory tables).
- ANALYZE TABLE buffer sizes for calculating distinct column values
Other things:
- Add THD::ram_limitation() to provide consistent memory limitations
in all code that used variables.tmp_memory_table_size as buffers.
If tmp_memory_table_size == 0, then 8192 is used.
This replaces Item_sum::ram_limitation which used 1024 as min buffer,
which is way to little for any practical case.
- Added security guard in heap_prepare_hp_create_info to ensure that
max_table_size is calculated same way as in MariaDB server.
- Fixed initial memory allocations for Item_func_group::concat which
allocated 'max allowed memory' at start. Now it allocates only 1/16
of that memory at start.
