The binlog-in-engine code limits the maximum size of engine binlog writes by
forcing OOB-spill, eg. to avoid exceeding InnoDB max mtr size. However there
was a case missed: When the transaction is larger than binlog_max_spill_size
but smaller than the binlog_cache_size, there was no check to force
OOB-spilling, and the code would do a too large binlog write that asserts or
fails inside InnoDB.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
This merges the patch series that fixes a lot of missing validation
checks for read_log_event(). The merge includes a few added fixes that
are needed in new code introduced in 12.3.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
PURGE BINARY LOGS could end with a warning instead of deleting files, as
the test below shows when run on the base of this commit. The warning
names a wrong reason. It follows an automatic purge that was refused by
the slave guard, slave_connections_needed_for_purge: the automatic purge
does not delete a binlog while it might still be needed by a slave, not
necessarily a connected one, that is while fewer than the variable's
number of slaves are connected.
Since MDEV-34504 the manual PURGE is meant to ignore the guard.
Yet it still protects a binlog that a slave is reading.
Technically this was possible because the automatic and the manual purge
share a code path, can_purge_log(), which cached the refusal and replayed
it, under the reason of the first arm below, to any later purge:
if (is_active(name) ||
(!is_relay_log && waiting_for_slave_to_change_binlog &&
purge_sending_new_binlog_file == sending_new_binlog_file &&
!strcmp(name, purge_binlog_name)))
reason= "it is the current active binlog";
So "active" had to be read as "active or busy". The fix removes this
duality too: the two arms are now separate and have their own reasons.
The main issue is fixed by narrowing the busy arm with an additional
`!interactive &&` conjunct, so that a manual PURGE never consults the
cached refusal.
Running the test below on an unfixed 11.4 server (source tree at commit
42038e1145) fails with "Result length mismatch" and, in part 2, this
diff against the recorded result:
PURGE BINARY LOGS TO 'master-bin.000006';
+Warnings:
+Note 1375 Binary log 'master-bin.000003' is not purged because it is the current active binlog
SHOW WARNINGS;
Level Code Message
+Note 1375 Binary log 'master-bin.000003' is not purged because it is the current active binlog
show binary logs;
Log_name File_size
+master-bin.000003 #
+master-bin.000004 #
+master-bin.000005 #
master-bin.000006 #
master-bin.000007 #
master-bin.000008 #
Test binlog.binlog_purge_stale_refusal_cache: the test sets
slave_connections_needed_for_purge=1, binlog_expire_logs_seconds=0 and
max_binlog_total_size=0 first and restores them at the end. It starts
with RESET MASTER so that the binlog names, 000003 and so on, do not depend
on the tests run before it in the same mtr environment. Part 1 is a control with
nothing cached; part 2 makes an automatic purge refuse 000003 (the files
must be older than binlog_expire_logs_seconds, hence --real_sleep 2 before
the rotation) and then runs PURGE BINARY LOGS TO 'master-bin.000006',
which must remove 000003..000005.
As a workaround, a PURGE could work if the following settings are made
before it, in this order:
SET GLOBAL binlog_expire_logs_seconds=0, slave_connections_needed_for_purge=0;
The second setting invalidates the cached refusal. The first one keeps
the purge that this setting triggers from refusing and caching again.
Note that both settings change the server's policy, not only this one
PURGE: the time-based expiry and the slave guard are off until they are
restored.
Issue:
CHANGE MASTER ... FOR CHANNEL with a channel name within
MAX_CONNECTION_NAME can still overflow the OS file name limit once
escaped into the relay log file name. Relay_log_info::init() then
fails to open the relay log, leaving its index file unopened, but
Master_info_index::remove_master_info() unconditionally calls
reset_logs() on it during CHANGE MASTER's error cleanup, which hits
the assertion in reinit_io_cache().
Solution:
Guard the reset_logs() call in remove_master_info() with is_open(),
so a relay log that was never opened is never passed to it. Also use
MY_SAFE_PATH in open_index_file() so an over-length name fails
deterministically instead of silently falling back to a mangled one.
When using binlog-in-engine, the binlog trx cache is memory-only and does
not have an underlying temporary file backing the cache. The binlog code
must handle this and avoid calling into IO_CACHE code that manipulates the
underlying file. However, one case was forgotten where the binlog caches are
truncated during COM_CHANGE_USER / THD::change_user() (connection pool
reset), and the code would try to seek() on an invalid file handle. This
crashes on Windows and is just plain wrong on any platform.
Thanks to Vlad for analysis and test case.
Reviewed-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
With log_bin=ON a transaction is committed via two-phase commit (the
binary log is the second participant), so it passes through the InnoDB
XA-prepare state. While a donor is held in BLOCK_COMMIT for a mariabackup
backup, its parallel appliers (wsrep_slave_threads > 1) leave one or more
such writesets prepared-but-not-yet-committed, and the snapshot captures
them. On a freshly SST'd joiner nothing resolves these prepared
transactions: binlog crash recovery does not run (the joiner has no in-use
binlog to recover from), and the wsrep continuity-based commit is inactive
because wsrep_emulate_bin_log is FALSE when log_bin is ON. The leftover
prepared transactions then abort startup with "Found <N> prepared
transactions!". Note this does not depend on the prepared set being
non-contiguous - even a contiguous run aborts, because nothing commits
or rolls it back.
Rollback these transactions in xarecover_handlerton(). If rollback fails
flag error to cause unireg_abort().
After a mariabackup SST the joiner could fail with
ER_GTID_STRICT_OUT_OF_ORDER (error 1950)
while re-binlogging transactions received over IST.
The cause is that the binary log copied from the donor carries a
Gtid_list whose position can be ahead of the storage-engine snapshot
because getting snapshot and binlog are no mutually atomic, so transactions
can be present in the copied binlog that are not committed in the copied
engine snapshot.
After the SST the joiner reports the (committed) engine position to the
cluster, IST resends those transactions, and re-binlogging them under
gtid_strict_mode=ON collides with the ahead Gtid_list -> error 1950.
(MDEV-34483 made the engine snapshot stop short of the binlog, which is
what exposed this.)
The copied binary log carries no transactions the joiner needs - only a
Gtid_list - so instead of shipping and then having to truncate/reconcile
it, the joiner now starts a fresh binary log and seeds its GTID position
from the storage-engine checkpoint during recovery. That checkpoint is
the committed cluster position, i.e. exactly where IST resumes, so the
joiner's binary log stays in lockstep with the rest of the cluster and
no out-of-order GTID can occur.
This works for both wsrep_gtid_mode settings; only the binlog domain of
the cluster stream differs:
- wsrep_gtid_mode=ON : wsrep_gtid_domain_id (cluster writes are
re-tagged to it), which is the domain stored in the checkpoint;
- wsrep_gtid_mode=OFF: gtid_domain_id (cluster writes keep the node's
configured domain).
Async-replica positions (mysql.gtid_slave_pos) are part of the engine
snapshot and survive the SST unchanged, so a Galera node can still serve
as an async master or replica across the SST.
This commit:
- sql/log.cc: adds wsrep_seed_binlog_gtid_state(), called from
do_binlog_recovery() when the joiner has no binary log, seeding the
binlog GTID state for the cluster domain to the SE checkpoint position.
- scripts/wsrep_sst_mariabackup.sh: no longer moves the donor's binary
log into place on the joiner.
- extra/mariabackup: backward compatibility: keep shipping binlog file
in SST but
- on donor fix the race between rotation and shipping so that the file
shipped is the one that had been rotated;
- on joiner discard shipped binlog in favour of one generated by
wsrep_seed_binlog_gtid_state().
- sql/wsrep_sst.cc: logs the position actually adopted from storage
(the authoritative post-SST position) rather than the script-reported
one.
- sql/handler.cc: downgrades the "Discovered discontinuity in recovered
wsrep transaction XIDs" message in wsrep_order_and_check_continuity()
from warning to debug level. With parallel appliers a snapshot
routinely captures prepared XIDs that are not contiguous with the
engine checkpoint, so this is normal during SST recovery and of no
value in regular operation; the transactions past the checkpoint are
re-delivered by the cluster (IST/SST) regardless.
- Adds an MDEV-38147 MTR test reproducing the issue.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Protect all READ_ONLY sysvars from run-time changes.
Put them into a separate section and use my_virtual_mem_protect()
to make this section read-only before the server starts accepting
connections.
Verify that they're all protected in the sys_var constructor.
One exception: opt_noacl (--skip-grant-tables) can be changed
from 1 to 0 on FLUSH PRIVILEGES. Let's briefly drop the
protection for this 1->0 change. It can be needed only once
in a server lifetime and only if it was started with --skip-grant-tables
On shutdown the protection is removed, making variables writable
again because shutdown resets some of them during the cleanup
Assisted-By: Claude:claude-4.8-opus
Don't attempt to rotate any old-style binlog files if trying to set
@@binlog_checksum. The @@binlog_checksum variable is unused with the
new binlog, and there are no old-style binlog files to rotate.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
`DBUG_EXECUTE_IF` is a macro that uses
the infamous `do {…} while(0)` “idiom”.
This detail causes the `break` in the code block to terminate the
`while(0)` behind the macro instead of the `for (;;)` outside.
Consequently, this fault simulation does not terminate the normal flow,
which in turn overwrites the simulated fault with a non-fault outcome.
Since this fault simulation is the first step of the loop,
this commit fixes the fault injection by moving it before the loop.
Remove the unscoped #pragma GCC diagnostic ignored
"-Wdeprecated-declarations" from include/violite.h and replace all
sprintf/vsprintf calls with snprintf/vsnprintf across the codebase.
Where possible, pass actual buffer sizes as function parameters
instead of using hardcoded constants or recomputing strlen().
Notable API changes:
- get_date(): added to_len parameter for buffer size
- nice_time(), end_timer() in client/mysql.cc: added buff_size
- generate_new_name(): added name_size parameter
- calc_md5(): added buffer_size parameter (fixes off-by-one)
- MD5_HASH_TO_STRING macro: added _size parameter
- Added MYSQL_UDF_MAX_RESULT_LENGTH constant in mysql_com.h
Also fixed a pre-existing bug in sql/rpl_mi.cc where the ellipsis
"..." was written to the wrong buffer (dbuff instead of buff).
Vendored code is excluded: extra/readline, extra/wolfssl,
wsrep-lib, libmariadb, storage/mroonga/vendor, zlib.
When adding in variable type binlog_row_event_fragment_threshold, it
was made a ulong to mimick max_allowed_packet and
slave_max_allowed_packet variables (as they are intertwined). However,
using ulong is bad practice as it has a different type depending on the
architecture. This is shown by test sys_vars.sysvars_server_notembedded
where the variable shows as `BIGINT UNSIGNED` on 64-bit systems, and
`INT UNSIGNED` on 32-bit systems.
This patch updates the variable to be uint (32-bit), as there is a
fundamental limitation of 4GB on event size, so the new
Partial_rows_log_event event couldn't exceed that size anyway.
Signed-off-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
This patch solves two problems:
1. Rows log events cannot be transmitted to the slave if their
size exceeds slave_max_packet_size (max 1GB at the time of
writing this patch, i.e. MariaDB 12.3)
2. Rows log events cannot be binlogged if they are larger than
4GB because the common binlog event header field event_len is
32-bits.
This patch adds support for fragmenting large Rows_log_events
through a new event type, Partial_rows_log_event. When any given
instantiation of a Rows_log_event (e.g. Write_rows_log_event, etc)
is too large to be sent to a replica (i.e. larger than the value
slave_max_allowed_packet, as configured on a replica), then the rows
event must be fragmented into sub-events (i.e.
Partial_rows_log_events), so the event can be transmitted to the
replica. The replica will then take the content of each of these
Partial_rows_log_events, and join them together into a large
Rows_log_event to be executed as normal. Partial_rows_log_events
are written to the binary log sequentially, and the replica assembles
the events in the order they are binlogged.
To control the size of each Partial_rows_log_event, a new system
variable is added: binlog_row_event_fragment_threshold. All
Partial_rows_log_events within the same group will have this size,
except for the last, which will take up the remaining length of the
underlying Rows_log_event.
Each Partial_rows_log_event stores its sequence number (seq_no) in the
overall series of fragments, the total number of fragments needed to
re-assemble the Rows_log_event (total_fragments), a uchar for flags for
embedding extra data, and any additional data as specified by the
flags. Currently, only the first event in a grouping will have
additional data: it will set the first bit in the flags
(FL_ORIG_EVENT_SIZE) to indicate it will be storing the total size of
the underlying Rows_log_event.
The cached Rows_log_event data is fragmented into
Partial_rows_log_events as follows. The primary will still generate a
Rows_log_event to write to the binlog; however, during the actual
writing process, the raw data of the rows event is split into
fragments, each covering some continuous section of the rows data. A
Partial_rows_log_event is created for each continuous section, and the
Partial_rows_log_events are written sequentially in-place of the
too-large Rows_log_event. The original data to be fragmented will
include a header and data header; however, will not include a checksum,
as each Partial_rows_log_event will have a checksum for validation, as
well as a sequence_number and total number of fragments to ensure all
fragments are present.
The re-assembly and execution of the original Rows_log_event on the
replica happens in Partial_rows_log_event::do_apply_event(). The rgi
is extended with a memory buffer that holds all data for the
original Rows_log_event. As each Partial_rows_log_event is
ingested, its Rows_log_event content is appended to this memory
buffer. Once the last fragment has added its content, a new
Rows_log_event is created using that buffer, and executed.
A new error message is added to indicate that the slave has received an
invalid stream of Partial_rows_log_events:
ER_PARTIAL_ROWS_LOG_EVENT_BAD_STREAM.
Note this commit only adds the server logic for fragmentic and
assembling events, the client logic (mysqlbinlog) is in the next
commit.
Alternative designs considered were:
1. Alternative 1: Change the master-slave communication protocol
such that the master would send events in chunks of size
slave_max_allowed_packet. Though this is still a valid idea,
and would solve the first problem described in this commit
message, this would still leave the limitation that
Rows_log_events could not exceed 4GB. Eventually, this change
should still be addressed (e.g. in MDEV-37853), and for users
of binlog-in-engine (MDEV-34705) which supports out-of-band
binlogging of events, this MDEV-32570 work will be superseded.
2. Alternative 2: Create a generic “Container_log_event” with the
intention to embed various other types of event data for
various purposes, with flags that describe the purpose of a
given container. This seemed overboard, as there is already a
generic Log_event framework that provides the necessary
abstractions to fragment/reassemble events without adding in
extra abstractions.
3. Alternative 3: Add a flag to Rows_log_event with semantics to
overwrite/correct the event_len field of the common event
header to use a 64-bit field stored in the data_header of the
Rows_log_event; and also do alternative 1, so the master would
send the large (> 4GB) rows event in chunks. This approach
would add too much complexity (changing both the binlogging
and transport layer); as well as introduce inconsistency to
the event definition (event_len and next_event_position would
no longer have consistent meanings).
Reviewed-by: Hemant Dangi <hemant.dangi@mariadb.com>
Acked-by: Kristian Nielsen <knielsen@knielsen-hq.org>
Signed-off-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
Implement an improved binlog implementation that is integrated into
the storage engine. The new implementation is enabled with the
--binlog-storage-engine option. Initially the InnoDB storage engine
implements the binlog.
Integrating the binlog in the storage engine improves performance,
since it makes the InnoDB redo log the single source of truth and
avoids the need for expensive two-phase commit between binlog and
engine. It also makes it possible to disable durability (set
--innodb-flush-log-at-trx-commit=0) to further improve performance,
while still preserving the ability to recover the binlog and database
into a consistent state after a crash.
The new binlog implementation also greatly improves the internal
design and implementation of the binlog, and enables future
enhancements for replication.
This is a squash of the original 11.4-based patch series.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
When ddl log recovery needs to binlog during the crash recovery, the
GTID was binlogged without the required "ddl" marker. This caused wrong
behaviour on the slave when using parallel replication.
Fixed by explicitly marking the "current statement" as DDL when binlogging
in ddl log crash recovery.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
This patch fixes that ALTER TABLE can call wakeup_subsequent_commits() too
early and allow following event groups to commit out-of-order in parallel
replication. Fixed by calling suspend_subsequent_commits() at the start
of the ALTER.
Could be seen as an assertion:
!tmp_gco->next_gco || tmp_gco->last_sub_id > sub_id
(Normally this is prevented because an ALTER TABLE will run in its own GCO,
and thus no following event groups can even start; however the missing DDL
mark caused by MDEV-38429 made this visible. And calling
wakeup_subsequent_commits() too early is wrong in any case).
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
compiling mariadb from a git tree
Fixed sprintf deprecation warnings compiling on MacOSX.
Replaced some sprintf calls with equivalent snprintf calls,
enough so that "normal" compile on MacOSX (as documented
in the docs) completes without warnings.
Two transactions could binlog their completions in opposite to how it
is done in Engine. That is is rare situations ROLLBACK in Engine of
the dependency parent transaction could be scheduled by the
transaction before its binlogging. That give a follower dependency
child one get binlogged ahead of the parent.
For fixing this bug its necessary to ensure the binlogging phase is
always first one in the internal one-phase rollback protocol.
The commit combines
1. a code polishing piece over a part of MDEV-21117 that
made binlog handlerton always commit first in no-2pc cases and
2. the same rule now applies to the rollback.
An added test demonstrates how the child could otherwise reach binlog
before its parent.
Two transactions could binlog their completions in opposite to how it
is done in Engine. That is is rare situations ROLLBACK in Engine of
the dependency parent transaction could be scheduled by the
transaction before its binlogging. That give a follower dependency
child one get binlogged ahead of the parent.
For fixing this bug its necessary to ensure the binlogging phase is
always first one in the internal one-phase rollback protocol.
The commit makes sure the binlog handlerton always rollbacks as first
handlerton in no-2pc cases.
An added test demonstrates how the child could otherwise reach binlog
before its parent.
The binary log could be corrupted when committing a large transaction
(i.e. one whose data exceeds the binlog_cache_size limit and spills
into a tmp file) in binlog_format=row if the server's --tmp-dir is
full. The corruption that happens is only the GTID of the errored
transaction would be written into the binary log, without any
body/finalizing events. This would happen because the content of the
transaction wasn't flushed at the proper time, and the transaction's
binlog cache data was not durable while trying to copy the content
from the binlog cache file into the binary log itself. While switching
the tmp file from a WRITE_CACHE to a READ_CACHE, the server would see
there is still data to flush in the cache, and first try to flush it.
This is not a valid time to flush that data to the temporary file
though, as:
1. The GTID event has already been written directly to the binary
log. So if this flushing fails, it leaves the binary log in a
corrupted state.
2. This is done during group commit, and will slow down other
concurrent transactions, which are otherwise ready to commit.
This patch fixes these issues by ensuring all transaction data is
fully flushed to its temporary file (if used) before starting any
critical paths, i.e. in binlog_flush_cache(). Note that if the binlog
cache is solely in-memory, this flush-to-temporary-file is skipped.
Reviewed-by: Andrei Elkin <andrei.elkin@mariadb.com>
Signed-off-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
This bug happens when using galera, max_tmp_session_space_usage and a
transaction that uses temporary file of a size between 'binlog-cache-size'
(default 32K) and 64K followed by a change user command.
In the case of a transaction of size between 32k and 64k, the server don't
truncate the binary log file on disk to speed up things, which is fine.
The bug was that change_user() reset the tmp_space_used variable and the
next time the temporary_file was truncated, we got a negative value for
tmp_space_used, which caused the error "Local temporary space limit
reached".
Fixed by not resetting tmp_space_used in init(), which is called by
change_user().
Other things
- Truncate binary log cache files when change_user() is called. This
makes the new users session not depending on log file sizes by previous
user.
- Some new ASSERT and DBUG_ENTER
- Fixed typo in DBUG output in ma_pagecache.c
`LOG_EVENT_OFFSET` is a duplicate of `EVENT_TYPE_OFFSET`
and `BIN_LOG_HEADER_SIZE` depending on the context.
While there, centralize `BIN_LOG_HEADER_SIZE` to `sql/rpl_constants.h`
(previously from `sql/unireg.h` and `client/mysqlbinlog.cc`)
Reviewed-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
- Removed duplicate words, like "the the" and "to to"
- Removed duplicate lines (one double sort line found in mysql.cc)
- Fixed some typos found while searching for duplicate words.
Command used to find duplicate words:
egrep -rI "\s([a-zA-Z]+)\s+\1\s" | grep -v param
Thanks to Artjoms Rimdjonoks for the command and pointing out the
spelling errors.
Ensure that Annotate_rows is always written direct after GTID information,
before any table_map events.
Before this patch, the following problems existed when mixing
transactional and not transactional tables in the same statement:
- Annotate rows could be written after row events or in the next GTID
event.
- See rpl_row_mixing_engines
- Annotate_rows was not always written to binary log in case of error
with a transactional table (rolled back) but a not transactional
table was updated.
- See sp_trans_log, binlog_row_mix_innodb_myisam
Fixed by writing the Annotate_rows event into the non transactional
cache if there are not transactional tables used. If not, write the
event into the transactional cache.
Replication can stop in error if a Heartbeat log event is sent to a
replica during rotation. There are two bugs at play:
1. Prior to MDEV-30128 (added in 11.0), there is a bug when checking
legacy events. When the replica rotates its relay logs, it
initializes its Format_description_log_event with binlog version 3
(this is hard-coded). So immediately after rotation (and until a
new Format_descriptor with binlog_format 4 is sent from the
master), the IO thread is expecting binlog_format 3 (i.e. it will
call queue_old_event() for incoming events). This invalidates any
events that are sent with an event type higher than 14. In theory,
we wouldn't expect any events to be sent in-between a rotate and
the next format descriptor log event, but if a long enough period
of time passes between then, the primary will generate and send a
Heartbeat event (of type 27). In such case, the slave will see the
heartbeat event of type 27, see it is higher than 14, and result
in an error mentioning 'Found invalid event in binary log', with
the expected log coordinates of the new log (which is
optimistically populated from the Rotate log event, not the new
event).
2. In all versions of MariaDB (11.0+), there is a bug when checking
the state of a Heartbeat log event, in that it doesn't consider a
rotated binary log. The check is meant to ensure that the
heartbeat provided by the master (i.e. the state of the master) is
greater than or equal to the state of the slave. In other words,
it checks that the slave isn't ahead of the master. However, if
the filename provided by the master heartbeat event is different
than the filename saved for the slave's state, the check always
fails. This is broken, because when the master rotates its logs,
the new binary log file will have a different filename (i.e. an
incremented index counter suffix). For example, if the master
rotates its binary logs from master-bin.000002 to
master-bin.000003, master-bin.000003 is ahead of
master-bin.000002, but the slave will see a difference between the
filenames and fail the check.
To fix the first problem, this patch disallows passing a heartbeat
event into queue_old_event (which is the source of the error, as it
tries to parse a heartbeat log event). This function (queue_old_event)
was removed with MDEV-30128, so bypassing it for heartbeat events is
not consequential (and it is already also done for
Format_description_events, which are not supported in old binlog file
versions). Note that backporting all of MDEV-30128 was also considered,
but this is less risky for GA.
To fix the second problem, we simply ignore heartbeat events on the
slave if the filenames don't match. This is because during rotation,
it can appear that the slave is ahead of the master, which breaks the
validity of the check (i.e. the check is to ensure the master is
ahead of the slave).
Additionally note that this patch restores a heartbeat check that was
incorrectly removed in 780db8e252
Reviewed-by: Andrei Elkin <andrei.elkin@mariadb.com>
Signed-off-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
