MDEV-32570 added fragmentation of large row events. When binlog
compression is enabled (log_bin_compress=ON), the compression is
bypassed, and these binary log events remain as regular row events, when
they should be compressed row events.
This is because the compression happens during the Log_event::write()
function, whereas row data fragmentation happens before
Log_event::write() is called. The Rows_log_event super-class stores the
row data buffers to be written to disk. The regular Rows_log_event
sub-classes's implementations of Log_event::write() write this data
as-is. The compressed sub-classes's implementation of ::write()
over-write these buffers with the compressed rows data before writing to
disk.
To fragment a large row event, the server fragments the Rows_log_event's
row data buffers, to be written by multiple Partial_rows_log_event's,
and each Partial_rows_log_event::write() call writes its portion of the
row data buffer to disk. However, this happens before compression ever
has a chance to take place, and thereby, an event that should be
compressed, never is.
MDEV-39762 added event structure validation for compressed events, and
discovered that these events carry the compressed event type, but are
not actually compressed. Event validation thereby fails.
This patch adds a workaround to override the event type of the
fragmented row events to be regular row events, to be consistent with
the actual on-disk content. The underlying problem still needs to be
addressed though, and is tracked by MDEV-40851.
Signed-off-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
Handle corrupted row events by reporting whether the field length could
not be determined from metadata or whether the field extends past the
row buffer instead of falling out through debug-only assertion paths
or ad hoc exits during flashback row conversion. Propagate flashback
conversion failures through the normal mysqlbinlog error path so debug
builds produce the intended diagnostics cleanly.
Add replication tests covering corrupted BLOB metadata and corrupted
BLOB length prefixes using server-side debug injection to write broken
binlog contents and mysqltest-friendly mysqlbinlog invocation patterns.
Reviewed-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
Reviewed-by: Georgi Kodinov <joro@mariadb.org>
If the replication IO Thread receives a Rotate event following a Format
Description event (FDE) with no post-header length for Rotate events,
the Rotate event’s parser constructor indexes
the FDE’s post-header lengths array out of bounds.
This commit defends against this situation by checking before the
constructor that the FDE describes Rotate events as recognized at all.
In practice, because the Binlog Dump thread generates a Fake
`ROTATE_EVENT` **before** sending the FDE, it has pinned
Rotate events’ post-header length to 8 regardless of FDEs.
This fix solution considers that the FDE’s description should still be
respected, matching the constructor.
Reviewed-by: Kristian Nielsen <knielsen@knielsen-hq.org>
Neither the binary-parsing Format Description Event constructor
nor the constructor-bypassing `get_checksum_alg()` function
validated the content length of the passed event buffer.
If they receive an FDE with undersized contents,
they would obtain corrupt results from an erronous memory location,
if not outright crash the program with that memory error.
This commit fixes both sites by adding content length checks.
Because the goal is not to solve the existence of two binary parsers,
`get_checksum_alg()` receives a check duplicated from the Format
Description constructor and is no longer a function that never errors.
Note, it is implementation detail that `get_checksum_alg()`’s
fix catches the invalidity before any code reaches the
parser-contructor’s fix, though the latter would come
to effect if we refactor `get_checksum_alg()` away.
Reviewed-by: Kristian Nielsen <knielsen@knielsen-hq.org>
The binary-parsing Format Description Event constructor did not
validate content length beyond the superclass `is_valid()` call.
When parsing a malformed FDE, to load the content fields added in
FDE v4 that are missing in this not-FDE, the parser constructor
would read from erroneous memory locations beyond the buffer.
If this did not outright crash the program, this would corrupt the FDE.
With the Format Description playing a critical role in determining how
to parse the events to follow, a corrupted FDE would also corrupt (or
trigger a crash in) the parsing of subsequent non-FDE events as well.
This commit fills in the validation with a FDE-specific guard.
It adds the FDE constant `ST_POST_HEADER_LEN_OFFSET` to assist
with comparing to the correct minimum size in the future.
(Both of these points are designed to merge with the
superclass’s guard as part of the MDEV-30128 merger).
Reviewed-by: Kristian Nielsen <knielsen@knielsen-hq.org>
A slave can crash when its master sends an event shorter than the
event's own header. A truncated heartbeat makes debug builds fail an
assertion in Binary_string::realloc_raw(). On 64-bit release builds it
stops the slave IO thread with an error message that omits the log
file name. If the master also declares an oversized common header in
its format description event, the slave allocates and fills nearly 4GB
instead. On 32-bit release builds it writes that error message past
the end of a stack buffer. An event under four bytes crashes every
build, because the slave checksums roughly 16 EiB and reads far past
the end of the packet.
Neither queue_event() nor the Heartbeat_log_event constructor bounded
the length the master sent. queue_event() handed that length to
event_checksum_test(), which subtracted the checksum length from it,
and the result wrapped on an event under four bytes. The constructor
computed the log file name length as event_len minus the header
lengths, and on a truncated heartbeat that subtraction wrapped to a
value near 4GB. queue_event() rejected the heartbeat as invalid, so
its error path appended the log file name to the error message using
the wrapped length.
Add the missing bound in both places. queue_event() now rejects an event
shorter than the common header before anything reads that header, and
the slave IO thread stops with an error. The Heartbeat_log_event
constructor now compares event_len against the combined header lengths
before the subtraction. A heartbeat that clears the first check but
still stops inside its headers keeps ident_len at 0 and log_ident at
NULL, so the error path appends nothing. That constructor check also
precedes the read of the extended log position, which previously ran on
a short event before any validation. A short event from the master now
stops the slave IO thread with an error instead of crashing the server.
Reviewed-by: Kristian Nielsen <knielsen@knielsen-hq.org>
Signed-off-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
A crafted DECIMAL user variable event makes bin2decimal() allocate and
copy a size taken entirely from two attacker-supplied bytes.
bin2decimal() has no parameter telling it how many bytes are available
(strings/decimal.c:1424):
int bin_size= decimal_bin_size(precision, scale);
sanity(to);
d_copy= (uchar*) my_alloca(bin_size);
memcpy(d_copy, from, bin_size);
decimal_bin_size() guards its inputs with DBUG_ASSERT(precision > 0)
and DBUG_ASSERT(scale <= precision) only, so release builds proceed with
anything. my_alloca is real alloca in this build.
Precision and scale arrive as val[0] and val[1]. Three call sites
pass them through:
* the applier, User_var_log_event::do_apply_event
(sql/log_event_server.cc:4563), which checks only val_len < 3
* pack_info (sql/log_event_server.cc:4356), which checks nothing, not
even val_len >= 3
* mysqlbinlog (sql/log_event_client.cc:2489), which checks nothing
The constructor does bound val_len against the event
(sql/log_event.cc:3064), so val_len is honest, but nothing ties
decimal_bin_size(val[0], val[1]) to it. val_len of 3 with a precision
of 65 already asks for 30 bytes where 1 is available.
Fixed by adding guards against malformed precision, scale and
val_len.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* restore result=NULL after free in mysqlshow.c (to prevent double-free)
* remove DBUG_EVALUATE_IF again
* remove useless 'stop slave' from a test
* restore charset encoding in mysqltest.{test,result}
* correct error for unparseable BINLOG data is ER_SYNTAX_ERROR now
* restore old libfmt on rhel-7, gcc 4.8.5 cannot compile anything newer
Add bounds checks to prevent out-of-bounds memory reads when parsing
Table_map_log_event optional metadata and constructor fields.
The checks prevent OOB reads but do not attempt to throw errors for
every possible corruption. Replication fundamentally assumes a trusted
source with valid data. The goal here is hardening against accidental
corruption (truncated events, garbled metadata lengths). On detecting
corruption, the slave either falls back gracefully to positional column
mapping (parse_* functions) or rejects the event entirely (constructor),
rather than crashing or leaking heap contents via OOB reads.
parse_* functions: Add if (unlikely(p > end)) return after every
net_field_length() call. Ensures decoded values are validated before
being stored to struct fields, preventing information disclosure from
OOB reads. On early return, the slave gracefully falls back to
positional column mapping.
Table_map_log_event constructor: Add bounds checks before memcpy calls
for m_colcnt, m_field_metadata_size, and m_null_bits. On overflow, frees
allocated memory and sets m_memory=NULL causing is_valid() to return
false. Adds read-side DBUG_EXECUTE_IF injection points for testing.
Functions fixed:
- parse_default_charset
- parse_column_charset
- parse_column_name
- parse_set_str_value
- parse_geometry_type
- parse_simple_pk
- parse_pk_with_prefix
- Table_map_log_event constructor (m_colcnt, m_field_metadata, m_null_bits)
MTR test rpl_table_map_log_event_overflow covers all affected code paths
with 9 test cases using debug injection.
This patch to mysqldump is only merged partially:
commit 666ff1db36
Author: Monty <monty@mariadb.org>
Date: Fri May 15 16:39:35 2026 +0300
MDEV-39519 Change mariadb-dump to use SHOW REPLICA STATUS instead of SHOW SLAVE STATUS
The functionality for --dump-slave cannot be merged as the underlying
functionality was removed in this commit:
commit 21c940d599
Author: ParadoxV5 <paradox.ver5@gmail.com>
Date: Thu Dec 4 21:22:20 2025 -0700
MDEV-37146 mariadb-dump: Replace SHOW SLAVE STATUS usage with information_schema.SLAVE_STATUS
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
If the first Partial_rows_log_event in a group provides an
original_event_size that is smaller than the actual size of the
underlying Rows_log_event, the slave will not allocate enough memory
to hold the actual Rows_log_event content, but still try to copy
that much memory, resulting in the slave crashing.
This patch adds validation checking for this condition, and will error
with code ER_PARTIAL_ROWS_LOG_EVENT_BAD_STREAM when detected.
Additionally:
* A new validation is added to ensure that the slave supports
re-constructing the size of the original event on the master (i.e. if
repliating from a 64-bit to a 32-bit system). A new test is added to
show the error message is correct. Note that the error code used here is
ER_OUTOFMEMORY, and that isn't exactly correct; but the actual error
isn't practical enough to warrant its own new error code.
* The error message for ER_PARTIAL_ROWS_LOG_EVENT_BAD_STREAM
is generalized to better support various types of broken streams. The
previous message was hard-coded to be about detecting out-of-sequence
fragments. The new message is updated to allow general "found %s" and
"expected %s" conditions.
Signed-off-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
Setting slave_skip_errors to 1677 (ER_SLAVE_CONVERSION_FAILED) doesn't work
as intended. The slave aborts replication on column mismatch. This fix
makes it log a warning and set the problematic field to the default
value gracefully.
Reviewed-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
MDEV-29804: Query_log_event::pack_info() was displaying inverted values
for foreign_key_checks, check_constraint_checks and unique_checks flags
in SHOW BINLOG EVENTS.
MDEV-32447: check_constraint_checks was not included in
OPTIONS_WRITTEN_TO_BINLOG, causing replication to break when
check_constraint_checks=OFF was used in non-ROW binlog formats.
Reviewed-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
Reviewed-by: Georgi Kodinov <joro@mariadb.org>
Remove the unscoped #pragma GCC diagnostic ignored
"-Wdeprecated-declarations" from include/violite.h and replace all
sprintf/vsprintf calls with snprintf/vsnprintf across the codebase.
Where possible, pass actual buffer sizes as function parameters
instead of using hardcoded constants or recomputing strlen().
Notable API changes:
- get_date(): added to_len parameter for buffer size
- nice_time(), end_timer() in client/mysql.cc: added buff_size
- generate_new_name(): added name_size parameter
- calc_md5(): added buffer_size parameter (fixes off-by-one)
- MD5_HASH_TO_STRING macro: added _size parameter
- Added MYSQL_UDF_MAX_RESULT_LENGTH constant in mysql_com.h
Also fixed a pre-existing bug in sql/rpl_mi.cc where the ellipsis
"..." was written to the wrong buffer (dbuff instead of buff).
Vendored code is excluded: extra/readline, extra/wolfssl,
wsrep-lib, libmariadb, storage/mroonga/vendor, zlib.
This affects row-based replication for UPDATE/DELETE on tables with no
primary key (and no non-NULL unique index).
There was a typo / incorrect merge from the following commit:
commit 3abce27e9d
Author: unknown <knielsen@knielsen-hq.org>
Date: Mon Dec 27 22:37:37 2010 +0100
Merge Percona patch row_based_replication_without_primary_key.patch into MariaDB.
Instead of referencing the index selected for applying the event, part
of the code was referencing the first index (in internal order) in the
table. This is code that checks if an index lookup is unique, or if it
requires a range scan. This could lead the code to incorrectly do a
unique lookup instead of comparing the pre-image against each row in a
range scan, thus applying the event to the wrong row and causing
replication to diverge.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
Fix rpl suite tests added by MDEV-25039.
rpl_foreign_key_lock_table_insert.test is removed altogether because it
is unclear what the purpose of the test is. The changes of the patch
were done on the slave, yet all operations in the test were done on the
master. Nothing different could happen on the slave because it is
configured to be serial, so all transactions would run sequentially
anyway, and no validations were performed.
rpl_foreign_key_ddl_insert.test was renamed to
rpl_row_foreign_key_mdl.test and the test itself was re-written to be
a minimal test case to ensure that MDL locking behavior is different
pre- and post- patch. A few problems with the original test:
* No foreign-key locking was done on the slave because the table
engine was not InnoDB.
* rpl_fk_ddl.inc had inconsistent validation checking. I.e., the child
query validation checks were done on the master (which is incorrect)
and because the slave was configured to be serial, the two
transactions could not run concurrently on the slave anyway.
Issue:
On galera write node INSERT statements does not acquire MDL locks on it's all child
tables and thereby wsrep certification keys are also added for limited tables, but
on applier nodes it does acquire MDL locks for all child tables. This can result
into MDL BF-BF conflict on applier node when transactions referring to parent and
child tables are executed concurrently. For example:
Tables with foreign keys: t1<-t2<-t3<-t4
Conflicting transactions: INSERT t1 and DROP TABLE t4
Wsrep certification keys taken on write node:
- for INSERT t1: t1 and t2
- for DROP TABLE t4: t4
On applier node MDL BF-BF conflict happened between two transaction because
MDL locks on t1, t2, t3 and t4 were taken for INSERT t1, which conflicted
with MDL lock on t4 taken by DROP TABLE t4.
The Wsrep certification keys helps in resolving this MDL BF-BF conflict by
prioritizing and scheduling concurrent transactions. But to generate Wsrep
certification keys it needs to open and take MDL locks on all the child tables.
On applier nodes Write_rows event is implicitly a REPLACE, deleting all conflicting
rows which can cause cascading FK actions and locks on foreign key children tables.
Solution:
For Galera applier nodes the Write_rows event is considered pure INSERT
which will never cause cascading FK actions and locks on foreign key children tables.
Don't pass uninitialized values in function call. MSAN complains about this
(even when the called function never accesses the uninitialized values, and
even when the function is constexpr .oO).
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
When replicating tables with different structures between a master and
slave (via binlog_row_metadata=FULL, i.e. MDEV-36290), if the table
didn't have any keys, replication could break from a
Can't find record in '<table>', Error_code: 1032;
error. This was due to the table's internal tracking of provided values
(i.e. TABLE::has_value_set) persisting across transactions. I.e., when
applying a new row event, the has_value_set bitset would start in a
state indicative of the last event. Then, if the new row event required
first finding a row, the has_value_set could represent a larger bitset
than what was actually unpacked from the master. More specifically, if
the last row event inserted something into the table, where slave-only
columns would be given default/auto-generated values, these would
update the TABLE::has_value_set. However, when the next row event would
come in, those auto-generated values from the last transaction would
not be present, but the row-search would include them to look-up the
row, and not be able to find anything.
This is fixed by resetting the table's internal state of provided
values (TABLE::has_value_set) before unpacking the row data. Doing so
revealed a bug where unpacked fields which explicitly provided NULL
values would skip indicating that an explicit value was provided
(and thereby couldn't be found by find_row()). To fix this, the
slave-side field will always call has_explicit_value() if it is
present in the packed data.
Test case based on work from: Deepthi Sreenivas <deepthi.sreenivas@mariadb.com>
Signed-off-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
Reviewed-by: Monty <monty@mariadb.com>
One can have data loss in multi-master setups when 1) both masters
update the same table, 2) ALTER TABLE is run on one master which
re-arranges the column ordering, and 3) transactions are binlogged
in ROW binlog_format.
This is because the slave assumes that all columns are in the same
order on the master and slave and all columns on the master also
exists on the slave. This happens even if binlog_row_metadata=FULL is
used. If this is not the case, this will lead to silent data loss.
A new option for slave_type_conversions bit field,
ERROR_IF_MISSING_FIELD, has been added, along with a new error,
ER_SLAVE_INCOMPATIBLE_TABLE_DEF. This allows the user to define if
the slave should abort replication if it is missing some field that
existed on the master. The option is off by default to keep things
compatible with earlier versions.
If a field is missing on the slave and log_warnings >= 1, a warning
will be logged to the error log.
This patch fixes this, when binlog_row_metadata=FULL is used on the
master, by mapping fields with identical names on the master and slave.
If slave has fields that does not exist in the row event, these will
be set to their default value.
The main idea is that we added two conversion tables:
m_tabledef.master_to_slave_map[master_column_index] -> slave_column_index
and m_tabledef.master_to_slave_error[master_column_index] which contains
an error number if the master_column does not exist on the slave or
it is not possible to convert the master data to the slave column.
master_to_slave_error[#] contains 0 if the column exists and is compatible.
General code changes:
- Instead of looping over row fields in the order of slave table
we are instead looping over fields in the order of the binary log.
- We are using table->write_set to know which fields should be updated
on the slave. This is reflected in unpack_row
- We are calling TABLE::mark_columns_per_binlog_row_image() to ensure
that rpl_write_set is properly set. This is needed if the slave also
is doing binary logging.
- Before replication aborted if the master and slave tables were too
different. Now replication is only aborted if the row actually uses
columns that does not exists on the slave (and ERROR_IF_MISSING_FIELD
is used) or uses columns that cannot be converted.
- Instead of giving errors in compatible_with(), used when table is
accessed by first the row event, we are instead giving errors
when we examine a row event and notice that it is accessing
a not existing or not compatible field.
Other code changes:
- Removed conv_table argument from compatible_with() and store it
directly in RPL_TABLE_LIST->m_conv_table
- table_def::compatible_with() returns now 1 on error (not 0).
- Remove m_width and skip arguments from prepare_record() as we are
now using table->write_set() to check which elements need a default
value.
- Moved DBUG_ENTER() to it's proper place (after variable
declarations) in a few functions.
- Some changes in unpack_row():
- Replaced null_mask and null_ptr with an indexed bit check for
simplicity.
- Removed check of rgi == null and table_found which never worked.
- Updated comments to reflect current code.
- Indentation changes as the code now uses 'continue' instead of
'if-else' in the main loop.
- The code to throw away 'extra master fields' is not needed as we
are now looping over fields in binary log, not over fields in
slave table.
- Simplified get_table_data(TABLE *table_arg) by returning found
table_list.
- Errors for row events are now initialized in compatible_with(),
checked in check_wrong_column_usage() and reported in
give_compatibility_error().
Co-authored-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
This patch solves two problems:
1. Rows log events cannot be transmitted to the slave if their
size exceeds slave_max_packet_size (max 1GB at the time of
writing this patch, i.e. MariaDB 12.3)
2. Rows log events cannot be binlogged if they are larger than
4GB because the common binlog event header field event_len is
32-bits.
This patch adds support for fragmenting large Rows_log_events
through a new event type, Partial_rows_log_event. When any given
instantiation of a Rows_log_event (e.g. Write_rows_log_event, etc)
is too large to be sent to a replica (i.e. larger than the value
slave_max_allowed_packet, as configured on a replica), then the rows
event must be fragmented into sub-events (i.e.
Partial_rows_log_events), so the event can be transmitted to the
replica. The replica will then take the content of each of these
Partial_rows_log_events, and join them together into a large
Rows_log_event to be executed as normal. Partial_rows_log_events
are written to the binary log sequentially, and the replica assembles
the events in the order they are binlogged.
To control the size of each Partial_rows_log_event, a new system
variable is added: binlog_row_event_fragment_threshold. All
Partial_rows_log_events within the same group will have this size,
except for the last, which will take up the remaining length of the
underlying Rows_log_event.
Each Partial_rows_log_event stores its sequence number (seq_no) in the
overall series of fragments, the total number of fragments needed to
re-assemble the Rows_log_event (total_fragments), a uchar for flags for
embedding extra data, and any additional data as specified by the
flags. Currently, only the first event in a grouping will have
additional data: it will set the first bit in the flags
(FL_ORIG_EVENT_SIZE) to indicate it will be storing the total size of
the underlying Rows_log_event.
The cached Rows_log_event data is fragmented into
Partial_rows_log_events as follows. The primary will still generate a
Rows_log_event to write to the binlog; however, during the actual
writing process, the raw data of the rows event is split into
fragments, each covering some continuous section of the rows data. A
Partial_rows_log_event is created for each continuous section, and the
Partial_rows_log_events are written sequentially in-place of the
too-large Rows_log_event. The original data to be fragmented will
include a header and data header; however, will not include a checksum,
as each Partial_rows_log_event will have a checksum for validation, as
well as a sequence_number and total number of fragments to ensure all
fragments are present.
The re-assembly and execution of the original Rows_log_event on the
replica happens in Partial_rows_log_event::do_apply_event(). The rgi
is extended with a memory buffer that holds all data for the
original Rows_log_event. As each Partial_rows_log_event is
ingested, its Rows_log_event content is appended to this memory
buffer. Once the last fragment has added its content, a new
Rows_log_event is created using that buffer, and executed.
A new error message is added to indicate that the slave has received an
invalid stream of Partial_rows_log_events:
ER_PARTIAL_ROWS_LOG_EVENT_BAD_STREAM.
Note this commit only adds the server logic for fragmentic and
assembling events, the client logic (mysqlbinlog) is in the next
commit.
Alternative designs considered were:
1. Alternative 1: Change the master-slave communication protocol
such that the master would send events in chunks of size
slave_max_allowed_packet. Though this is still a valid idea,
and would solve the first problem described in this commit
message, this would still leave the limitation that
Rows_log_events could not exceed 4GB. Eventually, this change
should still be addressed (e.g. in MDEV-37853), and for users
of binlog-in-engine (MDEV-34705) which supports out-of-band
binlogging of events, this MDEV-32570 work will be superseded.
2. Alternative 2: Create a generic “Container_log_event” with the
intention to embed various other types of event data for
various purposes, with flags that describe the purpose of a
given container. This seemed overboard, as there is already a
generic Log_event framework that provides the necessary
abstractions to fragment/reassemble events without adding in
extra abstractions.
3. Alternative 3: Add a flag to Rows_log_event with semantics to
overwrite/correct the event_len field of the common event
header to use a 64-bit field stored in the data_header of the
Rows_log_event; and also do alternative 1, so the master would
send the large (> 4GB) rows event in chunks. This approach
would add too much complexity (changing both the binlogging
and transport layer); as well as introduce inconsistency to
the event definition (event_len and next_event_position would
no longer have consistent meanings).
Reviewed-by: Hemant Dangi <hemant.dangi@mariadb.com>
Acked-by: Kristian Nielsen <knielsen@knielsen-hq.org>
Signed-off-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
To prepare for MDEV-32570, the Rows_log_event::write_data_body() is
split into two functions:
1. write_data_body_metadata(), which writes the context of the rows
data (i.e. width, cols, and cols_ai), which will only be written
for the first event fragment.
2. write_data_body_rows(), which allows the writing of the rows data
to be fragmented by parameterizing the writing of the rows data to
start at a given offset and only write a certain length. This lets
each row fragment (for MDEV-32570) to contain a chunk of the rows
data
Signed-off-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
