A checksum error in
`mariadb-binlog --verify-binlog-checksum --force-read` previously
resulted in both an error message and an Unknown event substitute,
the latter of which failed an assertion in debug builds or
became forgotten (memory leak) in non-debug (release) builds.
Since `mariadb-binlog --force-read` outputs “Unknown event”s
rather than errors in other invalid event cases,
this commit removes the error status from this situation to match.
Reviewed-by: Kristian Nielsen <knielsen@knielsen-hq.org>
This commit reörders code so the checksum is populated after
`mariadb-binlog --force` generates `Unknown_log_event` substitutes.
Previously, `mariadb-binlog --force` inconsistently omitted those
checksums from the output even if the checksum is presumably usable.
After merging to 11.4 (MDEV-31273), this commit will also fix «
MDEV-40542 MSAN use-of-uninitialized-value on Unknown_log_event::read_checksum_alg
», which was exposed by MDEV-31273’s removal
of the base `Log_event::checksum_alg` field.
Reviewed-by: Kristian Nielsen <knielsen@knielsen-hq.org>
MariaDB recognizes Version 2 Rows Events from MySQL, including the
format of the “extra data” field added in this version. (MDEV-5115)
When parsing this extra data according to the format, whether this data
has sufficient length was only checked by assertions in the
`Rows_log_event` constructor and the `mariadb-binlog --verbose` printer.
When parsing an event with malformed extra data, these assertions
* would straight up terminate the program in debug builds.
* were stripped in non-debug (release) builds.
This would render the parser defenseless to reading from erroneous
memory locations outside of the containing event, which will either
crash the program or, for `mariadb-binlog --verbose`, snapshot the
running memory to be exposed when outputting the event.
This commit replaces those assertions with an actual validity check.
Since MariaDB does not generate v2 Rows Events,
the included test uses a handcrafted binlog file.
Reviewed-by: Kristian Nielsen <knielsen@knielsen-hq.org>
Neither the binary-parsing Format Description Event constructor
nor the constructor-bypassing `get_checksum_alg()` function
validated the content length of the passed event buffer.
If they receive an FDE with undersized contents,
they would obtain corrupt results from an erronous memory location,
if not outright crash the program with that memory error.
This commit fixes both sites by adding content length checks.
Because the goal is not to solve the existence of two binary parsers,
`get_checksum_alg()` receives a check duplicated from the Format
Description constructor and is no longer a function that never errors.
Note, it is implementation detail that `get_checksum_alg()`’s
fix catches the invalidity before any code reaches the
parser-contructor’s fix, though the latter would come
to effect if we refactor `get_checksum_alg()` away.
Reviewed-by: Kristian Nielsen <knielsen@knielsen-hq.org>
The binary-parsing Format Description Event constructor did not
validate content length beyond the superclass `is_valid()` call.
When parsing a malformed FDE, to load the content fields added in
FDE v4 that are missing in this not-FDE, the parser constructor
would read from erroneous memory locations beyond the buffer.
If this did not outright crash the program, this would corrupt the FDE.
With the Format Description playing a critical role in determining how
to parse the events to follow, a corrupted FDE would also corrupt (or
trigger a crash in) the parsing of subsequent non-FDE events as well.
This commit fills in the validation with a FDE-specific guard.
It adds the FDE constant `ST_POST_HEADER_LEN_OFFSET` to assist
with comparing to the correct minimum size in the future.
(Both of these points are designed to merge with the
superclass’s guard as part of the MDEV-30128 merger).
Reviewed-by: Kristian Nielsen <knielsen@knielsen-hq.org>
A slave can crash or leak heap content when its master sends a Table_map
event that declares columns whose types need more field metadata than
the event carries. Building its description of the table, the slave
reads past the end of the field metadata the event holds. The bytes
beyond the buffer are decoded as column metadata.
A Table_map event carries a column count, one type byte per column,
and a block of field metadata. A column's type fixes how many metadata
bytes it consumes: two for the string, enum, set, bit, varchar, and
decimal types, one for the blob, float, and high precision temporal
types, none for the rest. The master writes exactly that many bytes
per column, so the metadata block a well formed event carries is the
sum over its columns. The slave sizes the metadata block from the
event and, while building its table_def, walks the columns in order
and reads each type's metadata bytes from that block.
The slave checked only that the metadata block was no larger than two
bytes per column. It never checked the block against the column types
themselves. table_def() then walked every declared column and indexed
the metadata block for each, so a column whose type needed metadata
the block did not hold read past the end of it. A column count large
enough carried that read far past the allocation.
The Table_map reader now sums the metadata the declared column types
consume and rejects the event when the block is smaller than that
sum, so is_valid() reports the event as invalid and the SQL thread
stops with a relay log read failure before any column is decoded. A
block larger than the sum is still accepted, because every consumer
reads only the bytes the types call for. A column type this slave
does not recognize counts as zero in the sum, so Table_map events
from a newer master using such a type still parse. A Table_map whose
metadata cannot cover its column types now stops the slave with an
error instead of reading past the buffer.
Note that the regression test does not crash a regular debug build;
an ASAN build is required to show the invalid memory access. The 24
column table map it injects overruns the metadata allocation by a
bounded amount that lands in mapped heap, which a debug build reads
without faulting and then stops the SQL thread with
ER_SLAVE_CONVERSION_FAILED on the column type mismatch.
Reviewed-by: Kristian Nielsen <knielsen@knielsen-hq.org>
Signed-off-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
A slave can crash when its master sends a Table_map event whose table
name is longer than an identifier can be. The SQL thread applying the
event copies the name into a fixed buffer and overruns it, and the SQL
thread crashes.
A Table_map event carries its database and table names each behind a
one-byte length. The constructor that parses the event sizes a buffer
for each name from its declared length, and do_apply_event() copies
both names out of the event into NAME_LEN+1 byte buffers with strmov().
strmov() copies up to the terminating null. A real identifier is at
most NAME_LEN bytes, so the name fits the buffer and carries that null.
The constructor never bounded the two lengths. Each length is read as a
byte, so each can reach 255. The constructor copied each name with
strncpy() over the declared length, which writes no terminating null
when the source holds none. do_apply_event() then copied the name into
its NAME_LEN+1 byte buffer with strmov(), which ran past the buffer
until it reached a null elsewhere in the heap. pack_info() read the
same names with %s for SHOW RELAYLOG EVENTS, past the buffer in the
same way.
Bound the lengths in the constructor. It now rejects a Table_map event
whose declared database name, or whose declared table name, exceeds
NAME_LEN, before it sizes the name buffers, so is_valid() returns
false. It also terminates each name it copies, so a source that holds
no null does not leave the name unterminated. A master naming a table
with an oversized name now stops the slave's SQL thread with
ER_SLAVE_RELAY_LOG_READ_FAILURE instead of crashing it.
Reviewed-by: Kristian Nielsen <knielsen@knielsen-hq.org>
Signed-off-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
A replica can crash when a compressed event from its master carries a
corrupted uncompressed length. A compressed Query event whose length
is near 4GB overwrites the IO thread's stack with the event's content.
A compressed rows event of the same shape makes the replica ask for an
allocation of 4GB. A debug build that gets that memory then fails an
assertion in binlog_buf_uncompress(). On a build where a ulong is 32
bits wide, both events overwrite the IO thread's stack.
The IO thread uncompresses a compressed event before writing the event
to the relay log, and offers the uncompress function 4096 bytes of its
own stack to hold the result. The function computes how large the event
will be once uncompressed, and that size decides where the uncompressed
event goes. An event that fits in the stack buffer is uncompressed
there. A larger event is uncompressed into an allocation the function
makes at that size. With either buffer, the function tells zlib that
the room available is the uncompressed length the event declared.
query_event_uncompress() never bounded the length read out of the
event. The function added the header length to that value, rounded the
sum up for alignment, and cast the result to uint32. The cast dropped
the high bits of a sum above 4GB, so a length near 4GB produced a size
of a few dozen bytes. That size fit the stack buffer, so content of any
size went onto the IO thread's stack. row_log_event_uncompress()
computes that sum in a ulong, which wraps where a ulong is 32 bits
wide. The Query_compressed_log_event constructor and
Rows_log_event::uncompress_buf() size their allocations from the same
unbounded length.
Reject an uncompressed length above MAX_MAX_ALLOWED_PACKET at every
point the length is read, before any size is computed from it. No
master writes a larger length, because max_allowed_packet is capped at
1GB. query_event_uncompress() also keeps that size in a size_t, so the
allocation is made from the whole computed value. Both functions return
the error code for the IO thread to report, so a length over the limit
stops the IO thread with ER_TOO_BIG_FOR_UNCOMPRESS rather than the
generic uncompress error. A compressed event with a corrupted
uncompressed length now stops the replica's IO thread with an error
instead of crashing the replica.
Alexandra Raveala wrote the original patch and its first regression
test, contributed through PR 5413. Brandon Nesterenko saw the work
through for the 10.6 release, replacing the test with one that makes
the master write the corrupt length and reworking how the IO thread
reports the error.
Co-authored-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
Reviewed-by: Kristian Nielsen <knielsen@knielsen-hq.org>
Add bounds checks to prevent out-of-bounds memory reads when parsing
Table_map_log_event optional metadata and constructor fields.
The checks prevent OOB reads but do not attempt to throw errors for
every possible corruption. Replication fundamentally assumes a trusted
source with valid data. The goal here is hardening against accidental
corruption (truncated events, garbled metadata lengths). On detecting
corruption, the slave either falls back gracefully to positional column
mapping (parse_* functions) or rejects the event entirely (constructor),
rather than crashing or leaking heap contents via OOB reads.
parse_* functions: Add if (unlikely(p > end)) return after every
net_field_length() call. Ensures decoded values are validated before
being stored to struct fields, preventing information disclosure from
OOB reads. On early return, the slave gracefully falls back to
positional column mapping.
Table_map_log_event constructor: Add bounds checks before memcpy calls
for m_colcnt, m_field_metadata_size, and m_null_bits. On overflow, frees
allocated memory and sets m_memory=NULL causing is_valid() to return
false. Adds read-side DBUG_EXECUTE_IF injection points for testing.
Functions fixed:
- parse_default_charset
- parse_column_charset
- parse_column_name
- parse_set_str_value
- parse_geometry_type
- parse_simple_pk
- parse_pk_with_prefix
- Table_map_log_event constructor (m_colcnt, m_field_metadata, m_null_bits)
MTR test rpl_table_map_log_event_overflow covers all affected code paths
with 9 test cases using debug injection.
MDEV-29804: Query_log_event::pack_info() was displaying inverted values
for foreign_key_checks, check_constraint_checks and unique_checks flags
in SHOW BINLOG EVENTS.
MDEV-32447: check_constraint_checks was not included in
OPTIONS_WRITTEN_TO_BINLOG, causing replication to break when
check_constraint_checks=OFF was used in non-ROW binlog formats.
Reviewed-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
Reviewed-by: Georgi Kodinov <joro@mariadb.org>
Remove the unscoped #pragma GCC diagnostic ignored
"-Wdeprecated-declarations" from include/violite.h and replace all
sprintf/vsprintf calls with snprintf/vsnprintf across the codebase.
Where possible, pass actual buffer sizes as function parameters
instead of using hardcoded constants or recomputing strlen().
Notable API changes:
- get_date(): added to_len parameter for buffer size
- nice_time(), end_timer() in client/mysql.cc: added buff_size
- generate_new_name(): added name_size parameter
- calc_md5(): added buffer_size parameter (fixes off-by-one)
- MD5_HASH_TO_STRING macro: added _size parameter
- Added MYSQL_UDF_MAX_RESULT_LENGTH constant in mysql_com.h
Also fixed a pre-existing bug in sql/rpl_mi.cc where the ellipsis
"..." was written to the wrong buffer (dbuff instead of buff).
Vendored code is excluded: extra/readline, extra/wolfssl,
wsrep-lib, libmariadb, storage/mroonga/vendor, zlib.
Parameter optional_meteradata_len to Optional_metadata_fields should
have been a size_t rather than uint.
Signed-off-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
- Less memory allocations (less fragmentation), less memory usage,
faster performance, less code (both source and executed).
- Added option used by RPL_TABLE_LIST::create_column_mapping() to
only parse and allocate column names.
- Avoid duplicate memory allocations of column names.
- Some protection for out-of-memory (when allocating field names).
- More work is needed to remove all allocation problems in
Optional_metadata_field().
- All allocated strings are ending with \0, which makes code safer.
This patch solves two problems:
1. Rows log events cannot be transmitted to the slave if their
size exceeds slave_max_packet_size (max 1GB at the time of
writing this patch, i.e. MariaDB 12.3)
2. Rows log events cannot be binlogged if they are larger than
4GB because the common binlog event header field event_len is
32-bits.
This patch adds support for fragmenting large Rows_log_events
through a new event type, Partial_rows_log_event. When any given
instantiation of a Rows_log_event (e.g. Write_rows_log_event, etc)
is too large to be sent to a replica (i.e. larger than the value
slave_max_allowed_packet, as configured on a replica), then the rows
event must be fragmented into sub-events (i.e.
Partial_rows_log_events), so the event can be transmitted to the
replica. The replica will then take the content of each of these
Partial_rows_log_events, and join them together into a large
Rows_log_event to be executed as normal. Partial_rows_log_events
are written to the binary log sequentially, and the replica assembles
the events in the order they are binlogged.
To control the size of each Partial_rows_log_event, a new system
variable is added: binlog_row_event_fragment_threshold. All
Partial_rows_log_events within the same group will have this size,
except for the last, which will take up the remaining length of the
underlying Rows_log_event.
Each Partial_rows_log_event stores its sequence number (seq_no) in the
overall series of fragments, the total number of fragments needed to
re-assemble the Rows_log_event (total_fragments), a uchar for flags for
embedding extra data, and any additional data as specified by the
flags. Currently, only the first event in a grouping will have
additional data: it will set the first bit in the flags
(FL_ORIG_EVENT_SIZE) to indicate it will be storing the total size of
the underlying Rows_log_event.
The cached Rows_log_event data is fragmented into
Partial_rows_log_events as follows. The primary will still generate a
Rows_log_event to write to the binlog; however, during the actual
writing process, the raw data of the rows event is split into
fragments, each covering some continuous section of the rows data. A
Partial_rows_log_event is created for each continuous section, and the
Partial_rows_log_events are written sequentially in-place of the
too-large Rows_log_event. The original data to be fragmented will
include a header and data header; however, will not include a checksum,
as each Partial_rows_log_event will have a checksum for validation, as
well as a sequence_number and total number of fragments to ensure all
fragments are present.
The re-assembly and execution of the original Rows_log_event on the
replica happens in Partial_rows_log_event::do_apply_event(). The rgi
is extended with a memory buffer that holds all data for the
original Rows_log_event. As each Partial_rows_log_event is
ingested, its Rows_log_event content is appended to this memory
buffer. Once the last fragment has added its content, a new
Rows_log_event is created using that buffer, and executed.
A new error message is added to indicate that the slave has received an
invalid stream of Partial_rows_log_events:
ER_PARTIAL_ROWS_LOG_EVENT_BAD_STREAM.
Note this commit only adds the server logic for fragmentic and
assembling events, the client logic (mysqlbinlog) is in the next
commit.
Alternative designs considered were:
1. Alternative 1: Change the master-slave communication protocol
such that the master would send events in chunks of size
slave_max_allowed_packet. Though this is still a valid idea,
and would solve the first problem described in this commit
message, this would still leave the limitation that
Rows_log_events could not exceed 4GB. Eventually, this change
should still be addressed (e.g. in MDEV-37853), and for users
of binlog-in-engine (MDEV-34705) which supports out-of-band
binlogging of events, this MDEV-32570 work will be superseded.
2. Alternative 2: Create a generic “Container_log_event” with the
intention to embed various other types of event data for
various purposes, with flags that describe the purpose of a
given container. This seemed overboard, as there is already a
generic Log_event framework that provides the necessary
abstractions to fragment/reassemble events without adding in
extra abstractions.
3. Alternative 3: Add a flag to Rows_log_event with semantics to
overwrite/correct the event_len field of the common event
header to use a 64-bit field stored in the data_header of the
Rows_log_event; and also do alternative 1, so the master would
send the large (> 4GB) rows event in chunks. This approach
would add too much complexity (changing both the binlogging
and transport layer); as well as introduce inconsistency to
the event definition (event_len and next_event_position would
no longer have consistent meanings).
Reviewed-by: Hemant Dangi <hemant.dangi@mariadb.com>
Acked-by: Kristian Nielsen <knielsen@knielsen-hq.org>
Signed-off-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
Preparation for MDEV-32570. When fragmenting a large row event into
multiple smaller fragment events, each fragment event will have its own
checksum attached, thereby negating the need to also store the checksum
of the overall large event.
The existing code assumes all events will always have checksums, but
this won't be true for the rows events that are re-assembled on the
replicas. This patch prepares for this by splitting the logic which
reads in and creates Log_event objects into two pieces, one which
handles the checksum validation; and the other which reads the raw
event data (without the checksum) and creates the object.
All existing code is unchanged which uses the checksum-assuming version
of the event reader. MDEV-32570 will be the only case which will bypass
the checksum logic, and will directly create its rows log events from
memory without validating checksums (as the checksums will have already
been validated by each individual fragment event).
Signed-off-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
The functions to read row log events from buffers used 32-bit numeric
types to hold the length of the buffer. MDEV-32570 will add in support
for row events larger than what 32-bits can represent, and so this
patch changes the type for the length variable to size_t so larger
row events can be created from raw memory.
Signed-off-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
Implement an improved binlog implementation that is integrated into
the storage engine. The new implementation is enabled with the
--binlog-storage-engine option. Initially the InnoDB storage engine
implements the binlog.
Integrating the binlog in the storage engine improves performance,
since it makes the InnoDB redo log the single source of truth and
avoids the need for expensive two-phase commit between binlog and
engine. It also makes it possible to disable durability (set
--innodb-flush-log-at-trx-commit=0) to further improve performance,
while still preserving the ability to recover the binlog and database
into a consistent state after a crash.
The new binlog implementation also greatly improves the internal
design and implementation of the binlog, and enables future
enhancements for replication.
This is a squash of the original 11.4-based patch series.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
compiling mariadb from a git tree
Fixed sprintf deprecation warnings compiling on MacOSX.
Replaced some sprintf calls with equivalent snprintf calls,
enough so that "normal" compile on MacOSX (as documented
in the docs) completes without warnings.
(Re)ënable the `ATTRIBUTE_FORMAT` on `my_dbug.h`’s `_db_doprnt_`
(better known by its frontend `DBUG_PRINT`) and `ma_recovery_util.h`’s
`tprint` & `eprint` to leverage GCC `-Wformat` checking
c4bf4b7aef introduced `WAITING_FOR_BUGFIX_TO_VSPRINTF` to conditionally
(read: temporarily) disable their `ATTRIBUTE_FORMAT`s.
Whatever that bug was aside, MDEV-21978 Zulip suggested that the
preference for `%b` was probably intended, although c52e62a76f
reverted the one in `storage/maria/ma_recovery.c` back to `%s`.
All extension migrations (e.g., `%b` ➡ `%sB`)
in this commit were on `DBUG_PRINT`.
* rpl.rpl_system_versioning_partitions updated for MDEV-32188
* innodb.row_size_error_log_warnings_3 changed error for MDEV-33658
(checks are done in a different order)
In Log_event::read_log_event(), don't use IO_CACHE::error of the relay log's
IO_CACHE to signal an error back to the caller. When reading the active
relay log, this flag is also being used by the IO thread, and setting it can
randomly cause the IO thread to wrongly detect IO error on writing and
permanently disable the relay log.
This was seen sporadically in test case rpl.rpl_from_mysql80. The read
error set by the SQL thread in the IO_CACHE would be interpreted as a
write error by the IO thread, which would cause it to throw a fatal
error and close the relay log. And this would later cause CHANGE
MASTER to try to purge a closed relay log, resulting in nullptr crash.
SQL thread is not able to parse an event read from the relay log. This
can happen like here when replicating unknown events from a MySQL master,
potentially also for other reasons.
Also fix a mistake in my_b_flush_io_cache() introduced back in 2001
(fa09f2cd7e) where my_b_flush_io_cache() could wrongly return an error set
in IO_CACHE::error, even if the flush operation itself succeeded.
Also fix another sporadic failure in rpl.rpl_from_mysql80 where the outout
of MASTER_POS_WAIT() depended on timing of SQL and IO thread.
Reviewed-by: Monty <monty@mariadb.org>
Reviewed-by: Andrei Elkin <andrei.elkin@mariadb.com>
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
MDEV-29533 Crash when MariaDB is replica of MySQL 8.0
MySQL 8.0 has added the following new events in the MySQL binary log
PARTIAL_UPDATE_ROWS_EVENT
TRANSACTION_PAYLOAD_EVENT
HEARTBEAT_LOG_EVENT_V2
- PARTIAL_UPDATE_ROWS_EVENT is used by MySQL to generate update
statements using JSON_SET, JSON_REPLACE and JSON_REMOVE to make
update of JSON columns more efficient. These events can be
disabled by setting 'binlog-row-value-options=""'
- TRANSACTION_PAYLOAD_EVENT is used by MySQL to signal that a
row event is compressed. It an be disably by setting
'binlog_transaction_compression=0'.
- HEARTBEAT_LOG_EVENT_V2 is written to the binary log many times
per seconds. It can be ignored by the server.
What this patch does:
- If PARTIAL_UPDATE_ROWS_EVENT or TRANSACTION_PAYLOAD_EVENT is found,
the server will stop with an error message of how to disable the
MySQL server to generate such events.
- HEARTBEAT_LOG_EVENT_V2 events are ignored.
- mariadb-binlog will write the name of the new events.
- mariadb-binlog will stop if PARTIAL_UPDATE_ROWS_EVENT or
TRANSACTION_PAYLOAD_EVENT is found, unless --force is given.
- Fixes a crash in mariadb-binlog if a character set unknown to
MariaDB is found. (MDEV-29533)
From Kristian Nielsen:
- Add test case for MySQL 8.0 to MariaDB replication and fixed a
a small typo in post_header_len initialization.
Reviewer: knielsen@mariadb.org
