This commit rewrites the rpl_master_has_bug() mechanism to solve a problem
with invalid memory access. The rpl_master_has_bug() mechanism detects
certain bugs depending on the master version, and uses that to enable
specific work-arounds on the slave. The problem was that
rpl_master_has_bug() accessed Relay_log_info::description_event_for_exec
that is not valid to access from concurrent parallel replication worker
threads, only from the SQL driver thread. Thus it could use the wrong event
or access invalid/freed memory.
This patch instead computes a bitmask of detected bugs when the SQL driver
thread processes the format description event, and reads that bitmask with
an atomic load from the worker threads. The bitmask of bugs can only change
when the master restarts with a new version, and we do not replicate events
concurrently across a format description event from a master restart. Thus,
the bitmask is safe to read concurrently from the Relay_log_info object
without locking.
This also avoids an expensive match of each entry in the bug list against
the master server version done for every single call to
rpl_master_has_bug(), which could be quite expensive when done eg. per field
in row events as in Field_string::compatible_field_size().
Also remove redundant conditional in table_def::compatible_with().
Thanks to Andrei Elkin for the idea to safely read the bitmask
concurrently from the Relay_log_info.
Reviewed-by: Andrei Elkin <andrei.elkin@pp.inet.fi>
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
When finding the row to modify for a row event, and when not using
rnd_pos_by_record() to locate the row, the code would use
table->use_all_columns(), which makes the read_set and write_set point
to the table->s->all_set in the table share. This caused problems when
other code later modified bits in the read_set or write_set, which
ends up wrongly modifying the table share.
We can just use bitmap_set_all(table->read_set) to mark to read all
columns and leave the possibility to later change the bits as needed.
This code changes in this patch must be null-merged from 10.11 to
11.4, as there the problem is fixed differently.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
If the replication IO Thread receives a Rotate event following a Format
Description event (FDE) with no post-header length for Rotate events,
the Rotate event’s parser constructor indexes
the FDE’s post-header lengths array out of bounds.
This commit defends against this situation by checking before the
constructor that the FDE describes Rotate events as recognized at all.
In practice, because the Binlog Dump thread generates a Fake
`ROTATE_EVENT` **before** sending the FDE, it has pinned
Rotate events’ post-header length to 8 regardless of FDEs.
This fix solution considers that the FDE’s description should still be
respected, matching the constructor.
Reviewed-by: Kristian Nielsen <knielsen@knielsen-hq.org>
Neither the binary-parsing Format Description Event constructor
nor the constructor-bypassing `get_checksum_alg()` function
validated the content length of the passed event buffer.
If they receive an FDE with undersized contents,
they would obtain corrupt results from an erronous memory location,
if not outright crash the program with that memory error.
This commit fixes both sites by adding content length checks.
Because the goal is not to solve the existence of two binary parsers,
`get_checksum_alg()` receives a check duplicated from the Format
Description constructor and is no longer a function that never errors.
Note, it is implementation detail that `get_checksum_alg()`’s
fix catches the invalidity before any code reaches the
parser-contructor’s fix, though the latter would come
to effect if we refactor `get_checksum_alg()` away.
Reviewed-by: Kristian Nielsen <knielsen@knielsen-hq.org>
The binary-parsing Format Description Event constructor did not
validate content length beyond the superclass `is_valid()` call.
When parsing a malformed FDE, to load the content fields added in
FDE v4 that are missing in this not-FDE, the parser constructor
would read from erroneous memory locations beyond the buffer.
If this did not outright crash the program, this would corrupt the FDE.
With the Format Description playing a critical role in determining how
to parse the events to follow, a corrupted FDE would also corrupt (or
trigger a crash in) the parsing of subsequent non-FDE events as well.
This commit fills in the validation with a FDE-specific guard.
It adds the FDE constant `ST_POST_HEADER_LEN_OFFSET` to assist
with comparing to the correct minimum size in the future.
(Both of these points are designed to merge with the
superclass’s guard as part of the MDEV-30128 merger).
Reviewed-by: Kristian Nielsen <knielsen@knielsen-hq.org>
A slave can crash when its master sends an event shorter than the
event's own header. A truncated heartbeat makes debug builds fail an
assertion in Binary_string::realloc_raw(). On 64-bit release builds it
stops the slave IO thread with an error message that omits the log
file name. If the master also declares an oversized common header in
its format description event, the slave allocates and fills nearly 4GB
instead. On 32-bit release builds it writes that error message past
the end of a stack buffer. An event under four bytes crashes every
build, because the slave checksums roughly 16 EiB and reads far past
the end of the packet.
Neither queue_event() nor the Heartbeat_log_event constructor bounded
the length the master sent. queue_event() handed that length to
event_checksum_test(), which subtracted the checksum length from it,
and the result wrapped on an event under four bytes. The constructor
computed the log file name length as event_len minus the header
lengths, and on a truncated heartbeat that subtraction wrapped to a
value near 4GB. queue_event() rejected the heartbeat as invalid, so
its error path appended the log file name to the error message using
the wrapped length.
Add the missing bound in both places. queue_event() now rejects an event
shorter than the common header before anything reads that header, and
the slave IO thread stops with an error. The Heartbeat_log_event
constructor now compares event_len against the combined header lengths
before the subtraction. A heartbeat that clears the first check but
still stops inside its headers keeps ident_len at 0 and log_ident at
NULL, so the error path appends nothing. That constructor check also
precedes the read of the extended log position, which previously ran on
a short event before any validation. A short event from the master now
stops the slave IO thread with an error instead of crashing the server.
Reviewed-by: Kristian Nielsen <knielsen@knielsen-hq.org>
Signed-off-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
A crafted DECIMAL user variable event makes bin2decimal() allocate and
copy a size taken entirely from two attacker-supplied bytes.
bin2decimal() has no parameter telling it how many bytes are available
(strings/decimal.c:1424):
int bin_size= decimal_bin_size(precision, scale);
sanity(to);
d_copy= (uchar*) my_alloca(bin_size);
memcpy(d_copy, from, bin_size);
decimal_bin_size() guards its inputs with DBUG_ASSERT(precision > 0)
and DBUG_ASSERT(scale <= precision) only, so release builds proceed with
anything. my_alloca is real alloca in this build.
Precision and scale arrive as val[0] and val[1]. Three call sites
pass them through:
* the applier, User_var_log_event::do_apply_event
(sql/log_event_server.cc:4563), which checks only val_len < 3
* pack_info (sql/log_event_server.cc:4356), which checks nothing, not
even val_len >= 3
* mysqlbinlog (sql/log_event_client.cc:2489), which checks nothing
The constructor does bound val_len against the event
(sql/log_event.cc:3064), so val_len is honest, but nothing ties
decimal_bin_size(val[0], val[1]) to it. val_len of 3 with a precision
of 65 already asks for 30 bytes where 1 is available.
Fixed by adding guards against malformed precision, scale and
val_len.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* restore result=NULL after free in mysqlshow.c (to prevent double-free)
* remove DBUG_EVALUATE_IF again
* remove useless 'stop slave' from a test
* restore charset encoding in mysqltest.{test,result}
* correct error for unparseable BINLOG data is ER_SYNTAX_ERROR now
* restore old libfmt on rhel-7, gcc 4.8.5 cannot compile anything newer
Add bounds checks to prevent out-of-bounds memory reads when parsing
Table_map_log_event optional metadata and constructor fields.
The checks prevent OOB reads but do not attempt to throw errors for
every possible corruption. Replication fundamentally assumes a trusted
source with valid data. The goal here is hardening against accidental
corruption (truncated events, garbled metadata lengths). On detecting
corruption, the slave either falls back gracefully to positional column
mapping (parse_* functions) or rejects the event entirely (constructor),
rather than crashing or leaking heap contents via OOB reads.
parse_* functions: Add if (unlikely(p > end)) return after every
net_field_length() call. Ensures decoded values are validated before
being stored to struct fields, preventing information disclosure from
OOB reads. On early return, the slave gracefully falls back to
positional column mapping.
Table_map_log_event constructor: Add bounds checks before memcpy calls
for m_colcnt, m_field_metadata_size, and m_null_bits. On overflow, frees
allocated memory and sets m_memory=NULL causing is_valid() to return
false. Adds read-side DBUG_EXECUTE_IF injection points for testing.
Functions fixed:
- parse_default_charset
- parse_column_charset
- parse_column_name
- parse_set_str_value
- parse_geometry_type
- parse_simple_pk
- parse_pk_with_prefix
- Table_map_log_event constructor (m_colcnt, m_field_metadata, m_null_bits)
MTR test rpl_table_map_log_event_overflow covers all affected code paths
with 9 test cases using debug injection.
MDEV-29804: Query_log_event::pack_info() was displaying inverted values
for foreign_key_checks, check_constraint_checks and unique_checks flags
in SHOW BINLOG EVENTS.
MDEV-32447: check_constraint_checks was not included in
OPTIONS_WRITTEN_TO_BINLOG, causing replication to break when
check_constraint_checks=OFF was used in non-ROW binlog formats.
Reviewed-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
Reviewed-by: Georgi Kodinov <joro@mariadb.org>
Remove the unscoped #pragma GCC diagnostic ignored
"-Wdeprecated-declarations" from include/violite.h and replace all
sprintf/vsprintf calls with snprintf/vsnprintf across the codebase.
Where possible, pass actual buffer sizes as function parameters
instead of using hardcoded constants or recomputing strlen().
Notable API changes:
- get_date(): added to_len parameter for buffer size
- nice_time(), end_timer() in client/mysql.cc: added buff_size
- generate_new_name(): added name_size parameter
- calc_md5(): added buffer_size parameter (fixes off-by-one)
- MD5_HASH_TO_STRING macro: added _size parameter
- Added MYSQL_UDF_MAX_RESULT_LENGTH constant in mysql_com.h
Also fixed a pre-existing bug in sql/rpl_mi.cc where the ellipsis
"..." was written to the wrong buffer (dbuff instead of buff).
Vendored code is excluded: extra/readline, extra/wolfssl,
wsrep-lib, libmariadb, storage/mroonga/vendor, zlib.
This affects row-based replication for UPDATE/DELETE on tables with no
primary key (and no non-NULL unique index).
There was a typo / incorrect merge from the following commit:
commit 3abce27e9d
Author: unknown <knielsen@knielsen-hq.org>
Date: Mon Dec 27 22:37:37 2010 +0100
Merge Percona patch row_based_replication_without_primary_key.patch into MariaDB.
Instead of referencing the index selected for applying the event, part
of the code was referencing the first index (in internal order) in the
table. This is code that checks if an index lookup is unique, or if it
requires a range scan. This could lead the code to incorrectly do a
unique lookup instead of comparing the pre-image against each row in a
range scan, thus applying the event to the wrong row and causing
replication to diverge.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
Fix rpl suite tests added by MDEV-25039.
rpl_foreign_key_lock_table_insert.test is removed altogether because it
is unclear what the purpose of the test is. The changes of the patch
were done on the slave, yet all operations in the test were done on the
master. Nothing different could happen on the slave because it is
configured to be serial, so all transactions would run sequentially
anyway, and no validations were performed.
rpl_foreign_key_ddl_insert.test was renamed to
rpl_row_foreign_key_mdl.test and the test itself was re-written to be
a minimal test case to ensure that MDL locking behavior is different
pre- and post- patch. A few problems with the original test:
* No foreign-key locking was done on the slave because the table
engine was not InnoDB.
* rpl_fk_ddl.inc had inconsistent validation checking. I.e., the child
query validation checks were done on the master (which is incorrect)
and because the slave was configured to be serial, the two
transactions could not run concurrently on the slave anyway.
Issue:
On galera write node INSERT statements does not acquire MDL locks on it's all child
tables and thereby wsrep certification keys are also added for limited tables, but
on applier nodes it does acquire MDL locks for all child tables. This can result
into MDL BF-BF conflict on applier node when transactions referring to parent and
child tables are executed concurrently. For example:
Tables with foreign keys: t1<-t2<-t3<-t4
Conflicting transactions: INSERT t1 and DROP TABLE t4
Wsrep certification keys taken on write node:
- for INSERT t1: t1 and t2
- for DROP TABLE t4: t4
On applier node MDL BF-BF conflict happened between two transaction because
MDL locks on t1, t2, t3 and t4 were taken for INSERT t1, which conflicted
with MDL lock on t4 taken by DROP TABLE t4.
The Wsrep certification keys helps in resolving this MDL BF-BF conflict by
prioritizing and scheduling concurrent transactions. But to generate Wsrep
certification keys it needs to open and take MDL locks on all the child tables.
On applier nodes Write_rows event is implicitly a REPLACE, deleting all conflicting
rows which can cause cascading FK actions and locks on foreign key children tables.
Solution:
For Galera applier nodes the Write_rows event is considered pure INSERT
which will never cause cascading FK actions and locks on foreign key children tables.
- use shared key for sequence update certification
- employ native replication's code to apply changes for sequences
which handles all corner cases properly
- fix the tests to allow more transactions using sequences to be
accepted
That way the sequence is always updated to the maximum value
independent of the order of updates, and shared certification keys
allow to improve acceptance ratio of concurrent transactions that
use sequences. It's reflected in the test changes.
In Query_log_event::do_apply_event there is comparison is used
character set in event same as cached charset and if not used
charset is changed. Unfortunately, it was done only if thread is
replica thread.
Fixed by adding condition for Galera applier thread so that
comparison is done leading to charset change if event
had different charset.
Applying of write set containing many row events causes
applier `thd->mem_root` to grow until applying is over.
This may contribute to memory fragmentation problem, producing
large chunks of memory which are not released back to operating
system.
In order to reduce memory fragmentation, free `thd->mem_root`
in `Rows_log_event::do_apply_event()` after applying a row event
with `F_STMT_END` set.
When binlog_row_image=MINIMAL, UPDATE_ROWS_EVENT may change columns
that are not in the before image. Such columns had their bit set in
table->write_set, but was missing their bit in table->read_set.
As part of this patch, bitmap_union() is extended to handle bitmaps of
different sizes, similar to bitmap_intersect().
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
Replication filtering is not safe for cluster consistency.
There is problem when two or more filtering rules are
provided by user. Here, we disable replication filtering
inside a Galera cluster for now. To support safe replication
filereing in the Galera cluster it should also effect SST/IST.
Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>
We had a protection against it, by allowing versioned delete if:
trx->id != table->vers_start_id()
For replace this check fails: replace calls ha_delete_row(record[2]), but
table->vers_start_id() returns the value from record[0], which is irrelevant.
The same problem hits Field::is_max, which may have checked the wrong record.
Fix:
* Refactor Field::is_max to optionally accept a pointer as an argument.
* Refactor vers_start_id and vers_end_id to always accept a pointer to the
record. there is a difference with is_max is that is_max accepts the pointer to
the
field data, rather than to the record.
Method val_int() would be too effortful to refactor to accept the argument, so
instead the value in record is fetched directly, like it is done in
Field_longlong.
See also MDEV-30046.
Idempotent write_row works same as REPLACE: if there is a duplicating
record in the table, then it will be deleted and re-inserted, with the
same update optimization.
The code in Rows:log_event::write_row was basically copy-pasted from
write_record.
What's done:
REPLACE operation was unified across replication and sql. It is now
representred as a Write_record class, that holds the whole state, and allows
re-using some resources in between the row writes.
Replace, IODKU and single insert implementations are split across different
methods, reluting in a much cleaner code.
The entry point is preserved as a single Write_record::write_record() call.
The implementation to call is chosen on the constructor stage.
This allowed several optimizations to be done:
1. The table key list is not iterated for every row. We find last unique key in
the order of checking once and preserve it across the rows. See last_uniq_key().
2. ib_handler::referenced_by_foreign_key acquires a global lock. This call was
done per row as well. Not all the table config that allows optimized replace is
folded into a single boolean field can_optimize. All the fields to check are
even stored in a single register on a 64-bit platform.
3. DUP_REPLACE and DUP_UPDATE cases now have one less level of indirection
4. modified_non_trans_tables is checked and set only when it's really needed.
5. Obsolete bitmap manipulations are removed.
Also:
* Unify replace initialization step across implementations:
add prepare_for_replace and finalize_replace
* alloca is removed in favor of mem_root allocation. This memory is reused
across the rows.
* An rpl-related callback is added to the replace branch, meaning that an extra
check is made per row replace even for the common case. It can be avoided with
templates if considered a problem.
MDEV-6247 added PROCESSLIST states for when a Replication
SQL thread processes Row events, including a WSRep variant
that dynamically includes the Galera Sequence Number.
MDEV-7409 further expanded on it by adding the table name to the states.
However, PROCESSLIST __cannot__ support generated states.
Because it loads the state texts asynchronously,
only permanently static strings are safe.
Even thread-local memory can become invalid when the thread terminates,
which can happen in the middle of generating a PROCESSLIST.
To prioritize memory safety, this commit reverts both variants to
static strings as the non-WSRep variant was before MDEV-7409.
* __Fully__ revert MDEV-7409 (d9898c9a71)
* Remove the WSRep override from MDEV-6247
* Remove `THD::wsrep_info` and its compiler
flag `WSREP_PROC_INFO` as they are now unused
This commit also includes small optimizations
from MDEV-36839’s previous draft, #4133.
Reviewed-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
This commit fixes a bug in MSAN verification in debug builds
related to the member of a lex->definer structure field that may
be uninitialized for some SET statements. The bug is caused by
debug output in a wsrep-specific insert and does not affect
server functionality.
In Log_event::read_log_event(), don't use IO_CACHE::error of the relay log's
IO_CACHE to signal an error back to the caller. When reading the active
relay log, this flag is also being used by the IO thread, and setting it can
randomly cause the IO thread to wrongly detect IO error on writing and
permanently disable the relay log.
This was seen sporadically in test case rpl.rpl_from_mysql80. The read
error set by the SQL thread in the IO_CACHE would be interpreted as a
write error by the IO thread, which would cause it to throw a fatal
error and close the relay log. And this would later cause CHANGE
MASTER to try to purge a closed relay log, resulting in nullptr crash.
SQL thread is not able to parse an event read from the relay log. This
can happen like here when replicating unknown events from a MySQL master,
potentially also for other reasons.
Also fix a mistake in my_b_flush_io_cache() introduced back in 2001
(fa09f2cd7e) where my_b_flush_io_cache() could wrongly return an error set
in IO_CACHE::error, even if the flush operation itself succeeded.
Also fix another sporadic failure in rpl.rpl_from_mysql80 where the outout
of MASTER_POS_WAIT() depended on timing of SQL and IO thread.
Reviewed-by: Monty <monty@mariadb.org>
Reviewed-by: Andrei Elkin <andrei.elkin@mariadb.com>
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
Two-Phase ALTER added a sa_seq_no field, but `Gtid_log_event::write()`'s
size calculation doesn't have an addend in its name.
This patch resizes the buffer to match `write()`'s code.
Parallel slave failed to retry in retry_event_group() with error
WSREP: Parallel slave worker failed at wsrep_before_command() hook
Fix wsrep transaction cleanup/restart in retry_event_group() to properly
clean up previous transaction by calling wsrep_after_statement().
Also move call to reset error after call to wsrep_after_statement()
to make sure that it remains effective.
Add a MTR test galera_as_slave_parallel_retry to reproduce the error
when the fix is not present.
Other issues which were detected when testing with sysbench:
Check if parallel slave is killed for retry before waiting for prior
commits in THD::wsrep_parallel_slave_wait_for_prior_commit(). This
is required with slave-parallel-mode=optimistic to avoid deadlock
when a slave later in commit order manages to reach prepare phase
before a lock conflict is detected.
Suppress wsrep applier specific warning for slave threads.
Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>
Fill in the `todo:gtid` in `check_and_remove_stale_alter`
(Note that `Master_info::master_id` is a `ulong`,
unlike `rpl_gtid::server_id`)
> We could have caught this before MDEV-11675 was
> published if we'd had this validation earlier 😇 .
> ⸺ Brandon, reply in #3360 (MDEV-21978)
Co-authored-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
That PR uncovered countless issues on `my_snprintf` uses.
This commit backports a squashed subset of their fixes.
(Excludes previous parts #3485 and #3493)
Problem was missing thd->set_time() before binlog event
execution in wsrep_apply_events.
Removed part of earlier commit 1363580 because it had
nothing to do with VERSIONED tables.
Note that this commit does not contain mtr-testcase
because actual timestamps on binlog file depends the
actual time when events are executed and how long
their execution takes.
Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>
Problem was that we did not found that table was partitioned
and then we should find what is actual underlaying storage
engine.
We should not use RSU for !InnoDB tables.
Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>
There are two problems.
First, replication fails when XA transactions are used where the
slave has replicate_do_db set and the client has touched a different
database when running DML such as inserts. This is because XA
commands are not treated as keywords, and are thereby not exempt
from the replication filter. The effect of this is that during an XA
transaction, if its logged “use db” from the master is filtered out
by the replication filter, then XA END will be ignored, yet its
corresponding XA PREPARE will be executed in an invalid state,
thereby breaking replication.
Second, if the slave replicates an XA transaction which results in
an empty transaction, the XA START through XA PREPARE first phase of
the transaction won’t be binlogged, yet the XA COMMIT will be
binlogged. This will break replication in chain configurations.
The first problem is fixed by treating XA commands in
Query_log_event as keywords, thus allowing them to bypass the
replication filter. Note that Query_log_event::is_trans_keyword() is
changed to accept a new parameter to define its mode, to either
check for XA commands or regular transaction commands, but not both.
In addition, mysqlbinlog is adapted to use this mode so its
--database filter does not remove XA commands from its output.
The second problem fixed by overwriting the XA state in the XID
cache to be XA_ROLLBACK_ONLY, so at commit time, the server knows to
rollback the transaction and skip its binlogging. If the xid cache
is cleared before an XA transaction receives its completion command
(e.g. on server shutdown), then before reporting ER_XAER_NOTA when
the completion command is executed, the filter is first checked if
the database is ignored, and if so, the error is ignored.
Reviewed By:
============
Kristian Nielsen <knielsen@knielsen-hq.org>
Andrei Elkin <andrei.elkin@mariadb.com>
