The bug was that FLUSH BINARY LOGS (or RESET MASTER) could run concurrently
with a binlog write that spans multiple pages. This results in trying to
write to a page that no longer exists, causing crash, assertion, or other
badness.
The problem was incorrect locking on the server level to prevent this
concurrent usage. In old binlog, binlog write is protected by LOCK_log. But
in new binlog, binlog write is part of engine (ordered) commit, and is
protected by LOCK_commit_ordered.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
Fix race where trx_group_commit_leader() was accessing the group commit
queue after waking up participants, which can invalidate the queue. Instead
do the remaining operations in the individual thread for each group commit
participant.
Also fix a problem where entries could be inserted out-of-order in the
pending LSN fifo, when the queue was empty after removing a later LSN, and
then an earlier LSN got inserted. This could move back the durable binlog
offset, causing slaves to not receive events.
Seen as sporadic failures of test case
binlog_in_engine.mariabackup_slave_provision_nolock.
A few other test tweaks to make them robust to sporadic failures.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
Write a single format description event to the engine binlog at server
startup.
This format description event - like for the legacy binlog - is used to
inform the slave server about the master restart. This is used by the slave
to drop any temporary tables that were binlogged by the master before the
restart, and are now implicitly dropped by the restart.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
Change the page size used for the new-style binlog to 16kB, to reduce the
impact of per-page overhead in binlog flushing and reading.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
Refactor the code to use binlog_chunk_reader for reading a GTID state
record, getting rid of the duplicate logic in the old special-purpose GTID
state reading code. This also removes the assumption that GTID state fits in
a single page (untested for now though).
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
When updating non-transactional tables inside a multi-statement transaction,
and binlog_direct_non_transactional_updates=1, then the non-transactional
updates are binlogged directly through the statement cache while the
transaction cache is still being added to in the main transaction.
Thus, move the engine_binlog_info out from binlog_cache_mngr and into the
individual stmt/trx binlog_cache_data, so that we can have separate
engine_binlog_info active for the statement and the transaction cache.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
Support for SAVEPOINT, ROLLBACK TO SAVEPOINT, rolling back a failed
statement (keeping active transaction), and rolling back transaction.
For savepoints (and start-of-statement), if the binlog data to be rolled
back is still in the in-memory part of trx cache we can just truncate the
cache to the point.
But if we need to spill cache contents as out-of-band data containing one or
more savepoints/start-of-statement point, then split the spill at each point
and inform the engine of the savepoints.
In InnoDB, at savepoint set, save the state of the forest of perfect binary
trees being built. Then at rollback, restore the appropriate state.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
Instead of returning only one chunk at a time, make
ha_innodb_binlog_reader::read_data() try to read all chunks on the page.
This reduces the number of times each reader has to latch pages in the page
fifo, which contends for a global mutex also shared with the writer.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
This patch makes replication crash-safe with the new binlog implementation,
even when --innodb-flush-log-at-trx-commit=0|2. The point is to not send any
binlog events to the slave until they have become durable on master, thus
avoiding that a slave may replicate a transaction that is lost during master
recovery, diverging the slave from the master.
Keep track of which point in the binlog has been durably synced to disk
(meaning the corresponding LSN has been durably synced to disk in the InnoDB
redo log). Each write to the binlog inserts an entry with offset and
corresponding LSN in a FIFO. Dump threads will first read only up to the
durable point in the binlog. A dump thread will then check the LSN fifo, and
do an InnoDB redo log sync if anything is pending. Then the FIFO is emptied
of any LSNs that have now become durable, and the durable point in the
binlog is updated and reading the binlog can continue.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
If the event group fitted in the binlog cache without the GTID event but not
with, the code would attempt to spill part of the GTID event as out-of-band
data, which is not correct. In release builds this would hang the server as
the spilling would try to lock an already owned mutex.
Fix by checking if the GTID event fits, and spilling any non-GTID data as
oob if it does not.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
After temporarily releasing the mutex during wait in
fsp_binlog_page_fifo::do_fdatasync(), the state may have changed, so be
sure to re-check to avoid fdatasync() on a now stale fh.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
Fix that spilling of out-of-band data to the binlog could happen
concurrently with binlog group commit, by holding LOCK_commit_ordered
over all binlog writes now.
Fix silly use-after-free bug where data was accessed in the old buffer after
realloc().
Improve the wording of the error when specifying an argument for --log-bin.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
Some basic improvements to the binlog-specific page fifo to hopefully get
reasonable scalabitily as a starting point.
The fifo is still protected by a global mutex, but some effort is taken to
reduce the duration a thread is holding the mutex.
Use a cyclic array instead of a linked list so pages can be looked up in
constant time. And cache allocated page objects to avoid repeated
malloc/free while holding the mutex.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
The file_no and page_no values are not really needed in the page object,
so remove them to save a bit of memory.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
InnoDB binlog files are now backed up along with other InnoDB data by
mariadb-backup.
The files are copied after backup locks have been released. Backup files
created later than the backup LSN are skipped. Then during --prepare, any
data missing from the hot-copied binlog files will be restored by the
binlog recovery code, and any excess data written after the backup LSN will
be zeroed out.
A couple test cases test taking a consistent backup of a server with active
traffic during the backup, by provisioning a slave from the restored binlog
position and checking that the slave can replicate from the original master
and get identical data.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
Keep track of, for each binlog file, how many open transactions have
out-of-band data starting in that file. Then at the start of each new binlog
file, in the header page, record the file_no of the earliest file that this
file might contain commit records with references back to OOB records in
that earlier file.
Use this in PURGE BINARY LOGS, so that when a dump thread (slave connection)
is active in file number N, and that file (or a later one) may require
looking back in an earlier file number M for out-of-band records, purge will
stop already at file number M. This way, we avoid that purge accidentally
deletes some binlog file that a dump thread would later get an error on
because it needs to read out-of-band data.
This patch also includes placeholder data for a similar facility for XA
references. The actual implementation of support for XA is for later though.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
Mostly various fixes to avoid initializing or creating any data or files for
the legacy binlog.
A possible later refinement could be to sub-class the binlog class
differently for legacy and in-engine binlogs, writing separate virtual
functions for behaviour that differ, extracting common functionality into
sub-methods. This could remove some if (opt_binlog_engine_hton)
conditionals.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
A reader could latch a page that was currently being flushed to disk, while
the flushing thread is temporarily releasing the mutex. If the page was
complete with data when the flushing started, the flush thread would not
correctly wait for the reader to release the latch, and the page could be
freed while the reader was still using it.
Also adjust a couple assertions to reflect the addition of the file header
page as page 0.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
- Update the end_offset for file_no=N+1 before file_no=N, so that a reader
does not have a window where it can see a record partially written,
present in N but not N+1.
- Fix a case when the writer changes to the next file exactly at the same
time that the reader loads the current state. In this case the reader
detects a potentially inconsistent state read and retries, and the code
forgot to release its latch on the page.
- Fix a race when flush_one_page() runs in parallel with RESET MASTER.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
Fix another race, when a RESET MASTER happens during flushing of the binlog
for log checkpoint.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
When a page is flushed to disk, the page fifo mutex is temporarily released;
this opens a window where a binlog write can add data to a page that is
partially flushed to disk due to a checkpoint request. When this happened,
the page could be written to disk corrupted, with only part of the data
written.
Fixed in this patch by detecting when a parallel update happens while
the mutex is released during page write, and in this case re-trying the
write until all data is correctly written.
Also fix another race that could cause a missed wakeup when multiple threads
were simultaneously trying to flush pages, due to using
pthread_cond_signal() and not pthread_cond_broadcast(). If the thread that
pthread_cond_signal() decides to wake up cannot proceed (due to waiting on a
different latch for example), no threads will wake up and the system can get
hung.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
Now the first page of each binlog tablespace file is reserved as a file
header, replacing the use of extra fields in the first gtid state record of
the file. The header is primarily used during recovery, especially to get
the file LSN before which no redo should be applied to the file.
Using a dedicated page makes it possible to durably sync the file header to
disk after RESET MASTER (and at first server startup) and not have it
overwritten (and potentially corrupted) later; this guarantees that the
recovery will have at least one file header to look at to determine from
which LSN to apply redo records.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
Add test case binlog_in_engine.recover with a first very simple recovery
test.
The test currently fails during InnoDB recovery:
2025-03-02 11:35:44 0 [ERROR] InnoDB: Missing FILE_DELETE or FILE_MODIFY for [page id: space=4294967281, page number=0] at 62894; set innodb_force_recovery=1 to ignore the record.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
Some smaller refactoring and additions to prepare for new approach to
recovery of binlog tablespaces.
Store at the head of each binlog file the start LSN and the file size.
The final page of a binlog file is now not released in the page fifo until
mtr is committed. This ensures that all changes to a binlog file are redo
logged when the tablespace is closed, which simplifies things as then at
most the two most recent binlog files will need redo records to be
re-applied during recovery.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
Enable mtr_t::write_binlog() in the code and fix a few associated things
around it. Recovery tests still ToDo:.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
In preparation for a simplified, lower-level recovery of binlog files
implemented in InnoDB, remove use of InnoDB tablespaces and buffer pool from
the binlog code. Instead, a custom binlog page fifo replaces the general
buffer pool for binlog pages, and tablespaces are replaced by simple file_no
references.
The new binlog page fifo is deliberately naively written in this commit for
simplicity, until the new recovery is complete and proven with tests; later
it can be improved for better efficiency and scalability. This first version
uses a simple global mutex, linear scans of linked lists, repeated
alloc/free of pages, and simple backgrund flush thread that uses
synchroneous pwrite() one page after another. Error handling is also mostly
omitted in this first version.
The page header/footer is not changed in this commit, nor is the pagesize,
to be done in a later patch.
The call to mtr_t::write_binlog() is currently commented-out in function
fsp_log_binlog_write() as it asserts in numerous places. To be enabled when
those asserts are fixed. For the same reason, the code does not yet
implement binlog_write_up_to(lsn_t lsn), to be done once mtr_t operations
are working.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
Still ToDo: is to restrict auto-purge so that it does not purge any binlog
file with out-of-band data that might still be needed by a connected slave.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
Fix missing WORDS_BIGENDIAN define in ut0compr_int.cc.
Fix misaligned read buffer for O_DIRECT.
Fix wrong/missing update_binlog_end_pos() in binlog group commit.
Fix race where active_binlog_file_no incremented too early.
Fix wrong assertion when reader reaches the very start of (active+1).
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
Enable binlog_in_engine as a default suite.
Fix embedded and Windows build failures.
Use sql_print_(error|warning) over ib::error() and ib::warn().
Use small_vector<> for the innodb_binlog_oob_reader instead of a custom
implementation.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
