This is based on a commit 281a0c9998
by Monty.
According to Monty, this:
- Galera should now be supported (needs testing)
- Enables ddl logging (so we can use it in the future)
- Flushes binary logs (we still must add code to copy them)
- mdl locks are consistent between mariadb-backup and backup command
- Give errors if backup command is done under a transaction, global
read lock or lock tables.
- Retry for MDL_BACKUP_WAIT_DDL (needed for
backup.backup_ddl_concurrent_verify)
Things to do (in addition to the things in Monty's earlier review):
- At backup_stage stage start, force rotate of aria log files. This
allows us to copy all old logs without any locks
- Copy all transactional tables and old aria logs under BACKUP_START (as
- maria-backup does)
- Copy the active aria log file under block commit (Only one file as aria
log rotation is disabled while backup is running.
- Improve speed of copying aria tables by copy files in up to 1M
blocks and run checksum on the blocks and only re-read blocks with
- wrong checksum.
- Copy non transactional files under BACKUP_START.
mariadb-backup --backup always uses a dedicated log_copying_thread()
that eagerly copies the log from the server. Let us do the same
in multi-threaded BACKUP SERVER TO, unless innodb_log_file_buffering=OFF
(which prevents arbitrary-size reads from the log file).
backup_sink:🆔 The thread identifier (0 to CONCURRENT-1)
InnoDB_backup::context::tracked: Log file queue.
innodb_backup_checkpoint_pmem(), innodb_backup_checkpoint():
Enqueue or detach the old log file.
innodb_backup_log_tracking(): Determine if log-tracking backup
is active, preventing SET GLOBAL innodb_log_file_buffering and
SET GLOBAL innodb_log_file_write_through.
InnoDB_backup::log_track(), Keep copying the log until we
run out of InnoDB data files to copy.
handlerton::backup_file: Check if a file should be
included in the backup. Implemented for ENGINE=Aria
in maria_backup_file().
aria_backup_start(): Copy the Aria log files
(FIXME: currently, single-threaded)
backup::copy_or_stream(): Copy or stream a file.
backup_context: Process-wide BACKUP SERVER context.
Handles the directory traversal and copying of
files for built-in storage engines that do not
implement this backup interface.
backup_target_phase. Wrap backup_context.
backup_target_phase::step(),
backup_context::step(): Process a file
from a directory scan, or by invoking
handlerton::backup_step().
The following SQL statements will be introduced:
BACKUP SERVER TO '/path/to/directory' [ 1 CONCURRENT ];
BACKUP SERVER WITH [ 1 CONCURRENT ] 'command';
In place of the 1, any positive number of threads may be specified.
For the first variant, '/path/to' must exist and '/path/to/directory'
must not exist; that is where the backup will be written to.
For BACKUP SERVER WITH, the string mariadb-backup-
will be prepended to the specified command and quoted.
Path separators or quotes are not allowed in command.
We expect the mariadb-backup-command to be in the
search PATH of the mariadbd process.
For now, we distribute no such script or program;
the user has to write one, something like this:
exec zstdmt|ssh backup@remote.example.com "exec cat > $1.tar.zstd"
The standard input of that command will be in a format compatible with
GNU tar --format=oldgnu (and also BSD tar variants that are also part of
Microsoft Windows and Apple macOS). The command is expected to optionally
compress and encrypt the stream and redirect it to a file on a local or
a remote server. The BACKUP SERVER WITH will append an additional argument,
a positive base-ten number in ASCII, starting with 1, to identify the
current thread. In this way, there can be multiple concurrent streams.
The InnoDB write-ahead log will be streamed near the end, because
the stream format requires the file sizes to be declared upfront.
TODO: implement the following:
In BACKUP SERVER TO ... 2 CONCURRENT or more, one thread will be
responsible for copying the InnoDB write-ahead log from the beginning
of the backup. Other files will be copied by other threads.
Note: In single-threaded BACKUP SERVER TO, the parameter
innodb_log_recovery_start that is written out to backup.cnf is
STRICTLY NECESSARY TO AVOID CORRUPTION during recovery! By default,
InnoDB crash recovery starts from the latest available log
checkpoint. However, for restoring a backup, recovery must start from
the checkpoint that was the latest when the backup was
started. Starting recovery from a possible later checkpoint will
result in a corrupted database!
The backup or the first stream will contain a file backup.cnf, which
includes parameters needed for restoring the backup. Currently,
these are innodb_log_recovery_start and innodb_log_recovery_target.
If innodb_log_recovery_target>0, InnoDB will be in read-only mode,
not allowing any writes to persistent files other than via the log
application.
To restore a streaming backup made with BACKUP SERVER WITH, an empty
directory needs to be created and all streams be extracted there using
the standard tar utility of the operating system, optionally after
undoing any encryption or compression that had been added by the
backup command. Then, the backup is prepared or MariaDB server started
up on the extracted directory, similar to as if the BACKUP SERVER TO
statement had been used.
The following will be implemented separately:
MDEV-39061 mariadb-backup compatible wrapper script for BACKUP SERVER
MDEV-40163 Partial backup and restore
MDEV-39091 Back up ENGINE=RocksDB
MDEV-40333 Concurrent DDL for Aria tables in BACKUP SERVER
The implementation introduces a basic multi-threaded driver
Sql_cmd_backup, storage engine interfaces, and basic copying of the
storage engines InnoDB, Aria, MyISAM, MERGE (MyISAM), Archive, CSV.
backup_target: A structured data type to represent a target directory.
On Microsoft Windows, we must use directory paths because there is
no variant of CopyFileEx() that would work on file handles.
backup_sink: Wraps a per-thread output stream as well as storage engine
specific context.
handlerton::backup_start(), handlerton::backup_end(): Invoked at the
start or end of a backup phase, in the thread that executes a
BACKUP SERVER statement.
handlerton::backup_step(): A backup step that can be invoked from
multiple threads concurrently, between the execution of the corresponding
handlerton::backup_start() and handlerton::backup_end() of the same
phase.
copy_entire_file(): A file copying service for POSIX systems.
copy_mmap(): A zero-copy alternative to backup::copy(), to copy from a
memory-mapped buffer.
copy_file_range_try(): A wrapper for Linux copy_file_range(2), which
may fail with EOPNOTSUPP or EXDEV and thus require a fallback to
copy_mmap() or backup::copy().
backup::copy(): A partial or sparse file-copying service. On other
platforms than FreeBSD or Microsoft Windows, there are shortcut
alternatives to this. Note: On Linux we never invoke sendfile(2) for
copying between files, because can be much slower than the
alternatives.
backup_stream_append_plain(): A wrapper of backup::append(), which is
the streaming equivalent of backup::copy().
backup_stream_zeropad(): Zero-pad the last tar block if needed.
backup_stream_append_async(): A variant of backup_stream_append_plain()
where the source file region is guaranteed to be immutable after the
call returns. Zero-copy mmap(2) or Linux sendfile(2) are inherently
risky for copying data files that may be modified in place, because it
could introduce a race condition between a page write that runs
concurrently with a child process that is reading the data from the
pipe.
backup::append(): On systems where we can determine the size of the
pipe buffer, invoke backup_stream_append_async() for the initial
write, and pread_write() for the last part, to guarantee that the data
written by the zero-copy shortcut will have been consumed before the
call returns and the caller is able to resume writes to the source
region.
pread_write(): On 64-bit systems, allocate a buffer of up to 1 MiB.
This is the "slow path" of copying or streaming files.
struct Aria_backup: Context for multi-threaded backup,
comprising directory handles, a mutex and enum Aria_backup_status.
aria_backup_start(): Prepare the context for aria_backup_step().
Most files are copied in BACKUP_PHASE_NO_DDL after flush_tables(thd,
FLUSH_NON_TRANS_TABLES) has been invoked. All ENGINE=Aria files
(including TRANSACTIONAL=0) are copied in
BACKUP_PHASE_NO_COMMIT. Thanks to Andrzej Jarząbek for writing
test cases and suggesting this logic.
aria_backup_step(): Copy one non-ACID file. Acquires
Aria_backup::mutex, traverses directories to construct one file name,
releases the mutex, and copies the file if one was found.
aria_backup_data(): Copy one data file. On Microsoft Windows, this
assumes that the current directory is the datadir. This assumption
would not hold in the Embedded Server library, which is not supported
on Microsoft Windows.
aria_backup_log(): Copy one ENGINE=Aria log file.
aria_backup_end(): Finish a copying phase and clean up the context.
InnoDB_backup::init(): Wait for a possible previous BACKUP SERVER
operation to reach the very end of InnoDB_backup::context::cleanup()
so that the context can be safely reused.
InnoDB_backup::queue: Collection of tablespace IDs and payload sizes
at the start of the backup, and the log_sys.first_lsn of log files
that have to be included in the backup. If any data file is created or
extended while the backup is executing, we must have the corresponding
write-ahead-log entries that we are copying since the latest
checkpoint that was completed when the backup started. If any
tablespaces are deleted during the backup, we may or may not copy
them, and the application of a FILE_DELETE record will remove them.
Similarly, applying FILE_RENAME or FILE_CREATE records will rename or
create files during recovery as needed.
log_sys.backup: Whether BACKUP SERVER is in progress. The purpose of this
is to make BACKUP SERVER prevent the concurrent execution of
SET GLOBAL innodb_log_archive=OFF or SET GLOBAL innodb_log_file_size
when innodb_log_archive=OFF.
log_sys.archived_checkpoint: Keep track of the earliest available
checkpoint, corresponding to log_sys.archived_lsn. This reflects
SET GLOBAL innodb_log_recovery_start (which is settable now), for
incremental backup.
fil_system.have_all_spaces: Whether all tablespace metadata is guaranteed
to be known. To speed up startup, InnoDB does not normally open
all tablespace files.
fil_space_t::create_lsn: Change to Atomic_relaxed and use this to
indicate tablespace creation LSN, in addition to indicate undo
tablespace rebuild LSN.
fil_space_t::backup_end: The first page number that is not being backed up
(by default 0, to indicate that no backup is in progress).
fil_space_t::BACKUP_BATCH_SIZE: The number of preceding pages that will be
covered by fil_space_t::backup_end. This is the unit of "page range locking"
during InnoDB backup.
buf_page_t::write_fix_try(), buf_page_t::write_unfix_try(): Try to set
or unset a fake "write fix" on a page, to prevent concurrent flush()
during a backup batch. The atomic operations may run concurrently with
set_reinit() and set_freed(). The fake "write fix" does not prevent
any concurrent read or write of the page data in the buffer pool; it
only blocks writes to the underlying data file.
buf_page_t::flush(): Atomically test and set write fix, and
skip the operation if the fake "write fix" was set.
buf_page_t::set_freed(), buf_page_t::set_reinit(): Employ a
compare-and-exchange loop to accommodate for the "write fix".
innodb_backup_batch_wait(): Look up any pages that we are about to
back up. For any dirty pages, invoke buf_page_t::write_fix_try() to
try to set a fake "write fix" lock-free. If the page is currently
write-fixed between buf_page_t::flush() and
buf_page_t::write_complete(), acquire and release a page U-latch to
wait for the conflicting write to complete.
InnoDB_backup::backup_batch_start(),
InnoDB_backup::backup_batch_stop(): Adjust fil_space_t::backup_end and
fake "write fix" of dirty pages to protect the copying of a range of
pages from the underlying file.
InnoDB_backup::commit(): Enqueue the remaining log to be copied.
InnoDB_backup::checkpoint_complete(): If backup is running and
commit() has not been called, add each completed innodb_archive_log=ON
file to the queue. Else, skip or delete, as appropriate.
log_t::backup_start(): If we were running with innodb_log_archive=ON,
ensure that the latest file is a valid recovery starting point.
That is, wait for the latest log checkpoint to be within the file.
buf_flush_list_space(): Check for concurrent backup before writing each
page. This is inefficient, but this function may be invoked from multiple
threads concurrently, and it cannot be changed easily, especially for
fil_crypt_thread().
fil_ibd_create(): Set fil_space_t::create_lsn after the file has been
created.
dict_load_tablespaces(): Determine the size of each file if
upgrade==true. Backup depends on that.
buf_dblwr_t::begin(), buf_dblwr_t::end(), buf_dblwr_t::size():
Accessors to allow BACKUP SERVER to skip the contents of the
doublewrite buffer in the system tablespace. It is only useful for
crash recovery in case a data page had been incompletely written by
the time the server was killed. If the server is killed during a
backup, the backup will be incomplete and unusable anyway.
Furthermore, the page range locking makes page writes and backup
mutually exclusive.
