Issue: mariadb-backup SST unconditionally passes socat's "commonname="
option; some socat builds don't register it, so parseopts() rejects
it as unknown regardless of value, breaking all SSL-encrypted SST.
Solution: probe the socat binary once for commonname support and
drop the option when unsupported.
Backported the fix done by Pekka Lampio for mariaDB 11.4 in PR
https://github.com/mariadb-corporation/codership-mariadb-server/pull/543
The PR has a fix for wsrep_sst_rsync script and new mtr test:
galera_3nodes.galera_mdev_36677" to check that the rsync SST method of Galera
works correctly also when the joiner node store InnoDB log files in a dedicated
directory separate from the data dictionary
Note: merging this PR to 11.4 may not be fully functional as there are other
changes in the rsync SST script. Take a look at the original 11.4 PR when merging.
With log_bin=ON a transaction is committed via two-phase commit (the
binary log is the second participant), so it passes through the InnoDB
XA-prepare state. While a donor is held in BLOCK_COMMIT for a mariabackup
backup, its parallel appliers (wsrep_slave_threads > 1) leave one or more
such writesets prepared-but-not-yet-committed, and the snapshot captures
them. On a freshly SST'd joiner nothing resolves these prepared
transactions: binlog crash recovery does not run (the joiner has no in-use
binlog to recover from), and the wsrep continuity-based commit is inactive
because wsrep_emulate_bin_log is FALSE when log_bin is ON. The leftover
prepared transactions then abort startup with "Found <N> prepared
transactions!". Note this does not depend on the prepared set being
non-contiguous - even a contiguous run aborts, because nothing commits
or rolls it back.
Rollback these transactions in xarecover_handlerton(). If rollback fails
flag error to cause unireg_abort().
After a mariabackup SST the joiner could fail with
ER_GTID_STRICT_OUT_OF_ORDER (error 1950)
while re-binlogging transactions received over IST.
The cause is that the binary log copied from the donor carries a
Gtid_list whose position can be ahead of the storage-engine snapshot
because getting snapshot and binlog are no mutually atomic, so transactions
can be present in the copied binlog that are not committed in the copied
engine snapshot.
After the SST the joiner reports the (committed) engine position to the
cluster, IST resends those transactions, and re-binlogging them under
gtid_strict_mode=ON collides with the ahead Gtid_list -> error 1950.
(MDEV-34483 made the engine snapshot stop short of the binlog, which is
what exposed this.)
The copied binary log carries no transactions the joiner needs - only a
Gtid_list - so instead of shipping and then having to truncate/reconcile
it, the joiner now starts a fresh binary log and seeds its GTID position
from the storage-engine checkpoint during recovery. That checkpoint is
the committed cluster position, i.e. exactly where IST resumes, so the
joiner's binary log stays in lockstep with the rest of the cluster and
no out-of-order GTID can occur.
This works for both wsrep_gtid_mode settings; only the binlog domain of
the cluster stream differs:
- wsrep_gtid_mode=ON : wsrep_gtid_domain_id (cluster writes are
re-tagged to it), which is the domain stored in the checkpoint;
- wsrep_gtid_mode=OFF: gtid_domain_id (cluster writes keep the node's
configured domain).
Async-replica positions (mysql.gtid_slave_pos) are part of the engine
snapshot and survive the SST unchanged, so a Galera node can still serve
as an async master or replica across the SST.
This commit:
- sql/log.cc: adds wsrep_seed_binlog_gtid_state(), called from
do_binlog_recovery() when the joiner has no binary log, seeding the
binlog GTID state for the cluster domain to the SE checkpoint position.
- scripts/wsrep_sst_mariabackup.sh: no longer moves the donor's binary
log into place on the joiner.
- extra/mariabackup: backward compatibility: keep shipping binlog file
in SST but
- on donor fix the race between rotation and shipping so that the file
shipped is the one that had been rotated;
- on joiner discard shipped binlog in favour of one generated by
wsrep_seed_binlog_gtid_state().
- sql/wsrep_sst.cc: logs the position actually adopted from storage
(the authoritative post-SST position) rather than the script-reported
one.
- sql/handler.cc: downgrades the "Discovered discontinuity in recovered
wsrep transaction XIDs" message in wsrep_order_and_check_continuity()
from warning to debug level. With parallel appliers a snapshot
routinely captures prepared XIDs that are not contiguous with the
engine checkpoint, so this is normal during SST recovery and of no
value in regular operation; the transactions past the checkpoint are
re-delivered by the cluster (IST/SST) regardless.
- Adds an MDEV-38147 MTR test reproducing the issue.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* don't create mariadb-wsrep-new-cluster in the mariadbd-writable path,
the server should not be able to poison the environment with OUTFILE.
Create it in /run
* As in /run it must be deleted by root, let galera_new_cluster delete
it, not the service
* wsrep-start-position cannot be created by root, so avoid a file
for it at all
Assisted-By: Claude:claude-5-opus
Issue:
When ssl-mode required encryption but the means to perform it was missing,
the SST scripts silently fell back to a cleartext transfer:
- wsrep_sst_rsync: ran over plain TCP when the 'stunnel' binary was absent.
- wsrep_sst_mariabackup: socat used a cleartext socket when ssl-mode was set
but no usable cert/key was found (encrypt stayed 0).
Solution:
Abort the SST instead of falling back to an unencrypted transfer when
ssl-mode is not DISABLED but encryption cannot be set up:
- wsrep_sst_rsync: derive the implicit ssl-mode from the SSL config even
when stunnel is absent, then abort with ENOENT if ssl-mode is active
and the stunnel binary is not found.
- wsrep_sst_mariabackup: after reading the SSL configuration, abort with
EINVAL if ssl-mode is not DISABLED but encrypt resolved to 0 (no usable
cert/key).
Reviewed-by: Jan Lindström <jan.lindstrom@mariadb.com>
Issue:
The mariabackup SST script enabled a peer certificate name check
for any ssl-mode starting with VERIFY, so VERIFY_CA behaved like
VERIFY_IDENTITY, unlike the rsync script.
Solution:
For VERIFY_CA verify the chain only - drop the socat
"commonname" check in mariabackup and gate the rsync checkHost/checkIP
on VERIFY_IDENTITY.
Issue:
Verify_ca_matches_cert ran "openssl verify -CAfile <ca> <cert>", which
treats only the first cert in the file as the target. For a leaf+intermediate
bundle against a root-only CA, openssl cannot reach the root, so the check
aborts the SST (exit 22).
Solution:
Pass the cert file as -untrusted so bundled intermediates complete
the chain to the CA. Trust anchor is unchanged (still -CAfile only).
Issue: The SST TLS options are not role-specific, so a node cannot
present a different certificate as donor (TLS client) than as joiner
(TLS server), which some environments require.
Solution: In the shared check_server_ssl_config, add role-specific
options that override the generic ones: ssl-server-{ca,capath,cert,key}
for the joiner and ssl-client-{ca,capath,cert,key} for the donor.
Applies to both mariabackup and rsync SST.
An appropriately privileged user (with SUPER privileges) could
execute shell commands as the uid of the mariadbd process
because the values of the system variable wsrep_sst_auth,
which can be modified at runtime, were not properly
sanitized when used to construct a shell command.
Combined rsync and mariabackup test cases and added
test case for incorrect values for wsrep_sst_auth.
Issue:
wsrep_sst_rsync.sh interpolated WSREP_SST_OPT_REMOTE_USER and
WSREP_SST_OPT_REMOTE_PSWD verbatim. Because both values originate from
the joiner side of the SST request, a newline in either could splice
an extra directive into the donor-written stunnel.conf (silently
downgrading peer-cert verification) or an extra line into the rsync
magic file. MDEV-39413 had introduced safe() for the same threat class
in wsrep_sst_mariabackup but did not extend it to the rsync script.
Solution:
Routing the rsync interpolations through safe() closes the gap, and
extending safe() to also reject tab and newline ensures multi-line
values cannot survive into a config-file heredoc.
scripts/fill_help_tables.sql from
https://github.com/mariadb-corporation/mariadb-docs/ as of
3fb208fcdfed06beb52ef9e305017de2387ff012
Other URL changes are taken from the HTTP redirection URL
from the old KB.
Some test cases where impacted:
* ps_missed_cmds{,_bin_prot} - used a topic_id of 0 that is
now used, used MAX_INT instead.
* range_notembedded used the help data as a test case. As its not
a stable table data, replaces with static data. Test case valid
by ensuring the ANALYZE EXPLAIN and optimizer trace had the same
structure. Also valided the MDEV-24739/MDEV-25069 bug fixes from
where the test case originaled, when reverted, exhibited the
previous behavour before the bug fixes.
introduce safe() wrapper for parameters that can later be interpolated
into command line.
Use as
- var="foo$BAR"
+ var="foo$(safe BAR)"
A parameter is safe, if it contains no spaces, single quotes or backticks
After moving the systemd service to using environment files
instead of `systemctl set-environment` in 11.6 (MDEV-19210),
they (wsrep-new-cluster and wsrep-start-position) are located
in /var/lib/mysql along with the socket file in
Fedora/RHEL-based distros. This causes them to have incorrect
selinux permissions and therefore be not readable by systemd.
A solution is to generate these files in the run directory,
instead, which already has correct selinux label mysqld_var_run_t
mysql-selinux-1.0.12. Dissociating these files and the socket
in CMake configs can also prove useful for other things.
This also corrects some of the duplicate code in the build
scripts and made INSTALL_RUNDATADIR into a proper location
and used this for the tmpfiles where the temporary files
are created.
Debian's location is /run/mysqld/ matching its INSTALL_UNIX_ADDRDIR,
which is now a temporary location controlled by tmpfiles.
Fix security issue where temporary output files containing SQL commands
and password hashes were created with default permissions (world-readable).
Modified prepare() function to create and umask the $output $config and
$command files. It is more secure than chmod as files can be opened, before
the chmod is applied, and then read when its its later populated.
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.
After moving the systemd service to using environment files
instead of `systemctl set-environment` in 11.6 (MDEV-19210),
they (wsrep-new-cluster and wsrep-start-position) are located
in /var/lib/mysql along with the socket file in
Fedora/RHEL-based distros. This causes them to have incorrect
selinux permissions and therefore be not readable by systemd.
A solution is to generate these files in the run directory,
instead, which already has correct selinux label mysqld_var_run_t
mysql-selinux-1.0.12. Dissociating these files and the socket
in CMake configs can also prove useful for other things.
This also corrects some of the duplicate code in the build
scripts and made INSTALL_RUNDATADIR into a proper location
and used this for the tmpfiles where the temporary files
are created.
Debian's location is /run/mysqld/ matching its INSTALL_UNIX_ADDRDIR,
which is now a temporary location controlled by tmpfiles.
From downstream bug RHEL-116757, the use of ss to map all the
usage of ports to pids and process names under selinux is rather
limited by the mysqld_exec_t limitations.
The checking of the mapping is a bit excessive in a selinux
environment where binding is limited.
We replace the use of ss under selinux to raw reading of the
/proc/tcp{,} interfaces of the linux that aren't selinux restricted.
Paths are set to "canonical" form and historical practice where
/lib comes before /usr/lib and /bin comes before /usr/bin. This
eliminates the need to skip setting LD_LIBRARY_PATH for FreeBSD,
which may help those users whose LD_LIBRARY_PATH is not configured
properly.
Bootstrapping nodes on FreeBSD 15.0 with mariadb-backup currently broken.
We should not need LD_LIBRARY_PATH trickery on FreeBSD, so skip setting
it same as on Darwin. Alternatively putting /lib before /usr/lib also
fixes it.
WSREP_SST: [INFO] Preparing the backup at /var/db/mysql/.sst (20250608 14:44:04.000)
WSREP_SST: [INFO] Evaluating /usr/local/bin/mariadb-backup --prepare ......
WSREP_SST: [ERROR] Cleanup after exit with status: 1 (20250608 14:44:04.000)
With the following error in /var/db/mysql/mariabackup.prepare.log:
ld-elf.so.1: /usr/local/bin/mariadb-backup: Undefined symbol "_ZNSt3__117bad_function_callD1Ev"
I did not look for the root cause in FreeBSD 15.0 that changed this
behaviour. Calling mariadb-backup directly from the shell works fine on
FreeBSD 14.x and 15.x when LD_LIBRARY_PATH is not set (when using
FreeBSD packages or installing from FreeBSD ports). Setting
LD_LIBRARY_PATH manually like the script does, breaks mariadb-backup
with the "Undefined symbol" error on FreeBSD 15.0 but not on 14.x.
Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>
-Replaced incorrect string comparision in views for is_hashed and is_old
as these columns are integer and not string
Affected Views:
-innodb_buffer_stats_by_schema
-innodb_buffer_stats_by_table
-x_innodb_buffer_stats_by_schema
-x_innodb_buffer_stats_by_table
As of CMake 3.24 CMAKE_COMPILER_IS_GNU(CC|CXX) are deprecated and should
be replaced with CMAKE_(C|CXX)_COMPILER_ID which were introduced with
CMake 2.6.
Delaying scripts on joiner after SST/IST has been made
a common debug feature for all suitable SST/IST methods.
Also some minor fixes have been made for new tests.
After cluster vote to evict a node that failed a transaction,
current master can't commit anymore.
Error voting for joiner in the JOINED state was broken because
the group-wide commit cut (implicit SUCCESS vote) was not taken
into account when processing error vote request from the JOINED
node.
This commit adds 3 MTR tests to verify the fix in the galera
library works as designed.
Requires Galera library commit 91f0090a05e96c3cc353b80d961ede45cefb9279
(galera library version > 26.4.19).
Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>
Mariabackup (mariadb-backup) supports the --use-memory option that
sets the buffer pool size for innodb. However, current SST scripts
do not use this option. This commit adds support for this option,
the value for which can be specified via the "use_memory" parameter
in the configuration file in the [sst], [mariabackup] or [xtrabackup]
sections (supported only for compatibility with old configurations).
In addition, if the innodb_buffer_pool_size option is specified in
the user configuration (in the main server configuration sections)
or passed to the SST scripts or the server via arguments, its value
is also passed to mariadb-backup as the value for the --use-memory
option.
A new section name [mariabackup] has also been added, which can
be used instead of the deprecated [xtrabackup] (the section name
"mariabackup" was specified in the documentation, but was not
actually supported by SST scripts before this commit).
