With log_bin=ON a transaction is committed via two-phase commit (the
binary log is the second participant), so it passes through the InnoDB
XA-prepare state. While a donor is held in BLOCK_COMMIT for a mariabackup
backup, its parallel appliers (wsrep_slave_threads > 1) leave one or more
such writesets prepared-but-not-yet-committed, and the snapshot captures
them. On a freshly SST'd joiner nothing resolves these prepared
transactions: binlog crash recovery does not run (the joiner has no in-use
binlog to recover from), and the wsrep continuity-based commit is inactive
because wsrep_emulate_bin_log is FALSE when log_bin is ON. The leftover
prepared transactions then abort startup with "Found <N> prepared
transactions!". Note this does not depend on the prepared set being
non-contiguous - even a contiguous run aborts, because nothing commits
or rolls it back.
Rollback these transactions in xarecover_handlerton(). If rollback fails
flag error to cause unireg_abort().
After a mariabackup SST the joiner could fail with
ER_GTID_STRICT_OUT_OF_ORDER (error 1950)
while re-binlogging transactions received over IST.
The cause is that the binary log copied from the donor carries a
Gtid_list whose position can be ahead of the storage-engine snapshot
because getting snapshot and binlog are no mutually atomic, so transactions
can be present in the copied binlog that are not committed in the copied
engine snapshot.
After the SST the joiner reports the (committed) engine position to the
cluster, IST resends those transactions, and re-binlogging them under
gtid_strict_mode=ON collides with the ahead Gtid_list -> error 1950.
(MDEV-34483 made the engine snapshot stop short of the binlog, which is
what exposed this.)
The copied binary log carries no transactions the joiner needs - only a
Gtid_list - so instead of shipping and then having to truncate/reconcile
it, the joiner now starts a fresh binary log and seeds its GTID position
from the storage-engine checkpoint during recovery. That checkpoint is
the committed cluster position, i.e. exactly where IST resumes, so the
joiner's binary log stays in lockstep with the rest of the cluster and
no out-of-order GTID can occur.
This works for both wsrep_gtid_mode settings; only the binlog domain of
the cluster stream differs:
- wsrep_gtid_mode=ON : wsrep_gtid_domain_id (cluster writes are
re-tagged to it), which is the domain stored in the checkpoint;
- wsrep_gtid_mode=OFF: gtid_domain_id (cluster writes keep the node's
configured domain).
Async-replica positions (mysql.gtid_slave_pos) are part of the engine
snapshot and survive the SST unchanged, so a Galera node can still serve
as an async master or replica across the SST.
This commit:
- sql/log.cc: adds wsrep_seed_binlog_gtid_state(), called from
do_binlog_recovery() when the joiner has no binary log, seeding the
binlog GTID state for the cluster domain to the SE checkpoint position.
- scripts/wsrep_sst_mariabackup.sh: no longer moves the donor's binary
log into place on the joiner.
- extra/mariabackup: backward compatibility: keep shipping binlog file
in SST but
- on donor fix the race between rotation and shipping so that the file
shipped is the one that had been rotated;
- on joiner discard shipped binlog in favour of one generated by
wsrep_seed_binlog_gtid_state().
- sql/wsrep_sst.cc: logs the position actually adopted from storage
(the authoritative post-SST position) rather than the script-reported
one.
- sql/handler.cc: downgrades the "Discovered discontinuity in recovered
wsrep transaction XIDs" message in wsrep_order_and_check_continuity()
from warning to debug level. With parallel appliers a snapshot
routinely captures prepared XIDs that are not contiguous with the
engine checkpoint, so this is normal during SST recovery and of no
value in regular operation; the transactions past the checkpoint are
re-delivered by the cluster (IST/SST) regardless.
- Adds an MDEV-38147 MTR test reproducing the issue.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* don't create mariadb-wsrep-new-cluster in the mariadbd-writable path,
the server should not be able to poison the environment with OUTFILE.
Create it in /run
* As in /run it must be deleted by root, let galera_new_cluster delete
it, not the service
* wsrep-start-position cannot be created by root, so avoid a file
for it at all
Assisted-By: Claude:claude-5-opus
Issue:
When ssl-mode required encryption but the means to perform it was missing,
the SST scripts silently fell back to a cleartext transfer:
- wsrep_sst_rsync: ran over plain TCP when the 'stunnel' binary was absent.
- wsrep_sst_mariabackup: socat used a cleartext socket when ssl-mode was set
but no usable cert/key was found (encrypt stayed 0).
Solution:
Abort the SST instead of falling back to an unencrypted transfer when
ssl-mode is not DISABLED but encryption cannot be set up:
- wsrep_sst_rsync: derive the implicit ssl-mode from the SSL config even
when stunnel is absent, then abort with ENOENT if ssl-mode is active
and the stunnel binary is not found.
- wsrep_sst_mariabackup: after reading the SSL configuration, abort with
EINVAL if ssl-mode is not DISABLED but encrypt resolved to 0 (no usable
cert/key).
Reviewed-by: Jan Lindström <jan.lindstrom@mariadb.com>
Issue:
The mariabackup SST script enabled a peer certificate name check
for any ssl-mode starting with VERIFY, so VERIFY_CA behaved like
VERIFY_IDENTITY, unlike the rsync script.
Solution:
For VERIFY_CA verify the chain only - drop the socat
"commonname" check in mariabackup and gate the rsync checkHost/checkIP
on VERIFY_IDENTITY.
Issue:
Verify_ca_matches_cert ran "openssl verify -CAfile <ca> <cert>", which
treats only the first cert in the file as the target. For a leaf+intermediate
bundle against a root-only CA, openssl cannot reach the root, so the check
aborts the SST (exit 22).
Solution:
Pass the cert file as -untrusted so bundled intermediates complete
the chain to the CA. Trust anchor is unchanged (still -CAfile only).
Issue: The SST TLS options are not role-specific, so a node cannot
present a different certificate as donor (TLS client) than as joiner
(TLS server), which some environments require.
Solution: In the shared check_server_ssl_config, add role-specific
options that override the generic ones: ssl-server-{ca,capath,cert,key}
for the joiner and ssl-client-{ca,capath,cert,key} for the donor.
Applies to both mariabackup and rsync SST.
An appropriately privileged user (with SUPER privileges) could
execute shell commands as the uid of the mariadbd process
because the values of the system variable wsrep_sst_auth,
which can be modified at runtime, were not properly
sanitized when used to construct a shell command.
Combined rsync and mariabackup test cases and added
test case for incorrect values for wsrep_sst_auth.
The upgrade SQL assumes the functions will resolve.
With a non-default path this isn't true and results in:
ERROR 1305 (42000) at line 3211: FUNCTION extract_schema_from_file_name does not exist
Resolve by setting PATH = 'CURRENT_SCHEMA' for the upgrade.
Idea thanks to Sergei Golubchik
Issue:
wsrep_sst_rsync.sh interpolated WSREP_SST_OPT_REMOTE_USER and
WSREP_SST_OPT_REMOTE_PSWD verbatim. Because both values originate from
the joiner side of the SST request, a newline in either could splice
an extra directive into the donor-written stunnel.conf (silently
downgrading peer-cert verification) or an extra line into the rsync
magic file. MDEV-39413 had introduced safe() for the same threat class
in wsrep_sst_mariabackup but did not extend it to the rsync script.
Solution:
Routing the rsync interpolations through safe() closes the gap, and
extending safe() to also reject tab and newline ensures multi-line
values cannot survive into a config-file heredoc.
scripts/fill_help_tables.sql from
https://github.com/mariadb-corporation/mariadb-docs/ as of
3fb208fcdfed06beb52ef9e305017de2387ff012
Other URL changes are taken from the HTTP redirection URL
from the old KB.
Some test cases where impacted:
* ps_missed_cmds{,_bin_prot} - used a topic_id of 0 that is
now used, used MAX_INT instead.
* range_notembedded used the help data as a test case. As its not
a stable table data, replaces with static data. Test case valid
by ensuring the ANALYZE EXPLAIN and optimizer trace had the same
structure. Also valided the MDEV-24739/MDEV-25069 bug fixes from
where the test case originaled, when reverted, exhibited the
previous behavour before the bug fixes.
In the build directory the galera sst scripts where
without executable permissions.
This is as a result of [1]. In this patch removing
WSREP_SST_SCRIPTS out of SERVER_SCRIPTS resulting
in the missing from BIN_SCRIPTS which all had an
explicit chmod executed on the script.
Corrected by explictly changing the permission on
the script as would have been done previously.
Thanks Marko Mäkelä for reporting.
[1] efb70285b908be57cc50934a6c8f994773d7583
introduce safe() wrapper for parameters that can later be interpolated
into command line.
Use as
- var="foo$BAR"
+ var="foo$(safe BAR)"
A parameter is safe, if it contains no spaces, single quotes or backticks
The wsrep SST scripts weren't part of the server-galera
component so CPack didn't put them in the MariaDB-server-galera
package.
The previous existing per file component wasn't used so this
was simplified.
Following the previous commit, we apply the same
to RPM.
Add new package Mariadb-server-galera to RPM build
recude mariadb-server package dependencies.
Galera cluster scripts and libraries are currently
in package mariadb-server. Move Galera cluster
scripts and libraries to new package mariadb-server-galera
as most of the people does not need them and
it allows to slim down mariadb-server package
The MariaDB-test package unlike its Debian
equal, didn't depend on the MariaDB-server
(and now it does), except if WITH_WSREP is
specified in which case it depends on the
MariaDB-server-galera package
Also from a865a7c23b
lsof, socat and pv are RECOMMENDS.
* lsof/socat - when selinux is enabled, these aren't used
* pv - the galera sst scripts will silently ignore its absence
Updated/Edited by: Daniel Black <daniel@mariadb.org>
