Failure:
- Filesystem flush before transfer only checks symlinks directly under
the data directory, missing a per-table DATA/INDEX DIRECTORY symlink
one level deeper (MyISAM, Aria), and missing InnoDB's per-table DATA
DIRECTORY entirely, since that uses a '.isl' text file, not a symlink.
- The database directory transfer recreates symlinks as symlinks. The
joiner's rsync daemon cannot keep an absolute symlink target as-is,
so a MyISAM or Aria table placed by DATA DIRECTORY or INDEX DIRECTORY
comes out unreadable after the SST.
Fix:
- Extend the flush's symlink search to two levels deep, and add a scan
for '.isl' files, flushing the filesystem each one points to.
- Transfer the database directories with --copy-links instead of
--links, so a table file reached through a symlink is copied by
content instead of recreated as a broken symlink.
Test:
- MDEV-40727.test creates an Aria table with DATA DIRECTORY and INDEX
DIRECTORY and an InnoDB table with DATA DIRECTORY, forces a full SST,
then checks the donor's log for the flush of both external
directories and checksums every table in database test between the
two nodes.
Assisted-by: https://mariadb.org/governance/governance-ai-policy/
The rsync SST donor ran a bare sync() between the table flush and the
transfer. sync() flushes every mounted filesystem on the host, so its
duration depends on I/O unrelated to the SST. The donor holds FTWRL,
LOCK_log and a paused provider at that point, so the whole server is
frozen until sync() returns.
Flush only the filesystems that hold the files to transfer: the data
directory, an absolute binlog directory, and each symlinked database
directory.
Use the full sync() where 'sync -f' (syncfs) is not available.
Replace the remaining Knowledge Base links with their MariaDB
Documentation equivalents, and fix the 14 help table URLs pointing at
/README pages that do not exist. Only URLs change in
fill_help_tables.sql.
Merging upward: fill_help_tables.sql conflicts at 10.11->11.4 and
11.8->12.3. At those two merges keep the target branch's version,
since 11.4, 11.8 and 12.3 each carry their own URL fix. From 12.3 to
13.1 and 13.1 to main it merges cleanly; take the incoming change.
.github/pull_request_template.md is deleted in 12.3; keep the deletion.
Replace the remaining Knowledge Base links with their MariaDB
Documentation equivalents, including the 838 URLs in the help tables.
Only URLs change in fill_help_tables.sql.
Merging upward: fill_help_tables.sql conflicts at 10.11->11.4 and
11.8->12.3. At those two merges keep the target branch's version,
since 11.4, 11.8 and 12.3 each carry their own URL fix. From 12.3 to
13.1 and 13.1 to main it merges cleanly; take the incoming change.
.github/pull_request_template.md is deleted in 12.3; keep the deletion.
mariadb-upgrade silently dropped EXECUTE grants on sys schema stored
functions and procedures. The sys schema install scripts reinstalled
every routine with DROP FUNCTION/PROCEDURE IF EXISTS followed by
CREATE. DROP cascades to delete the routine's rows in
mysql.procs_priv, so any EXECUTE grant a DBA had issued on e.g.
sys.table_exists or sys.quote_identifier was lost every time
mariadb-upgrade reinstalled the sys schema, even though the routine
itself came back unchanged.
Fix: replace DROP ... IF EXISTS + CREATE with CREATE OR REPLACE in
all 53 sys_schema function/procedure files and in the two templates
(templates/function.sql, templates/procedure.sql) so future routines
follow the same pattern. CREATE OR REPLACE PROCEDURE/FUNCTION goes
through sp_drop_routine_internal(), which only deletes the
mysql.proc row and never reaches sp_revoke_privileges() (that is
only called from the explicit DROP PROCEDURE/FUNCTION statement),
so mysql.procs_priv is left untouched and existing grants survive.
This mirrors the pattern already used by sys schema views
(CREATE OR REPLACE ... VIEW, since MDEV-9077), which never had this
problem.
Four of the converted files (functions/format_path.sql,
functions/ps_is_account_enabled_57.sql,
procedures/ps_setup_reset_to_default.sql,
procedures/ps_trace_thread_57.sql) are not referenced by
scripts/sys_schema/CMakeLists.txt; they were converted anyway for
consistency and have no behavioural effect.
As a side effect, a pre-existing UDF whose name collides with a sys
routine name is no longer destroyed before the reinstall fails:
DROP FUNCTION IF EXISTS resolved the UDF namespace first, silently
dropping the UDF and then failing on ER_SP_ALREADY_EXISTS anyway;
CREATE OR REPLACE fails immediately on ER_UDF_EXISTS with the UDF
intact.
scripts/maria_add_gis_sp.sql.in and the sys_config triggers were
deliberately left untouched: the GIS procedures are only
re-installed at bootstrap time (mariadb-upgrade instead patches
their definer in place via UPDATE), and triggers carry no
procs_priv rows, so neither is on the code path this bug is about.
Added mysql-test/main/mysql_upgrade_sys_routine_grants.test, which
grants EXECUTE on sys.table_exists and sys.quote_identifier,
overwrites both routine bodies with a marker to prove the upgrade
actually reinstalls them (rather than the sys schema install being
skipped), runs mariadb-upgrade, and checks both that the grants
survived and that the real routine bodies came back.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Issue: mariadb-backup SST unconditionally passes socat's "commonname="
option; some socat builds don't register it, so parseopts() rejects
it as unknown regardless of value, breaking all SSL-encrypted SST.
Solution: probe the socat binary once for commonname support and
drop the option when unsupported.
Backported the fix done by Pekka Lampio for mariaDB 11.4 in PR
https://github.com/mariadb-corporation/codership-mariadb-server/pull/543
The PR has a fix for wsrep_sst_rsync script and new mtr test:
galera_3nodes.galera_mdev_36677" to check that the rsync SST method of Galera
works correctly also when the joiner node store InnoDB log files in a dedicated
directory separate from the data dictionary
Note: merging this PR to 11.4 may not be fully functional as there are other
changes in the rsync SST script. Take a look at the original 11.4 PR when merging.
With log_bin=ON a transaction is committed via two-phase commit (the
binary log is the second participant), so it passes through the InnoDB
XA-prepare state. While a donor is held in BLOCK_COMMIT for a mariabackup
backup, its parallel appliers (wsrep_slave_threads > 1) leave one or more
such writesets prepared-but-not-yet-committed, and the snapshot captures
them. On a freshly SST'd joiner nothing resolves these prepared
transactions: binlog crash recovery does not run (the joiner has no in-use
binlog to recover from), and the wsrep continuity-based commit is inactive
because wsrep_emulate_bin_log is FALSE when log_bin is ON. The leftover
prepared transactions then abort startup with "Found <N> prepared
transactions!". Note this does not depend on the prepared set being
non-contiguous - even a contiguous run aborts, because nothing commits
or rolls it back.
Rollback these transactions in xarecover_handlerton(). If rollback fails
flag error to cause unireg_abort().
After a mariabackup SST the joiner could fail with
ER_GTID_STRICT_OUT_OF_ORDER (error 1950)
while re-binlogging transactions received over IST.
The cause is that the binary log copied from the donor carries a
Gtid_list whose position can be ahead of the storage-engine snapshot
because getting snapshot and binlog are no mutually atomic, so transactions
can be present in the copied binlog that are not committed in the copied
engine snapshot.
After the SST the joiner reports the (committed) engine position to the
cluster, IST resends those transactions, and re-binlogging them under
gtid_strict_mode=ON collides with the ahead Gtid_list -> error 1950.
(MDEV-34483 made the engine snapshot stop short of the binlog, which is
what exposed this.)
The copied binary log carries no transactions the joiner needs - only a
Gtid_list - so instead of shipping and then having to truncate/reconcile
it, the joiner now starts a fresh binary log and seeds its GTID position
from the storage-engine checkpoint during recovery. That checkpoint is
the committed cluster position, i.e. exactly where IST resumes, so the
joiner's binary log stays in lockstep with the rest of the cluster and
no out-of-order GTID can occur.
This works for both wsrep_gtid_mode settings; only the binlog domain of
the cluster stream differs:
- wsrep_gtid_mode=ON : wsrep_gtid_domain_id (cluster writes are
re-tagged to it), which is the domain stored in the checkpoint;
- wsrep_gtid_mode=OFF: gtid_domain_id (cluster writes keep the node's
configured domain).
Async-replica positions (mysql.gtid_slave_pos) are part of the engine
snapshot and survive the SST unchanged, so a Galera node can still serve
as an async master or replica across the SST.
This commit:
- sql/log.cc: adds wsrep_seed_binlog_gtid_state(), called from
do_binlog_recovery() when the joiner has no binary log, seeding the
binlog GTID state for the cluster domain to the SE checkpoint position.
- scripts/wsrep_sst_mariabackup.sh: no longer moves the donor's binary
log into place on the joiner.
- extra/mariabackup: backward compatibility: keep shipping binlog file
in SST but
- on donor fix the race between rotation and shipping so that the file
shipped is the one that had been rotated;
- on joiner discard shipped binlog in favour of one generated by
wsrep_seed_binlog_gtid_state().
- sql/wsrep_sst.cc: logs the position actually adopted from storage
(the authoritative post-SST position) rather than the script-reported
one.
- sql/handler.cc: downgrades the "Discovered discontinuity in recovered
wsrep transaction XIDs" message in wsrep_order_and_check_continuity()
from warning to debug level. With parallel appliers a snapshot
routinely captures prepared XIDs that are not contiguous with the
engine checkpoint, so this is normal during SST recovery and of no
value in regular operation; the transactions past the checkpoint are
re-delivered by the cluster (IST/SST) regardless.
- Adds an MDEV-38147 MTR test reproducing the issue.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* don't create mariadb-wsrep-new-cluster in the mariadbd-writable path,
the server should not be able to poison the environment with OUTFILE.
Create it in /run
* As in /run it must be deleted by root, let galera_new_cluster delete
it, not the service
* wsrep-start-position cannot be created by root, so avoid a file
for it at all
Assisted-By: Claude:claude-5-opus
Issue:
When ssl-mode required encryption but the means to perform it was missing,
the SST scripts silently fell back to a cleartext transfer:
- wsrep_sst_rsync: ran over plain TCP when the 'stunnel' binary was absent.
- wsrep_sst_mariabackup: socat used a cleartext socket when ssl-mode was set
but no usable cert/key was found (encrypt stayed 0).
Solution:
Abort the SST instead of falling back to an unencrypted transfer when
ssl-mode is not DISABLED but encryption cannot be set up:
- wsrep_sst_rsync: derive the implicit ssl-mode from the SSL config even
when stunnel is absent, then abort with ENOENT if ssl-mode is active
and the stunnel binary is not found.
- wsrep_sst_mariabackup: after reading the SSL configuration, abort with
EINVAL if ssl-mode is not DISABLED but encrypt resolved to 0 (no usable
cert/key).
Reviewed-by: Jan Lindström <jan.lindstrom@mariadb.com>
Issue:
The mariabackup SST script enabled a peer certificate name check
for any ssl-mode starting with VERIFY, so VERIFY_CA behaved like
VERIFY_IDENTITY, unlike the rsync script.
Solution:
For VERIFY_CA verify the chain only - drop the socat
"commonname" check in mariabackup and gate the rsync checkHost/checkIP
on VERIFY_IDENTITY.
Issue:
Verify_ca_matches_cert ran "openssl verify -CAfile <ca> <cert>", which
treats only the first cert in the file as the target. For a leaf+intermediate
bundle against a root-only CA, openssl cannot reach the root, so the check
aborts the SST (exit 22).
Solution:
Pass the cert file as -untrusted so bundled intermediates complete
the chain to the CA. Trust anchor is unchanged (still -CAfile only).
Issue: The SST TLS options are not role-specific, so a node cannot
present a different certificate as donor (TLS client) than as joiner
(TLS server), which some environments require.
Solution: In the shared check_server_ssl_config, add role-specific
options that override the generic ones: ssl-server-{ca,capath,cert,key}
for the joiner and ssl-client-{ca,capath,cert,key} for the donor.
Applies to both mariabackup and rsync SST.
An appropriately privileged user (with SUPER privileges) could
execute shell commands as the uid of the mariadbd process
because the values of the system variable wsrep_sst_auth,
which can be modified at runtime, were not properly
sanitized when used to construct a shell command.
Combined rsync and mariabackup test cases and added
test case for incorrect values for wsrep_sst_auth.
Issue:
wsrep_sst_rsync.sh interpolated WSREP_SST_OPT_REMOTE_USER and
WSREP_SST_OPT_REMOTE_PSWD verbatim. Because both values originate from
the joiner side of the SST request, a newline in either could splice
an extra directive into the donor-written stunnel.conf (silently
downgrading peer-cert verification) or an extra line into the rsync
magic file. MDEV-39413 had introduced safe() for the same threat class
in wsrep_sst_mariabackup but did not extend it to the rsync script.
Solution:
Routing the rsync interpolations through safe() closes the gap, and
extending safe() to also reject tab and newline ensures multi-line
values cannot survive into a config-file heredoc.
scripts/fill_help_tables.sql from
https://github.com/mariadb-corporation/mariadb-docs/ as of
3fb208fcdfed06beb52ef9e305017de2387ff012
Other URL changes are taken from the HTTP redirection URL
from the old KB.
Some test cases where impacted:
* ps_missed_cmds{,_bin_prot} - used a topic_id of 0 that is
now used, used MAX_INT instead.
* range_notembedded used the help data as a test case. As its not
a stable table data, replaces with static data. Test case valid
by ensuring the ANALYZE EXPLAIN and optimizer trace had the same
structure. Also valided the MDEV-24739/MDEV-25069 bug fixes from
where the test case originaled, when reverted, exhibited the
previous behavour before the bug fixes.
introduce safe() wrapper for parameters that can later be interpolated
into command line.
Use as
- var="foo$BAR"
+ var="foo$(safe BAR)"
A parameter is safe, if it contains no spaces, single quotes or backticks
In case if the view mysql.user was created (e.g. in 10.6) with a
pre- 18edb0959f server with an unexpected
character_set_client or collation_connection, e.g.
utf8mb3 and utf8mb3_general_ci, mysql_upgrade did not fix it to the
expected latin1 and latin1_swedish_ci.
Since 11.8 this could often lead to "Illegax mix of collations" errors
when querying mysql.user, because since 11.8 the default collation for
utf8mb3 is utf8mb3_uca1400_ai_ci, according to the default
@@character_set_collations. For consistency, it's better to fix the
problem starting from 11.4.
For example:
MariaDB [test]> select user,host,is_role from
-> mysql.user where is_role='N';
ERROR 1267 (HY000): Illegal mix of collations (utf8mb3_general_ci,COERCIBLE)
and (utf8mb3_uca1400_ai_ci,COERCIBLE) for operation '='
Fixing mariadb_system_tables_fix.sql to drop the view if it has non-standard
character set or collations, so it gets recreated again correctly by
the CREATE statement in mariadb_system_tables.sql
After moving the systemd service to using environment files
instead of `systemctl set-environment` in 11.6 (MDEV-19210),
they (wsrep-new-cluster and wsrep-start-position) are located
in /var/lib/mysql along with the socket file in
Fedora/RHEL-based distros. This causes them to have incorrect
selinux permissions and therefore be not readable by systemd.
A solution is to generate these files in the run directory,
instead, which already has correct selinux label mysqld_var_run_t
mysql-selinux-1.0.12. Dissociating these files and the socket
in CMake configs can also prove useful for other things.
This also corrects some of the duplicate code in the build
scripts and made INSTALL_RUNDATADIR into a proper location
and used this for the tmpfiles where the temporary files
are created.
Debian's location is /run/mysqld/ matching its INSTALL_UNIX_ADDRDIR,
which is now a temporary location controlled by tmpfiles.
Fix security issue where temporary output files containing SQL commands
and password hashes were created with default permissions (world-readable).
Modified prepare() function to create and umask the $output $config and
$command files. It is more secure than chmod as files can be opened, before
the chmod is applied, and then read when its its later populated.
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.
After moving the systemd service to using environment files
instead of `systemctl set-environment` in 11.6 (MDEV-19210),
they (wsrep-new-cluster and wsrep-start-position) are located
in /var/lib/mysql along with the socket file in
Fedora/RHEL-based distros. This causes them to have incorrect
selinux permissions and therefore be not readable by systemd.
A solution is to generate these files in the run directory,
instead, which already has correct selinux label mysqld_var_run_t
mysql-selinux-1.0.12. Dissociating these files and the socket
in CMake configs can also prove useful for other things.
This also corrects some of the duplicate code in the build
scripts and made INSTALL_RUNDATADIR into a proper location
and used this for the tmpfiles where the temporary files
are created.
Debian's location is /run/mysqld/ matching its INSTALL_UNIX_ADDRDIR,
which is now a temporary location controlled by tmpfiles.
From downstream bug RHEL-116757, the use of ss to map all the
usage of ports to pids and process names under selinux is rather
limited by the mysqld_exec_t limitations.
The checking of the mapping is a bit excessive in a selinux
environment where binding is limited.
We replace the use of ss under selinux to raw reading of the
/proc/tcp{,} interfaces of the linux that aren't selinux restricted.
