Issue:
The Commit: 0584846 'Add TL_FIRST_WRITE in SQL layer for determining R/W'
limits INSERT statements on write nodes to acquire MDL locks on it's all child
tables and thereby wsrep certification keys added, but on applier nodes it does
acquire MDL locks for all child tables. This can result into MDL BF-BF conflict
on applier node when transactions referring to parent and child tables are
executed concurrently. For example:
Tables with foreign keys: t1<-t2<-t3<-t4
Conflicting transactions: INSERT t1 and DROP TABLE t4
Wsrep certification keys taken on write node:
- for INSERT t1: t1 and t2
- for DROP TABLE t4: t4
On applier node MDL BF-BF conflict happened between two transaction because
MDL locks on t1, t2, t3 and t4 were taken for INSERT t1, which conflicted
with MDL lock on t4 taken by DROP TABLE t4.
The Wsrep certification keys helps in resolving this MDL BF-BF conflict by
prioritizing and scheduling concurrent transactions. But to generate Wsrep
certification keys it needs to open and take MDL locks on all the child tables.
The Commit: 0584846 change limits MDL lock to be taken on all child nodes for
read-only FK checks (INSERT t1). But this doesn't works on applier nodes
because Write_rows_log_event event logged for INSERT is also used to record
update (check Write_rows_log_event::get_trg_event_map()), and
therefore MDL locks is taken for all the child tables on applier node
for update and insert event.
Solution:
Additional keys for the referenced/foreign table needs to be added
to avoid potential MDL conflicts with concurrent update and DDLs.
Problem was that for partitioned tables base table storage engine
is DB_TYPE_PARTITION_DB and naturally different than DB_TYPE_INNODB
so operation was not allowed in Galera.
Fixed by requesting implementing storage engine for partitioned
tables i.e. table->file->partition_ht() or if that does not exist
we can use base table storage engine. Resulting storage engine
type is then used on condition is operation allowed when
wsrep_mode=DISALLOW_LOCAL_GTID or not. Operations to InnoDB
storage engine i.e DB_TYPE_INNODB should be allowed.
Cluster configuration was incorrect e.g. wsrep_node_address
was missing. Therefore, Galera replication was not properly
initialized and TOI is not supported.
Fix is to check when user tries to start Galera replication
with wsrep_on=ON that Galera replication is properly
initialized and node is ready to receive operations. If
Galera replication is not properly initialized return
a error.
Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>
This commit fixes a bug where Aria tables are used in
(master->slave1->slave2) and a backup is taken on slave2. In this case
it is possible that the replication position in the backup, stored in
mysql.gtid_slave_pos, will be wrong. This will lead to replication
errors if one is trying to use the backup as a new slave.
Analyze:
Replicated row events are committed with trans_commit_stmt() and
thd->transaction->all.ha_list != 0.
This means that backup_commit_lock is not taken for Aria tables,
which means the rows are committed and binary logged on the slave
under BLOCK_COMMIT which should not happen.
This issue does not occur on the master as thd->transaction->all.ha_list
is == 0 under AUTO_COMMIT, which sets 'is_real_trans' and 'rw_trans'
which in turn causes backup_commit_lock to be taken.
Fixed by checking in ha_check_and_coalesce_trx_read_only() if all handlers
supports rollback and if not, then wait for BLOCK_COMMIT also for
statement commit.
Issue:
Mariadb acquires additional MDL locks on UPDATE/INSERT/DELETE statements
on table with foreign keys. For example, table t1 references t2, an
UPDATE to t1 will MDL lock t2 in addition to t1.
A replica may deliver an ALTER t1 and UPDATE t2 concurrently for
applying. Then the UPDATE may acquire MDL lock for t1, followed by a
conflict when the ALTER attempts to MDL lock on t1. Causing a BF-BF
conflict.
Solution:
Additional keys for the referenced/foreign table needs to be added
to avoid potential MDL conflicts with concurrent update and DDLs.
Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>
Problem was that initial GTID was set on wsrep_before_prepare
out-of-order. In practice GTID was set to same as previous
executed transaction GTID. In recovery valid GTID was found
from prepared transaction and this transaction is committed
leading to fact that same GTID was executed twice.
This is fixed by setting invalid GTID at wsrep_before_prepare
and later in wsrep_before_commit actual correct GTID is set
and this setting is done while we are in commit monitor i.e.
assigment is done in order of replication.
In recovery if prepared transaction is found we check its
GTID, if it is invalid transaction will be rolled back
and if it is valid it will be committed.
Initialize gtid seqno from recovered seqno when
bootstrapping a new cluster.
Added two test cases for both mariabackup and rsync SST methods
to show that GTIDs remain consistent on cluster and that
all expected rows are in the table.
Added tests for wsrep GTID recovery with binlog on and off.
Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>
Problem was missing case from wsrep_handle_mdl_conflict. Test case
was trying to confirm that LOCK TABLE thread is not BF-aborted.
However as case was missing it was BF-aborted. Test case passed
because BF-aborting takes time and used wait condition might
see expected thread status before it was BF-aborted. Test naturally
failed if BF-aborting was done early enough.
Fix is to add missing case for SQLCOM_LOCK_TABLES to
wsrep_handle_mdl_conflict.
Note that using LOCK TABLE is still not recomended on cluster
because it could cause cluster hang.
This is a 10.5 specific commit that will then be overridden by
another one for 10.6+.
Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>
Problem was missing case from wsrep_handle_mdl_conflict. Test case
was trying to confirm that LOCK TABLE thread is not BF-aborted.
However as case was missing it was BF-aborted. Test case passed
because BF-aborting takes time and used wait condition might
see expected thread status before it was BF-aborted. Test naturally
failed if BF-aborting was done early enough.
Fix is to add missing case for SQLCOM_LOCK_TABLES to
wsrep_handle_mdl_conflict.
Note that using LOCK TABLE is still not recomended on cluster
because it could cause cluster hang.
Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>
Fix a regression that caused assertion thd->is_error() after
sync wait failures. If wsrep_sync_wait() fails make sure a appropriate
error is set. Partially revert 75dd0246f8.
Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>
Problem was incorrect handling of partitioned tables,
because db_type == DB_TYPE_PARTITION_DB
wsrep_should_replicate_ddl incorrectly marked
DDL as not replicatable. However, in partitioned
tables we should check implementing storage engine
from table->file->partition_ht() if available because
if partition handler is InnoDB all DDL should be allowed
even with wsrep_strict_ddl. For other storage engines
DDL should not be allowed and error should be issued.
This is 10.5 version of the fix.
Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>
Issue:
Mariadb Galera cluster fails to replicate from
Mysql 5.7 when configured with MASTER_USE_GTID=no
option for CHANGE MASTER.
HOST: mysql, mysql 5.7.44 binlog_format=ROW
HOST: m1, mariadb 10.6 GALERA NODE replicating from HOST mysql, Using_Gtid: No (log file and position)
HOST: m2 mariadb 10.6 GALERA NODE
HOST: m3 mariadb 10.6 GALERA NODE
Error on m1:
2024-05-22 16:11:07 1 [ERROR] WSREP: Vote 0 (success) on 78cebda7-1876-11ef-896b-8a58fca50d36:2565 is inconsistent with group. Leaving cluster.
Error on m2 and m3:
2024-05-22 16:11:06 2 [ERROR] Error in Log_event::read_log_event(): 'Found invalid event in binary log', data_len: 42, event_type: -94
2024-05-22 16:11:06 2 [ERROR] WSREP: applier could not read binlog event, seqno: 2565, len: 482
It fails in Gtid_log_event::is_valid() check on
secondary node when sequence number sent from primary
is 0. On primary for applier or slave thread sequence
number is set to 0, when both thd->variables.gtid_seq_no
and thd->variables.wsrep_gtid_seq_no have value 0.
Solution:
Skip adding Gtid Event on primary for applier or
slave thread when both thd->variables.gtid_seq_no
and thd->variables.wsrep_gtid_seq_no have value 0.
Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>
When client connections use threadpool, i.e. configuration has:
thread_handling = pool-of-threads it turned out that during wsrep
replication shutdown, not all client connections could be closed.
Reason was that some client threads has stmt_da in state DA_EOF,
and this state was earlier used to detect if client connection
was issuing SHUTDOWN command.
To fix this, the connection executing SHUTDOWN is now detected by
looking at the actual command being executed:
thd->get_command() == COM_SHUTDOWN
During replication shutdown, all other connections but the
SHUTDOWN executor, are terminated.
This commit has new mtr test galera.galera_threadpool, which
opens a number of threadpool client connections, and then
restarts the node to verify that connections in threadpool are
terminated during shutdown.
Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>
DROP TABLE on child and UPDATE of parent table can cause an MDL BF-BF
conflict when applied concurrently.
DROP TABLE takes MDL locks on both child and its parent table, however
it only it did not add certification keys for the parent table.
This patch adds the following:
* Append certification keys corresponding to all parent tables
before DROP TABLE replication.
* Fix wsrep_append_fk_parent_table() so that it works when it is
given a table list containing temporary tables.
* Make sure function wsrep_append_fk_parent_table() is only called
for local transaction. That was not the case for ALTER TABLE.
* Add a test case that verifies that UPDATE parent depends on
preceeding DROP TABLE child.
* Adapt galera_ddl_fk_conflict test to work with DROP TABLE as well.
Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>
