database/mariadb
1
0
Fork 0
mirror of https://github.com/MariaDB/server.git synced 2025-07-30 16:24:05 +03:00
Code Activity

MDEV-24966 Galera multi-master regression

Browse Source 
After the merging of MDEV-24915, 10.6 branch has regressions with handling of
concurrent write load against two or more cluster nodes. These regressions may
surface as cluster hanging, node crashes or data inconsistency. With some test
scenarios, the only visible symptom could be that the BF victim aborting happens
only by innodb lock wait timeout expiration. This would result only to poor
performance (by default 50 sec hang for each BF conflict), and could be somewhat
difficult to diagnose.

This pull request has following fixes to handle concurrent write load from
multiple nodes:

In lock_wait_wsrep_kill(), the victim trx was expected to be only in
TRX_STATE_ACTIVE state. With the delayed BF conflict handling, it can happen
that victim has advanced into pre commit state. This was fixed by choosing
victim both in TRX_STATE_ACTIVE and TRX_STATE_PREPARED states.

Victim transaction may be in several different states at the time of detected
lock conflict, and due to delayed BF aborting practice in MDEV-24915, the victim
may advance further before the actual BF aborting takes place. The BF aborting
in MDEV-24915 did not wake the victim, if it was in the state of waiting for
some other lock (than the one that was blocking the high priority thread).
This anomaly caused the innodb lock wait timeout expiration delays and poor
performance symptom. To fix this, lock_wait_wsrep_kill() now looks if
victim is in lock waiting state, and uses lock_cancel_waiting_and_release()
to cancel this lock wait.

wsrep_bf_abort() checks if the victim has active transaction (in wsrep-lib),
and starts a new transaction if there was no active transaction before.
Due to late BF aborting, the victim may have e.g. failed in certification
and is already aborting or has aborted at this stage. This has caused
problems in testing where BF aborter tries to BF abort himself.
The fix in wsrep_bf_abort() now skips the BF abort, if victim is aborting
or has aborted. Victim may not have started transaction yet in wsrep context,
but it may have acquired MDL locks (due to DDL execution), and this has
caused BF conflict. Such case does not require aborting in wsrep or
replication provider state.

BF aborting could cause BF-BF conflict scenario, if victim was already aborted
and changed to replayer having high priority as well. This BF-BF conflict
scenario is now avoided in lock_wait_wsrep() where we now check if blocking
lock holder is also high priority and is ordered before, caller should wait
for the lock in this situation.

The natural innodb deadlock resolving algorithm could pick BF thread as
deadlock victim. This is fixed by giving max weigh to BF threads in
Deadlock::report().

MDEV-24341 has changed excution paths in do_command() and this affects BF
aborted victim execution. This PR fixes one assert in do_command():
 DBUG_ASSERT(!thd->async_state.pending_ops())
Which fired if the thd was BF aborted earlier. This assert is now changed
to allow pending_ops() if thd was BF aborted before.

With these fixes, long term highly conflicting write load could be run against
to node cluster. If binlogging is configured, log_slave_updates should be
also set.
This commit is contained in:
sjaakola
2021-02-28 19:33:30 +02:00
committed by Jan Lindström
parent f74704c7d9
commit a1e70388c4
13 changed files with 129 additions and 501 deletions
Download Patch File Download Diff File Expand all files Collapse all files

176
mysql-test/suite/wsrep/t/variables_debug.test
View File

@ -1,176 +0,0 @@
--source include/have_wsrep.inc
--source include/force_restart.inc
--source include/have_innodb.inc
--source include/galera_have_debug_sync.inc
call mtr.add_suppression("WSREP: Initial position was provided by configuration or SST, avoiding override");
SET @wsrep_provider_options_saved= @@global.wsrep_provider_options;
SET @wsrep_cluster_address_saved= @@global.wsrep_cluster_address;
--echo
--echo # MDEV#5534: mysql_tzinfo_to_sql generates wrong query
--echo #
--echo # Testing wsrep_replicate_myisam variable.
--error ER_INCORRECT_GLOBAL_LOCAL_VAR
SELECT @@session.wsrep_replicate_myisam;
SELECT @@global.wsrep_replicate_myisam;
--error ER_GLOBAL_VARIABLE
SET SESSION wsrep_replicate_myisam= ON;
SET GLOBAL wsrep_replicate_myisam= ON;
# Reset it back.
SET GLOBAL wsrep_replicate_myisam= OFF;
SET GLOBAL wsrep_provider=none;
--echo #
--echo # MDEV#5790: SHOW GLOBAL STATUS LIKE does not show the correct list of
--echo # variables when using "_"
--echo #
CALL mtr.add_suppression("WSREP: Could not open saved state file for reading.*");
--disable_query_log
eval SET GLOBAL wsrep_provider= '$WSREP_PROVIDER';
--enable_query_log
--replace_column 2 #
SHOW GLOBAL STATUS LIKE 'wsrep%';
--echo
--replace_column 2 #
SHOW GLOBAL STATUS LIKE 'wsrep_%';
--replace_column 2 #
SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment';
--echo # Should show nothing.
SHOW STATUS LIKE 'x';
# Reset it back.
SET GLOBAL wsrep_provider=none;
--disable_query_log
eval SET GLOBAL wsrep_provider= '$WSREP_PROVIDER';
--enable_query_log
# The following 2 variables are used by mariabackup
# SST.
--echo
--replace_column 2 #
SHOW STATUS LIKE 'wsrep_local_state_uuid';
--echo
--replace_column 2 #
SHOW STATUS LIKE 'wsrep_last_committed';
# Reset it back.
SET GLOBAL wsrep_provider=none;
--echo
--echo #
--echo # MDEV#6206: wsrep_slave_threads subtracts from max_connections
--echo #
call mtr.add_suppression("WSREP: Failed to get provider options");
--disable_query_log
eval SET GLOBAL wsrep_provider= '$WSREP_PROVIDER';
--enable_query_log
--replace_regex /.*libgalera_smm.*/libgalera_smm.so/
SELECT @@global.wsrep_provider;
SELECT @@global.wsrep_slave_threads;
SELECT @@global.wsrep_cluster_address;
SELECT @@global.wsrep_on;
SHOW STATUS LIKE 'threads_connected';
SHOW STATUS LIKE 'wsrep_thread_count';
--echo
--disable_query_log
eval SET GLOBAL wsrep_provider= '$WSREP_PROVIDER';
--enable_query_log
--replace_regex /.*libgalera_smm.*/libgalera_smm.so/
SELECT @@global.wsrep_provider;
SELECT @@global.wsrep_cluster_address;
SELECT @@global.wsrep_on;
SHOW STATUS LIKE 'threads_connected';
SHOW STATUS LIKE 'wsrep_thread_count';
--echo
--echo # Setting wsrep_cluster_address triggers the creation of
--echo # applier/rollbacker threads.
SET GLOBAL wsrep_cluster_address= 'gcomm://';
--echo # Wait for applier thread to get created 1.
--let $wait_condition = SELECT VARIABLE_VALUE = 1 FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_applier_thread_count';
--source include/wait_condition.inc
--echo # Wait for applier thread to get created 2.
--let $wait_condition = SELECT VARIABLE_VALUE = 1 FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_rollbacker_thread_count';
--source include/wait_condition.inc
SELECT VARIABLE_VALUE AS EXPECT_1 FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_applier_thread_count';
SELECT VARIABLE_VALUE AS EXPECT_1 FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_rollbacker_thread_count';
SELECT VARIABLE_VALUE AS EXPECT_2 FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_thread_count';
--replace_regex /.*libgalera_smm.*/libgalera_smm.so/
SELECT @@global.wsrep_provider;
SELECT @@global.wsrep_cluster_address;
SELECT @@global.wsrep_on;
SHOW STATUS LIKE 'threads_connected';
SHOW STATUS LIKE 'wsrep_thread_count';
--echo
SET @wsrep_slave_threads_saved= @@global.wsrep_slave_threads;
SET GLOBAL wsrep_slave_threads= 10;
--echo # Wait for 9 applier threads to get created.
--let $wait_condition = SELECT VARIABLE_VALUE = 10 FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_applier_thread_count';
--source include/wait_condition.inc
SELECT VARIABLE_VALUE AS EXPECT_10 FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_applier_thread_count';
SELECT VARIABLE_VALUE AS EXPECT_1 FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_rollbacker_thread_count';
SELECT VARIABLE_VALUE AS EXPECT_11 FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_thread_count';
SHOW STATUS LIKE 'threads_connected';
#
# privileges for wsrep_on
#
set wsrep_on=0;
set wsrep_on=1;
--source include/wait_until_connected_again.inc
create user test@localhost;
connect con1,localhost,test;
set auto_increment_increment=10;
--error ER_SPECIFIC_ACCESS_DENIED_ERROR
set wsrep_on=0;
disconnect con1;
connection default;
drop user test@localhost;
--echo #
--echo # MDEV#6411: Setting set @@global.wsrep_sst_auth=NULL causes crash
--echo #
SET @wsrep_sst_auth_saved= @@global.wsrep_sst_auth;
SET @@global.wsrep_sst_auth= 'user:pass';
SELECT @@global.wsrep_sst_auth;
SET @@global.wsrep_sst_auth= '';
SELECT @@global.wsrep_sst_auth;
SET @@global.wsrep_sst_auth= NULL;
SELECT @@global.wsrep_sst_auth;
SET @@global.wsrep_sst_auth= @wsrep_sst_auth_saved;
# Reset (for mtr internal checks)
--disable_query_log
SET GLOBAL wsrep_slave_threads= @wsrep_slave_threads_saved;
eval SET GLOBAL wsrep_provider= '$WSREP_PROVIDER';
SET GLOBAL wsrep_cluster_address= @wsrep_cluster_address_saved;
SET GLOBAL wsrep_provider_options= @wsrep_provider_options_saved;
--enable_query_log
--source include/galera_wait_ready.inc
--echo # End of test.