database/mariadb
1
0
Fork 0
mirror of https://github.com/MariaDB/server.git synced 2025-07-29 05:21:33 +03:00
Code Activity

MDEV-4991: GTID binlog indexing

Browse Source 
Improve the performance of slave connect using B+-Tree indexes on each binlog
file. The index allows fast lookup of a GTID position to the corresponding
offset in the binlog file, as well as lookup of a position to find the
corresponding GTID position.

This eliminates a costly sequential scan of the starting binlog file
to find the GTID starting position when a slave connects. This is
especially costly if the binlog file is not cached in memory (IO
cost), or if it is encrypted or a lot of slaves connect simultaneously
(CPU cost).

The size of the index files is generally less than 1% of the binlog data, so
not expected to be an issue.

Most of the work writing the index is done as a background task, in
the binlog background thread. This minimises the performance impact on
transaction commit. A simple global mutex is used to protect index
reads and (background) index writes; this is fine as slave connect is
a relatively infrequent operation.

Here are the user-visible options and status variables. The feature is on by
default and is expected to need no tuning or configuration for most users.

binlog_gtid_index
  On by default. Can be used to disable the indexes for testing purposes.

binlog_gtid_index_page_size (default 4096)
  Page size to use for the binlog GTID index. This is the size of the nodes
  in the B+-tree used internally in the index. A very small page-size (64 is
  the minimum) will be less efficient, but can be used to stress the
  BTree-code during testing.

binlog_gtid_index_span_min (default 65536)
  Control sparseness of the binlog GTID index. If set to N, at most one
  index record will be added for every N bytes of binlog file written.
  This can be used to reduce the number of records in the index, at
  the cost only of having to scan a few more events in the binlog file
  before finding the target position

Two status variables are available to monitor the use of the GTID indexes:

  Binlog_gtid_index_hit
  Binlog_gtid_index_miss

The "hit" status increments for each successful lookup in a GTID index.
The "miss" increments when a lookup is not possible. This indicates that the
index file is missing (eg. binlog written by old server version
without GTID index support), or corrupt.

Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
This commit is contained in:
Kristian Nielsen
2023-09-08 13:12:49 +02:00
parent 20741b9237
commit d039346a7a
32 changed files with 4315 additions and 256 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
mysql-test/suite/binlog_encryption/encrypted_master_switch_to_unencrypted_gtid.result
View File

@ -6,6 +6,7 @@ connection server_2;
include/stop_slave.inc
CHANGE MASTER TO MASTER_USE_GTID=SLAVE_POS;
call mtr.add_suppression(" Got fatal error 1236 from master when reading data from binary log: 'Could not set up decryption for binlog.'");
call mtr.add_suppression(" Got fatal error 1236 from master when reading data from binary log: 'Could not decrypt binlog: encryption key error");
#####################################################
# Part 1: unencrypted master
#####################################################
@ -58,10 +59,11 @@ INSERT INTO table3_no_encryption SELECT NULL,NOW(),b FROM table3_no_encryption;
connection server_2;
start slave;
include/wait_for_slave_io_error.inc [errno=1236]
# Ensuring slave was unable to replicate any transactions..
# Ensuring slave was unable to replicate any encrypted transactions..
# ..success
SHOW TABLES;
Tables_in_test
table1_no_encryption
include/stop_slave_sql.inc
reset slave;
##########
@ -80,5 +82,7 @@ COUNT(*)
4
DROP TABLE table1_no_encryption, table2_to_encrypt, table3_no_encryption;
connection server_2;
RESET MASTER;
SET GLOBAL gtid_slave_pos= '';
include/start_slave.inc
include/rpl_end.inc

10
mysql-test/suite/binlog_encryption/encrypted_master_switch_to_unencrypted_gtid.test
View File

@ -36,6 +36,7 @@
CHANGE MASTER TO MASTER_USE_GTID=SLAVE_POS;
--enable_connect_log
call mtr.add_suppression(" Got fatal error 1236 from master when reading data from binary log: 'Could not set up decryption for binlog.'");
call mtr.add_suppression(" Got fatal error 1236 from master when reading data from binary log: 'Could not decrypt binlog: encryption key error");
--echo #####################################################
--echo # Part 1: unencrypted master
@ -55,6 +56,7 @@ FLUSH BINARY LOGS;
SET binlog_format=ROW;
INSERT INTO table1_no_encryption SELECT NULL,NOW(),b FROM table1_no_encryption;
INSERT INTO table1_no_encryption SELECT NULL,NOW(),b FROM table1_no_encryption;
--let $last_unencrypted_gtid= `SELECT @@gtid_binlog_pos`
# Make sure that binary logs are not encrypted
@ -120,11 +122,11 @@ start slave;
--let $slave_io_errno= 1236
--source include/wait_for_slave_io_error.inc
--echo # Ensuring slave was unable to replicate any transactions..
--echo # Ensuring slave was unable to replicate any encrypted transactions..
--let $gsp= `SELECT @@global.gtid_slave_pos`
if (`SELECT strcmp("$gsp","")`)
if (`SELECT strcmp("$gsp","$last_unencrypted_gtid")`)
{
die Slave without encryption configured should fail to read encrypted binlog;
die Slave without encryption configured should fail to read encrypted binlog (expected $last_unencrypted_gtid but got $gsp);
}
--echo # ..success
@ -151,5 +153,7 @@ DROP TABLE table1_no_encryption, table2_to_encrypt, table3_no_encryption;
--connection server_2
--disable_connect_log
RESET MASTER;
SET GLOBAL gtid_slave_pos= '';
--source include/start_slave.inc
--source include/rpl_end.inc