database/mariadb
1
0
Fork 0
mirror of https://github.com/MariaDB/server.git synced 2025-07-30 16:24:05 +03:00
Code Activity

MDEV-11738: Mariadb uses 100% of several of my 8 cpus doing nothing

Browse Source 
MDEV-11581: Mariadb starts InnoDB encryption threads
when key has not changed or data scrubbing turned off

Background: Key rotation is based on background threads
(innodb-encryption-threads) periodically going through
all tablespaces on fil_system. For each tablespace
current used key version is compared to max key age
(innodb-encryption-rotate-key-age). This process
naturally takes CPU. Similarly, in same time need for
scrubbing is investigated. Currently, key rotation
is fully supported on Amazon AWS key management plugin
only but InnoDB does not have knowledge what key
management plugin is used.

This patch re-purposes innodb-encryption-rotate-key-age=0
to disable key rotation and background data scrubbing.
All new tables are added to special list for key rotation
and key rotation is based on sending a event to
background encryption threads instead of using periodic
checking (i.e. timeout).

fil0fil.cc: Added functions fil_space_acquire_low()
to acquire a tablespace when it could be dropped concurrently.
This function is used from fil_space_acquire() or
fil_space_acquire_silent() that will not print
any messages if we try to acquire space that does not exist.
fil_space_release() to release a acquired tablespace.
fil_space_next() to iterate tablespaces in fil_system
using fil_space_acquire() and fil_space_release().
Similarly, fil_space_keyrotation_next() to iterate new
list fil_system->rotation_list where new tables.
are added if key rotation is disabled.
Removed unnecessary functions fil_get_first_space_safe()
fil_get_next_space_safe()

fil_node_open_file(): After page 0 is read read also
crypt_info if it is not yet read.

btr_scrub_lock_dict_func()
buf_page_check_corrupt()
buf_page_encrypt_before_write()
buf_merge_or_delete_for_page()
lock_print_info_all_transactions()
row_fts_psort_info_init()
row_truncate_table_for_mysql()
row_drop_table_for_mysql()
    Use fil_space_acquire()/release() to access fil_space_t.

buf_page_decrypt_after_read():
    Use fil_space_get_crypt_data() because at this point
    we might not yet have read page 0.

fil0crypt.cc/fil0fil.h: Lot of changes. Pass fil_space_t* directly
to functions needing it and store fil_space_t* to rotation state.
Use fil_space_acquire()/release() when iterating tablespaces
and removed unnecessary is_closing from fil_crypt_t. Use
fil_space_t::is_stopping() to detect when access to
tablespace should be stopped. Removed unnecessary
fil_space_get_crypt_data().

fil_space_create(): Inform key rotation that there could
be something to do if key rotation is disabled and new
table with encryption enabled is created.
Remove unnecessary functions fil_get_first_space_safe()
and fil_get_next_space_safe(). fil_space_acquire()
and fil_space_release() are used instead. Moved
fil_space_get_crypt_data() and fil_space_set_crypt_data()
to fil0crypt.cc.

fsp_header_init(): Acquire fil_space_t*, write crypt_data
and release space.

check_table_options()
	Renamed FIL_SPACE_ENCRYPTION_* TO FIL_ENCRYPTION_*

i_s.cc: Added ROTATING_OR_FLUSHING field to
information_schema.innodb_tablespace_encryption
to show current status of key rotation.
This commit is contained in:
Jan Lindström
2017-03-14 12:56:01 +02:00
parent a2f34809e5
commit 50eb40a2a8
58 changed files with 3220 additions and 2857 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
mysql-test/suite/encryption/t/encrypt_and_grep.opt
View File

@ -1,8 +1,7 @@
--innodb-encrypt-tables=ON
--innodb-encrypt-log=ON
--innodb-encryption-rotate-key-age=15
--innodb-encryption-threads=4
--innodb-encryption-threads=1
--innodb-tablespaces-encryption
--innodb-max-dirty-pages-pct=0.001

29
mysql-test/suite/encryption/t/encrypt_and_grep.test
View File

@ -1,5 +1,5 @@
-- source include/have_innodb.inc
-- source include/have_example_key_management_plugin.inc
-- source include/have_file_key_management_plugin.inc
# embedded does not support restart
-- source include/not_embedded.inc
@ -30,7 +30,10 @@ insert t3 values (repeat('dummy', 42));
--let $wait_condition=SELECT COUNT(*) = 1 FROM INFORMATION_SCHEMA.INNODB_TABLESPACES_ENCRYPTION WHERE MIN_KEY_VERSION = 0
--source include/wait_condition.inc
--sleep 5
SELECT NAME FROM INFORMATION_SCHEMA.INNODB_TABLESPACES_ENCRYPTION WHERE MIN_KEY_VERSION = 0;
SELECT NAME FROM INFORMATION_SCHEMA.INNODB_TABLESPACES_ENCRYPTION WHERE MIN_KEY_VERSION <> 0;
--source include/shutdown_mysqld.inc
--let SEARCH_PATTERN=foobar
--echo # t1 yes on expecting NOT FOUND
@ -49,15 +52,21 @@ insert t3 values (repeat('dummy', 42));
-- let SEARCH_FILE=$ib1_IBD
-- source include/search_pattern_in_file.inc
-- source include/start_mysqld.inc
--echo # Now turn off encryption and wait for threads to decrypt everything
SET GLOBAL innodb_encryption_threads = 4;
SET GLOBAL innodb_encryption_threads = 1;
SET GLOBAL innodb_encrypt_tables = off;
--echo # Wait max 10 min for key encryption threads to decrypt all spaces
--let $wait_timeout= 600
--let $wait_condition=SELECT COUNT(*) = 1 FROM INFORMATION_SCHEMA.INNODB_TABLESPACES_ENCRYPTION WHERE MIN_KEY_VERSION <> 0;
--let $wait_condition=SELECT COUNT(*) = 5 FROM INFORMATION_SCHEMA.INNODB_TABLESPACES_ENCRYPTION WHERE MIN_KEY_VERSION = 0 AND KEY_ROTATION_PAGE_NUMBER IS NULL;
--source include/wait_condition.inc
--sleep 5
SELECT NAME FROM INFORMATION_SCHEMA.INNODB_TABLESPACES_ENCRYPTION WHERE MIN_KEY_VERSION = 0;
SELECT NAME FROM INFORMATION_SCHEMA.INNODB_TABLESPACES_ENCRYPTION WHERE MIN_KEY_VERSION <> 0;
--source include/shutdown_mysqld.inc
--let SEARCH_PATTERN=foobar
--echo # t1 yes on expecting NOT FOUND
@ -76,6 +85,8 @@ SET GLOBAL innodb_encrypt_tables = off;
-- let SEARCH_FILE=$ib1_IBD
-- source include/search_pattern_in_file.inc
-- source include/start_mysqld.inc
--echo # Now turn on encryption and wait for threads to encrypt all spaces
SET GLOBAL innodb_encryption_threads = 4;
SET GLOBAL innodb_encrypt_tables = on;
@ -84,7 +95,11 @@ SET GLOBAL innodb_encrypt_tables = on;
--let $wait_timeout= 600
--let $wait_condition=SELECT COUNT(*) = 1 FROM INFORMATION_SCHEMA.INNODB_TABLESPACES_ENCRYPTION WHERE MIN_KEY_VERSION = 0;
--source include/wait_condition.inc
--sleep 5
SELECT NAME FROM INFORMATION_SCHEMA.INNODB_TABLESPACES_ENCRYPTION WHERE MIN_KEY_VERSION = 0;
SELECT NAME FROM INFORMATION_SCHEMA.INNODB_TABLESPACES_ENCRYPTION WHERE MIN_KEY_VERSION <> 0;
--source include/shutdown_mysqld.inc
--let SEARCH_PATTERN=foobar
--echo # t1 yes on expecting NOT FOUND
@ -103,6 +118,6 @@ SET GLOBAL innodb_encrypt_tables = on;
-- let SEARCH_FILE=$ib1_IBD
-- source include/search_pattern_in_file.inc
--echo # TODO: add shutdown + grep tests
-- source include/start_mysqld.inc
drop table t1, t2, t3;

5
mysql-test/suite/encryption/t/innodb-key-rotation-disable.opt Normal file
View File

@ -0,0 +1,5 @@
--innodb-encrypt-tables
--innodb-encrypt-log
--innodb-encryption-rotate-key-age=0
--innodb-encryption-threads=4
--innodb-tablespaces-encryption

102
mysql-test/suite/encryption/t/innodb-key-rotation-disable.test Normal file
View File

@ -0,0 +1,102 @@
-- source include/have_innodb.inc
-- source include/have_file_key_management_plugin.inc
# not embedded because of restarts
-- source include/not_embedded.inc
SELECT NAME FROM INFORMATION_SCHEMA.INNODB_TABLESPACES_ENCRYPTION WHERE MIN_KEY_VERSION <> 0;
SELECT NAME FROM INFORMATION_SCHEMA.INNODB_TABLESPACES_ENCRYPTION WHERE MIN_KEY_VERSION = 0;
--disable_query_log
--disable_warnings
let $innodb_compression_algorithm_orig=`SELECT @@innodb_compression_algorithm`;
let $innodb_file_format_orig = `SELECT @@innodb_file_format`;
let $innodb_file_per_table_orig = `SELECT @@innodb_file_per_table`;
let $encryption = `SELECT @@innodb_encrypt_tables`;
SET GLOBAL innodb_file_format = `Barracuda`;
SET GLOBAL innodb_file_per_table = ON;
# zlib
set global innodb_compression_algorithm = 1;
--enable_warnings
--enable_query_log
create database enctests;
use enctests;
create table t1(a int not null primary key, b char(200)) engine=innodb;
create table t2(a int not null primary key, b char(200)) engine=innodb row_format=compressed;
create table t3(a int not null primary key, b char(200)) engine=innodb page_compressed=yes;
create table t4(a int not null primary key, b char(200)) engine=innodb encrypted=yes;
create table t5(a int not null primary key, b char(200)) engine=innodb encrypted=yes row_format=compressed;
create table t6(a int not null primary key, b char(200)) engine=innodb encrypted=yes page_compressed=yes;
create table t7(a int not null primary key, b char(200)) engine=innodb encrypted=no;
create table t8(a int not null primary key, b char(200)) engine=innodb encrypted=no row_format=compressed;
create table t9(a int not null primary key, b char(200)) engine=innodb encrypted=no page_compressed=yes;
insert into t1 values (1, 'secredmessage');
insert into t2 values (1, 'secredmessage');
insert into t3 values (1, 'secredmessagecompressedaaaaaaaaabbbbbbbbbbbbbbccccccccccccccc');
insert into t4 values (1, 'secredmessage');
insert into t5 values (1, 'secredmessage');
insert into t6 values (1, 'secredmessagecompressedaaaaaaaaabbbbbbbbbbbbbbccccccccccccccc');
insert into t7 values (1, 'publicmessage');
insert into t8 values (1, 'publicmessage');
insert into t9 values (1, 'pugliccompressedaaaaaaaaabbbbbbbbbbbbbbccccccccccccccc');
--echo # should list tables t1-t6
SELECT NAME,ENCRYPTION_SCHEME,CURRENT_KEY_ID FROM INFORMATION_SCHEMA.INNODB_TABLESPACES_ENCRYPTION WHERE MIN_KEY_VERSION <> 0 AND NAME LIKE 'enctests%';
--echo # should list tables t7-t9
SELECT NAME,ENCRYPTION_SCHEME,CURRENT_KEY_ID FROM INFORMATION_SCHEMA.INNODB_TABLESPACES_ENCRYPTION WHERE MIN_KEY_VERSION = 0 and NAME LIKE 'enctests%';
--error ER_WRONG_VALUE_FOR_VAR
SET GLOBAL innodb_encrypt_tables=OFF;
--error ER_WRONG_VALUE_FOR_VAR
SET GLOBAL innodb_encrypt_tables=ON;
--let $MYSQLD_DATADIR=`select @@datadir`
-- source include/shutdown_mysqld.inc
--let SEARCH_RANGE = 10000000
--let SEARCH_PATTERN=secred
--echo # t1 default on expecting NOT FOUND
-- let SEARCH_FILE=$MYSQLD_DATADIR/enctests/t1.ibd
-- source include/search_pattern_in_file.inc
--echo # t2 default on expecting NOT FOUND
-- let SEARCH_FILE=$MYSQLD_DATADIR/enctests/t2.ibd
-- source include/search_pattern_in_file.inc
--echo # t3 default on expecting NOT FOUND
-- let SEARCH_FILE=$MYSQLD_DATADIR/enctests/t3.ibd
-- source include/search_pattern_in_file.inc
--echo # t4 on expecting NOT FOUND
-- let SEARCH_FILE=$MYSQLD_DATADIR/enctests/t4.ibd
-- source include/search_pattern_in_file.inc
--echo # t5 on expecting NOT FOUND
-- let SEARCH_FILE=$MYSQLD_DATADIR/enctests/t5.ibd
-- source include/search_pattern_in_file.inc
--echo # t6 on expecting NOT FOUND
-- let SEARCH_FILE=$MYSQLD_DATADIR/enctests/t6.ibd
-- source include/search_pattern_in_file.inc
--let SEARCH_PATTERN=public
--echo # t7 off expecting FOUND
-- let SEARCH_FILE=$MYSQLD_DATADIR/enctests/t7.ibd
-- source include/search_pattern_in_file.inc
--echo # t8 row compressed expecting NOT FOUND
-- let SEARCH_FILE=$MYSQLD_DATADIR/enctests/t8.ibd
-- source include/search_pattern_in_file.inc
--echo # t9 page compressed expecting NOT FOUND
-- let SEARCH_FILE=$MYSQLD_DATADIR/enctests/t9.ibd
-- source include/search_pattern_in_file.inc
-- source include/start_mysqld.inc
use test;
drop database enctests;
# reset system
--disable_query_log
--disable_warnings
EVAL SET GLOBAL innodb_compression_algorithm = $innodb_compression_algorithm_orig;
EVAL SET GLOBAL innodb_file_per_table = $innodb_file_per_table_orig;
EVAL SET GLOBAL innodb_file_format = $innodb_file_format_orig;
set global innodb_compression_algorithm = DEFAULT;
--enable_warnings
--enable_query_log

116
mysql-test/suite/encryption/t/innodb_onlinealter_encryption.test
View File

@ -1,26 +1,12 @@
-- source include/have_innodb.inc
-- source include/have_example_key_management_plugin.inc
-- source include/not_valgrind.inc
-- source include/have_file_key_management_plugin.inc
# test uses restart
-- source include/not_embedded.inc
-- source include/not_windows.inc
--disable_query_log
let $innodb_file_format_orig = `SELECT @@innodb_file_format`;
let $innodb_file_per_table_orig = `SELECT @@innodb_file_per_table`;
--enable_query_log
--disable_warnings
SET GLOBAL innodb_file_format = `Barracuda`;
SET GLOBAL innodb_file_per_table = ON;
--let $MYSQLD_DATADIR=`select @@datadir`
--let SEARCH_RANGE = 10000000
--let t1_IBD = $MYSQLD_DATADIR/test/t1.ibd
--let t2_IBD = $MYSQLD_DATADIR/test/t2.ibd
--let t3_IBD = $MYSQLD_DATADIR/test/t3.ibd
--let t4_IBD = $MYSQLD_DATADIR/test/t4.ibd
--let t5_IBD = $MYSQLD_DATADIR/test/t5.ibd
--let t6_IBD = $MYSQLD_DATADIR/test/t6.ibd
--let t7_IBD = $MYSQLD_DATADIR/test/t7.ibd
--enable_warnings
CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY, a VARCHAR(255)) ENGINE=InnoDB encrypted=yes;
CREATE TABLE t2 (id INT NOT NULL PRIMARY KEY, a VARCHAR(255)) ENGINE=InnoDB;
@ -50,7 +36,7 @@ delimiter ;//
commit;
set autocommit=0;
call innodb_insert_proc(15000);
call innodb_insert_proc(1500);
commit;
set autocommit=1;
@ -58,37 +44,48 @@ set autocommit=1;
--let $wait_timeout= 600
--let $wait_condition=SELECT COUNT(*) = 0 FROM INFORMATION_SCHEMA.INNODB_TABLESPACES_ENCRYPTION WHERE MIN_KEY_VERSION = 0
--source include/wait_condition.inc
--let $MYSQLD_DATADIR=`select @@datadir`
--sleep 10
--source include/shutdown_mysqld.inc
--source include/wait_until_disconnected.inc
--let SEARCH_RANGE = 10000000
--let SEARCH_PATTERN=foobar
--echo # t1 yes on expecting NOT FOUND
-- let SEARCH_FILE=$t1_IBD
-- let SEARCH_FILE=$MYSQLD_DATADIR/test/t1.ibd
-- source include/search_pattern_in_file.inc
--let SEARCH_PATTERN=temp
--echo # t2 ... on expecting NOT FOUND
-- let SEARCH_FILE=$t2_IBD
-- let SEARCH_FILE=$MYSQLD_DATADIR/test/t2.ibd
-- source include/search_pattern_in_file.inc
--let SEARCH_PATTERN=barfoo
--echo # t3 ... on expecting NOT FOUND
-- let SEARCH_FILE=$t3_IBD
-- let SEARCH_FILE=$MYSQLD_DATADIR/test/t3.ibd
-- source include/search_pattern_in_file.inc
--let SEARCH_PATTERN=repeat
--echo # t4 ... on expecting NOT FOUND
-- let SEARCH_FILE=$t4_IBD
-- let SEARCH_FILE=$MYSQLD_DATADIR/test/t4.ibd
-- source include/search_pattern_in_file.inc
--let SEARCH_PATTERN=author
--echo # t5 ... on expecting NOT FOUND
-- let SEARCH_FILE=$t5_IBD
-- let SEARCH_FILE=$MYSQLD_DATADIR/test/t5.ibd
-- source include/search_pattern_in_file.inc
--let SEARCH_PATTERN=mangled
--echo # t6 ... on expecting NOT FOUND
-- let SEARCH_FILE=$t6_IBD
-- let SEARCH_FILE=$MYSQLD_DATADIR/test/t6.ibd
-- source include/search_pattern_in_file.inc
--let SEARCH_PATTERN=mysql
--echo # t7 ... on expecting NOT FOUND
-- let SEARCH_FILE=$t7_IBD
-- let SEARCH_FILE=$MYSQLD_DATADIR/test/t7.ibd
-- source include/search_pattern_in_file.inc
-- source include/start_mysqld.inc
--disable_warnings
SET GLOBAL innodb_file_format = `Barracuda`;
SET GLOBAL innodb_file_per_table = ON;
--enable_warnings
ALTER TABLE t1 ADD COLUMN b int default 2;
ALTER TABLE t2 ADD COLUMN b int default 2;
ALTER TABLE t7 ADD COLUMN b int default 2;
@ -110,82 +107,39 @@ SHOW CREATE TABLE t5;
SHOW CREATE TABLE t6;
SHOW CREATE TABLE t7;
--sleep 10
--let SEARCH_PATTERN=foobar
--echo # t1 yes on expecting NOT FOUND
-- let SEARCH_FILE=$t1_IBD
-- source include/search_pattern_in_file.inc
--let SEARCH_PATTERN=temp
--echo # t2 ... on expecting NOT FOUND
-- let SEARCH_FILE=$t2_IBD
-- source include/search_pattern_in_file.inc
--let SEARCH_PATTERN=barfoo
--echo # t3 ... on expecting NOT FOUND
-- let SEARCH_FILE=$t3_IBD
-- source include/search_pattern_in_file.inc
--let SEARCH_PATTERN=repeat
--echo # t4 ... on expecting NOT FOUND
-- let SEARCH_FILE=$t4_IBD
-- source include/search_pattern_in_file.inc
--let SEARCH_PATTERN=author
--echo # t5 ... on expecting NOT FOUND
-- let SEARCH_FILE=$t5_IBD
-- source include/search_pattern_in_file.inc
--let SEARCH_PATTERN=mangled
--echo # t6 ... on expecting NOT FOUND
-- let SEARCH_FILE=$t6_IBD
-- source include/search_pattern_in_file.inc
--let SEARCH_PATTERN=mysql
--echo # t7 ... on expecting NOT FOUND
-- let SEARCH_FILE=$t7_IBD
-- source include/search_pattern_in_file.inc
--echo # Restarting server
-- source include/restart_mysqld.inc
--echo # Done restarting server
select count(1) from t1;
select count(1) from t2;
select count(1) from t3;
select count(1) from t4;
select count(1) from t5;
select count(1) from t6;
select count(1) from t7;
--source include/shutdown_mysqld.inc
--source include/wait_until_disconnected.inc
--let SEARCH_PATTERN=foobar
--echo # t1 yes on expecting NOT FOUND
-- let SEARCH_FILE=$t1_IBD
-- let SEARCH_FILE=$MYSQLD_DATADIR/test/t1.ibd
-- source include/search_pattern_in_file.inc
--let SEARCH_PATTERN=temp
--echo # t2 ... on expecting NOT FOUND
-- let SEARCH_FILE=$t2_IBD
-- let SEARCH_FILE=$MYSQLD_DATADIR/test/t2.ibd
-- source include/search_pattern_in_file.inc
--let SEARCH_PATTERN=barfoo
--echo # t3 ... on expecting NOT FOUND
-- let SEARCH_FILE=$t3_IBD
-- let SEARCH_FILE=$MYSQLD_DATADIR/test/t3.ibd
-- source include/search_pattern_in_file.inc
--let SEARCH_PATTERN=repeat
--echo # t4 ... on expecting NOT FOUND
-- let SEARCH_FILE=$t4_IBD
-- let SEARCH_FILE=$MYSQLD_DATADIR/test/t4.ibd
-- source include/search_pattern_in_file.inc
--let SEARCH_PATTERN=author
--echo # t5 ... on expecting NOT FOUND
-- let SEARCH_FILE=$t5_IBD
-- let SEARCH_FILE=$MYSQLD_DATADIR/test/t5.ibd
-- source include/search_pattern_in_file.inc
--let SEARCH_PATTERN=mangled
--echo # t6 ... on expecting NOT FOUND
-- let SEARCH_FILE=$t6_IBD
-- let SEARCH_FILE=$MYSQLD_DATADIR/test/t6.ibd
-- source include/search_pattern_in_file.inc
--let SEARCH_PATTERN=mysql
--echo # t7 ... on expecting NOT FOUND
-- let SEARCH_FILE=$t7_IBD
-- let SEARCH_FILE=$MYSQLD_DATADIR/test/t7.ibd
-- source include/search_pattern_in_file.inc
-- source include/start_mysqld.inc
DROP PROCEDURE innodb_insert_proc;
DROP TABLE t1, t2, t3, t4, t5, t6, t7;
# reset system
--disable_query_log
EVAL SET GLOBAL innodb_file_per_table = $innodb_file_per_table_orig;
EVAL SET GLOBAL innodb_file_format = $innodb_file_format_orig;
--enable_query_log