1
0
mirror of https://github.com/MariaDB/server.git synced 2025-07-27 18:02:13 +03:00

MDEV-13564 Mariabackup does not work with TRUNCATE

Implement undo tablespace truncation via normal redo logging.

Implement TRUNCATE TABLE as a combination of RENAME to #sql-ib name,
CREATE, and DROP.

Note: Orphan #sql-ib*.ibd may be left behind if MariaDB Server 10.2
is killed before the DROP operation is committed. If MariaDB Server 10.2
is killed during TRUNCATE, it is also possible that the old table
was renamed to #sql-ib*.ibd but the data dictionary will refer to the
table using the original name.

In MariaDB Server 10.3, RENAME inside InnoDB is transactional,
and #sql-* tables will be dropped on startup. So, this new TRUNCATE
will be fully crash-safe in 10.3.

ha_mroonga::wrapper_truncate(): Pass table options to the underlying
storage engine, now that ha_innobase::truncate() will need them.

rpl_slave_state::truncate_state_table(): Before truncating
mysql.gtid_slave_pos, evict any cached table handles from
the table definition cache, so that there will be no stale
references to the old table after truncating.

== TRUNCATE TABLE ==

WL#6501 in MySQL 5.7 introduced separate log files for implementing
atomic and crash-safe TRUNCATE TABLE, instead of using the InnoDB
undo and redo log. Some convoluted logic was added to the InnoDB
crash recovery, and some extra synchronization (including a redo log
checkpoint) was introduced to make this work. This synchronization
has caused performance problems and race conditions, and the extra
log files cannot be copied or applied by external backup programs.

In order to support crash-upgrade from MariaDB 10.2, we will keep
the logic for parsing and applying the extra log files, but we will
no longer generate those files in TRUNCATE TABLE.

A prerequisite for crash-safe TRUNCATE is a crash-safe RENAME TABLE
(with full redo and undo logging and proper rollback). This will
be implemented in MDEV-14717.

ha_innobase::truncate(): Invoke RENAME, create(), delete_table().
Because RENAME cannot be fully rolled back before MariaDB 10.3
due to missing undo logging, add some explicit rename-back in
case the operation fails.

ha_innobase::delete(): Introduce a variant that takes sqlcom as
a parameter. In TRUNCATE TABLE, we do not want to touch any
FOREIGN KEY constraints.

ha_innobase::create(): Add the parameters file_per_table, trx.
In TRUNCATE, the new table must be created in the same transaction
that renames the old table.

create_table_info_t::create_table_info_t(): Add the parameters
file_per_table, trx.

row_drop_table_for_mysql(): Replace a bool parameter with sqlcom.

row_drop_table_after_create_fail(): New function, wrapping
row_drop_table_for_mysql().

dict_truncate_index_tree_in_mem(), fil_truncate_tablespace(),
fil_prepare_for_truncate(), fil_reinit_space_header_for_table(),
row_truncate_table_for_mysql(), TruncateLogger,
row_truncate_prepare(), row_truncate_rollback(),
row_truncate_complete(), row_truncate_fts(),
row_truncate_update_system_tables(),
row_truncate_foreign_key_checks(), row_truncate_sanity_checks():
Remove.

row_upd_check_references_constraints(): Remove a check for
TRUNCATE, now that the table is no longer truncated in place.

The new test innodb.truncate_foreign uses DEBUG_SYNC to cover some
race-condition like scenarios. The test innodb-innodb.truncate does
not use any synchronization.

We add a redo log subformat to indicate backup-friendly format.
MariaDB 10.4 will remove support for the old TRUNCATE logging,
so crash-upgrade from old 10.2 or 10.3 to 10.4 will involve
limitations.

== Undo tablespace truncation ==

MySQL 5.7 implements undo tablespace truncation. It is only
possible when innodb_undo_tablespaces is set to at least 2.
The logging is implemented similar to the WL#6501 TRUNCATE,
that is, using separate log files and a redo log checkpoint.

We can simply implement undo tablespace truncation within
a single mini-transaction that reinitializes the undo log
tablespace file. Unfortunately, due to the redo log format
of some operations, currently, the total redo log written by
undo tablespace truncation will be more than the combined size
of the truncated undo tablespace. It should be acceptable
to have a little more than 1 megabyte of log in a single
mini-transaction. This will be fixed in MDEV-17138 in
MariaDB Server 10.4.

recv_sys_t: Add truncated_undo_spaces[] to remember for which undo
tablespaces a MLOG_FILE_CREATE2 record was seen.

namespace undo: Remove some unnecessary declarations.

fil_space_t::is_being_truncated: Document that this flag now
only applies to undo tablespaces. Remove some references.

fil_space_t::is_stopping(): Do not refer to is_being_truncated.
This check is for tablespaces of tables. Potentially used
tablespaces are never truncated any more.

buf_dblwr_process(): Suppress the out-of-bounds warning
for undo tablespaces.

fil_truncate_log(): Write a MLOG_FILE_CREATE2 with a nonzero
page number (new size of the tablespace in pages) to inform
crash recovery that the undo tablespace size has been reduced.

fil_op_write_log(): Relax assertions, so that MLOG_FILE_CREATE2
can be written for undo tablespaces (without .ibd file suffix)
for a nonzero page number.

os_file_truncate(): Add the parameter allow_shrink=false
so that undo tablespaces can actually be shrunk using this function.

fil_name_parse(): For undo tablespace truncation,
buffer MLOG_FILE_CREATE2 in truncated_undo_spaces[].

recv_read_in_area(): Avoid reading pages for which no redo log
records remain buffered, after recv_addr_trim() removed them.

trx_rseg_header_create(): Add a FIXME comment that we could write
much less redo log.

trx_undo_truncate_tablespace(): Reinitialize the undo tablespace
in a single mini-transaction, which will be flushed to the redo log
before the file size is trimmed.

recv_addr_trim(): Discard any redo logs for pages that were
logged after the new end of a file, before the truncation LSN.
If the rec_list becomes empty, reduce n_addrs. After removing
any affected records, actually truncate the file.

recv_apply_hashed_log_recs(): Invoke recv_addr_trim() right before
applying any log records. The undo tablespace files must be open
at this point.

buf_flush_or_remove_pages(), buf_flush_dirty_pages(),
buf_LRU_flush_or_remove_pages(): Add a parameter for specifying
the number of the first page to flush or remove (default 0).

trx_purge_initiate_truncate(): Remove the log checkpoints, the
extra logging, and some unnecessary crash points. Merge the code
from trx_undo_truncate_tablespace(). First, flush all to-be-discarded
pages (beyond the new end of the file), then trim the space->size
to make the page allocation deterministic. At the only remaining
crash injection point, flush the redo log, so that the recovery
can be tested.
This commit is contained in:
Marko Mäkelä
2018-08-28 13:43:06 +03:00
parent 59950df533
commit 055a3334ad
83 changed files with 907 additions and 6947 deletions

View File

@ -1,451 +0,0 @@
####################################################################
# TC to check truncate table statement atomicity for single #
# tablespace #
# Sceanrio covered: #
# 1. Debug points added for worklog #
# 2. Table with differnt row types #
# 3. Transactional statement. #
####################################################################
--source include/have_innodb.inc
--source include/have_debug.inc
--source include/big_test.inc
--source include/have_innodb_16k.inc
# Valgrind would result in a "long semaphore wait" inside InnoDB
--source include/not_valgrind.inc
# Embedded server does not support crashing
--source include/not_embedded.inc
# Avoid CrashReporter popup on Mac
--source include/not_crashrep.inc
#-----------------------------------------------------------------------
--disable_query_log
let $MYSQL_DATA_DIR= `select @@datadir`;
let $data_directory = data directory='$MYSQL_TMP_DIR/alt_dir';
let $innodb_file_per_table_orig=`select @@innodb_file_per_table`;
call mtr.add_suppression("InnoDB.*table did not exist in the InnoDB data dictionary.*");
call mtr.add_suppression("InnoDB: A page in the doublewrite buffer is not within space bounds.*");
call mtr.add_suppression("InnoDB: Cannot create file.*");
call mtr.add_suppression("InnoDB: Error number 17 means 'File exists'.*");
call mtr.add_suppression("InnoDB: A page in the doublewrite buffer is not within space bounds");
call mtr.add_suppression("InnoDB: Error: table .* does not exist in the InnoDB internal");
--enable_query_log
#-----------------------------------------------------------------------
set global innodb_file_per_table=on;
--echo # Verify that 'TRUNCATE TABLE' statement works fine and the size
--echo # of .ibd file is equal to the initial size after truncation.
#-----------------------------------------------------------------------
drop table if exists t1,t2,t3,t4,t6;
let $cnt = 6;
while ($cnt) {
# table with basic data type + primary ,secondary,composite,prefix index
create table t1(c1 int not null,
c2 int not null,
c3 char(255) not null,
c4 text(500) not null,
c5 blob(500) not null,
c6 varchar(500) not null,
c7 varchar(500) not null,
c8 datetime,
c9 decimal(5,3),
primary key (c1),
index (c3,c4(50),c5(50)),
index (c2))
engine=innodb row_format=redundant;
create table t2(c1 int not null,
c2 int not null,
c3 char(255) not null,
c4 text(500) not null,
c5 blob(500) not null,
c6 varchar(500) not null,
c7 varchar(500) not null,
c8 datetime,
c9 decimal(5,3),
primary key (c1),
index (c3,c4(50),c5(50)),
index (c2))
engine=innodb row_format=compact;
# with row type , key block size = 4K
create table t3(c1 int not null,
c2 int not null,
c3 char(255) not null,
c4 text(500) not null,
c5 blob(500) not null,
c6 varchar(500) not null,
c7 varchar(500) not null,
c8 datetime,
c9 decimal(5,3),
primary key (c1),
index (c3,c4(50),c5(50)),
index (c2))
engine=innodb row_format=compressed key_block_size=4;
create table t4(c1 int not null,
c2 int not null,
c3 char(255) not null,
c4 text(500) not null,
c5 blob(500) not null,
c6 varchar(500) not null,
c7 varchar(500) not null,
c8 datetime,
c9 decimal(5,3),
primary key (c1),
index (c3,c4(50),c5(50)),
index (c2))
engine=innodb row_format=dynamic;
create temporary table t5(c1 int not null,
c2 int not null,
c3 char(255) not null,
c4 text(500) not null,
c5 blob(500) not null,
c6 varchar(500) not null,
c7 varchar(500) not null,
c8 datetime,
c9 decimal(5,3),
primary key (c1),
index (c3,c4(50),c5(50)),
index (c2))
engine=innodb;
create table t6 ( a int ) engine = innodb;
insert into t6 values (50),(100),(150);
--disable_query_log
--disable_result_log
let $n=5;
# load created tables.
while ($n)
{
start transaction;
eval insert ignore into t1 values(
$n, $n,
repeat(concat(' tc3_',$n), 42),
repeat(concat(' tc4_',$n), 300),
repeat(concat(' tc5_',$n), 300),
repeat(concat(' tc6_',$n), 300),
repeat(concat(' tc7_',$n), 300),
now(), (100.55+$n));
eval insert ignore into t2 values(
$n, $n,
repeat(concat(' tc3_',$n), 42),
repeat(concat(' tc4_',$n), 300),
repeat(concat(' tc5_',$n), 300),
repeat(concat(' tc6_',$n), 300),
repeat(concat(' tc7_',$n), 300),
now(), (100.55+$n));
eval insert ignore into t3 values(
$n, $n,
repeat(concat(' tc3_',$n), 42),
repeat(concat(' tc4_',$n), 300),
repeat(concat(' tc5_',$n), 300),
repeat(concat(' tc6_',$n), 300),
repeat(concat(' tc7_',$n), 300),
now(), (100.55+$n));
eval insert ignore into t4 values(
$n, $n,
repeat(concat(' tc3_',$n), 42),
repeat(concat(' tc4_',$n), 300),
repeat(concat(' tc5_',$n), 300),
repeat(concat(' tc6_',$n), 300),
repeat(concat(' tc7_',$n), 300),
now(), (100.55+$n));
eval insert ignore into t5 values(
$n, $n,
repeat(concat(' tc3_',$n), 42),
repeat(concat(' tc4_',$n), 300),
repeat(concat(' tc5_',$n), 300),
repeat(concat(' tc6_',$n), 300),
repeat(concat(' tc7_',$n), 300),
now(), (100.55+$n));
if ($n <= 3)
{
commit;
}
if ($n > 3)
{
rollback;
}
dec $n;
}
# validate loading of the tables.
--enable_result_log
--enable_query_log
select count(*) from t1;
select count(*) from t2;
select count(*) from t3;
select count(*) from t4;
select count(*) from t5;
select count(*) from t6;
# set the debug crash point and exercise them.
if ($cnt == 6)
{
set session debug="+d,ib_trunc_crash_during_drop_index_temp_table";
--echo "---debug ib_trunc_crash_during_drop_index_temp_table point---"
}
if ($cnt == 5)
{
set session debug="+d,ib_trunc_crash_drop_reinit_done_create_to_start";
--echo "---debug ib_trunc_crash_drop_reinit_done_create_to_start---"
}
if ($cnt >= 5) {
--echo # Write file to make mysql-test-run.pl expect crash and restart
--exec echo "wait" > $MYSQLTEST_VARDIR/tmp/mysqld.1.expect
--echo # Run the crashing query
--error 2013
truncate table t5;
--source include/wait_until_disconnected.inc
--enable_reconnect
--echo # Restart the MySQL server
--exec echo "restart" > $MYSQLTEST_VARDIR/tmp/mysqld.1.expect
--source include/wait_until_connected_again.inc
--disable_reconnect
select count(*) from t1;
select count(*) from t2;
select count(*) from t3;
select count(*) from t4;
--error ER_NO_SUCH_TABLE
select count(*) from t5;
select count(*) from t6;
}
# set the debug crash point and exercise them.
if ($cnt == 6)
{
set session debug="+d,ib_trunc_crash_on_drop_of_sec_index";
--echo "---debug ib_trunc_crash_on_drop_of_sec_index point---"
}
if ($cnt == 5)
{
set session debug="+d,ib_trunc_crash_on_create_of_sec_index";
--echo "---debug ib_trunc_crash_on_create_of_sec_index---"
}
if ($cnt == 4)
{
set session debug="+d,ib_trunc_crash_before_log_removal";
--echo "---debug ib_trunc_crash_before_log_removal point---"
}
if ($cnt == 3)
{
set session debug="+d,ib_trunc_crash_after_truncate_done";
--echo "---debug ib_trunc_crash_after_truncate_done point---"
}
if ($cnt == 2)
{
set session debug="+d,ib_trunc_crash_after_truncate_done";
--echo "---debug ib_trunc_crash_after_truncate_done point---"
}
if ($cnt == 1)
{
set session debug="+d,ib_trunc_crash_after_redo_log_write_complete";
--echo "---debug ib_trunc_crash_after_redo_log_write_complete point---"
}
--echo # Write file to make mysql-test-run.pl expect crash and restart
--exec echo "wait" > $MYSQLTEST_VARDIR/tmp/mysqld.1.expect
--echo # Run the crashing query
--error 2013
truncate table t1;
--source include/wait_until_disconnected.inc
--enable_reconnect
--echo # Restart the MySQL server
--exec echo "restart" > $MYSQLTEST_VARDIR/tmp/mysqld.1.expect
--source include/wait_until_connected_again.inc
--disable_reconnect
select count(*) from t1;
select count(*) from t2;
select count(*) from t3;
select count(*) from t4;
--error ER_NO_SUCH_TABLE
select count(*) from t5;
select count(*) from t6;
if ($cnt == 6)
{
set session debug="+d,ib_trunc_crash_on_drop_of_sec_index";
--echo "---debug ib_trunc_crash_on_drop_of_sec_index point---"
}
if ($cnt == 5)
{
set session debug="+d,ib_trunc_crash_on_create_of_sec_index";
--echo "---debug ib_trunc_crash_on_create_of_sec_index---"
}
if ($cnt == 4)
{
set session debug="+d,ib_trunc_crash_before_log_removal";
--echo "---debug ib_trunc_crash_before_log_removal point---"
}
if ($cnt == 3)
{
set session debug="+d,ib_trunc_crash_after_truncate_done";
--echo "---debug ib_trunc_crash_after_truncate_done point---"
}
if ($cnt == 2)
{
set session debug="+d,ib_trunc_crash_after_truncate_done";
--echo "---debug ib_trunc_crash_after_truncate_done point---"
}
if ($cnt == 1)
{
set session debug="+d,ib_trunc_crash_after_redo_log_write_complete";
--echo "---debug ib_trunc_crash_after_redo_log_write_complete point---"
}
--echo # Write file to make mysql-test-run.pl expect crash and restart
--exec echo "wait" > $MYSQLTEST_VARDIR/tmp/mysqld.1.expect
--echo # Run the crashing query
--error 2013
truncate table t2;
--source include/wait_until_disconnected.inc
--enable_reconnect
--echo # Restart the MySQL server
--exec echo "restart" > $MYSQLTEST_VARDIR/tmp/mysqld.1.expect
--source include/wait_until_connected_again.inc
--disable_reconnect
select count(*) from t1;
select count(*) from t2;
select count(*) from t3;
select count(*) from t4;
--error ER_NO_SUCH_TABLE
select count(*) from t5;
select count(*) from t6;
if ($cnt == 6)
{
set session debug="+d,ib_trunc_crash_on_drop_of_sec_index";
--echo "---debug ib_trunc_crash_on_drop_of_sec_index point---"
}
if ($cnt == 5)
{
set session debug="+d,ib_trunc_crash_on_create_of_sec_index";
--echo "---debug ib_trunc_crash_on_create_of_sec_index---"
}
if ($cnt == 4)
{
set session debug="+d,ib_trunc_crash_before_log_removal";
--echo "---debug ib_trunc_crash_before_log_removal point---"
}
if ($cnt == 3)
{
set session debug="+d,ib_trunc_crash_after_truncate_done";
--echo "---debug ib_trunc_crash_after_truncate_done point---"
}
if ($cnt == 2)
{
set session debug="+d,ib_trunc_crash_after_truncate_done";
--echo "---debug ib_trunc_crash_after_truncate_done point---"
}
if ($cnt == 1)
{
set session debug="+d,ib_trunc_crash_after_redo_log_write_complete";
--echo "---debug ib_trunc_crash_after_redo_log_write_complete point---"
}
--echo # Write file to make mysql-test-run.pl expect crash and restart
--exec echo "wait" > $MYSQLTEST_VARDIR/tmp/mysqld.1.expect
--echo # Run the crashing query
--error 2013
truncate table t3;
--source include/wait_until_disconnected.inc
--enable_reconnect
--echo # Restart the MySQL server
--exec echo "restart" > $MYSQLTEST_VARDIR/tmp/mysqld.1.expect
--source include/wait_until_connected_again.inc
--disable_reconnect
select count(*) from t1;
select count(*) from t2;
select count(*) from t3;
select count(*) from t4;
--error ER_NO_SUCH_TABLE
select count(*) from t5;
select count(*) from t6;
if ($cnt == 6)
{
set session debug="+d,ib_trunc_crash_on_drop_of_sec_index";
--echo "---debug ib_trunc_crash_on_drop_of_sec_index point---"
}
if ($cnt == 5)
{
set session debug="+d,ib_trunc_crash_on_create_of_sec_index";
--echo "---debug ib_trunc_crash_on_create_of_sec_index---"
}
if ($cnt == 4)
{
set session debug="+d,ib_trunc_crash_before_log_removal";
--echo "---debug ib_trunc_crash_before_log_removal point---"
}
if ($cnt == 3)
{
set session debug="+d,ib_trunc_crash_after_truncate_done";
--echo "---debug ib_trunc_crash_after_truncate_done point---"
}
if ($cnt == 2)
{
set session debug="+d,ib_trunc_crash_after_truncate_done";
--echo "---debug ib_trunc_crash_after_truncate_done point---"
}
if ($cnt == 1)
{
set session debug="+d,ib_trunc_crash_after_redo_log_write_complete";
--echo "---debug ib_trunc_crash_after_redo_log_write_complete point---"
}
--echo # Write file to make mysql-test-run.pl expect crash and restart
--exec echo "wait" > $MYSQLTEST_VARDIR/tmp/mysqld.1.expect
--echo # Run the crashing query
--error 2013
truncate table t4;
--source include/wait_until_disconnected.inc
--enable_reconnect
--echo # Restart the MySQL server
--exec echo "restart" > $MYSQLTEST_VARDIR/tmp/mysqld.1.expect
--source include/wait_until_connected_again.inc
--disable_reconnect
select count(*) from t1;
select count(*) from t2;
select count(*) from t3;
select count(*) from t4;
--error ER_NO_SUCH_TABLE
select count(*) from t5;
select count(*) from t6;
drop table t1, t2, t3, t4, t6;
dec $cnt;
--disable_query_log
eval set global innodb_file_per_table=$innodb_file_per_table_orig;
--enable_query_log
}

View File

@ -1,25 +0,0 @@
#
# WL#6501: make truncate table atomic
#
# TC tries to hit crash point during truncate of
# compressed non-temp table residing in single tablespace
# with page-size=16k
--source include/have_innodb.inc
--source include/have_innodb_16k.inc
--source include/have_debug.inc
--source include/big_test.inc
# Valgrind would complain about memory leaks when we crash on purpose.
--source include/not_valgrind.inc
# Embedded server does not support crashing
--source include/not_embedded.inc
# Avoid CrashReporter popup on Mac
--source include/not_crashrep.inc
let $wl6501_file_per_table = 1;
let $wl6501_row_fmt = compressed;
let $wl6501_kbs = 16;
--source suite/innodb/include/innodb_wl6501_crash.inc

View File

@ -1,27 +0,0 @@
#
# WL#6501: make truncate table atomic
#
# TC tries to hit crash point during truncate of
# compressed non-temp table residing in single tablespace.
# with page-size=4k
--source include/have_innodb.inc
--source include/have_innodb_4k.inc
--source include/have_debug.inc
--source include/big_test.inc
# Valgrind would complain about memory leaks when we crash on purpose.
--source include/not_valgrind.inc
# Embedded server does not support crashing
--source include/not_embedded.inc
# Avoid CrashReporter popup on Mac
--source include/not_crashrep.inc
let $wl6501_file_per_table = 1;
let $wl6501_row_fmt = compressed;
let $wl6501_kbs = 4;
--source suite/innodb/include/innodb_wl6501_crash.inc
let $wl6501_temp = temporary;
--source suite/innodb/include/innodb_wl6501_crash_temp.inc

View File

@ -1,25 +0,0 @@
#
# WL#6501: make truncate table atomic
#
# TC tries to hit crash point during truncate of
# compressed non-temp table residing in single tablespace.
# with page-size=8k
--source include/have_innodb.inc
--source include/have_innodb_8k.inc
--source include/have_debug.inc
--source include/big_test.inc
# Valgrind would complain about memory leaks when we crash on purpose.
--source include/not_valgrind.inc
# Embedded server does not support crashing
--source include/not_embedded.inc
# Avoid CrashReporter popup on Mac
--source include/not_crashrep.inc
let $wl6501_file_per_table = 1;
let $wl6501_row_fmt = compressed;
let $wl6501_kbs = 8;
--source suite/innodb/include/innodb_wl6501_crash.inc

View File

@ -1,37 +0,0 @@
#
# WL#6501: make truncate table atomic
#
# load table with some significiant amount of data
# and then try truncate
--source include/have_innodb.inc
--source include/have_debug.inc
--source include/big_test.inc
--source include/have_innodb_16k.inc
# Valgrind would complain about memory leaks when we crash on purpose.
--source include/not_valgrind.inc
# Embedded server does not support crashing
--source include/not_embedded.inc
# Avoid CrashReporter popup on Mac
--source include/not_crashrep.inc
# Single-Tablespace/Non-Compressed
let $wl6501_file_per_table = 1;
let $wl6501_row_fmt = compact;
let $wl6501_kbs = 16;
--source suite/innodb_zip/include/innodb_wl6501_scale.inc
# Single-Tablespace/Compressed
let $wl6501_file_per_table = 1;
let $wl6501_row_fmt = compressed;
let $wl6501_kbs = 16;
--source suite/innodb_zip/include/innodb_wl6501_scale.inc
# System-Tablespace/Non-Compressed
let $wl6501_file_per_table = 0;
let $wl6501_row_fmt = compact;
let $wl6501_kbs = 16;
--source suite/innodb_zip/include/innodb_wl6501_scale.inc