mirror of
https://github.com/MariaDB/server.git
synced 2025-11-12 10:22:39 +03:00
055a3334ad
Implement undo tablespace truncation via normal redo logging. Implement TRUNCATE TABLE as a combination of RENAME to #sql-ib name, CREATE, and DROP. Note: Orphan #sql-ib*.ibd may be left behind if MariaDB Server 10.2 is killed before the DROP operation is committed. If MariaDB Server 10.2 is killed during TRUNCATE, it is also possible that the old table was renamed to #sql-ib*.ibd but the data dictionary will refer to the table using the original name. In MariaDB Server 10.3, RENAME inside InnoDB is transactional, and #sql-* tables will be dropped on startup. So, this new TRUNCATE will be fully crash-safe in 10.3. ha_mroonga::wrapper_truncate(): Pass table options to the underlying storage engine, now that ha_innobase::truncate() will need them. rpl_slave_state::truncate_state_table(): Before truncating mysql.gtid_slave_pos, evict any cached table handles from the table definition cache, so that there will be no stale references to the old table after truncating. == TRUNCATE TABLE == WL#6501 in MySQL 5.7 introduced separate log files for implementing atomic and crash-safe TRUNCATE TABLE, instead of using the InnoDB undo and redo log. Some convoluted logic was added to the InnoDB crash recovery, and some extra synchronization (including a redo log checkpoint) was introduced to make this work. This synchronization has caused performance problems and race conditions, and the extra log files cannot be copied or applied by external backup programs. In order to support crash-upgrade from MariaDB 10.2, we will keep the logic for parsing and applying the extra log files, but we will no longer generate those files in TRUNCATE TABLE. A prerequisite for crash-safe TRUNCATE is a crash-safe RENAME TABLE (with full redo and undo logging and proper rollback). This will be implemented in MDEV-14717. ha_innobase::truncate(): Invoke RENAME, create(), delete_table(). Because RENAME cannot be fully rolled back before MariaDB 10.3 due to missing undo logging, add some explicit rename-back in case the operation fails. ha_innobase::delete(): Introduce a variant that takes sqlcom as a parameter. In TRUNCATE TABLE, we do not want to touch any FOREIGN KEY constraints. ha_innobase::create(): Add the parameters file_per_table, trx. In TRUNCATE, the new table must be created in the same transaction that renames the old table. create_table_info_t::create_table_info_t(): Add the parameters file_per_table, trx. row_drop_table_for_mysql(): Replace a bool parameter with sqlcom. row_drop_table_after_create_fail(): New function, wrapping row_drop_table_for_mysql(). dict_truncate_index_tree_in_mem(), fil_truncate_tablespace(), fil_prepare_for_truncate(), fil_reinit_space_header_for_table(), row_truncate_table_for_mysql(), TruncateLogger, row_truncate_prepare(), row_truncate_rollback(), row_truncate_complete(), row_truncate_fts(), row_truncate_update_system_tables(), row_truncate_foreign_key_checks(), row_truncate_sanity_checks(): Remove. row_upd_check_references_constraints(): Remove a check for TRUNCATE, now that the table is no longer truncated in place. The new test innodb.truncate_foreign uses DEBUG_SYNC to cover some race-condition like scenarios. The test innodb-innodb.truncate does not use any synchronization. We add a redo log subformat to indicate backup-friendly format. MariaDB 10.4 will remove support for the old TRUNCATE logging, so crash-upgrade from old 10.2 or 10.3 to 10.4 will involve limitations. == Undo tablespace truncation == MySQL 5.7 implements undo tablespace truncation. It is only possible when innodb_undo_tablespaces is set to at least 2. The logging is implemented similar to the WL#6501 TRUNCATE, that is, using separate log files and a redo log checkpoint. We can simply implement undo tablespace truncation within a single mini-transaction that reinitializes the undo log tablespace file. Unfortunately, due to the redo log format of some operations, currently, the total redo log written by undo tablespace truncation will be more than the combined size of the truncated undo tablespace. It should be acceptable to have a little more than 1 megabyte of log in a single mini-transaction. This will be fixed in MDEV-17138 in MariaDB Server 10.4. recv_sys_t: Add truncated_undo_spaces[] to remember for which undo tablespaces a MLOG_FILE_CREATE2 record was seen. namespace undo: Remove some unnecessary declarations. fil_space_t::is_being_truncated: Document that this flag now only applies to undo tablespaces. Remove some references. fil_space_t::is_stopping(): Do not refer to is_being_truncated. This check is for tablespaces of tables. Potentially used tablespaces are never truncated any more. buf_dblwr_process(): Suppress the out-of-bounds warning for undo tablespaces. fil_truncate_log(): Write a MLOG_FILE_CREATE2 with a nonzero page number (new size of the tablespace in pages) to inform crash recovery that the undo tablespace size has been reduced. fil_op_write_log(): Relax assertions, so that MLOG_FILE_CREATE2 can be written for undo tablespaces (without .ibd file suffix) for a nonzero page number. os_file_truncate(): Add the parameter allow_shrink=false so that undo tablespaces can actually be shrunk using this function. fil_name_parse(): For undo tablespace truncation, buffer MLOG_FILE_CREATE2 in truncated_undo_spaces[]. recv_read_in_area(): Avoid reading pages for which no redo log records remain buffered, after recv_addr_trim() removed them. trx_rseg_header_create(): Add a FIXME comment that we could write much less redo log. trx_undo_truncate_tablespace(): Reinitialize the undo tablespace in a single mini-transaction, which will be flushed to the redo log before the file size is trimmed. recv_addr_trim(): Discard any redo logs for pages that were logged after the new end of a file, before the truncation LSN. If the rec_list becomes empty, reduce n_addrs. After removing any affected records, actually truncate the file. recv_apply_hashed_log_recs(): Invoke recv_addr_trim() right before applying any log records. The undo tablespace files must be open at this point. buf_flush_or_remove_pages(), buf_flush_dirty_pages(), buf_LRU_flush_or_remove_pages(): Add a parameter for specifying the number of the first page to flush or remove (default 0). trx_purge_initiate_truncate(): Remove the log checkpoints, the extra logging, and some unnecessary crash points. Merge the code from trx_undo_truncate_tablespace(). First, flush all to-be-discarded pages (beyond the new end of the file), then trim the space->size to make the page allocation deterministic. At the only remaining crash injection point, flush the redo log, so that the recovery can be tested.
265 lines
7.1 KiB
Plaintext
265 lines
7.1 KiB
Plaintext
--source include/have_innodb.inc
|
|
--source include/have_debug_sync.inc
|
|
--source include/have_debug.inc
|
|
--source include/have_partition.inc
|
|
|
|
set default_storage_engine=innodb;
|
|
set @old_dbug=@@global.debug_dbug;
|
|
# Ensure that the history list length will actually be decremented by purge.
|
|
SET @saved_frequency = @@GLOBAL.innodb_purge_rseg_truncate_frequency;
|
|
SET GLOBAL innodb_purge_rseg_truncate_frequency = 1;
|
|
|
|
CREATE TABLE `t` (
|
|
`a` BLOB,
|
|
`b` BLOB,
|
|
`c` BLOB GENERATED ALWAYS AS (CONCAT(a,b)) VIRTUAL,
|
|
`h` VARCHAR(10) DEFAULT NULL,
|
|
`i` int
|
|
) ENGINE=InnoDB;
|
|
|
|
INSERT INTO t VALUES (REPEAT('g', 16000), REPEAT('x', 16000), DEFAULT, "kk", 1);
|
|
INSERT INTO t VALUES (REPEAT('a', 16000), REPEAT('b', 16000), DEFAULT, "mm", 2);
|
|
|
|
CREATE INDEX idx ON t(c(100));
|
|
|
|
SET global debug_dbug="+d,ib_purge_virtual_index_callback";
|
|
UPDATE t SET a = REPEAT('m', 16000) WHERE a like "aaa%";
|
|
--source ../../innodb/include/wait_all_purged.inc
|
|
SET global debug_dbug=@old_dbug;
|
|
DROP TABLE t;
|
|
|
|
CREATE TABLE t (
|
|
a TINYBLOB,
|
|
b TINYBLOB,
|
|
c TINYBLOB GENERATED ALWAYS AS (CONCAT(a,b)) VIRTUAL,
|
|
h VARCHAR(10) DEFAULT NULL,
|
|
i INT
|
|
) ROW_FORMAT=COMPACT ENGINE=InnoDB;
|
|
|
|
INSERT INTO t VALUES (REPEAT('g', 100), REPEAT('x', 100), DEFAULT, "kk", 1);
|
|
INSERT INTO t VALUES (REPEAT('a', 100), REPEAT('b', 100), DEFAULT, "mm", 2);
|
|
|
|
CREATE INDEX idx ON t(c(100));
|
|
|
|
SET global debug_dbug="+d,ib_purge_virtual_index_callback";
|
|
UPDATE t SET a = REPEAT('m', 100) WHERE a like "aaa%";
|
|
--source ../../innodb/include/wait_all_purged.inc
|
|
SET global debug_dbug=@old_dbug;
|
|
DROP TABLE t;
|
|
|
|
|
|
CREATE TABLE t1 (
|
|
id INT NOT NULL,
|
|
store_id INT NOT NULL,
|
|
x INT GENERATED ALWAYS AS (id + store_id)
|
|
)
|
|
PARTITION BY RANGE (store_id) (
|
|
PARTITION p0 VALUES LESS THAN (6),
|
|
PARTITION p1 VALUES LESS THAN (11),
|
|
PARTITION p2 VALUES LESS THAN (16),
|
|
PARTITION p3 VALUES LESS THAN (21)
|
|
);
|
|
|
|
insert into t1 values(1, 2, default);
|
|
insert into t1 values(3, 4, default);
|
|
|
|
insert into t1 values(3, 12, default);
|
|
insert into t1 values(4, 18, default);
|
|
|
|
CREATE INDEX idx ON t1(x);
|
|
|
|
SET global debug_dbug="+d,ib_purge_virtual_index_callback";
|
|
UPDATE t1 SET id = 10 WHERE id = 1;
|
|
--source ../../innodb/include/wait_all_purged.inc
|
|
SET global debug_dbug=@old_dbug;
|
|
DROP TABLE t1;
|
|
|
|
#
|
|
# BUG#22082762 RE-ENABLE SUPPORT FOR ADDING VIRTUAL INDEX WHILE DROPPING VIRTUAL COLUMN
|
|
#
|
|
--source include/count_sessions.inc
|
|
|
|
connect (con1,localhost,root,,);
|
|
connection default;
|
|
|
|
# Test adding virtual index on newly added virtual column
|
|
CREATE TABLE t1 (a INT, b INT);
|
|
|
|
INSERT INTO t1(a, b) VALUES (1, 1), (2, 2), (3, 3);
|
|
|
|
connection con1;
|
|
--echo # disable purge
|
|
CREATE TABLE t0 (a INT) ENGINE=InnoDB;
|
|
BEGIN; SELECT * FROM t0;
|
|
|
|
connection default;
|
|
DELETE FROM t1 WHERE a = 1;
|
|
|
|
UPDATE t1 SET a = 4, b = 4 WHERE a = 3;
|
|
|
|
INSERT INTO t1(a, b) VALUES (5, 5);
|
|
|
|
SET DEBUG_SYNC= 'inplace_after_index_build SIGNAL uncommitted WAIT_FOR purged';
|
|
--error ER_ALTER_OPERATION_NOT_SUPPORTED_REASON
|
|
ALTER TABLE t1 ADD COLUMN c INT GENERATED ALWAYS AS(a+b), ADD INDEX idx (c), ALGORITHM=INPLACE, LOCK=NONE;
|
|
send ALTER TABLE t1 ADD COLUMN c INT GENERATED ALWAYS AS(a+b), ADD INDEX idx (c), ALGORITHM=INPLACE, LOCK=SHARED;
|
|
|
|
connection con1;
|
|
SET DEBUG_SYNC= 'now WAIT_FOR uncommitted';
|
|
|
|
--echo # enable purge
|
|
COMMIT;
|
|
|
|
--echo # wait for purge to process the deleted records.
|
|
--source ../../innodb/include/wait_all_purged.inc
|
|
|
|
SET DEBUG_SYNC= 'now SIGNAL purged';
|
|
|
|
connection default;
|
|
--echo /* connection default */ ALTER TABLE t1 ADD COLUMN c INT GENERATED ALWAYS AS(a+b), ADD INDEX idx (c), ALGORITHM=INPLACE, LOCK=SHARED;
|
|
--reap
|
|
SHOW CREATE TABLE t1;
|
|
|
|
SELECT * FROM t1;
|
|
|
|
DROP TABLE t1;
|
|
|
|
# Test adding index on existing virtual column
|
|
CREATE TABLE t1 (a INT, b INT, c INT GENERATED ALWAYS AS(a+b));
|
|
|
|
INSERT INTO t1(a, b) VALUES (1, 1), (2, 2), (3, 3), (4, 4);
|
|
|
|
connection con1;
|
|
--echo # disable purge
|
|
BEGIN; SELECT * FROM t0;
|
|
|
|
connection default;
|
|
DELETE FROM t1 WHERE a = 1;
|
|
|
|
UPDATE t1 SET a = 2, b = 2 WHERE a = 5;
|
|
|
|
INSERT INTO t1(a, b) VALUES (6, 6);
|
|
|
|
SET DEBUG_SYNC= 'inplace_after_index_build SIGNAL uncommitted WAIT_FOR purged';
|
|
send ALTER TABLE t1 ADD INDEX idx (c), ALGORITHM=INPLACE, LOCK=NONE;
|
|
|
|
connection con1;
|
|
SET DEBUG_SYNC= 'now WAIT_FOR uncommitted';
|
|
|
|
DELETE FROM t1 WHERE a = 3;
|
|
|
|
UPDATE t1 SET a = 7, b = 7 WHERE a = 4;
|
|
|
|
INSERT INTO t1(a, b) VALUES (8, 8);
|
|
|
|
--echo # enable purge
|
|
COMMIT;
|
|
|
|
--echo # wait for purge to process the deleted/updated records.
|
|
let $wait_all_purged=1;
|
|
--source ../../innodb/include/wait_all_purged.inc
|
|
let $wait_all_purged=0;
|
|
|
|
SET DEBUG_SYNC= 'now SIGNAL purged';
|
|
|
|
disconnect con1;
|
|
|
|
connection default;
|
|
--echo /* connection default */ ALTER TABLE t1 ADD INDEX idx (c), ALGORITHM=INPLACE, LOCK=NONE;
|
|
--reap
|
|
SHOW CREATE TABLE t1;
|
|
|
|
SELECT * FROM t1;
|
|
|
|
DROP TABLE t0, t1;
|
|
|
|
#
|
|
# test MDLs with purge
|
|
#
|
|
create table t (a blob, b blob, c blob as (concat(a,b)), h varchar(10), index (c(100)));
|
|
insert t(a,b,h) values (repeat('g', 16000), repeat('x', 16000), "kk");
|
|
insert t(a,b,h) values (repeat('a', 16000), repeat('b', 16000), "mm");
|
|
set global debug_dbug="+d,ib_purge_virtual_index_callback";
|
|
connect(prevent_purge, localhost, root);
|
|
start transaction with consistent snapshot;
|
|
connection default;
|
|
update t set a = repeat('m', 16000) where a like "aaa%";
|
|
connect(lock_table, localhost, root);
|
|
lock table t write;
|
|
connection prevent_purge;
|
|
commit;
|
|
connection default;
|
|
--source ../../innodb/include/wait_all_purged.inc
|
|
disconnect lock_table;
|
|
start transaction with consistent snapshot;
|
|
commit;
|
|
--source ../../innodb/include/wait_all_purged.inc
|
|
set global debug_dbug=@old_dbug;
|
|
drop table t;
|
|
|
|
--echo #
|
|
--echo # MDEV-15165 InnoDB purge for index on virtual column
|
|
--echo # is trying to access an incomplete record
|
|
--echo #
|
|
CREATE TABLE t1(
|
|
u INT PRIMARY KEY, b BLOB, ug INT GENERATED ALWAYS AS (u) VIRTUAL,
|
|
INDEX bug(b(100),ug)
|
|
) ENGINE=InnoDB;
|
|
INSERT INTO t1 (u,b) VALUES(1,REPEAT('a',16384));
|
|
connection prevent_purge;
|
|
start transaction with consistent snapshot;
|
|
connection default;
|
|
DELETE FROM t1;
|
|
SET DEBUG_SYNC='blob_write_middle SIGNAL halfway WAIT_FOR purged';
|
|
send INSERT INTO t1 (u,b) VALUES(1,REPEAT('a',16384));
|
|
connection prevent_purge;
|
|
SET DEBUG_SYNC='now WAIT_FOR halfway';
|
|
COMMIT;
|
|
--source ../../innodb/include/wait_all_purged.inc
|
|
SET DEBUG_SYNC='now SIGNAL purged';
|
|
|
|
connection default;
|
|
reap;
|
|
DROP TABLE t1;
|
|
|
|
CREATE TABLE t1 (y YEAR, vy YEAR AS (y) VIRTUAL UNIQUE, pk INT PRIMARY KEY)
|
|
ENGINE=InnoDB;
|
|
|
|
INSERT INTO t1 (pk,y) VALUES (1,2022);
|
|
CREATE TABLE t2(f1 INT NOT NULL, PRIMARY KEY(f1))ENGINE=InnoDB;
|
|
|
|
SET GLOBAL debug_dbug = '+d,ib_purge_virtual_index_callback';
|
|
|
|
BEGIN;
|
|
INSERT INTO t2(f1) VALUES(1);
|
|
connection prevent_purge;
|
|
SET DEBUG_SYNC=RESET;
|
|
start transaction with consistent snapshot;
|
|
connection default;
|
|
COMMIT;
|
|
|
|
connect(truncate,localhost,root,,);
|
|
REPLACE INTO t1(pk, y) SELECT pk,y FROM t1;
|
|
send TRUNCATE TABLE t1;
|
|
|
|
connection prevent_purge;
|
|
COMMIT;
|
|
SET DEBUG_SYNC='now SIGNAL purge_start';
|
|
disconnect prevent_purge;
|
|
|
|
connection default;
|
|
SET DEBUG_SYNC='now WAIT_FOR purge_start';
|
|
--source ../../innodb/include/wait_all_purged.inc
|
|
SET GLOBAL debug_dbug=@old_dbug;
|
|
|
|
connection truncate;
|
|
reap;
|
|
disconnect truncate;
|
|
|
|
connection default;
|
|
DROP TABLE t1, t2;
|
|
|
|
--source include/wait_until_count_sessions.inc
|
|
set debug_sync=reset;
|
|
SET GLOBAL innodb_purge_rseg_truncate_frequency = @saved_frequency;
|