database/mariadb
1
0
Fork 0
mirror of https://github.com/MariaDB/server.git synced 2025-11-10 23:02:54 +03:00
Code Activity
Files
3a82c256daf3a075bdaba103445652f0116b8a8b
mariadb/mysql-test/suite/innodb/t/log_corruption.test
Marko Mäkelä 685d958e38 MDEV-14425 Improve the redo log for concurrency 
The InnoDB redo log used to be formatted in blocks of 512 bytes.
The log blocks were encrypted and the checksum was calculated while
holding log_sys.mutex, creating a serious scalability bottleneck.

We remove the fixed-size redo log block structure altogether and
essentially turn every mini-transaction into a log block of its own.
This allows encryption and checksum calculations to be performed
on local mtr_t::m_log buffers, before acquiring log_sys.mutex.
The mutex only protects a memcpy() of the data to the shared
log_sys.buf, as well as the padding of the log, in case the
to-be-written part of the log would not end in a block boundary of
the underlying storage. For now, the "padding" consists of writing
a single NUL byte, to allow recovery and mariadb-backup to detect
the end of the circular log faster.

Like the previous implementation, we will overwrite the last log block
over and over again, until it has been completely filled. It would be
possible to write only up to the last completed block (if no more
recent write was requested), or to write dummy FILE_CHECKPOINT records
to fill the incomplete block, by invoking the currently disabled
function log_pad(). This would require adjustments to some logic around
log checkpoints, page flushing, and shutdown.

An upgrade after a crash of any previous version is not supported.
Logically empty log files from a previous version will be upgraded.

An attempt to start up InnoDB without a valid ib_logfile0 will be
refused. Previously, the redo log used to be created automatically
if it was missing. Only with with innodb_force_recovery=6, it is
possible to start InnoDB in read-only mode even if the log file
does not exist. This allows the contents of a possibly corrupted
database to be dumped.

Because a prepared backup from an earlier version of mariadb-backup
will create a 0-sized log file, we will allow an upgrade from such
log files, provided that the FIL_PAGE_FILE_FLUSH_LSN in the system
tablespace looks valid.

The 512-byte log checkpoint blocks at 0x200 and 0x600 will be replaced
with 64-byte log checkpoint blocks at 0x1000 and 0x2000.

The start of log records will move from 0x800 to 0x3000. This allows us
to use 4096-byte aligned blocks for all I/O in a future revision.

We extend the MDEV-12353 redo log record format as follows.

(1) Empty mini-transactions or extra NUL bytes will not be allowed.
(2) The end-of-minitransaction marker (a NUL byte) will be replaced
with a 1-bit sequence number, which will be toggled each time when the
circular log file wraps back to the beginning.
(3) After the sequence bit, a CRC-32C checksum of all data
(excluding the sequence bit) will written.
(4) If the log is encrypted, 8 bytes will be written before
the checksum and included in it. This is part of the
initialization vector (IV) of encrypted log data.
(5) File names, page numbers, and checkpoint information will not be
encrypted. Only the payload bytes of page-level log will be encrypted.
The tablespace ID and page number will form part of the IV.
(6) For padding, arbitrary-length FILE_CHECKPOINT records may be written,
with all-zero payload, and with the normal end marker and checksum.
The minimum size is 7 bytes, or 7+8 with innodb_encrypt_log=ON.

In mariadb-backup and in Galera snapshot transfer (SST) scripts, we will
no longer remove ib_logfile0 or create an empty ib_logfile0. Server startup
will require a valid log file. When resizing the log, we will create
a logically empty ib_logfile101 at the current LSN and use an atomic rename
to replace ib_logfile0 with it. See the test innodb.log_file_size.

Because there is no mandatory padding in the log file, we are able
to create a dummy log file as of an arbitrary log sequence number.
See the test mariabackup.huge_lsn.

The parameter innodb_log_write_ahead_size and the
INFORMATION_SCHEMA.INNODB_METRICS counter log_padded will be removed.

The minimum value of innodb_log_buffer_size will be increased to 2MiB
(because log_sys.buf will replace recv_sys.buf) and the increment
adjusted to 4096 bytes (the maximum log block size).

The following INFORMATION_SCHEMA.INNODB_METRICS counters will be removed:

os_log_fsyncs
os_log_pending_fsyncs
log_pending_log_flushes
log_pending_checkpoint_writes

The following status variables will be removed:

Innodb_os_log_fsyncs (this is included in Innodb_data_fsyncs)
Innodb_os_log_pending_fsyncs (this was limited to at most 1 by design)

log_sys.get_block_size(): Return the physical block size of the log file.
This is only implemented on Linux and Microsoft Windows for now, and for
the power-of-2 block sizes between 64 and 4096 bytes (the minimum and
maximum size of a checkpoint block). If the block size is anything else,
the traditional 512-byte size will be used via normal file system
buffering.

If the file system buffers can be bypassed, a message like the following
will be issued:

InnoDB: File system buffers for log disabled (block size=512 bytes)
InnoDB: File system buffers for log disabled (block size=4096 bytes)

This has been tested on Linux and Microsoft Windows with both sizes.

On Linux, only enable O_DIRECT on the log for innodb_flush_method=O_DSYNC.
Tests in 3 different environments where the log is stored in a device
with a physical block size of 512 bytes are yielding better throughput
without O_DIRECT. This could be due to the fact that in the event the
last log block is being overwritten (if multiple transactions would
become durable at the same time, and each of will write a small
number of bytes to the last log block), it should be faster to re-copy
data from log_sys.buf or log_sys.flush_buf to the kernel buffer,
to be finally written at fdatasync() time.

The parameter innodb_flush_method=O_DSYNC will imply O_DIRECT for
data files. This option will enable O_DIRECT on the log file on Linux.
It may be unsafe to use when the storage device does not support
FUA (Force Unit Access) mode.

When the server is compiled WITH_PMEM=ON, we will use memory-mapped
I/O for the log file if the log resides on a "mount -o dax" device.
We will identify PMEM in a start-up message:

InnoDB: log sequence number 0 (memory-mapped); transaction id 3

On Linux, we will also invoke mmap() on any ib_logfile0 that resides
in /dev/shm, effectively treating the log file as persistent memory.
This should speed up "./mtr --mem" and increase the test coverage of
PMEM on non-PMEM hardware. It also allows users to estimate how much
the performance would be improved by installing persistent memory.
On other tmpfs file systems such as /run, we will not use mmap().

mariadb-backup: Eliminated several variables. We will refer
directly to recv_sys and log_sys.

backup_wait_for_lsn(): Detect non-progress of
xtrabackup_copy_logfile(). In this new log format with
arbitrary-sized blocks, we can only detect log file overrun
indirectly, by observing that the scanned log sequence number
is not advancing.

xtrabackup_copy_logfile(): On PMEM, do not modify the sequence bit,
because we are not allowed to modify the server's log file, and our
memory mapping is read-only.

trx_flush_log_if_needed_low(): Do not use the callback on pmem.
Using neither flush_lock nor write_lock around PMEM writes seems
to yield the best performance. The pmem_persist() calls may
still be somewhat slower than the pwrite() and fdatasync() based
interface (PMEM mounted without -o dax).

recv_sys_t::buf: Remove. We will use log_sys.buf for parsing.

recv_sys_t::MTR_SIZE_MAX: Replaces RECV_SCAN_SIZE.

recv_sys_t::file_checkpoint: Renamed from mlog_checkpoint_lsn.

recv_sys_t, log_sys_t: Removed many data members.

recv_sys.lsn: Renamed from recv_sys.recovered_lsn.
recv_sys.offset: Renamed from recv_sys.recovered_offset.
log_sys.buf_size: Replaces srv_log_buffer_size.

recv_buf: A smart pointer that wraps log_sys.buf[recv_sys.offset]
when the buffer is being allocated from the memory heap.

recv_ring: A smart pointer that wraps a circular log_sys.buf[] that is
backed by ib_logfile0. The pointer will wrap from recv_sys.len
(log_sys.file_size) to log_sys.START_OFFSET. For the record that
wraps around, we may copy file name or record payload data to
the auxiliary buffer decrypt_buf in order to have a contiguous
block of memory. The maximum size of a record is less than
innodb_page_size bytes.

recv_sys_t::parse(): Take the smart pointer as a template parameter.
Do not temporarily add a trailing NUL byte to FILE_ records, because
we are not supposed to modify the memory-mapped log file. (It is
attached in read-write mode already during recovery.)

recv_sys_t::parse_mtr(): Wrapper for recv_sys_t::parse().

recv_sys_t::parse_pmem(): Like parse_mtr(), but if PREMATURE_EOF would be
returned on PMEM, use recv_ring to wrap around the buffer to the start.

mtr_t::finish_write(), log_close(): Do not enforce log_sys.max_buf_free
on PMEM, because it has no meaning on the mmap-based log.

log_sys.write_to_buf: Count writes to log_sys.buf. Replaces
srv_stats.log_write_requests and export_vars.innodb_log_write_requests.
Protected by log_sys.mutex. Updated consistently in log_close().
Previously, mtr_t::commit() conditionally updated the count,
which was inconsistent.

log_sys.write_to_log: Count swaps of log_sys.buf and log_sys.flush_buf,
for writing to log_sys.log (the ib_logfile0). Replaces
srv_stats.log_writes and export_vars.innodb_log_writes.
Protected by log_sys.mutex.

log_sys.waits: Count waits in append_prepare(). Replaces
srv_stats.log_waits and export_vars.innodb_log_waits.

recv_recover_page(): Do not unnecessarily acquire
log_sys.flush_order_mutex. We are inserting the blocks in arbitary
order anyway, to be adjusted in recv_sys.apply(true).

We will change the definition of flush_lock and write_lock to
avoid potential false sharing. Depending on sizeof(log_sys) and
CPU_LEVEL1_DCACHE_LINESIZE, the flush_lock and write_lock could
share a cache line with each other or with the last data members
of log_sys.

Thanks to Matthias Leich for providing https://rr-project.org traces
for various failures during the development, and to
Thirunarayanan Balathandayuthapani for his help in debugging
some of the recovery code. And thanks to the developers of the
rr debugger for a tool without which extensive changes to InnoDB
would be very challenging to get right.

Thanks to Vladislav Vaintroub for useful feedback and
to him, Axel Schwenke and Krunal Bauskar for testing the performance.
2022-01-21 16:03:47 +02:00

635 lines
25 KiB
Plaintext
Raw Blame History

--source include/have_innodb.inc
--source include/have_innodb_16k.inc
--disable_query_log
call mtr.add_suppression("InnoDB: Upgrade after a crash is not supported");
call mtr.add_suppression("InnoDB: Plugin initialization aborted");
call mtr.add_suppression("Plugin 'InnoDB' init function returned error");
call mtr.add_suppression("Plugin 'InnoDB' registration as a STORAGE ENGINE failed");
call mtr.add_suppression("InnoDB: Unsupported redo log format");
call mtr.add_suppression("InnoDB: No valid checkpoint was found");
call mtr.add_suppression("InnoDB: Invalid log header checksum");
call mtr.add_suppression("InnoDB: Missing MLOG_CHECKPOINT");
call mtr.add_suppression("InnoDB: MLOG_FILE_NAME incorrect");
call mtr.add_suppression("InnoDB: ############### CORRUPT LOG RECORD FOUND");
call mtr.add_suppression("InnoDB: Log scan aborted at LSN");
call mtr.add_suppression("InnoDB: Missing MLOG_FILE_NAME or MLOG_FILE_DELETE before MLOG_CHECKPOINT for tablespace 42\\r?$");
call mtr.add_suppression("InnoDB: Obtaining redo log encryption key version 1 failed");
call mtr.add_suppression("InnoDB: Decrypting checkpoint failed");
call mtr.add_suppression("InnoDB: Log file .*ib_logfile1 is of different size 2097152 bytes than other log files 1048576 bytes!");
--enable_query_log
let bugdir= $MYSQLTEST_VARDIR/tmp/log_corruption;
--mkdir $bugdir
--let SEARCH_FILE = $MYSQLTEST_VARDIR/log/mysqld.1.err
let $check_no_innodb=SELECT * FROM INFORMATION_SCHEMA.ENGINES
WHERE engine = 'innodb'
AND support IN ('YES', 'DEFAULT', 'ENABLED');
write_file $bugdir/ib_buffer_pool;
EOF
--let $dirs= --innodb-data-home-dir=$bugdir --innodb-log-group-home-dir=$bugdir
perl;
do "$ENV{MTR_SUITE_DIR}/../innodb/include/crc32.pl";
# Create a dummy system tablespace file using the default innodb_page_size=16k
die unless open OUT, ">", "$ENV{bugdir}/ibdata1";
binmode OUT;
# We calculate innodb_checksum_algorithm=crc32 for the pages.
# The following bytes are excluded:
# bytes 0..3 (the checksum is stored there)
# bytes 26..37 (encryption key version, post-encryption checksum, tablespace id)
# bytes $page_size-8..$page_size-1 (checksum, LSB of FIL_PAGE_LSN)
my $polynomial = 0x82f63b78; # CRC-32C
# Tablespace header page with valid FSP_SIZE=768 pages.
# Also, write a dummy FSEG_MAGIC_N at offset 60 to keep fseg_inode_try_get()
# happy when fseg_n_reserved_pages() is following an invalid pointer
# from the all-zero change buffer header page (page 3).
## FIL_PAGE_OFFSET
my $head = pack("Nx[18]", 0);
## FSP_PAGE_SIZE, # FSEG_MAGIC_N
my $body = pack("x[8]Nx[10]Nx[16312]", 768, 97937874);
my $ck = mycrc32($head, 0, $polynomial) ^ mycrc32($body, 0, $polynomial);
print OUT pack("N",$ck).$head.pack("x[12]").$body.pack("Nx[4]",$ck);
# Dummy pages 1..6.
$body = pack("x[16338]");
for (my($page) = 1; $page < 7; $page++)
{
## FIL_PAGE_OFFSET
$head = pack("Nx[18]", $page);
$ck = mycrc32($head, 0, $polynomial) ^ mycrc32($body, 0, $polynomial);
print OUT pack("N",$ck).$head.pack("x[16350]Nx[4]",$ck);
}
# Dictionary header page (page 7).
## FIL_PAGE_OFFSET
$head = pack("Nx[18]", 7);
## DICT_HDR_TABLES,DICT_HDR_INDEXES
$body = pack("x[32]Nx[8]Nx[16290]", 8, 9);
$ck = mycrc32($head, 0, $polynomial) ^ mycrc32($body, 0, $polynomial);
print OUT pack("N",$ck).$head.pack("x[12]").$body.pack("Nx[4]",$ck);
# Empty SYS_TABLES page (page 8).
## FIL_PAGE_OFFSET, FIL_PAGE_PREV, FIL_PAGE_NEXT, FIL_PAGE_TYPE
$head = pack("NNNx[8]n", 8, ~0, ~0, 17855);
## PAGE_N_DIR_SLOTS, PAGE_HEAP_TOP, PAGE_INDEX_ID == DICT_TABLES_ID
$body = pack("nnx[31]Cx[20]", 2, 124, 1);
$body .= pack("nxnn", 0x801, 3, 116) . "infimum";
$body .= pack("xnxnxx", 0x901, 0x803) . "supremum";
$body .= pack("x[16248]nn", 116, 101);
$ck = mycrc32($head, 0, $polynomial) ^ mycrc32($body, 0, $polynomial);
print OUT pack("N",$ck).$head.pack("x[12]").$body.pack("Nx[4]",$ck);
# Empty SYS_INDEXES page (page 9).
## FIL_PAGE_OFFSET, FIL_PAGE_PREV, FIL_PAGE_NEXT, FIL_PAGE_TYPE
$head = pack("NNNx[8]n", 9, ~0, ~0, 17855);
## PAGE_N_DIR_SLOTS, PAGE_HEAP_TOP, PAGE_INDEX_ID == DICT_INDEXES_ID
$body = pack("nnx[31]Cx[20]", 2, 124, 3);
$body .= pack("nxnn", 0x801, 3, 116) . "infimum";
$body .= pack("xnxnxx", 0x901, 0x803) . "supremum";
$body .= pack("x[16248]nn", 116, 101);
$ck = mycrc32($head, 0, $polynomial) ^ mycrc32($body, 0, $polynomial);
print OUT pack("N",$ck).$head.pack("x[12]").$body.pack("Nx[4]",$ck);
print OUT chr(0) x (759 * 16384);
close OUT or die;
# Create a redo log from before MariaDB 10.2.2/MySQL 5.7.9
die unless open OUT, ">", "$ENV{bugdir}/ib_logfile0";
binmode OUT;
print OUT pack("x[9]nx[5]", 0x1286), " ", chr(0) x 492;
print OUT pack("x[13]nCNnxxNNx[256]", 0x1286, 12, 0x80c, 0xf0, ~0, ~0);
print OUT pack("H*x[1240]", "590DBAACFE922582");
print OUT pack("H*x[5]", "800009440022000c0000000138");
print OUT pack("H*x[475]H*", "12860cb7809781e80006626f677573", "089C0ADA");
print OUT chr(0) x 2094592;
close OUT or die;
EOF
--echo # redo log from before MariaDB 10.2.2/MySQL 5.7.9
--let $restart_parameters= $dirs
--source include/restart_mysqld.inc
eval $check_no_innodb;
--source include/shutdown_mysqld.inc
let SEARCH_PATTERN=InnoDB: Upgrade after a crash is not supported. This redo log was created before MariaDB 10\\.2\\.2\\.;
--source include/search_pattern_in_file.inc
--echo # redo log from before MariaDB 10.2.2, with corrupted log checkpoint
--move_file $bugdir/ib_logfile0 $bugdir/ib_logfile
perl;
# Create a dummy second redo log file
die unless open OUT, ">", "$ENV{bugdir}/ib_logfile0";
binmode OUT;
print OUT chr(0) x 1048576;
close OUT or die;
EOF
--source include/start_mysqld.inc
eval $check_no_innodb;
--source include/shutdown_mysqld.inc
let SEARCH_PATTERN=InnoDB: Upgrade after a crash is not supported. This redo log was created before MariaDB 10\\.2\\.2, and we did not find a valid checkpoint;
--source include/search_pattern_in_file.inc
--echo # redo log from before MariaDB 10.2.2, with corrupted log block
--remove_file $bugdir/ib_logfile0
--move_file $bugdir/ib_logfile $bugdir/ib_logfile0
# Create a dummy second redo log file that does not contain payload.
--copy_file $bugdir/ib_logfile0 $bugdir/ib_logfile1
perl;
die unless open OUT, "+<", "$ENV{bugdir}/ib_logfile0";
binmode OUT;
die unless seek(OUT, 0x800, 0);
print OUT pack("Nx[508]", 0xc0deba5e);
close OUT or die;
EOF
--source include/start_mysqld.inc
eval $check_no_innodb;
--source include/shutdown_mysqld.inc
--source include/search_pattern_in_file.inc
--echo # empty redo log from before MariaDB 10.2.2
perl;
die unless open OUT, "+<", "$ENV{bugdir}/ib_logfile0";
binmode OUT;
die unless seek(OUT, 0x800, 0);
print OUT pack("NnnNx[496]N", 0x80000944, 12, 12, 0, 0xb2a);
close OUT or die;
EOF
--let $restart_parameters= $dirs --innodb-force-recovery=5 --innodb-log-file-size=2m
--source include/start_mysqld.inc
SELECT COUNT(*) FROM INFORMATION_SCHEMA.ENGINES
WHERE engine = 'innodb'
AND support IN ('YES', 'DEFAULT', 'ENABLED');
--source include/shutdown_mysqld.inc
--let SEARCH_PATTERN= InnoDB: Upgrading redo log:
--source include/search_pattern_in_file.inc
--let $restart_parameters= $dirs
--echo # Corrupted multi-file redo log from before MariaDB 10.2.2
--move_file $bugdir/ib_logfile0 $bugdir/ib_logfile
perl;
die unless open OUT, ">", "$ENV{bugdir}/ib_logfile0";
binmode OUT;
print OUT pack("x[9]nx[5]", 0x1286), " ", chr(0) x 492;
print OUT pack("x[13]nCNnxxNNx[256]", 0x1286, 12, 0x20000c, 0xf0, ~0, ~0);
print OUT pack("H*x[1240]", "B41DCAC4525D1217");
print OUT pack("H*x[5]", "800009440022000c0000000138");
print OUT pack("H*x[475]H*", "12860cb7809781e80006626f677573", "089C0ADA");
print OUT chr(0) x 1046016;
close OUT or die;
die unless open OUT, ">", "$ENV{bugdir}/ib_logfile1";
binmode OUT;
die unless seek(OUT, 0xFF800, 0);
print OUT chr(0) x 2048;
close OUT or die;
EOF
--let $restart_parameters= $dirs --innodb-force-recovery=5 --innodb-log-file-size=2m
--source include/start_mysqld.inc
SELECT COUNT(*) FROM INFORMATION_SCHEMA.ENGINES
WHERE engine = 'innodb'
AND support IN ('YES', 'DEFAULT', 'ENABLED');
--source include/shutdown_mysqld.inc
let SEARCH_PATTERN=InnoDB: Upgrade after a crash is not supported. This redo log was created before MariaDB 10\\.2\\.2, and it appears corrupted;
--source include/search_pattern_in_file.inc
--echo # Empty multi-file redo log (wrong offset) from before MariaDB 10.2.2
perl;
die unless open OUT, "+<", "$ENV{bugdir}/ib_logfile1";
binmode OUT;
die unless seek(OUT, 0xFF800, 0);
print OUT pack("NnnNx[496]N", 0x80000944, 12, 12, 0, 0xb2a);
print OUT chr(0) x 1536;
close OUT or die;
EOF
--let $restart_parameters= $dirs --innodb-force-recovery=5 --innodb-log-file-size=2m
--source include/start_mysqld.inc
SELECT COUNT(*) FROM INFORMATION_SCHEMA.ENGINES
WHERE engine = 'innodb'
AND support IN ('YES', 'DEFAULT', 'ENABLED');
--source include/shutdown_mysqld.inc
--let SEARCH_PATTERN= Upgrade after a crash is not supported. This redo log was created before MariaDB 10\\.2\\.2, and we did not find a valid checkpoint\\.
--source include/search_pattern_in_file.inc
--echo # Multi-file redo log with size mismatch from after MariaDB 10.2.2
perl;
do "$ENV{MTR_SUITE_DIR}/../innodb/include/crc32.pl";
my $polynomial = 0x82f63b78; # CRC-32C
die unless open OUT, "+<", "$ENV{bugdir}/ib_logfile0";
binmode OUT;
$_= pack("Nx[5]nx[5]", 1, 0x1286) . "BogoDB 1.2.3.4" . chr(0) x 478;
print OUT $_, pack("N", mycrc32($_, 0, $polynomial));
# checkpoint page 1 and all-zero checkpoint 2
$_= pack("x[13]nCNNx[484]", 0x1286, 12, 0, 0x20080c);
print OUT $_, pack("N", mycrc32($_, 0, $polynomial));
print OUT chr(0) x 1024;
close OUT or die;
die unless open OUT, ">", "$ENV{bugdir}/ib_logfile1";
binmode OUT;
die unless seek(OUT, 0xFF800, 0);
print OUT chr(0) x 2048;
close OUT or die;
EOF
--echo # Corrupted multi-file redo log from after MariaDB 10.2.2
--let $restart_parameters= $dirs --innodb-force-recovery=5 --innodb-log-file-size=2m
--source include/start_mysqld.inc
SELECT COUNT(*) FROM INFORMATION_SCHEMA.ENGINES
WHERE engine = 'innodb'
AND support IN ('YES', 'DEFAULT', 'ENABLED');
--source include/shutdown_mysqld.inc
--source include/search_pattern_in_file.inc
--let SEARCH_PATTERN=InnoDB: No valid checkpoint was found; the log was created with BogoDB 1\\.2\\.3\\.4\\.
--source include/search_pattern_in_file.inc
perl;
die unless open OUT, ">", "$ENV{bugdir}/ib_logfile1";
binmode OUT;
die unless seek(OUT, 0x1FFFFF, 0);
print OUT chr(0);
close OUT or die;
EOF
--let $restart_parameters= $dirs --innodb-force-recovery=5 --innodb-log-file-size=2m
--source include/start_mysqld.inc
SELECT COUNT(*) FROM INFORMATION_SCHEMA.ENGINES
WHERE engine = 'innodb'
AND support IN ('YES', 'DEFAULT', 'ENABLED');
--source include/shutdown_mysqld.inc
--let SEARCH_PATTERN=InnoDB: Log file .*ib_logfile1 is of different size 2097152 bytes than other log files 1048576 bytes!
--let SEARCH_PATTERN=InnoDB: No valid checkpoint was found; the log was created with BogoDB 1\\.2\\.3\\.4\\.
--source include/search_pattern_in_file.inc
--echo # Empty multi-file redo log from after MariaDB 10.2.2
perl;
do "$ENV{MTR_SUITE_DIR}/../innodb/include/crc32.pl";
my $polynomial = 0x82f63b78; # CRC-32C
die unless open OUT, "+<", "$ENV{bugdir}/ib_logfile1";
binmode OUT;
die unless seek(OUT, 0x800, 0); # the first 2048 bytes are unused!
$_= pack("Nnnx[500]", 0x80000944, 12, 12);
print OUT $_, pack("N", mycrc32($_, 0, $polynomial));
close OUT or die;
EOF
--let $restart_parameters= $dirs --innodb-force-recovery=5 --innodb-log-file-size=2m
--source include/start_mysqld.inc
SELECT COUNT(*) FROM INFORMATION_SCHEMA.ENGINES
WHERE engine = 'innodb'
AND support IN ('YES', 'DEFAULT', 'ENABLED');
--source include/shutdown_mysqld.inc
--source include/search_pattern_in_file.inc
--let $restart_parameters= $dirs
--remove_file $bugdir/ib_logfile0
--move_file $bugdir/ib_logfile $bugdir/ib_logfile0
--echo # redo log from "after" MariaDB 10.2.2, but with invalid header checksum
perl;
die unless open OUT, "+<", "$ENV{bugdir}/ib_logfile0";
binmode OUT;
print OUT pack("N",2);
close OUT or die;
EOF
--source include/start_mysqld.inc
eval $check_no_innodb;
--source include/shutdown_mysqld.inc
let SEARCH_PATTERN=InnoDB: Invalid log header checksum;
--source include/search_pattern_in_file.inc
--echo # distant future redo log format, with valid header checksum
perl;
die unless open OUT, "+<", "$ENV{bugdir}/ib_logfile0";
binmode OUT;
print OUT pack("Nx[5]nx[5]", ~0, 0x1286);
print OUT "malicious intentions, or perhaps not";
print OUT pack("x[456]N", 0x8a1bb475);
close OUT or die;
EOF
--source include/start_mysqld.inc
eval $check_no_innodb;
--source include/shutdown_mysqld.inc
let SEARCH_PATTERN=InnoDB: Unsupported redo log format. The redo log was created with malicious intentions, or perhaps\.;
--source include/search_pattern_in_file.inc
--echo # valid header, but old-format checkpoint blocks
perl;
die unless open OUT, "+<", "$ENV{bugdir}/ib_logfile0";
binmode OUT;
# header block
print OUT pack("Nx[5]nx[5]", 1, 0x1286);
print OUT "malicious intentions, or perhaps not";
print OUT pack("x[456]N", 0xd42d53a2);
# old-format checkpoint block 1
print OUT pack("x[13]nCNnxxNNx[256]", 0x1286, 12, 0x80c, 0xf0, ~0, ~0);
print OUT pack("H*x[1240]", "590DBAACFE922582");
print OUT pack("H*x[5]", "800009440022000c0000000138");
print OUT pack("H*x[475]H*", "12860cb7809781e80006626f677573", "089C0ADA");
close OUT or die;
EOF
--source include/start_mysqld.inc
eval $check_no_innodb;
--source include/shutdown_mysqld.inc
--let SEARCH_PATTERN=InnoDB: No valid checkpoint was found; the log was created with malicious intentions, or perhaps\\.
--source include/search_pattern_in_file.inc
--echo # valid header, valid checkpoint 1, all-zero (invalid) checkpoint 2, invalid block checksum
perl;
die unless open OUT, "+<", "$ENV{bugdir}/ib_logfile0";
binmode OUT;
die unless seek(OUT, 0x210, 0);
print OUT pack("NNx[264]", 0, 0x80c);
print OUT pack("NNx[212]N", 0x590dbaac, 0xfe922582, 0xc72d49c4);
close OUT or die;
EOF
# Anything below innodb_force_recovery=6 must find a valid redo log.
# Missing tablespace files are tolerated already with innodb_force_recovery=1.
--let $restart_parameters= $dirs --innodb-force-recovery=5
--source include/start_mysqld.inc
eval $check_no_innodb;
--source include/shutdown_mysqld.inc
let SEARCH_PATTERN=InnoDB: Invalid log header checksum;
--source include/search_pattern_in_file.inc
let SEARCH_PATTERN=InnoDB: Upgrade after a crash is not supported\. The redo log was created with malicious intentions, or perhaps, and it appears corrupted\.;
--source include/search_pattern_in_file.inc
--echo # same, but with current-version header
perl;
die unless open OUT, "+<", "$ENV{bugdir}/ib_logfile0";
binmode OUT;
print OUT pack("Nx[5]nx[5]", 103, 0x1286), "MariaDB 10.3.1";
print OUT pack("x[478]N", 0x85021a0f);
close OUT or die;
EOF
--source include/start_mysqld.inc
eval $check_no_innodb;
--source include/shutdown_mysqld.inc
let SEARCH_PATTERN=InnoDB: Invalid log header checksum;
--source include/search_pattern_in_file.inc
--echo # --innodb-force-recovery=6 (skip the entire redo log)
--let $restart_parameters= $dirs --innodb-force-recovery=6
--source include/start_mysqld.inc
eval $check_no_innodb;
--source include/shutdown_mysqld.inc
--let SEARCH_PATTERN=\\[Note\\] InnoDB: log sequence number 0.*; transaction id 0
--source include/search_pattern_in_file.inc
--echo # valid header, valid checkpoint 1, all-zero (invalid) checkpoint 2, invalid block number
perl;
die unless open OUT, "+<", "$ENV{bugdir}/ib_logfile0";
binmode OUT;
# header block
print OUT pack("Nx[5]nx[5]", 103, 0x1286), "MariaDB 10.3.1";
print OUT pack("x[478]N", 0x85021a0f);
# checkpoint page 1 and all-zero checkpoint 2
print OUT pack("x[13]nCNNx[264]", 0x1286, 12, 0, 0x80c);
print OUT pack("H*x[212]Nx[1024]", "590DBAACFE922582", 0xc72d49c4);
# redo log data
print OUT pack("H*x[5]", "C0DEBA5E0022000c0000000138");
print OUT pack("H*x[475]H*", "12860cb7809781e80006626f677573", "089C0ADA");
close OUT or die;
EOF
--copy_file $bugdir/ib_logfile0 $bugdir/ib_logfile
# Anything below innodb_force_recovery=6 must find an invalid redo log.
# Missing tablespace files are tolerated already with innodb_force_recovery=1.
--let $restart_parameters= $dirs --innodb-force-recovery=5
--source include/start_mysqld.inc
eval $check_no_innodb;
--source include/shutdown_mysqld.inc
--echo # --innodb-force-recovery=6 (skip the entire redo log)
--let $restart_parameters= $dirs --innodb-force-recovery=6
--source include/start_mysqld.inc
eval $check_no_innodb;
--source include/shutdown_mysqld.inc
--echo # Test a corrupted MLOG_FILE_NAME record.
--echo # valid header, valid checkpoint 1, all-zero (invalid) checkpoint 2
--remove_file $bugdir/ib_logfile0
--move_file $bugdir/ib_logfile $bugdir/ib_logfile0
perl;
die unless open OUT, "+<", "$ENV{bugdir}/ib_logfile0";
binmode OUT;
die unless seek(OUT, 0x800, 0);
print OUT pack("N", 0x80000944);
die unless seek(OUT, 0x9fc, 0);
print OUT pack("N", 0xc86474db);
close OUT or die;
EOF
--let $restart_parameters= $dirs
--source include/start_mysqld.inc
eval $check_no_innodb;
--source include/shutdown_mysqld.inc
let SEARCH_PATTERN=\\[ERROR\\] InnoDB: Upgrade after a crash is not supported\. The redo log was created with MariaDB 10\.3\.1\.;
--source include/search_pattern_in_file.inc
--echo # Test a corrupted MLOG_FILE_NAME record.
--echo # valid header, invalid checkpoint 1, valid checkpoint 2, invalid block
perl;
die unless open OUT, "+<", "$ENV{bugdir}/ib_logfile0";
binmode OUT;
# header block
print OUT pack("Nx[5]nx[5]", 103, 0x1286), "MariaDB 10.3.1";
print OUT pack("x[478]N", 0x85021a0f);
# invalid (all-zero) checkpoint page 1 and an empty log page
print OUT chr(0) x 1024;
# valid checkpoint block 2
print OUT pack("x[13]nCNNx[264]", 0x1286, 12, 0, 0x80c);
print OUT pack("H*x[212]N", "590DBAACFE922582", 0xc72d49c4);
# log page
print OUT pack("H*", "800009440022000c00000001");
# MLOG_CHECKPOINT record
print OUT pack("H*", "38000000000012860c");
# MLOG_FILE_NAME record
print OUT pack("H*", "b7809781e80006"), "bigot";
# padding and wrong log block checksum
print OUT pack("x[475]N", 150151);
close OUT or die;
EOF
--source include/start_mysqld.inc
eval $check_no_innodb;
--source include/shutdown_mysqld.inc
let SEARCH_PATTERN=InnoDB: Invalid log header checksum
--source include/search_pattern_in_file.inc
let SEARCH_PATTERN=\\[ERROR\\] InnoDB: Upgrade after a crash is not supported\. The redo log was created with MariaDB 10\.3\.1, and it appears corrupted\.;
--source include/search_pattern_in_file.inc
--echo # valid header, invalid checkpoint 1, valid checkpoint 2, invalid log record
perl;
die unless open OUT, "+<", "$ENV{bugdir}/ib_logfile0";
binmode OUT;
die unless seek(OUT, 0x9fc, 0);
print OUT pack("N", 2454333373);
close OUT or die;
EOF
--source include/start_mysqld.inc
eval $check_no_innodb;
--source include/shutdown_mysqld.inc
let SEARCH_PATTERN=\\[ERROR\\] InnoDB: Upgrade after a crash is not supported\. The redo log was created with MariaDB 10\.3\.1\.;
--source include/search_pattern_in_file.inc
--echo # 10.2 missing MLOG_FILE_NAME or MLOG_FILE_DELETE before MLOG_CHECKPOINT
perl;
die unless open OUT, "+<", "$ENV{bugdir}/ib_logfile0";
binmode OUT;
# header block
print OUT pack("Nx[5]nx[5]", 1, 0x1286);
print OUT "ibbackup was here!!!1!";
print OUT pack("x[470]N", 0x52b54540);
# invalid (all-zero) checkpoint page 1 and an empty log page
print OUT chr(0) x 1024;
# valid checkpoint block 2
print OUT pack("x[12]NNNx[264]", 0x12860c, 0, 0x80c);
# pointer to the MLOG_CHECKPOINT record, and checkpoint page checksum
print OUT pack("H*x[204]NNN", "590DBAACFE922582", 0x128612, 0, 0x101741b);
# log page
print OUT pack("H*", "800009440022000c00000001");
# MLOG_1BYTE record for missing tablespace 42 page 123
print OUT pack("CCCnC", 128 | 1, 42, 123, 4567, 89);
# MLOG_CHECKPOINT record
print OUT pack("CNN", 56, 0, 0x12860c);
# padding (MLOG_DUMMY_RECORD) and block checksum
print OUT " " x 481, pack("N", 0x644e6db8);
close OUT or die;
EOF
--source include/start_mysqld.inc
eval $check_no_innodb;
--source include/shutdown_mysqld.inc
let SEARCH_PATTERN=\\[ERROR\\] InnoDB: Upgrade after a crash is not supported\. The redo log was created with ibbackup was here!!!1!\.;
--source include/search_pattern_in_file.inc
--echo # 10.3 missing MLOG_FILE_NAME or MLOG_FILE_DELETE before MLOG_CHECKPOINT
perl;
die unless open OUT, "+<", "$ENV{bugdir}/ib_logfile0";
binmode OUT;
print OUT pack("Nx[5]nx[5]", 103, 0x1286), "MariaDB 10.3.1";
print OUT pack("x[478]N", 0x85021a0f);
close OUT or die;
EOF
--source include/start_mysqld.inc
eval $check_no_innodb;
--source include/shutdown_mysqld.inc
let SEARCH_PATTERN=\\[ERROR\\] InnoDB: Upgrade after a crash is not supported\. The redo log was created with MariaDB 10\.3\.1\.;
--source include/search_pattern_in_file.inc
--echo # Empty 10.3 redo log
perl;
die unless open OUT, "+<", "$ENV{bugdir}/ib_logfile0";
binmode OUT;
die unless seek(OUT, 0x800, 0);
print OUT pack("NnnNx[496]N", 0x80000944, 12, 12, 1, 0x46c8a2a2);
close OUT or die;
EOF
--let $restart_parameters= $dirs --innodb-force-recovery=5 --innodb-log-file-size=2m
--source include/start_mysqld.inc
SELECT COUNT(*) FROM INFORMATION_SCHEMA.ENGINES
WHERE engine = 'innodb'
AND support IN ('YES', 'DEFAULT', 'ENABLED');
--source include/shutdown_mysqld.inc
--let SEARCH_PATTERN= InnoDB: log sequence number 1213964\\b.*; transaction id 0
--source include/search_pattern_in_file.inc
--echo # Empty 10.2 redo log
perl;
die unless open OUT, "+<", "$ENV{bugdir}/ib_logfile0";
binmode OUT;
# header block
print OUT pack("Nx[5]nx[5]", 1, 0x1286);
print OUT "ibbackup was here!!!1!";
print OUT pack("x[470]N", 0x52b54540);
# In encryption.innodb_log_corruption the previous step would
# replace the block with an encrypted one and update the checkpoint.
# Restore them.
# invalid (all-zero) checkpoint page 1 and an empty log page
print OUT chr(0) x 1024;
# valid checkpoint block 2
print OUT pack("x[12]NNNx[264]", 0x12860c, 0, 0x80c);
# pointer to the MLOG_CHECKPOINT record, and checkpoint page checksum
print OUT pack("H*x[204]NNN", "590DBAACFE922582", 0x128612, 0, 0x101741b);
# log page
print OUT pack("NnnNx[496]N", 0x80000944, 12, 12, 1, 0x46c8a2a2);
close OUT or die;
EOF
--source include/start_mysqld.inc
SELECT COUNT(*) FROM INFORMATION_SCHEMA.ENGINES
WHERE engine = 'innodb'
AND support IN ('YES', 'DEFAULT', 'ENABLED');
--source include/shutdown_mysqld.inc
--let SEARCH_PATTERN= InnoDB: Upgrading redo log:
--source include/search_pattern_in_file.inc
--echo # Empty 10.5 redo log
perl;
die unless open OUT, "+<", "$ENV{bugdir}/ib_logfile0";
binmode OUT;
# header block
print OUT pack("Nx[5]nx[5]", 0x50485953, 0x1286);
print OUT "ibbackup was here!!!1!";
print OUT pack("x[470]N", 0x677700cf);
# invalid (all-zero) checkpoint page 1 and an empty log page
print OUT chr(0) x 1024;
# valid checkpoint block 2
print OUT pack("x[12]NNNx[264]", 0x12860c, 0, 0x80c);
# pointer to the FILE_CHECKPOINT record, and checkpoint page checksum
print OUT pack("H*x[204]NNN", "590DBAACFE922582", 0x128612, 0, 0x101741b);
# log page
print OUT pack("NnnNx[496]N", 0x80000944, 12, 12, 1, 0x46c8a2a2);
close OUT or die;
EOF
--source include/start_mysqld.inc
SELECT COUNT(*) FROM INFORMATION_SCHEMA.ENGINES
WHERE engine = 'innodb'
AND support IN ('YES', 'DEFAULT', 'ENABLED');
--source include/shutdown_mysqld.inc
--let SEARCH_PATTERN= InnoDB: Upgrading redo log:
--source include/search_pattern_in_file.inc
--echo # Minimal MariaDB 10.1.21 encrypted redo log
perl;
die unless open OUT, "+<", "$ENV{bugdir}/ib_logfile0";
binmode OUT;
# header and empty checkpoint page 1 (not encrypted)
print OUT pack("x[9]nx[5]A[4]x[46]Cx[1469]", 0x186e, "", 2);
# checkpoint page 2 (not encrypted)
print OUT pack("NNNNNNNN", 0, 1, 0, 0x186e16, 0x816, 0x1000000, ~0, ~0);
print OUT pack("x[6]Cx[249]", 0x20);
print OUT pack("H*x[12]nNNn", "FDE012DE6D3BD6AC", 0x201, 1, 1, 0xEFC);
print OUT pack("H*x[162]",
"75774015174BE4723904C88E110E3B50FF5C634167DB058D02A415D2A4A3");
# log block (encrypted)
print OUT pack("H*", "80000c380016000c000000015cf22e8aff355642045605c22b97f7cba94cc9932b6032b9c828ef85269024127d3863193b44e52d22eccdd0a8b197ce837f6b530c3042a022738bbac803adc5e7607333409b35c1f6351601a9da83d9b0bc3f389fb288d2bd6a0783066e3b07cae0e276fe83aa522d3f2b59b9549e2338031674383bbef87d7a42f415870fa33685afd3894a9d8214d60b5cde0c776489f45ad0c759e0495bc5d0c91018750070ed21ec1a41bcb9054007851df28e1da15fb662ab219bd558f1569b22671c943bfb0c12e97b604feb0d96a53c05c868e2eea9d4692cd0b6f4625a52adc8251ac955957e33d8c32eb964e416b246b950321a71d5b2e8e95c26e00333abe24ca827cd2eb95cff7b21dce639073f526614816fa2151afdb967aa049b4625cd70881fbd73827c1eb6f70447dcb7755055fa939a2c8ec38fc24f135c22f9f9691bd34a91be7047e5fd68d2e68b0d072cb69c253e14c7111b950b9d6df327f3ffb2a0dc6eff045d64baeb3c75c4ae4aeb41235757f40291540148d79deaff983851247134d7513a550ed7e21bf2e209d8dec51b252f0e9ba6fa33cc2ad0b28d05c04442dac9d233d0ca93d8ef0b81ba82e7396ffaec979b2df97f7249a20d8cb8036b582c245f27e64ac12c8b1d46248b670936893c7c737ba8a4ecf623689c18d79755865626993a43d607611d0736d1c626ebf88946");
close OUT or die;
EOF
--let $restart_parameters= $dirs --innodb-force-recovery=5
--source include/start_mysqld.inc
if (!$no_cleanup) {
eval $check_no_innodb;
--source include/shutdown_mysqld.inc
--let SEARCH_PATTERN= InnoDB: Obtaining redo log encryption key version 1 failed
--source include/search_pattern_in_file.inc
--let SEARCH_PATTERN= InnoDB: Decrypting checkpoint failed
--source include/search_pattern_in_file.inc
--list_files $bugdir
--remove_files_wildcard $bugdir
--rmdir $bugdir
--let $restart_parameters=
--source include/start_mysqld.inc
}
View Git Blame Copy Permalink