The setting innodb_safe_truncate=ON reduces compatibility with older
versions of MariaDB and backup tools in two ways.
First, we will be writing TRX_UNDO_RENAME_TABLE records, which older
versions do not know about. These records could be misinterpreted if
a DDL transaction was recovered and would be rolled back.
Such rollback is only possible if the server was killed while
an incomplete DDL transaction was persisted. On transaction completion,
the insert_undo log pages would only be repurposed for new undo log
allocations, and their contents would not matter. So, older versions
will not have a problem with innodb_safe_truncate=ON if the server was
shut down cleanly.
Second, to prevent such recovery failure, innodb_safe_truncate=ON will
cause a modification of the redo log format identifier, which will
prevent older versions from starting up after a crash. MariaDB Server
versions older than 10.2.13 will refuse to start up altogether, even
after clean shutdown.
A server restart with innodb_safe_truncate=OFF will restore compatibility
with older server and backup versions.
Rename the 10.2-specific configuration option innodb_unsafe_truncate
to innodb_safe_truncate, and invert its value.
The default (for now) is innodb_safe_truncate=OFF, to avoid
disrupting users with an undo and redo log format change within
a Generally Available (GA) release series.
While MariaDB Server 10.2 is not really guaranteed to be compatible
with Percona XtraBackup 2.4 (for example, the MySQL 5.7 undo log format
change that could be present in XtraBackup, but was reverted from
MariaDB in MDEV-12289), we do not want to disrupt users who have
deployed xtrabackup and MariaDB Server 10.2 in their environments.
With this change, MariaDB 10.2 will continue to use the backup-unsafe
TRUNCATE TABLE code, so that neither the undo log nor the redo log
formats will change in an incompatible way.
Undo tablespace truncation will keep using the redo log only. Recovery
or backup with old code will fail to shrink the undo tablespace files,
but the contents will be recovered just fine.
In the MariaDB Server 10.2 series only, we introduce the configuration
parameter innodb_unsafe_truncate and make it ON by default. To allow
MariaDB Backup (mariabackup) to work properly with TRUNCATE TABLE
operations, use loose_innodb_unsafe_truncate=OFF.
MariaDB Server 10.3.10 and later releases will always use the
backup-safe TRUNCATE TABLE, and this parameter will not be
added there.
recv_recovery_rollback_active(): Skip row_mysql_drop_garbage_tables()
unless innodb_unsafe_truncate=OFF. It is too unsafe to drop orphan
tables if RENAME operations are not transactional within InnoDB.
LOG_HEADER_FORMAT_10_3: Replaces LOG_HEADER_FORMAT_CURRENT.
log_init(), log_group_file_header_flush(),
srv_prepare_to_delete_redo_log_files(),
innobase_start_or_create_for_mysql(): Choose the redo log format
and subformat based on the value of innodb_unsafe_truncate.
This concludes the merge of all applicable InnoDB changes from
MySQL 5.7.23, with the exception of a performance fix, which we
plan to rewrite in MariaDB later in such a way that it does not
involve changing the storage engine API:
MDEV-16849 Extending indexed VARCHAR column should be instantaneous
Introduce the configuration option innodb_log_optimize_ddl
for controlling whether native index creation or table-rebuild
in InnoDB should keep optimizing the redo log
(and writing MLOG_INDEX_LOAD records to ensure that
concurrent backup would fail).
By default, we have innodb_log_optimize_ddl=ON, that is,
the default behaviour that was introduced in MariaDB 10.2.2
(with the merge of InnoDB from MySQL 5.7) will be unchanged.
BtrBulk::m_trx: Replaces m_trx_id. We must be able to check for
KILL QUERY even if !m_flush_observer (innodb_log_optimize_ddl=OFF).
page_cur_insert_rec_write_log(): Declare globally, so that this
can be called from PageBulk::insert().
row_merge_insert_index_tuples(): Remove the unused parameter trx_id.
row_merge_build_indexes(): Enable or disable redo logging based on
the innodb_log_optimize_ddl parameter.
PageBulk::init(), PageBulk::insert(), PageBulk::finish(): Write
redo log records if needed. For ROW_FORMAT=COMPRESSED, redo log
will be written in PageBulk::compress() unless we called
m_mtr.set_log_mode(MTR_LOG_NO_REDO).
The parameter innodb_lock_schedule_algorithm was introduced in
MariaDB Server 10.1.19, 10.2.13, 10.3.4 as part of MDEV-11039.
In MariaDB 10.1, the default value of the parameter is 'fcfs',
that is, the existing algorithm is used by default. But in
later versions of MariaDB Server, the parameter was 'vats',
enabling the new algorithm.
Because the new algorithm is triggering a debug assertion failure
that suggests corruption of the transactional lock data structures,
we will revert to the old algorithm by default until we have
resolved the problem.
InnoDB in Debian uses utf8mb4 as default character set since
version 10.0.20-2. This leads to major pain due to keys longer
than 767 bytes.
MariaDB 10.2 (and MySQL 5.7) introduced the setting
innodb_default_row_format that is DYNAMIC by default. These
versions also changed the default values of the parameters
innodb_large_prefix=ON and innodb_file_format=Barracuda.
This would allow longer column index prefixes to be created.
The original purpose of these parameters was to allow InnoDB
to be downgraded to MySQL 5.1, which is long out of support.
Every InnoDB version since MySQL 5.5 does support operation
with the relaxed limits.
We backport the parameter innodb_default_row_format to
MariaDB 10.1, but we will keep its default value at COMPACT.
This allows MariaDB 10.1 to be configured so that CREATE TABLE
is less likely to encounter a problem with the limitation:
loose_innodb_large_prefix=ON
loose_innodb_default_row_format=DYNAMIC
(Note that the setting innodb_large_prefix was deprecated in
MariaDB 10.2 and removed in MariaDB 10.3.)
The only observable difference in the behaviour with the default
settings should be that ROW_FORMAT=DYNAMIC tables can be created
both in the system tablespace and in .ibd files, no matter what
innodb_file_format has been assigned to. Unlike MariaDB 10.2,
we are not changing the default value of innodb_file_format,
so ROW_FORMAT=COMPRESSED tables cannot be created without
changing the parameter.
Two follow-up tasks were filed for MySQL 5.7.21 changes that
were not applied here:
MDEV-15179 performance_schema.file_instances does not reflect RENAME TABLE
MDEV-14222 Unnecessary 'cascade' memory allocation for every updated
row when there is no FOREIGN KEY
This is 10.1 version where no merge error exists.
wsrep_on_check
New check function. Galera can't be enabled
if innodb-lock-schedule-algorithm=VATS.
innobase_kill_query
In Galera async kill we could own lock mutex.
innobase_init
If Variance-Aware-Transaction-Sheduling Algorithm (VATS) is
used on Galera we refuse to start InnoDB.
Changed innodb-lock-schedule-algorithm as read-only parameter
as it was designed to be.
lock_rec_other_has_expl_req,
lock_rec_other_has_conflicting,
lock_rec_lock_slow
lock_table_other_has_incompatible
lock_rec_insert_check_and_lock
Change pointer to conflicting lock to normal pointer as this
pointer contents could be changed later.
The InnoDB purge subsystem can be best stopped by opening a read view,
for example by START TRANSACTION WITH CONSISTENT SNAPSHOT.
To ensure that everything is purged, use wait_all_purged.inc,
which waits for the History list length in SHOW ENGINE INNODB STATUS
to reach 0. Setting innodb_purge_run_now never guaranteed this.
Only a relevant subset of the InnoDB changes was merged.
In particular, two follow-up bug fixes for the bugs that
were introduced in 5.7.18 but not MariaDB 10.2.7 were omitted.
Because MariaDB 10.2.7 omitted the risky change
Bug#23481444 OPTIMISER CALL ROW_SEARCH_MVCC() AND READ THE INDEX
APPLIED BY UNCOMMITTED ROWS
we do not need the follow-up fixes that were introduced in
MySQL 5.6.37 and MySQL 5.7.19:
Bug#25175249 ASSERTION: (TEMPL->IS_VIRTUAL && !FIELD) || ...
Bug#25793677 INNODB: FAILING ASSERTION: CLUST_TEMPL_FOR_SEC || LEN
The option innodb_log_compressed_pages was contributed by
Facebook to MySQL 5.6. It was disabled in the 5.6.10 GA release
due to problems that were fixed in 5.6.11, which is when the
option was enabled.
The option was set to innodb_log_compressed_pages=ON by default
(disabling the feature), because safety was considered more
important than speed. The option innodb_log_compressed_pages=OFF
can *CORRUPT* ROW_FORMAT=COMPRESSED tables on crash recovery
if the zlib deflate function is behaving differently (producing
a different amount of compressed data) from how it behaved
when the redo log records were written (prior to the crash recovery).
In MDEV-6935, the default value was changed to
innodb_log_compressed_pages=OFF. This is inherently unsafe, because
there are very many different environments where MariaDB can be
running, using different zlib versions. While zlib can decompress
data just fine, there are no guarantees that different versions will
always compress the same data to the exactly same size. To avoid
problems related to zlib upgrades or version mismatch, we must
use a safe default setting.
This will reduce the write performance for users of
ROW_FORMAT=COMPRESSED tables. If you configure
innodb_log_compressed_pages=ON, please make sure that you will
always cleanly shut down InnoDB before upgrading the server
or zlib.
Significantly reduce the amount of InnoDB, XtraDB and Mariabackup
code changes by defining pfs_os_file_t as something that is
transparently compatible with os_file_t.
Simplify the tests that are present in MySQL 5.7. Make the table
smaller while generating enough undo log. Do not unnecessarily
drop tables.
trx_purge_initiate_truncate(): Remove two crash injection points
(before and after normal redo log checkpoint), because they are
not adding any value. Clarify some messages.
trx_sys_create_rsegs(): Display the number of active undo tablespaces.
srv_undo_tablespaces_init(): When initializing the data files, do not
leave srv_undo_tablespaces_active at 0.
Do not display that number; let trx_sys_create_rsegs() display it once
the final number is known.
innodb_params_adjust(): Adjust parameters after startup.
innobase_init(): Do not allow innodb_max_undo_size to be less
than SRV_UNDO_TABLESPACE_SIZE_IN_PAGES. This avoids unnecessary
repeated truncation of undo tablespaces when using
innodb_page_size=32k or innodb_page_size=64k.
MySQL 5.7 reduced the maximum number of innodb_undo_tablespaces
from 126 to 95 when it reserved 32 persistent rollback segments
for the temporary undo logs. Since MDEV-12289 restored all 128
persistent rollback segments for persistent undo logs, the
reasonable maximum value of innodb_undo_tablespaces is 127
(not 126 or 95). This is because out of the 128 rollback segments,
the first one will always be created in the system tablespace
and the remaining ones can be created in dedicated undo tablespaces.
MDEV-11581: Mariadb starts InnoDB encryption threads
when key has not changed or data scrubbing turned off
Background: Key rotation is based on background threads
(innodb-encryption-threads) periodically going through
all tablespaces on fil_system. For each tablespace
current used key version is compared to max key age
(innodb-encryption-rotate-key-age). This process
naturally takes CPU. Similarly, in same time need for
scrubbing is investigated. Currently, key rotation
is fully supported on Amazon AWS key management plugin
only but InnoDB does not have knowledge what key
management plugin is used.
This patch re-purposes innodb-encryption-rotate-key-age=0
to disable key rotation and background data scrubbing.
All new tables are added to special list for key rotation
and key rotation is based on sending a event to
background encryption threads instead of using periodic
checking (i.e. timeout).
fil0fil.cc: Added functions fil_space_acquire_low()
to acquire a tablespace when it could be dropped concurrently.
This function is used from fil_space_acquire() or
fil_space_acquire_silent() that will not print
any messages if we try to acquire space that does not exist.
fil_space_release() to release a acquired tablespace.
fil_space_next() to iterate tablespaces in fil_system
using fil_space_acquire() and fil_space_release().
Similarly, fil_space_keyrotation_next() to iterate new
list fil_system->rotation_list where new tables.
are added if key rotation is disabled.
Removed unnecessary functions fil_get_first_space_safe()
fil_get_next_space_safe()
fil_node_open_file(): After page 0 is read read also
crypt_info if it is not yet read.
btr_scrub_lock_dict_func()
buf_page_check_corrupt()
buf_page_encrypt_before_write()
buf_merge_or_delete_for_page()
lock_print_info_all_transactions()
row_fts_psort_info_init()
row_truncate_table_for_mysql()
row_drop_table_for_mysql()
Use fil_space_acquire()/release() to access fil_space_t.
buf_page_decrypt_after_read():
Use fil_space_get_crypt_data() because at this point
we might not yet have read page 0.
fil0crypt.cc/fil0fil.h: Lot of changes. Pass fil_space_t* directly
to functions needing it and store fil_space_t* to rotation state.
Use fil_space_acquire()/release() when iterating tablespaces
and removed unnecessary is_closing from fil_crypt_t. Use
fil_space_t::is_stopping() to detect when access to
tablespace should be stopped. Removed unnecessary
fil_space_get_crypt_data().
fil_space_create(): Inform key rotation that there could
be something to do if key rotation is disabled and new
table with encryption enabled is created.
Remove unnecessary functions fil_get_first_space_safe()
and fil_get_next_space_safe(). fil_space_acquire()
and fil_space_release() are used instead. Moved
fil_space_get_crypt_data() and fil_space_set_crypt_data()
to fil0crypt.cc.
fsp_header_init(): Acquire fil_space_t*, write crypt_data
and release space.
check_table_options()
Renamed FIL_SPACE_ENCRYPTION_* TO FIL_ENCRYPTION_*
i_s.cc: Added ROTATING_OR_FLUSHING field to
information_schema.innodb_tablespace_encryption
to show current status of key rotation.
Because the default build configuration of the server will remain
at -DWITH_INNODB_AHI=ON, we want to test the instrumentation.
We make and revert the test adjustments in separate commits on purpose,
so that this commit can be easily reverted later if the default
build configuration is changed to -DWITH_INNODB_AHI=OFF.
The InnoDB adaptive hash index is sometimes degrading the performance of
InnoDB, and it is sometimes disabled to get more consistent performance.
We should have a compile-time option to disable the adaptive hash index.
Let us introduce two options:
OPTION(WITH_INNODB_AHI "Include innodb_adaptive_hash_index" ON)
OPTION(WITH_INNODB_ROOT_GUESS "Cache index root block descriptors" ON)
where WITH_INNODB_AHI always implies WITH_INNODB_ROOT_GUESS.
As part of this change, the misleadingly named function
trx_search_latch_release_if_reserved(trx) will be replaced with the macro
trx_assert_no_search_latch(trx) that will be empty unless
BTR_CUR_HASH_ADAPT is defined (cmake -DWITH_INNODB_AHI=ON).
We will also remove the unused column
INFORMATION_SCHEMA.INNODB_TRX.TRX_ADAPTIVE_HASH_TIMEOUT.
In MariaDB Server 10.1, it used to reflect the value of
trx_t::search_latch_timeout which could be adjusted during
row_search_for_mysql(). In 10.2, there is no such field.
Other than the removal of the unused column TRX_ADAPTIVE_HASH_TIMEOUT,
this is an almost non-functional change to the server when using the
default build options.
Some tests are adjusted so that they will work with both
-DWITH_INNODB_AHI=ON and -DWITH_INNODB_AHI=OFF. The test
innodb.innodb_monitor has been renamed to innodb.monitor
in order to track MySQL 5.7, and the duplicate tests
sys_vars.innodb_monitor_* are removed.
MDEV-7618 introduced configuration parameter innodb_instrument_semaphores
in MariaDB Server 10.1. The parameter seems to only affect the rw-lock
X-latch acquisition. Extra fields are added to rw_lock_t to remember one
X-latch holder or waiter. These fields are not being consulted or reported
anywhere. This is basically only adding code bloat.
If the intention is to debug hangs or deadlocks, we have better tools for
that in the debug server, and for the non-debug server, core dumps can
reveal a lot. For example, the mini-transaction memo records the
currently held buffer block or index rw-locks, to be released at
mtr_t::commit().
The configuration parameter innodb_instrument_semaphores will be
deprecated in 10.2.5 and removed in 10.3.0.
rw_lock_t: Remove the members lock_name, file_name, line, thread_id
which did not affect any output.
The InnoDB redo log consists of a list of files that logically form
a bigger file, as if the individual files were concatenated together.
The first file will always be written on redo log checkpoint, because
the two checkpoint pages are at the start of the single logical
redo log file.
There is no technical reason why InnoDB requires at least 2 files
to exist. Let us reduce the minimum number to 1. In that way,
restoring from backups will become easier, since InnoDB can directly
deal with a single backed-up redo log file.
Oracle introduced a Memcached plugin interface to the InnoDB
storage engine in MySQL 5.6. That interface is essentially a
fork of Memcached development snapshot 1.6.0-beta1 of an old
development branch 'engine-pu'.
To my knowledge, there have not been any updates to the Memcached code
between MySQL 5.6 and 5.7; only bug fixes and extensions related to
the Oracle modifications.
The Memcached plugin is not part of the MariaDB Server. Therefore it
does not make sense to include the InnoDB interfaces for the Memcached
plugin, or to have any related configuration parameters:
innodb_api_bk_commit_interval
innodb_api_disable_rowlock
innodb_api_enable_binlog
innodb_api_enable_mdl
innodb_api_trx_level
Removing this code in one commit makes it possible to easily restore
it, in case it turns out to be needed later.
Remove the debug parameter innodb_force_recovery_crash that was
introduced into MySQL 5.6 by me in WL#6494 which allowed InnoDB
to resize the redo log on startup.
Let innodb.log_file_size actually start up the server, but ensure
that the InnoDB storage engine refuses to start up in each of the
scenarios.
Problem was that implementation merged from 10.1 was incompatible
with InnoDB 5.7.
buf0buf.cc: Add functions to return should we punch hole and
how big.
buf0flu.cc: Add written page to IORequest
fil0fil.cc: Remove unneeded status call and add test is
sparse files and punch hole supported by file system when
tablespace is created. Add call to get file system
block size. Used file node is added to IORequest. Added
functions to check is punch hole supported and setting
punch hole.
ha_innodb.cc: Remove unneeded status variables (trim512-32768)
and trim_op_saved. Deprecate innodb_use_trim and
set it ON by default. Add function to set innodb-use-trim
dynamically.
dberr.h: Add error code DB_IO_NO_PUNCH_HOLE
if punch hole operation fails.
fil0fil.h: Add punch_hole variable to fil_space_t and
block size to fil_node_t.
os0api.h: Header to helper functions on buf0buf.cc and
fil0fil.cc for os0file.h
os0file.h: Remove unneeded m_block_size from IORequest
and add bpage to IORequest to know actual size of
the block and m_fil_node to know tablespace file
system block size and does it support punch hole.
os0file.cc: Add function punch_hole() to IORequest
to do punch_hole operation,
get the file system block size and determine
does file system support sparse files (for punch hole).
page0size.h: remove implicit copy disable and
use this implicit copy to implement copy_from()
function.
buf0dblwr.cc, buf0flu.cc, buf0rea.cc, fil0fil.cc, fil0fil.h,
os0file.h, os0file.cc, log0log.cc, log0recv.cc:
Remove unneeded write_size parameter from fil_io
calls.
srv0mon.h, srv0srv.h, srv0mon.cc: Remove unneeded
trim512-trim32678 status variables. Removed
these from monitor tests.
Change default to zlib, this has effect only if user has
explicitly requested page compression and then user
naturally expects that pages are really compressed
if they can be compressed.
- Atomic writes are enabled by default
- Automatically detect if device supports atomic write and use it if
atomic writes are enabled
- Remove ATOMIC WRITE options from CREATE TABLE
- Atomic write is a device option, not a table options as the table may
crash if the media changes
- Add support for SHANNON SSD cards
