Let us make innodb_buffer_pool_filename a read-only variable
so that a malicious user cannot cause an important file to be
deleted on InnoDB shutdown. An attempt to delete a directory
will fail because it is not a regular file, but what if the
variable pointed to (say) ibdata1, ib_logfile0 or some *.ibd file?
It does not seem to make much sense for this parameter to be
configurable in the first place, but we will not change that in order
to avoid breaking compatibility.
The deprecated parameters will be removed:
innodb_defragment
innodb_defragment_n_pages
innodb_defragment_stats_accuracy
innodb_defragment_fill_factor_n_recs
innodb_defragment_fill_factor
innodb_defragment_frequency
The mysql.innodb_index_stats.stat_name values 'n_page_split' and
'n_pages_freed' will lose their special meaning.
The related changes to OPTIMIZE TABLE in InnoDB will be removed as well.
The parameter innodb_optimize_fulltext_only will retain its special
meaning in OPTIMIZE TABLE.
Tested by: Matthias Leich
Starting with commit baf276e6d4 (MDEV-19229)
the parameter innodb_undo_tablespaces can be increased from its
previous default value 0 while allowing an upgrade from old databases.
We will change the default setting to innodb_undo_tablespaces=3
so that the space occupied by possible bursts of undo log records
can be reclaimed after SET GLOBAL innodb_undo_log_truncate=ON.
We will not enable innodb_undo_log_truncate by default, because it
causes some observable performance degradation.
Special thanks to Thirunarayanan Balathandayuthapani for diagnosing
and fixing a number of bugs related to this new default setting.
Tested by: Matthias Leich, Axel Schwenke, Vladislav Vaintroub
(with both values of innodb_undo_log_truncate)
The purpose of the change buffer was to reduce random disk access,
which could be useful on rotational storage, but maybe less so on
solid-state storage.
When we wished to
(1) insert a record into a non-unique secondary index,
(2) delete-mark a secondary index record,
(3) delete a secondary index record as part of purge (but not ROLLBACK),
and the B-tree leaf page where the record belongs to is not in the buffer
pool, we inserted a record into the change buffer B-tree, indexed by
the page identifier. When the page was eventually read into the buffer
pool, we looked up the change buffer B-tree for any modifications to the
page, applied these upon the completion of the read operation. This
was called the insert buffer merge.
We remove the change buffer, because it has been the source of
various hard-to-reproduce corruption bugs, including those fixed in
commit 5b9ee8d819 and
commit 165564d3c3 but not limited to them.
A downgrade will fail with a clear message starting with
commit db14eb16f9 (MDEV-30106).
buf_page_t::state: Merge IBUF_EXIST to UNFIXED and
WRITE_FIX_IBUF to WRITE_FIX.
buf_pool_t::watch[]: Remove.
trx_t: Move isolation_level, check_foreigns, check_unique_secondary,
bulk_insert into the same bit-field. The only purpose of
trx_t::check_unique_secondary is to enable bulk insert into an
empty table. It no longer enables insert buffering for UNIQUE INDEX.
btr_cur_t::thr: Remove. This field was originally needed for change
buffering. Later, its use was extended to cover SPATIAL INDEX.
Much of the time, rtr_info::thr holds this field. When it does not,
we will add parameters to SPATIAL INDEX specific functions.
ibuf_upgrade_needed(): Check if the change buffer needs to be updated.
ibuf_upgrade(): Merge and upgrade the change buffer after all redo log
has been applied. Free any pages consumed by the change buffer, and
zero out the change buffer root page to mark the upgrade completed,
and to prevent a downgrade to an earlier version.
dict_load_tablespaces(): Renamed from
dict_check_tablespaces_and_store_max_id(). This needs to be invoked
before ibuf_upgrade().
btr_cur_open_at_rnd_pos(): Specialize for use in persistent statistics.
The change buffer merge does not need this function anymore.
btr_page_alloc(): Renamed from btr_page_alloc_low(). We no longer
allocate any change buffer pages.
btr_cur_open_at_rnd_pos(): Specialize for use in persistent statistics.
The change buffer merge does not need this function anymore.
row_search_index_entry(), btr_lift_page_up(): Add a parameter thr
for the SPATIAL INDEX case.
rtr_page_split_and_insert(): Specialized from btr_page_split_and_insert().
rtr_root_raise_and_insert(): Specialized from btr_root_raise_and_insert().
Note: The support for upgrading from the MySQL 3.23 or MySQL 4.0
change buffer format that predates the MySQL 4.1 introduction of
the option innodb_file_per_table was removed in MySQL 5.6.5
as part of mysql/mysql-server@69b6241a79
and MariaDB 10.0.11 as part of 1d0f70c2f8.
In the tests innodb.log_upgrade and innodb.log_corruption, we create
valid (upgraded) change buffer pages.
Tested by: Matthias Leich
We introduce the following settable Boolean global variables:
innodb_log_file_write_through: Whether writes to ib_logfile0 are
write-through (disabling any caching, as in O_SYNC or O_DSYNC).
innodb_data_file_write_through: Whether writes to any InnoDB data files
(including the temporary tablespace) are write-through.
innodb_data_file_buffering: Whether the file system cache is enabled
for InnoDB data files.
All these parameters are OFF by default, that is, the file system cache
will be disabled, but any hardware caching is enabled, that is,
explicit calls to fsync(), fdatasync() or similar functions are needed.
On systems that support FUA it may make sense to enable write-through,
to avoid extra system calls.
If the deprecated read-only start-up parameter is set to one of the
following values, then the values of the 4 Boolean flags (the above 3
plus innodb_log_file_buffering) will be set as follows:
O_DSYNC:
innodb_log_file_write_through=ON, innodb_data_file_write_through=ON,
innodb_data_file_buffering=OFF, and
(if supported) innodb_log_file_buffering=OFF.
fsync, littlesync, nosync, or (Microsoft Windows specific) normal:
innodb_log_file_write_through=OFF, innodb_data_file_write_through=OFF,
and innodb_data_file_buffering=ON.
Note: fsync() or fdatasync() will only be disabled if the separate
parameter debug_no_sync (in the code, my_disable_sync) is set.
In mariadb-backup, the parameter innodb_flush_method will be ignored.
The Boolean parameters can be modified by SET GLOBAL while the
server is running. This will require reopening the ib_logfile0
or all currently open InnoDB data files.
We will open files straight in O_DSYNC or O_SYNC mode when applicable.
Data files we will try to open straight in O_DIRECT mode when the
page size is at least 4096 bytes. For atomically creating data files,
we will invoke os_file_set_nocache() to enable O_DIRECT afterwards,
because O_DIRECT is not supported on some file systems. We will also
continue to invoke os_file_set_nocache() on ib_logfile0 when
innodb_log_file_buffering=OFF can be fulfilled.
For reopening the ib_logfile0, we use the same logic that was developed
for online log resizing and reused for updates of
innodb_log_file_buffering.
Reopening all data files is implemented in the new function
fil_space_t::reopen_all().
Reviewed by: Vladislav Vaintroub
Tested by: Matthias Leich
In commit c4c8830709 (MDEV-28111) we disabled
the file system cache on the InnoDB write-ahead log file (ib_logfile0)
by default on Linux.
It turns out that especially with innodb_flush_trx_log_at_commit=2,
writing to the log via the file system cache typically improves throughput,
especially on slow storage or at a small number of concurrent transactions.
For other values of innodb_flush_log_at_trx_commit, direct writes were
observed to be mostly but not always faster. Whether it pays off to
disable the file system cache on the log may depend on the type of storage,
the workload, and the operating system kernel version.
On Linux and Microsoft Windows, we will introduce the settable Boolean
global variable innodb_log_file_buffering that indicates whether the
file system cache on the redo log file is enabled. The default value is
innodb_log_file_buffering=OFF. If the server is started up with
innodb_flush_log_at_trx_commit=2, the value will be changed to
innodb_log_file_buffering=ON.
When a persistent memory interface is being used for the log,
the value cannot be changed from innodb_log_file_buffering=OFF.
On Linux, when the physical block size cannot be determined
to be a power of 2 between 64 and 4096 bytes, the file system cache
cannot be disabled, and innodb_log_file_buffering=ON cannot be changed.
Server log messages will indicate whether the file system cache is
enabled for the redo log:
[Note] InnoDB: Buffered log writes (block size=512 bytes)
[Note] InnoDB: File system buffers for log disabled (block size=512 bytes)
After this change, the startup parameter innodb_flush_method will no
longer control whether O_DIRECT will be set on the redo log on Linux.
On other operating systems that support O_DIRECT, no interface has been
implemented for controlling the file system cache for the redo log.
The innodb_flush_method values O_DIRECT, O_DIRECT_NO_FSYNC, O_DSYNC
will enable O_DIRECT for data files, not the log.
Tested by: Matthias Leich, Axel Schwenke
The parameter innodb_prefix_index_cluster_optimization used to enable an
optimization that was added in cb37c55768
and was disabled by default.
We will unconditionally enable the extension and mark the parameter
as deprecated.
Related to this, the counters
Innodb_secondary_index_triggered_cluster_reads and
Innodb_secondary_index_triggered_cluster_reads_avoided
allowed to determine the usefulness of this optimization.
Now that the configuration parameter is disabled, the counters
do not serve any useful purpose and can be removed.
row_search_with_covering_prefix(): Fix a bug that caused an
incorrect result to be returned.
INNODB_VERSION_STR: Replaced with PACKAGE_VERSION (non-functional change).
INNODB_VERSION_SHORT: Replaced with direct use of
MYSQL_VERSION_MAJOR << 8 | MYSQL_VERSION_MINOR.
check_version(): Simplify the mariadb-backup version check,
and require the server version to be MariaDB 10.8 or later,
because that is when the InnoDB redo log format was last changed.
This is a backport of commit 4489a89c71
in order to remove the test innodb.redo_log_during_checkpoint
that would cause trouble in the DBUG subsystem invoked by
safe_mutex_lock() via log_checkpoint(). Before
commit 7cffb5f6e8
these mutexes were of different type.
The following options were introduced in
commit 2e814d4702 (mariadb-10.2.2)
and have little use:
innodb_disable_resize_buffer_pool_debug had no effect even in
MariaDB 10.2.2 or MySQL 5.7.9. It was introduced in
mysql/mysql-server@5c4094cf49
to work around a problem that was fixed in
mysql/mysql-server@2957ae4f99
(but the parameter was not removed).
innodb_page_cleaner_disabled_debug and innodb_master_thread_disabled_debug
are only used by the test innodb.redo_log_during_checkpoint
that will be removed as part of this commit.
innodb_dict_stats_disabled_debug is only used by that test,
and it is redundant because one could simply use
innodb_stats_persistent=OFF or the STATS_PERSISTENT=0 attribute
of the table in the test to achieve the same effect.
We will remove the parameter innodb_disallow_writes because it is badly
designed and implemented. The parameter was never allowed at startup.
It was only internally used by Galera snapshot transfer.
If a user executed
SET GLOBAL innodb_disallow_writes=ON;
the server could hang even on subsequent read operations.
During Galera snapshot transfer, we will block writes
to implement an rsync friendly snapshot, as follows:
sst_flush_tables() will acquire a global lock by executing
FLUSH TABLES WITH READ LOCK, which will block any writes
at the high level.
sst_disable_innodb_writes(), invoked via ha_disable_internal_writes(true),
will suspend or disable InnoDB background tasks or threads that could
initiate writes. As part of this, log_make_checkpoint() will be invoked
to ensure that anything in the InnoDB buf_pool.flush_list will be written
to the data files. This has the nice side effect that the Galera joiner
will avoid crash recovery.
The changes to sql/wsrep.cc and to the tests are based on a prototype
that was developed by Jan Lindström.
Reviewed by: Jan Lindström
We support online log resizing by replicating the current ib_logfile0
to a new file ib_logfile101, which will eventually replace the
ib_logfile0 on the first applicable log checkpoint.
Unless the log is located in a persistent memory file system (PMEM),
an attempt to SET GLOBAL innodb_log_file_size to less than
innodb_log_buffer_size will be refused. (With PMEM, a.k.a. mmap()
based log, that parameter has no meaning.)
Should the server be killed while the log was being resized,
both files ib_logfile0 and ib_logfile101 may exist on startup,
and since commit 3b06415cb8
the extra file ib_logfile101 will be removed.
We will initiate checkpoint flushing by invoking buf_flush_ahead(),
to let buf_flush_page_cleaner() write out pages until the
buf_flush_async_lsn target has been reached.
On a log checkpoint, if the new checkpoint LSN is not older than
log_sys.resize_lsn (the start LSN of the ib_logfile101),
we can switch files and complete the log resizing. Else, we will
attempt to switch files on the next checkpoint.
Log resizing can be aborted by killing the connection that is
executing the SET GLOBAL statement.
If the ib_logfile101 wraps around to the beginning, we must
advance the log_sys.resize_lsn. In the resized log file,
the sequence bit will always be written as 1 (no wrap-around).
The log will be duplicated in log_t::resize_write(), invoked by
mtr_t::finish_write().
When the log is being written via system calls (not PMEM), the initial
log_sys.resize_lsn is the current log_sys.first_lsn, plus an integer
multiple of log_sys.block_size, corresponding to the LSN at the start
of the block that was written by log_sys.write_lsn. The log_sys.resize_buf
will be of the same size as the log_sys.buf. During resizing, the
contents of log_sys.buf and log_sys.resize_buf will be identical,
except that the sequence bit of each mini-transaction will always be 1 in
log_sys.resize_buf. If resizing is in progress, log_t::write_buf()
will write log_sys.resize_buf to log_sys.resize_log (ib_logfile101).
If the file would wrap around, the buffer will be written to
log_sys.START_OFFSET and the log_sys.resize_lsn advanced accordingly.
When using mmap() on /dev/shm or a PMEM mount -o dax file system,
the initial log_sys.resize_lsn will be the log_sys.lsn at the time
the resizing is initiated. If the log file wraps around during resizing,
then the log_sys.resize_lsn will be advanced by
(log_sys.resize_target - log_sys.START_OFFSET).
log_t::resize_start(), log_t::resize_abort(), log_t::write_checkpoint():
Unless the log is mmap() based, acquire flush_lock and write_lock.
In any case, acquire exclusive log_sys.latch to prevent race conditions.
log_t::resize_rename(): Renamed from log_t::rename_resized(),
and moved some code to the previous sole caller srv_start().
Thanks to Vladislav Vaintroub for helpful review comments
and to Matthias Leich for testing this, in particular, testing
crash recovery, multiple concurrent SET GLOBAL innodb_log_file_size
and frequently killed connections.
As a follow-up to MDEV-27734 Set innodb_change_buffering=none by default
we mark the option innodb_change_buffering deprecated, to inform users
of its future removal.
A prominent bottleneck in mtr_t::commit() is log_sys.mutex between
log_sys.append_prepare() and log_close().
User-visible change: The minimum innodb_log_file_size will be
increased from 1MiB to 4MiB so that some conditions can be
trivially satisfied.
log_sys.latch (log_latch): Replaces log_sys.mutex and
log_sys.flush_order_mutex. Copying mtr_t::m_log to
log_sys.buf is protected by a shared log_sys.latch.
Writes from log_sys.buf to the file system will be protected
by an exclusive log_sys.latch.
log_sys.lsn_lock: Protects the allocation of log buffer
in log_sys.append_prepare().
sspin_lock: A simple spin lock, for log_sys.lsn_lock.
Thanks to Vladislav Vaintroub for suggesting this idea, and for
reviewing these changes.
mariadb-backup: Replace some use of log_sys.mutex with recv_sys.mutex.
buf_pool_t::insert_into_flush_list(): Implement sorting of flush_list
because ordering is otherwise no longer guaranteed. Ordering by LSN
is needed for the proper operation of redo log checkpoints.
log_sys.append_prepare(): Advance log_sys.lsn and log_sys.buf_free by
the length, and return the old values. Also increment write_to_buf,
which was previously done in log_close().
mtr_t::finish_write(): Obtain the buffer pointer from
log_sys.append_prepare().
log_sys.buf_free: Make the field Atomic_relaxed,
to simplify log_flush_margin(). Use only loads and stores
to avoid costly read-modify-write atomic operations.
buf_pool.flush_list_requests: Replaces
export_vars.innodb_buffer_pool_write_requests
and srv_stats.buf_pool_write_requests.
Protected by buf_pool.flush_list_mutex.
buf_pool_t::insert_into_flush_list(): Do not invoke page_cleaner_wakeup().
Let the caller do that after a batch of calls.
recv_recover_page(): Invoke a minimal part of
buf_pool.insert_into_flush_list().
ReleaseBlocks::modified: A number of pages added to buf_pool.flush_list.
ReleaseBlocks::operator(): Merge buf_flush_note_modification() here.
log_t::set_capacity(): Renamed from log_set_capacity().
The aim of the InnoDB change buffer is to avoid delays when a leaf page
of a secondary index is not present in the buffer pool, and a record needs
to be inserted, delete-marked, or purged. Instead of reading the page into
the buffer pool for making such a modification, we may insert a record to
the change buffer (a special index tree in the InnoDB system tablespace).
The buffered changes are guaranteed to be merged if the index page
actually needs to be read later.
The change buffer could be useful when the database is stored on a
rotational medium (hard disk) where random seeks are slower than
sequential reads or writes.
Obviously, the change buffer will cause write amplification, due to
potentially large amount of metadata that is being written to the
change buffer. We will have to write redo log records for modifying
the change buffer tree as well as the user tablespace. Furthermore,
in the user tablespace, we must maintain a change buffer bitmap page
that uses 2 bits for estimating the amount of free space in pages,
and 1 bit to specify whether buffered changes exist. This bitmap needs
to be updated on every operation, which could reduce performance.
Even if the change buffer were free of bugs such as MDEV-24449
(potentially causing the corruption of any page in the system tablespace)
or MDEV-26977 (corruption of secondary indexes due to a currently
unknown reason), it will make diagnosis of other data corruption harder.
Because of all this, it is best to disable the change buffer by default.
In commit 9bc874a594 (MDEV-23497)
the configuration option innodb_read_only_compressed was introduced
to giver users advance notice of a plan to remove ROW_FORMAT=COMPRESSED
support for InnoDB.
Based on user feedback, this plan has been scrapped.
Even though ROW_FORMAT=COMPRESSED is a dead end and causes some
overhead for InnoDB data structures, we can live with that.
Now that we know that some users really want to keep using
ROW_FORMAT=COMPRESSED, the previous default value of the parameter
innodb_read_only_compressed=ON should be changed to OFF, to allow
smooth upgrades to 10.6 and later versions, without requiring users
to update any configuration file.
