- Change thd->backup_commit_lock from MDL_request* to MDL_ticket*,
and convert related call sites in handler.cc, log.cc, sql_class.cc,
and xa.cc to use MDL_ACQUIRE_LOCK.
- Convert reload_acl_and_cache() in sql_reload.cc, purge_master_logs()
and reset_master() in sql_repl.cc to MDL_ACQUIRE_LOCK, holding the
ticket in a local MDL_ticket* (these functions used a local
MDL_request originally and should not touch thd->backup_commit_lock).
- Convert acquire_lock() in partition_info.cc to MDL_ACQUIRE_LOCK.
- Add MDL_REQUEST_LIST_ADD() helper for enqueuing lock requests into
MDL_request_list, and convert call sites in sql_base.cc and sp.cc.
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.
Added ADAPTIVE_HASH_INDEX=DEFAULT|YES|NO table and index option to InnoDB.
The table and index options only have an effect if InnoDB adaptive hash
index feature is enabled.
- Having the ADAPTIVE_HASH_INDEX TABLE option set to NO will disable
adaptive hash index for all indexes in the table that do not have
the index option adaptive_hash_index=yes.
- Having the ADAPTIVE_HASH_INDEX TABLE option set to YES will enable the
adaptive hash index for all indexes in the table that do not have
the index option adaptive_hash_index=no.
- Using adaptive_hash_index=default deletes the old setting.
- One can also use OFF/ON as the options. This is to make it work similar
as other existing options.
- innodb.adaptive_hash_index has been changed from a bool to an enum with
values OFF, ON and IF_SPECIFIED. If IF_SPECIFIED is used, adaptive
hash index is only used for tables and indexes that specify
adaptive_hash_index=on.
- The following new options can be used to further optimize adaptive hash
index for an index (default is unset/auto for all of them):
- complete_fields:
- 0 to the number of columns the key is defined on (max 64)
- bytes_from_incomplete_field:
- This is only usable for memcmp()-comparable index fields, such as
VARBINARY or INT. For example, a 3-byte prefix on an INT will
return an identical hash value for 0‥255, another one for 256‥511,
and so on.
- Range is min 0 max 16383.
- for_equal_hash_point_to_last_record:
- Default is unset/auto, NO points to the first record, known as
left_side in the code; YES points to the last record.
Example: we have an INT column with the values 1,4,10 and bytes=3,
will that hash value point to the record 1 or the record 10?
Note: all values will necessarily have the same hash value
computed on the big-endian byte prefix 0x800000, for all of the
values 0x80000001, 0x80000004, 0x8000000a. InnoDB inverts the
sign bit in order to have memcmp()-compatible comparison.
Example:
CREATE TABLE t1 (a int primary key, b varchar(100), c int,
index (b) adaptive_hash_index=no, index (c))
engine=innodb, adaptive_hash_index=yes;
Notable changes in InnoDB
- btr_search.enabled was changed from a bool to a ulong enum to be
able to handle options OFF, ON and IF_SPECIFIED. ulong is needed
to compile with MariaDB enum variables.
- To be able to find all instances where btr_search.enabled was used
I changed all code to use btr_search.get_enabled() when accessing
the value and used btr_search.is_enabled(index) to test if AHI is
enabled for the index.
- btr_search.enable() was changed to always take two parameters,
resize and value of enabled. This was needed as enabled can now
have values OFF (0), ON (1), and IF_SPECIFIED (2).
- store all AHI-related options in per-index dict_index_t::ahi
bit-packed 32-bit atomic field dict_index_t::ahi::enabled_fixed_mask
- static assertions and debug assertions ensure that all options fit
into the 32-bit field
- packing details:
- enabled, adaptive_hash_index (first 2 bits)
- fields, complete_fields (7 bits)
- bytes, bytes_from_incomplete_field (14 bits)
- left, ~for_equal_hash_point_to_last_record (1 bit)
- have_fields, fields set flag (1 bit)
- have_bytes, bytes set flag (1 bit)
- have_left, left set flag (last 1 bit)
- 5 bits spare after have_left
- manipulation of the bit-packed field avoids usage of branches or
conditional instructions to minimize the performance impact of
the new options
- in btr_search_update_hash_ref() apply the per-index AHI options
using bit-masking to override internal heuristic values with user
preferences
- add innodb.index_ahi_option test:
- test a combination of per-table and per-index AHI options
- use a stored procedure which checks if AHI is used during a burst of
index lookups checking delta in adaptive_hash_searches InnoDB
monitor variable
- test that the maximum number of fields per (secondary) index is 64
(32 PK + 32 secondary key columns)
- test that crash recovery retains table/index metadata
- test AHI with partitioned tables
- verify that changing AHI preferences for table/index is instant DDL
- boundary checks for complete_fields and bytes_from_incomplete_field
- add innodb.index_ahi_option_debug test:
- test debug builds with index_ahi_option_debug_check debug variable
enabled to verify that the proper per-index AHI options are applied
during index lookups
- test that "illegal" per-index AHI options are non-destructive and
just lead to automatic parameter clamping and possibly no AHI usage
- test toggling global AHI status with concurrent AHI access
Visible user changes:
- select @@global.adaptive_hash_index will now return a string instead
of 0 or 1.
Other notable changes:
- In sql/create_options.cc:
- In function set_one_value() avoid reading after the end of the
current string.
Co-authored-by: Monty <monty@mariadb.org>
Co-authored-by: Marko Mäkelä <marko.makela@mariadb.com>
Co-authored-by: Alessandro Vetere <iminelink@gmail.com>
Co-authored-by: Thirunarayanan Balathandayuthapani <thiru@mariadb.com>
Add new acquire_lock() overloads on MDL_context that combine MDL_request
initialization and lock acquisition into a single call, returning
MDL_ticket* directly. Convert applicable call sites across the codebase
to use the new ACQUIRE_LOCK macro.
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.
When ALTER TABLE uses the copy algorithm, ha_create_table() is
invoked to build the shadow table. A failure there was previously
reported as "Can't create table", which is misleading because the
user never issued a CREATE.
Fix (per review by @midenok): keep errno 1005 / SQLSTATE HY000
unchanged and only swap the message text when HA_CREATE_TMP_ALTER
is set, via a new translatable template ER_CANT_CREATE_TMP_ALTER_TABLE
in errmsg-utf8.txt (no SQLSTATE clause -- it is a message template
rather than a wire error code).
The user-visible error becomes:
ERROR 1005 (HY000): Can't create temporary table for ALTER TABLE \
`db`.`tbl` (errno: N "...")
No existing error handler, stored procedure, tool, or application
that matches on errno 1005 or SQLSTATE HY000 is affected.
Notes on the 12.3 rebase:
* In 12.3 the ER_CANT_CREATE_TABLE reporting logic has been moved
from ha_create_table() into its new helper ha_create_table_from_share().
The HA_CREATE_TMP_ALTER branch is added in the helper so that both
the main shadow table and any high-level (vector) index sub-tables
go through the same path. Vector index sub-tables use a zero-
initialised index_cinfo, so HA_CREATE_TMP_ALTER is never set for
them and they still get the plain "Can't create table" message.
Review follow-ups (Round 6, @midenok):
* Moved the test content from suite/innodb/t/alter_table_error_message.test
into main/alter_table.test (no duplicate --include directives);
dropped the extra .test/.result files accordingly.
* Added the standard "End of 12.3 tests" section marker at the end.
* Removed the unnecessary DROP TABLE IF EXISTS cluster_t; in the
cleanup step (the test is deterministic, the table no longer
exists at that point).
Review follow-ups (Round 7, test re-record + errmsg formatting):
* Fixed ER_CANT_CREATE_TMP_ALTER_TABLE template to use the exact
ER_CANT_CREATE_TABLE format specifiers, so error frames produced
by my_error() are byte-identical to the existing error (only the
fixed prefix changes).
* Re-recorded innodb.innodb-index.result and innodb.foreign_key.result
which now observe the new "Can't create temporary table for ALTER
TABLE" message in ALTER TABLE error paths that used to print
"Can't create table".
* No code changes outside of the intentional message swap; all other
.result files remain untouched.
Review follow-ups (Round 8, additional test re-record from CI):
* Re-recorded innodb.table_flags.result for the MDEV-38079 scenario
(ALTER TABLE t1 PAGE_COMPRESSED=1, ALGORITHM=COPY under
INNODB_DEFAULT_ROW_FORMAT=REDUNDANT) -- this ALTER path now reports
the new "Can't create temporary table for ALTER TABLE" message.
Verified on all 10 page size / checksum combinations (4k/8k/16k/
32k/64k x strict_crc32/strict_full_crc32).
* Re-recorded parts.engine_defined_part_attributes.result for the
ALTER TABLE `t4` PARTITION BY RANGE(...) ENCRYPTED path. The four
remaining "Can't create table" occurrences in the same file refer
to pure CREATE TABLE failures and correctly retain the original
message.
* No code or test-case logic changes; only .result re-recordings.
Signed-off-by: a18792721831 <735809899@qq.com>
CLA: a18792721831 <735809899@qq.com>
Reason:
=======
Following the changes in MDEV-38734, the server no longer marks
all indexed virtual columns during an UPDATE operation.
Consequently, ha_innobase::update() populates the upd_t vector
with old_vrow but omits the actual data for these virtual columns.
Despite this omission, trx_undo_page_report_modify() continues to
write metadata for indexed virtual columns into the undo log. Because
the actual values are missing from the update vector, the undo log
entry is recorded without the historical data for these columns.
When the purge thread processes the undo log to reconstruct a
previous record state for MVCC, it identifies an indexed virtual
column but finds no associated data.
The purge thread incorrectly interprets this missing data as a NULL
value, rather than a "missing/unrecorded" value. The historical
record is reconstructed with an incorrect NULL for the virtual column.
This causes the purge thread to incorrectly identify and purge
records that are not actually delete-marked, leading to abort
of server.
Solution:
=========
ha_innobase::column_bitmaps_signal(): Revert the column-marking
logic to the state prior to commit a4e4a56720, ensuring all
indexed virtual columns are unconditionally marked during an UPDATE.
The previous "optimization" attempted to manually detect indexed
column changes before marking virtual columns. The manual check
for indexed column modifications is redundant. InnoDB already
provides the UPD_NODE_NO_ORD_CHANGE flag within row_upd_step().
This flag is being used in trx_undo_page_report_modify() and
trx_undo_read_undo_rec() should log or read virtual column values.
Refactored column_bitmaps_signal() to accept a mark_for_update
parameter that controls when indexed virtual columns are marked.
TABLE::mark_columns_needed_for_update() is the only place that needs
mark_for_update=true because only UPDATE operations need to mark
indexed virtual columns for InnoDB's undo logging mechanism.
INSERT operation is already handled by
TABLE::mark_virtual_columns_for_write(insert_fl=true).
Even the commit a4e4a56720 changes are
going to affect TABLE::mark_virtual_column_for_write(false) and
It is called during UPDATE operation, and that's when
column_bitmaps_signal() needs to mark indexed virtual columns.
Online DDL has separate code path which is handled by
row_log_mark_virtual_cols() for all DML operations
The method Type_handler::Column_definition_fix_attributes() was an
addition for Type_handler::Column_definition_set_attributes().
The code in virtual implementations of Column_definition_fix_attributes()
has now been joined to corresponding virtual implementations of
Column_definition_set_attributes().
After this change everything is done in Column_definition_set_attributes().
This makes the code clearer. Previous implementation was rather confusing.
For example, for temporal data types:
(1) After Column_definition_set_attributes():
- Column_definition::length meant the fractional precision
- Column_definition::decimals was set to 0
(2) After Column_definition_fix_attributes():
- Column_definition::length means the full character length
- Column_definition::decimals means the fractional precision
Now everything gets set to (2) right in Column_definition_set_attributes().
Problem:
=======
InnoDB DDL does ALTER TABLE PAGE_COMPRESSED=1 because:
1. check_if_supported_inplace_alter() reads srv_file_per_table
during precheck and does allow INSTANT operation.
2. User does change innodb_file_per_table later.
But prepare phase read srv_file_per_table again during execution.
3. If the global variable changed between these phases, causes
server abort for page_compressed tables.
Solution:
=========
- Add a file_per_table member to Alter_inplace_info to
capture the innodb_file_per_table value during the
precheck phase and use that consistent value throughout
the entire ALTER TABLE operation.
- Converted boolean flags (online, ignore, error_if_not_empty,
mdl_exclusive_after_prepare) from individual boolean
members to bitfields packed in a single byte
- Reduced Alter_inplace_info structure size by 24 bytes
remove copying of part_elem->option_struct to table->s->option_struct.
It meant that most of the time a handler had no access to its options
(that's why some handlers resorted to a complex process of guessing the
correct partition) and couldn't work concurrenty. Instead
* introduce handler::option_struct which will point to
table->s->option_struct or part_elem->option_struct as appropriate,
the handler can use it any time as needed. A handler should not use
table->s->option_struct unless it really knows what it's doing and has
verified that it doesn't break partitioning
* rename option_struct in table->s to option_struct_table and in
part_elem to option_struct_part to emphasize it's semantics and
prevent incorrect usage.
* fix engines accordingly.
Implement an improved binlog implementation that is integrated into
the storage engine. The new implementation is enabled with the
--binlog-storage-engine option. Initially the InnoDB storage engine
implements the binlog.
Integrating the binlog in the storage engine improves performance,
since it makes the InnoDB redo log the single source of truth and
avoids the need for expensive two-phase commit between binlog and
engine. It also makes it possible to disable durability (set
--innodb-flush-log-at-trx-commit=0) to further improve performance,
while still preserving the ability to recover the binlog and database
into a consistent state after a crash.
The new binlog implementation also greatly improves the internal
design and implementation of the binlog, and enables future
enhancements for replication.
This is a squash of the original 11.4-based patch series.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
...just like the when deleting the .frm file in the same function.
This improves consistency: internal duplicate deletion of .par file
and .frm file both fail silently.
Two transactions could binlog their completions in opposite to how it
is done in Engine. That is is rare situations ROLLBACK in Engine of
the dependency parent transaction could be scheduled by the
transaction before its binlogging. That give a follower dependency
child one get binlogged ahead of the parent.
For fixing this bug its necessary to ensure the binlogging phase is
always first one in the internal one-phase rollback protocol.
The commit combines
1. a code polishing piece over a part of MDEV-21117 that
made binlog handlerton always commit first in no-2pc cases and
2. the same rule now applies to the rollback.
An added test demonstrates how the child could otherwise reach binlog
before its parent.
Two transactions could binlog their completions in opposite to how it
is done in Engine. That is is rare situations ROLLBACK in Engine of
the dependency parent transaction could be scheduled by the
transaction before its binlogging. That give a follower dependency
child one get binlogged ahead of the parent.
For fixing this bug its necessary to ensure the binlogging phase is
always first one in the internal one-phase rollback protocol.
The commit makes sure the binlog handlerton always rollbacks as first
handlerton in no-2pc cases.
An added test demonstrates how the child could otherwise reach binlog
before its parent.
@@new_mode is a set of flags to control introduced features.
Flags are by default off. Setting a flag in @@new_mode will introduce
a new different server behaviour and/or set of features.
We also introduce a new option 'hidden_values' into some system variable
types to hide options that we do not wish to show as options.
- Don't print hidden values in mysqld --help output.
- Make get_options() use the same logic as check_new_mode_value() does.
- Setting @@new_mode=ALL shouldn't give warnings.
(cherry picked from commit f7387cb13d)
In the BASE of this patch when a slave parallel worker proceeds from
the wait-for-prior-commit stage into retrying it may have its
backup-lock related sub-state, specifically `THD::backup_commit_lock`,
not reset, that is the pointer dangling.
That caused segfault at the pointer's dereferencing in the worker retrying.
The reason THD::backup_commit_lock is left dangling was unexpected
state of THD having non-NULL of `THD::backup_commit_lock` and NULL of `mdl_backup.ticket`.
This combination turns out possible when the slave worker is killed
for retry *and* few instruction later it does not succeed
to (re-)acquire the Backup MDL at exiting from
`MYSQL_BIN_LOG::queue_for_group_commit()`.
While it did not succeed it also did not expose that fact with
timing out from `MDL_context::acquire_lock` and it did not have to,
as it before to start waiting it found itself killed.
The bug is fixed with amendment of `backup_commit_lock` reset condition
at the end of `ha_commit_trans()`. The amended reset remains careful to
affect only the stack allocated lock.
A test is added to confirm the fixes with reproducing all stages
described above. In the patch BASE it causes the segfault.
@@new_mode is a set of flags to control introduced features.
Flags are by default off. Setting a flag in @@new_mode will introduce
a new different server behaviour and/or set of features.
We also introduce a new option 'hidden_values' into some system variable
types to hide options that we do not wish to show as options.
- Don't print hidden values in mysqld --help output.
- Make get_options() use the same logic as check_new_mode_value() does.
- Setting @@new_mode=ALL shouldn't give warnings.
We had a protection against it, by allowing versioned delete if:
trx->id != table->vers_start_id()
For replace this check fails: replace calls ha_delete_row(record[2]), but
table->vers_start_id() returns the value from record[0], which is irrelevant.
The same problem hits Field::is_max, which may have checked the wrong record.
Fix:
* Refactor Field::is_max to optionally accept a pointer as an argument.
* Refactor vers_start_id and vers_end_id to always accept a pointer to the
record. there is a difference with is_max is that is_max accepts the pointer to
the
field data, rather than to the record.
Method val_int() would be too effortful to refactor to accept the argument, so
instead the value in record is fetched directly, like it is done in
Field_longlong.
See also MDEV-30046.
Idempotent write_row works same as REPLACE: if there is a duplicating
record in the table, then it will be deleted and re-inserted, with the
same update optimization.
The code in Rows:log_event::write_row was basically copy-pasted from
write_record.
What's done:
REPLACE operation was unified across replication and sql. It is now
representred as a Write_record class, that holds the whole state, and allows
re-using some resources in between the row writes.
Replace, IODKU and single insert implementations are split across different
methods, reluting in a much cleaner code.
The entry point is preserved as a single Write_record::write_record() call.
The implementation to call is chosen on the constructor stage.
This allowed several optimizations to be done:
1. The table key list is not iterated for every row. We find last unique key in
the order of checking once and preserve it across the rows. See last_uniq_key().
2. ib_handler::referenced_by_foreign_key acquires a global lock. This call was
done per row as well. Not all the table config that allows optimized replace is
folded into a single boolean field can_optimize. All the fields to check are
even stored in a single register on a 64-bit platform.
3. DUP_REPLACE and DUP_UPDATE cases now have one less level of indirection
4. modified_non_trans_tables is checked and set only when it's really needed.
5. Obsolete bitmap manipulations are removed.
Also:
* Unify replace initialization step across implementations:
add prepare_for_replace and finalize_replace
* alloca is removed in favor of mem_root allocation. This memory is reused
across the rows.
* An rpl-related callback is added to the replace branch, meaning that an extra
check is made per row replace even for the common case. It can be avoided with
templates if considered a problem.
