InnoDB does not allow creating multiple FULLTEXT INDEX
in ALGORITHM=INPLACE. This constraint was not being properly
enforced after MariaDB started to support ALGORITHM=INSTANT
and instant ADD COLUMN.
As a side effect of this bug, we again allow ALGORITHM=INPLACE
to rebuild a table when one FULLTEXT INDEX survives.
Also, we are returning a more accurate reason for refusing LOCK=NONE.
innobase_fulltext_exist(): Return the number of fulltext indexes.
ha_innobase::check_if_supported_inplace_alter(): If the table
needs to be rebuilt, refuse the operation if multiple fulltext
indexes would remain.
dict_table_t::init_instant(): Correctly initialize the length of
variable-length instantly dropped columns.
row_ins_index_entry_set_vals(): For variable-length instantly dropped
columns, write 0 bytes of data. For dropped fixed-length NOT NULL
columns, write the fixed length of NUL bytes as data.
dict_table_t::instant_column(): Correctly compute the value of
metadata_changed. The original computation in
commit 003720755f would essentially
invoke memcmp(x,x,y), which can only return 0.
This is a regression after MDEV-13671.
The bug is related to key part prefix lengths wich are stored in SYS_FIELDS.
Storage format is not obvious and was handled incorrectly which led to data
dictionary corruption.
SYS_FIELDS.POS actually contains prefix length too in case if any key part
has prefix length.
innobase_rename_column_try(): fixed prefixes handling
Tests for prefixed indexes added too.
Closes#1063
The fix for MDEV-17901 did not cover cases where the AUTO_INCREMENT
column was not dropped, but some other columns before it were.
commit_cache_norebuild(): Revert the MDEV-17901 fix.
dict_index_t::clear_instant_alter(): Update table->persistent_autoinc.
innobase_instant_try(): Only try to update the hidden metadata
record if the number of columns is changing (increasing) or
a metadata BLOB is being added due to permuting or dropping columns
for the first time.
dict_table_t::instant_column(), ha_innobase_inplace_ctx::instant_column():
Return whether the metadata record needs to be updated.
This assertion should have been relaxed when implementing the first part of
MDEV-15563: instant removal of NOT NULL attribute for ROW_FORMAT=REDUNDANT
tables.
For ROW_FORMAT=REDUNDANT, there is no bitmap of null columns;
the null flags are encoded in the end offset of each field.
We do not really care about the number of fields that can be NULL.
This is basically re-applying 8fe34dd45f.
Assertions about ROW_FORMAT not changing during instant ALTER TABLE can
fail if the MariaDB and InnoDB data dictionaries get out of sync,
for example if innodb_default_row_format is used instead of specifying
ROW_FORMAT in the CREATE TABLE statement. In this case, ALTER_OPTIONS
would not necessarily be set. We would create the ctx->instant_table
with a ROW_FORMAT based on altered_table. When applying it to
the ctx->old_table, we will preserve the original ROW_FORMAT.
dict_table_t::init_instant(): Initialize instant->non_pk_col_map[].
Refactored from dict_table_t::instant_column().
Also, assert that the n_nullable for the clustered index is initialized
correctly.
Allow instant changes of columns in ROW_FORMAT=REDUNDANT
from NOT NULL to NULL.
Later, this may be implemented for ROW_FORMAT=COMPACT or DYNAMIC,
but in that case any indexes on the table must be rebuilt.
dict_table_t::prepare_instant(): Add some debug assertions,
and relax a debug assertion so that the number of fields is
allowed not to change.
dict_index_t::instant_add_field(): Relax a debug assertion,
allowing a column to change from NOT NULL to NULL.
dict_table_t::instant_column(): Add debug assertions.
instant_alter_column_possible(): Allow ALTER_COLUMN_NULLABLE
when applicable.
innodb_insert_sys_columns(): Add the parameter bool update=false
to run UPDATE instead of INSERT.
innobase_instant_add_col(): Remove; let the only caller invoke
innodb_insert_sys_columns() directly.
innobase_instant_try(): Update the SYS_COLUMNS record if the
column is changed. Only convert the table to the instant ALTER TABLE
format if necessary. For ALTER_COLUMN_NULLABLE in ROW_FORMAT=REDUNDANT,
there is no data format change.
ha_innobase::prepare_inplace_alter_table(): check max column length for every
index in a table, not just added in this particular ALTER TABLE with ADD INDEX ones.
We failed to reset the dict_table_t::persistent_autoinc after
instantly dropping an AUTO_INCREMENT column, causing a bogus
call to row_parse_int() on a subsequent insert.
The fix of MDEV-17793 was updating SYS_INDEXES.TABLE_ID in order
to make the table invisible to purge (lazily delete old undo log
records).
By design of InnoDB, an update of TABLE_ID cannot be rolled back,
because the rollback would effectively drop all indexes of the table
due to the internal 'trigger' on SYS_INDEXES modifications.
So, we revert the code change of MDEV-17793 and instead fix
MDEV-17793 in a different way: by tweaking the undo log parsing
during purge.
The MDEV-17793 bug scenario is that a table becomes empty and
a third instant ALTER TABLE is executed before purge processes
the undo log record for the second instant ALTER TABLE. After
this point, when purge sees the record, the table could have
a mismatching number of rows.
The test case works with this alternative fix. But what about
a scenario where a fourth instant ALTER TABLE arrives before
purge processes the second one? Could anything bad happen?
Purge is only doing two things: First, free any BLOBs that
were affected by the update record, and then, reset the
DB_TRX_ID,DB_ROLL_PTR if a matching record is found.
For the hidden metadata record, the only BLOB that we update
is the hidden metadata BLOB that was introduced by MDEV-15562.
Any other BLOBs (for the initial default values of instantly
added columns) are never updated.
So, in our scenario, the metadata BLOB that was created by
the first instant ALTER TABLE (if it involved dropping or
permuting columns) would be freed by purge when it is processing
the undo record of the second ALTER TABLE. The BLOB value that
was written by the second ALTER TABLE should be freed when
the table is emptied. This is currently not done: MDEV-17383
should fix that. There is no possibility of double-free, because
purge would only free old values of BLOBs.
What about MVCC and other callers of trx_undo_update_rec_get_update()?
The answer is simple: they should never be accessing the hidden
metadata record in the first place.
dict_table_t::reassign_id(): Remove.
btr_cur_pessimistic_delete(): Clarify a comment.
row_mysql_table_id_reassign(), row_discard_tablespace_for_mysql():
Add comments explaining that the operation cannot be rolled back.
trx_undo_update_rec_get_update(): Avoid out-of-bounds access when
parsing a metadata record. Avoid unnecessary memory allocation when
filtering out fields from the update vector.
row_undo_ins_remove_clust_rec(): When rolling back the insert
of a metadata record, also reset the root page state within the
same mini-transaction.
dict_table_t::rollback_instant(): Do not allow index->n_core_fields
to be larger than index->n_fields. This logic exists in the 10.3
branch but was incorrectly removed as part of MDEV-15562 in 10.4.0.
This regression was introduced in MDEV-11369 instant ADD COLUMN.
prepare_inplace_alter_table_dict(): Avoid dereferencing ctx->new_table
after row_create_table_for_mysql() returns a failure code, because
it will have freed the memory.
dict_table_t::prepare_instant(): Remove too strict assertions that
could fail if the MariaDB and InnoDB data dictionaries are out of sync.
That the dictionaries got out of sync during TRUNCATE TABLE
is a separate bug was fixed in 10.2 by
commit 971e1d8677.
Several race conditions between MDEV-15562 instant ALTER TABLE and purge
were observed.
The most obvious race condition resulted in a reported assertion failure in
dict_index_t::instant_add_field(): instant.n_core_fields == n_core_fields
would not hold if the table was emptied by purge after the time
dict_table_t::prepare_instant() was called.
During purge, it can turn out that the table is logically empty, only
containing a metadata record. If the metadata record is of the type
created by MDEV-11369 instant ADD COLUMN, it can be removed and
dict_index_t::clear_instant_add() can be called. This will convert the
table to the canonical non-instant format. (If the metadata record is
of the MDEV-15562 type, then it can only be deleted if the table becomes
empty as the result of rollback of an instant ALTER TABLE operation.)
row_purge_remove_clust_if_poss_low(): Add a debug check that ensures
that purge can never remove a MDEV-15562 metadata record.
ha_innobase::open(): Add a comment about the necessity of rolling
back any recovered instant ALTER TABLE transaction on the table.
instant_metadata_lock(): An auxiliary function to acquire a page latch
on the metadata record, to prevent race conditions.
dict_table_t::prepare_instant(), dict_index_t::instant_add_field(),
dict_table_t::rollback_instant(), innobase_instant_try():
Invoke instant_metadata_lock() in order to prevent race conditions.
dict_index_t::instant_add_field(): Correct debug assertions.
The == was guaranteed by code in dict_table_t::prepare_instant()
that was introduced in MDEV-15562. Due to the race condition,
we could occasionally have <=, but never >= like the code was
after MDEV-11369.
ha_innobase_inplace_ctx::instant_column(): Wrapper for
dict_table_t::instant_column(). Add debug assertions.
InnoDB in MySQL 5.7 introduced two new parameters to the function
dict_hdr_get_new_id(), to allow redo logging to be disabled when
assigning identifiers to temporary tables or during the
backup-unfriendly TRUNCATE TABLE that was replaced in MariaDB
by MDEV-13564.
Now that MariaDB 10.4.0 removed the crash recovery code for the
backup-unfriendly TRUNCATE, we can revert dict_hdr_get_new_id()
to be used only for persistent data structures.
dict_table_assign_new_id(): Remove. This was a simple 2-line function
that was called from few places.
dict_table_open_on_id_low(): Declare in the only file where it
is called.
dict_sys_t::temp_id_hash: A separate lookup table for temporary tables.
Table names will be in the common dict_sys_t::table_hash.
dict_sys_t::get_temporary_table_id(): Assign a temporary table ID.
dict_sys_t::get_table(): Look up a persistent table.
dict_sys_t::get_temporary_table(): Look up a temporary table.
dict_sys_t::temp_table_id: The sequence of temporary table identifiers.
Starts from DICT_HDR_FIRST_ID, so that we can continue to simply compare
dict_table_t::id to a few constants for the persistent hard-coded
data dictionary tables.
undo_node_t::state: Distinguish temporary and persistent tables.
lock_check_dict_lock(), lock_get_table_id(): Assert that there cannot
be locks on temporary tables.
row_rec_to_index_entry_impl(): Assert that there cannot be metadata
records on temporary tables.
row_undo_ins_parse_undo_rec(): Distinguish temporary and persistent tables.
Move some assertions from the only caller. Return whether the table was
found.
row_undo_ins(): Add some assertions.
row_undo_mod_clust(), row_undo_mod(): Do not assign node->state.
Let row_undo() do that.
row_undo_mod_parse_undo_rec(): Distinguish temporary and persistent tables.
Move some assertions from the only caller. Return whether the table was
found.
row_undo_try_truncate(): Renamed and simplified from trx_roll_try_truncate().
row_undo_rec_get(): Replaces trx_roll_pop_top_rec_of_trx() and
trx_roll_pop_top_rec(). Fetch an undo log record, and assign undo->state
accordingly.
trx_undo_truncate_end(): Acquire the rseg->mutex only for the minimum
required duration, and release it between mini-transactions.
There was a race condition between ALTER TABLE and purge.
If a table turns out to be logically empty when instant ALTER TABLE
is executing, we will convert the table to the canonical format,
to avoid overhead during subsequent accesses, and to allow the
data file to be imported into older versions of MariaDB.
It could happen that at the time the table is logically empty,
there still exists an undo log record for updating the hidden
metadata record for an earlier instant ALTER TABLE operation.
If the table was converted to the canonical format before
purge processes this undo log record, the undo log record
could be referring to index fields that no longer exist,
causing a crash.
To prevent the race condition, we must delete the old undo log records.
We do this lazily by assigning a new table ID, so that the table lookup
for the old undo log records will fail.
dict_table_t::reassign_id(): Reassign the table_id to
effectively lazily delete old undo log records.
innobase_instant_try(): Invoke index->table->reassign_id() before
index->clear_instant_alter().
