Updated tests: cases with bugs or which cannot be run
with the cursor-protocol were excluded with
"--disable_cursor_protocol"/"--enable_cursor_protocol"
Fix for v.10.5
In any test that uses wait_all_purged.inc, ensure that InnoDB tables
will be created without persistent statistics.
This is a follow-up to commit cd04673a17
after a similar failure was observed in the innodb_zip.blob test.
The motivation of introducing the parameter
innodb_purge_rseg_truncate_frequency in
mysql/mysql-server@28bbd66ea5 and
mysql/mysql-server@8fc2120fed
seems to have been to avoid stalls due to freeing undo log pages
or truncating undo log tablespaces. In MariaDB Server,
innodb_undo_log_truncate=ON should be a much lighter operation
than in MySQL, because it will not involve any log checkpoint.
Another source of performance stalls should be
trx_purge_truncate_rseg_history(), which is shrinking the history list
by freeing the undo log pages whose undo records have been purged.
To alleviate that, we will introduce a purge_truncation_task that will
offload this from the purge_coordinator_task. In that way, the next
innodb_purge_batch_size pages may be parsed and purged while the pages
from the previous batch are being freed and the history list being shrunk.
The processing of innodb_undo_log_truncate=ON will still remain the
responsibility of the purge_coordinator_task.
purge_coordinator_state::count: Remove. We will ignore
innodb_purge_rseg_truncate_frequency, and act as if it had been
set to 1 (the maximum shrinking frequency).
purge_coordinator_state::do_purge(): Invoke an asynchronous task
purge_truncation_callback() to free the undo log pages.
purge_sys_t::iterator::free_history(): Free those undo log pages
that have been processed. This used to be a part of
trx_purge_truncate_history().
purge_sys_t::clone_end_view(): Take a new value of purge_sys.head
as a parameter, so that it will be updated while holding exclusive
purge_sys.latch. This is needed for race-free access to the field
in purge_truncation_callback().
Reviewed by: Vladislav Lesin
Problem:
=======
InnoDB alter fails before applying instant operation. So rollback
assigns wrong column to the secondary index field. It leads
to the assert failure in the consecutive alter.
Fix:
===
InnoDB shouldn't do rollback of instant operation when it fails
before applying instant operation.
failed in dtuple_convert_big_rec
In dtuple_convert_big_rec(), InnoDB fails to consider the
instant metadata blob while choosing the variable length
field.
ha_innobase::check_if_supported_inplace_alter(): Do not allow
ALGORITHM=INSTANT for operations that avoid a table rebuild
but involve dropping (or creating) secondary indexes.
If a table is altered using the MDEV-11369/MDEV-15562/MDEV-13134
ALGORITHM=INSTANT, it can force the table to use a non-canonical
format:
* A hidden metadata record at the start of the clustered index
is used to store each column's DEFAULT value. This makes it possible
to add new columns that have default values without rebuilding the table.
* Starting with MDEV-15562 in MariaDB Server 10.4, a BLOB in the
hidden metadata record is used to store column mappings. This makes
it possible to drop or reorder columns without rebuilding the table.
This also makes it possible to add columns to any position or drop
columns from any position in the table without rebuilding the table.
If a column is dropped without rebuilding the table, old records
will contain garbage in that column's former position, and new records
will be written with NULL values, empty strings, or dummy values.
This is generally not a problem. However, there may be cases where
users may want to avoid putting a table into this format.
For example, users may want to ensure that future UPDATE operations
after an ADD COLUMN will be performed in-place, to reduce write
amplification. (Instantly added columns are essentially always
variable-length.) Users might also want to avoid bugs similar to
MDEV-19916, or they may want to be able to export tables to
older versions of the server.
We will introduce the option innodb_instant_alter_column_allowed,
with the following values:
* never (0): Do not allow instant add/drop/reorder,
to maintain format compatibility with MariaDB 10.x and MySQL 5.x.
If the table (or partition) is not in the canonical format, then
any ALTER TABLE (even one that does not involve instant column
operations) will force a table rebuild.
* add_last (1, default in 10.3): Store a hidden metadata record that
allows columns to be appended to the table instantly (MDEV-11369).
In 10.4 or later, if the table (or partition) is not in this format,
then any ALTER TABLE (even one that does not involve column changes)
will force a table rebuild.
Starting with 10.4:
* add_drop_reorder (2, default): Like 'add_last', but allow the
metadata record to store a column map, to support instant
add/drop/reorder of columns (MDEV-15562).
commit 08ba388713 of MDEV-12353
introduced an incorrect assumption, which was documented by
the failing assertion.
After instant ADD COLUMN, we can have a null (and in-place) UPDATE
of NULL to NULL. No data needs to be written for such updates.
For ROW_FORMAT=REDUNDANT, we reserve space for the NULL values,
and to be compatible with existing behaviour, we will zerofill
the unused data bytes when updating to NULL value.
Revert part of commit 6cedb671e9
because it turns out to be theoretically impossible to parse a
ROW_FORMAT=COMPACT or ROW_FORMAT=DYNAMIC metadata record where
the variable-length fields in the PRIMARY KEY have been written
as nonempty strings.
btr_cur_instant_init_low(): Accurately parse the metadata record
header for ROW_FORMAT=DYNAMIC and ROW_FORMAT=COMPACT. CHAR columns
used to be unnecessarily written as nonempty strings of bytes.
In the test innodb.instant_alter,4k we would be flagging an error
for too large row size. That error was previously only being reported
if the table was being rebuilt. Thus, this merge is fixing a small
omission in MDEV-11369 (instant ADD COLUMN).
row_upd_build_difference_binary(): Correctly handle the
case where columns (or clustered index fields) have been added
since the 'entry' was originally created. In this case,
the update vector must replace any missing columns with the
default values of the instantly added columns.
The prtype & DATA_LONG_TRUE_VARCHAR flag only plays a role when
converting between InnoDB internal format and the MariaDB SQL layer
row format. Ideally this flag would never have been persisted in the
InnoDB data dictionary.
There were bogus assertion failures when an instant ADD, DROP, or
column reordering was combined with a change of extending a VARCHAR
from less than 256 bytes to more than 255 bytes. Such changes are
allowed starting with MDEV-15563 in MariaDB 10.4.3.
dict_table_t::instant_column(), dict_col_t::same_format(): Ignore
the DATA_LONG_TRUE_VARCHAR flag, because it does not affect the
persistent storage format.
This is follow-up to MDEV-18048: Relax a too strict debug assertion
This assertion should have been relaxed when implementing the first part of
MDEV-15563: instant removal of NOT NULL attribute for ROW_FORMAT=REDUNDANT
tables.
For ROW_FORMAT=REDUNDANT, there is no bitmap of null columns;
the null flags are encoded in the end offset of each field.
We do not really care about the number of fields that can be NULL.
If we instantly change the size of a fixed-length field
and treat it as kind-of variable-length, then we will need
conversions between old column values and new ones.
I tried adding such a conversion to row_build(), but then I
noticed that more conversions would be needed, because
old values still appeared in a freshly rebuilt secondary index,
causing a mismatch when trying to search with the correct
longer value that was converted in my provisional fix to row_build().
So, we will revert the essential part of
MDEV-15563: Instant ROW_FORMAT=REDUNDANT column extension
(commit 22feb179ae), but not
remove any tests.
innobase_build_col_map_add(): Do not assume that old_field->pack_length()
equals to field->pack_length(). Fix submitted by Aleksey Midenkov.
innobase_instant_try(): Assert that the column length of fixed-length
NOT NULL columns is only changing for ROW_FORMAT=REDUNDANT.
The Create_field::charset can contain garbage for columns
that the SQL layer does not consider as being string columns.
InnoDB considers BIT a string column for historical reasons
(and backward compatibility with old persistent InnoDB metadata),
and therefore it checked the charset.
The Field::charset() consistently is my_charset_bin for BIT,
so we can trust that one.
Field_str::is_equal(): Do not allow instant conversions between
BIT (which is stored big-endian) and integer types (which can
be stored big-endian or little-endian, depending on storage engine).
row_sel_field_store_in_mysql_format_func(): Properly extend
narrower integer and DATA_FIXBINARY values to the current format.
DATA_FIXBINARY was incorrectly padded with 0x20 instead of 0.
instant_alter_column_possible(): Add the other MDEV-17459 work-around
condition. The existence of fulltext indexes only prevents instant
DROP COLUMN or changing the order of columns. Other forms of instant
ALTER TABLE are no problem.
Before commit 4e7ee166a9 that merged
the MDEV-18295 fix from 10.3, the work-around of MDEV-17459 in
instant_alter_column_possible() was categorically refusing any
ALGORITHM=INSTANT if any FULLTEXT INDEX was present. After that commit,
a related condition was only present in prepare_inplace_alter_table_dict()
but not in the other callers of instant_alter_column_possible().
dict_table_t::init_instant(): Correctly initialize the length of
variable-length instantly dropped columns.
row_ins_index_entry_set_vals(): For variable-length instantly dropped
columns, write 0 bytes of data. For dropped fixed-length NOT NULL
columns, write the fixed length of NUL bytes as data.
btr_cur_pessimistic_insert(): Do not attempt to convert the
metadata BLOB to external storage if it has already been converted.
It could have been converted by btr_cur_pessimistic_update().
Relax some over-zealous assertions.
dtuple_convert_big_rec(): Assert that the metadata BLOB has
not been converted yet.
dict_table_t::instant_column(): Correctly compute the value of
metadata_changed. The original computation in
commit 003720755f would essentially
invoke memcmp(x,x,y), which can only return 0.
The fix for MDEV-17901 did not cover cases where the AUTO_INCREMENT
column was not dropped, but some other columns before it were.
commit_cache_norebuild(): Revert the MDEV-17901 fix.
dict_index_t::clear_instant_alter(): Update table->persistent_autoinc.
innobase_instant_try(): Only try to update the hidden metadata
record if the number of columns is changing (increasing) or
a metadata BLOB is being added due to permuting or dropping columns
for the first time.
dict_table_t::instant_column(), ha_innobase_inplace_ctx::instant_column():
Return whether the metadata record needs to be updated.
This assertion should have been relaxed when implementing the first part of
MDEV-15563: instant removal of NOT NULL attribute for ROW_FORMAT=REDUNDANT
tables.
For ROW_FORMAT=REDUNDANT, there is no bitmap of null columns;
the null flags are encoded in the end offset of each field.
We do not really care about the number of fields that can be NULL.
Allow instant changes of columns in ROW_FORMAT=REDUNDANT
from NOT NULL to NULL.
Later, this may be implemented for ROW_FORMAT=COMPACT or DYNAMIC,
but in that case any indexes on the table must be rebuilt.
dict_table_t::prepare_instant(): Add some debug assertions,
and relax a debug assertion so that the number of fields is
allowed not to change.
dict_index_t::instant_add_field(): Relax a debug assertion,
allowing a column to change from NOT NULL to NULL.
dict_table_t::instant_column(): Add debug assertions.
instant_alter_column_possible(): Allow ALTER_COLUMN_NULLABLE
when applicable.
innodb_insert_sys_columns(): Add the parameter bool update=false
to run UPDATE instead of INSERT.
innobase_instant_add_col(): Remove; let the only caller invoke
innodb_insert_sys_columns() directly.
innobase_instant_try(): Update the SYS_COLUMNS record if the
column is changed. Only convert the table to the instant ALTER TABLE
format if necessary. For ALTER_COLUMN_NULLABLE in ROW_FORMAT=REDUNDANT,
there is no data format change.
btr_cur_pessimistic_insert(): Convert the metadata field of the metadata
record into BLOB before inserting, just like btr_cur_optimistic_insert()
does.
We failed to reset the dict_table_t::persistent_autoinc after
instantly dropping an AUTO_INCREMENT column, causing a bogus
call to row_parse_int() on a subsequent insert.
dict_index_t::clear_instant_alter(): Remove a loop that became
redundant in commit ae2004c616
and caused corruption of n_nullable when dropping the last column
which was not declared NOT NULL.
dict_index_t::clear_instant_alter(): Correctly move all fields
corresponding to instantly dropped columns to the end of the array.
This fixes a regression that was introduced in
commit 5aaee3746e.
dict_index_t::reconstruct_fields(): Correctly permute the fields
of the columns. The code was totally wrong in MDEV-15562.
It would only work when columns are added last or dropped,
but not when columns are permuted.
