1
0
mirror of https://github.com/MariaDB/server.git synced 2025-11-27 05:41:41 +03:00
Commit Graph

53 Commits

Author SHA1 Message Date
Thirunarayanan Balathandayuthapani
4cd92143ea MDEV-25630 Rollback of instant operation adds wrong column to secondary index
Problem:
=======
  InnoDB alter fails before applying instant operation. So rollback
assigns wrong column to the secondary index field. It leads
to the assert failure in the consecutive alter.

Fix:
===
  InnoDB shouldn't do rollback of instant operation when it fails
before applying instant operation.
2021-04-28 03:56:24 +05:30
Thirunarayanan Balathandayuthapani
8f4a3bf07c MDEV-25057 Assertion `n_fields < dtuple_get_n_fields(entry)'
failed in dtuple_convert_big_rec

In dtuple_convert_big_rec(), InnoDB fails to consider the
instant metadata blob while choosing the variable length
field.
2021-03-09 19:37:27 +05:30
Marko Mäkelä
474290540a MDEV-22465: DROP indexed COLUMN is wrongly claimed to be ALGORITHM=INSTANT
ha_innobase::check_if_supported_inplace_alter(): Do not allow
ALGORITHM=INSTANT for operations that avoid a table rebuild
but involve dropping (or creating) secondary indexes.
2020-05-05 20:32:14 +03:00
Marko Mäkelä
e2f1f88fa6 Merge 10.3 into 10.4 2020-03-30 14:50:23 +03:00
Marko Mäkelä
b092d35f13 MDEV-20590 Introduce a file format constraint to ALTER TABLE
If a table is altered using the MDEV-11369/MDEV-15562/MDEV-13134
ALGORITHM=INSTANT, it can force the table to use a non-canonical
format:

* A hidden metadata record at the start of the clustered index
is used to store each column's DEFAULT value. This makes it possible
to add new columns that have default values without rebuilding the table.

* Starting with MDEV-15562 in MariaDB Server 10.4, a BLOB in the
hidden metadata record is used to store column mappings. This makes
it possible to drop or reorder columns without rebuilding the table.
This also makes it possible to add columns to any position or drop
columns from any position in the table without rebuilding the table.

If a column is dropped without rebuilding the table, old records
will contain garbage in that column's former position, and new records
will be written with NULL values, empty strings, or dummy values.

This is generally not a problem. However, there may be cases where
users may want to avoid putting a table into this format.
For example, users may want to ensure that future UPDATE operations
after an ADD COLUMN will be performed in-place, to reduce write
amplification. (Instantly added columns are essentially always
variable-length.) Users might also want to avoid bugs similar to
MDEV-19916, or they may want to be able to export tables to
older versions of the server.

We will introduce the option innodb_instant_alter_column_allowed,
with the following values:

* never (0): Do not allow instant add/drop/reorder,
to maintain format compatibility with MariaDB 10.x and MySQL 5.x.
If the table (or partition) is not in the canonical format, then
any ALTER TABLE (even one that does not involve instant column
operations) will force a table rebuild.

* add_last (1, default in 10.3): Store a hidden metadata record that
allows columns to be appended to the table instantly (MDEV-11369).
In 10.4 or later, if the table (or partition) is not in this format,
then any ALTER TABLE (even one that does not involve column changes)
will force a table rebuild.

Starting with 10.4:

* add_drop_reorder (2, default): Like 'add_last', but allow the
metadata record to store a column map, to support instant
add/drop/reorder of columns (MDEV-15562).
2020-03-30 12:41:59 +03:00
Marko Mäkelä
6a8a4c19e2 MDEV-21485 ASAN use-after-poison in dfield_get_len or Assertion `pos < index->n_def' failed
The server would crash when instantly reordering the columns of a
table whose all columns belong to the PRIMARY KEY.
2020-01-15 18:28:52 +02:00
Oleksandr Byelkin
a15234bf4b Merge branch '10.3' into 10.4 2019-12-09 15:09:41 +01:00
Marko Mäkelä
c4ed1bee5b MDEV-21172 Memory leak after failed ADD PRIMARY KEY 2019-12-05 08:54:14 +01:00
Marko Mäkelä
3eda03d0fe MDEV-21148: Assertion index->n_core_fields + n_add >= index->n_fields
Revert part of commit 6cedb671e9
because it turns out to be theoretically impossible to parse a
ROW_FORMAT=COMPACT or ROW_FORMAT=DYNAMIC metadata record where
the variable-length fields in the PRIMARY KEY have been written
as nonempty strings.
2019-11-26 20:46:25 +02:00
Marko Mäkelä
6cedb671e9 MDEV-21088 Table cannot be loaded after instant ADD/DROP COLUMN
btr_cur_instant_init_low(): Accurately parse the metadata record
header for ROW_FORMAT=DYNAMIC and ROW_FORMAT=COMPACT. CHAR columns
used to be unnecessarily written as nonempty strings of bytes.
2019-11-20 14:12:53 +08:00
Marko Mäkelä
746ee78535 MDEV-20949: Merge 10.3 into 10.4 2019-11-14 13:22:29 +02:00
Marko Mäkelä
4ded5fb9ac MDEV-20949: Merge 10.2 into 10.3
In the test innodb.instant_alter,4k we would be flagging an error
for too large row size. That error was previously only being reported
if the table was being rebuilt. Thus, this merge is fixing a small
omission in MDEV-11369 (instant ADD COLUMN).
2019-11-14 11:26:49 +02:00
Marko Mäkelä
db4a27ab73 Merge 10.3 into 10.4 2019-08-31 06:53:45 +03:00
Marko Mäkelä
17336f6d30 MDEV-20066 Wrong value on instantly added column after DELETE and UPDATE
row_upd_build_difference_binary(): Correctly handle the
case where columns (or clustered index fields) have been added
since the 'entry' was originally created. In this case,
the update vector must replace any missing columns with the
default values of the instantly added columns.
2019-08-30 14:37:50 +03:00
Marko Mäkelä
e9c1701e11 Merge 10.3 into 10.4 2019-07-25 18:42:06 +03:00
Eugene Kosov
0f83c8878d Merge 10.2 into 10.3 2019-07-16 18:39:21 +03:00
Eugene Kosov
a0230bc76d MDEV-18266 Changing an index comment unnecessarily rebuilds index
ALTER_CHANGE_INDEX_COMMENT: new handler flag added

Compare_keys::EqualButComment: new outcome of compare_keys_but_name()
2019-07-10 21:36:29 +03:00
Marko Mäkelä
bb970dda77 MDEV-18719 Assertion (c.prtype ^ o->prtype) & ... failed on ALTER TABLE
The prtype & DATA_LONG_TRUE_VARCHAR flag only plays a role when
converting between InnoDB internal format and the MariaDB SQL layer
row format. Ideally this flag would never have been persisted in the
InnoDB data dictionary.

There were bogus assertion failures when an instant ADD, DROP, or
column reordering was combined with a change of extending a VARCHAR
from less than 256 bytes to more than 255 bytes. Such changes are
allowed starting with MDEV-15563 in MariaDB 10.4.3.

dict_table_t::instant_column(), dict_col_t::same_format(): Ignore
the DATA_LONG_TRUE_VARCHAR flag, because it does not affect the
persistent storage format.
2019-02-25 16:14:30 +02:00
Marko Mäkelä
4932aba921 MDEV-18649 Assertion supremum[7] == index.n_core_null_bytes failed
This is follow-up to MDEV-18048: Relax a too strict debug assertion

This assertion should have been relaxed when implementing the first part of
MDEV-15563: instant removal of NOT NULL attribute for ROW_FORMAT=REDUNDANT
tables.

For ROW_FORMAT=REDUNDANT, there is no bitmap of null columns;
the null flags are encoded in the end offset of each field.
We do not really care about the number of fields that can be NULL.
2019-02-20 11:40:16 +02:00
Marko Mäkelä
2c74799d64 MDEV-18627 Wrong result after instant size change of integer
If we instantly change the size of a fixed-length field
and treat it as kind-of variable-length, then we will need
conversions between old column values and new ones.

I tried adding such a conversion to row_build(), but then I
noticed that more conversions would be needed, because
old values still appeared in a freshly rebuilt secondary index,
causing a mismatch when trying to search with the correct
longer value that was converted in my provisional fix to row_build().

So, we will revert the essential part of
MDEV-15563: Instant ROW_FORMAT=REDUNDANT column extension
(commit 22feb179ae), but not
remove any tests.
2019-02-18 18:30:01 +02:00
Marko Mäkelä
33fd3998d2 MDEV-18596 Crash in row_mysql_store_col_in_innobase_format() on MODIFY/ADD column
innobase_build_col_map_add(): Do not assume that old_field->pack_length()
equals to field->pack_length(). Fix submitted by Aleksey Midenkov.

innobase_instant_try(): Assert that the column length of fixed-length
NOT NULL columns is only changing for ROW_FORMAT=REDUNDANT.
2019-02-18 14:22:01 +02:00
Marko Mäkelä
a12d0d5a56 MDEV-18609 Assertion !is_string || (*af)->charset() == cf->charset failed
The Create_field::charset can contain garbage for columns
that the SQL layer does not consider as being string columns.
InnoDB considers BIT a string column for historical reasons
(and backward compatibility with old persistent InnoDB metadata),
and therefore it checked the charset.

The Field::charset() consistently is my_charset_bin for BIT,
so we can trust that one.
2019-02-18 11:44:20 +02:00
Marko Mäkelä
790b6f5ae2 MDEV-18598: Wrong results after instant integer conversions
Field_str::is_equal(): Do not allow instant conversions between
BIT (which is stored big-endian) and integer types (which can
be stored big-endian or little-endian, depending on storage engine).

row_sel_field_store_in_mysql_format_func(): Properly extend
narrower integer and DATA_FIXBINARY values to the current format.
DATA_FIXBINARY was incorrectly padded with 0x20 instead of 0.
2019-02-17 01:05:31 +02:00
Marko Mäkelä
e1af460146 MDEV-18579 Assertion !ctx->online || num_fts_index == 0
instant_alter_column_possible(): Add the other MDEV-17459 work-around
condition. The existence of fulltext indexes only prevents instant
DROP COLUMN or changing the order of columns. Other forms of instant
ALTER TABLE are no problem.

Before commit 4e7ee166a9 that merged
the MDEV-18295 fix from 10.3, the work-around of MDEV-17459 in
instant_alter_column_possible() was categorically refusing any
ALGORITHM=INSTANT if any FULLTEXT INDEX was present. After that commit,
a related condition was only present in prepare_inplace_alter_table_dict()
but not in the other callers of instant_alter_column_possible().
2019-02-15 13:56:04 +02:00
Marko Mäkelä
71261e3188 MDEV-18315/MDEV-18316: Assertion failures on instant DROP COLUMN
dict_table_t::prepare_instant(): Correctly assign instantly
dropped columns to instant->dropped[].
2019-01-21 15:25:37 +02:00
Marko Mäkelä
5fb4e4ab39 MDEV-18149 Crash after DROP COLUMN of AUTO_INCREMENT column on nonempty table
commit_cache_norebuild(): Restore the MDEV-17901 fix that was reverted
as part of the MDEV-18076/MDEV-18077 fix. It was not redundant after all.
2019-01-17 14:16:59 +02:00
Marko Mäkelä
edba047080 MDEV-18218 Assertion `0' failed in btr_page_reorganize_low upon DROP COLUMN
dict_table_t::init_instant(): Correctly initialize the length of
variable-length instantly dropped columns.

row_ins_index_entry_set_vals(): For variable-length instantly dropped
columns, write 0 bytes of data. For dropped fixed-length NOT NULL
columns, write the fixed length of NUL bytes as data.
2019-01-16 18:46:32 +02:00
Marko Mäkelä
301bd62b25 MDEV-17820 Assertion failed on instant DROP COLUMN
btr_cur_pessimistic_insert(): Do not attempt to convert the
metadata BLOB to external storage if it has already been converted.
It could have been converted by btr_cur_pessimistic_update().
Relax some over-zealous assertions.

dtuple_convert_big_rec(): Assert that the metadata BLOB has
not been converted yet.
2019-01-10 00:45:34 +02:00
Marko Mäkelä
30da40bb8c MDEV-18160/MDEV-18162 Assertion failure or crash after DROP COLUMN
dict_table_t::instant_column(): Correctly compute the value of
metadata_changed. The original computation in
commit 003720755f would essentially
invoke memcmp(x,x,y), which can only return 0.
2019-01-08 00:57:19 +02:00
Marko Mäkelä
88b7b8199a MDEV-18076/MDEV-18077 Crash on AUTO_INCREMENT column after instant DROP
The fix for MDEV-17901 did not cover cases where the AUTO_INCREMENT
column was not dropped, but some other columns before it were.

commit_cache_norebuild(): Revert the MDEV-17901 fix.

dict_index_t::clear_instant_alter(): Update table->persistent_autoinc.
2018-12-27 16:31:27 +02:00
Marko Mäkelä
003720755f MDEV-18033/MDEV-18034: Assertion failed on NOT NULL removal
innobase_instant_try(): Only try to update the hidden metadata
record if the number of columns is changing (increasing) or
a metadata BLOB is being added due to permuting or dropping columns
for the first time.

dict_table_t::instant_column(), ha_innobase_inplace_ctx::instant_column():
Return whether the metadata record needs to be updated.
2018-12-27 16:31:27 +02:00
Marko Mäkelä
a8eb0c76bf MDEV-18048: Relax a too strict debug assertion
This assertion should have been relaxed when implementing the first part of
MDEV-15563: instant removal of NOT NULL attribute for ROW_FORMAT=REDUNDANT
tables.

For ROW_FORMAT=REDUNDANT, there is no bitmap of null columns;
the null flags are encoded in the end offset of each field.
We do not really care about the number of fields that can be NULL.
2018-12-21 17:43:01 +02:00
Marko Mäkelä
ccb1acbd3c MDEV-18035 Failing assertion on DELETE
instant_alter_column_possible(): Do not allow instant removal of NOT NULL
attribute from a column that belongs to the key of the clustered index.
2018-12-21 17:43:01 +02:00
Marko Mäkelä
7a27db778e MDEV-15563: Instantly change a column to NULL
Allow instant changes of columns in ROW_FORMAT=REDUNDANT
from NOT NULL to NULL.

Later, this may be implemented for ROW_FORMAT=COMPACT or DYNAMIC,
but in that case any indexes on the table must be rebuilt.

dict_table_t::prepare_instant(): Add some debug assertions,
and relax a debug assertion so that the number of fields is
allowed not to change.

dict_index_t::instant_add_field(): Relax a debug assertion,
allowing a column to change from NOT NULL to NULL.

dict_table_t::instant_column(): Add debug assertions.

instant_alter_column_possible(): Allow ALTER_COLUMN_NULLABLE
when applicable.

innodb_insert_sys_columns(): Add the parameter bool update=false
to run UPDATE instead of INSERT.

innobase_instant_add_col(): Remove; let the only caller invoke
innodb_insert_sys_columns() directly.

innobase_instant_try(): Update the SYS_COLUMNS record if the
column is changed. Only convert the table to the instant ALTER TABLE
format if necessary. For ALTER_COLUMN_NULLABLE in ROW_FORMAT=REDUNDANT,
there is no data format change.
2018-12-13 22:23:49 +02:00
Marko Mäkelä
c1e695c90e MDEV-17763 Assertion `len == 20U' failed in rec_convert_dtuple_to_rec_comp on DROP COLUMN
btr_cur_pessimistic_insert(): Convert the metadata field of the metadata
record into BLOB before inserting, just like btr_cur_optimistic_insert()
does.
2018-12-12 12:15:40 +02:00
Marko Mäkelä
dc6ad59765 MDEV-17901 Crash after instant DROP COLUMN of AUTO_INCREMENT column
We failed to reset the dict_table_t::persistent_autoinc after
instantly dropping an AUTO_INCREMENT column, causing a bogus
call to row_parse_int() on a subsequent insert.
2018-12-11 20:50:22 +02:00
Marko Mäkelä
9fdb8c59eb MDEV-17721: Corrupted data dictionary after instant DROP COLUMN
dict_index_t::clear_instant_alter(): Remove a loop that became
redundant in commit ae2004c616
and caused corruption of n_nullable when dropping the last column
which was not declared NOT NULL.
2018-11-21 13:59:04 +02:00
Marko Mäkelä
ae2004c616 MDEV-17721: Corrupted data dictionary after instant DROP COLUMN
dict_index_t::clear_instant_alter(): Correctly move all fields
corresponding to instantly dropped columns to the end of the array.
This fixes a regression that was introduced in
commit 5aaee3746e.
2018-11-21 12:24:49 +02:00
Marko Mäkelä
5aaee3746e MDEV-17721: Corrupted data dictionary after instant DROP COLUMN
dict_index_t::clear_instant_alter(): Correctly sort the index fields
to correspond to the columns of the table.
2018-11-20 13:32:20 +02:00
Marko Mäkelä
16d43150ae MDEV-17721 Corrupted data dictionary after instant ADD COLUMN
dict_index_t::reconstruct_fields(): Correctly permute the fields
of the columns. The code was totally wrong in MDEV-15562.
It would only work when columns are added last or dropped,
but not when columns are permuted.
2018-11-16 16:35:56 +02:00
Marko Mäkelä
f85012246c MDEV-17735 Assertion failure in row_parse_int() on first ADD/DROP COLUMN
row_ins_clust_index_entry_low(): Do not attempt to read an AUTO_INCREMENT
column value from a metadata record, because it does not make any sense.
Moreover, the field offset would be off by one in case the AUTO_INCREMENT
column is not part of the PRIMARY KEY, because the MDEV-15562 metadata
record would contain an extra field at index->first_user_field().

On MariaDB Server 10.3 after MDEV-11369, we would unnecessarily read
a dummy AUTO_INCREMENT value from the metadata record, but that value
would always be written as NULL or 0, so there is no problem.
2018-11-16 13:28:37 +02:00
Marko Mäkelä
0e5a4ac253 MDEV-15662 Instant DROP COLUMN or changing the order of columns
Allow ADD COLUMN anywhere in a table, not only adding as the
last column.

Allow instant DROP COLUMN and instant changing the order of columns.

The added columns will always be added last in clustered index records.
In new records, instantly dropped columns will be stored as NULL or
empty when possible.

Information about dropped and reordered columns will be written in
a metadata BLOB (mblob), which is stored before the first 'user' field
in the hidden metadata record at the start of the clustered index.
The presence of mblob is indicated by setting the delete-mark flag in
the metadata record.

The metadata BLOB stores the number of clustered index fields,
followed by an array of column information for each field.
For dropped columns, we store the NOT NULL flag, the fixed length,
and for variable-length columns, whether the maximum length exceeded
255 bytes. For non-dropped columns, we store the column position.

Unlike with MDEV-11369, when a table becomes empty, it cannot
be converted back to the canonical format. The reason for this is
that other threads may hold cached objects such as
row_prebuilt_t::ins_node that could refer to dropped or reordered
index fields.

For instant DROP COLUMN and ROW_FORMAT=COMPACT or ROW_FORMAT=DYNAMIC,
we must store the n_core_null_bytes in the root page, so that the
chain of node pointer records can be followed in order to reach the
leftmost leaf page where the metadata record is located.
If the mblob is present, we will zero-initialize the strings
"infimum" and "supremum" in the root page, and use the last byte of
"supremum" for storing the number of null bytes (which are allocated
but useless on node pointer pages). This is necessary for
btr_cur_instant_init_metadata() to be able to navigate to the mblob.

If the PRIMARY KEY contains any variable-length column and some
nullable columns were instantly dropped, the dict_index_t::n_nullable
in the data dictionary could be smaller than it actually is in the
non-leaf pages. Because of this, the non-leaf pages could use more
bytes for the null flags than the data dictionary expects, and we
could be reading the lengths of the variable-length columns from the
wrong offset, and thus reading the child page number from wrong place.
This is the result of two design mistakes that involve unnecessary
storage of data: First, it is nonsense to store any data fields for
the leftmost node pointer records, because the comparisons would be
resolved by the MIN_REC_FLAG alone. Second, there cannot be any null
fields in the clustered index node pointer fields, but we nevertheless
reserve space for all the null flags.

Limitations (future work):

MDEV-17459 Allow instant ALTER TABLE even if FULLTEXT INDEX exists
MDEV-17468 Avoid table rebuild on operations on generated columns
MDEV-17494 Refuse ALGORITHM=INSTANT when the row size is too large

btr_page_reorganize_low(): Preserve any metadata in the root page.
Call lock_move_reorganize_page() only after restoring the "infimum"
and "supremum" records, to avoid a memcmp() assertion failure.

dict_col_t::DROPPED: Magic value for dict_col_t::ind.

dict_col_t::clear_instant(): Renamed from dict_col_t::remove_instant().
Do not assert that the column was instantly added, because we
sometimes call this unconditionally for all columns.
Convert an instantly added column to a "core column". The old name
remove_instant() could be mistaken to refer to "instant DROP COLUMN".

dict_col_t::is_added(): Rename from dict_col_t::is_instant().

dtype_t::metadata_blob_init(): Initialize the mblob data type.

dtuple_t::is_metadata(), dtuple_t::is_alter_metadata(),
upd_t::is_metadata(), upd_t::is_alter_metadata(): Check if info_bits
refer to a metadata record.

dict_table_t::instant: Metadata about dropped or reordered columns.

dict_table_t::prepare_instant(): Prepare
ha_innobase_inplace_ctx::instant_table for instant ALTER TABLE.
innobase_instant_try() will pass this to dict_table_t::instant_column().
On rollback, dict_table_t::rollback_instant() will be called.

dict_table_t::instant_column(): Renamed from instant_add_column().
Add the parameter col_map so that columns can be reordered.
Copy and adjust v_cols[] as well.

dict_table_t::find(): Find an old column based on a new column number.

dict_table_t::serialise_columns(), dict_table_t::deserialise_columns():
Convert the mblob.

dict_index_t::instant_metadata(): Create the metadata record
for instant ALTER TABLE. Invoke dict_table_t::serialise_columns().

dict_index_t::reconstruct_fields(): Invoked by
dict_table_t::deserialise_columns().

dict_index_t::clear_instant_alter(): Move the fields for the
dropped columns to the end, and sort the surviving index fields
in ascending order of column position.

ha_innobase::check_if_supported_inplace_alter(): Do not allow
adding a FTS_DOC_ID column if a hidden FTS_DOC_ID column exists
due to FULLTEXT INDEX. (This always required ALGORITHM=COPY.)

instant_alter_column_possible(): Add a parameter for InnoDB table,
to check for additional conditions, such as the maximum number of
index fields.

ha_innobase_inplace_ctx::first_alter_pos: The first column whose position
is affected by instant ADD, DROP, or changing the order of columns.

innobase_build_col_map(): Skip added virtual columns.

prepare_inplace_add_virtual(): Correctly compute num_to_add_vcol.
Remove some unnecessary code. Note that the call to
innodb_base_col_setup() should be executed later.

commit_try_norebuild(): If ctx->is_instant(), let the virtual
columns be added or dropped by innobase_instant_try().

innobase_instant_try(): Fill in a zero default value for the
hidden column FTS_DOC_ID (to reduce the work needed in MDEV-17459).
If any columns were dropped or reordered (or added not last),
delete any SYS_COLUMNS records for the following columns, and
insert SYS_COLUMNS records for all subsequent stored columns as well
as for all virtual columns. If any virtual column is dropped, rewrite
all virtual column metadata. Use a shortcut only for adding
virtual columns. This is because innobase_drop_virtual_try()
assumes that the dropped virtual columns still exist in ctx->old_table.

innodb_update_cols(): Renamed from innodb_update_n_cols().

innobase_add_one_virtual(), innobase_insert_sys_virtual(): Change
the return type to bool, and invoke my_error() when detecting an error.

innodb_insert_sys_columns(): Insert a record into SYS_COLUMNS.
Refactored from innobase_add_one_virtual() and innobase_instant_add_col().

innobase_instant_add_col(): Replace the parameter dfield with type.

innobase_instant_drop_cols(): Drop matching columns from SYS_COLUMNS
and all columns from SYS_VIRTUAL.

innobase_add_virtual_try(), innobase_drop_virtual_try(): Let
the caller invoke innodb_update_cols().

innobase_rename_column_try(): Skip dropped columns.

commit_cache_norebuild(): Update table->fts->doc_col.

dict_mem_table_col_rename_low(): Skip dropped columns.

trx_undo_rec_get_partial_row(): Skip dropped columns.

trx_undo_update_rec_get_update(): Handle the metadata BLOB correctly.

trx_undo_page_report_modify(): Avoid out-of-bounds access to record fields.
Log metadata records consistently.
Apparently, the first fields of a clustered index may be updated
in an update_undo vector when the index is ID_IND of SYS_FOREIGN,
as part of renaming the table during ALTER TABLE. Normally, updates of
the PRIMARY KEY should be logged as delete-mark and an insert.

row_undo_mod_parse_undo_rec(), row_purge_parse_undo_rec():
Use trx_undo_metadata.

row_undo_mod_clust_low(): On metadata rollback, roll back the root page too.

row_undo_mod_clust(): Relax an assertion. The delete-mark flag was
repurposed for ALTER TABLE metadata records.

row_rec_to_index_entry_impl(): Add the template parameter mblob
and the optional parameter info_bits for specifying the desired new
info bits. For the metadata tuple, allow conversion between the original
format (ADD COLUMN only) and the generic format (with hidden BLOB).
Add the optional parameter "pad" to determine whether the tuple should
be padded to the index fields (on ALTER TABLE it should), or whether
it should remain at its original size (on rollback).

row_build_index_entry_low(): Clean up the code, removing
redundant variables and conditions. For instantly dropped columns,
generate a dummy value that is NULL, the empty string, or a
fixed length of NUL bytes, depending on the type of the dropped column.

row_upd_clust_rec_by_insert_inherit_func(): On the update of PRIMARY KEY
of a record that contained a dropped column whose value was stored
externally, we will be inserting a dummy NULL or empty string value
to the field of the dropped column. The externally stored column would
eventually be dropped when purge removes the delete-marked record for
the old PRIMARY KEY value.

btr_index_rec_validate(): Recognize the metadata record.

btr_discard_only_page_on_level(): Preserve the generic instant
ALTER TABLE metadata.

btr_set_instant(): Replaces page_set_instant(). This sets a clustered
index root page to the appropriate format, or upgrades from
the MDEV-11369 instant ADD COLUMN to generic ALTER TABLE format.

btr_cur_instant_init_low(): Read and validate the metadata BLOB page
before reconstructing the dictionary information based on it.

btr_cur_instant_init_metadata(): Do not read any lengths from the
metadata record header before reading the BLOB. At this point, we
would not actually know how many nullable fields the metadata record
contains.

btr_cur_instant_root_init(): Initialize n_core_null_bytes in one
of two possible ways.

btr_cur_trim(): Handle the mblob record.

row_metadata_to_tuple(): Convert a metadata record to a data tuple,
based on the new info_bits of the metadata record.

btr_cur_pessimistic_update(): Invoke row_metadata_to_tuple() if needed.
Invoke dtuple_convert_big_rec() for metadata records if the record is
too large, or if the mblob is not yet marked as externally stored.

btr_cur_optimistic_delete_func(), btr_cur_pessimistic_delete():
When the last user record is deleted, do not delete the
generic instant ALTER TABLE metadata record. Only delete
MDEV-11369 instant ADD COLUMN metadata records.

btr_cur_optimistic_insert(): Avoid unnecessary computation of rec_size.

btr_pcur_store_position(): Allow a logically empty page to contain
a metadata record for generic ALTER TABLE.

REC_INFO_DEFAULT_ROW_ADD: Renamed from REC_INFO_DEFAULT_ROW.
This is for the old instant ADD COLUMN (MDEV-11369) only.

REC_INFO_DEFAULT_ROW_ALTER: The more generic metadata record,
with additional information for dropped or reordered columns.

rec_info_bits_valid(): Remove. The only case when this would fail
is when the record is the generic ALTER TABLE metadata record.

rec_is_alter_metadata(): Check if a record is the metadata record
for instant ALTER TABLE (other than ADD COLUMN). NOTE: This function
must not be invoked on node pointer records, because the delete-mark
flag in those records may be set (it is garbage), and then a debug
assertion could fail because index->is_instant() does not necessarily
hold.

rec_is_add_metadata(): Check if a record is MDEV-11369 ADD COLUMN metadata
record (not more generic instant ALTER TABLE).

rec_get_converted_size_comp_prefix_low(): Assume that the metadata
field will be stored externally. In dtuple_convert_big_rec() during
the rec_get_converted_size() call, it would not be there yet.

rec_get_converted_size_comp(): Replace status,fields,n_fields with tuple.

rec_init_offsets_comp_ordinary(), rec_get_converted_size_comp_prefix_low(),
rec_convert_dtuple_to_rec_comp(): Add template<bool mblob = false>.
With mblob=true, process a record with a metadata BLOB.

rec_copy_prefix_to_buf(): Assert that no fields beyond the key and
system columns are being copied. Exclude the metadata BLOB field.

rec_convert_dtuple_to_metadata_comp(): Convert an alter metadata tuple
into a record.

row_upd_index_replace_metadata(): Apply an update vector to an
alter_metadata tuple.

row_log_allocate(): Replace dict_index_t::is_instant()
with a more appropriate condition that ignores dict_table_t::instant.
Only a table on which the MDEV-11369 ADD COLUMN was performed
can "lose its instantness" when it becomes empty. After
instant DROP COLUMN or reordering columns, we cannot simply
convert the table to the canonical format, because the data
dictionary cache and all possibly existing references to it
from other client connection threads would have to be adjusted.

row_quiesce_write_index_fields(): Do not crash when the table contains
an instantly dropped column.

Thanks to Thirunarayanan Balathandayuthapani for discussing the design
and implementing an initial prototype of this.
Thanks to Matthias Leich for testing.
2018-10-19 18:57:23 +03:00
Marko Mäkelä
2fa4ed031c MDEV-17483 Insert on delete-marked record can wrongly inherit old values for instantly added column
row_ins_clust_index_entry_low(): Do not call dtuple_t::trim()
before row_ins_clust_index_entry_by_modify(), so that the values
of all columns will be available in row_upd_build_difference_binary().
If applicable, the tuple can be trimmed in btr_cur_optimistic_update()
or btr_cur_pessimistic_update(), which will be called by
row_ins_clust_index_entry_by_modify().
2018-10-17 18:55:46 +03:00
Marko Mäkelä
aba5c72be2 MDEV-17196 Crash during instant ADD COLUMN with long DEFAULT value
A debug assertion would fail if an instant ADD COLUMN operation
involves splitting the leftmost leaf page and storing a default
value off-page. Another debug assertion could fail if the
default value does not fit in an undo log page.

btr_cur_pessimistic_update(): Invoke rec_offs_make_valid()
in order to prevent rec_offs_validate() assertion failure.

innobase_add_instant_try(): Invoke btr_cur_pessimistic_update()
with the BTR_KEEP_POS_FLAG, which is the correct course of action
when BLOBs may need to be written. Whenever returning true,
ensure that my_error() will have been called.
2018-09-14 15:06:58 +03:00
Marko Mäkelä
aab5c557cf MDEV-16830 Crash in ALTER TABLE DROP FOREIGN KEY
ha_innobase::inplace_alter_table(): Do nothing if INNOBASE_ALTER_INSTANT
flags (such as DROP FOREIGN KEY) was present.

Also, use ALTER_OPTIONS instead of the alias ALTER_CHANGE_CREATE_OPTION.

This bug was caused by MDEV-11369, MDEV-13134 or related work.
2018-08-03 17:41:31 +03:00
Marko Mäkelä
73a10cbcc5 MDEV-16065 Assertion failed in btr_pcur_restore_position_func on UPDATE
btr_pcur_store_position(): Assert that the 'default row' record never
is the only record in a page. (If that would happen, an empty
root page would be re-created in the non-instant format, not containing
the special record.) When the cursor is positioned on the page infimum,
never use the 'default row' as the BTR_PCUR_BEFORE reference.
(This is additional cleanup, not fixing the bug.)

rec_copy_prefix_to_buf(): When converting a record prefix to
the non-instant-add format, copy the original number of null flags.
Rename the variable instant_len to instant_omit, and introduce a
few more variables to make the code easiser to read.

Note: In purge, rec_copy_prefix_to_buf() is also used for storing the
persistent cursor position on a 'default row' record. The stored record
reference will be garbage, but row_search_on_row_ref() will do special
handling to reposition the cursor on the 'default row', based on
ref->info_bits.

innodb.dml_purge: Also cover the 'default row'.
2018-05-02 15:44:52 +03:00
Marko Mäkelä
fe79ac5b0e MDEV-14837 Duplicate primary keys are allowed after ADD COLUMN / UPDATE
This bug affected tables where the PRIMARY KEY contains variable-length
columns, and ROW_FORMAT is COMPACT or DYNAMIC.

rec_init_offsets_comp_ordinary(): Do not short-cut the parsing
of the record header for records that contain explicit values
for instantly added columns.

rec_copy_prefix_to_buf(): Copy more header for records that
contain explicit values for instantly added columns.
2018-01-09 13:48:41 +02:00
Marko Mäkelä
34f2f4fa43 MDEV-14660 Assertion failure in lock_move_rec_list_start() after instant ADD COLUMN
lock_move_rec_list_start(): Relax a too strict assertion.
This function can be invoked on the leftmost leaf page, after all.
So, the first record of each page can be a 'default row' record,
but the 'default row' record must never be locked.
2017-12-15 13:52:27 +02:00
Marko Mäkelä
1773116fe0 Use ST_AsText() to get textual result 2017-11-23 15:02:26 +02:00
Marko Mäkelä
761e6574c8 MDEV-14396 Assertion failed in create_option_need_rebuild
Relax a too strict debug assertion, and add a test.
2017-11-15 02:41:45 +02:00