1
0
mirror of https://github.com/MariaDB/server.git synced 2025-12-09 08:01:34 +03:00
Commit Graph

428 Commits

Author SHA1 Message Date
Oleksandr Byelkin
93ac7ae70f Merge branch '10.3' into 10.4 2019-02-21 14:40:52 +01:00
Marko Mäkelä
2c74799d64 MDEV-18627 Wrong result after instant size change of integer
If we instantly change the size of a fixed-length field
and treat it as kind-of variable-length, then we will need
conversions between old column values and new ones.

I tried adding such a conversion to row_build(), but then I
noticed that more conversions would be needed, because
old values still appeared in a freshly rebuilt secondary index,
causing a mismatch when trying to search with the correct
longer value that was converted in my provisional fix to row_build().

So, we will revert the essential part of
MDEV-15563: Instant ROW_FORMAT=REDUNDANT column extension
(commit 22feb179ae), but not
remove any tests.
2019-02-18 18:30:01 +02:00
Marko Mäkelä
22feb179ae MDEV-15563: Instant ROW_FORMAT=REDUNDANT column extension
This was developed by Aleksey Midenkov based on my design.

In the original InnoDB storage format (that was retroactively named
ROW_FORMAT=REDUNDANT in MySQL 5.0.3), the length of each index field
is stored explicitly.

Because of this, we can and now will allow instant conversion from
VARCHAR to CHAR or VARBINARY to BINARY of equal or greater size,
as well as instant conversion of TINYINT to SMALLINT to MEDIUMINT
to INT to BIGINT (while not changing between signed and unsigned).

Theoretically, we could allow changing from an unsigned integer to
a bigger unsigned integer, as well as changing CHAR to VARCHAR, but
that would require additional metadata and conversions whenever
reading old records.

Field_str::is_equal(), Field_varstring::is_equal(), Field_num::is_equal():
Return the new result IS_EQUAL_PACK_LENGTH_EXT if the table advertises
HA_EXTENDED_TYPES_CONVERSION capability and we are considering the
above-mentioned conversions.

ALTER_COLUMN_EQUAL_PACK_LENGTH_EXT: A new ALTER TABLE flag, similar
to ALTER_COLUMN_EQUAL_PACK_LENGTH but requiring conversions when
reading the data. The Field::is_equal() result IS_EQUAL_PACK_LENGTH_EXT
will map to this flag.

dtype_get_fixed_size_low(): For BINARY, CHAR and integer columns
in ROW_FORMAT=REDUNDANT, return 0 (variable length) from now on.

dtype_get_sql_null_size(): Keep returning the current size for
BINARY, CHAR and integer columns, so that in ROW_FORMAT=REDUNDANT
it will remain possible to update in place between NULL and NOT NULL
values.

btr_index_rec_validate(): Relax a CHECK TABLE length check for
ROW_FORMAT=REDUNDANT tables.

btr_cur_instant_init_low(): No longer trust fixed_len
for ROW_FORMAT=REDUNDANT tables.

We cannot rely on fixed_len anymore because the record can have shorter
length from before instant extension. Note that importing such tablespace
into earlier MariaDB versions produces ER_TABLE_SCHEMA_MISMATCH when
using a .cfg file.
2019-02-13 17:39:05 +02:00
Oleksandr Byelkin
dcc838168f Merge branch '10.3' into bb-10.3-merge 2019-02-12 12:05:10 +01:00
Marko Mäkelä
9f56dd7382 Merge 10.3 into 10.4 2019-02-11 17:55:25 +02:00
Marko Mäkelä
4e7ee166a9 MDEV-18295 IMPORT TABLESPACE fails with instant-altered tables
When importing a tablespace, we must initialize dummy DEFAULT NULL
values for any instantly added columns in order to avoid a debug
assertion failure when PageConverter::update_records() invokes
rec_get_offsets(). Finally, when the operation completes, we must
evict and reload the table definition, so that the correct
default values for instantly added columns will be loaded.

ha_innobase::discard_or_import_tablespace(): On successful
IMPORT TABLESPACE, evict and reload the table definition,
so that btr_cur_instant_init() will load the correct metadata.

PageConverter::update_index_page(): Fill in dummy DEFAULT NULL values
for instantly added columns. These will be replaced upon the
completion of the operation by evicting and reloading the metadata.

row_discard_tablespace(): Invoke dict_table_t::remove_instant().
After DISCARD TABLESPACE, the table is no longer in "instant ALTER"
format, because there is no data file attached.
2019-02-11 14:42:48 +02:00
Oleksandr Byelkin
65c5ef9b49 dirty merge 2019-02-07 13:59:31 +01:00
Marko Mäkelä
0a1c3477bf MDEV-18493 Remove page_size_t
MySQL 5.7 introduced the class page_size_t and increased the size of
buffer pool page descriptors by introducing this object to them.

Maybe the intention of this exercise was to prepare for a future
where the buffer pool could accommodate multiple page sizes.
But that future never arrived, not even in MySQL 8.0. It is much
easier to manage a pool of a single page size, and typically all
storage devices of an InnoDB instance benefit from using the same
page size.

Let us remove page_size_t from MariaDB Server. This will make it
easier to remove support for ROW_FORMAT=COMPRESSED (or make it a
compile-time option) in the future, just by removing various
occurrences of zip_size.
2019-02-07 12:21:35 +02:00
Marko Mäkelä
a249e57b68 Merge 10.1 into 10.2
Temporarily disable a test for
commit 2175bfce3e
because fixing it in 10.2 requires updating libmariadb.
2019-02-03 17:22:05 +02:00
Marko Mäkelä
213ece2f2e Merge 10.1 into 10.1
This is joint work with Oleksandr Byelkin.
2019-02-02 13:00:15 +02:00
Oleksandr Byelkin
94b68b35f4 Reverting part of da34c7de5d that was already fixed by MDEV-17531 by Marko 2019-01-28 15:39:27 +01:00
Marko Mäkelä
0474be0aab Merge 10.3 into 10.4 2019-01-18 13:58:33 +02:00
Marko Mäkelä
a0d3ead83a Merge 10.2 into 10.3 2019-01-18 10:43:31 +02:00
Marko Mäkelä
37ffdb44ef Merge 10.1 into 10.2 2019-01-18 06:51:52 +02:00
Marko Mäkelä
1d72db45a8 MDEV-18237 InnoDB: Unable to drop FTS index aux table and further errors (possibly bogus)
row_drop_table_for_mysql(): Fix a regression introduced in MDEV-16515.
Similar to the follow-up fixes MDEV-16647 and MDEV-17470, we must make
the internal tables of FULLTEXT INDEX immune to kills, to avoid noise
and resource leakage on DROP TABLE or ALTER TABLE. (Orphan internal tables
would be dropped at the next InnoDB startup only.)
2019-01-18 06:46:39 +02:00
Marko Mäkelä
734510a44d Merge 10.3 into 10.4 2019-01-06 17:43:02 +02:00
Sergei Golubchik
6bb11efa4a Merge branch '10.2' into 10.3 2019-01-03 13:09:41 +01:00
Sergei Golubchik
aeefd26ecb Merge branch '10.0' into 10.1 2018-12-29 23:44:45 +01:00
Marko Mäkelä
33caaba5c8 Merge 10.1 into 10.2 2018-12-28 17:40:38 +02:00
Eugene Kosov
c5a5eaa9a9 MDEV-17470 Orphan temporary files after interrupted ALTER cause InnoDB: Operating system error number 17 and eventual fatal error 71
Orphan #sql* tables may remain after ALTER TABLE
was interrupted by timeout or KILL or client disconnect.

This is a regression caused by MDEV-16515.

Similar to temporary tables (MDEV-16647), we had better ignore the
KILL when dropping the original table in the final part of ALTER TABLE.

Closes #1020
2018-12-28 17:05:48 +02:00
Marko Mäkelä
b74eb5a5fe row_drop_table_for_mysql(): Correct a parameter to innobase_format_name()
This fixes a regression that was introduced in MySQL 5.6.6
in an error handling code path, in the following change:

commit 024f363d6b5f09b20d1bba411af55be95c7398d3
Author: kevin.lewis@oracle.com <>
Date:   Fri Jun 15 09:01:42 2012 -0500

    Bug #14169459 INNODB; DROP TABLE DOES NOT DELETE THE IBD FILE
    FOR A TEMPORARY TABLE.
2018-12-28 12:28:16 +02:00
Marko Mäkelä
1b31d8852c MDEV-17899: Fix a regression from MDEV-17793
The fix of MDEV-17793 was updating SYS_INDEXES.TABLE_ID in order
to make the table invisible to purge (lazily delete old undo log
records).

By design of InnoDB, an update of TABLE_ID cannot be rolled back,
because the rollback would effectively drop all indexes of the table
due to the internal 'trigger' on SYS_INDEXES modifications.

So, we revert the code change of MDEV-17793 and instead fix
MDEV-17793 in a different way: by tweaking the undo log parsing
during purge.

The MDEV-17793 bug scenario is that a table becomes empty and
a third instant ALTER TABLE is executed before purge processes
the undo log record for the second instant ALTER TABLE. After
this point, when purge sees the record, the table could have
a mismatching number of rows.

The test case works with this alternative fix. But what about
a scenario where a fourth instant ALTER TABLE arrives before
purge processes the second one? Could anything bad happen?

Purge is only doing two things: First, free any BLOBs that
were affected by the update record, and then, reset the
DB_TRX_ID,DB_ROLL_PTR if a matching record is found.
For the hidden metadata record, the only BLOB that we update
is the hidden metadata BLOB that was introduced by MDEV-15562.
Any other BLOBs (for the initial default values of instantly
added columns) are never updated.

So, in our scenario, the metadata BLOB that was created by
the first instant ALTER TABLE (if it involved dropping or
permuting columns) would be freed by purge when it is processing
the undo record of the second ALTER TABLE. The BLOB value that
was written by the second ALTER TABLE should be freed when
the table is emptied. This is currently not done: MDEV-17383
should fix that. There is no possibility of double-free, because
purge would only free old values of BLOBs.

What about MVCC and other callers of trx_undo_update_rec_get_update()?
The answer is simple: they should never be accessing the hidden
metadata record in the first place.

dict_table_t::reassign_id(): Remove.

btr_cur_pessimistic_delete(): Clarify a comment.

row_mysql_table_id_reassign(), row_discard_tablespace_for_mysql():
Add comments explaining that the operation cannot be rolled back.

trx_undo_update_rec_get_update(): Avoid out-of-bounds access when
parsing a metadata record. Avoid unnecessary memory allocation when
filtering out fields from the update vector.
2018-12-11 13:45:24 +02:00
Marko Mäkelä
fcb6bb4bcc Merge 10.3 into 10.4 2018-12-10 13:16:03 +02:00
Marko Mäkelä
1d18665e0b Merge 10.2 into 10.3 2018-12-10 12:28:31 +02:00
Marko Mäkelä
0d7cf06af5 MDEV-17938 ALTER TABLE reports ER_TABLESPACE_EXISTS after failed ALTER TABLE
There was a race condition in the error handling of ALTER TABLE when
the table contains FULLTEXT INDEX.

During the error handling of an erroneous ALTER TABLE statement,
when InnoDB would drop the internally created tables for FULLTEXT INDEX,
it could happen that one of the hidden tables was being concurrently
accessed by a background thread. Because of this, InnoDB would defer
the drop operation to the background.

However, related to MDEV-13564 backup-safe TRUNCATE TABLE and its
prerequisite MDEV-14585, we had to make the background drop table queue
crash-safe by renaming the table to a temporary name before enqueueing it.
This renaming was introduced in a follow-up of the MDEV-13407 fix.
As part of this rename operation, we were unnecessarily parsing the
current SQL statement, because the same rename operation could also be
executed as part of ALTER TABLE via ha_innobase::rename_table().

If an ALTER TABLE statement was being refused due to incorrectly formed
FOREIGN KEY constraint, then it could happen that the renaming of the hidden
internal tables for FULLTEXT INDEX could also fail, triggering a host of
error log messages, and causing a subsequent table-rebuilding ALTER TABLE
operation to fail due to the tablespace already existing.

innobase_rename_table(), row_rename_table_for_mysql(): Add the parameter
use_fk for suppressing the parsing of FOREIGN KEY constraints. It
will only be passed as use_fk=true by ha_innobase::rename_table(),
which can be invoked as part of ALTER TABLE...ALGORITHM=COPY.
2018-12-10 11:44:39 +02:00
Marko Mäkelä
b374246730 Merge 10.3 into 10.4 2018-11-30 11:05:46 +02:00
Marko Mäkelä
0abd2766b1 Merge 10.2 into 10.3
Also, related to MDEV-15522, MDEV-17304, MDEV-17835,
remove the Galera xtrabackup tests, because xtrabackup never worked
with MariaDB Server 10.3 due to InnoDB redo log format changes.
2018-11-30 09:38:56 +02:00
Marko Mäkelä
be998bfdc5 MDEV-17859: Clean up the FOREIGN KEY handling
dict_create_add_foreigns_to_dictionary(): Do not commit the transaction.
The operation can still fail in dict_load_foreigns(), and we want
to be able to roll back the transaction.

create_table_info_t::create_table(): Never reset m_drop_before_rollback,
and never commit the transaction. We use a single point of rollback
in ha_innobase::create(). Merge the logic from
row_table_add_foreign_constraints().
2018-11-29 09:16:48 +02:00
Marko Mäkelä
7dcbc33db5 Merge 10.3 into 10.4 2018-11-26 17:20:07 +02:00
Marko Mäkelä
1afed20774 Merge 10.2 into 10.3
The test galera_sst_mariabackup_table_options was disabled,
because the server refuses to start up due to wrong parameters.
2018-11-26 14:05:15 +02:00
Marko Mäkelä
2a31b82831 MDEV-17816 Crash in TRUNCATE TABLE when table creation fails
The error handling in the MDEV-13564 TRUNCATE TABLE was broken
when an error occurred during table creation.

row_create_index_for_mysql(): Do not drop the table on error.

fts_create_one_common_table(), fts_create_one_index_table():
Do drop the table on error.

create_index(), create_table_info_t::create_table():
Let the caller handle the index creation errors.

ha_innobase::create(): If create_table_info_t::create_table()
fails, drop the incomplete table, roll back the transaction,
and finally return an error to the caller.
2018-11-26 12:50:27 +02:00
Marko Mäkelä
06e5f28f9f MDEV-12266: Remove a level of pointer indirection
Replace table->space->id with table->space_id.
2018-11-22 17:10:26 +02:00
Marko Mäkelä
4be0855cf5 MDEV-17794 Do not assign persistent ID for temporary tables
InnoDB in MySQL 5.7 introduced two new parameters to the function
dict_hdr_get_new_id(), to allow redo logging to be disabled when
assigning identifiers to temporary tables or during the
backup-unfriendly TRUNCATE TABLE that was replaced in MariaDB
by MDEV-13564.

Now that MariaDB 10.4.0 removed the crash recovery code for the
backup-unfriendly TRUNCATE, we can revert dict_hdr_get_new_id()
to be used only for persistent data structures.

dict_table_assign_new_id(): Remove. This was a simple 2-line function
that was called from few places.

dict_table_open_on_id_low(): Declare in the only file where it
is called.

dict_sys_t::temp_id_hash: A separate lookup table for temporary tables.
Table names will be in the common dict_sys_t::table_hash.

dict_sys_t::get_temporary_table_id(): Assign a temporary table ID.

dict_sys_t::get_table(): Look up a persistent table.

dict_sys_t::get_temporary_table(): Look up a temporary table.

dict_sys_t::temp_table_id: The sequence of temporary table identifiers.
Starts from DICT_HDR_FIRST_ID, so that we can continue to simply compare
dict_table_t::id to a few constants for the persistent hard-coded
data dictionary tables.

undo_node_t::state: Distinguish temporary and persistent tables.

lock_check_dict_lock(), lock_get_table_id(): Assert that there cannot
be locks on temporary tables.

row_rec_to_index_entry_impl(): Assert that there cannot be metadata
records on temporary tables.

row_undo_ins_parse_undo_rec(): Distinguish temporary and persistent tables.
Move some assertions from the only caller. Return whether the table was
found.

row_undo_ins(): Add some assertions.

row_undo_mod_clust(), row_undo_mod(): Do not assign node->state.
Let row_undo() do that.

row_undo_mod_parse_undo_rec(): Distinguish temporary and persistent tables.
Move some assertions from the only caller. Return whether the table was
found.

row_undo_try_truncate(): Renamed and simplified from trx_roll_try_truncate().

row_undo_rec_get(): Replaces trx_roll_pop_top_rec_of_trx() and
trx_roll_pop_top_rec(). Fetch an undo log record, and assign undo->state
accordingly.

trx_undo_truncate_end(): Acquire the rseg->mutex only for the minimum
required duration, and release it between mini-transactions.
2018-11-22 15:42:52 +02:00
Marko Mäkelä
dde2ca4aa1 Merge 10.3 into 10.4 2018-11-19 20:22:33 +02:00
Marko Mäkelä
fd58bb71e2 Merge 10.2 into 10.3 2018-11-19 18:45:53 +02:00
Marko Mäkelä
ff88e4bb8a Remove many redundant #include from InnoDB 2018-11-19 11:42:14 +02:00
Marko Mäkelä
f92d223fe2 MDEV-17750: Remove dict_index_get_sys_col_pos()
dict_index_t::db_trx_id(): Return the position of DB_TRX_ID.
Only valid for the clustered index.

dict_index_t::db_roll_ptr(): Return the position of DB_ROLL_PTR.
Only valid for the clustered index.

dict_index_get_sys_col_pos(): Remove. This was performing unnecessarily
complex computations, which only made sense for DB_ROW_ID, which would
exist either as the first field in the clustered index or as the last
field in a secondary index (only when a DB_ROW_ID column is materialised).

row_sel_store_row_id_to_prebuilt(): Remove, and replace with simpler code.

row_upd_index_entry_sys_field(): Remove.

btr_cur_log_sys(): Replaces row_upd_write_sys_vals_to_log().

btr_cur_write_sys(): Write DB_TRX_ID,DB_ROLL_PTR to a data tuple.
2018-11-16 20:37:06 +02:00
Marko Mäkelä
074c684099 Merge 10.3 into 10.4 2018-11-06 16:24:16 +02:00
Marko Mäkelä
df563e0c03 Merge 10.2 into 10.3
main.derived_cond_pushdown: Move all 10.3 tests to the end,
trim trailing white space, and add an "End of 10.3 tests" marker.
Add --sorted_result to tests where the ordering is not deterministic.

main.win_percentile: Add --sorted_result to tests where the
ordering is no longer deterministic.
2018-11-06 09:40:39 +02:00
Marko Mäkelä
32062cc61c Merge 10.1 into 10.2 2018-11-06 08:41:48 +02:00
Marko Mäkelä
a5cbdd63bc Fix innodb.table_flags,debug 2018-10-31 12:09:08 +02:00
Sergei Golubchik
a6e0000494 Merge branch '10.0' into 10.1 2018-10-31 10:53:22 +01:00
Sergei Golubchik
44f6f44593 Merge branch '10.0' into 10.1 2018-10-30 15:10:01 +01:00
Marko Mäkelä
dc91ea5bb7 MDEV-12023 Assertion failure sym_node->table != NULL on startup
row_drop_table_for_mysql(): Avoid accessing non-existing dictionary tables.

dict_create_or_check_foreign_constraint_tables(): Add debug instrumentation
for creating and dropping a table before the creation of any non-core
dictionary tables.

trx_purge_add_update_undo_to_history(): Adjust a debug assertion, so that
it will not fail due to the test instrumentation.
2018-10-30 15:53:55 +02:00
Marko Mäkelä
6ced789186 MDEV-12023 Assertion failure sym_node->table != NULL on startup
row_drop_table_for_mysql(): Avoid accessing non-existing dictionary tables.

dict_create_or_check_foreign_constraint_tables(): Add debug instrumentation
for creating and dropping a table before the creation of any non-core
dictionary tables.

trx_purge_add_update_undo_to_history(): Adjust a debug assertion, so that
it will not fail due to the test instrumentation.
2018-10-30 13:29:19 +02:00
Sergei Golubchik
87d852f102 Merge branch 'merge/merge-innodb-5.6' into 10.0 2018-10-28 01:22:18 +02:00
Sergei Golubchik
da34c7de5d 5.6.42 2018-10-27 21:05:16 +02:00
Marko Mäkelä
5dd3b52f95 MDEV-17531 Crash in RENAME TABLE with FOREIGN KEY and FULLTEXT INDEX
In RENAME TABLE, when an error occurs while renaming FOREIGN KEY
constraint, that error would be overwritten when renaming the
InnoDB internal tables related to FULLTEXT INDEX.

row_rename_table_for_mysql(): Do not attempt to rename the internal
tables if an error already occurred.

This problem was originally reported as Oracle Bug#27545888.
2018-10-25 13:03:29 +03:00
Marko Mäkelä
0e5a4ac253 MDEV-15662 Instant DROP COLUMN or changing the order of columns
Allow ADD COLUMN anywhere in a table, not only adding as the
last column.

Allow instant DROP COLUMN and instant changing the order of columns.

The added columns will always be added last in clustered index records.
In new records, instantly dropped columns will be stored as NULL or
empty when possible.

Information about dropped and reordered columns will be written in
a metadata BLOB (mblob), which is stored before the first 'user' field
in the hidden metadata record at the start of the clustered index.
The presence of mblob is indicated by setting the delete-mark flag in
the metadata record.

The metadata BLOB stores the number of clustered index fields,
followed by an array of column information for each field.
For dropped columns, we store the NOT NULL flag, the fixed length,
and for variable-length columns, whether the maximum length exceeded
255 bytes. For non-dropped columns, we store the column position.

Unlike with MDEV-11369, when a table becomes empty, it cannot
be converted back to the canonical format. The reason for this is
that other threads may hold cached objects such as
row_prebuilt_t::ins_node that could refer to dropped or reordered
index fields.

For instant DROP COLUMN and ROW_FORMAT=COMPACT or ROW_FORMAT=DYNAMIC,
we must store the n_core_null_bytes in the root page, so that the
chain of node pointer records can be followed in order to reach the
leftmost leaf page where the metadata record is located.
If the mblob is present, we will zero-initialize the strings
"infimum" and "supremum" in the root page, and use the last byte of
"supremum" for storing the number of null bytes (which are allocated
but useless on node pointer pages). This is necessary for
btr_cur_instant_init_metadata() to be able to navigate to the mblob.

If the PRIMARY KEY contains any variable-length column and some
nullable columns were instantly dropped, the dict_index_t::n_nullable
in the data dictionary could be smaller than it actually is in the
non-leaf pages. Because of this, the non-leaf pages could use more
bytes for the null flags than the data dictionary expects, and we
could be reading the lengths of the variable-length columns from the
wrong offset, and thus reading the child page number from wrong place.
This is the result of two design mistakes that involve unnecessary
storage of data: First, it is nonsense to store any data fields for
the leftmost node pointer records, because the comparisons would be
resolved by the MIN_REC_FLAG alone. Second, there cannot be any null
fields in the clustered index node pointer fields, but we nevertheless
reserve space for all the null flags.

Limitations (future work):

MDEV-17459 Allow instant ALTER TABLE even if FULLTEXT INDEX exists
MDEV-17468 Avoid table rebuild on operations on generated columns
MDEV-17494 Refuse ALGORITHM=INSTANT when the row size is too large

btr_page_reorganize_low(): Preserve any metadata in the root page.
Call lock_move_reorganize_page() only after restoring the "infimum"
and "supremum" records, to avoid a memcmp() assertion failure.

dict_col_t::DROPPED: Magic value for dict_col_t::ind.

dict_col_t::clear_instant(): Renamed from dict_col_t::remove_instant().
Do not assert that the column was instantly added, because we
sometimes call this unconditionally for all columns.
Convert an instantly added column to a "core column". The old name
remove_instant() could be mistaken to refer to "instant DROP COLUMN".

dict_col_t::is_added(): Rename from dict_col_t::is_instant().

dtype_t::metadata_blob_init(): Initialize the mblob data type.

dtuple_t::is_metadata(), dtuple_t::is_alter_metadata(),
upd_t::is_metadata(), upd_t::is_alter_metadata(): Check if info_bits
refer to a metadata record.

dict_table_t::instant: Metadata about dropped or reordered columns.

dict_table_t::prepare_instant(): Prepare
ha_innobase_inplace_ctx::instant_table for instant ALTER TABLE.
innobase_instant_try() will pass this to dict_table_t::instant_column().
On rollback, dict_table_t::rollback_instant() will be called.

dict_table_t::instant_column(): Renamed from instant_add_column().
Add the parameter col_map so that columns can be reordered.
Copy and adjust v_cols[] as well.

dict_table_t::find(): Find an old column based on a new column number.

dict_table_t::serialise_columns(), dict_table_t::deserialise_columns():
Convert the mblob.

dict_index_t::instant_metadata(): Create the metadata record
for instant ALTER TABLE. Invoke dict_table_t::serialise_columns().

dict_index_t::reconstruct_fields(): Invoked by
dict_table_t::deserialise_columns().

dict_index_t::clear_instant_alter(): Move the fields for the
dropped columns to the end, and sort the surviving index fields
in ascending order of column position.

ha_innobase::check_if_supported_inplace_alter(): Do not allow
adding a FTS_DOC_ID column if a hidden FTS_DOC_ID column exists
due to FULLTEXT INDEX. (This always required ALGORITHM=COPY.)

instant_alter_column_possible(): Add a parameter for InnoDB table,
to check for additional conditions, such as the maximum number of
index fields.

ha_innobase_inplace_ctx::first_alter_pos: The first column whose position
is affected by instant ADD, DROP, or changing the order of columns.

innobase_build_col_map(): Skip added virtual columns.

prepare_inplace_add_virtual(): Correctly compute num_to_add_vcol.
Remove some unnecessary code. Note that the call to
innodb_base_col_setup() should be executed later.

commit_try_norebuild(): If ctx->is_instant(), let the virtual
columns be added or dropped by innobase_instant_try().

innobase_instant_try(): Fill in a zero default value for the
hidden column FTS_DOC_ID (to reduce the work needed in MDEV-17459).
If any columns were dropped or reordered (or added not last),
delete any SYS_COLUMNS records for the following columns, and
insert SYS_COLUMNS records for all subsequent stored columns as well
as for all virtual columns. If any virtual column is dropped, rewrite
all virtual column metadata. Use a shortcut only for adding
virtual columns. This is because innobase_drop_virtual_try()
assumes that the dropped virtual columns still exist in ctx->old_table.

innodb_update_cols(): Renamed from innodb_update_n_cols().

innobase_add_one_virtual(), innobase_insert_sys_virtual(): Change
the return type to bool, and invoke my_error() when detecting an error.

innodb_insert_sys_columns(): Insert a record into SYS_COLUMNS.
Refactored from innobase_add_one_virtual() and innobase_instant_add_col().

innobase_instant_add_col(): Replace the parameter dfield with type.

innobase_instant_drop_cols(): Drop matching columns from SYS_COLUMNS
and all columns from SYS_VIRTUAL.

innobase_add_virtual_try(), innobase_drop_virtual_try(): Let
the caller invoke innodb_update_cols().

innobase_rename_column_try(): Skip dropped columns.

commit_cache_norebuild(): Update table->fts->doc_col.

dict_mem_table_col_rename_low(): Skip dropped columns.

trx_undo_rec_get_partial_row(): Skip dropped columns.

trx_undo_update_rec_get_update(): Handle the metadata BLOB correctly.

trx_undo_page_report_modify(): Avoid out-of-bounds access to record fields.
Log metadata records consistently.
Apparently, the first fields of a clustered index may be updated
in an update_undo vector when the index is ID_IND of SYS_FOREIGN,
as part of renaming the table during ALTER TABLE. Normally, updates of
the PRIMARY KEY should be logged as delete-mark and an insert.

row_undo_mod_parse_undo_rec(), row_purge_parse_undo_rec():
Use trx_undo_metadata.

row_undo_mod_clust_low(): On metadata rollback, roll back the root page too.

row_undo_mod_clust(): Relax an assertion. The delete-mark flag was
repurposed for ALTER TABLE metadata records.

row_rec_to_index_entry_impl(): Add the template parameter mblob
and the optional parameter info_bits for specifying the desired new
info bits. For the metadata tuple, allow conversion between the original
format (ADD COLUMN only) and the generic format (with hidden BLOB).
Add the optional parameter "pad" to determine whether the tuple should
be padded to the index fields (on ALTER TABLE it should), or whether
it should remain at its original size (on rollback).

row_build_index_entry_low(): Clean up the code, removing
redundant variables and conditions. For instantly dropped columns,
generate a dummy value that is NULL, the empty string, or a
fixed length of NUL bytes, depending on the type of the dropped column.

row_upd_clust_rec_by_insert_inherit_func(): On the update of PRIMARY KEY
of a record that contained a dropped column whose value was stored
externally, we will be inserting a dummy NULL or empty string value
to the field of the dropped column. The externally stored column would
eventually be dropped when purge removes the delete-marked record for
the old PRIMARY KEY value.

btr_index_rec_validate(): Recognize the metadata record.

btr_discard_only_page_on_level(): Preserve the generic instant
ALTER TABLE metadata.

btr_set_instant(): Replaces page_set_instant(). This sets a clustered
index root page to the appropriate format, or upgrades from
the MDEV-11369 instant ADD COLUMN to generic ALTER TABLE format.

btr_cur_instant_init_low(): Read and validate the metadata BLOB page
before reconstructing the dictionary information based on it.

btr_cur_instant_init_metadata(): Do not read any lengths from the
metadata record header before reading the BLOB. At this point, we
would not actually know how many nullable fields the metadata record
contains.

btr_cur_instant_root_init(): Initialize n_core_null_bytes in one
of two possible ways.

btr_cur_trim(): Handle the mblob record.

row_metadata_to_tuple(): Convert a metadata record to a data tuple,
based on the new info_bits of the metadata record.

btr_cur_pessimistic_update(): Invoke row_metadata_to_tuple() if needed.
Invoke dtuple_convert_big_rec() for metadata records if the record is
too large, or if the mblob is not yet marked as externally stored.

btr_cur_optimistic_delete_func(), btr_cur_pessimistic_delete():
When the last user record is deleted, do not delete the
generic instant ALTER TABLE metadata record. Only delete
MDEV-11369 instant ADD COLUMN metadata records.

btr_cur_optimistic_insert(): Avoid unnecessary computation of rec_size.

btr_pcur_store_position(): Allow a logically empty page to contain
a metadata record for generic ALTER TABLE.

REC_INFO_DEFAULT_ROW_ADD: Renamed from REC_INFO_DEFAULT_ROW.
This is for the old instant ADD COLUMN (MDEV-11369) only.

REC_INFO_DEFAULT_ROW_ALTER: The more generic metadata record,
with additional information for dropped or reordered columns.

rec_info_bits_valid(): Remove. The only case when this would fail
is when the record is the generic ALTER TABLE metadata record.

rec_is_alter_metadata(): Check if a record is the metadata record
for instant ALTER TABLE (other than ADD COLUMN). NOTE: This function
must not be invoked on node pointer records, because the delete-mark
flag in those records may be set (it is garbage), and then a debug
assertion could fail because index->is_instant() does not necessarily
hold.

rec_is_add_metadata(): Check if a record is MDEV-11369 ADD COLUMN metadata
record (not more generic instant ALTER TABLE).

rec_get_converted_size_comp_prefix_low(): Assume that the metadata
field will be stored externally. In dtuple_convert_big_rec() during
the rec_get_converted_size() call, it would not be there yet.

rec_get_converted_size_comp(): Replace status,fields,n_fields with tuple.

rec_init_offsets_comp_ordinary(), rec_get_converted_size_comp_prefix_low(),
rec_convert_dtuple_to_rec_comp(): Add template<bool mblob = false>.
With mblob=true, process a record with a metadata BLOB.

rec_copy_prefix_to_buf(): Assert that no fields beyond the key and
system columns are being copied. Exclude the metadata BLOB field.

rec_convert_dtuple_to_metadata_comp(): Convert an alter metadata tuple
into a record.

row_upd_index_replace_metadata(): Apply an update vector to an
alter_metadata tuple.

row_log_allocate(): Replace dict_index_t::is_instant()
with a more appropriate condition that ignores dict_table_t::instant.
Only a table on which the MDEV-11369 ADD COLUMN was performed
can "lose its instantness" when it becomes empty. After
instant DROP COLUMN or reordering columns, we cannot simply
convert the table to the canonical format, because the data
dictionary cache and all possibly existing references to it
from other client connection threads would have to be adjusted.

row_quiesce_write_index_fields(): Do not crash when the table contains
an instantly dropped column.

Thanks to Thirunarayanan Balathandayuthapani for discussing the design
and implementing an initial prototype of this.
Thanks to Matthias Leich for testing.
2018-10-19 18:57:23 +03:00
Marko Mäkelä
6319c0b541 MDEV-13564: Replace innodb_unsafe_truncate with innodb_safe_truncate
Rename the 10.2-specific configuration option innodb_unsafe_truncate
to innodb_safe_truncate, and invert its value.

The default (for now) is innodb_safe_truncate=OFF, to avoid
disrupting users with an undo and redo log format change within
a Generally Available (GA) release series.
2018-10-11 15:10:13 +03:00