1
0
mirror of https://github.com/MariaDB/server.git synced 2025-11-15 09:02:33 +03:00
Commit Graph

1619 Commits

Author SHA1 Message Date
Sergei Golubchik
67240858b2 Merge branch '10.1' into 10.2 2018-12-30 18:30:29 +01:00
Sergei Golubchik
aeefd26ecb Merge branch '10.0' into 10.1 2018-12-29 23:44:45 +01:00
Eugene Kosov
802ce9672f MDEV-18041 Database corruption after renaming a prefix-indexed column
This is a regression after MDEV-13671.

The bug is related to key part prefix lengths wich are stored in SYS_FIELDS.
Storage format is not obvious and was handled incorrectly which led to data
dictionary corruption.

SYS_FIELDS.POS actually contains prefix length too in case if any key part
has prefix length.

innobase_rename_column_try(): fixed prefixes handling

Tests for prefixed indexes added too.

Closes #1063
2018-12-29 22:57:05 +02:00
Marko Mäkelä
0b73b96f9d Merge 10.1 into 10.2 2018-12-29 11:05:26 +02:00
Marko Mäkelä
68143c8905 MDEV-17470: Fix the test for --embedded 2018-12-29 10:57:26 +02:00
Marko Mäkelä
33caaba5c8 Merge 10.1 into 10.2 2018-12-28 17:40:38 +02:00
Eugene Kosov
c5a5eaa9a9 MDEV-17470 Orphan temporary files after interrupted ALTER cause InnoDB: Operating system error number 17 and eventual fatal error 71
Orphan #sql* tables may remain after ALTER TABLE
was interrupted by timeout or KILL or client disconnect.

This is a regression caused by MDEV-16515.

Similar to temporary tables (MDEV-16647), we had better ignore the
KILL when dropping the original table in the final part of ALTER TABLE.

Closes #1020
2018-12-28 17:05:48 +02:00
Marko Mäkelä
c5bb6024a7 MDEV-18039 Assertion failed in btr_node_ptr_max_size for VARCHAR(0)
btr_node_ptr_max_size(): Do not reserve extra space for indexed VARCHAR(0)
columns.
2018-12-21 12:12:57 +02:00
Marko Mäkelä
dbb39a778d MDEV-17958: Make innochecksum follow the build option
Innochecksum was being built as if WITH_INNODB_BUG_ENDIAN_CRC32:BOOL=OFF
had been specified.

Also, clean up tests:

innodb.innochecksum: Useless; superceded by innodb_zip.innochecksum.
innodb.innodb_zip_innochecksum: Remove; duplicated innodb_zip.innochecksum.
innodb.innodb_zip_innochecksum2: Remove; duplicated innodb_zip.innochecksum_2.
innodb.innodb_zip_innochecksum3: Remove; duplicated innodb_zip.innochecksum_3.

No test case was added. I tested manually by adding debug instrumentation
to both innochecksum and buf_page_is_checksum_valid_crc32() to make
innochecksum write the buggy crc32, and to get warnings for falling back
to the buggy checksum. Automating this would require that tests be
adjusted depending on the build options.
2018-12-14 13:52:13 +02:00
Marko Mäkelä
e3dda3d95e MDEV-17989 InnoDB: Failing assertion: dict_tf2_is_valid(flags, flags2)
With innodb_default_row_format=redundant, InnoDB would crash when
using table options that are incompatible with ROW_FORMAT=REDUNDANT.

create_table_info_t::m_default_row_format: Cache the value of
innodb_default_row_format.

create_table_info_t::check_table_options(): Validate ROW_TYPE_DEFAULT
with m_default_row_format.

create_table_info_t::innobase_table_flags(): Use the
cached m_default_row_format.

create_table_info_t: Never read m_form->s->row_type.
Use m_create_info->row_type instead.

dict_tf_set(): Never set invalid flags for ROW_FORMAT=REDUNDANT.

ha_innobase::truncate(): Set info.row_type based on the ROW_FORMAT
of the current table.
2018-12-13 21:25:12 +02:00
Marko Mäkelä
2e5aea4bab Merge 10.1 into 10.2 2018-12-13 15:47:38 +02:00
Marko Mäkelä
621041b676 Merge 10.0 into 10.1
Also, apply the MDEV-17957 changes to encrypted page checksums,
and remove error message output from the checksum function,
because these messages would be useless noise when mariabackup
is retrying reads of corrupted-looking pages, and not that
useful during normal server operation either.

The error messages in fil_space_verify_crypt_checksum()
should be refactored separately.
2018-12-13 13:37:21 +02:00
Marko Mäkelä
c913cd2b66 MDEV-17885 TRUNCATE on temporary table causes ER_GET_ERRNO
ha_innobase::truncate(): Because CREATE TEMPORARY TABLE
allows invalid table options when innodb_file_per_table=1,
do allow them also in TRUNCATE for temporary tables.
2018-12-12 16:31:34 +02:00
Marko Mäkelä
e0aebf5cf1 MDEV-17815 Assertion failed in btr_node_ptr_max_size for CHAR(0)
btr_node_ptr_max_size(): Treat CHAR(0) from SQL as a special case.
The InnoDB internal SQL parser maps the type "CHAR" to DATA_VARCHAR,
but MariaDB does allow CHAR(0) with an empty value, and does enforce
the length limitation.
2018-12-12 13:21:05 +02:00
Marko Mäkelä
db1210f939 Merge 10.1 into 10.2 2018-12-12 12:13:43 +02:00
Marko Mäkelä
f77f8f6d1a Merge 10.0 into 10.1 2018-12-12 10:48:53 +02:00
Eugene Kosov
d956709b4b MDEV-17833 ALTER TABLE is not enforcing prefix index size limit
ha_innobase::prepare_inplace_alter_table(): check max column length for every
index in a table, not just added in this particular ALTER TABLE with ADD INDEX ones.
2018-12-11 22:03:44 +03:00
Thirunarayanan Balathandayuthapani
f2c7972a3d MDEV-17432 Assertion `lock_trx_has_sys_table_locks(trx) == 0' failed upon ALTER TABLE .. ADD FOREIGN KEY
- This is a regression of commit b26e603aeb. While dropping
the incompletely created table, InnoDB shouldn't consider that operation as non-atomic one.
2018-12-03 01:12:04 +05:30
Marko Mäkelä
b26e603aeb MDEV-17859 Operating system errors in file operations after failed CREATE
This is a regression due to MDEV-17816.

When creating a table fails, we must roll back the dictionary
transaction. Because the rollback may rename tables, and because
InnoDB lacks proper undo logging for CREATE operations, we must
drop the incompletely created table before rolling back the
transaction, which could include a RENAME operation.
But, we must not blindly drop the table by name; after all,
the operation could have failed because another table by the
same name already existed.

create_table_info_t::m_drop_before_rollback: A flag that is set
if the table needs to be dropped before transaction rollback.

create_table_info_t::create_table(): Remove some duplicated
error handling.

ha_innobase::create(): On error, only drop the table if it was
actually created.
2018-11-28 15:17:56 +02:00
Marko Mäkelä
971e1d8677 MDEV-17831 TRUNCATE TABLE removes ROW_FORMAT=COMPRESSED
If a table had a KEY_BLOCK_SIZE attribute, but no ROW_FORMAT,
it would be created as ROW_FORMAT=COMPRESSED in InnoDB.
However, TRUNCATE TABLE would lose the KEY_BLOCK_SIZE attribute
and create the table with the innodb_default_row_format (DYNAMIC).

This is a regression that was introduced by MDEV-13564.

update_create_info_from_table(): Copy also KEY_BLOCK_SIZE.
2018-11-26 16:41:09 +02:00
Marko Mäkelä
2a31b82831 MDEV-17816 Crash in TRUNCATE TABLE when table creation fails
The error handling in the MDEV-13564 TRUNCATE TABLE was broken
when an error occurred during table creation.

row_create_index_for_mysql(): Do not drop the table on error.

fts_create_one_common_table(), fts_create_one_index_table():
Do drop the table on error.

create_index(), create_table_info_t::create_table():
Let the caller handle the index creation errors.

ha_innobase::create(): If create_table_info_t::create_table()
fails, drop the incomplete table, roll back the transaction,
and finally return an error to the caller.
2018-11-26 12:50:27 +02:00
Marko Mäkelä
3fbee66499 MDEV-17645 innodb.log_file_name_debug does not clean up after itself
The test innodb.log_file_name_debug failed to ensure that
the bogus redo log record that its debug injection emitted
would be consumed by a redo log checkpoint before running a
subsequent test, which could perform crash recovery.

Add an extra shutdown to ensure that a redo log checkpoint is
generated. In this way, the following will succeed:

./mtr --no-reorder innodb.log_file_name_debug innodb.read_only_recovery
2018-11-08 14:50:32 +02:00
Marko Mäkelä
32062cc61c Merge 10.1 into 10.2 2018-11-06 08:41:48 +02:00
Marko Mäkelä
d63e198061 Merge 10.0 into 10.1 2018-11-05 12:15:17 +02:00
Eugene Kosov
03977e8273 MDEV-13671 InnoDB should use case-insensitive column name comparisons like the rest of the server
Problem affects INPLACE ALTER rename columns.

innobase_rename_column_try(): some strcmp() was replaced with my_strcasecmp(),
queries to update data dictionary was updated to not match column name case.
2018-11-05 11:59:59 +02:00
Marko Mäkelä
8a346f31b9 MDEV-17073 INSERT…ON DUPLICATE KEY UPDATE became more deadlock-prone
thd_rpl_stmt_based(): A new predicate to check if statement-based
replication is active. (This can also hold when replication is not
in use, but binlog is.)

que_thr_stop(), row_ins_duplicate_error_in_clust(),
row_ins_sec_index_entry_low(), row_ins(): On a duplicate key error,
only lock all index records when statement-based replication is in use.
2018-11-02 18:52:39 +02:00
Sergei Golubchik
a6e0000494 Merge branch '10.0' into 10.1 2018-10-31 10:53:22 +01:00
Sergei Golubchik
44f6f44593 Merge branch '10.0' into 10.1 2018-10-30 15:10:01 +01:00
Marko Mäkelä
dc91ea5bb7 MDEV-12023 Assertion failure sym_node->table != NULL on startup
row_drop_table_for_mysql(): Avoid accessing non-existing dictionary tables.

dict_create_or_check_foreign_constraint_tables(): Add debug instrumentation
for creating and dropping a table before the creation of any non-core
dictionary tables.

trx_purge_add_update_undo_to_history(): Adjust a debug assertion, so that
it will not fail due to the test instrumentation.
2018-10-30 15:53:55 +02:00
Marko Mäkelä
6ced789186 MDEV-12023 Assertion failure sym_node->table != NULL on startup
row_drop_table_for_mysql(): Avoid accessing non-existing dictionary tables.

dict_create_or_check_foreign_constraint_tables(): Add debug instrumentation
for creating and dropping a table before the creation of any non-core
dictionary tables.

trx_purge_add_update_undo_to_history(): Adjust a debug assertion, so that
it will not fail due to the test instrumentation.
2018-10-30 13:29:19 +02:00
Marko Mäkelä
30c3d6db32 MDEV-17533 Merge new release of InnoDB 5.6.42 to 10.0
Also, add a test for a bug that does not seem to affect MariaDB.
2018-10-25 13:05:23 +03:00
Marko Mäkelä
2549f98289 MDEV-17532 Performance_schema reports wrong directory for the temporary files of ALTER TABLE…ALGORITHM=INPLACE
row_merge_file_create_low(): Pass the directory of the temporary file
to the PSI_FILE_CALL.
2018-10-25 13:04:41 +03:00
Marko Mäkelä
5dd3b52f95 MDEV-17531 Crash in RENAME TABLE with FOREIGN KEY and FULLTEXT INDEX
In RENAME TABLE, when an error occurs while renaming FOREIGN KEY
constraint, that error would be overwritten when renaming the
InnoDB internal tables related to FULLTEXT INDEX.

row_rename_table_for_mysql(): Do not attempt to rename the internal
tables if an error already occurred.

This problem was originally reported as Oracle Bug#27545888.
2018-10-25 13:03:29 +03:00
Marko Mäkelä
a21e01a53d MDEV-17541 KILL QUERY during lock wait in FOREIGN KEY check causes hang
row_ins_check_foreign_constraint(): Do not overwrite hard errors
with the soft error DB_LOCK_WAIT. This prevents an infinite
wait loop when DB_INTERRUPTED was returned. For DB_LOCK_WAIT,
row_insert_for_mysql() would keep invoking row_ins_step() and the
transaction would remain active until the server shutdown is initiated.
2018-10-25 10:35:46 +03:00
Marko Mäkelä
853a0a4368 MDEV-13564: Set innodb_safe_truncate=ON by default
The setting innodb_safe_truncate=ON reduces compatibility with older
versions of MariaDB and backup tools in two ways.

First, we will be writing TRX_UNDO_RENAME_TABLE records, which older
versions do not know about. These records could be misinterpreted if
a DDL transaction was recovered and would be rolled back.
Such rollback is only possible if the server was killed while
an incomplete DDL transaction was persisted. On transaction completion,
the insert_undo log pages would only be repurposed for new undo log
allocations, and their contents would not matter. So, older versions
will not have a problem with innodb_safe_truncate=ON if the server was
shut down cleanly.

Second, to prevent such recovery failure, innodb_safe_truncate=ON will
cause a modification of the redo log format identifier, which will
prevent older versions from starting up after a crash. MariaDB Server
versions older than 10.2.13 will refuse to start up altogether, even
after clean shutdown.

A server restart with innodb_safe_truncate=OFF will restore compatibility
with older server and backup versions.
2018-10-17 17:35:38 +03:00
Marko Mäkelä
81a5b6ccd5 MDEV-17433 Allow InnoDB start up with empty ib_logfile0 from mariabackup --prepare
A prepared backup from Mariabackup does not really need to contain any
redo log file, because all log will have been applied to the data files.

When the user copies a prepared backup to a data directory (overwriting
existing files), it could happen that the data directory already contained
redo log files from the past. mariabackup --copy-back) would delete the
old redo log files, but a user’s own copying script might not do that.
To prevent corruption caused by mixing an old redo log file with data
files from a backup, starting with MDEV-13311, Mariabackup would create
a zero-length ib_logfile0 that would prevent startup.

Actually, there is no need to prevent InnoDB from starting up when a
single zero-length file ib_logfile0 is present. Only if there exist
multiple data files of different lengths, then we should refuse to
start up due to inconsistency. A single zero-length ib_logfile0 should
be treated as if the log files were missing: create new log files
according to the configuration.

open_log_file(): Remove. There is no need to open the log files
at this point, because os_file_get_status() already determined
the size of the file.

innobase_start_or_create_for_mysql(): Move the creation of new
log files a little later, not when finding out that the first log
file does not exist, but after finding out that it does not exist
or it exists as a zero-length file.
2018-10-11 23:00:48 +03:00
Marko Mäkelä
6319c0b541 MDEV-13564: Replace innodb_unsafe_truncate with innodb_safe_truncate
Rename the 10.2-specific configuration option innodb_unsafe_truncate
to innodb_safe_truncate, and invert its value.

The default (for now) is innodb_safe_truncate=OFF, to avoid
disrupting users with an undo and redo log format change within
a Generally Available (GA) release series.
2018-10-11 15:10:13 +03:00
Marko Mäkelä
3448ceb02a MDEV-13564: Implement innodb_unsafe_truncate=ON for compatibility
While MariaDB Server 10.2 is not really guaranteed to be compatible
with Percona XtraBackup 2.4 (for example, the MySQL 5.7 undo log format
change that could be present in XtraBackup, but was reverted from
MariaDB in MDEV-12289), we do not want to disrupt users who have
deployed xtrabackup and MariaDB Server 10.2 in their environments.

With this change, MariaDB 10.2 will continue to use the backup-unsafe
TRUNCATE TABLE code, so that neither the undo log nor the redo log
formats will change in an incompatible way.

Undo tablespace truncation will keep using the redo log only. Recovery
or backup with old code will fail to shrink the undo tablespace files,
but the contents will be recovered just fine.

In the MariaDB Server 10.2 series only, we introduce the configuration
parameter innodb_unsafe_truncate and make it ON by default. To allow
MariaDB Backup (mariabackup) to work properly with TRUNCATE TABLE
operations, use loose_innodb_unsafe_truncate=OFF.

MariaDB Server 10.3.10 and later releases will always use the
backup-safe TRUNCATE TABLE, and this parameter will not be
added there.

recv_recovery_rollback_active(): Skip row_mysql_drop_garbage_tables()
unless innodb_unsafe_truncate=OFF. It is too unsafe to drop orphan
tables if RENAME operations are not transactional within InnoDB.

LOG_HEADER_FORMAT_10_3: Replaces LOG_HEADER_FORMAT_CURRENT.

log_init(), log_group_file_header_flush(),
srv_prepare_to_delete_redo_log_files(),
innobase_start_or_create_for_mysql(): Choose the redo log format
and subformat based on the value of innodb_unsafe_truncate.
2018-10-11 08:17:04 +03:00
Marko Mäkelä
00b6c7d8fc MDEV-16946 innodb.alter_kill failed in buildbot with wrong result
Ensure that no redo log checkpoint occurs in a critical section
of a recovery test.
2018-10-10 06:31:43 +03:00
Marko Mäkelä
1ff22b2062 MDEV-17289: Skip the test for non-debug server 2018-10-06 13:43:13 +03:00
Thirunarayanan Balathandayuthapani
33fadbfefc MDEV-17289: Add a test case 2018-10-05 17:36:31 +03:00
Marko Mäkelä
304857764b MDEV-13564 Mariabackup does not work with TRUNCATE
Implement undo tablespace truncation via normal redo logging.

Implement TRUNCATE TABLE as a combination of RENAME to #sql-ib name,
CREATE, and DROP.
2018-09-25 17:29:43 +03:00
Oleksandr Byelkin
28f08d3753 Merge branch '10.1' into 10.2 2018-09-14 08:47:22 +02:00
Marko Mäkelä
75f8e86f57 MDEV-17158 TRUNCATE is not atomic after MDEV-13564
It turned out that ha_innobase::truncate() would prematurely
commit the transaction already before the completion of the
ha_innobase::create(). All of this must be atomic.

innodb.truncate_crash: Use the correct DEBUG_SYNC point, and
tolerate non-truncation of the table, because the redo log
for the TRUNCATE transaction commit might be flushed due to
some InnoDB background activity.

dict_build_tablespace_for_table(): Merge to the function
dict_build_table_def_step().

dict_build_table_def_step(): If a table is being created during
an already started data dictionary transaction (such as TRUNCATE),
persistently write the table_id to the undo log header before
creating any file. In this way, the recovery of TRUNCATE will be
able to delete the new file before rolling back the rename of
the original table.

dict_table_rename_in_cache(): Add the parameter replace_new_file,
used as part of rolling back a TRUNCATE operation.

fil_rename_tablespace_check(): Add the parameter replace_new.
If the parameter is set and a file identified by new_path exists,
remove a possible tablespace and also the file.

create_table_info_t::create_table_def(): Remove some debug assertions
that no longer hold. During TRUNCATE, the transaction will already
have been started (and performed a rename operation) before the
table is created. Also, remove a call to dict_build_tablespace_for_table().

create_table_info_t::create_table(): Add the parameter create_fk=true.
During TRUNCATE TABLE, do not add FOREIGN KEY constraints to the
InnoDB data dictionary, because they will also not be removed.

row_table_add_foreign_constraints(): If trx=NULL, do not modify
the InnoDB data dictionary, but only load the FOREIGN KEY constraints
from the data dictionary.

ha_innobase::create(): Lock the InnoDB data dictionary cache only
if no transaction was passed by the caller. Unlock it in any case.

innobase_rename_table(): Add the parameter commit = true.
If !commit, do not lock or unlock the data dictionary cache.

ha_innobase::truncate(): Lock the data dictionary before invoking
rename or create, and let ha_innobase::create() unlock it and
also commit or roll back the transaction.

trx_undo_mark_as_dict(): Renamed from trx_undo_mark_as_dict_operation()
and declared global instead of static.

row_undo_ins_parse_undo_rec(): If table_id is set, this must
be rolling back the rename operation in TRUNCATE TABLE, and
therefore replace_new_file=true.
2018-09-10 14:59:58 +03:00
Marko Mäkelä
73ed19e44f MDEV-14585 Automatically remove #sql- tables in InnoDB dictionary during recovery
This is a backport of the following commits:
commit b4165985c9
commit 69e88de0fe
commit 40f4525f43
commit 656f66def2

Now that MDEV-14717 made RENAME TABLE crash-safe within InnoDB,
it should be safe to drop the #sql- tables within InnoDB during
crash recovery. These tables can be one of two things:

(1) #sql-ib related to deferred DROP TABLE (follow-up to MDEV-13407)
or to table-rebuilding ALTER TABLE...ALGORITHM=INPLACE
(since MDEV-14378, only related to the intermediate copy of a table),

(2) #sql- related to the intermediate copy of a table during
ALTER TABLE...ALGORITHM=COPY

We will not drop tables whose name starts with #sql2, because
the server can be killed during an ALGORITHM=COPY operation at
a point where the original table was renamed to #sql2 but the
finished intermediate copy was not yet renamed from #sql-
to the original table name.

If an old version of MariaDB Server before 10.2.13 (MDEV-11415)
was killed while ALTER TABLE...ALGORITHM=COPY was in progress,
after recovery there could be undo log records for some records that were
inserted into an intermediate copy of the table. Due to these undo log
records, InnoDB would resurrect locks at recovery, and the intermediate
table would be locked while we are trying to drop it. This would cause
a call to row_rename_table_for_mysql(), either from
row_mysql_drop_garbage_tables() or from the rollback of a RENAME
operation that was part of the ALTER TABLE.

row_rename_table_for_mysql(): Do not attempt to parse FOREIGN KEY
constraints when renaming from #sql-something to #sql-something-else,
because it does not make any sense.

row_drop_table_for_mysql(): When deferring DROP TABLE due to locks,
do not rename the table if its name already starts with the #sql-
prefix, which is what row_mysql_drop_garbage_tables() uses.
Previously, the too strict prefix #sql-ib was used, and some
tables were renamed unnecessarily.
2018-09-07 22:10:03 +03:00
Marko Mäkelä
8dcacd3b01 Follow-up to MDEV-13407 innodb.drop_table_background failed in buildbot with "Tablespace for table exists"
This is a backport of commit 88aff5f471.

The InnoDB background DROP TABLE queue is something that we should
really remove, but are unable to until we remove dict_operation_lock
so that DDL and DML operations can be combined in a single transaction.

Because the queue is not persistent, it is not crash-safe. We should
in some way ensure that the deferred-dropped tables will be dropped
after server restart.

The existence of two separate transactions complicates the error handling
of CREATE TABLE...SELECT. We should really not break locks in DROP TABLE.

Our solution to these problems is to rename the table to a temporary
name, and to drop such-named tables on InnoDB startup. Also, the
queue will use table IDs instead of names from now on.

check-testcase.test: Ignore #sql-ib*.ibd files, because tables may enter
the background DROP TABLE queue shortly before the test finishes.

innodb.drop_table_background: Test CREATE...SELECT and the creation of
tables whose file name starts with #sql-ib.

innodb.alter_crash: Adjust the recovery, now that the #sql-ib tables
will be dropped on InnoDB startup.

row_mysql_drop_garbage_tables(): New function, to drop all #sql-ib tables
on InnoDB startup.

row_drop_table_for_mysql_in_background(): Remove an unnecessary and
misplaced call to log_buffer_flush_to_disk(). (The call should have been
after the transaction commit. We do not care about flushing the redo log
here, because the table would be dropped again at server startup.)

Remove the entry from the list after the table no longer exists.

If server shutdown has been initiated, empty the list without actually
dropping any tables. They will be dropped again on startup.

row_drop_table_for_mysql(): Do not call lock_remove_all_on_table().
Instead, if locks exist, defer the DROP TABLE until they do not exist.
If the table name does not start with #sql-ib, rename it to that prefix
before adding it to the background DROP TABLE queue.
2018-09-07 22:10:03 +03:00
Marko Mäkelä
754727bb8a MDEV-14378 In ALGORITHM=INPLACE, use a common name for the intermediate tables or partitions
This is a backport of commit 07e9ff1fe1.

Allow DROP TABLE `#mysql50##sql-...._.` to drop tables that were
being rebuilt by ALGORITHM=INPLACE

NOTE: If the server is killed after the table-rebuilding ALGORITHM=INPLACE
commits inside InnoDB but before the .frm file has been replaced, then
the recovery will involve something else than DROP TABLE.

NOTE: If the server is killed in a true inplace ALTER TABLE commits
inside InnoDB but before the .frm file has been replaced, then we
are really out of luck. To properly handle that situation, we would
need a transactional mysql.ddl_fixup table that directs recovery to
rename or remove files.

prepare_inplace_alter_table_dict(): Use the altered_table->s->table_name
for generating the new_table_name.

table_name_t::part_suffix: The start of the partition name suffix.

table_name_t::dbend(): Return the end of the schema name.

table_name_t::dblen(): Return the length of the schema name, in bytes.

table_name_t::basename(): Return the name without the schema name.

table_name_t::part(): Return the partition name, or NULL if none.

row_drop_table_for_mysql(): Assert for #sql, not #sql-ib.
2018-09-07 22:10:03 +03:00
Marko Mäkelä
cf2a4426a2 MDEV-14717 RENAME TABLE in InnoDB is not crash-safe
This is a backport of commit 0bc36758ba
and commit 9eb3fcc9fb.

InnoDB in MariaDB 10.2 appears to only write MLOG_FILE_RENAME2
redo log records during table-rebuilding ALGORITHM=INPLACE operations.
We must write the records for any .ibd file renames, so that the
operations are crash-safe.

If InnoDB is killed during a RENAME TABLE operation, it can happen that
the transaction for updating the data dictionary will be rolled back.
But, nothing will roll back the renaming of the .ibd file
(the MLOG_FILE_RENAME2 only guarantees roll-forward), or for that matter,
the renaming of the dict_table_t::name in the dict_sys cache. We introduce
the undo log record TRX_UNDO_RENAME_TABLE to fix this.

fil_space_for_table_exists_in_mem(): Remove the parameters
adjust_space, table_id and some code that was trying to work around
these deficiencies.

fil_name_write_rename(): Write a MLOG_FILE_RENAME2 record.

dict_table_rename_in_cache(): Invoke fil_name_write_rename().

trx_undo_rec_copy(): Set the first 2 bytes to the length of the
copied undo log record.

trx_undo_page_report_rename(), trx_undo_report_rename():
Write a TRX_UNDO_RENAME_TABLE record with the old table name.

row_rename_table_for_mysql(): Invoke trx_undo_report_rename()
before modifying any data dictionary tables.

row_undo_ins_parse_undo_rec(): Roll back TRX_UNDO_RENAME_TABLE
by invoking dict_table_rename_in_cache(), which will take care
of both renaming the table and the file.

ha_innobase::truncate(): Remove a work-around.
2018-09-07 22:10:02 +03:00
Marko Mäkelä
e67b1070bb MDEV-17049 Enable innodb_undo tests on buildbot
Remove the innodb_undo suite, and move and adapt the tests.
Remove unnecessary restarts, and add innodb_page_size_small.inc
for combinations.

innodb.undo_truncate is the merge of innodb_undo.truncate
and innodb_undo.truncate_multi_client.

Add the global status variable innodb_undo_truncations.
Without this, the test innodb.undo_truncate would occasionally
report that truncation did not happen. The test was only waiting
for the history list length to reach 0, but the undo tablespace
truncation would only take place some time after that.

Undo tablespace truncation will only occasionally occur with
innodb_page_size=32k, and typically never occur (with this amount
of undo log operations) with innodb_page_size=64k. We disable
these combinations.

innodb.undo_truncate_recover was formerly called
innodb_undo.truncate_recover.
2018-09-07 22:10:02 +03:00
Marko Mäkelä
055a3334ad MDEV-13564 Mariabackup does not work with TRUNCATE
Implement undo tablespace truncation via normal redo logging.

Implement TRUNCATE TABLE as a combination of RENAME to #sql-ib name,
CREATE, and DROP.

Note: Orphan #sql-ib*.ibd may be left behind if MariaDB Server 10.2
is killed before the DROP operation is committed. If MariaDB Server 10.2
is killed during TRUNCATE, it is also possible that the old table
was renamed to #sql-ib*.ibd but the data dictionary will refer to the
table using the original name.

In MariaDB Server 10.3, RENAME inside InnoDB is transactional,
and #sql-* tables will be dropped on startup. So, this new TRUNCATE
will be fully crash-safe in 10.3.

ha_mroonga::wrapper_truncate(): Pass table options to the underlying
storage engine, now that ha_innobase::truncate() will need them.

rpl_slave_state::truncate_state_table(): Before truncating
mysql.gtid_slave_pos, evict any cached table handles from
the table definition cache, so that there will be no stale
references to the old table after truncating.

== TRUNCATE TABLE ==

WL#6501 in MySQL 5.7 introduced separate log files for implementing
atomic and crash-safe TRUNCATE TABLE, instead of using the InnoDB
undo and redo log. Some convoluted logic was added to the InnoDB
crash recovery, and some extra synchronization (including a redo log
checkpoint) was introduced to make this work. This synchronization
has caused performance problems and race conditions, and the extra
log files cannot be copied or applied by external backup programs.

In order to support crash-upgrade from MariaDB 10.2, we will keep
the logic for parsing and applying the extra log files, but we will
no longer generate those files in TRUNCATE TABLE.

A prerequisite for crash-safe TRUNCATE is a crash-safe RENAME TABLE
(with full redo and undo logging and proper rollback). This will
be implemented in MDEV-14717.

ha_innobase::truncate(): Invoke RENAME, create(), delete_table().
Because RENAME cannot be fully rolled back before MariaDB 10.3
due to missing undo logging, add some explicit rename-back in
case the operation fails.

ha_innobase::delete(): Introduce a variant that takes sqlcom as
a parameter. In TRUNCATE TABLE, we do not want to touch any
FOREIGN KEY constraints.

ha_innobase::create(): Add the parameters file_per_table, trx.
In TRUNCATE, the new table must be created in the same transaction
that renames the old table.

create_table_info_t::create_table_info_t(): Add the parameters
file_per_table, trx.

row_drop_table_for_mysql(): Replace a bool parameter with sqlcom.

row_drop_table_after_create_fail(): New function, wrapping
row_drop_table_for_mysql().

dict_truncate_index_tree_in_mem(), fil_truncate_tablespace(),
fil_prepare_for_truncate(), fil_reinit_space_header_for_table(),
row_truncate_table_for_mysql(), TruncateLogger,
row_truncate_prepare(), row_truncate_rollback(),
row_truncate_complete(), row_truncate_fts(),
row_truncate_update_system_tables(),
row_truncate_foreign_key_checks(), row_truncate_sanity_checks():
Remove.

row_upd_check_references_constraints(): Remove a check for
TRUNCATE, now that the table is no longer truncated in place.

The new test innodb.truncate_foreign uses DEBUG_SYNC to cover some
race-condition like scenarios. The test innodb-innodb.truncate does
not use any synchronization.

We add a redo log subformat to indicate backup-friendly format.
MariaDB 10.4 will remove support for the old TRUNCATE logging,
so crash-upgrade from old 10.2 or 10.3 to 10.4 will involve
limitations.

== Undo tablespace truncation ==

MySQL 5.7 implements undo tablespace truncation. It is only
possible when innodb_undo_tablespaces is set to at least 2.
The logging is implemented similar to the WL#6501 TRUNCATE,
that is, using separate log files and a redo log checkpoint.

We can simply implement undo tablespace truncation within
a single mini-transaction that reinitializes the undo log
tablespace file. Unfortunately, due to the redo log format
of some operations, currently, the total redo log written by
undo tablespace truncation will be more than the combined size
of the truncated undo tablespace. It should be acceptable
to have a little more than 1 megabyte of log in a single
mini-transaction. This will be fixed in MDEV-17138 in
MariaDB Server 10.4.

recv_sys_t: Add truncated_undo_spaces[] to remember for which undo
tablespaces a MLOG_FILE_CREATE2 record was seen.

namespace undo: Remove some unnecessary declarations.

fil_space_t::is_being_truncated: Document that this flag now
only applies to undo tablespaces. Remove some references.

fil_space_t::is_stopping(): Do not refer to is_being_truncated.
This check is for tablespaces of tables. Potentially used
tablespaces are never truncated any more.

buf_dblwr_process(): Suppress the out-of-bounds warning
for undo tablespaces.

fil_truncate_log(): Write a MLOG_FILE_CREATE2 with a nonzero
page number (new size of the tablespace in pages) to inform
crash recovery that the undo tablespace size has been reduced.

fil_op_write_log(): Relax assertions, so that MLOG_FILE_CREATE2
can be written for undo tablespaces (without .ibd file suffix)
for a nonzero page number.

os_file_truncate(): Add the parameter allow_shrink=false
so that undo tablespaces can actually be shrunk using this function.

fil_name_parse(): For undo tablespace truncation,
buffer MLOG_FILE_CREATE2 in truncated_undo_spaces[].

recv_read_in_area(): Avoid reading pages for which no redo log
records remain buffered, after recv_addr_trim() removed them.

trx_rseg_header_create(): Add a FIXME comment that we could write
much less redo log.

trx_undo_truncate_tablespace(): Reinitialize the undo tablespace
in a single mini-transaction, which will be flushed to the redo log
before the file size is trimmed.

recv_addr_trim(): Discard any redo logs for pages that were
logged after the new end of a file, before the truncation LSN.
If the rec_list becomes empty, reduce n_addrs. After removing
any affected records, actually truncate the file.

recv_apply_hashed_log_recs(): Invoke recv_addr_trim() right before
applying any log records. The undo tablespace files must be open
at this point.

buf_flush_or_remove_pages(), buf_flush_dirty_pages(),
buf_LRU_flush_or_remove_pages(): Add a parameter for specifying
the number of the first page to flush or remove (default 0).

trx_purge_initiate_truncate(): Remove the log checkpoints, the
extra logging, and some unnecessary crash points. Merge the code
from trx_undo_truncate_tablespace(). First, flush all to-be-discarded
pages (beyond the new end of the file), then trim the space->size
to make the page allocation deterministic. At the only remaining
crash injection point, flush the redo log, so that the recovery
can be tested.
2018-09-07 22:10:02 +03:00