PROBLEM
-------
Whenever an fts table is created it registers itself in a queue which
is operated by a background thread whose job is to optimize the
fts tables in background. Additionally we place these fts tables in
non-LRU list so that they cannot be evicted from cache. But in the
scenario when a node is brought up which is already having fts
tables ,we first try to load the fts tables in dictionary ,but we skip
the part where it is added in background queue and in non-LRU list because
the background thread is not yet created,so these tables are loaded
but they can be evicted from the cache. Now coming to the deadlock scenario
1. A Server background thread is trying to evict a table from the cache
because the cache is full,so it scans the LRU list for the tables it can
evict.It finds the fts table (because of the reason explained above)
can be evicted and it takes the dict_sys->mutex (this is a system wide mutex)
submits a request to the background thread to remove this table from queue
and waits it to be completed.
2. In the mean time fts_optimize_thread() is processing another job
in the queue and needs dict_sys->mutex for a small amount of time,
but it cannot get it because it is blocked by the first background thread.
So Thread 1 is waiting for its job to be completed by Thread 2,whereas Thread 2
is waiting for dict_sys->mutex held by thread 1 ,causing the deadlock.
FIX
Problem:
=======
Multiple insert statement in table contains FULLTEXT KEY and a
FTS_DOC_ID column aborts the server if the FTS_DOC_ID exceeds
FTS_DOC_ID_MAX_STEP.
Solution:
========
Remove the exception for first committed insert statement.
Reviewed-by: Jimmy Yang<jimmy.yang@oracle.com>
RB: 18023
Introduced new alter algorithm type called NOCOPY & INSTANT for
inplace alter operation.
NOCOPY - Algorithm refuses any alter operation that would
rebuild the clustered index. It is a subset of INPLACE algorithm.
INSTANT - Algorithm allow any alter operation that would
modify only meta data. It is a subset of NOCOPY algorithm.
Introduce new variable called alter_algorithm. The values are
DEFAULT(0), COPY(1), INPLACE(2), NOCOPY(3), INSTANT(4)
Message to deprecate old_alter_table variable and make it alias
for alter_algorithm variable.
alter_algorithm variable for slave is always set to default.
- Inplace alter shouldn't support if the number of newly added fts index
exceeds 1 even though the table undergoes rebuild. It is a regression of
MDEV-14016
- Galera tests that was not updated with connection change
messages
- Test where out of memory error was changed (We are now using the
standard out of memory error in most places)
- Removed tokudb tests that uses include files that doesn't exist
in MariaDB
- Removed not supported mariadb startup option from option file
To disable debug instrumentation, save and restore the original value
of the variable DEBUG_DBUG. Assigning -d,... will enable the output of
a lot of unrelated DBUG messages to the server error log.
MariaDB inherits the MySQL limitation that ALGORITHM=INPLACE cannot
create more than one FULLTEXT INDEX at a time. As part of the MDEV-11369
Instant ADD COLUMN refactoring, MariaDB 10.3.2 accidentally stopped
enforcing the restriction.
Actually, it is a bug in MySQL 5.6 and MariaDB 10.0 that an ALTER TABLE
statement with multiple ADD FULLTEXT INDEX but without explicit
ALGORITHM=INPLACE would return in an error message, rather than
executing the operation with ALGORITHM=COPY.
ha_innobase::check_if_supported_inplace_alter(): Enforce the restriction
on multiple FULLTEXT INDEX.
prepare_inplace_alter_table_dict(): Replace some code with debug
assertions. A "goto error_handled" at this point would result in
another error, because the reference count of ctx->new_table would be 0.
Issue
=====
The original issue was that the size of a fil_per_table tablespace was calculated
incorrectly during truncate in the presence of an fts index. This incorrect calculation
was fixed as part of BUG#25053705 along with a testcase to reproduce the bug. The
assert that was added as part of it to reproduce the bug was wrong and resulted in
this bug.
Fix
===
Although the assert was removed earlier in a seperate commit as it was blocking the
ntest, this patch replaces the other parts of the code that were added to reproduce
the bug and replaces it with code that tries to reproduce the bug in a different way.
The new code basically tries to tweak conditions so as to simulate the random read
where a page that doesn't exist is tried to be read.
RB: 15890
Reviewed-by: Jimmy Yang <Jimmy.Yang@oracle.com>
Reviewed-by: Satya Bodapati <satya.bodapati@oracle.com>
fts_get_next_doc_id(): Assign the first and subsequent FTS_DOC_ID
in the same way: by post-incrementing the cached value.
If there is a user-specified FTS_DOC_ID, do not touch the internal
sequence.
The field fts_token->position is not initialized in
row_merge_fts_doc_tokenize(). We cannot have that field
without changing the fulltext parser plugin ABI
(adding st_mysql_ftparser_boolean_info::position,
as it was done in MySQL 5.7 in WL#6943).
The InnoDB fulltext parser plugins "ngram" and "Mecab" that were
introduced in MySQL 5.7 do depend on that field. But the simple_parser
does not. Apparently, simple_parser is leaving the field as 0.
So, in our fix we will assume that the missing position field is 0.
In my merge of the MySQL fix for Oracle Bug#23333990 / WL#9513
I overlooked some subsequent revisions to the test, and I also
failed to notice that the test is actually always failing.
Oracle introduced the parameter innodb_stats_include_delete_marked
but failed to consistently take it into account in FOREIGN KEY
constraints that involve CASCADE or SET NULL.
When innodb_stats_include_delete_marked=ON, obviously the purge of
delete-marked records should update the statistics as well.
One more omission was that statistics were never updated on ROLLBACK.
We are fixing that as well, properly taking into account the
parameter innodb_stats_include_delete_marked.
dict_stats_analyze_index_level(): Simplify an expression.
(Using the ternary operator with a constant operand is unnecessary
obfuscation.)
page_scan_method_t: Revert the change done by Oracle. Instead,
examine srv_stats_include_delete_marked directly where it is needed.
dict_stats_update_if_needed(): Renamed from
row_update_statistics_if_needed().
row_update_for_mysql_using_upd_graph(): Assert that the table statistics
are initialized, as guaranteed by ha_innobase::open(). Update the
statistics in a consistent way, both for FOREIGN KEY triggers and
for the main table. If FOREIGN KEY constraints exist, do not dereference
a freed pointer, but cache the proper value of node->is_delete so that
it matches prebuilt->table.
row_purge_record_func(): Update statistics if
innodb_stats_include_delete_marked=ON.
row_undo_ins(): Update statistics (on ROLLBACK of a fresh INSERT).
This is independent of the parameter; the record is not delete-marked.
row_undo_mod(): Update statistics on the ROLLBACK of updating key columns,
or (if innodb_stats_include_delete_marked=OFF) updating delete-marks.
innodb.innodb_stats_persistent: Renamed and extended from
innodb.innodb_stats_del_mark. Reduced the unnecessarily large dataset
from 262,144 to 32 rows. Test both values of the configuration
parameter innodb_stats_include_delete_marked.
Test that purge is updating the statistics.
innodb_fts.innodb_fts_multiple_index: Adjust the result. The test
is performing a ROLLBACK of an INSERT, which now affects the statistics.
include/wait_all_purged.inc: Moved from innodb.innodb_truncate_debug
to its own file.
namely, restart_mysqld_with_option.inc and kill_and_restart_mysqld.inc -
use restart_mysqld.inc instead.
Also remove innodb_wl6501_crash_stripped.inc that wasn't used anywhere.
The test is not expected to crash. With a non-debug server,
Valgrind completes in reasonable time without any failure.
Also, it does not make sense to store and restore parameters
when the parameters are already being restored by a server restart.
Before killing the server, ensure that the redo log for the
incomplete transaction is flushed, so that the AUTO_INCREMENT
sequence will always be updated. Usually the INSERT
transaction would not have persisted the sequence before the
server was killed, but sometimes it could happen, causing
result mismatch.
Note: This test used to be called innodb_fts.innodb_fts_misc_debug.
Do not effectively set DEBUG_DBUG='d' by setting DEBUG_DBUG='-d,...'.
Instead, restore the saved value of DEBUG_DBUG.
Also, split the test innodb_fts.innodb_fts_misc_debug into
innodb_fts.crash_recovery and innodb_fts.misc_debug, and enable
these tests for --valgrind, the latter test for --embedded,
and the former tests for the non-debug server.
Do not effectively set DEBUG_DBUG='d' by setting DEBUG_DBUG='-d,...'.
Instead, restore the saved value of DEBUG_DBUG.
Also, split the test innodb_fts.innodb_fts_misc_debug into
innodb_fts.crash_recovery and innodb_fts.misc_debug, and enable
these tests for --valgrind, the latter test for --embedded,
and the former tests for the non-debug server.
crashes server
This bug is the result of merging the Oracle MySQL follow-up fix
BUG#22963169 MYSQL CRASHES ON CREATE FULLTEXT INDEX
without merging the base bug fix:
Bug#79475 Insert a token of 84 4-bytes chars into fts index causes
server crash.
Unlike the above mentioned fixes in MySQL, our fix will not change
the storage format of fulltext indexes in InnoDB or XtraDB
when a character encoding with mbmaxlen=2 or mbmaxlen=3
and the length of a word is between 128 and 84*mbmaxlen bytes.
The Oracle fix would allocate 2 length bytes for these cases.
Compatibility with other MySQL and MariaDB releases is ensured by
persisting the used maximum length in the SYS_COLUMNS table in the
InnoDB data dictionary.
This fix also removes some unnecessary strcmp() calls when checking
for the legacy default collation my_charset_latin1
(my_charset_latin1.name=="latin1_swedish_ci").
fts_create_one_index_table(): Store the actual length in bytes.
This metadata will be written to the SYS_COLUMNS table.
fts_zip_initialize(): Initialize only the first byte of the buffer.
Actually the code should not even care about this first byte, because
the length is set as 0.
FTX_MAX_WORD_LEN: Define as HA_FT_MAXCHARLEN * 4 aka 336 bytes,
not as 254 bytes.
row_merge_create_fts_sort_index(): Set the actual maximum length of the
column in bytes, similar to fts_create_one_index_table().
row_merge_fts_doc_tokenize(): Remove the redundant parameter word_dtype.
Use the actual maximum length of the column. Calculate the extra_size
in the same way as row_merge_buf_encode() does.
This test and result had been modified in WL#6204, but I missed it,
because the test had been renamed from
innodb_fts.innodb_fts_plugin to innodb_fts.plugin.
crashes server
This bug is the result of merging the Oracle MySQL follow-up fix
BUG#22963169 MYSQL CRASHES ON CREATE FULLTEXT INDEX
without merging the base bug fix:
Bug#79475 Insert a token of 84 4-bytes chars into fts index causes
server crash.
Unlike the above mentioned fixes in MySQL, our fix will not change
the storage format of fulltext indexes in InnoDB or XtraDB
when a character encoding with mbmaxlen=2 or mbmaxlen=3
and the length of a word is between 128 and 84*mbmaxlen bytes.
The Oracle fix would allocate 2 length bytes for these cases.
Compatibility with other MySQL and MariaDB releases is ensured by
persisting the used maximum length in the SYS_COLUMNS table in the
InnoDB data dictionary.
This fix also removes some unnecessary strcmp() calls when checking
for the legacy default collation my_charset_latin1
(my_charset_latin1.name=="latin1_swedish_ci").
fts_create_one_index_table(): Store the actual length in bytes.
This metadata will be written to the SYS_COLUMNS table.
fts_zip_initialize(): Initialize only the first byte of the buffer.
Actually the code should not even care about this first byte, because
the length is set as 0.
FTX_MAX_WORD_LEN: Define as HA_FT_MAXCHARLEN * 4 aka 336 bytes,
not as 254 bytes.
row_merge_create_fts_sort_index(): Set the actual maximum length of the
column in bytes, similar to fts_create_one_index_table().
row_merge_fts_doc_tokenize(): Remove the redundant parameter word_dtype.
Use the actual maximum length of the column. Calculate the extra_size
in the same way as row_merge_buf_encode() does.
