This test was using a sleep of 1 second in an attempt to ensure that the
timestamp that is part of an InnoDB status string would increase.
This not only prolongs the test execution time by 1+1 seconds, but it
also is inaccurate. It is possible that the actual sleep duration is
less than a second.
Let us wait for the creation of the file ib_buffer_pool and then wait
for the buffer pool dump completion. In that way, the test can complete
in a dozen or two milliseconds (1% of the previous duration) and work
more reliably.
Omit `state` when selecting processlist to verify which threads are running.
The state changes as threads are running (enter_state()), and this causes
sporadic test failures.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
Let us remove a test that frequently fails with a result difference.
This test had been added in fc279d7ea2
to cover a bug in thd_destructor_proxy(), which was replaced with
simpler logic in 5e62b6a5e0 (MDEV-16264).
- Split the doublewrite test into two test (doublewrite,
doublewrite_debug) to reduce the execution time of the test
- Removed big_test tag for the newly added test case
- Made doublewrite test as non-debug test
- Added search pattern to make sure that InnoDB uses doublewrite buffer
- Replaced all kill_mysqld.inc with shutdown_mysqld.inc and
zero shutdown timeout
- Removed the case where fsp_flags got corrupted. Because from commit
3da5d047b8 (MDEV-31851) onwards,
doublewrite buffer removes the conversion the fsp flags from buggy
10.1 format
Thanks to Marko Mäkelä for providing the non-debug test
srv_export_innodb_status(): Update
export_vars.innodb_buffer_pool_read_requests
with buf_pool.stat.n_page_gets. This is caused due
to incorrect merge commit 44c9008ba6
Problem:
========
- InnoDB should have two keys on the same column for the self
referencing foreign key relation.
Solution:
=========
- Allow self referential foreign key relation to work with one
key.
InnoDB could return off-by-1 estimates for the involved tables.
This would cause off-by-many difference in join output cardinality
for the top-level SELECT, and so different query plan for the subquery.
The fix: Introduce mysql-test/include/innodb_stable_estimates.{inc,opt}
which disables InnoDB's background statistics collection, and use it.
row_upd_clust_rec_by_insert(): If we are resuming from a lock wait,
reset the 'disowned' flag of the BLOB pointers in 'entry' that we
copied from 'rec' on which we had invoked btr_cur_disown_inherited_fields()
before the lock wait started. In this way, the inserted record with
the updated PRIMARY KEY value will have the BLOB ownership associated
with itself, like it is supposed to be.
Note: If the lock wait had been aborted, then rollback would have
invoked btr_cur_unmark_extern_fields() and no corruption would be possible.
Reviewed by: Vladislav Lesin
Tested by: Matthias Leich
trx_t::commit_in_memory(): Empty the detailed_error string, so that
FOREIGN KEY error messages from an earlier transaction will not be
wrongly reused in ha_innobase::get_error_message().
Reviewed by: Thirunarayanan Balathandayuthapani
mysql_prepare_alter_table(): Alter table should check whether
foreign key exists when it expected to exists and
report the error in early stage
dict_foreign_parse_drop_constraints(): Don't throw error if the
foreign key constraints doesn't exist when if exists is given
in the statement.
Let us wait for the completion of purge before testing the KILL of
CREATE INDEX c2d ON t1(c2), so that there will be no table handle
acquisition by a purge task before the operation is rolled back.
Also, let us make the test compatible with ./mtr --repeat,
and convert variable_value from string to integer so that any
comparisons will be performed correctly.
Let us make the test compatible with ./mtr --repeat
and convert variable_value to integer, so that comparisons like
16>9 will work as intended, instead of being compared as '16'<'9'.
In any test that uses wait_all_purged.inc, ensure that InnoDB tables
will be created without persistent statistics.
This is a follow-up to commit cd04673a17
after a similar failure was observed in the innodb_zip.blob test.
Problem:
========
- InnoDB fails to free the allocated buffer of stored cursor
when information schema query is interrupted.
Solution:
=========
- In case of error handling, information schema query should free
the allocated buffer to store the cursor.
Let us tolerate multiple "Memory pressure event freed"
in case there a real memory pressure event occurred
in addition to the one that this test simulates.
Also, clean up some SET variables.
buf_page_t::set_os_unused(): Remove the system call that had been added in
commit 16c9718758 and revised in
commit c1fd082e9c for Microsoft Windows.
buf_pool_t::garbage_collect(): A new function to collect any garbage
from the InnoDB buffer pool that can be removed without writing any
log or data files. This will also invoke madvise() for all of buf_pool.free.
To trigger this the following MDEV is implemented:
MDEV-24670 avoid OOM by linux kernel co-operative memory management
To avoid frequent triggers that caused the MDEV-31953 regression, while
still preserving the 10.11 functionality of non-greedy kernel memory
usage, memory triggers are used.
On the triggering of memory pressure, if supported in the Linux kernel,
trigger the garbage collection of the innodb buffer pool.
The hard coded triggers occur where there is:
* some memory pressure in 5 of the last 10 seconds
* a full stall on memory pressure for 10ms in the last 2 seconds
The kernel will trigger only one in each of these time windows. To avoid
mariadb being in a constant state of memory garbage collection, this has
been limited to once per minute.
For a small set of kernels in 2023 (6.5, 6.6), there was a limit requiring
CAP_SYS_RESOURCE that was lifted[1] to support the use case of user
memory pressure. It not currently possible to set CAP_SYS_RESOURCES in
a systemd service as its setting a capability inside a usernamespace.
Running under systemd v254+ requires the default MemoryPressureWatch=auto
(or alternately "on").
Functionality was tested in a 6.4 kernel Fedora successfully under a
systemd service.
Running in a container requires that (unmask=)/sys/fs/cgroup be writable
by the mariadbd process.
To aid testing, the buf_pool_resize was a convient trigger point on
which to trigger garbage collection.
ref [1]: https://lore.kernel.org/all/CAMw=ZnQ56cm4Txgy5EhGYvR+Jt4s-KVgoA9_65HKWVMOXp7a9A@mail.gmail.com/T/#m3bd2a73c5ee49965cb73a830b1ccaa37ccf4e427
Co-Author: Daniel Black (on memory pressure trigger)
Reviewed by: Marko Mäkelä, Vladislav Vaintroub, Vladislav Lesin,
Thirunarayanan Balathandayuthapani
Tested by: Matthias Leich
dict_find_max_space_id(): Return SELECT MAX(SPACE) FROM SYS_TABLES.
dict_check_tablespaces_and_store_max_id(): In the normal case
(no encryption plugin has been loaded and the change buffer is empty),
invoke dict_find_max_space_id() and do not open any .ibd files.
If a std::set<uint32_t> has been specified, open the files whose
tablespace ID is mentioned. Else, open all data files that are identified
by SYS_TABLES records.
fil_ibd_open(): Remove a call to os_file_get_last_error() that can
report a misleading error, such as EINVAL inside my_realpath() that is
not an actual error. This could be invoked when a data file is found
but the FSP_SPACE_FLAGS are incorrect, such as is the case for
table test.td in
./mtr --mysqld=--innodb-buffer-pool-dump-at-shutdown=0 innodb.table_flags
buf_load(): If any tablespaces could not be found, invoke
dict_check_tablespaces_and_store_max_id() on the missing tablespaces.
dict_load_tablespace(): Try to load the tablespace unless it was found
to be futile. This fixes failures related to FTS_*.ibd files for
FULLTEXT INDEX.
btr_cur_t::search_leaf(): Prevent a crash when the tablespace
does not exist. This was caught by the test innodb_fts.fts_concurrent_insert
when the change to dict_load_tablespaces() was not present.
We modify a few tests to ensure that tables will not be loaded at startup.
For some fault injection tests this means that the corrupted tables
will not be loaded, because dict_load_tablespace() would perform stricter
checks than dict_check_tablespaces_and_store_max_id().
Tested by: Matthias Leich
Reviewed by: Thirunarayanan Balathandayuthapani
ha_innobase::extra(): Do not invoke log_buffer_flush_to_disk()
if high_level_read_only holds.
log_buffer_flush_to_disk(): Remove an assertion that duplicates one
at the start of log_write_up_to().
This test occasionally fails with a failure to purge history.
Let us try to purge everything before starting the interesting part,
to make that occasional failure go away.
The patch for "MDEV-25440: Indexed CHAR ... broken with NO_PAD collations"
fixed these scenarios from MDEV-26743:
- Basic latin letter vs equal accented letter
- Two letters vs equal (but space padded) expansion
However, this scenario was still broken:
- Basic latin letter (but followed by an ignorable character)
vs equal accented letter
Fix:
When processing for a NOPAD collation a string with trailing ignorable
characters, like:
'<non-ignorable><ignorable><ignorable>'
the string gets virtually converted to:
'<non-ignorable><ignorable><ignorable><space><space><space>...'
After the fix the code works differently in these two cases:
1. <space> fits into the "nchars" limit
2. <space> does not fit into the "nchars" limit
Details:
1. If "nchars" is large enough (4+ in this example),
return weights as follows:
'[weight-for-non-ignorable, 1 char] [weight-for-space-character, 3 chars]'
i.e. the weight for the virtual trailing space character now indicates
that it corresponds to total 3 characters:
- two ignorable characters
- one virtual trailing space character
2. If "nchars" is small (3), then the virtual trailing space character
does not fit into the "nchar" limit, so return 0x00 as weight, e.g.:
'[weight-for-non-ignorable, 1 char] [0x00, 2 chars]'
Adding corresponding MTR tests and unit tests.
recv_group_scan_log_recs(): Set the debug flag recv_sys.after_apply
after actually completing the log scan.
In the test, suppress some errors that may be reported when
the crash recovery of RENAME TABLE t1 TO t2 is preceded by
copying t2.ibd to t1.ibd.
This imports and adapts a number of MySQL 5.7 test cases that are
applicable to MariaDB.
Some tests for old bug fixes are not that relevant because the code
has been refactored since then (especially starting with
MariaDB Server 10.6), and the tests would not reproduce the
original bug if the fix was reverted.
In the test innodb_fts.opt, there are many duplicate MATCH ranks, which
would make the results nondeterministic. The test was stabilized by
changing some LIMIT clauses or by adding sorted_result in those cases
where the purpose of a test was to show that no sorting took place
in the server.
In the test innodb_fts.phrase, MySQL 5.7 would generate FTS_DOC_ID that
are 1 larger than in MariaDB. In innodb_fts.index_table the difference is 2.
This is because in MariaDB, fts_get_next_doc_id() post-increments
cache->next_doc_id, while MySQL 5.7 pre-increments it.
Reviewed by: Thirunarayanan Balathandayuthapani
Problem:
=======
- InnoDB fails to find the foreign key index for the newly
added foreign key relation. This is caused by commit
5f09b53bdb (MDEV-31086).
FIX:
===
In check_col_is_in_fk_indexes(), while iterating through
the newly added foreign key relationship, InnoDB should
consider that foreign key relation may not have foreign index
when foreign key check is disabled.
- Introduced the variable "innodb_truncate_temporary_tablespace_now"
to shrink the temporary tablespace.
Steps for shrinking the temporary tablespace:
1) Find the last used extent in temporary tablespace
by iterating through the BITMAP in extent descriptor pages
2) If the last used extent is lesser than user specified size
then set desired target size to user specified size
3) Store the page contents of "to be modified" extent
descriptor pages, latches the "to be modified" extent
descriptor pages and check for buffer pool memory availability
4) Update the FSP_SIZE and FSP_FREE_LIMIT in header page
5) Remove the "to be truncated" pages from FSP_FREE and
FSP_FREE_FRAG list
6) Reset the bitmap in the last descriptor pages for the
"to be truncated" pages.
7) Clear the freed range in temporary tablespace which
are to be truncated.
8) Evict the "to be truncated" temporary tablespace pages
from LRU list.
9) In case of multiple files, calculate the truncated last
file size and do truncation in last file
10) Commit the mini-transaction for shrinking the tablespace
