* no automatic --loose-skip-innodb added by mtr based on the test name.
instead loose-skip-innodb is now in the default_mysqld.cnf
* have_innodb_plugin.inc is changed to give a verbose "skip" message
(instead of "require: true")
* My::Suite class. It's support in mtr, and everywhere
* support for suite.pm
* when sorting tests, take combinations into account
* support for SUITENAME_COMBINATIONS
* no special treatment for innodb_plugin in mtr_cases.pm
* two special pre-created config groups: ENV and OPT
* allow option names to start from #
* allow magic option to have an argument
* remove dead code
* fix @-substitution to works as expected
* new processes take the value of $opt_verbose automatically, no need to pass it to a constructor
* innodb_plugin suite uses suite.pm and combinations file to test as much as possible
(innodb plugin, xtradb plugin, xtradb static - whatever available)
* besides test-master.opt and test-slave.opt a test.opt file is also
loaded, both for master and slave
* .opt files for all included files are loaded too
* progress report in the xterm titlebar
FLUSH TABLES <list> WITH READ LOCK are incompatible" to
be pushed as separate patch.
Replaced thread state name "Waiting for table", which was
used by threads waiting for a metadata lock or table flush,
with a set of names which better reflect types of resources
being waited for.
Also replaced "Table lock" thread state name, which was used
by threads waiting on thr_lock.c table level lock, with more
elaborate "Waiting for table level lock", to make it
more consistent with other thread state names.
Updated test cases and their results according to these
changes.
Fixed sys_vars.query_cache_wlock_invalidate_func test to not
to wait for timeout of wait_condition.inc script.
mysql-test/r/query_cache.result:
Added test coverage for query_cache_wlock_invalidate
behavior for implicitly locked tables.
mysql-test/suite/sys_vars/r/query_cache_wlock_invalidate_func.result:
Fixed sys_vars.query_cache_wlock_invalidate_func test to not
to wait for timeout of wait_condition.inc script. Reverted
changes to test which introduced timeout and replaced waiting
condition with a more appropriate one.
Test coverage for query_cache_wlock_invalidate behavior for
implicitly locked tables was added to query_cache.test.
mysql-test/suite/sys_vars/t/query_cache_wlock_invalidate_func.test:
Fixed sys_vars.query_cache_wlock_invalidate_func test to not
to wait for timeout of wait_condition.inc script. Reverted
changes to test which introduced timeout and replaced waiting
condition with a more appropriate one.
Test coverage for query_cache_wlock_invalidate behavior for
implicitly locked tables was added to query_cache.test.
mysql-test/t/query_cache.test:
Added test coverage for query_cache_wlock_invalidate
behavior for implicitly locked tables.
mysys/thr_lock.c:
Replaced "Table lock" thread state name, which was used by
threads waiting on thr_lock.c table level lock, with more
elaborate "Waiting for table level lock", to make it
consistent with thread state names which are used while
waiting for metadata locks and table flush.
sql/mdl.cc:
Replaced thread state name "Waiting for table", which was
used by threads waiting for a metadata lock or table flush,
with a set of names which better reflect types of resources
being waited for.
To implement this:
- Adjusted MDL_wait::timed_wait() to take thread state name
as parameter.
- Introduced method of MDL_key class which allows to get
thread state name to be used while waiting for resource
corresponding to the key and changed code to use it.
Added array translating namespaces to thread state names
as part of this change.
sql/mdl.h:
To implement this:
- Adjusted MDL_wait::timed_wait() to take thread state name
as parameter.
- Introduced method of MDL_key class which allows to get
thread state name to be used while waiting for resource
corresponding to the key and changed code to use it.
Added array translating namespaces to thread state names
as part of this change.
sql/sql_base.cc:
Replaced thread state name "Waiting for table", which was
used by threads waiting for table flush, with a more elaborate
"Waiting for table flush".
mysql-test/suite/percona/percona_log_slow_slave_statements.test:
Replaced 'grep | wc -l' with 'grep -c' as output of wc -l is not portable (for example on Mac)
mysql-test/suite/percona/percona_slow_query_log-control_global_slow.result:
Replaced 'grep | wc -l' with 'grep -c' as output of wc -l is not portable (for example on Mac)
mysql-test/suite/percona/percona_slow_query_log-control_global_slow.test:
Replaced 'grep | wc -l' with 'grep -c' as output of wc -l is not portable (for example on Mac)
mysql-test/suite/percona/percona_slow_query_log-log_slow_filter.result:
Replaced 'grep | wc -l' with 'grep -c' as output of wc -l is not portable (for example on Mac)
mysql-test/suite/percona/percona_slow_query_log-log_slow_filter.test:
Replaced 'grep | wc -l' with 'grep -c' as output of wc -l is not portable (for example on Mac)
mysql-test/suite/percona/percona_slow_query_log-log_slow_verbosity.result:
Replaced 'grep | wc -l' with 'grep -c' as output of wc -l is not portable (for example on Mac)
mysql-test/suite/percona/percona_slow_query_log-log_slow_verbosity.test:
Replaced 'grep | wc -l' with 'grep -c' as output of wc -l is not portable (for example on Mac)
mysql-test/suite/percona/percona_slow_query_log-long_query_time.result:
Replaced 'grep | wc -l' with 'grep -c' as output of wc -l is not portable (for example on Mac)
mysql-test/suite/percona/percona_slow_query_log-long_query_time.test:
Replaced 'grep | wc -l' with 'grep -c' as output of wc -l is not portable (for example on Mac)
mysql-test/suite/percona/percona_slow_query_log-microseconds_in_slow_query_log.result:
Replaced 'grep | wc -l' with 'grep -c' as output of wc -l is not portable (for example on Mac)
mysql-test/suite/percona/percona_slow_query_log-microseconds_in_slow_query_log.test:
Replaced 'grep | wc -l' with 'grep -c' as output of wc -l is not portable (for example on Mac)
mysql-test/suite/percona/percona_slow_query_log-min_examined_row_limit.result:
Replaced 'grep | wc -l' with 'grep -c' as output of wc -l is not portable (for example on Mac)
mysql-test/suite/percona/percona_slow_query_log-min_examined_row_limit.test:
Replaced 'grep | wc -l' with 'grep -c' as output of wc -l is not portable (for example on Mac)
mysql-test/suite/percona/percona_slow_query_log-use_global_long_query_time.result:
Replaced 'grep | wc -l' with 'grep -c' as output of wc -l is not portable (for example on Mac)
mysql-test/suite/percona/percona_slow_query_log-use_global_long_query_time.test:
Replaced 'grep | wc -l' with 'grep -c' as output of wc -l is not portable (for example on Mac)
mysql-test/suite/rpl/r/rpl_stm_until.result:
Updated results
mysql-test/suite/rpl/t/rpl_stm_until.test:
Fixed random failure in test suite: master could be reading data from master-bin.00001 at the same time as test suite was changing it.
With statement- or mixed-mode logging, "LOAD DATA INFILE" queries
are written to the binlog using special types of log events.
When mysqlbinlog reads such events, it re-creates the file in a
temporary directory with a generated filename and outputs a
"LOAD DATA INFILE" query where the filename is replaced by the
generated file. The temporary file is not deleted by mysqlbinlog
after termination.
To fix the problem, in mixed mode we go to row-based. In SBR, we
document it to remind user the tmpfile is left in a temporary
directory.
mysql-test/suite/binlog/r/binlog_mixed_load_data.result:
Test result for BUG#34283.
mysql-test/suite/binlog/t/binlog_killed_simulate.test:
Updated for BUg#34283
mysql-test/suite/binlog/t/binlog_mixed_load_data.test:
Added the test file to verify that 'load data infile...' statement
will go to row-based in mixed mode.
mysql-test/suite/binlog/t/binlog_stm_blackhole.test:
Updated for BUg#34283
mysql-test/suite/rpl/r/rpl_innodb_mixed_dml.result:
Updated for Bug#34283
mysql-test/suite/rpl/t/rpl_loaddata.test:
Updated for Bug#34283
mysql-test/suite/rpl/t/rpl_loaddata_fatal.test:
Updated for Bug#34283
mysql-test/suite/rpl/t/rpl_loaddata_map.test:
Updated for Bug#34283
mysql-test/suite/rpl/t/rpl_slave_load_remove_tmpfile.test:
Updated for Bug#34283
mysql-test/suite/rpl/t/rpl_stm_log.test:
Updated for B
mysys/stacktrace.c:
Auto merge
sql/sql_lex.cc:
Auto merg
sql/sql_load.cc:
Added code to go to row-based in mixed mode for
'load data infile ...' statement
With statement- or mixed-mode logging, "LOAD DATA INFILE" queries
are written to the binlog using special types of log events.
When mysqlbinlog reads such events, it re-creates the file in a
temporary directory with a generated filename and outputs a
"LOAD DATA INFILE" query where the filename is replaced by the
generated file. The temporary file is not deleted by mysqlbinlog
after termination.
To fix the problem, in mixed mode we go to row-based. In SBR, we
document it to remind user the tmpfile is left in a temporary
directory.
mysql-test/extra/rpl_tests/rpl_loaddata.test:
Updated for Bug#34283
mysql-test/suite/binlog/r/binlog_mixed_load_data.result:
Test result for BUG#34283.
mysql-test/suite/binlog/t/binlog_killed_simulate.test:
Updated for Bug#34283
mysql-test/suite/binlog/t/binlog_mixed_load_data.test:
Added the test file to verify that 'load data infile...' statement
will go to row-based in mixed mode.
mysql-test/suite/binlog/t/binlog_stm_blackhole.test:
Updated for Bug#34283
mysql-test/suite/rpl/r/rpl_innodb_mixed_dml.result:
Updated for Bug#34283
mysql-test/suite/rpl/t/rpl_loaddata_fatal.test:
Updated for Bug#34283
mysql-test/suite/rpl/t/rpl_loaddata_map.test:
Updated for Bug#34283
mysql-test/suite/rpl/t/rpl_slave_load_remove_tmpfile.test:
Updated for Bug#34283
mysql-test/suite/rpl/t/rpl_stm_log.test:
Updated for Bug#34283
sql/sql_load.cc:
Added code to go to row-based in mixed mode for
'load data infile ...' statement
A CREATE...SELECT that fails is written to the binary log if a non-transactional
statement is updated. If the logging format is ROW, the CREATE statement and the
changes are written to the binary log as distinct events and by consequence the
created table is not rolled back in the slave.
In this patch, we opted to let the slave goes out of sync by not writting to the
binary log the CREATE statement. We do this by simply reseting the binary log's
cache.
mysql-test/suite/rpl/r/rpl_drop.result:
Added a test case.
mysql-test/suite/rpl/t/rpl_drop.test:
Added a test case.
sql/log.cc:
Introduced a function to clean up the cache.
sql/log.h:
Introduced a function to clean up the cache.
sql/sql_insert.cc:
Cleaned up the binary log cache if a CREATE...SELECT fails.
Speed up some PBXT tests by adding begin...commit around creating of testing tables.
include/my_base.h:
Fixed wrong constant
mysql-test/mysql-test-run.pl:
Print MariaDB instead of MySQL
mysql-test/r/range.result:
Move test that required partitions to parts.optimizer
mysql-test/suite/innodb_plugin/t/disabled.def:
Disable test that causes valgrind warning about not released memory in xtradb
mysql-test/suite/parts/r/optimizer.result:
Moved from range.result
mysql-test/suite/parts/t/optimizer.test:
Moved from range.test
mysql-test/suite/pbxt/r/join_nested.result:
Updated results after optimizer changes
mysql-test/suite/pbxt/r/renamedb.result:
Updated test for new error message
mysql-test/suite/pbxt/t/check.test:
Speed up test
mysql-test/suite/pbxt/t/count_distinct2.test:
Speed up test
mysql-test/suite/pbxt/t/derived.test:
Speed up test
mysql-test/suite/pbxt/t/renamedb.test:
Updated test for new error message
mysql-test/suite/rpl/r/rpl_log_pos.result:
Updated results
mysql-test/suite/rpl/t/rpl_log_pos.test:
Update test to read from a position that has 'known wrong' data.
The orignal test read a timestamp, so the error message could differ between runs.
mysql-test/suite/rpl/t/rpl_temporary_errors.test:
Sync to slave to make test predictable
mysql-test/t/events_time_zone.test:
Extend wait to make test predictable
mysql-test/t/range.test:
Move test that required partitions to parts.optimizer
sql/sql_list.h:
Fixed compiler warning
sql/sql_load.cc:
buffer was not freed in some error conditions
tests/mysql_client_test.c:
Fixed compiler warning
/*![:version:] Query Code */, where [:version:] is a sequence of 5
digits representing the mysql server version(e.g /*!50200 ... */),
is a special comment that the query in it can be executed on those
servers whose versions are larger than the version appearing in the
comment. It leads to a security issue when slave's version is larger
than master's. A malicious user can improve his privileges on slaves.
Because slave SQL thread is running with SUPER privileges, so it can
execute queries that he/she does not have privileges on master.
This bug is fixed with the logic below:
- To replace '!' with ' ' in the magic comments which are not applied on
master. So they become common comments and will not be applied on slave.
- Example:
'INSERT INTO t1 VALUES (1) /*!10000, (2)*/ /*!99999 ,(3)*/
will be binlogged as
'INSERT INTO t1 VALUES (1) /*!10000, (2)*/ /* 99999 ,(3)*/
mysql-test/suite/rpl/t/rpl_conditional_comments.test:
Test the patch for this bug.
sql/mysql_priv.h:
Rename inBuf as rawBuf and remove the const limitation.
sql/sql_lex.cc:
To replace '!' with ' ' in the magic comments which are not applied on
master.
sql/sql_lex.h:
Remove the const limitation on parameter buff, as it can be modified in the function since
this patch.
Add member function yyUnput for Lex_input_stream. It set a character back the query buff.
sql/sql_parse.cc:
Rename inBuf as rawBuf and remove the const limitation.
sql/sql_partition.cc:
Remove the const limitation on parameter part_buff, as it can be modified in the function since
this patch.
sql/sql_partition.h:
Remove the const limitation on parameter part_buff, as it can be modified in the function since
this patch.
sql/table.h:
Remove the const limitation on variable partition_info, as it can be modified since
this patch.
Problem: when SHOW BINLOG EVENTS was issued, it increased the value of
@@session.max_allowed_packet. This allowed a non-root user to increase
the amount of memory used by her thread arbitrarily. Thus, it removes
the bound on the amount of system resources used by a client, so it
presents a security risk (DoS attack).
Fix: it is correct to increase the value of @@session.max_allowed_packet
while executing SHOW BINLOG EVENTS (see BUG 30435). However, the
increase should only be temporary. Thus, the fix is to restore the value
when SHOW BINLOG EVENTS ends.
The value of @@session.max_allowed_packet is also increased in
mysql_binlog_send (i.e., the binlog dump thread). It is not clear if this
can cause any trouble, since normally the client that issues
COM_BINLOG_DUMP will not issue any other commands that would be affected
by the increased value of @@session.max_allowed_packet. However, we
restore the value just in case.
mysql-test/suite/rpl/r/rpl_packet.result:
update result file
mysql-test/suite/rpl/t/rpl_packet.test:
Add test that verifies that @@session.max_allowed_packet does not change
when issuing SHOW BINLOG EVENTS.
Make previous sub-test clean up.
Add comments listing the bugs in this test case.
sql/sql_repl.cc:
Restore the old value of thd->variables.max_allowed_packet at the
end of mysql_binlog_send and mysql_show_binlog_events.
DROP USER
RENAME USER CURRENT_USER() ...
GRANT ... TO CURRENT_USER()
REVOKE ... FROM CURRENT_USER()
ALTER DEFINER = CURRENT_USER() EVENTbut, When these statements are binlogged, CURRENT_USER() just is binlogged
as 'CURRENT_USER()', it is not expanded to the real user name. When slave
executes the log event, 'CURRENT_USER()' is expand to the user of slave
SQL thread, but SQL thread's user name always NULL. This breaks the replication.
After this patch, session's user will be written into query log events
if these statements call CURREN_USER() or 'ALTER EVENT' does not assign a definer.
mysql-test/include/diff_tables.inc:
Expend its abilities.
Now it can diff not only in sessions of 'master' and 'slave', but
other sessions as well.
sql/log_event.cc:
session's user will be written into Query_log_event, if is_current_user_used() is TRUE.
On slave SQL thread, Only thd->invoker is written into Query_log_event,
if it exists.
sql/sql_acl.cc:
On slave SQL thread, grantor should copy from thd->invoker, if it exists
sql/sql_class.h:
On slave SQL thread, thd->invoker is used to store the applying event's
invoker.
BUG#54872 MBR: replication failure caused by using tmp table inside transaction
Changed criteria to classify a statement as unsafe in order to reduce the
number of spurious warnings. So a statement is classified as unsafe when
there is on-going transaction at any point of the execution if:
1. The mixed statement is about to update a transactional table and
a non-transactional table.
2. The mixed statement is about to update a temporary transactional
table and a non-transactional table.
3. The mixed statement is about to update a transactional table and
read from a non-transactional table.
4. The mixed statement is about to update a temporary transactional
table and read from a non-transactional table.
5. The mixed statement is about to update a non-transactional table
and read from a transactional table when the isolation level is
lower than repeatable read.
After updating a transactional table if:
6. The mixed statement is about to update a non-transactional table
and read from a temporary transactional table.
7. The mixed statement is about to update a non-transactional table
and read from a temporary transactional table.
8. The mixed statement is about to update a non-transactionala table
and read from a temporary non-transactional table.
9. The mixed statement is about to update a temporary non-transactional
table and update a non-transactional table.
10. The mixed statement is about to update a temporary non-transactional
table and read from a non-transactional table.
11. A statement is about to update a non-transactional table and the
option variables.binlog_direct_non_trans_update is OFF.
The reason for this is that locks acquired may not protected a concurrent
transaction of interfering in the current execution and by consequence in
the result. So the patch reduced the number of spurious unsafe warnings.
Besides we fixed a regression caused by BUG#51894, which makes temporary
tables to go into the trx-cache if there is an on-going transaction. In
MIXED mode, the patch for BUG#51894 ignores that the trx-cache may have
updates to temporary non-transactional tables that must be written to the
binary log while rolling back the transaction.
So we fix this problem by writing the content of the trx-cache to the
binary log while rolling back a transaction if a non-transactional
temporary table was updated and the binary logging format is MIXED.
switching binlog format to ROW
BUG 52616 fixed the case which the user would switch from STMT to
ROW binlog format, but the server would silently ignore it. After
that fix thd->is_current_stmt_binlog_format_row() reports correct
value at logging time and events are logged in ROW (as expected)
instead of STMT as they were previously and wrongly logged.
However, the fix was only partially complete, because on
disconnect, at THD cleanup, the implicit logging of temporary
tables is conditionally performed. If the binlog_format==ROW and
thd->is_current_stmt_binlog_format_row() is true then DROPs are
not logged. Given that the user can switch from STMT to ROW, this
is wrong because the server cannot tell, just by relying on the
ROW binlog format, that the tables have been dropped before. This
is effectively similar to the MIXED scenario when a switch from
STMT to ROW is triggered.
We fix this by removing this condition from
close_temporary_tables.
mysql-test/extra/binlog_tests/drop_temp_table.test:
Added binlog test case.
mysql-test/suite/binlog/r/binlog_row_drop_tmp_tbl.result:
Result changes because:
- there is a missing drop on three temporary tables
- it now contains results for the test added
mysql-test/suite/binlog/r/binlog_row_mix_innodb_myisam.result:
Result now contains the implicit drop for the temporary table.
mysql-test/suite/binlog/r/binlog_stm_drop_tmp_tbl.result:
Result file changed because it now contains results for added
test case.
mysql-test/suite/rpl/r/rpl_drop_temp.result:
Result file changed because it now contains results for added
test case.
mysql-test/suite/rpl/t/rpl_drop_temp.test:
Added replication test case.
sql/sql_base.cc:
Removed the condition that would make the server to skip
logging implicit drops when ROW binary log format mode was
in use.
Additionally, deployed DBUG_ENTER/RETURN macros.
DROP USER
RENAME USER CURRENT_USER() ...
GRANT ... TO CURRENT_USER()
REVOKE ... FROM CURRENT_USER()
ALTER DEFINER = CURRENT_USER() EVENTbut, When these statements are binlogged, CURRENT_USER() just is binlogged
as 'CURRENT_USER()', it is not expanded to the real user name. When slave
executes the log event, 'CURRENT_USER()' is expand to the user of slave
SQL thread, but SQL thread's user name always NULL. This breaks the replication.
After this patch, session's user will be written into query log events
if these statements call CURREN_USER() or 'ALTER EVENT' does not assign a definer.
mysql-test/include/diff_tables.inc:
Expend its abilities.
Now it can diff not only in sessions of 'master' and 'slave', but
other sessions as well.
mysql-test/include/rpl_diff_tables.inc:
Diff the same table between master and slaves.
sql/log_event.cc:
session's user will be written into Query_log_event, if is_current_user_used() is TRUE.
On slave SQL thread, Only thd->variables.current_user is written into Query_log_event,
if it exists.
sql/sql_acl.cc:
On slave SQL thread, grantor should copy from thd->variables.current_user, if it exists
sql/sql_class.h:
On slave SQL thread, thd->variables.current_user is used to store the applying event's
invoker.
DML flow and SAVEPOINT
The problem was that replication could break if a transaction involving
both transactional and non-transactional tables was rolled back to a
savepoint. It broke if a concurrent connection tried to drop a
transactional table which was locked after the savepoint was set.
This DROP TABLE completed when ROLLBACK TO SAVEPOINT was executed as the
lock on the table was dropped by the transaction. When the slave later
tried to apply the binlog, it would fail as the table would already
have been dropped.
The reason for the problem is that transactions involving both
transactional and non-transactional tables are written fully to the
binlog during ROLLBACK TO SAVEPOINT. At the same time, metadata locks
acquired after a savepoint, were released during ROLLBACK TO SAVEPOINT.
This allowed a second connection to drop a table only used between
SAVEPOINT and ROLLBACK TO SAVEPOINT. Which caused the transaction binlog
to refer to a non-existing table when it was written during ROLLBACK
TO SAVEPOINT.
This patch fixes the problem by not releasing metadata locks when
ROLLBACK TO SAVEPOINT is executed if binlogging is enabled.
The default storage engine is changed from MyISAM to
InnoDB, in all builds except for the embedded server.
In addition, the following system variables are
changed:
* innodb_file_per_table is enabled
* innodb_strict_mode is enabled
* innodb_file_format_name_update is changed
to 'Barracuda'
The test suite is changed so that tests that do not
explicitly include the have_innodb.inc are run with
--default-storage-engine=MyISAM. This is to ease the
transition, so that most regression tests are run
with the same engine as before.
Some tests are disabled for the embedded server
regression test, as the output of certain statements
will be different that for the regular server
(i.e SELECT @@default_storage_engine). This is to
ease transition.
mysql-test/mysql-test-run.pl:
The regression test suite now adds a
--default-storage-engine=MyISAM for all non-innodb
tests. This behaviour can be controlled by the
default-myisam switch in mysql-test-run
mysql-test/t/bootstrap-master.opt:
The bootstrap test can only be run without InnoDB
as it starts several mysqld instances on the same
datadir. This is possible with MyISAM, but not
with InnoDB.
storage/innobase/CMakeLists.txt:
Build InnoDB per default
storage/innobase/handler/ha_innodb.cc:
Change default values for system variables
Enable file_per_table
Enable strict_mode
Upgrade default file format to Barracuda
* fully support --mysqld=--plugin-load=xxxx
* uniformly support all loadable plugins, no need to hard-code
every new plugin in mtr
* autodetect MTR_VS_CONFIG on windows
locks for DML statements and changes the way MDL locks
are acquired/granted in contended case.
Instead of backing-off when a lock conflict is encountered
and waiting for it to go away before restarting open_tables()
process we now wait for lock to be released without releasing
any previously acquired locks. If conflicting lock goes away
we resume opening tables. If waiting leads to a deadlock we
try to resolve it by backing-off and restarting open_tables()
immediately.
As result both waiting for possibility to acquire and
acquiring of a metadata lock now always happen within the
same MDL API call. This has allowed to make release of a lock
and granting it to the most appropriate pending request an
atomic operation.
Thanks to this it became possible to wake up during release
of lock only those waiters which requests can be satisfied
at the moment as well as wake up only one waiter in case
when granting its request would prevent all other requests
from being satisfied. This solves thundering herd problem
which occured in cases when we were releasing some lock and
woke up many waiters for SNRW or X locks (this was the issue
in bug#52289 "performance regression for MyISAM in sysbench
OLTP_RW test".
This also allowed to implement more fair (FIFO) scheduling
among waiters with the same priority.
It also opens the door for introducing new types of requests
for metadata locks such as low-prio SNRW lock which is
necessary in order to support LOCK TABLES LOW_PRIORITY WRITE.
Notice that after this sometimes can report ER_LOCK_DEADLOCK
error in cases in which it has not happened before.
Particularly we will always report this error if waiting for
conflicting lock has happened in the middle of transaction
and resulted in a deadlock. Before this patch the error was
not reported if deadlock could have been resolved by backing
off all metadata locks acquired by the current statement.
mysql-test/r/mdl_sync.result:
Added test coverage for some aspects of deadlock handling in
metadata locking subsystem.
Adjusted test case after removing back-off in general case
when conflicting metadata lock is encountered during
open_tables() (now this happens only if waiting for
conflicting lock to go away leads to a deadlock).
mysql-test/r/sp_sync.result:
Adjusted test case after removing back-off in general case
when conflicting metadata lock is encountered during
open_tables() (now this happens only if waiting for
conflicting lock to go away leads to a deadlock).
mysql-test/suite/perfschema/r/dml_setup_instruments.result:
Adjusted test results after renaming MDL_context::
m_waiting_for_lock rwlock to m_LOCK_waiting_for.
mysql-test/suite/rpl/r/rpl_sp.result:
Adjusted test case after implementing new approach to
acquiring metadata locks in open_tables(). We no longer
release all MDL locks acquired by statement before waiting
for conflicting lock to go away. As result DROP FUNCTION
statement has to wait for DML statement which managed to
acquire metadata lock on function being dropped and now
waits for other conflicting metadata lock to go away.
mysql-test/suite/rpl/t/rpl_sp.test:
Adjusted test case after implementing new approach to
acquiring metadata locks in open_tables(). We no longer
release all MDL locks acquired by statement before waiting
for conflicting lock to go away. As result DROP FUNCTION
statement has to wait for DML statement which managed to
acquire metadata lock on function being dropped and now
waits for other conflicting metadata lock to go away.
mysql-test/t/mdl_sync.test:
Added test coverage for some aspects of deadlock handling in
metadata locking subsystem.
Adjusted test case after removing back-off in general case
when conflicting metadata lock is encountered during
open_tables() (now this happens only if waiting for
conflicting lock to go away leads to a deadlock).
mysql-test/t/sp_sync.test:
Adjusted test case after removing back-off in general case
when conflicting metadata lock is encountered during
open_tables() (now this happens only if waiting for
conflicting lock to go away leads to a deadlock).
sql/mdl.cc:
Changed MDL subsystem to support new approach to acquring
metadata locks in open tables and more fair and efficient
scheduling of metadata locks. To implement this:
- Made releasing of the lock and granting it to the most
appropriate pending request atomic operation. As result it
became possible to wake up only those waiters requests from
which can be satisfied at the moment as well as wake-up
only one waiter in case when granting its request would
prevent all other requests from being satisfied.
This solved thundering herd problem which occured in cases
when we were releasing some lock and woke up many waiters
for SNRW or X locks (this was the issue in Bug #52289
"performance regression for MyISAM in sysbench OLTP_RW
test".
To emphasize above changes wake_up_waiters() was renamed
to MDL_context::reschedule_waiters().
- Changed code to add tickets for new requests to the back of
waiters queue and to select tickets to be satisfied from
the head of the queue if possible (this makes scheduling of
requests with the same priority fair). To be able to do
this efficiently we now use for waiting and granted queues
version of I_P_List class which provides fast push_back()
method.
- Members and methods of MDL_context related to sending
and waiting for signal were moved to separate MDL_wait
class.
- Since in order to avoid race conditions we must grant the
lock only to the context which was not chosen as a victim
of deadlock, killed or aborted due to timeout
MDL_wait::set_status() (former awake()) was changed not to
send signal if signal slot is already occupied and to
indicate this fact through its return value. As another
consequence MDL_wait::timed_wait() method was changed to
handle timeout (optionally) and abort due to kill as
signals which make signal slot occupied.
- Renamed MDL_context::acquire_lock_impl() to acquire_lock().
Changed it to be able correctly process requests for shared
locks when there are open HANDLERs, made this method more
optimized for acquisition of shared locks. As part of this
change moved code common between try_acquire_lock() and
acquire_lock() to new try_acquire_lock_impl() method.
Also adjusted acquire_lock()'s code to take into account
the fact that in cases when lock is granted as result of
MDL_context::reschedule_waiters() call (i.e. when it is
granted after waiting for lock conflict to go away)
updating MDL_lock state is responsibility of the thread
calling reschedule_waiters().
- Changed MDL_context::find_deadlock() to send VICTIM
signal even if victim is the context which has initiated
deadlock detection. This is required in order to avoid
races in cases when the same context simultaneously is
chosen as a victim and its request for lock is satisfied.
As result return value of this method became unnecessary
and it was changed to return void.
Adjusted MDL_lock::find_deadlock() method to take into
account that now there can be a discrepancy between
MDL_context::m_waiting_for value being set and real state
of the ticket this member points to.
- Renamed MDL_context::m_waiting_for_lock to m_LOCK_waiting_for
and MDL_context::stop_waiting() to done_waiting_for().
- Finally, removed MDL_context::wait_for_lock() method.
sql/mdl.h:
Changed MDL subsystem to support new approach to acquring
metadata locks in open tables and more fair and efficient
scheduling of metadata locks. To implement this:
- Members and methods of MDL_context related to sending
and waiting for signal were moved to separate MDL_wait
class.
- Since now in order to avoid race conditions we must grant
the lock only to the context which was not chosen as a
victim of deadlock, killed or aborted due to timeout
MDL_wait::set_status (former awake()) was changed not to
send signal if signal slot is already occupied and to
indicate this fact through its return value.
Also NORMAL_WAKE_UP signal became GRANTED, and timeouts
and aborts due to kill became full blown signals rather
than simple return values.
- MDL_wait::timed_wait() now takes extra parameter that
indicates whether signal should be set if timeout is
reached.
- Enabled fast push_back() operation in MDL_context::m_tickets
list to make move_ticket_after_trans_sentinel() method more
efficient.
- Removed MDL_context::wait_for_lock() method.
- Renamed MDL_context::m_waiting_for_lock to m_LOCK_waiting_for
and MDL_context::stop_waiting() to done_waiting_for().
- MDL_context::acquire_lock_impl() became acquire_lock().
- Introduced MDL_context::try_acquire_lock_impl() as a
place for code shared by try_acquire_lock and
acquire_lock().
- Due to fact that now VICTIM signal is sent even if victim
is the context which has initiated deadlock detection
find_deadlock() no longer needs a return value.
sql/sql_base.cc:
Implemented new approach to acquiring metadata locks in
open_tables(). We no longer perform back-off when conflicting
metadata lock is encountered. Instead we wait for this lock
to go away while holding all locks which were acquired so
far. Back-off is only used in situation when further waiting
will cause a deadlock which could be avoided by performing
back-off and restarting open_tables() process. Absence of
waiting between back-off and restart of acquiring metadata
locks can't lead to livelocks as MDL subsystem was changed
to make release of lock and granting it to waiting lock
an atomic action, so back-off will automatically give way
to other participants of deadlock loop.
Accordingly:
- open_table_get_mdl_lock() and open_and_process_routine()
were changed to wait for conflicting metadata lock to
go away without back-off. Only if such wait leads to a
deadlock back-off is requested. As part of this change
new error handler class was introduced which converts,
if possible, ER_LOCK_DEADLOCK error to a request for
back-off and re-start of open_tables() process.
- Open_table_context::recover_from_failed_open() was changed
not to wait in case of metadata lock conflict. Instead we
immediately proceed to re-acquiring locks.
- Open_table_context::request_backoff_action() now always
emits error if back-off is requested in the middle of
transaction as we can't be sure that releasing lock
which were acquired only by current statement will
resolve a deadlock. Before this patch such situations were
successfully detected thanks to the fact that we called
MDL_context::wait_for_lock() method in
recover_from_failed_open().
- In order to avoid deadlocks open_tables() code was adjusted
to flush open HANDLERs for which there are pending requests
for X locks before restarting the process of acquiring
metadata locks.
- Changed close_tables_for_reopen() not to reset MDL_request
for tables belonging to the tail of prelocking list. It is
no longer necessary as these MDL_request objects won't be
used for any waiting.
- Adjusted comment in tdc_wait_for_old_version() to avoid
mentioning removed MDL_context::wait_for_lock() method.
sql/sql_base.h:
As we no longer wait for conflicting metadata lock away in
Open_table_context::recover_from_failed_open() method,
Open_table_context::OT_WAIT_MDL_LOCK action was renamed to
OT_MDL_CONFLICT.
Also Open_table_context::m_failed_mdl_request became
unnecessary and was removed.
sql/sql_plist.h:
Extended I_P_List template to support efficient push_back()
operation if it is parameterized with an appropriate policy
class.
sql/sql_show.cc:
Adjusted code after removal of MDL_context::wait_for_lock()
method. Now if one needs to acquire metadata lock with waiting
one has to use a variant of MDL_context::acquire_lock() method.
When using Unique Keys with nullable parts in RBR, the slave can
choose the wrong row to update. This happens because a table with
an unique key containing nullable parts cannot strictly guarantee
uniqueness. As stated in the manual, for all engines, a UNIQUE
index allows multiple NULL values for columns that can contain
NULL.
We fix this at the slave by extending the checks before assuming
that the row found through an unique index is is the correct
one. This means that when a record (R) is fetched from the storage
engine and a key that is not primary (K) is used, the server does
the following:
- If K is unique and has no nullable parts, it returns R;
- Otherwise, if any field in the before image that is part of K
is null do an index scan;
- If there is no NULL field in the BI part of K, then return R.
A side change: renamed the existing test case file and added a
test case covering the changes in this patch.
breaks
When a "CREATE TEMPORARY TABLE SELECT * FROM" was executed the OPTION_KEEP_LOG was
not set into the thd->variables.option_bits. For that reason, if the transaction
had updated only transactional engines and was rolled back at the end (.e.g due to
a deadlock) the changes were not written to the binary log, including the creation
of the temporary table.
To fix the problem, we have set the OPTION_KEEP_LOG into the
thd->variables.option_bits when a "CREATE TEMPORARY TABLE
SELECT * FROM" is executed.
Fixed some bugs introduced in 5.1.47
Disabled some tests until we have merged with latest Xtradb
configure.in:
Added testing if valgrind/memcheck.h exists
storage/pbxt/src/ha_pbxt.cc:
LOCK_plugin is not anymore locked in init
