bug #57006 "Deadlock between HANDLER and FLUSH TABLES WITH READ
LOCK" and bug #54673 "It takes too long to get readlock for
'FLUSH TABLES WITH READ LOCK'".
The first bug manifested itself as a deadlock which occurred
when a connection, which had some table open through HANDLER
statement, tried to update some data through DML statement
while another connection tried to execute FLUSH TABLES WITH
READ LOCK concurrently.
What happened was that FTWRL in the second connection managed
to perform first step of GRL acquisition and thus blocked all
upcoming DML. After that it started to wait for table open
through HANDLER statement to be flushed. When the first connection
tried to execute DML it has started to wait for GRL/the second
connection creating deadlock.
The second bug manifested itself as starvation of FLUSH TABLES
WITH READ LOCK statements in cases when there was a constant
stream of concurrent DML statements (in two or more
connections).
This has happened because requests for protection against GRL
which were acquired by DML statements were ignoring presence of
pending GRL and thus the latter was starved.
This patch solves both these problems by re-implementing GRL
using metadata locks.
Similar to the old implementation acquisition of GRL in new
implementation is two-step. During the first step we block
all concurrent DML and DDL statements by acquiring global S
metadata lock (each DML and DDL statement acquires global IX
lock for its duration). During the second step we block commits
by acquiring global S lock in COMMIT namespace (commit code
acquires global IX lock in this namespace).
Note that unlike in old implementation acquisition of
protection against GRL in DML and DDL is semi-automatic.
We assume that any statement which should be blocked by GRL
will either open and acquires write-lock on tables or acquires
metadata locks on objects it is going to modify. For any such
statement global IX metadata lock is automatically acquired
for its duration.
The first problem is solved because waits for GRL become
visible to deadlock detector in metadata locking subsystem
and thus deadlocks like one in the first bug become impossible.
The second problem is solved because global S locks which
are used for GRL implementation are given preference over
IX locks which are acquired by concurrent DML (and we can
switch to fair scheduling in future if needed).
Important change:
FTWRL/GRL no longer blocks DML and DDL on temporary tables.
Before this patch behavior was not consistent in this respect:
in some cases DML/DDL statements on temporary tables were
blocked while in others they were not. Since the main use cases
for FTWRL are various forms of backups and temporary tables are
not preserved during backups we have opted for consistently
allowing DML/DDL on temporary tables during FTWRL/GRL.
Important change:
This patch changes thread state names which are used when
DML/DDL of FTWRL is waiting for global read lock. It is now
either "Waiting for global read lock" or "Waiting for commit
lock" depending on the stage on which FTWRL is.
Incompatible change:
To solve deadlock in events code which was exposed by this
patch we have to replace LOCK_event_metadata mutex with
metadata locks on events. As result we have to prohibit
DDL on events under LOCK TABLES.
This patch also adds extensive test coverage for interaction
of DML/DDL and FTWRL.
Performance of new and old global read lock implementations
in sysbench tests were compared. There were no significant
difference between new and old implementations.
Fixed a bug in Aria when two threads was inserting into the same table and row page and one thread did an abort becasue of duplicate key.
mysys/thr_lock.c:
Fixed valgrind warning
sql/sql_base.cc:
Remove not used variable
storage/maria/ma_bitmap.c:
Added ma_bitmap_lock() & ma_bitmap_unlock() to protect against two threads using the bitmap at the same time.
More DBUG_PRINT()
storage/maria/ma_blockrec.c:
Fixed a bug in Aria when two threads was inserting into the same table and row page and one thread did an abort becasue of duplicate key.
Fix was that we block other threads to modify the bitmap while we are removing the row with a duplicate key.
storage/maria/ma_blockrec.h:
Added ma_bitmap_lock() & ma_bitmap_unlock() to protect against two threads using the bitmap at the same time.
storage/maria/maria_def.h:
Changed flush_all_requested to be a counter.
storage/myisam/mi_locking.c:
Fixed compiler error on windows (typo).
Added locking of lock mutex when updating status in external_unlock() for Aria and MyISAM tables.
Fixed that 'source' command doesn't cause mysql command line tool to exit on error.
DEBUG_EXECUTE() and DEBUG_EVALUATE_IF() should not execute things based on wildcards. (Allows one to run --debug with mysql-test-run scripts that uses @debug)
Fixed several core dump, deadlock and crashed table bugs in handling of LOCK TABLE with MERGE tables:
- Added priority of locks to avoid crashes with MERGE tables.
- Added thr_lock_merge() to allow one to merge two results of thr_lock().
Fixed 'not found row' bug in REPLACE with Aria tables.
Mark MyISAM tables that are part of MERGE with HA_OPEN_MERGE_TABLE and set the locks to have priority THR_LOCK_MERGE_PRIV.
- By sorting MERGE tables last in thr_multi_unlock() it's safer to release and relock them many times (can happen when TRIGGERS are created)
Avoid printing (null) in debug file (to easier find out wrong NULL pointer usage with %s).
client/mysql.cc:
Fixed that 'source' command doesn't cause mysql command line tool to exit on error.
client/mysqltest.cc:
Don't send NULL to fn_format(). (Can cause crash on Solaris when using --debug)
dbug/dbug.c:
DEBUG_EXECUTE() and DEBUG_EVALUATE_IF() should not execute things based on wildcards.
include/my_base.h:
Added flag to signal if one opens a MERGE table.
Added extra() command to signal that one is not part of a MERGE table anymore.
include/thr_lock.h:
Added priority for locks (needed to fix bug in thr_lock when using MERGE tables)
Added option to thr_unlock() if get_status() should be called.
Added prototype for thr_merge_locks().
mysql-test/mysql-test-run.pl:
Ignore crashed table warnings for tables named 'crashed'.
mysql-test/r/merge.result:
Renamed triggers to make debugging easier.
Added some CHECK TABLES to catch errors earlier.
Additional tests.
mysql-test/r/merge_debug.result:
Test of error handling when reopening MERGE tables.
mysql-test/r/udf_query_cache.result:
Added missing flush status
mysql-test/suite/parts/r/partition_repair_myisam.result:
Update results
mysql-test/t/merge.test:
Renamed triggers to make debugging easier.
Added some CHECK TABLES to catch errors earlier.
Additional tests.
mysql-test/t/merge_debug.test:
Test of error handling when reopening MERGE tables.
mysql-test/t/udf_query_cache.test:
Added missing flush status
mysys/my_getopt.c:
Removed not used variable
mysys/my_symlink2.c:
Changed (null) to (NULL) to make it easier to find NULL arguments to DBUG_PRINT() functions.
(On linux, NULL to sprintf is printed 'null')
mysys/thr_lock.c:
Added priority of locks to avoid crashes with MERGE tables.
Added thr_lock_merge() to allow one to merge two results of thr_lock().
- This is needed for MyISAM as all locked table must share the same status. If not, you will not see newly inserted rows in other instances of the table.
If calling thr_unlock() with THR_UNLOCK_UPDATE_STATUS, call update_status() and restore_status() for the locks. This is needed in some rare cases where we call thr_unlock() followed by thr_lock() without calling external_unlock/external_lock in between.
Simplify loop in thr_multi_lock().
Added 'start_trans', which is called at end of thr_multi_lock() when all locks are taken.
- This was needed by Aria to ensure that transaction is started when we got all locks, not at get_status(). Without this, some rows could not be visible when we lock two tables at the same time, causing REPLACE using two tables to fail unexpectedly.
sql/handler.cc:
Add an assert() in handler::print_error() for "impossible errors" (like table is crashed) when --debug-assert-if-crashed-table is used.
sql/lock.cc:
Simplify mysql_lock_tables() code if get_lock_data() returns 0 locks.
Added new parameter to thr_multi_unlock()
In mysql_unlock_read_tables(), call first externa_unlock(), then thr_multi_unlock(); This is same order as we do in mysql_unlock_tables().
Don't abort locks in mysql_lock_abort() for merged tables when a MERGE table is deleted; Would cause a spin lock.
Added call to thr_merge_locks() in mysql_lock_merge() to ensure consistency in thr_locks().
- New locks of same type and table is stored after the old lock to ensure that we get the status from the original lock.
sql/mysql_priv.h:
Added debug_assert_if_crashed_table
sql/mysqld.cc:
Added --debug-assert-if-crashed-table
sql/parse_file.cc:
Don't print '(null)' in DBUG_PRINT of no dir given
sql/set_var.cc:
Increase default size of buffer for @debug variable.
sql/sql_base.cc:
In case of error from reopen_table() in reopen_tables(), call unlock_open_table() and restart loop.
- This fixed bug when we twice deleted same table from open_cache.
Don't take name lock for already name locked table in open_unireg_entry().
- Fixed bug when doing repair in reopen_table().
- In detach_merge_children(), always detach if 'clear_refs' is given. We can't trust parent->children_attached as this function can be called twice, first time with clear_refs set to 0.
sql/sql_class.cc:
Changed printing of (null) to "" in set_thd_proc_info()
sql/sql_parse.cc:
Added DBUG
sql/sql_trigger.cc:
Don't call unlink_open_table() if reopen_table() fails as the table may already be freed.
storage/maria/ma_bitmap.c:
Fixed DBUG_ASSERT() in allocate_tail()
storage/maria/ma_blockrec.c:
Fixed wrong calculation of row length for very small rows in undo_row_update().
- Fixes ASSERT() when doing undo.
storage/maria/ma_blockrec.h:
Added _ma_block_start_trans() and _ma_block_start_trans_no_versioning()
storage/maria/ma_locking.c:
Call _ma_update_status_with_lock() when releasing write locks.
- Fixes potential problem with updating status without the proper lock.
storage/maria/ma_open.c:
Changed to use start_trans() instead of get_status() to ensure that we see all rows in all locked tables when we got the locks.
- Fixed 'not found row' bug in REPLACE with Aria tables.
storage/maria/ma_state.c:
Added _ma_update_status_with_lock() and _ma_block_start_trans().
This is to ensure that we see all rows in all locked tables when we got the locks.
storage/maria/ma_state.h:
Added _ma_update_status_with_lock()
storage/maria/ma_write.c:
More DBUG_PRINT
storage/myisam/mi_check.c:
Fixed error message
storage/myisam/mi_extra.c:
Added HA_EXTRA_DETACH_CHILD:
- Detach MyISAM table to not be part of MERGE table (remove flag & lock priority).
storage/myisam/mi_locking.c:
Call mi_update_status_with_lock() when releasing write locks.
- Fixes potential problem with updating status without the proper lock.
Change to use new HA_OPEN_MERGE_TABLE flag to test if MERGE table.
Added mi_fix_status(), called by thr_merge().
storage/myisam/mi_open.c:
Added marker if part of MERGE table.
Call mi_fix_status() in thr_lock() for transactional tables.
storage/myisam/myisamdef.h:
Change my_once_flag to uint, as it stored different values than just 0/1
Added 'open_flag' to store state given to mi_open()
storage/myisammrg/ha_myisammrg.cc:
Add THR_LOCK_MERGE_PRIV to THR_LOCK_DATA to get MERGE locks sorted after other types of locks.
storage/myisammrg/myrg_locking.c:
Remove windows specific code.
storage/myisammrg/myrg_open.c:
Use HA_OPEN_MERGE_TABLE to mi_open().
Set HA_OPEN_MERGE_TABLE for linked MyISAM tables.
storage/xtradb/buf/buf0buf.c:
Fixed compiler warning
storage/xtradb/buf/buf0lru.c:
Initialize variable that could be used not initialized.
sql_show.cc during rqg_info_schema test on Windows".
Ensure we do not access freed memory when filling
information_schema.views when one of the views
could not be properly opened.
mysql-test/r/information_schema.result:
Update results - a fix for Bug#56540.
mysql-test/t/information_schema.test:
Add a test case for Bug#56540
sql/sql_base.cc:
Push an error into the Diagnostics area
when we return an error.
This directs get_all_tables() to the execution
branch which doesn't involve 'process_table()'
when no table/view was opened.
sql/sql_show.cc:
Do not try to access underlying table fields
when opening of a view failed. The underlying
table is closed in that case, and accessing
its fields may lead to dereferencing a damaged
pointer.
sql_show.cc during rqg_info_schema test on Windows".
Ensure we do not access freed memory when filling
information_schema.views when one of the views
could not be properly opened.
|| thd->in_sub_stmt || (thd->state..
Don't rollback statement transactions if we are in a sub-statement.
This could for example happen for open_ltable() when opening the
general log during execution of a stored procedure.
|| thd->in_sub_stmt || (thd->state..
Don't rollback statement transactions if we are in a sub-statement.
This could for example happen for open_ltable() when opening the
general log during execution of a stored procedure.
thd->in_sub_stmt || (thd->state..
OPTIMIZE TABLE is not directly supported by InnoDB. Instead,
recreate and analyze of the table is done. After recreate,
the table is closed and locks are released before the table
is reopened and locks re-acquired for the analyze phase.
This assertion was triggered if OPTIMIZE TABLE failed to
acquire thr_lock locks before starting the analyze phase.
The assertion tests (among other things) that there no
active statement transaction. However, as part of acquiring
the thr_lock lock, external_lock() is called for InnoDB
tables and this causes a statement transaction to be started.
If thr_multi_lock() later fails (e.g. due to timeout),
the failure handling code causes this assert to be triggered.
This patch fixes the problem by doing rollback of the
current statement transaction in case open_ltable (used by
OPTIMIZE TABLE) fails to acquire thr_lock locks.
Test case added to lock_sync.test.
thd->in_sub_stmt || (thd->state..
OPTIMIZE TABLE is not directly supported by InnoDB. Instead,
recreate and analyze of the table is done. After recreate,
the table is closed and locks are released before the table
is reopened and locks re-acquired for the analyze phase.
This assertion was triggered if OPTIMIZE TABLE failed to
acquire thr_lock locks before starting the analyze phase.
The assertion tests (among other things) that there no
active statement transaction. However, as part of acquiring
the thr_lock lock, external_lock() is called for InnoDB
tables and this causes a statement transaction to be started.
If thr_multi_lock() later fails (e.g. due to timeout),
the failure handling code causes this assert to be triggered.
This patch fixes the problem by doing rollback of the
current statement transaction in case open_ltable (used by
OPTIMIZE TABLE) fails to acquire thr_lock locks.
Test case added to lock_sync.test.
After ALTER TABLE which changed only table's metadata, row-based
binlog sometimes got corrupted since the tablemap was unexpectedly
set to 0 for subsequent updates to the same table.
ALTER TABLE which changed only table's metadata always reset
table_map_id for the table share to 0. Despite the fact that
0 is a valid value for table_map_id, this step caused problems
as it could have created situation in which we had more than
one table share with table_map_id equal 0. If more than one
table with table_map_id are 0 were updated in the same statement,
updates to these different tables were written into the same
rows event. This caused slave server to crash.
This bug happens only on 5.1. It doesn't affect 5.5+.
This patch solves this problem by ensuring that ALTER TABLE
statements which change metadata only never reset table_map_id
to 0. To do this it changes reopen_table() to correctly use
refreshed table_map_id value instead of using the old one/
resetting it.
mysql-test/suite/rpl/r/rpl_alter.result:
Add test for BUG#56226
mysql-test/suite/rpl/t/rpl_alter.test:
Add test for BUG#56226
After ALTER TABLE which changed only table's metadata, row-based
binlog sometimes got corrupted since the tablemap was unexpectedly
set to 0 for subsequent updates to the same table.
ALTER TABLE which changed only table's metadata always reset
table_map_id for the table share to 0. Despite the fact that
0 is a valid value for table_map_id, this step caused problems
as it could have created situation in which we had more than
one table share with table_map_id equal 0. If more than one
table with table_map_id are 0 were updated in the same statement,
updates to these different tables were written into the same
rows event. This caused slave server to crash.
This bug happens only on 5.1. It doesn't affect 5.5+.
This patch solves this problem by ensuring that ALTER TABLE
statements which change metadata only never reset table_map_id
to 0. To do this it changes reopen_table() to correctly use
refreshed table_map_id value instead of using the old one/
resetting it.
argument of inline_mysql_mutex_init in sql_base.cc.
When initializing LOCK_dd_owns_lock_open mutex pass
correct PSI key instead of NULL value.
mysql-test/suite/perfschema/r/dml_setup_instruments.result:
Updated test results after adding P_S instrumentation
for LOCK_dd_owns_lock_open.
sql/sql_base.cc:
When initializing LOCK_dd_owns_lock_open mutex pass
correct PSI key instead of NULL value.
Fix some bugs where we stored values other than 0 or 1 in my_bool
Fixed some compiler warnings
client/mysql.cc:
Changed interrupted_query from my_bool to int, as we stored 2 in it.
client/mysqladmin.cc:
Changed return variable type to same type as function value type
client/mysqltest.cc:
Changed 'found' to int as we store other values than 0 or 1 into it
Changed type for parameter of set_reconnect() to match usage.
extra/libevent/evbuffer.c:
Added __attribute__((unused))
extra/libevent/event.c:
Added __attribute__((unused))
extra/libevent/signal.c:
Added __attribute__((unused))
sql/event_data_objects.h:
my_bool -> bool
sql/event_db_repository.cc:
my_bool -> bool
sql/event_db_repository.h:
my_bool -> bool
sql/event_parse_data.h:
my_bool -> bool
sql/events.cc:
my_bool -> bool
sql/events.h:
my_bool -> bool
sql/field.cc:
my_bool -> bool
sql/field.h:
my_bool -> bool
sql/hash_filo.h:
my_bool -> bool
sql/item.cc:
my_bool -> bool
sql/item.h:
my_bool -> bool
sql/item_cmpfunc.h:
my_bool -> bool
Changed result_for_null_param from my_bool to int as we stored -1 in it.
sql/item_func.cc:
my_bool -> bool
Modified udf wrapper functions so that the UDF functions would continue to use my_bool. (To keep compatibility with UDF:s)
sql/item_func.h:
my_bool -> bool
sql/item_subselect.h:
my_bool -> bool
sql/item_sum.cc:
Modified udf wrapper functions so that the UDF functions would continue to use my_bool. (To keep compatibility with UDF:s)
sql/parse_file.h:
my_bool -> bool
sql/rpl_mi.h:
my_bool -> bool
sql/sp_rcontext.h:
my_bool -> bool
sql/sql_analyse.h:
my_bool -> bool
sql/sql_base.cc:
Change some assignments so that we don't initialize bool variables with int's.
sql/sql_bitmap.h:
my_bool -> bool
sql/sql_cache.cc:
my_bool -> bool
sql/sql_cache.h:
my_bool -> bool
sql/sql_class.h:
my_bool -> bool
sql/sql_insert.cc:
Change some assignments so that we don't initialize bool variables with int's.
sql/sql_prepare.cc:
my_bool -> bool
sql/table.h:
my_bool -> bool
storage/maria/ma_check.c:
Removed duplicate assignment
strings/decimal.c:
Fixed wrong variable usage.
Don't do complex arithmetic on bool when simple works.
tables".
Attempting to issue an INSERT DELAYED statement for a MERGE
table might have caused a deadlock if it happened as part of
a transaction or under LOCK TABLES, and there was a concurrent
DDL or LOCK TABLES ... WRITE statement which tried to lock one
of its underlying tables.
The problem occurred when a delayed insert handler thread tried
to open a MERGE table and discovered that to do this it had also
to open all underlying tables and hence acquire metadata
locks on them. Since metadata locks on the underlying tables were
not pre-acquired by the connection thread executing INSERT DELAYED,
attempts to do so might lead to waiting. In this case the
connection thread had to wait for the delayed insert thread.
If the thread which was preventing the lock on the underlying table
from being acquired had to wait for the connection thread (due to
this or other metadata locks), a deadlock occurred.
This deadlock was not detected by the MDL deadlock detector since
waiting for the handler thread by the connection thread is not
represented in the wait-for graph.
This patch solves the problem by ensuring that the delayed
insert handler thread never tries to open underlying tables
of a MERGE table. Instead open_tables() is aborted right after
the parent table is opened and a ER_DELAYED_NOT_SUPPORTED
error is emitted (which is passed to the connection thread and
ultimately to the user).
mysql-test/r/merge.result:
Added test for bug #56251 "Deadlock with INSERT DELAYED and
MERGE tables".
mysql-test/t/merge.test:
Added test for bug #56251 "Deadlock with INSERT DELAYED and
MERGE tables".
sql/sql_base.cc:
Changed open_n_lock_single_table() to take prelocking strategy
as an argument instead of always using DML_prelocking_strategy.
sql/sql_base.h:
Changed open_n_lock_single_table() to take prelocking strategy
as an argument instead of always using DML_prelocking_strategy.
Added a version of this function which is compatible with old
signature.
sql/sql_insert.cc:
When opening MERGE table in delayed insert thread stop and emit
ER_DELAYED_NOT_SUPPORTED right after opening main table and
before opening underlying tables. This ensures that we won't
try to acquire metadata lock on underlying tables which might
lead to a deadlock.
This is achieved by using special prelocking strategy which
abort open_tables() process as soon as we discover that we
have opened table with engine which doesn't support delayed
inserts.
tables".
Attempting to issue an INSERT DELAYED statement for a MERGE
table might have caused a deadlock if it happened as part of
a transaction or under LOCK TABLES, and there was a concurrent
DDL or LOCK TABLES ... WRITE statement which tried to lock one
of its underlying tables.
The problem occurred when a delayed insert handler thread tried
to open a MERGE table and discovered that to do this it had also
to open all underlying tables and hence acquire metadata
locks on them. Since metadata locks on the underlying tables were
not pre-acquired by the connection thread executing INSERT DELAYED,
attempts to do so might lead to waiting. In this case the
connection thread had to wait for the delayed insert thread.
If the thread which was preventing the lock on the underlying table
from being acquired had to wait for the connection thread (due to
this or other metadata locks), a deadlock occurred.
This deadlock was not detected by the MDL deadlock detector since
waiting for the handler thread by the connection thread is not
represented in the wait-for graph.
This patch solves the problem by ensuring that the delayed
insert handler thread never tries to open underlying tables
of a MERGE table. Instead open_tables() is aborted right after
the parent table is opened and a ER_DELAYED_NOT_SUPPORTED
error is emitted (which is passed to the connection thread and
ultimately to the user).
table causes assert failure".
Attempting to use FLUSH TABLE table_list WITH READ LOCK
statement for a MERGE table led to an assertion failure if
one of its children was not present in the list of tables
to be flushed. The problem was not visible in non-debug builds.
The assertion failure was caused by the fact that in such
situations FLUSH TABLES table_list WITH READ LOCK implementation
tried to use (e.g. lock) such child tables without acquiring
metadata lock on them. This happened because when opening tables
we assumed metadata locks on all tables were already acquired
earlier during statement execution and a such assumption was
false for MERGE children.
This patch fixes the problem by ensuring at open_tables() time
that we try to acquire metadata locks on all tables to be opened.
For normal tables such requests are satisfied instantly since
locks are already acquired for them. For MERGE children metadata
locks are acquired in normal fashion.
Note that FLUSH TABLES merge_table WITH READ LOCK will lock for
read both the MERGE table and its children but will flush only
the MERGE table. To flush children one has to mention them in table
list explicitly. This is expected behavior and it is consistent with
usage patterns for this statement (e.g. in mysqlhotcopy script).
mysql-test/r/flush.result:
Added test case for bug #55273 "FLUSH TABLE tm WITH READ LOCK
for Merge table causes assert failure".
mysql-test/t/flush.test:
Added test case for bug #55273 "FLUSH TABLE tm WITH READ LOCK
for Merge table causes assert failure".
sql/sql_base.cc:
Changed lock_table_names() to support newly introduced
MYSQL_OPEN_SKIP_SCOPED_MDL_LOCK flag.
sql/sql_base.h:
Introduced MYSQL_OPEN_SKIP_SCOPED_MDL_LOCK flag for
open_tables() and lock_table_names() which allows to skip
acquiring of global and schema-scope locks when SNW, SNRW or
X metadata locks are acquired.
sql/sql_reload.cc:
Changed "FLUSH TABLES table_list WITH READ LOCK" code not to
cause assert about missing metadata locks when MERGE table is
flushed without one of its underlying tables.
To achieve this we no longer call open_and_lock_tables() with
MYSQL_OPEN_HAS_MDL_LOCK flag so this function automatically
acquires metadata locks on MERGE children if such lock has
not been already acquired at earlier stage. Instead we call
this function with MYSQL_OPEN_SKIP_SCOPED_MDL_LOCK flag to
suppress acquiring of global IX lock in order to keep FLUSH
TABLES table_list WITH READ LOCK compatible with FLUSH TABLE
WITH READ LOCK.
Also changed implementation to use lock_table_names() function
for pre-acquiring of metadata locks instead of custom code.
To implement this change moved setting of open_type member for
table list elements to parser.
sql/sql_yacc.yy:
Now we set acceptable type of table for FLUSH TABLES table_list
WITH READ LOCK at parsing time instead of execution time.
table causes assert failure".
Attempting to use FLUSH TABLE table_list WITH READ LOCK
statement for a MERGE table led to an assertion failure if
one of its children was not present in the list of tables
to be flushed. The problem was not visible in non-debug builds.
The assertion failure was caused by the fact that in such
situations FLUSH TABLES table_list WITH READ LOCK implementation
tried to use (e.g. lock) such child tables without acquiring
metadata lock on them. This happened because when opening tables
we assumed metadata locks on all tables were already acquired
earlier during statement execution and a such assumption was
false for MERGE children.
This patch fixes the problem by ensuring at open_tables() time
that we try to acquire metadata locks on all tables to be opened.
For normal tables such requests are satisfied instantly since
locks are already acquired for them. For MERGE children metadata
locks are acquired in normal fashion.
Note that FLUSH TABLES merge_table WITH READ LOCK will lock for
read both the MERGE table and its children but will flush only
the MERGE table. To flush children one has to mention them in table
list explicitly. This is expected behavior and it is consistent with
usage patterns for this statement (e.g. in mysqlhotcopy script).
MDL deadlock detector".
Deadlock could have occurred when workload containing mix
of DML, DDL and FLUSH TABLES statements affecting same
set of tables was executed in heavily concurrent environment.
This deadlock occurred when several connections tried to
perform deadlock detection in metadata locking subsystem.
The first connection started traversing wait-for graph,
encountered sub-graph representing wait for flush, acquired
LOCK_open and dived into sub-graph inspection. When it has
encounterd sub-graph corresponding to wait for metadata lock
and blocked while trying to acquire rd-lock on
MDL_lock::m_rwlock (*) protecting this subgraph, since some
other thread had wr-lock on it. When this wr-lock was released
it could have happened (if there was other pending wr-lock
against this rwlock) that rd-lock from the first connection
was left unsatisfied but at the same time new rd-lock request
from the second connection sneaked in and was satisfied (for
this to be possible second rd- request should come exactly
after wr-lock is released but before pending wr-lock manages
to grab rwlock, which is possible both on Linux and in our
own rwlock implementation). If this second connection
continued traversing wait-for graph and encountered sub-graph
representing wait for flush it tried to acquire LOCK_open
and thus deadlock was created.
This patch tries to workaround this problem but not allowing
deadlock detector to lock LOCK_open mutex if some other thread
doing deadlock detection already owns it and current search
depth is greater than 0. Instead deadlock is reported.
Other possible solutions are either known to have negative
effects on performance or require much more time for proper
implementation and testing.
No test case is provided as this bug is very hard to repeat
in MTR environment but is repeatable with the help of RQG
tests.
sql/mdl.cc:
Moved Deadlock_detection_visitor::m_current_search_depth to
parent class to make it available in
TABLE_SHARE::visit_subgraph().
Added MDL_wait_for_graph_visitor::abort_traversal() method
which allows to abort traversal of a wait-for graph and
report a deadlock.
sql/mdl.h:
Moved Deadlock_detection_visitor::m_current_search_depth to
parent class to make it available in
TABLE_SHARE::visit_subgraph().
Added MDL_wait_for_graph_visitor::abort_traversal() method
which allows to abort traversal of a wait-for graph and
report a deadlock.
sql/sql_base.cc:
Added dd_owns_lock_open counter and mutex protecting it to
track number of connections which do deadlock detection and
own or try to acquire LOCK_open.
sql/sql_base.h:
Added dd_owns_lock_open counter and mutex protecting it to
track number of connections which do deadlock detection and
own or try to acquire LOCK_open.
sql/table.cc:
Workaround bug #56405 but not allowing MDL deadlock detector
to lock LOCK_open mutex if some other thread doing deadlock
detection already owns it and current search depth is greater
than 0. Instead report deadlock.
MDL deadlock detector".
Deadlock could have occurred when workload containing mix
of DML, DDL and FLUSH TABLES statements affecting same
set of tables was executed in heavily concurrent environment.
This deadlock occurred when several connections tried to
perform deadlock detection in metadata locking subsystem.
The first connection started traversing wait-for graph,
encountered sub-graph representing wait for flush, acquired
LOCK_open and dived into sub-graph inspection. When it has
encounterd sub-graph corresponding to wait for metadata lock
and blocked while trying to acquire rd-lock on
MDL_lock::m_rwlock (*) protecting this subgraph, since some
other thread had wr-lock on it. When this wr-lock was released
it could have happened (if there was other pending wr-lock
against this rwlock) that rd-lock from the first connection
was left unsatisfied but at the same time new rd-lock request
from the second connection sneaked in and was satisfied (for
this to be possible second rd- request should come exactly
after wr-lock is released but before pending wr-lock manages
to grab rwlock, which is possible both on Linux and in our
own rwlock implementation). If this second connection
continued traversing wait-for graph and encountered sub-graph
representing wait for flush it tried to acquire LOCK_open
and thus deadlock was created.
This patch tries to workaround this problem but not allowing
deadlock detector to lock LOCK_open mutex if some other thread
doing deadlock detection already owns it and current search
depth is greater than 0. Instead deadlock is reported.
Other possible solutions are either known to have negative
effects on performance or require much more time for proper
implementation and testing.
No test case is provided as this bug is very hard to repeat
in MTR environment but is repeatable with the help of RQG
tests.
Fall back to use ALTER TABLE for engines that doesn't support REPAIR when doing repair for upgrade.
Nicer output from mysql_upgrade and mysql_check
Updated all arrays that used NAME_LEN to use SAFE_NAME_LEN to ensure that we don't break things accidently as names can now have a #mysql50# prefix.
client/mysql_upgrade.c:
If we are using verbose, also run mysqlcheck in verbose mode.
client/mysqlcheck.c:
Add more information if running in verbose mode
Print 'Needs upgrade' instead of complex error if table needs to be upgraded
Don't write connect information if verbose is not 2 or above
mysql-test/r/drop.result:
Updated test and results as we now support full table names
mysql-test/r/grant.result:
Now you get a correct error message if using #mysql with paths
mysql-test/r/show_check.result:
Update results as table names can temporarly be bigger than NAME_LEN (during upgrade)
mysql-test/r/upgrade.result:
Test upgrade for long table names.
mysql-test/suite/funcs_1/r/is_tables_is.result:
Updated old test result (had note been updated in a while)
mysql-test/t/drop.test:
Updated test and results as we now support full table names
mysql-test/t/grant.test:
Now you get a correct error message if using #mysql with paths
mysql-test/t/upgrade.test:
Test upgrade for long table names.
sql/ha_partition.cc:
NAME_LEN -> SAFE_NAME_LEN
sql/item.cc:
NAME_LEN -> SAFE_NAME_LEN
sql/log_event.cc:
NAME_LEN -> SAFE_NAME_LEN
sql/mysql_priv.h:
Added SAFE_NAME_LEN
sql/rpl_filter.cc:
NAME_LEN -> SAFE_NAME_LEN
sql/sp.cc:
NAME_LEN -> SAFE_NAME_LEN
sql/sp_head.cc:
NAME_LEN -> SAFE_NAME_LEN
sql/sql_acl.cc:
NAME_LEN -> SAFE_NAME_LEN
sql/sql_base.cc:
NAME_LEN -> SAFE_NAME_LEN
sql/sql_connect.cc:
NAME_LEN -> SAFE_NAME_LEN
sql/sql_parse.cc:
NAME_LEN -> SAFE_NAME_LEN
sql/sql_prepare.cc:
NAME_LEN -> SAFE_NAME_LEN
sql/sql_select.cc:
NAME_LEN -> SAFE_NAME_LEN
sql/sql_show.cc:
NAME_LEN -> SAFE_NAME_LEN
Enlarge table names for SHOW TABLES to also include optional #mysql50#
sql/sql_table.cc:
Fall back to use ALTER TABLE for engines that doesn't support REPAIR when doing repair for upgrade.
sql/sql_trigger.cc:
NAME_LEN -> SAFE_NAME_LEN
sql/sql_udf.cc:
NAME_LEN -> SAFE_NAME_LEN
sql/sql_view.cc:
NAME_LEN -> SAFE_NAME_LEN
sql/table.cc:
Fixed check_table_name() to not count #mysql50# as part of name
If #mysql50# is part of the name, don't allow path characters in name.
to allow temp table operations) -- prerequisite patch #3.
Rename open_temporary_table() to open_table_uncached().
open_temporary_table() will be introduced in following patches
to open temporary tables for a statement.
to allow temp table operations) -- prerequisite patch #3.
Rename open_temporary_table() to open_table_uncached().
open_temporary_table() will be introduced in following patches
to open temporary tables for a statement.
to allow temp table operations) -- prerequisite patch #2.
Introduce a new form of find_temporary_table() function:
find_temporary_table() by a table key. It will be used
in further patches.
Replace find_temporary_table(table_list->db, table_list->name)
by more appropiate find_temporary_table(table_list) across
the codebase.
to allow temp table operations) -- prerequisite patch #2.
Introduce a new form of find_temporary_table() function:
find_temporary_table() by a table key. It will be used
in further patches.
Replace find_temporary_table(table_list->db, table_list->name)
by more appropiate find_temporary_table(table_list) across
the codebase.
- Changed to still use bcmp() in certain cases becasue
- Faster for short unaligneed strings than memcmp()
- Bettern when using valgrind
- Changed to use my_sprintf() instead of sprintf() to get higher portability for old systems
- Changed code to use MariaDB version of select->skip_record()
- Removed -%::SCCS/s.% from Makefile.am:s to remove automake warnings
to allow temp table operations) -- prerequisite patch #1.
Move a piece of code that initialiazes TABLE instance
after it was successfully opened into a separate function.
This function will be reused in the following patches.
to allow temp table operations) -- prerequisite patch #1.
Move a piece of code that initialiazes TABLE instance
after it was successfully opened into a separate function.
This function will be reused in the following patches.
on CREATE TABLE .. SELECT I_S.PART
This assert was triggered if an InnoDB table was created using
CREATE TABLE ... AS SELECT where the query used an I_S table, and
a view existed in the database. It would also be triggered for
any statement changing an InnoDB table (e.g. INSERT, UPDATE, DELETE)
which had a subquery referencing an I_S table.
The assert was triggered if open_normal_and_derived_tables() failed
and a statement transaction had been started. This will usually not
happen as tables are opened before a statement transaction is started.
However, e.g. CREATE TABLE ... AS SELECT starts a transaction in order
to insert tuples into the new table. And if the subquery references
an I_S table, all current tables and views can be opened in order to
fill the I_S table on the fly. If a view is discovered, open will fail
as it is instructed to open tables only (OPEN_TABLE_ONLY). This would
cause the assert to be triggered.
The assert was added in the patch for Bug#52044 and was therefore
not in any released versions of the server.
This patch fixes the problem by adjusting the assert to take into
consideration the possibility of tables being opened as part of
an I_S query. This is similar to what is already done for
close_tables_for_reopen().
Test case added to information_schema_inno.test.
on CREATE TABLE .. SELECT I_S.PART
This assert was triggered if an InnoDB table was created using
CREATE TABLE ... AS SELECT where the query used an I_S table, and
a view existed in the database. It would also be triggered for
any statement changing an InnoDB table (e.g. INSERT, UPDATE, DELETE)
which had a subquery referencing an I_S table.
The assert was triggered if open_normal_and_derived_tables() failed
and a statement transaction had been started. This will usually not
happen as tables are opened before a statement transaction is started.
However, e.g. CREATE TABLE ... AS SELECT starts a transaction in order
to insert tuples into the new table. And if the subquery references
an I_S table, all current tables and views can be opened in order to
fill the I_S table on the fly. If a view is discovered, open will fail
as it is instructed to open tables only (OPEN_TABLE_ONLY). This would
cause the assert to be triggered.
The assert was added in the patch for Bug#52044 and was therefore
not in any released versions of the server.
This patch fixes the problem by adjusting the assert to take into
consideration the possibility of tables being opened as part of
an I_S query. This is similar to what is already done for
close_tables_for_reopen().
Test case added to information_schema_inno.test.
temp table
This patch introduces two key changes in the replication's behavior.
Firstly, it reverts part of BUG#51894 which puts any update to temporary tables
into the trx-cache. Now, updates to temporary tables are handled according to
the type of their engines as a regular table.
Secondly, an unsafe mixed statement, (i.e. a statement that access transactional
table as well non-transactional or temporary table, and writes to any of them),
are written into the trx-cache in order to minimize errors in the execution when
the statement logging format is in use.
Such changes has a direct impact on which statements are classified as unsafe
statements and thus part of BUG#53259 is reverted.
temp table
This patch introduces two key changes in the replication's behavior.
Firstly, it reverts part of BUG#51894 which puts any update to temporary tables
into the trx-cache. Now, updates to temporary tables are handled according to
the type of their engines as a regular table.
Secondly, an unsafe mixed statement, (i.e. a statement that access transactional
table as well non-transactional or temporary table, and writes to any of them),
are written into the trx-cache in order to minimize errors in the execution when
the statement logging format is in use.
Such changes has a direct impact on which statements are classified as unsafe
statements and thus part of BUG#53259 is reverted.
check_if_table_exists()
This assert was triggered when the server tried to load plugins
while running in embedded server mode. In embedded server mode,
check_if_table_exists() was used to check if mysql.plugin existed
so that ER_NO_SUCH_TABLE could be silently ignored.
The problem was that this check was done without acquiring a metadata
lock on mysql.plugin first. This triggered the assert.
This patch fixes the problem by removing the call to
check_if_table_exists() from plugin_load(). Instead an error handler
which traps ER_NO_SUCH_TABLE is installed before trying to open
mysql.plugin when running in embedded server mode.
No test coverage added since this assert was triggered by
existing tests running in embedded server mode.
sql/sql_base.cc:
Renamed Prelock_error_handler to No_such_table_error_handler
and moved the declaration to sql_base.h to make it usable
in plugin_load().
sql/sql_base.h:
Renamed Prelock_error_handler to No_such_table_error_handler
and moved the declaration to sql_base.h to make it usable
in plugin_load().
sql/sql_plugin.cc:
Removed call to check_if_table_exists() used to check for mysql.plugin
in plugin_load() for embedded server. Instead install error handler
which traps ER_NO_SUCH_TABLE during open_and_lock_tables().
check_if_table_exists()
This assert was triggered when the server tried to load plugins
while running in embedded server mode. In embedded server mode,
check_if_table_exists() was used to check if mysql.plugin existed
so that ER_NO_SUCH_TABLE could be silently ignored.
The problem was that this check was done without acquiring a metadata
lock on mysql.plugin first. This triggered the assert.
This patch fixes the problem by removing the call to
check_if_table_exists() from plugin_load(). Instead an error handler
which traps ER_NO_SUCH_TABLE is installed before trying to open
mysql.plugin when running in embedded server mode.
No test coverage added since this assert was triggered by
existing tests running in embedded server mode.
