database/mariadb
1
0
Fork 0
mirror of https://github.com/MariaDB/server.git synced 2025-07-30 16:24:05 +03:00
Code Activity

Implementation of simple deadlock detection for metadata locks.

Browse Source 
This change is supposed to reduce number of ER_LOCK_DEADLOCK
errors which occur when multi-statement transaction encounters
conflicting metadata lock in cases when waiting is possible.

The idea is not to fail ER_LOCK_DEADLOCK error immediately when
we encounter conflicting metadata lock. Instead we release all
metadata locks acquired by current statement and start to wait
until conflicting lock go away. To avoid deadlocks we use simple
empiric which aborts waiting with ER_LOCK_DEADLOCK error if it
turns out that somebody is waiting for metadata locks owned by
this transaction.

This patch also fixes bug #46273 "MySQL 5.4.4 new MDL: Bug#989
is not fully fixed in case of ALTER".

The bug was that concurrent execution of UPDATE or MULTI-UPDATE
statement as a part of multi-statement transaction that already
has used table being updated and ALTER TABLE statement might have
resulted of loss of isolation between this transaction and ALTER
TABLE statement, which manifested itself as changes performed by
ALTER TABLE becoming visible in transaction and wrong binary log
order as a consequence.

This problem occurred when UPDATE or MULTI-UPDATE's wait in
mysql_lock_tables() call was aborted due to metadata lock
upgrade performed by concurrent ALTER TABLE. After such abort all
metadata locks held by transaction were released but transaction
silently continued to be executed as if nothing has happened.

We solve this problem by changing our code not to release all
locks in such case. Instead we release only locks which were
acquired by current statement and then try to reacquire them
by restarting open/lock tables process. We piggyback on simple
deadlock detector implementation since this change has to be
done anyway for it.

mysql-test/include/handler.inc:
  After introduction of basic deadlock detector for metadata locks
  it became necessary to change parts of test for HANDLER statements
  which covered some of scenarios in which ER_LOCK_DEADLOCK error
  was detected in absence of real deadlock (with new deadlock detector
  this no longer happens).
  Also adjusted test to the fact that HANDLER READ for the table no
  longer will be blocked by ALTER TABLE for the same table which awaits
  for metadata lock upgrade (this is due to removal of mysql_lock_abort()
  from wait_while_table_is_used()).
mysql-test/r/handler_innodb.result:
  After introduction of basic deadlock detector for metadata locks
  it became necessary to change parts of test for HANDLER statements
  which covered some of scenarios in which ER_LOCK_DEADLOCK error
  was detected in absence of real deadlock (with new deadlock detector
  this no longer happens).
  Also adjusted test to the fact that HANDLER READ for the table no
  longer will be blocked by ALTER TABLE for the same table which awaits
  for metadata lock upgrade (this is due to removal of mysql_lock_abort()
  from wait_while_table_is_used()).
mysql-test/r/handler_myisam.result:
  After introduction of basic deadlock detector for metadata locks
  it became necessary to change parts of test for HANDLER statements
  which covered some of scenarios in which ER_LOCK_DEADLOCK error
  was detected in absence of real deadlock (with new deadlock detector
  this no longer happens).
  Also adjusted test to the fact that HANDLER READ for the table no
  longer will be blocked by ALTER TABLE for the same table which awaits
  for metadata lock upgrade (this is due to removal of mysql_lock_abort()
  from wait_while_table_is_used()).
mysql-test/r/mdl_sync.result:
  Added test coverage for basic deadlock detection in metadata
  locking subsystem and for bug #46273 "MySQL 5.4.4 new MDL:
  Bug#989 is not fully fixed in case of ALTER".
mysql-test/r/sp-lock.result:
  Adjusted test coverage for metadata locking for stored routines
  since after introduction of basic deadlock detector for metadata
  locks number of scenarios in which ER_LOCK_DEADLOCK error in
  absence of deadlock has decreased.
mysql-test/t/mdl_sync.test:
  Added test coverage for basic deadlock detection in metadata
  locking subsystem and for bug #46273 "MySQL 5.4.4 new MDL:
  Bug#989 is not fully fixed in case of ALTER".
mysql-test/t/sp-lock.test:
  Adjusted test coverage for metadata locking for stored routines
  since after introduction of basic deadlock detector for metadata
  locks number of scenarios in which ER_LOCK_DEADLOCK error in
  absence of deadlock has decreased.
sql/log_event_old.cc:
  close_tables_for_reopen() now takes one more argument which
  specifies at which point it should stop releasing metadata
  locks acquired by this connection.
sql/mdl.cc:
  Changed metadata locking subsystem to support basic deadlock detection
  with a help of the following simple empiric -- we assume that there is
  a deadlock if there is a connection which has to wait for a metadata
  lock which is currently acquired by some connection which is itself
  waiting to be able to acquire some shared metadata lock.
  
  To implement this change:
  - Added MDL_context::can_wait_lead_to_deadlock()/_impl() methods
    which allow to find out if there is someone waiting for metadata
    lock which is held by the connection and therefore deadlocks are
    possible if this connection is going to wait for some metadata lock.
    To do this added version of MDL_ticket::has_pending_conflicting_lock()
    method which assumes that its caller already owns LOCK_mdl mutex.
  - Changed MDL_context::wait_for_locks() to use one of the above methods
    to check if somebody is waiting for metadata lock owned by this
    context (and therefore deadlock is possible) and emit ER_LOCK_DEADLOCK
    error in this case. Also now we mark context of connections waiting
    inside of this method by setting MDL_context::m_is_waiting_in_mdl
    member. Thanks to this such connection could be waken up if some
    other connection starts waiting for one of its metadata locks and
    so a deadlock can occur.
  - Adjusted notify_shared_lock() to wake up connections which wait inside
    MDL_context::wait_for_locks() while holding shared metadata lock.
  - Changed MDL_ticket::upgrade_shared_lock_to_exclusive() to add
    temporary ticket for exclusive lock to MDL_lock::waiting queue, so
    request for metadata lock upgrade can be properly detected by our
    empiric.
    Also now this method invokes a callback which forces transactions
    holding shared metadata lock on the table to call MDL_context::
    can_wait_lead_to_deadlock() method even if they don't need any new
    metadata locks. Thanks to this such transactions can detect deadlocks/
    livelocks between MDL and table-level locks.
  
  Also reduced timeouts between calls to notify_shared_lock()
  in MDL_ticket::upgrade_shared_lock_to_exclusive() and
  MDL_context::acquire_exclusive_locks(). This was necessary
  to get rid of call to mysql_lock_abort() in wait_while_table_is_used().
  (Now we instead rely on notify_shared_lock() timely calling
  mysql_lock_abort_for_thread() for the table on which lock
  is being upgraded/acquired).
sql/mdl.h:
  - Added a version of MDL_ticket::has_pending_conflicting_lock() method
    to be used in situations when caller already has acquired LOCK_mdl
    mutex.
  - Added MDL_context::can_wait_lead_to_deadlock()/_impl() methods
    which allow to find out if there is someone waiting for metadata lock
    which is held by this connection and thus deadlocks are possible if
    this connections will start waiting for some metadata lock.
  - Added MDL_context::m_is_waiting_in_mdl member to mark connections
    waiting in MDL_context::wait_for_locks() method of metadata locking
    subsystem. Added getter method for this private member to make it
    accessible in notify_shared_lock() auxiliary so we can wake-up such
    connections if they hold shared metadata locks.
  - Finally, added mysql_abort_transactions_with_shared_lock() callback
    to be able force transactions which don't need any new metadata
    locks still call MDL_context::can_wait_lead_to_deadlock() and detect
    some of deadlocks between metadata locks and table-level locks.
sql/mysql_priv.h:
  close_tables_for_reopen() now takes one more argument which
  specifies at which point it should stop releasing metadata
  locks acquired by this connection.
sql/sql_base.cc:
  Changed approach to metadata locking for multi-statement transactions.
  We no longer fail ER_LOCK_DEADLOCK error immediately when we encounter
  conflicting metadata lock. Instead we release all metadata locks
  acquired by current statement and start to wait until conflicting
  locks to go away by calling MDL_context::wait_for_locks() method.
  To avoid deadlocks the latter implements simple empiric which aborts
  waiting with ER_LOCK_DEADLOCK error if it turns out that somebody
  is waiting for metadata locks owned by this transaction.
  
  To implement the change described above:
  - Introduced Open_table_context::m_start_of_statement_svp member to
    store state of metadata locks at the start of the statement.
  - Changed Open_table_context::request_backoff_action() not to
    fail with ER_LOCK_DEADLOCK immediately if back-off is requested
    due to conflicting metadata lock.
  - Added new argument for close_tables_for_reopen() procedure which
    allows to specify subset of metadata locks to be released.
  - Changed open_tables() not to release all metadata locks acquired
    by current transaction when metadata lock conflict is discovered.
    Instead we release only locks acquired by current statement.
  - Changed open_ltable() and open_and_lock_tables_derived() not to emit
    ER_LOCK_DEADLOCK error when mysql_lock_tables() is aborted in
    multi-statement transaction when somebody tries to acquire exclusive
    metadata lock on the table. Instead we release metadata locks acquired
    by current statement and try to wait until they can be re-acquired.
  - Adjusted tdc_wait_for_old_versions() to check if there is someone
    waiting for one of metadata locks held by this connection and run
    deadlock detection in order to avoid deadlocks in some
    situations.
  - Added mysql_abort_transactions_with_shared_lock() callback which
    allows to force transactions holding shared metadata lock on the
    table to call MDL_context::can_wait_lead_to_deadlock() even if they
    don't need any new metadata locks so they can detect potential
    deadlocks between metadata locking subsystem and table-level locks.
  - Adjusted wait_while_table_is_used() not to set TABLE::version to
    0 as it is now done only when necessary by the above-mentioned
    callback. Also removed unnecessary call to mysql_lock_abort().
    Instead we rely on code performing metadata lock upgrade aborting
    waits on the table-level lock for this table by calling
    mysql_lock_abort_for_thread() (invoked by
    mysql_notify_thread_having_shared_lock()). In future this should
    allow to reduce number of scenarios in which we produce
    ER_LOCK_DEADLOCK error even though no real deadlock exists.
sql/sql_class.h:
  Introduced Open_table_context::m_start_of_statement_svp member to
  store state of metadata locks at the start of the statement.
  Replaced Open_table_context::m_can_deadlock member with m_has_locks
  member to reflect the fact that we no longer unconditionally emit
  ER_LOCK_DEADLOCK error for transaction having some metadata locks
  when conflicting metadata lock is discovered.
sql/sql_insert.cc:
  close_tables_for_reopen() now takes one more argument which
  specifies at which point it should stop releasing metadata
  locks acquired by this connection.
sql/sql_plist.h:
  Made I_P_List_iterator<T, B> usable with const lists.
sql/sql_show.cc:
  close_tables_for_reopen() now takes one more argument which
  specifies at which point it should stop releasing metadata
  locks acquired by this connection.
sql/sql_update.cc:
  Changed UPDATE and MULTI-UPDATE code not to release all metadata
  locks when calls to mysql_lock_tables() are aborted. Instead we
  release only locks which are acquired by this statement and then
  try to reacquire them by calling open_tables(). This solves
  bug #46273 "MySQL 5.4.4 new MDL: Bug#989 is not fully fixed in
  case of ALTER".
This commit is contained in:
Dmitry Lenev
2009-12-30 20:53:30 +03:00
parent 1c327209e1
commit 0228c98936
17 changed files with 1189 additions and 262 deletions
Download Patch File Download Diff File Expand all files Collapse all files

165
sql/sql_base.cc
View File

@ -2142,25 +2142,13 @@ bool rename_temporary_table(THD* thd, TABLE *table, const char *db,
bool wait_while_table_is_used(THD *thd, TABLE *table,
enum ha_extra_function function)
{
enum thr_lock_type old_lock_type;
DBUG_ENTER("wait_while_table_is_used");
DBUG_PRINT("enter", ("table: '%s' share: 0x%lx db_stat: %u version: %lu",
table->s->table_name.str, (ulong) table->s,
table->db_stat, table->s->version));
/* Ensure no one can reopen table before it's removed */
pthread_mutex_lock(&LOCK_open);
table->s->version= 0;
pthread_mutex_unlock(&LOCK_open);
old_lock_type= table->reginfo.lock_type;
mysql_lock_abort(thd, table, TRUE); /* end threads waiting on lock */
if (table->mdl_ticket->upgrade_shared_lock_to_exclusive())
{
mysql_lock_downgrade_write(thd, table, old_lock_type);
DBUG_RETURN(TRUE);
}
pthread_mutex_lock(&LOCK_open);
tdc_remove_table(thd, TDC_RT_REMOVE_NOT_OWN,
@ -3722,9 +3710,10 @@ end_with_lock_open:
Open_table_context::Open_table_context(THD *thd)
:m_action(OT_NO_ACTION),
m_can_deadlock((thd->in_multi_stmt_transaction() ||
thd->mdl_context.lt_or_ha_sentinel())&&
thd->mdl_context.has_locks())
m_start_of_statement_svp(thd->mdl_context.mdl_savepoint()),
m_has_locks((thd->in_multi_stmt_transaction() ||
thd->mdl_context.lt_or_ha_sentinel()) &&
thd->mdl_context.has_locks())
{}
@ -3741,12 +3730,22 @@ Open_table_context::
request_backoff_action(enum_open_table_action action_arg)
{
/*
We have met a exclusive metadata lock or a old version of
table and we are inside a transaction that already hold locks.
We can't follow the locking protocol in this scenario as it
might lead to deadlocks.
We are inside a transaction that already holds locks and have
met a broken table or a table which needs re-discovery.
Performing any recovery action requires acquiring an exclusive
metadata lock on this table. Doing that with locks breaks the
metadata locking protocol and might lead to deadlocks,
so we report an error.
However, if we have only met a conflicting lock or an old
TABLE version, and just need to wait for the conflict to
disappear/old version to go away, allow waiting.
While waiting, we use a simple empiric to detect
deadlocks: we never wait on someone who's waiting too.
Waiting will be done after releasing metadata locks acquired
by this statement.
*/
if (m_can_deadlock)
if (m_has_locks && action_arg != OT_WAIT)
{
my_error(ER_LOCK_DEADLOCK, MYF(0));
return TRUE;
@ -4364,7 +4363,7 @@ restart:
elements from the table list (if MERGE tables are involved),
*/
TABLE_LIST *failed_table= *table_to_open;
close_tables_for_reopen(thd, start);
close_tables_for_reopen(thd, start, ot_ctx.start_of_statement_svp());
/*
Here we rely on the fact that 'tables' still points to the valid
@ -4414,7 +4413,8 @@ restart:
{
if (ot_ctx.can_recover_from_failed_open())
{
close_tables_for_reopen(thd, start);
close_tables_for_reopen(thd, start,
ot_ctx.start_of_statement_svp());
if (ot_ctx.recover_from_failed_open(thd, &rt->mdl_request, NULL))
goto err;
@ -4827,14 +4827,14 @@ retry:
while ((error= open_table(thd, table_list, thd->mem_root, &ot_ctx, 0)) &&
ot_ctx.can_recover_from_failed_open())
{
/* We can't back off with an open HANDLER, we don't wait with locks. */
/* We never have an open HANDLER or LOCK TABLES here. */
DBUG_ASSERT(thd->mdl_context.lt_or_ha_sentinel() == NULL);
/*
Even though we have failed to open table we still need to
call release_transactional_locks() to release metadata locks which
might have been acquired successfully.
*/
thd->mdl_context.release_transactional_locks();
thd->mdl_context.rollback_to_savepoint(ot_ctx.start_of_statement_svp());
table_list->mdl_request.ticket= 0;
if (ot_ctx.recover_from_failed_open(thd, &table_list->mdl_request,
table_list))
@ -4876,24 +4876,13 @@ retry:
{
if (refresh)
{
if (ot_ctx.can_deadlock())
{
my_error(ER_LOCK_DEADLOCK, MYF(0));
table= 0;
}
else
{
close_thread_tables(thd);
table_list->table= NULL;
table_list->mdl_request.ticket= NULL;
/*
We can't back off with an open HANDLER,
we don't wait with locks.
*/
DBUG_ASSERT(thd->mdl_context.lt_or_ha_sentinel() == NULL);
thd->mdl_context.release_transactional_locks();
goto retry;
}
close_thread_tables(thd);
table_list->table= NULL;
table_list->mdl_request.ticket= NULL;
/* We never have an open HANDLER or LOCK TABLES here. */
DBUG_ASSERT(thd->mdl_context.lt_or_ha_sentinel() == NULL);
thd->mdl_context.rollback_to_savepoint(ot_ctx.start_of_statement_svp());
goto retry;
}
else
table= 0;
@ -4941,7 +4930,16 @@ bool open_and_lock_tables_derived(THD *thd, TABLE_LIST *tables,
{
uint counter;
bool need_reopen;
bool has_locks= thd->mdl_context.has_locks();
/*
Remember the set of metadata locks which this connection
managed to acquire before the start of the current statement.
It can be either transaction-scope locks, or HANDLER locks,
or LOCK TABLES locks. If mysql_lock_tables() fails with
need_reopen request, we'll use it to instruct
close_tables_for_reopen() to release all locks of this
statement.
*/
MDL_ticket *start_of_statement_svp= thd->mdl_context.mdl_savepoint();
DBUG_ENTER("open_and_lock_tables_derived");
DBUG_PRINT("enter", ("derived handling: %d", derived));
@ -4960,13 +4958,7 @@ bool open_and_lock_tables_derived(THD *thd, TABLE_LIST *tables,
break;
if (!need_reopen)
DBUG_RETURN(TRUE);
if ((thd->in_multi_stmt_transaction() ||
thd->mdl_context.lt_or_ha_sentinel()) && has_locks)
{
my_error(ER_LOCK_DEADLOCK, MYF(0));
DBUG_RETURN(TRUE);
}
close_tables_for_reopen(thd, &tables);
close_tables_for_reopen(thd, &tables, start_of_statement_svp);
}
if (derived &&
(mysql_handle_derived(thd->lex, &mysql_derived_prepare) ||
@ -5280,6 +5272,8 @@ bool lock_tables(THD *thd, TABLE_LIST *tables, uint count,
flags, need_reopen)))
DBUG_RETURN(TRUE);
DEBUG_SYNC(thd, "after_lock_tables_takes_lock");
if (thd->lex->requires_prelocking() &&
thd->lex->sql_command != SQLCOM_LOCK_TABLES)
{
@ -5379,18 +5373,24 @@ bool lock_tables(THD *thd, TABLE_LIST *tables, uint count,
}
/*
/**
Prepare statement for reopening of tables and recalculation of set of
prelocked tables.
SYNOPSIS
close_tables_for_reopen()
thd in Thread context
tables in/out List of tables which we were trying to open and lock
@param[in] thd Thread context.
@param[in,out] tables List of tables which we were trying to open
and lock.
@param[in] start_of_statement_svp MDL savepoint which represents the set
of metadata locks which the current transaction
managed to acquire before execution of the current
statement and to which we should revert before
trying to reopen tables. NULL if no metadata locks
were held and thus all metadata locks should be
released.
*/
void close_tables_for_reopen(THD *thd, TABLE_LIST **tables)
void close_tables_for_reopen(THD *thd, TABLE_LIST **tables,
MDL_ticket *start_of_statement_svp)
{
TABLE_LIST *first_not_own_table= thd->lex->first_not_own_table();
TABLE_LIST *tmp;
@ -5425,13 +5425,7 @@ void close_tables_for_reopen(THD *thd, TABLE_LIST **tables)
for (tmp= first_not_own_table; tmp; tmp= tmp->next_global)
tmp->mdl_request.ticket= NULL;
close_thread_tables(thd);
/* We can't back off with an open HANDLERs, we must not wait with locks. */
DBUG_ASSERT(thd->mdl_context.lt_or_ha_sentinel() == NULL);
/*
Due to the above assert, this effectively releases *all* locks
of this session, so that we can safely wait on tables.
*/
thd->mdl_context.release_transactional_locks();
thd->mdl_context.rollback_to_savepoint(start_of_statement_svp);
}
@ -8413,6 +8407,8 @@ bool mysql_notify_thread_having_shared_lock(THD *thd, THD *in_use)
But in case a thread has an open HANDLER statement,
(and thus already grabbed a metadata lock), it gets
blocked only too late -- at the table cache level.
Starting from 5.5, this could also easily happen in
a multi-statement transaction.
*/
broadcast_refresh();
pthread_mutex_unlock(&LOCK_open);
@ -8420,6 +8416,28 @@ bool mysql_notify_thread_having_shared_lock(THD *thd, THD *in_use)
}
/**
Force transactions holding shared metadata lock on the table to call
MDL_context::can_wait_lead_to_deadlock() even if they don't need any
new metadata locks so they can detect potential deadlocks between
metadata locking subsystem and table-level locks.
@param mdl_key MDL key for the table on which we are upgrading
metadata lock.
*/
void mysql_abort_transactions_with_shared_lock(const MDL_key *mdl_key)
{
if (mdl_key->mdl_namespace() == MDL_key::TABLE)
{
pthread_mutex_lock(&LOCK_open);
tdc_remove_table(NULL, TDC_RT_REMOVE_UNUSED, mdl_key->db_name(),
mdl_key->name());
pthread_mutex_unlock(&LOCK_open);
}
}
/**
Remove all or some (depending on parameter) instances of TABLE and
TABLE_SHARE from the table definition cache.
@ -8525,6 +8543,25 @@ tdc_wait_for_old_versions(THD *thd, MDL_request_list *mdl_requests)
to broadcast on COND_refresh because of this.
*/
mysql_ha_flush(thd);
/*
Check if there is someone waiting for one of metadata locks
held by this connection and return an error if that's the
case, since this situation may lead to a deadlock.
This can happen, when, for example, this connection is
waiting for an old version of some table to go away and
another connection is trying to upgrade its shared
metadata lock to exclusive, and thus is waiting
for this to release its lock. We must check for
the condition on each iteration of the loop to remove
any window for a race.
*/
if (thd->mdl_context.can_wait_lead_to_deadlock())
{
my_error(ER_LOCK_DEADLOCK, MYF(0));
return TRUE;
}
pthread_mutex_lock(&LOCK_open);
MDL_request_list::Iterator it(*mdl_requests);