A pre-requisite patch for the fix for Bug#52044.

This patch also fixes Bug#55452 "SET PASSWORD is
replicated twice in RBR mode".

The goal of this patch is to remove the release of 
metadata locks from close_thread_tables().
This is necessary to not mistakenly release
the locks in the course of a multi-step
operation that involves multiple close_thread_tables()
or close_tables_for_reopen().

On the same token, move statement commit outside 
close_thread_tables().

Other cleanups:
Cleanup COM_FIELD_LIST.
Don't call close_thread_tables() in COM_SHUTDOWN -- there
are no open tables there that can be closed (we leave
the locked tables mode in THD destructor, and this
close_thread_tables() won't leave it anyway).

Make open_and_lock_tables() and open_and_lock_tables_derived()
call close_thread_tables() upon failure.
Remove the calls to close_thread_tables() that are now
unnecessary.

Simplify the back off condition in Open_table_context.

Streamline metadata lock handling in LOCK TABLES 
implementation.

Add asserts to ensure correct life cycle of 
statement transaction in a session.

Remove a piece of dead code that has also become redundant
after the fix for Bug 37521.

mysql-test/r/variables.result:
  Update results: set @@autocommit and statement transaction/
  prelocked mode.
mysql-test/r/view.result:
  A harmless change in CHECK TABLE <view> status for a broken view.
  If previously a failure to prelock all functions used in a view 
  would leave the connection in LTM_PRELOCKED mode, now we call
  close_thread_tables() from open_and_lock_tables()
  and leave prelocked mode, thus some check in mysql_admin_table() that
  works only in prelocked/locked tables mode is no longer activated.
mysql-test/suite/rpl/r/rpl_row_implicit_commit_binlog.result:
  Fixed Bug#55452 "SET PASSWORD is replicated twice in
  RBR mode": extra binlog events are gone from the
  binary log.
mysql-test/t/variables.test:
  Add a test case: set autocommit and statement transaction/prelocked
  mode.
sql/event_data_objects.cc:
  Simplify code in Event_job_data::execute().
  Move sp_head memory management to lex_end().
sql/event_db_repository.cc:
  Move the release of metadata locks outside
  close_thread_tables().
  Make sure we call close_thread_tables() when
  open_and_lock_tables() fails and remove extra
  code from the events data dictionary.
  Use close_mysql_tables(), a new internal
  function to properly close mysql.* tables
  in the data dictionary.
  Contract Event_db_repository::drop_events_by_field,
  drop_schema_events into one function.
  When dropping all events in a schema,
  make sure we don't mistakenly release all
  locks acquired by DROP DATABASE. These
  include locks on the database name
  and the global intention exclusive
  metadata lock.
sql/event_db_repository.h:
  Function open_event_table() does not require an instance 
  of Event_db_repository.
sql/events.cc:
  Use close_mysql_tables() instead of close_thread_tables()
  to bootstrap events, since the latter no longer
  releases metadata locks.
sql/ha_ndbcluster.cc:
  - mysql_rm_table_part2 no longer releases
  acquired metadata locks. Do it in the caller.
sql/ha_ndbcluster_binlog.cc:
  Deploy the new protocol for closing thread
  tables in run_query() and ndb_binlog_index
  code.
sql/handler.cc:
  Assert that we never call ha_commit_trans/
  ha_rollback_trans in sub-statement, which
  is now the case.
sql/handler.h:
  Add an accessor to check whether THD_TRANS object
  is empty (has no transaction started).
sql/log.cc:
  Update a comment.
sql/log_event.cc:
  Since now we commit/rollback statement transaction in 
  mysql_execute_command(), we need a mechanism to communicate
  from Query_log_event::do_apply_event() to mysql_execute_command()
  that the statement transaction should be rolled back, not committed.
  Ideally it would be a virtual method of THD. I hesitate
  to make THD a virtual base class in this already large patch.
  Use a thd->variables.option_bits for now.
  
  Remove a call to close_thread_tables() from the slave IO
  thread. It doesn't open any tables, and the protocol
  for closing thread tables is more complicated now.
  
  Make sure we properly close thread tables, however, 
  in Load_data_log_event, which doesn't
  follow the standard server execution procedure
  with mysql_execute_command().
  @todo: this piece should use Server_runnable
  framework instead.
  Remove an unnecessary call to mysql_unlock_tables().
sql/rpl_rli.cc:
  Update Relay_log_info::slave_close_thread_tables()
  to follow the new close protocol.
sql/set_var.cc:
  Remove an unused header.
sql/slave.cc:
  Remove an unnecessary call to
  close_thread_tables().
sql/sp.cc:
  Remove unnecessary calls to close_thread_tables()
  from SP DDL implementation. The tables will
  be closed by the caller, in mysql_execute_command().
  When dropping all routines in a database, make sure
  to not mistakenly drop all metadata locks acquired
  so far, they include the scoped lock on the schema.
sql/sp_head.cc:
  Correct the protocol that closes thread tables
  in an SP instruction.
  Clear lex->sphead before cleaning up lex
  with lex_end to make sure that we don't
  delete the sphead twice. It's considered
  to be "cleaner" and more in line with
  future changes than calling delete lex->sphead
  in other places that cleanup the lex.
sql/sp_head.h:
  When destroying m_lex_keeper of an instruction,
  don't delete the sphead that all lex objects
  share. 
  @todo: don't store a reference to routine's sp_head
  instance in instruction's lex.
sql/sql_acl.cc:
  Don't call close_thread_tables() where the caller will
  do that for us.
  Fix Bug#55452 "SET PASSWORD is replicated twice in RBR 
  mode" by disabling RBR replication in change_password()
  function.
  Use close_mysql_tables() in bootstrap and ACL reload
  code to make sure we release all metadata locks.
sql/sql_base.cc:
  This is the main part of the patch:
  - remove manipulation with thd->transaction
  and thd->mdl_context from close_thread_tables().
  Now this function is only responsible for closing
  tables, nothing else.
  This is necessary to be able to easily use
  close_thread_tables() in procedures, that
  involve multiple open/close tables, which all
  need to be protected continuously by metadata
  locks.
  Add asserts ensuring that TABLE object
  is only used when is protected by a metadata lock.
  Simplify the back off condition of Open_table_context,
  we no longer need to look at the autocommit mode.
  Make open_and_lock_tables() and open_normal_and_derived_tables()
  close thread tables and release metadata locks acquired so-far 
  upon failure. This simplifies their usage.
  Implement close_mysql_tables().
sql/sql_base.h:
  Add declaration for close_mysql_tables().
sql/sql_class.cc:
  Remove a piece of dead code that has also become redundant
  after the fix for Bug 37521.
  The code became dead when my_eof() was made a non-protocol method,
  but a method that merely modifies the diagnostics area.
  The code became redundant with the fix for Bug#37521, when 
  we started to cal close_thread_tables() before
  Protocol::end_statement().
sql/sql_do.cc:
  Do nothing in DO if inside a substatement
  (the assert moved out of trans_rollback_stmt).
sql/sql_handler.cc:
  Add comments.
sql/sql_insert.cc:
  Remove dead code. 
  Release metadata locks explicitly at the
  end of the delayed insert thread.
sql/sql_lex.cc:
  Add destruction of lex->sphead to lex_end(),
  lex "reset" method called at the end of each statement.
sql/sql_parse.cc:
  Move close_thread_tables() and other related
  cleanups to mysql_execute_command()
  from dispatch_command(). This has become
  possible after the fix for Bug#37521.
  Mark federated SERVER statements as DDL.
  
  Next step: make sure that we don't store
  eof packet in the query cache, and move
  the query cache code outside mysql_parse.
  
  Brush up the code of COM_FIELD_LIST.
  Remove unnecessary calls to close_thread_tables().
  
  When killing a query, don't report "OK"
  if it was a suicide.
sql/sql_parse.h:
  Remove declaration of a function that is now static.
sql/sql_partition.cc:
  Remove an unnecessary call to close_thread_tables().
sql/sql_plugin.cc:
  open_and_lock_tables() will clean up
  after itself after a failure.
  Move close_thread_tables() above
  end: label, and replace with close_mysql_tables(),
  which will also release the metadata lock
  on mysql.plugin.
sql/sql_prepare.cc:
  Now that we no longer release locks in close_thread_tables()
  statement prepare code has become more straightforward.
  Remove the now redundant check for thd->killed() (used
  only by the backup project) from Execute_server_runnable.
  Reorder code to take into account that now mysql_execute_command()
  performs lex->unit.cleanup() and close_thread_tables().
sql/sql_priv.h:
  Add a new option to server options to interact
  between the slave SQL thread and execution
  framework (hack). @todo: use a virtual
  method of class THD instead.
sql/sql_servers.cc:
  Due to Bug 25705 replication of 
  DROP/CREATE/ALTER SERVER is broken.
  Make sure at least we do not attempt to 
  replicate these statements using RBR,
  as this violates the assert in close_mysql_tables().
sql/sql_table.cc:
  Do not release metadata locks in mysql_rm_table_part2,
  this is done by the caller.
  Do not call close_thread_tables() in mysql_create_table(),
  this is done by the caller. 
  Fix a bug in DROP TABLE under LOCK TABLES when,
  upon error in wait_while_table_is_used() we would mistakenly
  release the metadata lock on a non-dropped table.
  Explicitly release metadata locks when doing an implicit
  commit.
sql/sql_trigger.cc:
  Now that we delete lex->sphead in lex_end(),
  zero the trigger's sphead in lex after loading
  the trigger, to avoid double deletion.
sql/sql_udf.cc:
  Use close_mysql_tables() instead of close_thread_tables().
sql/sys_vars.cc:
  Remove code added in scope of WL#4284 which would
  break when we perform set @@session.autocommit along
  with setting other variables and using tables or functions.
  A test case added to variables.test.
sql/transaction.cc:
  Add asserts.
sql/tztime.cc:
  Use close_mysql_tables() rather than close_thread_tables().

sql/sql_base.cc

@@ -1402,6 +1402,9 @@ void close_thread_tables(THD *thd)
     DEBUG_SYNC(thd, "before_close_thread_tables");
 #endif
 
+  DBUG_ASSERT(thd->transaction.stmt.is_empty() || thd->in_sub_stmt ||
+              (thd->state_flags & Open_tables_state::BACKUPS_AVAIL));
+
   /* Detach MERGE children after every statement. Even under LOCK TABLES. */
   for (table= thd->open_tables; table; table= table->next)
   {
@@ -1446,28 +1449,6 @@ void close_thread_tables(THD *thd)
     Mark all temporary tables used by this statement as free for reuse.
   */
   mark_temp_tables_as_free_for_reuse(thd);
-  /*
-    Let us commit transaction for statement. Since in 5.0 we only have
-    one statement transaction and don't allow several nested statement
-    transactions this call will do nothing if we are inside of stored
-    function or trigger (i.e. statement transaction is already active and
-    does not belong to statement for which we do close_thread_tables()).
-    TODO: This should be fixed in later releases.
-   */
-  if (!(thd->state_flags & Open_tables_state::BACKUPS_AVAIL))
-  {
-    thd->stmt_da->can_overwrite_status= TRUE;
-    thd->is_error() ? trans_rollback_stmt(thd) : trans_commit_stmt(thd);
-    thd->stmt_da->can_overwrite_status= FALSE;
-
-    /*
-      Reset transaction state, but only if we're not inside a
-      sub-statement of a prelocked statement.
-    */
-    if (thd->locked_tables_mode <= LTM_LOCK_TABLES ||
-        thd->lex->requires_prelocking())
-      thd->transaction.stmt.reset();
-  }
 
   if (thd->locked_tables_mode)
   {
@@ -1528,26 +1509,6 @@ void close_thread_tables(THD *thd)
   if (thd->open_tables)
     close_open_tables(thd);
 
-  /*
-    - If inside a multi-statement transaction,
-    defer the release of metadata locks until the current
-    transaction is either committed or rolled back. This prevents
-    other statements from modifying the table for the entire
-    duration of this transaction.  This provides commit ordering
-    and guarantees serializability across multiple transactions.
-    - If closing a system table, defer the release of metadata locks
-    to the caller. We have no sentinel in MDL subsystem to guard
-    transactional locks from system tables locks, so don't know
-    which locks are which here.
-    - If in autocommit mode, or outside a transactional context,
-    automatically release metadata locks of the current statement.
-  */
-  if (! thd->in_multi_stmt_transaction_mode() &&
-      ! (thd->state_flags & Open_tables_state::BACKUPS_AVAIL))
-  {
-    thd->mdl_context.release_transactional_locks();
-  }
-
   DBUG_VOID_RETURN;
 }
 
@@ -1562,7 +1523,14 @@ bool close_thread_table(THD *thd, TABLE **table_ptr)
   DBUG_ASSERT(table->key_read == 0);
   DBUG_ASSERT(!table->file || table->file->inited == handler::NONE);
   mysql_mutex_assert_not_owner(&LOCK_open);
-
+  /*
+    The metadata lock must be released after giving back
+    the table to the table cache.
+  */
+  DBUG_ASSERT(thd->mdl_context.is_lock_owner(MDL_key::TABLE,
+                                             table->s->db.str,
+                                             table->s->table_name.str,
+                                             MDL_SHARED));
   table->mdl_ticket= NULL;
 
   mysql_mutex_lock(&thd->LOCK_thd_data);
@@ -3188,6 +3156,7 @@ Locked_tables_list::init_locked_tables(THD *thd)
   return FALSE;
 }
 
+
 /**
   Leave LTM_LOCK_TABLES mode if it's been entered.
 
@@ -3224,7 +3193,12 @@ Locked_tables_list::unlock_locked_tables(THD *thd)
     }
     thd->leave_locked_tables_mode();
 
+    DBUG_ASSERT(thd->transaction.stmt.is_empty());
     close_thread_tables(thd);
+    /*
+      We rely on the caller to implicitly commit the
+      transaction and release transactional locks.
+    */
   }
   /*
     After closing tables we can free memory used for storing lock
@@ -3810,9 +3784,7 @@ Open_table_context::Open_table_context(THD *thd, uint flags)
              LONG_TIMEOUT : thd->variables.lock_wait_timeout),
    m_flags(flags),
    m_action(OT_NO_ACTION),
-   m_has_locks((thd->in_multi_stmt_transaction_mode() &&
-                thd->mdl_context.has_locks()) ||
-                thd->mdl_context.trans_sentinel())
+   m_has_locks(thd->mdl_context.has_locks())
 {}
 
 
@@ -5264,6 +5236,8 @@ TABLE *open_ltable(THD *thd, TABLE_LIST *table_list, thr_lock_type lock_type,
     table= 0;
 
 end:
+  if (table == NULL)
+    close_thread_tables(thd);
   thd_proc_info(thd, 0);
   DBUG_RETURN(table);
 }
@@ -5282,7 +5256,8 @@ end:
                               should work for this statement.
 
   @note
-    The lock will automaticaly be freed by close_thread_tables()
+    The thr_lock locks will automatically be freed by
+    close_thread_tables().
 
   @retval FALSE  OK.
   @retval TRUE   Error
@@ -5293,11 +5268,12 @@ bool open_and_lock_tables(THD *thd, TABLE_LIST *tables,
                           Prelocking_strategy *prelocking_strategy)
 {
   uint counter;
+  MDL_ticket *mdl_savepoint= thd->mdl_context.mdl_savepoint();
   DBUG_ENTER("open_and_lock_tables");
   DBUG_PRINT("enter", ("derived handling: %d", derived));
 
   if (open_tables(thd, &tables, &counter, flags, prelocking_strategy))
-    DBUG_RETURN(TRUE);
+    goto err;
 
   DBUG_EXECUTE_IF("sleep_open_and_lock_after_open", {
                   const char *old_proc_info= thd->proc_info;
@@ -5306,15 +5282,22 @@ bool open_and_lock_tables(THD *thd, TABLE_LIST *tables,
                   thd->proc_info= old_proc_info;});
 
   if (lock_tables(thd, tables, counter, flags))
-    DBUG_RETURN(TRUE);
+    goto err;
 
   if (derived &&
       (mysql_handle_derived(thd->lex, &mysql_derived_prepare) ||
        (thd->fill_derived_tables() &&
         mysql_handle_derived(thd->lex, &mysql_derived_filling))))
-    DBUG_RETURN(TRUE); /* purecov: inspected */
+    goto err;
 
   DBUG_RETURN(FALSE);
+err:
+  if (! thd->in_sub_stmt)
+    trans_rollback_stmt(thd);  /* Necessary if derived handling failed. */
+  close_thread_tables(thd);
+  /* Don't keep locks for a failed statement. */
+  thd->mdl_context.rollback_to_savepoint(mdl_savepoint);
+  DBUG_RETURN(TRUE);
 }
 
 
@@ -5340,13 +5323,24 @@ bool open_and_lock_tables(THD *thd, TABLE_LIST *tables,
 
 bool open_normal_and_derived_tables(THD *thd, TABLE_LIST *tables, uint flags)
 {
+  DML_prelocking_strategy prelocking_strategy;
   uint counter;
+  MDL_ticket *mdl_savepoint= thd->mdl_context.mdl_savepoint();
   DBUG_ENTER("open_normal_and_derived_tables");
   DBUG_ASSERT(!thd->fill_derived_tables());
-  if (open_tables(thd, &tables, &counter, flags) ||
+  if (open_tables(thd, &tables, &counter, flags, &prelocking_strategy) ||
       mysql_handle_derived(thd->lex, &mysql_derived_prepare))
-    DBUG_RETURN(TRUE); /* purecov: inspected */
+    goto end;
+
   DBUG_RETURN(0);
+end:
+  /* No need to rollback statement transaction, it's not started. */
+  DBUG_ASSERT(thd->transaction.stmt.is_empty());
+  close_thread_tables(thd);
+  /* Don't keep locks for a failed statement. */
+  thd->mdl_context.rollback_to_savepoint(mdl_savepoint);
+
+  DBUG_RETURN(TRUE); /* purecov: inspected */
 }
 
 
@@ -5607,6 +5601,14 @@ void close_tables_for_reopen(THD *thd, TABLE_LIST **tables,
     /* We have to cleanup translation tables of views. */
     tmp->cleanup_items();
   }
+  /*
+    No need to commit/rollback the statement transaction: it's
+    either not started or we're filling in an INFORMATION_SCHEMA
+    table on the fly, and thus mustn't manipulate with the
+    transaction of the enclosing statement.
+  */
+  DBUG_ASSERT(thd->transaction.stmt.is_empty() ||
+              (thd->state_flags & Open_tables_state::BACKUPS_AVAIL));
   close_thread_tables(thd);
   thd->mdl_context.rollback_to_savepoint(start_of_statement_svp);
 }
@@ -9034,7 +9036,8 @@ open_system_tables_for_read(THD *thd, TABLE_LIST *table_list,
                            MYSQL_LOCK_IGNORE_TIMEOUT))
   {
     lex->restore_backup_query_tables_list(&query_tables_list_backup);
-    goto error;
+    thd->restore_backup_open_tables_state(backup);
+    DBUG_RETURN(TRUE);
   }
 
   for (TABLE_LIST *tables= table_list; tables; tables= tables->next_global)
@@ -9045,11 +9048,6 @@ open_system_tables_for_read(THD *thd, TABLE_LIST *table_list,
   lex->restore_backup_query_tables_list(&query_tables_list_backup);
 
   DBUG_RETURN(FALSE);
-
-error:
-  close_system_tables(thd, backup);
-
-  DBUG_RETURN(TRUE);
 }
 
 
@@ -9072,6 +9070,38 @@ close_system_tables(THD *thd, Open_tables_backup *backup)
 }
 
 
+/**
+  A helper function to close a mysql.* table opened
+  in an auxiliary THD during bootstrap or in the main
+  connection, when we know that there are no locks
+  held by the connection due to a preceding implicit
+  commit.
+
+  This function assumes that there is no
+  statement transaction started for the operation
+  itself, since mysql.* tables are not transactional
+  and when they are used the binlog is off (DDL
+  binlogging is always statement-based.
+
+  We need this function since we'd like to not
+  just close the system table, but also release
+  the metadata lock on it.
+
+  Note, that in LOCK TABLES mode this function
+  does not release the metadata lock. But in this
+  mode the table can be opened only if it is locked
+  explicitly with LOCK TABLES.
+*/
+
+void
+close_mysql_tables(THD *thd)
+{
+  /* No need to commit/rollback statement transaction, it's not started. */
+  DBUG_ASSERT(thd->transaction.stmt.is_empty());
+  close_thread_tables(thd);
+  thd->mdl_context.release_transactional_locks();
+}
+
 /*
   Open and lock one system table for update.
 
@@ -9143,16 +9173,7 @@ open_log_table(THD *thd, TABLE_LIST *one_table, Open_tables_backup *backup)
     table->timestamp_field_type= TIMESTAMP_NO_AUTO_SET;
   }
   else
-  {
-    /*
-      If error in mysql_lock_tables(), open_ltable doesn't close the
-      table. Thread kill during mysql_lock_tables() is such error. But
-      open tables cannot be accepted when restoring the open tables
-      state.
-    */
-    close_thread_tables(thd);
     thd->restore_backup_open_tables_state(backup);
-  }
 
   thd->utime_after_lock= save_utime_after_lock;
   DBUG_RETURN(table);