A fix for Bug#26750 "valgrind leak in sp_head" (and post-review

fixes).

The legend: on a replication slave, in case a trigger creation
was filtered out because of application of replicate-do-table/
replicate-ignore-table rule, the parsed definition of a trigger was not 
cleaned up properly. LEX::sphead member was left around and leaked 
memory. Until the actual implementation of support of 
replicate-ignore-table rules for triggers by the patch for Bug 24478 it 
was never the case that "case SQLCOM_CREATE_TRIGGER"
was not executed once a trigger was parsed,
so the deletion of lex->sphead there worked and the memory did not leak.

The fix: 

The real cause of the bug is that there is no 1 or 2 places where
we can clean up the main LEX after parse. And the reason we 
can not have just one or two places where we clean up the LEX is
asymmetric behaviour of MYSQLparse in case of success or error. 

One of the root causes of this behaviour is the code in Item::Item()
constructor. There, a newly created item adds itself to THD::free_list
- a single-linked list of Items used in a statement. Yuck. This code
is unaware that we may have more than one statement active at a time,
and always assumes that the free_list of the current statement is
located in THD::free_list. One day we need to be able to explicitly
allocate an item in a given Query_arena.
Thus, when parsing a definition of a stored procedure, like
CREATE PROCEDURE p1() BEGIN SELECT a FROM t1; SELECT b FROM t1; END;
we actually need to reset THD::mem_root, THD::free_list and THD::lex
to parse the nested procedure statement (SELECT *).
The actual reset and restore is implemented in semantic actions
attached to sp_proc_stmt grammar rule.
The problem is that in case of a parsing error inside a nested statement
Bison generated parser would abort immediately, without executing the
restore part of the semantic action. This would leave THD in an 
in-the-middle-of-parsing state.
This is why we couldn't have had a single place where we clean up the LEX
after MYSQLparse - in case of an error we needed to do a clean up
immediately, in case of success a clean up could have been delayed.
This left the door open for a memory leak.

One of the following possibilities were considered when working on a fix:
- patch the replication logic to do the clean up. Rejected
as breaks module borders, replication code should not need to know the
gory details of clean up procedure after CREATE TRIGGER.
- wrap MYSQLparse with a function that would do a clean up.
Rejected as ideally we should fix the problem when it happens, not
adjust for it outside of the problematic code.
- make sure MYSQLparse cleans up after itself by invoking the clean up
functionality in the appropriate places before return. Implemented in 
this patch.
- use %destructor rule for sp_proc_stmt to restore THD - cleaner
than the prevoius approach, but rejected
because needs a careful analysis of the side effects, and this patch is 
for 5.0, and long term we need to use the next alternative anyway
- make sure that sp_proc_stmt doesn't juggle with THD - this is a 
large work that will affect many modules.

Cleanup: move main_lex and main_mem_root from Statement to its
only two descendants Prepared_statement and THD. This ensures that
when a Statement instance was created for purposes of statement backup,
we do not involve LEX constructor/destructor, which is fairly expensive.
In order to track that the transformation produces equivalent 
functionality please check the respective constructors and destructors
of Statement, Prepared_statement and THD - these members were
used only there.
This cleanup is unrelated to the patch.

sql/log_event.cc

@@ -3114,8 +3114,8 @@ int Load_log_event::exec_event(NET* net, struct st_relay_log_info* rli,
 
       ex.skip_lines = skip_lines;
       List<Item> field_list;
-      thd->main_lex.select_lex.context.resolve_in_table_list_only(&tables);
+      thd->lex->select_lex.context.resolve_in_table_list_only(&tables);
-      set_fields(tables.db, field_list, &thd->main_lex.select_lex.context);
+      set_fields(tables.db, field_list, &thd->lex->select_lex.context);
       thd->variables.pseudo_thread_id= thread_id;
       if (net)
       {

sql/mysqld.cc

@@ -1598,18 +1598,6 @@ static void network_init(void)
 
 #endif /*!EMBEDDED_LIBRARY*/
 
-void MYSQLerror(const char *s)
-{
-  THD *thd=current_thd;
-  char *yytext= (char*) thd->lex->tok_start;
-  /* "parse error" changed into "syntax error" between bison 1.75 and 1.875 */
-  if (strcmp(s,"parse error") == 0 || strcmp(s,"syntax error") == 0)
-    s=ER(ER_SYNTAX_ERROR);
-  my_printf_error(ER_PARSE_ERROR,  ER(ER_PARSE_ERROR), MYF(0), s,
-                  (yytext ? (char*) yytext : ""),
-                  thd->lex->yylineno);
-}
-
 
 #ifndef EMBEDDED_LIBRARY
 /*

sql/sql_class.cc

@@ -166,14 +166,10 @@ Open_tables_state::Open_tables_state(ulong version_arg)
 }
 
 
-/*
-  Pass nominal parameters to Statement constructor only to ensure that
-  the destructor works OK in case of error. The main_mem_root will be
-  re-initialized in init().
-*/
 
 THD::THD()
-  :Statement(CONVENTIONAL_EXECUTION, 0, ALLOC_ROOT_MIN_BLOCK_SIZE, 0),
+   :Statement(&main_lex, &main_mem_root, CONVENTIONAL_EXECUTION,
+              /* statement id */ 0),
    Open_tables_state(refresh_version),
    lock_id(&main_lock_id),
    user_time(0), in_sub_stmt(0), global_read_lock(0), is_fatal_error(0),
@@ -184,6 +180,12 @@ THD::THD()
 {
   ulong tmp;
 
+  /*
+    Pass nominal parameters to init_alloc_root only to ensure that
+    the destructor works OK in case of an error. The main_mem_root
+    will be re-initialized in init_for_queries().
+  */
+  init_sql_alloc(&main_mem_root, ALLOC_ROOT_MIN_BLOCK_SIZE, 0);
   stmt_arena= this;
   thread_stack= 0;
   db= 0;
@@ -443,6 +445,7 @@ THD::~THD()
 #ifndef DBUG_OFF
   dbug_sentry= THD_SENTRY_GONE;
 #endif  
+  free_root(&main_mem_root, MYF(0));
   DBUG_VOID_RETURN;
 }
 
@@ -1581,18 +1584,17 @@ void Query_arena::cleanup_stmt()
   Statement functions 
 */
 
-Statement::Statement(enum enum_state state_arg, ulong id_arg,
+Statement::Statement(LEX *lex_arg, MEM_ROOT *mem_root_arg,
-                     ulong alloc_block_size, ulong prealloc_size)
+                     enum enum_state state_arg, ulong id_arg)
-  :Query_arena(&main_mem_root, state_arg),
+  :Query_arena(mem_root_arg, state_arg),
   id(id_arg),
   set_query_id(1),
-  lex(&main_lex),
+  lex(lex_arg),
   query(0),
   query_length(0),
   cursor(0)
 {
   name.str= NULL;
-  init_sql_alloc(&main_mem_root, alloc_block_size, prealloc_size);
 }
 
 
@@ -1634,7 +1636,7 @@ void Statement::restore_backup_statement(Statement *stmt, Statement *backup)
 
 void THD::end_statement()
 {
-  /* Cleanup SQL processing state to resuse this statement in next query. */
+  /* Cleanup SQL processing state to reuse this statement in next query. */
   lex_end(lex);
   delete lex->result;
   lex->result= 0;
@@ -1675,12 +1677,6 @@ void THD::restore_active_arena(Query_arena *set, Query_arena *backup)
 
 Statement::~Statement()
 {
-  /*
-    We must free `main_mem_root', not `mem_root' (pointer), to work
-    correctly if this statement is used as a backup statement,
-    for which `mem_root' may point to some other statement.
-  */
-  free_root(&main_mem_root, MYF(0));
 }
 
 C_MODE_START

sql/sql_class.h

@@ -753,8 +753,10 @@ public:
 
 class Server_side_cursor;
 
-/*
+/**
-  State of a single command executed against this connection.
+  @class Statement
+  @brief State of a single command executed against this connection.
+
   One connection can contain a lot of simultaneously running statements,
   some of which could be:
    - prepared, that is, contain placeholders,
@@ -772,10 +774,6 @@ class Statement: public ilink, public Query_arena
   Statement(const Statement &rhs);              /* not implemented: */
   Statement &operator=(const Statement &rhs);   /* non-copyable */
 public:
-  /* FIXME: these must be protected */
-  MEM_ROOT main_mem_root;
-  LEX     main_lex;
-
   /*
     Uniquely identifies each statement object in thread scope; change during
     statement lifetime. FIXME: must be const
@@ -819,10 +817,10 @@ public:
 public:
 
   /* This constructor is called for backup statements */
-  Statement() { clear_alloc_root(&main_mem_root); }
+  Statement() {}
 
-  Statement(enum enum_state state_arg, ulong id_arg,
+  Statement(LEX *lex_arg, MEM_ROOT *mem_root_arg,
-            ulong alloc_block_size, ulong prealloc_size);
+            enum enum_state state_arg, ulong id_arg);
   virtual ~Statement();
 
   /* Assign execution context (note: not all members) of given stmt to self */
@@ -834,7 +832,7 @@ public:
 };
 
 
-/*
+/**
   Container for all statements created/used in a connection.
   Statements in Statement_map have unique Statement::id (guaranteed by id
   assignment in Statement::Statement)
@@ -914,6 +912,10 @@ bool xid_cache_insert(XID *xid, enum xa_states xa_state);
 bool xid_cache_insert(XID_STATE *xid_state);
 void xid_cache_delete(XID_STATE *xid_state);
 
+/**
+  @class Security_context
+  @brief A set of THD members describing the current authenticated user.
+*/
 
 class Security_context {
 public:
@@ -943,7 +945,7 @@ public:
 };
 
 
-/*
+/**
   A registry for item tree transformations performed during
   query optimization. We register only those changes which require
   a rollback to re-execute a prepared statement or stored procedure
@@ -954,7 +956,7 @@ struct Item_change_record;
 typedef I_List<Item_change_record> Item_change_list;
 
 
-/*
+/**
   Type of prelocked mode.
   See comment for THD::prelocked_mode for complete description.
 */
@@ -963,7 +965,7 @@ enum prelocked_mode_type {NON_PRELOCKED= 0, PRELOCKED= 1,
                           PRELOCKED_UNDER_LOCK_TABLES= 2};
 
 
-/*
+/**
   Class that holds information about tables which were opened and locked
   by the thread. It is also used to save/restore this information in
   push_open_tables_state()/pop_open_tables_state().
@@ -1048,14 +1050,17 @@ public:
   }
 };
 
-
+/**
-/* class to save context when executing a function or trigger */
+  @class Sub_statement_state
+  @brief Used to save context when executing a function or trigger
+*/
 
 /* Defines used for Sub_statement_state::in_sub_stmt */
 
 #define SUB_STMT_TRIGGER 1
 #define SUB_STMT_FUNCTION 2
 
+
 class Sub_statement_state
 {
 public:
@@ -1072,7 +1077,8 @@ public:
 };
 
 
-/*
+/**
+  @class THD
   For each client connection we create a separate thread with THD serving as
   a thread/connection descriptor
 */
@@ -1659,6 +1665,23 @@ public:
       *p_db_length= db_length;
     return FALSE;
   }
+private:
+  /**
+    The lex to hold the parsed tree of conventional (non-prepared) queries.
+    Whereas for prepared and stored procedure statements we use an own lex
+    instance for each new query, for conventional statements we reuse
+    the same lex. (@see mysql_parse for details).
+  */
+  LEX main_lex;
+  /**
+    This memory root is used for two purposes:
+    - for conventional queries, to allocate structures stored in main_lex
+    during parsing, and allocate runtime data (execution plan, etc.)
+    during execution.
+    - for prepared queries, only to allocate runtime data. The parsed
+    tree itself is reused between executions and thus is stored elsewhere.
+  */
+  MEM_ROOT main_mem_root;
 };
 
 

sql/sql_lex.cc

@@ -1648,6 +1648,36 @@ void st_select_lex::print_limit(THD *thd, String *str)
   }
 }
 
+/**
+  @brief Restore the LEX and THD in case of a parse error.
+
+  This is a clean up call that is invoked by the Bison generated
+  parser before returning an error from MYSQLparse. If your
+  semantic actions manipulate with the global thread state (which
+  is a very bad practice and should not normally be employed) and
+  need a clean-up in case of error, and you can not use %destructor
+  rule in the grammar file itself, this function should be used
+  to implement the clean up.
+*/
+
+void st_lex::cleanup_lex_after_parse_error(THD *thd)
+{
+  /*
+    Delete sphead for the side effect of restoring of the original
+    LEX state, thd->lex, thd->mem_root and thd->free_list if they
+    were replaced when parsing stored procedure statements.  We
+    will never use sphead object after a parse error, so it's okay
+    to delete it only for the sake of the side effect.
+    TODO: make this functionality explicit in sp_head class.
+    Sic: we must nullify the member of the main lex, not the
+    current one that will be thrown away
+  */
+  if (thd->lex->sphead);
+  {
+    delete thd->lex->sphead;
+    thd->lex->sphead= NULL;
+  }
+}
 
 /*
   Initialize (or reset) Query_tables_list object.

sql/sql_lex.h

@@ -1182,6 +1182,10 @@ typedef struct st_lex : public Query_tables_list
   {
     return context_stack.head();
   }
+  /*
+    Restore the LEX and THD in case of a parse error.
+  */
+  static void cleanup_lex_after_parse_error(THD *thd);
 
   void reset_n_backup_query_tables_list(Query_tables_list *backup);
   void restore_backup_query_tables_list(Query_tables_list *backup);

sql/sql_parse.cc

@@ -4506,9 +4506,6 @@ end_with_restore_list:
       clean up the environment.
     */
 create_sp_error:
-    lex->unit.cleanup();
-    delete lex->sphead;
-    lex->sphead= 0;
     if (sp_result != SP_OK )
       goto error;
     send_ok(thd);
@@ -4879,9 +4876,6 @@ create_sp_error:
     /* Conditionally writes to binlog. */
     res= mysql_create_or_drop_trigger(thd, all_tables, 1);
 
-    /* We don't care about trigger body after this point */
-    delete lex->sphead;
-    lex->sphead= 0;
     break;
   }
   case SQLCOM_DROP_TRIGGER:
@@ -5904,15 +5898,7 @@ void mysql_parse(THD *thd, char *inBuf, uint length)
       else
 #endif
       {
-	if (thd->net.report_error)
+	if (! thd->net.report_error)
-	{
-	  if (thd->lex->sphead)
-	  {
-	    delete thd->lex->sphead;
-	    thd->lex->sphead= NULL;
-	  }
-	}
-	else
 	{
           /*
             Binlog logs a string starting from thd->query and having length
@@ -5932,7 +5918,6 @@ void mysql_parse(THD *thd, char *inBuf, uint length)
 	  query_cache_end_of_result(thd);
 	}
       }
-      lex->unit.cleanup();
     }
     else
     {
@@ -5940,19 +5925,14 @@ void mysql_parse(THD *thd, char *inBuf, uint length)
       DBUG_PRINT("info",("Command aborted. Fatal_error: %d",
 			 thd->is_fatal_error));
 
-      /*
-        The first thing we do after parse error is freeing sp_head to
-        ensure that we have restored original memroot.
-      */
-      if (thd->lex->sphead)
-      {
-	/* Clean up after failed stored procedure/function */
-	delete thd->lex->sphead;
-	thd->lex->sphead= NULL;
-      }
       query_cache_abort(&thd->net);
-      lex->unit.cleanup();
     }
+    if (thd->lex->sphead)
+    {
+      delete thd->lex->sphead;
+      thd->lex->sphead= 0;
+    }
+    lex->unit.cleanup();
     thd->proc_info="freeing items";
     thd->end_statement();
     thd->cleanup_after_query();

sql/sql_prepare.cc

@@ -99,9 +99,12 @@ public:
 #endif
 };
 
-/******************************************************************************
+/****************************************************************************/
-  Prepared_statement: a statement that can contain placeholders
+
-******************************************************************************/
+/**
+  @class Prepared_statement
+  @brief Prepared_statement: a statement that can contain placeholders
+*/
 
 class Prepared_statement: public Statement
 {
@@ -141,6 +144,16 @@ public:
   bool execute(String *expanded_query, bool open_cursor);
   /* Destroy this statement */
   bool deallocate();
+private:
+  /**
+    Store the parsed tree of a prepared statement here.
+  */
+  LEX main_lex;
+  /**
+    The memory root to allocate parsed tree elements (instances of Item,
+    SELECT_LEX and other classes).
+  */
+  MEM_ROOT main_mem_root;
 };
 
 
@@ -2034,6 +2047,7 @@ void mysql_sql_stmt_prepare(THD *thd)
     delete stmt;
     DBUG_VOID_RETURN;
   }
+
   if (thd->stmt_map.insert(thd, stmt))
   {
     /* The statement is deleted and an error is set if insert fails */
@@ -2629,17 +2643,18 @@ Select_fetch_protocol_prep::send_data(List<Item> &fields)
 ****************************************************************************/
 
 Prepared_statement::Prepared_statement(THD *thd_arg, Protocol *protocol_arg)
-  :Statement(INITIALIZED, ++thd_arg->statement_id_counter,
+  :Statement(&main_lex, &main_mem_root,
-             thd_arg->variables.query_alloc_block_size,
+             INITIALIZED, ++thd_arg->statement_id_counter),
-             thd_arg->variables.query_prealloc_size),
   thd(thd_arg),
   result(thd_arg),
   protocol(protocol_arg),
   param_array(0),
   param_count(0),
   last_errno(0),
-   flags((uint) IS_IN_USE)
+  flags((uint) IS_IN_USE)
 {
+  init_alloc_root(&main_mem_root, thd_arg->variables.query_alloc_block_size,
+                  thd_arg->variables.query_prealloc_size);
   *last_error= '\0';
 }
 
@@ -2688,6 +2703,7 @@ Prepared_statement::~Prepared_statement()
   */
   free_items();
   delete lex->result;
+  free_root(&main_mem_root, MYF(0));
   DBUG_VOID_RETURN;
 }
 
@@ -2703,6 +2719,7 @@ void Prepared_statement::cleanup_stmt()
   DBUG_ENTER("Prepared_statement::cleanup_stmt");
   DBUG_PRINT("enter",("stmt: %p", this));
 
+  DBUG_ASSERT(lex->sphead == 0);
   /* The order is important */
   lex->unit.cleanup();
   cleanup_items(free_list);
@@ -2789,15 +2806,6 @@ bool Prepared_statement::prepare(const char *packet, uint packet_len)
   error= MYSQLparse((void *)thd) || thd->is_fatal_error ||
       thd->net.report_error || init_param_array(this);
 
-  /*
-    The first thing we do after parse error is freeing sp_head to
-    ensure that we have restored original memroot.
-  */
-  if (error && lex->sphead)
-  {
-    delete lex->sphead;
-    lex->sphead= NULL;
-  }
   /*
     While doing context analysis of the query (in check_prepared_statement)
     we allocate a lot of additional memory: for open tables, JOINs, derived
@@ -2823,12 +2831,18 @@ bool Prepared_statement::prepare(const char *packet, uint packet_len)
   if (error == 0)
     error= check_prepared_statement(this, name.str != 0);
 
-  /* Free sp_head if check_prepared_statement() failed. */
+  /*
-  if (error && lex->sphead)
+    Currently CREATE PROCEDURE/TRIGGER/EVENT are prohibited in prepared
+    statements: ensure we have no memory leak here if by someone tries
+    to PREPARE stmt FROM "CREATE PROCEDURE ..."
+  */
+  DBUG_ASSERT(lex->sphead == NULL || error != 0);
+  if (lex->sphead)
   {
     delete lex->sphead;
     lex->sphead= NULL;
   }
+
   lex_end(lex);
   cleanup_stmt();
   thd->restore_backup_statement(this, &stmt_backup);

sql/sql_trigger.cc

@@ -983,11 +983,8 @@ bool Table_triggers_list::check_n_load(THD *thd, const char *db,
 	thd->spcont= 0;
         if (MYSQLparse((void *)thd) || thd->is_fatal_error)
         {
-          /*
+          /* Currently sphead is always deleted in case of a parse error */
-            Free lex associated resources.
+          DBUG_ASSERT(lex.sphead == 0);
-            QQ: Do we really need all this stuff here ?
-          */
-          delete lex.sphead;
           goto err_with_lex_cleanup;
         }