Bug#53676: Unexpected errors and possible table

corruption on ADD PARTITION and LOCK TABLE
Bug#53770: Server crash at handler.cc:2076 on
           LOAD DATA after timed out COALESCE PARTITION

5.5 fix for:
Bug#51042: REORGANIZE PARTITION can leave table in an
           inconsistent state in case of crash
Needs to be back-ported to 5.1

5.5 fix for:
Bug#50418: DROP PARTITION does not interact with
           transactions

Main problem was non-persistent operations done
before meta-data lock was taken (53770+53676).
And 53676 needed to keep the table/partitions opened and locked
while copying the data to the new partitions.

Also added thorough tests to spot some additional bugs
in the ddl_log code, which could result in bad state
between the .frm and partitions.

Collapsed patch, includes all fixes required from the reviewers.

mysql-test/r/partition_innodb.result:
  updated result with new test
mysql-test/suite/parts/inc/partition_crash.inc:
  crash test include file
mysql-test/suite/parts/inc/partition_crash_add.inc:
  test all states in fast_alter_partition_table
  ADD PARTITION branch
mysql-test/suite/parts/inc/partition_crash_change.inc:
  test all states in fast_alter_partition_table
  CHANGE PARTITION branch
mysql-test/suite/parts/inc/partition_crash_drop.inc:
  test all states in fast_alter_partition_table
  DROP PARTITION branch
mysql-test/suite/parts/inc/partition_fail.inc:
  recovery test including injecting errors
mysql-test/suite/parts/inc/partition_fail_add.inc:
  test all states in fast_alter_partition_table
  ADD PARTITION branch
mysql-test/suite/parts/inc/partition_fail_change.inc:
  test all states in fast_alter_partition_table
  CHANGE PARTITION branch
mysql-test/suite/parts/inc/partition_fail_drop.inc:
  test all states in fast_alter_partition_table
  DROP PARTITION branch
mysql-test/suite/parts/inc/partition_mgm_crash.inc:
  include file that runs all crash and failure injection tests.
mysql-test/suite/parts/r/partition_debug_innodb.result:
  new test result file
mysql-test/suite/parts/r/partition_debug_myisam.result:
  new test result file
mysql-test/suite/parts/r/partition_special_innodb.result:
  updated result
mysql-test/suite/parts/r/partition_special_myisam.result:
  updated result
mysql-test/suite/parts/t/partition_debug_innodb-master.opt:
  opt file for using with crashing tests of partitioned innodb
mysql-test/suite/parts/t/partition_debug_innodb.test:
  partitioned innodb test that require debug builds
mysql-test/suite/parts/t/partition_debug_myisam-master.opt:
  opt file for using with crashing tests of partitioned myisam
mysql-test/suite/parts/t/partition_debug_myisam.test:
  partitioned myisam test that require debug builds
mysql-test/suite/parts/t/partition_special_innodb-master.opt:
  added innodb-file-per-table to easier verify partition status on disk
mysql-test/suite/parts/t/partition_special_innodb.test:
  added test case
mysql-test/suite/parts/t/partition_special_myisam.test:
  added test case
mysql-test/t/partition_innodb.test:
  added test case
sql/sql_base.cc:
  Moved alter_close_tables to sql_partition.cc
sql/sql_base.h:
  removed some non existing and duplicated functions.
sql/sql_partition.cc:
  fast_alter_partition_table:
  Spletted abort_and_upgrad_lock_and_close_table
  to its parts (wait_while_table_is_used and
  alter_close_tables) and always have
  wait_while_table_is_used before any persistent
  operations (including logs, which will be executed
  on failure) and alter_close_tables after
  create/read/write operations and before
  drop operations.
  
  moved alter_close_tables here from sql_base.cc
  
  Added error injections for better test coverage.
  
  write_log_final_change_partition:
  fixed a log_entry linking bug (delete_frm was not
  linked to change/drop partition)
  and drop partition must be executed before
  change partition (change partition can rename a
  partition to an old name, like REORG p1 INTO (p1,p2).
  
  write_log_add_change_partition:
  need to use drop_frm first, and relinking that entry
  and reusing its execute entry.
sql/sql_table.cc:
  added initialization of next_active_log_entry.
sql/table.h:
  removed a duplicate declaration.

mysql-test/r/partition_innodb.result

@@ -1,5 +1,67 @@
 drop table if exists t1, t2;
 #
+# Bug#50418: DROP PARTITION does not interact with transactions
+#
+CREATE TABLE t1 (
+id INT AUTO_INCREMENT NOT NULL,
+name CHAR(50) NOT NULL,
+myDate DATE NOT NULL,
+PRIMARY KEY (id, myDate),
+INDEX idx_date (myDate)
+) ENGINE=InnoDB
+PARTITION BY RANGE ( TO_DAYS(myDate) ) (
+PARTITION p0 VALUES LESS THAN (734028),
+PARTITION p1 VALUES LESS THAN (734029),
+PARTITION p2 VALUES LESS THAN (734030),
+PARTITION p3 VALUES LESS THAN MAXVALUE
+) ;
+INSERT INTO t1 VALUES 
+(NULL, 'Lachlan', '2009-09-13'),
+(NULL, 'Clint', '2009-09-13'),
+(NULL, 'John', '2009-09-14'),
+(NULL, 'Dave', '2009-09-14'),
+(NULL, 'Jeremy', '2009-09-15'),
+(NULL, 'Scott', '2009-09-15'),
+(NULL, 'Jeff', '2009-09-16'),
+(NULL, 'Joe', '2009-09-16');
+SET AUTOCOMMIT=0;
+SELECT * FROM t1 FOR UPDATE;
+id	name	myDate
+1	Lachlan	2009-09-13
+2	Clint	2009-09-13
+3	John	2009-09-14
+4	Dave	2009-09-14
+5	Jeremy	2009-09-15
+6	Scott	2009-09-15
+7	Jeff	2009-09-16
+8	Joe	2009-09-16
+UPDATE t1 SET name = 'Mattias' WHERE id = 7;
+SELECT * FROM t1 WHERE id = 7;
+id	name	myDate
+7	Mattias	2009-09-16
+# Connection con1
+SET lock_wait_timeout = 1;
+# After the patch it will wait and fail on timeout.
+ALTER TABLE t1 DROP PARTITION p3;
+ERROR HY000: Lock wait timeout exceeded; try restarting transaction
+SHOW WARNINGS;
+Level	Code	Message
+Error	1205	Lock wait timeout exceeded; try restarting transaction
+# Connection default
+SELECT * FROM t1;
+id	name	myDate
+1	Lachlan	2009-09-13
+2	Clint	2009-09-13
+3	John	2009-09-14
+4	Dave	2009-09-14
+5	Jeremy	2009-09-15
+6	Scott	2009-09-15
+7	Mattias	2009-09-16
+8	Joe	2009-09-16
+# No changes.
+COMMIT;
+DROP TABLE t1;
+#
 # Bug#51830: Incorrect partition pruning on range partition (regression)
 #
 CREATE TABLE t1 (a INT NOT NULL)