MDEV-11412 Ensure that table is truly dropped when using DROP TABLE

The used code is largely based on code from Tencent

The problem is that in some rare cases there may be a conflict between .frm
files and the files in the storage engine. In this case the DROP TABLE
was not able to properly drop the table.

Some MariaDB/MySQL forks has solved this by adding a FORCE option to
DROP TABLE. After some discussion among MariaDB developers, we concluded
that users expects that DROP TABLE should always work, even if the
table would not be consistent. There should not be a need to use a
separate keyword to ensure that the table is really deleted.

The used solution is:
- If a .frm table doesn't exists, try dropping the table from all storage
  engines.
- If the .frm table exists but the table does not exist in the engine
  try dropping the table from all storage engines.
- Update storage engines using many table files (.CVS, MyISAM, Aria) to
  succeed with the drop even if some of the files are missing.
- Add HTON_AUTOMATIC_DELETE_TABLE to handlerton's where delete_table()
  is not needed and always succeed. This is used by ha_delete_table_force()
  to know which handlers to ignore when trying to drop a table without
  a .frm file.

The disadvantage of this solution is that a DROP TABLE on a non existing
table will be a bit slower as we have to ask all active storage engines
if they know anything about the table.

Other things:
- Added a new flag MY_IGNORE_ENOENT to my_delete() to not give an error
  if the file doesn't exist. This simplifies some of the code.
- Don't clear thd->error in ha_delete_table() if there was an active
  error. This is a bug fix.
- handler::delete_table() will not abort if first file doesn't exists.
  This is bug fix to handle the case when a drop table was aborted in
  the middle.
- Cleaned up mysql_rm_table_no_locks() to ensure that if_exists uses
  same code path as when it's not used.
- Use non_existing_Table_error() to detect if table didn't exists.
  Old code used different errors tests in different position.
- Table_triggers_list::drop_all_triggers() now drops trigger file if
  it can't be parsed instead of leaving it hanging around (bug fix)
- InnoDB doesn't anymore print error about .frm file out of sync with
  InnoDB directory if .frm file does not exists. This change was required
  to be able to try to drop an InnoDB file when .frm doesn't exists.
- Fixed bug in mi_delete_table() where the .MYD file would not be dropped
  if the .MYI file didn't exists.
- Fixed memory leak in Mroonga when deleting non existing table
- Fixed memory leak in Connect when deleting non existing table

Bugs fixed introduced by the original version of this commit:
MDEV-22826 Presence of Spider prevents tables from being force-deleted from
           other engines

mysql-test/main/drop_table_force.test

@@ -0,0 +1,235 @@
+--source include/have_log_bin.inc
+--source include/have_innodb.inc
+--source include/have_archive.inc
+#
+# This test is based on the orginal test from Tencent for DROP TABLE ... FORCE
+# In MariaDB we did reuse the code but MariaDB does not require the FORCE
+# keyword to drop a table even if the .frm file or some engine files are
+# missing.
+# To make it easy to see the differences between the orginal code and
+# the new one, we have left some references to the original test case
+#
+
+CALL mtr.add_suppression("Operating system error number");
+CALL mtr.add_suppression("The error means the system cannot");
+CALL mtr.add_suppression("returned OS error 71");
+
+let $DATADIR= `select @@datadir`;
+
+--echo #Test1: table with missing .ibd can be dropped directly
+# drop table without ibd
+create table t1(a int)engine=innodb;
+--remove_file $DATADIR/test/t1.ibd
+drop table t1;
+--list_files  $DATADIR/test/
+
+# Original DROP TABLE .. FORCE required SUPER privilege. MariaDB doesn't
+--echo # Test droping table without frm without super privilege
+
+# create table t1 and rm frm
+create table t1(a int) engine=innodb;
+--remove_file $DATADIR/test/t1.frm
+
+# create test user
+create user test identified by '123456';
+grant all privileges on test.t1 to 'test'@'%'identified by '123456' with grant option;
+
+# connect as test
+connect (con_test, localhost, test,'123456', );
+--connection con_test
+
+# drop table with user test
+drop table t1;
+--error ER_BAD_TABLE_ERROR
+drop table t1;
+
+# connect as root
+--connection default
+
+--disconnect con_test
+drop user test;
+
+# check files in datadir about t1
+--list_files  $DATADIR/test/
+
+--echo #Test4: drop table can drop consistent table as well
+create table t1(a int) engine=innodb;
+drop table t1;
+
+# check files in datadir about t1
+--list_files  $DATADIR/test/
+
+--echo #Test5: drop table with triger, and with missing frm
+# create table t1 with triger and rm frm
+create table t1(a int)engine=innodb;
+create trigger t1_trg before insert on t1 for each row begin end;
+
+let $DATADIR= `select @@datadir`;
+--remove_file $DATADIR/test/t1.frm
+
+drop table t1;
+--error ER_BAD_TABLE_ERROR
+drop table t1;
+
+# check files in datadir about t1
+--list_files  $DATADIR/test/
+
+--echo #Test6: table with foreign key references can not be dropped
+# create table with foreign key reference and rm frm
+CREATE TABLE parent (id INT NOT NULL, PRIMARY KEY (id)) ENGINE=INNODB;
+CREATE TABLE child (id INT, parent_id INT, INDEX par_ind (parent_id), FOREIGN KEY (parent_id) REFERENCES parent(id) ON DELETE CASCADE) ENGINE=INNODB;
+--remove_file $DATADIR/test/parent.frm
+
+# parent can not be dropped when there are foreign key references
+--error ER_ROW_IS_REFERENCED_2
+drop table parent;
+
+# parent can be dropped when there are no foreign key references
+drop table child;
+drop table parent;
+
+# check files in datadir about child and parent
+--list_files  $DATADIR/test/
+
+--echo #Test7: drop table twice
+create table t1(a int)engine=innodb;
+--remove_file $DATADIR/test/t1.frm
+
+# first drop table will success
+drop table t1;
+
+# check files in datadir about t1
+--list_files  $DATADIR/test/
+
+# second drop with if exists will also ok
+drop table if exists t1;
+
+# check files in datadir about t1
+--list_files  $DATADIR/test/
+
+--echo #Test8: check compatibility with if exists
+create table t1(a int)engine=innodb;
+--remove_file $DATADIR/test/t1.frm
+
+# first drop will success
+drop table t1;
+
+# check files in datadir about t1
+--list_files  $DATADIR/test/
+
+# second drop with if exists will success
+drop table if exists t1;
+
+--echo #Test9: check compatibility with restrict/cascade
+# create table with foreign key reference and rm frm
+CREATE TABLE parent (id INT NOT NULL, PRIMARY KEY (id)) ENGINE=INNODB;
+CREATE TABLE child (id INT, parent_id INT, INDEX par_ind (parent_id), FOREIGN KEY (parent_id) REFERENCES parent(id) ON DELETE CASCADE) ENGINE=INNODB;
+
+# parent can not be dropped when there are foreign key references
+--error ER_ROW_IS_REFERENCED_2
+drop table parent;
+--error ER_ROW_IS_REFERENCED_2
+drop table parent restrict;
+--error ER_ROW_IS_REFERENCED_2
+drop table parent cascade;
+--error ER_ROW_IS_REFERENCED_2
+drop table parent;
+--error ER_ROW_IS_REFERENCED_2
+drop table parent restrict;
+--error ER_ROW_IS_REFERENCED_2
+drop table parent cascade;
+
+# parent can be dropped when there are no foreign key references
+drop table child;
+drop table parent;
+
+--echo #Test10: drop non-innodb engine table returns ok
+# create myisam table t1 and rm .frm
+create table t1(a int) engine=myisam;
+--remove_file $DATADIR/test/t1.frm
+--replace_result \\ /
+drop table t1;
+
+# create myisam table t1 and rm .MYD
+create table t1(a int) engine=myisam;
+--remove_file $DATADIR/test/t1.MYD
+--replace_result \\ /
+drop table t1;
+
+# create myisam table t1 and rm .MYI
+create table t1(a int) engine=myisam;
+--remove_file $DATADIR/test/t1.MYI
+--replace_result \\ /
+drop table t1;
+--list_files  $DATADIR/test/
+
+# create Aria table t1 and rm .frm and .MAD
+create table t1(a int) engine=aria;
+--remove_file $DATADIR/test/t1.frm
+--remove_file $DATADIR/test/t1.MAD
+--list_files  $DATADIR/test/
+--error ER_BAD_TABLE_ERROR
+drop table t1;
+--replace_result \\ /
+show warnings;
+--list_files  $DATADIR/test/
+
+# create Aria table t2 and rm .frm and .MAI
+create table t2(a int) engine=aria;
+flush tables;
+--remove_file $DATADIR/test/t2.frm
+--remove_file $DATADIR/test/t2.MAI
+--list_files  $DATADIR/test/
+--error ER_BAD_TABLE_ERROR
+drop table t2;
+--replace_result \\ /
+show warnings;
+--list_files  $DATADIR/test/
+
+# create Aria table t2 and rm .MAI and .MAD
+create table t2(a int) engine=aria;
+flush tables;
+--remove_file $DATADIR/test/t2.MAD
+--remove_file $DATADIR/test/t2.MAI
+--list_files  $DATADIR/test/
+--replace_result \\ /
+drop table t2;
+
+# create CVS table t2 and rm .frm
+create table t2(a int not null) engine=CSV;
+flush tables;
+--remove_file $DATADIR/test/t2.frm
+drop table t2;
+--list_files  $DATADIR/test/
+
+# create CVS table t2 and rm .frm
+create table t2(a int not null) engine=CSV;
+flush tables;
+--remove_file $DATADIR/test/t2.CSV
+drop table t2;
+--list_files  $DATADIR/test/
+
+# create Archive table t2 and rm 
+# Note that as Archive has discovery, removing the
+# ARZ will automatically remove the .frm
+
+create table t2(a int not null) engine=archive;
+flush tables;
+--error 1
+--remove_file $DATADIR/test/t2.frm
+select * from t2;
+flush tables;
+--remove_file $DATADIR/test/t2.ARZ
+--error ER_NO_SUCH_TABLE
+select * from t2;
+--list_files  $DATADIR/test/
+--replace_result \\ /
+--error ER_BAD_TABLE_ERROR
+drop table t2;
+
+create table t2(a int not null) engine=archive;
+flush tables;
+--remove_file $DATADIR/test/t2.ARZ
+--error ER_BAD_TABLE_ERROR
+drop table t2;
+--list_files  $DATADIR/test/