MDEV-5589 commit set up a policy to skip DROP TEMPORARY TABLE binary logging
in case the target table has not been "CREATEed" in binlog (no CREATE
Query-log-event was logged into the binary log).
It turns out that
1. the rule did not cover non-existing table DROPped with IF-EXISTS clause.
The logged-create knowledge for the non-existing one does not even need
MDEV-5589 patch, and
2. connection close disobeys it to trigger automatic DROP-IF-EXISTS
binlogging.
Either 1 or 2 or even both is/are also responsible for unexpected binlog
records observed in MDEV-17863, actually rendering a referred
@@global.read_only irrelevant as far as the described stored procedure
definition *and* the ROW binlog-format are concerned.
if we didn't write the CREATE TEMPORARY TABLE statement.
- Enable old code from one of my older changesets that didn't make into 10.0
- Fix test cased that failed as they expected DROP TEMPORARY TABLE in the log.
IN TIME RECOVERY FAILURE ON SLAVES
Problem:
DROP TEMP TABLE IF EXISTS commands can cause point
in time recovery (re-applying binlog) failures.
Analyses:
In RBR, 'DROP TEMPORARY TABLE' commands are
always binlogged by adding 'IF EXISTS' clauses.
Also, the slave SQL thread will not check replicate.* filter
rules for "DROP TEMPORARY TABLE IF EXISTS" queries.
If log-slave-updates is enabled on slave, these queries
will be binlogged in the format of "USE `db`;
DROP TEMPORARY TABLE IF EXISTS `t1`;" irrespective
of filtering rules and irrespective of the `db` existence.
When users try to recover slave from it's own binlog,
use `db` command might fail if `db` is not present on slave.
Fix:
At the time of writing the 'DROP TEMPORARY TABLE
IF EXISTS' query into the binlog, 'use `db`' will not be
present and the table name in the query will be a fully
qualified table name.
Eg:
'USE `db`; DROP TEMPORARY TABLE IF EXISTS `t1`;'
will be logged as
'DROP TEMPORARY TABLE IF EXISTS `db`.`t1`;'.
includes:
* remove some remnants of "Bug#14521864: MYSQL 5.1 TO 5.5 BUGS PARTITIONING"
* introduce LOCK_share, now LOCK_ha_data is strictly for engines
* rea_create_table() always creates .par file (even in "frm-only" mode)
* fix a 5.6 bug, temp file leak on dummy ALTER TABLE
Adjust full test suite to work with GTID.
Huge patch, mainly due to having to update .result file for all SHOW BINLOG
EVENTS and mysqlbinlog outputs, where the new GTID events pop up.
Everything was painstakingly checked to be still correct and valid .result
file updates.
and small collateral changes
mysql-test/lib/My/Test.pm:
somehow with "print" we get truncated writes sometimes
mysql-test/suite/perfschema/r/digest_table_full.result:
md5 hashes of statement digests differ, because yacc token codes are different in mariadb
mysql-test/suite/perfschema/r/dml_handler.result:
host table is not ported over yet
mysql-test/suite/perfschema/r/information_schema.result:
host table is not ported over yet
mysql-test/suite/perfschema/r/nesting.result:
this differs, because we don't rewrite general log queries, and multi-statement
packets are logged as a one entry. this result file is identical to what mysql-5.6.5
produces with the --log-raw option.
mysql-test/suite/perfschema/r/relaylog.result:
MariaDB modifies the binlog index file directly, while MySQL 5.6 has a feature "crash-safe binlog index" and modifies a special "crash-safe" shadow copy of the index file and then moves it over. That's why this test shows "NONE" index file writes in MySQL and "MANY" in MariaDB.
mysql-test/suite/perfschema/r/server_init.result:
MariaDB initializes the "manager" resources from the "manager" thread, and starts this thread only when --flush-time is not 0. MySQL 5.6 initializes "manager" resources unconditionally on server startup.
mysql-test/suite/perfschema/r/stage_mdl_global.result:
this differs, because MariaDB disables query cache when query_cache_size=0. MySQL does not
do that, and this causes useless mutex locks and waits.
mysql-test/suite/perfschema/r/statement_digest.result:
md5 hashes of statement digests differ, because yacc token codes are different in mariadb
mysql-test/suite/perfschema/r/statement_digest_consumers.result:
md5 hashes of statement digests differ, because yacc token codes are different in mariadb
mysql-test/suite/perfschema/r/statement_digest_long_query.result:
md5 hashes of statement digests differ, because yacc token codes are different in mariadb
mysql-test/suite/rpl/r/rpl_mixed_drop_create_temp_table.result:
will be updated to match 5.6 when alfranio.correia@oracle.com-20110512172919-c1b5kmum4h52g0ni and anders.song@greatopensource.com-20110105052107-zoab0bsf5a6xxk2y are merged
mysql-test/suite/rpl/r/rpl_non_direct_mixed_mixing_engines.result:
will be updated to match 5.6 when anders.song@greatopensource.com-20110105052107-zoab0bsf5a6xxk2y is merged
temp table
This patch introduces two key changes in the replication's behavior.
Firstly, it reverts part of BUG#51894 which puts any update to temporary tables
into the trx-cache. Now, updates to temporary tables are handled according to
the type of their engines as a regular table.
Secondly, an unsafe mixed statement, (i.e. a statement that access transactional
table as well non-transactional or temporary table, and writes to any of them),
are written into the trx-cache in order to minimize errors in the execution when
the statement logging format is in use.
Such changes has a direct impact on which statements are classified as unsafe
statements and thus part of BUG#53259 is reverted.
'CREATE TABLE IF NOT EXISTS ... SELECT' behaviour
BUG#47132, BUG#47442, BUG49494, BUG#23992 and BUG#48814 will disappear
automatically after the this patch.
BUG#55617 is fixed by this patch too.
This is the 5.5 part.
It implements:
- 'CREATE TABLE IF NOT EXISTS ... SELECT' statement will not insert
anything and binlog anything if the table already exists.
It only generate a warning that table already exists.
- A couple of test cases for the behavior changing.
switching binlog format to ROW
BUG 52616 fixed the case which the user would switch from STMT to
ROW binlog format, but the server would silently ignore it. After
that fix thd->is_current_stmt_binlog_format_row() reports correct
value at logging time and events are logged in ROW (as expected)
instead of STMT as they were previously and wrongly logged.
However, the fix was only partially complete, because on
disconnect, at THD cleanup, the implicit logging of temporary
tables is conditionally performed. If the binlog_format==ROW and
thd->is_current_stmt_binlog_format_row() is true then DROPs are
not logged. Given that the user can switch from STMT to ROW, this
is wrong because the server cannot tell, just by relying on the
ROW binlog format, that the tables have been dropped before. This
is effectively similar to the MIXED scenario when a switch from
STMT to ROW is triggered.
We fix this by removing this condition from
close_temporary_tables.
mysql-test/extra/binlog_tests/drop_temp_table.test:
Added binlog test case.
mysql-test/suite/binlog/r/binlog_row_drop_tmp_tbl.result:
Result changes because:
- there is a missing drop on three temporary tables
- it now contains results for the test added
mysql-test/suite/binlog/r/binlog_row_mix_innodb_myisam.result:
Result now contains the implicit drop for the temporary table.
mysql-test/suite/binlog/r/binlog_stm_drop_tmp_tbl.result:
Result file changed because it now contains results for added
test case.
mysql-test/suite/rpl/r/rpl_drop_temp.result:
Result file changed because it now contains results for added
test case.
mysql-test/suite/rpl/t/rpl_drop_temp.test:
Added replication test case.
sql/sql_base.cc:
Removed the condition that would make the server to skip
logging implicit drops when ROW binary log format mode was
in use.
Additionally, deployed DBUG_ENTER/RETURN macros.
The test was used to fail because of
UPDATE t3,t4 SET t3.a=t4.a + bug27417(1);
did not prescribe the order of two row operations implied by the update.
Fixed with forcing the order with adding a where condition w/o
affecting the former bug fixes logics.
mysql-test/extra/binlog_tests/mix_innodb_myisam_side_effects.test:
implemented the bug page suggestion to make a test deterministic.
mysql-test/suite/binlog/r/binlog_row_mix_innodb_myisam.result:
results are updated and (!) corrected.
mysql-test/suite/binlog/r/binlog_stm_mix_innodb_myisam.result:
results are updated
transaction
BUG#52616 Temp table prevents switch binlog format from STATEMENT to ROW
Before the WL#2687 and BUG#46364, every non-transactional change that happened
after a transactional change was written to trx-cache and flushed upon
committing the transaction. WL#2687 and BUG#46364 changed this behavior and
non-transactional changes are now written to the binary log upon committing
the statement.
A binary log event is identified as transactional or non-transactional through
a flag in the Log_event which is set taking into account the underlie storage
engine on what it is stems from. In the current bug, this flag was not being
set properly when the DROP TEMPORARY TABLE was executed.
However, while fixing this bug we figured out that changes to temporary tables
should be always written to the trx-cache if there is an on-going transaction.
Otherwise, binlog events in the reversed order would be produced.
Regarding concurrency, keeping changes to temporary tables in the trx-cache is
also safe as temporary tables are only visible to the owner connection.
In this patch, we classify the following statements as unsafe:
1 - INSERT INTO t_myisam SELECT * FROM t_myisam_temp
2 - INSERT INTO t_myisam_temp SELECT * FROM t_myisam
3 - CREATE TEMPORARY TABLE t_myisam_temp SELECT * FROM t_myisam
On the other hand, the following statements are classified as safe:
1 - INSERT INTO t_innodb SELECT * FROM t_myisam_temp
2 - INSERT INTO t_myisam_temp SELECT * FROM t_innodb
The patch also guarantees that transactions that have a DROP TEMPORARY are
always written to the binary log regardless of the mode and the outcome:
commit or rollback. In particular, the DROP TEMPORARY is extended with the
IF EXISTS clause when the current statement logging format is set to row.
Finally, the patch allows to switch from STATEMENT to MIXED/ROW when there
are temporary tables but the contrary is not possible.
mysql-test/extra/rpl_tests/rpl_binlog_max_cache_size.test:
Updated the test case because
CREATE TEMPORARY TABLE t_innodb_temp SELECT * FROM t_myisam is not unsafe.
mysql-test/extra/rpl_tests/rpl_implicit_commit_binlog.test:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/extra/rpl_tests/rpl_innodb.test:
Removed comments from the test case that became false after the patch.
mysql-test/extra/rpl_tests/rpl_loaddata.test:
Suppressed warning messages due to the following cases:
1 - INSERT INTO t_myisam SELECT * FROM t_myisam_temp
2 - INSERT INTO t_myisam_temp SELECT * FROM t_myisam
mysql-test/include/ctype_utf8_table.inc:
Suppressed warning messages due to the following cases:
1 - INSERT INTO t_myisam SELECT * FROM t_myisam_temp
2 - INSERT INTO t_myisam_temp SELECT * FROM t_myisam
mysql-test/r/ctype_cp932_binlog_stm.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/binlog/r/binlog_database.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/binlog/r/binlog_innodb_row.result:
Updated the result file.
mysql-test/suite/binlog/r/binlog_multi_engine.result:
Updated the unsafe message.
mysql-test/suite/binlog/r/binlog_row_binlog.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/binlog/r/binlog_row_drop_tmp_tbl.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/binlog/r/binlog_row_mix_innodb_myisam.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/binlog/r/binlog_stm_binlog.result:
Updated the result file.
mysql-test/suite/binlog/r/binlog_stm_mix_innodb_myisam.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/binlog/t/binlog_tmp_table.test:
Suppressed warning messages due to the following cases:
1 - INSERT INTO t_myisam SELECT * FROM t_myisam_temp
2 - INSERT INTO t_myisam_temp SELECT * FROM t_myisam
mysql-test/suite/ndb/r/ndb_binlog_format.result:
Updated the unsafe message.
mysql-test/suite/rpl/r/rpl_concurrency_error.result:
Updated the unsafe message.
mysql-test/suite/rpl/r/rpl_mixed_binlog_max_cache_size.result:
Updated the result file because
CREATE TEMPORARY TABLE t_innodb_temp SELECT * FROM t_myisam is not unsafe.
mysql-test/suite/rpl/r/rpl_mixed_implicit_commit_binlog.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/rpl/r/rpl_mixed_mixing_engines.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/rpl/r/rpl_mixed_row_innodb.result:
Added some comments to ease the understanding of the result file.
mysql-test/suite/rpl/r/rpl_non_direct_mixed_mixing_engines.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/rpl/r/rpl_non_direct_row_mixing_engines.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/rpl/r/rpl_non_direct_stm_mixing_engines.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/rpl/r/rpl_row_drop.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/rpl/r/rpl_row_implicit_commit_binlog.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/rpl/r/rpl_row_mixing_engines.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/rpl/r/rpl_stm_binlog_max_cache_size.result:
Updated the result file because
CREATE TEMPORARY TABLE t_innodb_temp SELECT * FROM t_myisam is not unsafe.
mysql-test/suite/rpl/r/rpl_stm_implicit_commit_binlog.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/rpl/r/rpl_stm_innodb.result:
Added some comments to ease the understanding of the result file.
mysql-test/suite/rpl/r/rpl_stm_mixing_engines.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/rpl/r/rpl_stm_stop_middle_group.result:
Updated the unsafe message.
mysql-test/suite/rpl/r/rpl_temp_temporary.result:
Added a test case.
mysql-test/suite/rpl/t/rpl000013.test:
Suppressed warning messages due to the following cases:
1 - INSERT INTO t_myisam SELECT * FROM t_myisam_temp
2 - INSERT INTO t_myisam_temp SELECT * FROM t_myisam
3 - CREATE TEMPORARY TABLE t_myisam_temp SELECT * FROM t_myisam
mysql-test/suite/rpl/t/rpl_misc_functions.test:
Suppressed warning messages.
mysql-test/suite/rpl/t/rpl_temp_table.test:
Suppressed warning messages due to the following cases:
1 - INSERT INTO t_myisam SELECT * FROM t_myisam_temp
2 - INSERT INTO t_myisam_temp SELECT * FROM t_myisam
3 - CREATE TEMPORARY TABLE t_myisam_temp SELECT * FROM t_myisam
mysql-test/suite/rpl/t/rpl_temp_temporary.test:
Added a test case.
mysql-test/suite/rpl/t/rpl_temporary.test:
Suppressed warning messages due to the following cases:
1 - INSERT INTO t_myisam SELECT * FROM t_myisam_temp
2 - INSERT INTO t_myisam_temp SELECT * FROM t_myisam
3 - CREATE TEMPORARY TABLE t_myisam_temp SELECT * FROM t_myisam
mysql-test/suite/rpl_ndb/r/rpl_ndb_row_implicit_commit_binlog.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/rpl_ndb/r/rpl_truncate_7ndb.result:
Updated the test case to remove references to positions
in the binary log.
mysql-test/suite/rpl_ndb/t/rpl_truncate_7ndb.test:
Updated the test case to remove references to positions
in the binary log.
mysql-test/t/create_select_tmp.test:
Suppressed warning messages due to the following cases:
1 - INSERT INTO t_myisam SELECT * FROM t_myisam_temp
2 - INSERT INTO t_myisam_temp SELECT * FROM t_myisam
3 - CREATE TEMPORARY TABLE t_myisam_temp SELECT * FROM t_myisam
mysql-test/t/ctype_cp932_binlog_stm.test:
Suppressed warning messages due to the following cases:
1 - INSERT INTO t_myisam SELECT * FROM t_myisam_temp
2 - INSERT INTO t_myisam_temp SELECT * FROM t_myisam
3 - CREATE TEMPORARY TABLE t_myisam_temp SELECT * FROM t_myisam
mysql-test/t/mysqlbinlog.test:
Suppressed warning messages due to the following cases:
1 - INSERT INTO t_myisam SELECT * FROM t_myisam_temp
2 - INSERT INTO t_myisam_temp SELECT * FROM t_myisam
3 - CREATE TEMPORARY TABLE t_myisam_temp SELECT * FROM t_myisam
sql/log.cc:
Improved the code by creating several functions to hide decision
on type of engine changed, commit/abort, etc:
. stmt_has_updated_non_trans_table
. trans_has_updated_non_trans_table
. ending_trans
Updated the binlog_rollback function and the use of the
OPTION_KEEP_LOG which indincates when a temporary table was
either created or dropped and as such the command must be
logged if not in MIXED mode and even while rolling back the
transaction.
sql/log.h:
Improved the code by creating several functions to hide decision
on type of engine changed, commit/abort, etc.
sql/log_event.cc:
Removed the setting of the OPTION_KEEP_LOG as it is related to CREATE
TEMPORARY and DROP TEMPORARY and not to the type of engine (i.e.
transactional or non-transactional).
sql/log_event_old.cc:
Removed the setting of the OPTION_KEEP_LOG as it is related to CREATE
TEMPORARY and DROP TEMPORARY and not to the type of engine (i.e.
transactional or non-transactional).
sql/share/errmsg-utf8.txt:
Updated the unsafe message.
sql/sql_class.cc:
Classifies the following statements as unsafe:
1 - INSERT INTO t_myisam SELECT * FROM t_myisam_temp
2 - INSERT INTO t_myisam_temp SELECT * FROM t_myisam
On the other hand, the following statements are classified as safe:
1 - INSERT INTO t_innodb SELECT * FROM t_myisam_temp
2 - INSERT INTO t_myisam_temp SELECT * FROM t_innodb
sql/sql_class.h:
It allows to switch from STATEMENT to MIXED/ROW when there are temporary
tables but the contrary is not possible.
sql/sql_table.cc:
Fixed the case that a DROP/DROP TEMPORARY that affects a temporary table in MIXED
mode is written as a DROP TEMPORARY TABLE IF EXISTS because the table may not exist in the slave and due to the IF EXISTS token an error will never happen
while processing the statement in the slave.
Removed a function that was not being used.
Conflicts:
Text conflict in mysql-test/suite/binlog/r/binlog_row_mix_innodb_myisam.result
Text conflict in sql/log.cc
Text conflict in sql/set_var.cc
Text conflict in sql/sql_class.cc
Conflicts:
Text conflict in client/mysqlbinlog.cc
Text conflict in mysql-test/Makefile.am
Text conflict in mysql-test/collections/default.daily
Text conflict in mysql-test/r/mysqlbinlog_row_innodb.result
Text conflict in mysql-test/suite/rpl/r/rpl_typeconv_innodb.result
Text conflict in mysql-test/suite/rpl/t/rpl_get_master_version_and_clock.test
Text conflict in mysql-test/suite/rpl/t/rpl_row_create_table.test
Text conflict in mysql-test/suite/rpl/t/rpl_slave_skip.test
Text conflict in mysql-test/suite/rpl/t/rpl_typeconv_innodb.test
Text conflict in mysys/charset.c
Text conflict in sql/field.cc
Text conflict in sql/field.h
Text conflict in sql/item.h
Text conflict in sql/item_func.cc
Text conflict in sql/log.cc
Text conflict in sql/log_event.cc
Text conflict in sql/log_event_old.cc
Text conflict in sql/mysqld.cc
Text conflict in sql/rpl_utility.cc
Text conflict in sql/rpl_utility.h
Text conflict in sql/set_var.cc
Text conflict in sql/share/Makefile.am
Text conflict in sql/sql_delete.cc
Text conflict in sql/sql_plugin.cc
Text conflict in sql/sql_select.cc
Text conflict in sql/sql_table.cc
Text conflict in storage/example/ha_example.h
Text conflict in storage/federated/ha_federated.cc
Text conflict in storage/myisammrg/ha_myisammrg.cc
Text conflict in storage/myisammrg/myrg_open.c
When mysqlbinlog was given the --database=X flag, it always printed
'ROLLBACK TO', but the corresponding 'SAVEPOINT' statement was not
printed. The replicated filter(replicated-do/ignore-db) and binlog
filter (binlog-do/ignore-db) has the same problem. They are solved
in this patch together.
After this patch, We always check whether the query is 'SAVEPOINT'
statement or not. Because this is a literal check, 'SAVEPOINT' and
'ROLLBACK TO' statements are also binlogged in uppercase with no
any comments.
The binlog before this patch can be handled correctly except one case
that any comments are in front of the keywords. for example:
/* bla bla */ SAVEPOINT a;
/* bla bla */ ROLLBACK TO a;
For temporary tables that are created with an engine that does
not provide the HTON_CAN_RECREATE, the truncate operation is
performed resorting to the optimized handler::ha_delete_all_rows
method. However, this means that the truncate will share
execution path, from mysql_delete, with truncate on regular
tables and other delete operations. As a consequence the truncate
operation, for the temporary table is logged, even if in row mode
because there is no distinction between this and the other delete
operations at binlogging time.
We fix this by checking if: (i) the binlog format, when the
truncate operation was issued, is ROW; (ii) if the operation is a
truncate; and (iii) if the table is a temporary table; before
writing to the binary log. If all three conditions are met, we
skip writing to the binlog. A side effect of this fix is that we
limit the scope of setting and resetting the
current_stmt_binlog_row_based. Now we just set and reset it
inside mysql_delete in the boundaries of the
handler::ha_write_row loop. This way we have access to
thd->current_stmt_binlog_row_based real value inside
mysql_delete.
mysql-test/suite/binlog/r/binlog_row_mix_innodb_myisam.result:
Updated result for spurious truncate table.
mysql-test/suite/binlog/t/binlog_row_innodb_truncate.test:
Test case.
sql/sql_delete.cc:
Added check in mysql_delete before writing the TRUNCATE statement
to the binary log. Additionally, removed the set/reset of
current_stmt_binlog_row_based so that it happens just in the
boundaries of the handler::ha_write_row loop inside mysql_delete.
As part of BUG@39934 fix, the public:
- THD::current_stmt_binlog_row_based
variable had been removed and replaced by a private variable:
- THD::current_stmt_binlog_format.
THD was refactored and some modifiers and accessors were
implemented for the new variable.
However, due to a bad merge, the
THD::current_stmt_binlog_row_based variable is back as a public
member of THD. This in itself is already potentially
harmful. What's even worse is that while merging some more
patches and resolving conflicts, the variable started being used
again, which is obviously wrong.
To fix this we:
1. remove the extraneous variable from sql_class.h
2. revert a bad merge for BUG#49132
3. merge BUG#49132 properly again (actually, making use of the
cset used to merge the original patch to mysql-pe).
In RBR, DDL statement will change binlog format to non row-based
format before it is binlogged, but the binlog format was not be
restored, and then manipulating a temporary table can not reset binlog
format to row-based format rightly. So that the manipulated statement
is binlogged with statement-based format.
To fix the problem, restore the state of binlog format after the DDL
statement is binlogged.
mysql-test/extra/rpl_tests/rpl_tmp_table_and_DDL.test:
Added the test file to verify if executing DDL statement before
trying to manipulate a temporary table causes row-based replication
to break with error 'table does not exist'.
mysql-test/suite/binlog/r/binlog_row_mix_innodb_myisam.result:
Correct the test result, all the above binlog event
should be row-based after the bug49132 is fixed IN RBR.
mysql-test/suite/ndb/r/ndb_tmp_table_and_DDL.result:
Test result for bug#49132 base on ndb engine.
mysql-test/suite/ndb/t/ndb_tmp_table_and_DDL.test:
Added the test file to verify if executing DDL statement before
trying to manipulate a temporary table causes row-based replication
to break with error 'table does not exist' base on ndb engine.
mysql-test/suite/rpl/r/rpl_tmp_table_and_DDL.result:
Test result for bug#49132 base on myisam engine.
mysql-test/suite/rpl/t/rpl_tmp_table_and_DDL.test:
Added the test file to verify if executing DDL statement before
trying to manipulate a temporary table causes row-based replication
to break with error 'table does not exist' base on myisam engine.
sql/event_db_repository.cc:
Added code to restore the state of binlog format after the DDL
statement is binlogged.
sql/events.cc:
Added code to restore the state of binlog format after the DDL
statement is binlogged.
sql/sp.cc:
Added code to restore the state of binlog format after the DDL
statement is binlogged.
sql/sql_acl.cc:
Added code to restore the state of binlog format after the DDL
statement is binlogged.
sql/sql_udf.cc:
Added code to restore the state of binlog format after the DDL
statement is binlogged.
escaped field names
When in mixed or statement mode, the master logs LOAD DATA
queries by resorting to an Execute_load_query_log_event. This
event does not contain the original query, but a rewritten
version of it, which includes the table field names. However, the
rewrite does not escape the field names. If these names match a
reserved keyword, then the slave will stop with a syntax error
when executing the event.
We fix this by escaping the fields names as it happens already
for the table name.
mysql-test/extra/rpl_tests/rpl_loaddata.test:
Added test case for the reported bug.
mysql-test/r/mysqlbinlog.result:
Changed result to support escaped field name.
mysql-test/suite/binlog/r/binlog_killed_simulate.result:
Changed result to support escaped field name.
mysql-test/suite/binlog/r/binlog_row_mix_innodb_myisam.result:
Changed result to support escaped field name.
mysql-test/suite/binlog/r/binlog_stm_blackhole.result:
Changed result to support escaped field name.
mysql-test/suite/binlog/r/binlog_stm_mix_innodb_myisam.result:
Changed result to support escaped field name.
mysql-test/suite/rpl/r/rpl_innodb_mixed_dml.result:
Changed result to support escaped field name.
mysql-test/suite/rpl/r/rpl_loaddata.result:
Added result for new test.
Changed show slave status positions which are now different because of
extra escape character in field names.
mysql-test/suite/rpl/r/rpl_loaddata_fatal.result:
Changed show slave status positions which are now different because of
extra escape character.
mysql-test/suite/rpl/r/rpl_loaddata_map.result:
Changed result to support escaped field name.
mysql-test/suite/rpl/r/rpl_stm_log.result:
Changed result to support escaped field name.
mysql-test/t/mysqlbinlog.test:
Changed positions which is now different because of extra escape
character in field names.
sql/sql_load.cc:
Appended escape characters before and after field names.
2617.31.12, 2617.31.15, 2617.31.15, 2617.31.16, 2617.43.1
- initial changeset that introduced the fix for
Bug#989 and follow up fixes for all test suite failures
introduced in the initial changeset.
------------------------------------------------------------
revno: 2617.31.1
committer: Davi Arnaut <Davi.Arnaut@Sun.COM>
branch nick: 4284-6.0
timestamp: Fri 2009-03-06 19:17:00 -0300
message:
Bug#989: If DROP TABLE while there's an active transaction, wrong binlog order
WL#4284: Transactional DDL locking
Currently the MySQL server does not keep metadata locks on
schema objects for the duration of a transaction, thus failing
to guarantee the integrity of the schema objects being used
during the transaction and to protect then from concurrent
DDL operations. This also poses a problem for replication as
a DDL operation might be replicated even thought there are
active transactions using the object being modified.
The solution is to defer the release of metadata locks until
a active transaction is either committed or rolled back. This
prevents other statements from modifying the table for the
entire duration of the transaction. This provides commitment
ordering for guaranteeing serializability across multiple
transactions.
- Incompatible change:
If MySQL's metadata locking system encounters a lock conflict,
the usual schema is to use the try and back-off technique to
avoid deadlocks -- this schema consists in releasing all locks
and trying to acquire them all in one go.
But in a transactional context this algorithm can't be utilized
as its not possible to release locks acquired during the course
of the transaction without breaking the transaction commitments.
To avoid deadlocks in this case, the ER_LOCK_DEADLOCK will be
returned if a lock conflict is encountered during a transaction.
Let's consider an example:
A transaction has two statements that modify table t1, then table
t2, and then commits. The first statement of the transaction will
acquire a shared metadata lock on table t1, and it will be kept
utill COMMIT to ensure serializability.
At the moment when the second statement attempts to acquire a
shared metadata lock on t2, a concurrent ALTER or DROP statement
might have locked t2 exclusively. The prescription of the current
locking protocol is that the acquirer of the shared lock backs off
-- gives up all his current locks and retries. This implies that
the entire multi-statement transaction has to be rolled back.
- Incompatible change:
FLUSH commands such as FLUSH PRIVILEGES and FLUSH TABLES WITH READ
LOCK won't cause locked tables to be implicitly unlocked anymore.
mysql-test/extra/binlog_tests/drop_table.test:
Add test case for Bug#989.
mysql-test/extra/binlog_tests/mix_innodb_myisam_binlog.test:
Fix test case to reflect the fact that transactions now hold
metadata locks for the duration of a transaction.
mysql-test/include/mix1.inc:
Fix test case to reflect the fact that transactions now hold
metadata locks for the duration of a transaction.
mysql-test/include/mix2.inc:
Fix test case to reflect the fact that transactions now hold
metadata locks for the duration of a transaction.
mysql-test/r/flush_block_commit.result:
Update test case result (WL#4284).
mysql-test/r/flush_block_commit_notembedded.result:
Update test case result (WL#4284).
mysql-test/r/innodb.result:
Update test case result (WL#4284).
mysql-test/r/innodb_mysql.result:
Update test case result (WL#4284).
mysql-test/r/lock.result:
Add test case result for an effect of WL#4284/Bug#989
(all locks should be released when a connection terminates).
mysql-test/r/mix2_myisam.result:
Update test case result (effects of WL#4284/Bug#989).
mysql-test/r/not_embedded_server.result:
Update test case result (effects of WL#4284/Bug#989).
Add a test case for interaction of WL#4284 and FLUSH PRIVILEGES.
mysql-test/r/partition_innodb_semi_consistent.result:
Update test case result (effects of WL#4284/Bug#989).
mysql-test/r/partition_sync.result:
Temporarily disable the test case for Bug#43867,
which will be fixed by a subsequent backport.
mysql-test/r/ps.result:
Add a test case for effect of PREPARE on transactional
locks: we take a savepoint at beginning of PREAPRE
and release it at the end. Thus PREPARE does not
accumulate metadata locks (Bug#989/WL#4284).
mysql-test/r/read_only_innodb.result:
Update test case result (effects of WL#4284/Bug#989).
mysql-test/suite/binlog/r/binlog_row_drop_tbl.result:
Add a test case result (WL#4284/Bug#989).
mysql-test/suite/binlog/r/binlog_row_mix_innodb_myisam.result:
Update test case result (effects of WL#4284/Bug#989).
mysql-test/suite/binlog/r/binlog_stm_drop_tbl.result:
Add a test case result (WL#4284/Bug#989).
mysql-test/suite/binlog/r/binlog_stm_mix_innodb_myisam.result:
Update test case result (effects of WL#4284/Bug#989).
mysql-test/suite/binlog/r/binlog_unsafe.result:
A side effect of Bug#989 -- slightly different table map ids.
mysql-test/suite/binlog/t/binlog_row_drop_tbl.test:
Add a test case for WL#4284/Bug#989.
mysql-test/suite/binlog/t/binlog_stm_drop_tbl.test:
Add a test case for WL#4284/Bug#989.
mysql-test/suite/binlog/t/binlog_stm_row.test:
Update to the new state name. This
is actually a follow up to another patch for WL#4284,
that changes Locked thread state to Table lock.
mysql-test/suite/ndb/r/ndb_index_ordered.result:
Remove result for disabled part of the test case.
mysql-test/suite/ndb/t/disabled.def:
Temporarily disable a test case (Bug#45621).
mysql-test/suite/ndb/t/ndb_index_ordered.test:
Disable a part of a test case (needs update to
reflect semantics of Bug#989).
mysql-test/suite/rpl/t/disabled.def:
Disable tests made meaningless by transactional metadata
locking.
mysql-test/suite/sys_vars/r/autocommit_func.result:
Add a commit (Bug#989).
mysql-test/suite/sys_vars/t/autocommit_func.test:
Add a commit (Bug#989).
mysql-test/t/flush_block_commit.test:
Fix test case to reflect the fact that transactions now hold
metadata locks for the duration of a transaction.
mysql-test/t/flush_block_commit_notembedded.test:
Fix test case to reflect the fact that transactions now hold
metadata locks for the duration of a transaction.
Add a test case for transaction-scope locks and the global
read lock (Bug#989/WL#4284).
mysql-test/t/innodb.test:
Fix test case to reflect the fact that transactions now hold
metadata locks for the duration of a transaction
(effects of Bug#989/WL#4284).
mysql-test/t/lock.test:
Add a test case for Bug#989/WL#4284.
mysql-test/t/not_embedded_server.test:
Add a test case for Bug#989/WL#4284.
mysql-test/t/partition_innodb_semi_consistent.test:
Replace TRUNCATE with DELETE, to not issue
an implicit commit of a transaction, and not depend
on metadata locks.
mysql-test/t/partition_sync.test:
Temporarily disable the test case for Bug#43867,
which needs a fix to be backported from 6.0.
mysql-test/t/ps.test:
Add a test case for semantics of PREPARE and transaction-scope
locks: metadata locks on tables used in PREPARE are enclosed into a
temporary savepoint, taken at the beginning of PREPARE,
and released at the end. Thus PREPARE does not effect
what locks a transaction owns.
mysql-test/t/read_only_innodb.test:
Fix test case to reflect the fact that transactions now hold
metadata locks for the duration of a transaction
(Bug#989/WL#4284).
Wait for the read_only statement to actually flush tables before
sending other concurrent statements that depend on its state.
mysql-test/t/xa.test:
Fix test case to reflect the fact that transactions now hold
metadata locks for the duration of a transaction
(Bug#989/WL#4284).
sql/ha_ndbcluster_binlog.cc:
Backport bits of changes of ha_ndbcluster_binlog.cc
from 6.0, to fix the failing binlog test suite with
WL#4284. WL#4284 implementation does not work
with 5.1 implementation of ndbcluster binlog index.
sql/log_event.cc:
Release metadata locks after issuing a commit.
sql/mdl.cc:
Style changes (WL#4284).
sql/mysql_priv.h:
Rename parameter to match the name used in the definition (WL#4284).
sql/rpl_injector.cc:
Release metadata locks on commit (WL#4284).
sql/rpl_rli.cc:
Remove assert made meaningless, metadata locks are released
at the end of the transaction.
sql/set_var.cc:
Close tables and release locks if autocommit mode is set.
sql/slave.cc:
Release metadata locks after a rollback.
sql/sql_acl.cc:
Don't implicitly unlock locked tables. Issue a implicit commit
at the end and unlock tables.
sql/sql_base.cc:
Defer the release of metadata locks when closing tables
if not required to.
Issue a deadlock error if the locking protocol requires
that a transaction re-acquire its locks.
Release metadata locks when closing tables for reopen.
sql/sql_class.cc:
Release metadata locks if the thread is killed.
sql/sql_parse.cc:
Release metadata locks after implicitly committing a active
transaction, or after explicit commits or rollbacks.
sql/sql_plugin.cc:
Allocate MDL request on the stack as the use of the table
is contained within the function. It will be removed from
the context once close_thread_tables is called at the end
of the function.
sql/sql_prepare.cc:
The problem is that the prepare phase of the CREATE TABLE
statement takes a exclusive metadata lock lock and this can
cause a self-deadlock the thread already holds a shared lock
on the table being that should be created.
The solution is to make the prepare phase take a shared
metadata lock when preparing a CREATE TABLE statement. The
execution of the statement will still acquire a exclusive
lock, but won't cause any problem as it issues a implicit
commit.
After some discussions with stakeholders it has been decided that
metadata locks acquired during a PREPARE statement must be released
once the statement is prepared even if it is prepared within a multi
statement transaction.
sql/sql_servers.cc:
Don't implicitly unlock locked tables. Issue a implicit commit
at the end and unlock tables.
sql/sql_table.cc:
Close table and release metadata locks after a admin operation.
sql/table.h:
The problem is that the prepare phase of the CREATE TABLE
statement takes a exclusive metadata lock lock and this can
cause a self-deadlock the thread already holds a shared lock
on the table being that should be created.
The solution is to make the prepare phase take a shared
metadata lock when preparing a CREATE TABLE statement. The
execution of the statement will still acquire a exclusive
lock, but won't cause any problem as it issues a implicit
commit.
sql/transaction.cc:
Release metadata locks after the implicitly committed due
to a new transaction being started. Also, release metadata
locks acquired after a savepoint if the transaction is rolled
back to the save point.
The problem is that in some cases transaction-long metadata locks
could be released before the transaction was committed. This could
happen when a active transaction was ended by a "START TRANSACTION"
or "BEGIN" statement, in which case the metadata locks would be
released before the actual commit of the active transaction.
The solution is to defer the release of metadata locks to after the
transaction has been implicitly committed. No test case is provided
as the effort to provide one is too disproportional to the size of
the fix.
Non-transactional updates that take place inside a transaction present problems
for logging because they are visible to other clients before the transaction
is committed, and they are not rolled back even if the transaction is rolled
back. It is not always possible to log correctly in statement format when both
transactional and non-transactional tables are used in the same transaction.
In the current patch, we ensure that such scenario is completely safe under the
ROW and MIXED modes.
Let
- T be a transactional table and N non-transactional table.
- B be begin, C commit and R rollback.
- N be a statement that accesses and changes only N-tables.
- T be a statement that accesses and changes only T-tables.
In RBR, changes to N-tables that happen early in a transaction are not immediately flushed
upon committing a statement. This behavior may, however, break consistency in the presence
of concurrency since changes done to N-tables become immediately visible to other
connections. To fix this problem, we do the following:
. B N N T C would log - B N C B N C B T C.
. B N N T R would log - B N C B N C B T R.
Note that we are not preserving history from the master as we are introducing a commit that
never happened. However, this seems to be more acceptable than the possibility of breaking
consistency in the presence of concurrency.
"load data" statements were written to the binlog as a mix of the original statement
and bits recreated from parse-info. This relied on implementation details and broke
with IGNORE_SPACES and versioned comments.
We now completely resynthesize the query for LOAD DATA for binlog (which among other
things normalizes them somewhat with regard to case, spaces, etc.).
We have already parsed the query properly, so we make use of that rather
than mix-and-match string literals and parsed items.
This should make us safe with regard to versioned comments, even those
spanning multiple tokens. Also no longer affected by IGNORE_SPACES.
mysql-test/r/mysqlbinlog.result:
LOAD DATA INFILE normalized
mysql-test/suite/binlog/r/binlog_killed_simulate.result:
LOAD DATA INFILE normalized
mysql-test/suite/binlog/r/binlog_row_mix_innodb_myisam.result:
LOAD DATA INFILE normalized
mysql-test/suite/binlog/r/binlog_stm_blackhole.result:
LOAD DATA INFILE normalized
mysql-test/suite/binlog/r/binlog_stm_mix_innodb_myisam.result:
LOAD DATA INFILE normalized
mysql-test/suite/rpl/r/rpl_innodb_mixed_dml.result:
LOAD DATA INFILE normalized
mysql-test/suite/rpl/r/rpl_loaddata.result:
LOAD DATA INFILE normalized
mysql-test/suite/rpl/r/rpl_loaddata_fatal.result:
LOAD DATA INFILE normalized; offsets adjusted to reflect that
mysql-test/suite/rpl/r/rpl_loaddata_map.result:
LOAD DATA INFILE normalized
mysql-test/suite/rpl/r/rpl_loaddatalocal.result:
test for #43746 - trying to break LOAD DATA part of parser
mysql-test/suite/rpl/r/rpl_stm_log.result:
LOAD DATA INFILE normalized
mysql-test/suite/rpl/t/rpl_loaddatalocal.test:
try to break the LOAD DATA part of the parser (test for #43746)
mysql-test/t/mysqlbinlog.test:
LOAD DATA INFILE normalized; adjust offsets to reflect that
sql/log_event.cc:
clean up Load_log_event::print_query and friends so they don't print
excess spaces. add support for printing charset names to print_query.
sql/log_event.h:
We already have three places where we synthesize LOAD DATA queries.
Better use one of those!
sql/sql_lex.h:
When binlogging LOAD DATA statements, we make up the statement to
be logged (from the parse-info, rather than substrings of the
original query) now. Consequently, we no longer need (string-)
pointers into the original query.
sql/sql_load.cc:
Completely rewrote write_execute_load_query_log_event() to synthesize the
LOAD DATA statement wholesale, rather than piece it together from
synthesized bits and literal excerpts from the original query. This
will not only give us a nice, normalized statement (all uppercase,
no excess spaces, etc.), it will also handle comments, including
versioned comments right, which is certainly more than we can say
about the previous incarnation.
sql/sql_yacc.yy:
We're no longer assembling LOAD DATA statements from bodyparts of the
original query, so some bookkeeping in the parser can go.
binlog
Mixing transactional (T) and non-transactional (N) tables on behalf of a
transaction may lead to inconsistencies among masters and slaves in STATEMENT
mode. The problem stems from the fact that although modifications done to
non-transactional tables on behalf of a transaction become immediately visible
to other connections they do not immediately get to the binary log and therefore
consistency is broken. Although there may be issues in mixing T and M tables in
STATEMENT mode, there are safe combinations that clients find useful.
In this bug, we fix the following issue. Mixing N and T tables in multi-level
(e.g. a statement that fires a trigger) or multi-table table statements (e.g.
update t1, t2...) were not handled correctly. In such cases, it was not possible
to distinguish when a T table was updated if the sequence of changes was N and T.
In a nutshell, just the flag "modified_non_trans_table" was not enough to reflect
that both a N and T tables were changed. To circumvent this issue, we check if an
engine is registered in the handler's list and changed something which means that
a T table was modified.
Check WL 2687 for a full-fledged patch that will make the use of either the MIXED or
ROW modes completely safe.
mysql-test/suite/binlog/r/binlog_row_mix_innodb_myisam.result:
Truncate statement is wrapped in BEGIN/COMMIT.
mysql-test/suite/binlog/r/binlog_stm_mix_innodb_myisam.result:
Truncate statement is wrapped in BEGIN/COMMIT.
Post-merge fixes. Changes to some result sets.
mysql-test/r/commit_1innodb.result:
TRUNCATE TABLE does not cause the binary log to do commits any more.
mysql-test/suite/binlog/r/binlog_row_mix_innodb_myisam.result:
TRUNCATE TABLE is not transactional, hence does not have BEGIN/COMMIT around itself.
mysql-test/suite/binlog/r/binlog_stm_mix_innodb_myisam.result:
TRUNCATE TABLE is not transactional, hence does not have BEGIN/COMMIT around itself.
mysql-test/suite/rpl/r/rpl_innodb_mixed_dml.result:
TRUNCATE TABLE is not transactional, hence does not have BEGIN/COMMIT around itself.
mysql-test/suite/rpl/r/rpl_truncate_2myisam.result:
STOP SLAVE was replaced with include file.
mysql-test/suite/rpl/r/rpl_truncate_3innodb.result:
STOP SLAVE was replaced with include file.
conflicts:
Text conflict in client/mysqltest.cc
Text conflict in mysql-test/include/wait_until_connected_again.inc
Text conflict in mysql-test/lib/mtr_report.pm
Text conflict in mysql-test/mysql-test-run.pl
Text conflict in mysql-test/r/events_bugs.result
Text conflict in mysql-test/r/log_state.result
Text conflict in mysql-test/r/myisam_data_pointer_size_func.result
Text conflict in mysql-test/r/mysqlcheck.result
Text conflict in mysql-test/r/query_cache.result
Text conflict in mysql-test/r/status.result
Text conflict in mysql-test/suite/binlog/r/binlog_index.result
Text conflict in mysql-test/suite/binlog/r/binlog_innodb.result
Text conflict in mysql-test/suite/rpl/r/rpl_packet.result
Text conflict in mysql-test/suite/rpl/t/rpl_packet.test
Text conflict in mysql-test/t/disabled.def
Text conflict in mysql-test/t/events_bugs.test
Text conflict in mysql-test/t/log_state.test
Text conflict in mysql-test/t/myisam_data_pointer_size_func.test
Text conflict in mysql-test/t/mysqlcheck.test
Text conflict in mysql-test/t/query_cache.test
Text conflict in mysql-test/t/rpl_init_slave_func.test
Text conflict in mysql-test/t/status.test
binlog_row_mix_innodb_myisam resutls are corrected.
The last operations prior the dup error is
TRUNCATE table t2;
Therefore after
--error ER_DUP_ENTRY
INSERT INTO t2 select * from t1;
table t2 must be empty, and that is what the updated results confirm.
mysql-test/suite/binlog/r/binlog_row_mix_innodb_myisam.result:
results changed due to Bug #33420 fixing.
after rollback on master
When starting a transaction with a statement containing changes
to both transactional tables and non-transactional tables, the
statement is considered as non-transactional and is therefore
written directly to the binary log. This behaviour was present
in 5.0, and has propagated to 5.1.
If a trigger containing a change of a non-transactional table is
added to a transactional table, any changes to the transactional
table is "tainted" as non-transactional.
This patch solves the problem by removing the existing "hack" that
allows non-transactional statements appearing first in a transaction
to be written directly to the binary log. Instead, anything inside
a transaction is treaded as part of the transaction and not written
to the binary log until the transaction is committed.
mysql-test/suite/rpl/t/rpl_row_create_table.test:
Removing positions from SHOW BINLOG EVENTS and using
reset_master_and_slave to start on a fresh binary log each time.
mysql-test/suite/rpl/t/rpl_slave_skip.test:
Adding explicit commit in AUTOCOMMIT=0 to make test work correctly.
mysql-test/suite/rpl/t/rpl_trigger.test:
Adding test case for BUG#40116.
sql/log.cc:
Changing commit logic in binlog_commit() to only commit when
committing a real transaction or committing a punch transaction.
