In RBR, DDL statement will change binlog format to non row-based
format before it is binlogged, but the binlog format was not be
restored, and then manipulating a temporary table can not reset binlog
format to row-based format rightly. So that the manipulated statement
is binlogged with statement-based format.
To fix the problem, restore the state of binlog format after the DDL
statement is binlogged.
mysql-test/extra/rpl_tests/rpl_tmp_table_and_DDL.test:
Added the test file to verify if executing DDL statement before
trying to manipulate a temporary table causes row-based replication
to break with error 'table does not exist'.
mysql-test/suite/binlog/r/binlog_row_mix_innodb_myisam.result:
Correct the test result, all the above binlog event
should be row-based after the bug49132 is fixed IN RBR.
mysql-test/suite/ndb/r/ndb_tmp_table_and_DDL.result:
Test result for bug#49132 base on ndb engine.
mysql-test/suite/ndb/t/ndb_tmp_table_and_DDL.test:
Added the test file to verify if executing DDL statement before
trying to manipulate a temporary table causes row-based replication
to break with error 'table does not exist' base on ndb engine.
mysql-test/suite/rpl/r/rpl_tmp_table_and_DDL.result:
Test result for bug#49132 base on myisam engine.
mysql-test/suite/rpl/t/rpl_tmp_table_and_DDL.test:
Added the test file to verify if executing DDL statement before
trying to manipulate a temporary table causes row-based replication
to break with error 'table does not exist' base on myisam engine.
sql/event_db_repository.cc:
Added code to restore the state of binlog format after the DDL
statement is binlogged.
sql/events.cc:
Added code to restore the state of binlog format after the DDL
statement is binlogged.
sql/sp.cc:
Added code to restore the state of binlog format after the DDL
statement is binlogged.
sql/sql_acl.cc:
Added code to restore the state of binlog format after the DDL
statement is binlogged.
sql/sql_udf.cc:
Added code to restore the state of binlog format after the DDL
statement is binlogged.
It is well-known that due to concurrency issues, a slave can become
inconsistent when a transaction contains updates to both transaction and
non-transactional tables.
In a nutshell, the current code-base tries to preserve causality among the
statements by writing non-transactional statements to the txn-cache which
is flushed upon commit. However, modifications done to non-transactional
tables on behalf of a transaction become immediately visible to other
connections but may not immediately get into the binary log and therefore
consistency may be broken.
In general, it is impossible to automatically detect causality/dependency
among statements by just analyzing the statements sent to the server. This
happen because dependency may be hidden in the application code and it is
necessary to know a priori all the statements processed in the context of
a transaction such as in a procedure. Moreover, even for the few cases that
we could automatically address in the server, the computation effort
required could make the approach infeasible.
So, in this patch we introduce the option
- "--binlog-direct-non-transactional-updates" that can be used to bypass
the current behavior in order to write directly to binary log statements
that change non-transactional tables.
Besides, it is used to enable the WL#2687 which is disabled by default.
mysql-test/extra/rpl_tests/rpl_binlog_max_cache_size.test:
Changes the result set as the STMT mode behaves as both the ROW and MIXED modes and as such uses the non-trx-cache and takes longer to fill the trx-cache up and trigger an error.
mysql-test/extra/rpl_tests/rpl_implicit_commit_binlog.test:
Changes the result set as the STMT mode behaves as both the ROW and MIXED modes and as such uses the non-trx-cache. It also fixes comments.
mysql-test/extra/rpl_tests/rpl_mixing_engines.test:
The STMT mode is unsafe when mixed-statements are executed thus making slaves to go out of sync. For that reason, it checks consistency if not in STMT mode.
mysql-test/include/default_mysqld.cnf:
Makes binlog-direct-non-transactional-updates "TRUE" by default in the test
cases.
mysql-test/r/mysqld--help-notwin.result:
Updates the result file with the new option.
mysql-test/r/mysqld--help-win.result:
Updates the result file with the new option.
mysql-test/suite/binlog/r/binlog_multi_engine.result:
Updates the result file because non-trx-changes are written ahead of the
transaction.
mysql-test/suite/binlog/r/binlog_switch_inside_trans.result:
Verifies if the user cannot change the opion binlog_direct_non_transactional_updates
within a transaction or a procedure/function/trigger.
mysql-test/suite/binlog/t/binlog_stm_mix_innodb_myisam-master.opt:
Sets binlog_direct_non_transactional_updates to FALSE in order to avoid changing
the test case and its result file.
mysql-test/suite/binlog/t/binlog_switch_inside_trans.test:
Verifies if the user cannot change the opion binlog_direct_non_transactional_updates
within a transaction or a procedure/function/trigger.
mysql-test/suite/ndb/r/ndb_binlog_format.result:
Updates the result file because non-trx-changes are written ahead of the
transaction.
mysql-test/suite/rpl/r/rpl_begin_commit_rollback.result:
Sets binlog_direct_non_transactional_updates to FALSE in order to avoid changing the test case and its result file.
mysql-test/suite/rpl/r/rpl_concurrency_error.result:
Updates the result file because non-trx-changes are written ahead of the
transaction
mysql-test/suite/rpl/r/rpl_mixed_implicit_commit_binlog.result:
Updates the result file because non-trx-changes are written ahead of the
transaction.
mysql-test/suite/rpl/r/rpl_mixed_mixing_engines.result:
Updates the result file because non-trx-changes are written ahead of the
transaction.
mysql-test/suite/rpl/r/rpl_stm_binlog_max_cache_size.result:
Changes the result set as the STMT mode behaves as both the ROW and MIXED modes and as such uses the non-trx-cache and takes longer to fill the trx-cache up and trigger an error.
mysql-test/suite/rpl/r/rpl_stm_implicit_commit_binlog.result:
Updates the result file because non-trx-changes are written ahead of the
transaction.
mysql-test/suite/rpl/r/rpl_stm_mixing_engines.result:
Updates the result file because non-trx-changes are written ahead of the
transaction.
mysql-test/suite/rpl/r/rpl_stm_start_stop_slave.result:
Sets binlog_direct_non_transactional_updates to FALSE in order to avoid changing the test case and its result file.
mysql-test/suite/rpl/r/rpl_stm_stop_middle_group.result:
Sets binlog_direct_non_transactional_updates to FALSE in order to avoid changing the test case and its result file.
mysql-test/suite/rpl/t/rpl_begin_commit_rollback.test:
Sets binlog_direct_non_transactional_updates to FALSE in order to avoid changing the test case and its result file.
mysql-test/suite/rpl/t/rpl_stm_start_stop_slave.test:
Sets binlog_direct_non_transactional_updates to FALSE in order to avoid changing the test case and its result file.
mysql-test/suite/rpl/t/rpl_stm_stop_middle_group.test:
Sets binlog_direct_non_transactional_updates to FALSE in order to avoid changing the test case and its result file.
sql/log.cc:
Verifies if changes should be written to either the trx-cache or non-trx-cache through the use of the function use_trans_cache(). It also organizes the code.
sql/log.h:
Changes the signature of some functions by adding the modifier "const" to the thd parameter. Specifically, the following functions are changed:
bool trans_has_updated_trans_table(const THD* thd);
bool stmt_has_updated_trans_table(const THD *thd);
bool use_trans_cache(const THD*, bool is_transactional);
sql/share/errmsg-utf8.txt:
Creates error messages to report when an user tries to change the new option
binlog_direct_non_transactional_updates within a transaction or a procedure/
function/trigger.
sql/share/errmsg.txt:
Creates error messages to report when an user tries to change the new option
binlog_direct_non_transactional_updates within a transaction or a procedure/
function/trigger.
sql/sql_class.h:
Adds the new option binlog_direct_non_transactional_updates.
sql/sys_vars.cc:
Adds the new option binlog_direct_non_transactional_updates.
support-files/my-small.cnf.sh:
Adds binlog-direct-non-transactional-updates to the example file. By default
the option is disabled.
On Unix systems TZ can be set to change the time zone for specific processes only.
But on Windows this does not fully work. It changes some aspects of time zones
in the system but not others (notably localtime() vs. file system time stamps).
This causes test failures in Windows in certain time zones (but not all),
where PURGE BEFORE DATE statements get the wrong files when TZ is set to
+03:00 by default.
Fix by only setting TZ in the small number of tests that really need it, and leave
it untouched in the rest.
Thanks to Alex Budovski for helping with this.
mysql-test/include/ps_conv.inc:
Explicitly set TIME_ZONE, as mysql-test-run.pl now does not set it.
mysql-test/mysql-test-run.pl:
Do not set TZ by default, as it causes problems on Windows.
mysql-test/suite/binlog/r/binlog_unsafe.result:
Explicitly set TIME_ZONE, as mysql-test-run.pl now does not set it.
mysql-test/suite/binlog/t/binlog_unsafe.test:
Explicitly set TIME_ZONE, as mysql-test-run.pl now does not set it.
mysql-test/suite/rpl/r/rpl_mixed_ddl_dml.result:
Explicitly set TIME_ZONE, as mysql-test-run.pl now does not set it.
mysql-test/suite/rpl/t/rpl_mixed_ddl_dml.test:
Explicitly set TIME_ZONE, as mysql-test-run.pl now does not set it.
mysql-test/t/mysqlbinlog2-master.opt:
mysql-test-run.pl no longer sets TZ by default, so set it explicitly for this particular test.
Conflicts:
Text conflict in .bzr-mysql/default.conf
Text conflict in mysql-test/suite/rpl/r/rpl_loaddata_fatal.result
Text conflict in mysql-test/suite/rpl/r/rpl_stm_log.result
Text conflict in mysql-test/t/mysqlbinlog.test
Text conflict in sql/sql_acl.cc
Text conflict in sql/sql_servers.cc
Text conflict in sql/sql_update.cc
Text conflict in support-files/mysql.spec.sh
Problem: The test case failed because: (i) warning text in
result file differed from the warning output by the
server, and (ii) binlog contents in result file did
not show the statements logged wrapped in BEGIN/COMMIT
as it is the case after WL 2687.
Solution: We update the result file, but first we change the
unsafe warning text to also refer to performance_schema
table(s). This required changing the result files for
existing test cases that provide output for warnings
related to ER_BINLOG_UNSAFE_SYSTEM_TABLE. "Grepping" in
result files, shows that only binlog_unsafe contained
reference to such a warning.
We also update the result file with the missing
BEGIN/COMMIT statements.
Problem: When RAND() is binlogged in statement mode, the seed is
binlogged too, so the replication slave generates the same
sequence of random numbers. This makes replication work in many
cases, but not in all cases: the order of rows is not guaranteed
for, e.g., UPDATE or INSERT...SELECT statements, so the row data
will be different if master and slave retrieve the rows in
different orders.
Fix: Mark RAND() as unsafe. It will generate a warning if
binlog_format=STATEMENT and switch to row-logging if
binlog_format=ROW.
mysql-test/extra/rpl_tests/rpl_row_func003.test:
updated test case to ignore new warnings
mysql-test/suite/binlog/r/binlog_unsafe.result:
updated result file
mysql-test/suite/binlog/t/binlog_unsafe.test:
Added test for RAND().
Also clarified some old comments.
mysql-test/suite/rpl/r/rpl_misc_functions.result:
updated result file
mysql-test/suite/rpl/r/rpl_nondeterministic_functions.result:
updated test case to ignore new warnings
mysql-test/suite/rpl/r/rpl_optimize.result:
updated result file
mysql-test/suite/rpl/r/rpl_row_func003.result:
updated result file
mysql-test/suite/rpl/t/rpl_misc_functions.test:
updated test case to ignore new warnings
mysql-test/suite/rpl/t/rpl_nondeterministic_functions.test:
updated test case to ignore new warnings
mysql-test/suite/rpl/t/rpl_optimize.test:
updated test case to ignore new warnings
mysql-test/suite/rpl/t/rpl_trigger.test:
updated test case to ignore new warnings
mysql-test/suite/rpl_ndb/r/rpl_ndb_func003.result:
updated result file
sql/item_create.cc:
Mark RAND() unsafe.
Conflicts:
Text conflict in .bzr-mysql/default.conf
Text conflict in mysql-test/extra/rpl_tests/rpl_loaddata.test
Text conflict in mysql-test/r/mysqlbinlog2.result
Text conflict in mysql-test/suite/binlog/r/binlog_stm_mix_innodb_myisam.result
Text conflict in mysql-test/suite/binlog/r/binlog_unsafe.result
Text conflict in mysql-test/suite/rpl/r/rpl_insert_id.result
Text conflict in mysql-test/suite/rpl/r/rpl_loaddata.result
Text conflict in mysql-test/suite/rpl/r/rpl_stm_auto_increment_bug33029.result
Text conflict in mysql-test/suite/rpl/r/rpl_udf.result
Text conflict in mysql-test/suite/rpl/t/rpl_slow_query_log.test
Text conflict in sql/field.h
Text conflict in sql/log.cc
Text conflict in sql/log_event.cc
Text conflict in sql/log_event_old.cc
Text conflict in sql/mysql_priv.h
Text conflict in sql/share/errmsg.txt
Text conflict in sql/sp.cc
Text conflict in sql/sql_acl.cc
Text conflict in sql/sql_base.cc
Text conflict in sql/sql_class.h
Text conflict in sql/sql_db.cc
Text conflict in sql/sql_delete.cc
Text conflict in sql/sql_insert.cc
Text conflict in sql/sql_lex.cc
Text conflict in sql/sql_lex.h
Text conflict in sql/sql_load.cc
Text conflict in sql/sql_table.cc
Text conflict in sql/sql_update.cc
Text conflict in sql/sql_view.cc
Conflict adding files to storage/innobase. Created directory.
Conflict because storage/innobase is not versioned, but has versioned children. Versioned directory.
Conflict adding file storage/innobase. Moved existing file to storage/innobase.moved.
Conflict adding files to storage/innobase/handler. Created directory.
Conflict because storage/innobase/handler is not versioned, but has versioned children. Versioned directory.
Contents conflict in storage/innobase/handler/ha_innodb.cc
For tables with metadata sizes ranging from 251 to 255 the size
of the event data (m_data_size) was being improperly calculated
in the Table_map_log_event constructor. This was due to the fact
that when writing the Table_map_log_event body (in
Table_map_log_event::write_data_body) a call to net_store_length
is made for packing the m_field_metadata_size. It happens that
net_store_length uses *one* byte for storing
m_field_metadata_size when it is smaller than 251 but *three*
bytes when it exceeds that value. BUG 42749 had already
pinpointed and fix this fact, but the fix was incomplete, as the
calculation in the Table_map_log_event constructor considers 255
instead of 251 as the threshold to increment m_data_size by
three. Thence, the window for having a mismatch between the
number of bytes written and the number of bytes accounted in the
event length (m_data_size) was left open for
m_field_metadata_size values between 251 and 255.
We fix this by changing the condition in the Table_map_log_event
constructor to match the one in the net_store_length, ie,
increment one byte if m_field_metadata_size < 251 and three if it
exceeds this value.
mysql-test/suite/rpl/r/rpl_row_tbl_metadata.result:
Updated result file.
mysql-test/suite/rpl/t/rpl_row_tbl_metadata.test:
Changes to the original test case: added slave and moved
file into the rpl suite.
New test case: replicates two tables one with 250 and
another with 252 metadata sizes. This exercises the usage
of 1 or 3 bytes while packing the m_field_metadata_size.
sql/log_event.cc:
Made the m_data_size calculation for the table map log event
to match the number of bytes used while packing the
m_field_metadata_size value (according to net_store_length
function in pack.c).
In auto-commit mode, updating both trx and non-trx tables (i.e. issuing a mixed
statement) causes the following sequence of events:
1 - "Flush trx changes" (MYSQL_BIN_LOG::write) - T1:
1.1 - mutex_lock (&LOCK_log)
1.2 - mutex_lock (&LOCK_prep_xids)
1.3 - increase prepared_xids
1.4 - mutex_unlock (&LOCK_prep_xids)
1.5 - mutex_unlock (&LOCK_log)
2 - "Flush non-trx changes" (MYSQL_BIN_LOG::write) - T1:
2.1 - mutex_lock (&LOCK_log)
2.2 - mutex_unlock (&LOCK_log)
3. "unlog" - T1
3.1 - mutex_lock (&LOCK_prep_xids)
3.2 - decrease prepared xids
3.3 - pthread_cond_signal(&COND_prep_xids);
3.4 - mutex_unlock (&LOCK_prep_xids)
The "FLUSH logs" command produces the following sequence of events:
1 - "FLUSH logs" command (MYSQL_BIN_LOG::new_file_impl) - user thread:
1.1 - mutex_lock (&LOCK_log)
1.2 - mutex_lock (&LOCK_prep_xids)
1.3 - while (prepared_xids) pthread_cond_wait(..., &LOCK_prep_xids);
1.4 - mutex_unlock (&LOCK_prep_xids)
1.5 - mutex_unlock (&LOCK_log)
A deadlock will arise if T1 flushes the trx changes and thus increases
prepared_xids but before it is able to continue the execution and flush the
non-trx changes, an user thread calls the "FLUSH logs" command and wait that
the prepared_xids is decreased and gets to zero. However, T1 cannot proceed
with the call to "Flush non-trx changes" because it will block in the mutex
"LOCK_log" and by consequence cannot complete the execution and call the
unlog to decrease the prepared_xids.
To fix the problem, we ensure that the non-trx changes are always flushed
before the trx changes.
Note that if you call "Flush non-trx changes" and a concurrent "FLUSH logs" is
issued, the "Flush non-trx changes" may block, but a deadlock will never happen
because the prepared_xids will eventually get to zero. Bottom line, there will
not be any transaction able to increase the prepared_xids because they will
block in the mutex "LOCK_log" (MYSQL_BIN_LOG::write) and those that increased
the prepared_xids will eventually commit and decrease the prepared_xids.
When @@session.binlog_format is modified inside a transaction,
it can cause slave to go out of sync.
To fix the problem, make the session variable 'binlog_format'
read-only inside a transaction.
mysql-test/suite/binlog/r/binlog_format_switch_inside_trans.result:
Test result for bug#47863.
mysql-test/suite/binlog/t/binlog_format_switch_inside_trans.test:
Added test file to verify if the session variable 'binlog_format'
is read-only inside a transaction and in sub-statements.
sql/set_var.cc:
Added code to make the session variable 'binlog_format'
read-only inside a transaction.
Bug#16565 mysqld --help --verbose does not order variablesBug#20413 sql_slave_skip_counter is not shown in show variables
Bug#20415 Output of mysqld --help --verbose is incomplete
Bug#25430 variable not found in SELECT @@global.ft_max_word_len;
Bug#32902 plugin variables don't know their names
Bug#34599 MySQLD Option and Variable Reference need to be consistent in formatting!
Bug#34829 No default value for variable and setting default does not raise error
Bug#34834 ? Is accepted as a valid sql mode
Bug#34878 Few variables have default value according to documentation but error occurs
Bug#34883 ft_boolean_syntax cant be assigned from user variable to global var.
Bug#37187 `INFORMATION_SCHEMA`.`GLOBAL_VARIABLES`: inconsistent status
Bug#40988 log_output_basic.test succeeded though syntactically false.
Bug#41010 enum-style command-line options are not honoured (maria.maria-recover fails)
Bug#42103 Setting key_buffer_size to a negative value may lead to very large allocations
Bug#44691 Some plugins configured as MYSQL_PLUGIN_MANDATORY in can be disabled
Bug#44797 plugins w/o command-line options have no disabling option in --help
Bug#46314 string system variables don't support expressions
Bug#46470 sys_vars.max_binlog_cache_size_basic_32 is broken
Bug#46586 When using the plugin interface the type "set" for options caused a crash.
Bug#47212 Crash in DBUG_PRINT in mysqltest.cc when trying to print octal number
Bug#48758 mysqltest crashes on sys_vars.collation_server_basic in gcov builds
Bug#49417 some complaints about mysqld --help --verbose output
Bug#49540 DEFAULT value of binlog_format isn't the default value
Bug#49640 ambiguous option '--skip-skip-myisam' (double skip prefix)
Bug#49644 init_connect and \0
Bug#49645 init_slave and multi-byte characters
Bug#49646 mysql --show-warnings crashes when server dies
CMakeLists.txt:
Bug#44691 Some plugins configured as MYSQL_PLUGIN_MANDATORY in can be disabled
client/mysql.cc:
don't crash with --show-warnings when mysqld dies
config/ac-macros/plugins.m4:
Bug#44691 Some plugins configured as MYSQL_PLUGIN_MANDATORY in can be disabled
include/my_getopt.h:
comments
include/my_pthread.h:
fix double #define
mysql-test/mysql-test-run.pl:
run sys_vars suite by default
properly recognize envirinment variables (e.g. MTR_MAX_SAVE_CORE) set to 0
escape gdb command line arguments
mysql-test/suite/sys_vars/r/rpl_init_slave_func.result:
init_slave+utf8 bug
mysql-test/suite/sys_vars/t/rpl_init_slave_func.test:
init_slave+utf8 bug
mysys/my_getopt.c:
Bug#34599 MySQLD Option and Variable Reference need to be consistent in formatting!
Bug#46586 When using the plugin interface the type "set" for options caused a crash.
Bug#49640 ambiguous option '--skip-skip-myisam' (double skip prefix)
mysys/typelib.c:
support for flagset
sql/ha_ndbcluster.cc:
backport from telco tree
sql/item_func.cc:
Bug#49644 init_connect and \0
Bug#49645 init_slave and multi-byte characters
sql/sql_builtin.cc.in:
Bug#44691 Some plugins configured as MYSQL_PLUGIN_MANDATORY in can be disabled
sql/sql_plugin.cc:
Bug#44691 Some plugins configured as MYSQL_PLUGIN_MANDATORY in can be disabled
Bug#32902 plugin variables don't know their names
Bug#44797 plugins w/o command-line options have no disabling option in --help
sql/sys_vars.cc:
all server variables are defined here
storage/myisam/ft_parser.c:
remove unnecessary updates of param->quot
storage/myisam/ha_myisam.cc:
myisam_* variables belong here
strings/my_vsnprintf.c:
%o and %llx
unittest/mysys/my_vsnprintf-t.c:
%o and %llx tests
vio/viosocket.c:
bugfix: fix @@wait_timeout to work with socket timeouts (vs. alarm thread)
Problem: The test was written before BUG#45827 was fixed.
The test contained code that assumed the wrong behavior,
pre-BUG#45827. Then, the fix for BUG#45827 was merged
from 5.1-rep+2 to 5.1-rep+3. Since the test case assumed
the wrong behavior, it failed. This should have been fixed
by making the test assume the correct behavior, but was
fixed by updating the result file to assert failure.
Fix 1: fix the test to assume correct behavior
(post-BUG#45827), update result file.
Fix 2: make test fail with 'die' instead of 'exit' when
wrong behavior is detected. Thus, the test cannot be
silenced with a wrong result file in case the behavior
will change again.
mysql-test/extra/rpl_tests/create_recursive_construct.inc:
Replaced 'exit' by 'die' to avoid similar
mysql-test/suite/binlog/r/binlog_unsafe.result:
Updated result file.
mysql-test/suite/binlog/t/binlog_unsafe.test:
Since BUG#45827 is now fixed, we need to update the test case
in two places where it expects the wrong result because of
BUG#45827.
escaped field names
When in mixed or statement mode, the master logs LOAD DATA
queries by resorting to an Execute_load_query_log_event. This
event does not contain the original query, but a rewritten
version of it, which includes the table field names. However, the
rewrite does not escape the field names. If these names match a
reserved keyword, then the slave will stop with a syntax error
when executing the event.
We fix this by escaping the fields names as it happens already
for the table name.
mysql-test/extra/rpl_tests/rpl_loaddata.test:
Added test case for the reported bug.
mysql-test/r/mysqlbinlog.result:
Changed result to support escaped field name.
mysql-test/suite/binlog/r/binlog_killed_simulate.result:
Changed result to support escaped field name.
mysql-test/suite/binlog/r/binlog_row_mix_innodb_myisam.result:
Changed result to support escaped field name.
mysql-test/suite/binlog/r/binlog_stm_blackhole.result:
Changed result to support escaped field name.
mysql-test/suite/binlog/r/binlog_stm_mix_innodb_myisam.result:
Changed result to support escaped field name.
mysql-test/suite/rpl/r/rpl_innodb_mixed_dml.result:
Changed result to support escaped field name.
mysql-test/suite/rpl/r/rpl_loaddata.result:
Added result for new test.
Changed show slave status positions which are now different because of
extra escape character in field names.
mysql-test/suite/rpl/r/rpl_loaddata_fatal.result:
Changed show slave status positions which are now different because of
extra escape character.
mysql-test/suite/rpl/r/rpl_loaddata_map.result:
Changed result to support escaped field name.
mysql-test/suite/rpl/r/rpl_stm_log.result:
Changed result to support escaped field name.
mysql-test/t/mysqlbinlog.test:
Changed positions which is now different because of extra escape
character in field names.
sql/sql_load.cc:
Appended escape characters before and after field names.
2617.31.12, 2617.31.15, 2617.31.15, 2617.31.16, 2617.43.1
- initial changeset that introduced the fix for
Bug#989 and follow up fixes for all test suite failures
introduced in the initial changeset.
------------------------------------------------------------
revno: 2617.31.1
committer: Davi Arnaut <Davi.Arnaut@Sun.COM>
branch nick: 4284-6.0
timestamp: Fri 2009-03-06 19:17:00 -0300
message:
Bug#989: If DROP TABLE while there's an active transaction, wrong binlog order
WL#4284: Transactional DDL locking
Currently the MySQL server does not keep metadata locks on
schema objects for the duration of a transaction, thus failing
to guarantee the integrity of the schema objects being used
during the transaction and to protect then from concurrent
DDL operations. This also poses a problem for replication as
a DDL operation might be replicated even thought there are
active transactions using the object being modified.
The solution is to defer the release of metadata locks until
a active transaction is either committed or rolled back. This
prevents other statements from modifying the table for the
entire duration of the transaction. This provides commitment
ordering for guaranteeing serializability across multiple
transactions.
- Incompatible change:
If MySQL's metadata locking system encounters a lock conflict,
the usual schema is to use the try and back-off technique to
avoid deadlocks -- this schema consists in releasing all locks
and trying to acquire them all in one go.
But in a transactional context this algorithm can't be utilized
as its not possible to release locks acquired during the course
of the transaction without breaking the transaction commitments.
To avoid deadlocks in this case, the ER_LOCK_DEADLOCK will be
returned if a lock conflict is encountered during a transaction.
Let's consider an example:
A transaction has two statements that modify table t1, then table
t2, and then commits. The first statement of the transaction will
acquire a shared metadata lock on table t1, and it will be kept
utill COMMIT to ensure serializability.
At the moment when the second statement attempts to acquire a
shared metadata lock on t2, a concurrent ALTER or DROP statement
might have locked t2 exclusively. The prescription of the current
locking protocol is that the acquirer of the shared lock backs off
-- gives up all his current locks and retries. This implies that
the entire multi-statement transaction has to be rolled back.
- Incompatible change:
FLUSH commands such as FLUSH PRIVILEGES and FLUSH TABLES WITH READ
LOCK won't cause locked tables to be implicitly unlocked anymore.
mysql-test/extra/binlog_tests/drop_table.test:
Add test case for Bug#989.
mysql-test/extra/binlog_tests/mix_innodb_myisam_binlog.test:
Fix test case to reflect the fact that transactions now hold
metadata locks for the duration of a transaction.
mysql-test/include/mix1.inc:
Fix test case to reflect the fact that transactions now hold
metadata locks for the duration of a transaction.
mysql-test/include/mix2.inc:
Fix test case to reflect the fact that transactions now hold
metadata locks for the duration of a transaction.
mysql-test/r/flush_block_commit.result:
Update test case result (WL#4284).
mysql-test/r/flush_block_commit_notembedded.result:
Update test case result (WL#4284).
mysql-test/r/innodb.result:
Update test case result (WL#4284).
mysql-test/r/innodb_mysql.result:
Update test case result (WL#4284).
mysql-test/r/lock.result:
Add test case result for an effect of WL#4284/Bug#989
(all locks should be released when a connection terminates).
mysql-test/r/mix2_myisam.result:
Update test case result (effects of WL#4284/Bug#989).
mysql-test/r/not_embedded_server.result:
Update test case result (effects of WL#4284/Bug#989).
Add a test case for interaction of WL#4284 and FLUSH PRIVILEGES.
mysql-test/r/partition_innodb_semi_consistent.result:
Update test case result (effects of WL#4284/Bug#989).
mysql-test/r/partition_sync.result:
Temporarily disable the test case for Bug#43867,
which will be fixed by a subsequent backport.
mysql-test/r/ps.result:
Add a test case for effect of PREPARE on transactional
locks: we take a savepoint at beginning of PREAPRE
and release it at the end. Thus PREPARE does not
accumulate metadata locks (Bug#989/WL#4284).
mysql-test/r/read_only_innodb.result:
Update test case result (effects of WL#4284/Bug#989).
mysql-test/suite/binlog/r/binlog_row_drop_tbl.result:
Add a test case result (WL#4284/Bug#989).
mysql-test/suite/binlog/r/binlog_row_mix_innodb_myisam.result:
Update test case result (effects of WL#4284/Bug#989).
mysql-test/suite/binlog/r/binlog_stm_drop_tbl.result:
Add a test case result (WL#4284/Bug#989).
mysql-test/suite/binlog/r/binlog_stm_mix_innodb_myisam.result:
Update test case result (effects of WL#4284/Bug#989).
mysql-test/suite/binlog/r/binlog_unsafe.result:
A side effect of Bug#989 -- slightly different table map ids.
mysql-test/suite/binlog/t/binlog_row_drop_tbl.test:
Add a test case for WL#4284/Bug#989.
mysql-test/suite/binlog/t/binlog_stm_drop_tbl.test:
Add a test case for WL#4284/Bug#989.
mysql-test/suite/binlog/t/binlog_stm_row.test:
Update to the new state name. This
is actually a follow up to another patch for WL#4284,
that changes Locked thread state to Table lock.
mysql-test/suite/ndb/r/ndb_index_ordered.result:
Remove result for disabled part of the test case.
mysql-test/suite/ndb/t/disabled.def:
Temporarily disable a test case (Bug#45621).
mysql-test/suite/ndb/t/ndb_index_ordered.test:
Disable a part of a test case (needs update to
reflect semantics of Bug#989).
mysql-test/suite/rpl/t/disabled.def:
Disable tests made meaningless by transactional metadata
locking.
mysql-test/suite/sys_vars/r/autocommit_func.result:
Add a commit (Bug#989).
mysql-test/suite/sys_vars/t/autocommit_func.test:
Add a commit (Bug#989).
mysql-test/t/flush_block_commit.test:
Fix test case to reflect the fact that transactions now hold
metadata locks for the duration of a transaction.
mysql-test/t/flush_block_commit_notembedded.test:
Fix test case to reflect the fact that transactions now hold
metadata locks for the duration of a transaction.
Add a test case for transaction-scope locks and the global
read lock (Bug#989/WL#4284).
mysql-test/t/innodb.test:
Fix test case to reflect the fact that transactions now hold
metadata locks for the duration of a transaction
(effects of Bug#989/WL#4284).
mysql-test/t/lock.test:
Add a test case for Bug#989/WL#4284.
mysql-test/t/not_embedded_server.test:
Add a test case for Bug#989/WL#4284.
mysql-test/t/partition_innodb_semi_consistent.test:
Replace TRUNCATE with DELETE, to not issue
an implicit commit of a transaction, and not depend
on metadata locks.
mysql-test/t/partition_sync.test:
Temporarily disable the test case for Bug#43867,
which needs a fix to be backported from 6.0.
mysql-test/t/ps.test:
Add a test case for semantics of PREPARE and transaction-scope
locks: metadata locks on tables used in PREPARE are enclosed into a
temporary savepoint, taken at the beginning of PREPARE,
and released at the end. Thus PREPARE does not effect
what locks a transaction owns.
mysql-test/t/read_only_innodb.test:
Fix test case to reflect the fact that transactions now hold
metadata locks for the duration of a transaction
(Bug#989/WL#4284).
Wait for the read_only statement to actually flush tables before
sending other concurrent statements that depend on its state.
mysql-test/t/xa.test:
Fix test case to reflect the fact that transactions now hold
metadata locks for the duration of a transaction
(Bug#989/WL#4284).
sql/ha_ndbcluster_binlog.cc:
Backport bits of changes of ha_ndbcluster_binlog.cc
from 6.0, to fix the failing binlog test suite with
WL#4284. WL#4284 implementation does not work
with 5.1 implementation of ndbcluster binlog index.
sql/log_event.cc:
Release metadata locks after issuing a commit.
sql/mdl.cc:
Style changes (WL#4284).
sql/mysql_priv.h:
Rename parameter to match the name used in the definition (WL#4284).
sql/rpl_injector.cc:
Release metadata locks on commit (WL#4284).
sql/rpl_rli.cc:
Remove assert made meaningless, metadata locks are released
at the end of the transaction.
sql/set_var.cc:
Close tables and release locks if autocommit mode is set.
sql/slave.cc:
Release metadata locks after a rollback.
sql/sql_acl.cc:
Don't implicitly unlock locked tables. Issue a implicit commit
at the end and unlock tables.
sql/sql_base.cc:
Defer the release of metadata locks when closing tables
if not required to.
Issue a deadlock error if the locking protocol requires
that a transaction re-acquire its locks.
Release metadata locks when closing tables for reopen.
sql/sql_class.cc:
Release metadata locks if the thread is killed.
sql/sql_parse.cc:
Release metadata locks after implicitly committing a active
transaction, or after explicit commits or rollbacks.
sql/sql_plugin.cc:
Allocate MDL request on the stack as the use of the table
is contained within the function. It will be removed from
the context once close_thread_tables is called at the end
of the function.
sql/sql_prepare.cc:
The problem is that the prepare phase of the CREATE TABLE
statement takes a exclusive metadata lock lock and this can
cause a self-deadlock the thread already holds a shared lock
on the table being that should be created.
The solution is to make the prepare phase take a shared
metadata lock when preparing a CREATE TABLE statement. The
execution of the statement will still acquire a exclusive
lock, but won't cause any problem as it issues a implicit
commit.
After some discussions with stakeholders it has been decided that
metadata locks acquired during a PREPARE statement must be released
once the statement is prepared even if it is prepared within a multi
statement transaction.
sql/sql_servers.cc:
Don't implicitly unlock locked tables. Issue a implicit commit
at the end and unlock tables.
sql/sql_table.cc:
Close table and release metadata locks after a admin operation.
sql/table.h:
The problem is that the prepare phase of the CREATE TABLE
statement takes a exclusive metadata lock lock and this can
cause a self-deadlock the thread already holds a shared lock
on the table being that should be created.
The solution is to make the prepare phase take a shared
metadata lock when preparing a CREATE TABLE statement. The
execution of the statement will still acquire a exclusive
lock, but won't cause any problem as it issues a implicit
commit.
sql/transaction.cc:
Release metadata locks after the implicitly committed due
to a new transaction being started. Also, release metadata
locks acquired after a savepoint if the transaction is rolled
back to the save point.
The problem is that in some cases transaction-long metadata locks
could be released before the transaction was committed. This could
happen when a active transaction was ended by a "START TRANSACTION"
or "BEGIN" statement, in which case the metadata locks would be
released before the actual commit of the active transaction.
The solution is to defer the release of metadata locks to after the
transaction has been implicitly committed. No test case is provided
as the effort to provide one is too disproportional to the size of
the fix.
This patch fixes three bugs as follows. First, aborting the server while purging
binary logs might generate orphan files due to how the purge operation was
implemented:
(purge routine - sql/log.cc - MYSQL_BIN_LOG::purge_logs)
1 - register the files to be removed in a temporary buffer.
2 - update the log-bin.index.
3 - flush the log-bin.index.
4 - erase the files whose names where register in the temporary buffer
in step 1.
Thus a failure while executing step 4 would generate an orphan file. Second,
a similar issue might happen while creating a new binary as follows:
(create routine - sql/log.cc - MYSQL_BIN_LOG::open)
1 - open the new log-bin.
2 - update the log-bin.index.
Thus a failure while executing step 1 would generate an orphan file.
To fix these issues, we record the files to be purged or created before really
removing or adding them. So if a failure happens such records can be used to
automatically remove dangling files. The new steps might be outlined as follows:
(purge routine - sql/log.cc - MYSQL_BIN_LOG::purge_logs)
1 - register the files to be removed in the log-bin.~rec~ placed in
the data directory.
2 - update the log-bin.index.
3 - flush the log-bin.index.
4 - delete the log-bin.~rec~.
(create routine - sql/log.cc - MYSQL_BIN_LOG::open)
1 - register the file to be created in the log-bin.~rec~
placed in the data directory.
2 - open the new log-bin.
3 - update the log-bin.index.
4 - delete the log-bin.~rec~.
(recovery routine - sql/log.cc - MYSQL_BIN_LOG::open_index_file)
1 - open the log-bin.index.
2 - open the log-bin.~rec~.
3 - for each file in log-bin.~rec~.
3.1 Check if the file is in the log-bin.index and if so ignore it.
3.2 Otherwise, delete it.
The third issue can be described as follows. The purge operation was allowing
to remove a file in use thus leading to the loss of data and possible
inconsistencies between the master and slave. Roughly, the routine was only
taking into account the dump threads and so if a slave was not connect the
file might be delete even though it was in use.
Text conflict in mysql-test/collections/default.experimental
Text conflict in mysql-test/r/show_check.result
Text conflict in mysql-test/r/sp-code.result
Text conflict in mysql-test/suite/binlog/r/binlog_tmp_table.result
Text conflict in mysql-test/suite/rpl/t/disabled.def
Text conflict in mysql-test/t/show_check.test
Text conflict in mysys/my_delete.c
Text conflict in sql/item.h
Text conflict in sql/item_cmpfunc.h
Text conflict in sql/log.cc
Text conflict in sql/mysqld.cc
Text conflict in sql/repl_failsafe.cc
Text conflict in sql/slave.cc
Text conflict in sql/sql_parse.cc
Text conflict in sql/sql_table.cc
Text conflict in sql/sql_yacc.yy
Text conflict in storage/myisam/ha_myisam.cc
Corrected results for
stm_auto_increment_bug33029.reject 2009-12-01
20:01:49.000000000 +0300
<andrei> @@ -42,9 +42,6 @@
<andrei> RETURN i;
<andrei> END//
<andrei> CALL p1();
<andrei> -Warnings:
<andrei> -Note 1592 Statement may not be safe to log in statement
format.
<andrei> -Note 1592 Statement may not be safe to log in statement
format.
There should be indeed no Note present because there is in fact autoincrement
top-level query in sp() that triggers inserting in yet another auto-inc table.
(todo: alert DaoGang to improve the test).
beyond unsigned long.
BUG#44779: binlog.binlog_max_extension may be causing failure on
next test in PB
NOTE1: this is the backport to next-mr.
NOTE2: already includes patch for BUG#44779.
Binlog file extensions would turn into negative numbers once the
variable used to hold the value reached maximum for signed
long. Consequently, incrementing value to the next (negative) number
would lead to .000000 extension, causing the server to fail.
This patch addresses this issue by not allowing negative extensions
and by returning an error on find_uniq_filename, when the limit is
reached. Additionally, warnings are printed to the error log when the
limit is approaching. FLUSH LOGS will also report warnings to the
user, if the extension number has reached the limit. The limit has been
set to 0x7FFFFFFF as the maximum.
mysql-test/suite/binlog/t/binlog_max_extension.test:
Test case added that checks the maximum available number for
binlog extensions.
sql/log.cc:
Changes to find_uniq_filename and test_if_number.
sql/log.h:
Added macros with values for MAX_LOG_UNIQUE_FN_EXT and
LOG_WARN_UNIQUE_FN_EXT_LEFT, as suggested in review.
