General overview:
The logic for switching to row format when binlog_format=MIXED had
numerous flaws. The underlying problem was the lack of a consistent
architecture.
General purpose of this changeset:
This changeset introduces an architecture for switching to row format
when binlog_format=MIXED. It enforces the architecture where it has
to. It leaves some bugs to be fixed later. It adds extensive tests to
verify that unsafe statements work as expected and that appropriate
errors are produced by problems with the selection of binlog format.
It was not practical to split this into smaller pieces of work.
Problem 1:
To determine the logging mode, the code has to take several parameters
into account (namely: (1) the value of binlog_format; (2) the
capabilities of the engines; (3) the type of the current statement:
normal, unsafe, or row injection). These parameters may conflict in
several ways, namely:
- binlog_format=STATEMENT for a row injection
- binlog_format=STATEMENT for an unsafe statement
- binlog_format=STATEMENT for an engine only supporting row logging
- binlog_format=ROW for an engine only supporting statement logging
- statement is unsafe and engine does not support row logging
- row injection in a table that does not support statement logging
- statement modifies one table that does not support row logging and
one that does not support statement logging
Several of these conflicts were not detected, or were detected with
an inappropriate error message. The problem of BUG#39934 was that no
appropriate error message was written for the case when an engine
only supporting row logging executed a row injection with
binlog_format=ROW. However, all above cases must be handled.
Fix 1:
Introduce new error codes (sql/share/errmsg.txt). Ensure that all
conditions are detected and handled in decide_logging_format()
Problem 2:
The binlog format shall be determined once per statement, in
decide_logging_format(). It shall not be changed before or after that.
Before decide_logging_format() is called, all information necessary to
determine the logging format must be available. This principle ensures
that all unsafe statements are handled in a consistent way.
However, this principle is not followed:
thd->set_current_stmt_binlog_row_based_if_mixed() is called in several
places, including from code executing UPDATE..LIMIT,
INSERT..SELECT..LIMIT, DELETE..LIMIT, INSERT DELAYED, and
SET @@binlog_format. After Problem 1 was fixed, that caused
inconsistencies where these unsafe statements would not print the
appropriate warnings or errors for some of the conflicts.
Fix 2:
Remove calls to THD::set_current_stmt_binlog_row_based_if_mixed() from
code executed after decide_logging_format(). Compensate by calling the
set_current_stmt_unsafe() at parse time. This way, all unsafe statements
are detected by decide_logging_format().
Problem 3:
INSERT DELAYED is not unsafe: it is logged in statement format even if
binlog_format=MIXED, and no warning is printed even if
binlog_format=STATEMENT. This is BUG#45825.
Fix 3:
Made INSERT DELAYED set itself to unsafe at parse time. This allows
decide_logging_format() to detect that a warning should be printed or
the binlog_format changed.
Problem 4:
LIMIT clause were not marked as unsafe when executed inside stored
functions/triggers/views/prepared statements. This is
BUG#45785.
Fix 4:
Make statements containing the LIMIT clause marked as unsafe at
parse time, instead of at execution time. This allows propagating
unsafe-ness to the view.
mysql-test/extra/rpl_tests/create_recursive_construct.inc:
Added auxiliary file used by binlog_unsafe.test to create and
execute recursive constructs
(functions/procedures/triggers/views/prepared statements).
mysql-test/extra/rpl_tests/rpl_foreign_key.test:
removed unnecessary set @@session.binlog_format
mysql-test/extra/rpl_tests/rpl_insert_delayed.test:
Filter out table id from table map events in binlog listing.
Got rid of $binlog_format_statement.
mysql-test/extra/rpl_tests/rpl_ndb_apply_status.test:
disable warnings around call to unsafe procedure
mysql-test/include/rpl_udf.inc:
Disabled warnings for code that generates warnings
for some binlog formats. That would otherwise cause
inconsistencies in the result file.
mysql-test/r/mysqldump.result:
Views are now unsafe if they contain a LIMIT clause.
That fixed BUG#45831. Due to BUG#45832, a warning is
printed for the CREATE VIEW statement.
mysql-test/r/sp_trans.result:
Unsafe statements in stored procedures did not give a warning if
binlog_format=statement. This is BUG#45824. Now they do, so this
result file gets a new warning.
mysql-test/suite/binlog/r/binlog_multi_engine.result:
Error message changed.
mysql-test/suite/binlog/r/binlog_statement_insert_delayed.result:
INSERT DELAYED didn't generate a warning when binlog_format=STATEMENT.
That was BUG#45825. Now there is a warning, so result file needs to be
updated.
mysql-test/suite/binlog/r/binlog_stm_ps.result:
Changed error message.
mysql-test/suite/binlog/r/binlog_unsafe.result:
updated result file:
- error message changed
- added test for most combinations of unsafe constructs invoked
from recursive constructs
- INSERT DELAYED now gives a warning (because BUG#45826 is fixed)
- INSERT..SELECT..LIMIT now gives a warning from inside recursive
constructs (because BUG#45785 was fixed)
- When a recursive construct (e.g., stored proc or function)
contains more than one statement, at least one of which is
unsafe, then all statements in the recursive construct give
warnings. This is a new bug introduced by this changeset.
It will be addressed in a post-push fix.
mysql-test/suite/binlog/t/binlog_innodb.test:
Changed error code for innodb updates with READ COMMITTED or
READ UNCOMMITTED transaction isolation level and
binlog_format=statement.
mysql-test/suite/binlog/t/binlog_multi_engine.test:
The error code has changed for statements where more than one
engine is involved and one of them is self-logging.
mysql-test/suite/binlog/t/binlog_unsafe-master.opt:
Since binlog_unsafe now tests unsafe-ness of UDF's, we need an extra
flag in the .opt file.
mysql-test/suite/binlog/t/binlog_unsafe.test:
- Clarified comment.
- Rewrote first part of test. Now it tests not only unsafe variables
and functions, but also unsafe-ness due to INSERT..SELECT..LIMIT,
INSERT DELAYED, insert into two autoinc columns, use of UDF's, and
access to log tables in the mysql database.
Also, in addition to functions, procedures, triggers, and prepared
statements, it now also tests views; and it constructs recursive
calls in two levels by combining these recursive constructs.
Part of the logic is in extra/rpl_tests/create_recursive_construct.inc.
- added tests for all special system variables that should not be unsafe.
- added specific tests for BUG#45785 and BUG#45825
mysql-test/suite/rpl/r/rpl_events.result:
updated result file
mysql-test/suite/rpl/r/rpl_extraColmaster_innodb.result:
updated result file
mysql-test/suite/rpl/r/rpl_extraColmaster_myisam.result:
updated result file
mysql-test/suite/rpl/r/rpl_foreign_key_innodb.result:
updated result file
mysql-test/suite/rpl/r/rpl_idempotency.result:
updated result file
mysql-test/suite/rpl/r/rpl_mix_found_rows.result:
Split rpl_found_rows.test into rpl_mix_found_rows.test (a new file) and
rpl_stm_found_rows.test (renamed rpl_found_rows.test). This file equals
the second half of the old rpl_found_rows.result, with the following
modifications:
- minor formatting changes
- additional initialization
mysql-test/suite/rpl/r/rpl_mix_insert_delayed.result:
Moved out code operating in mixed mode from rpl_stm_insert_delayed
(into rpl_mix_insert_delayed) and got rid of explicit setting of
binlog format.
mysql-test/suite/rpl/r/rpl_rbr_to_sbr.result:
updated result file
mysql-test/suite/rpl/r/rpl_row_idempotency.result:
Moved the second half of rpl_idempotency.test, which only
executed in row mode, to rpl_row_idempotency.test. This is
the new result file.
mysql-test/suite/rpl/r/rpl_row_insert_delayed.result:
Got rid of unnecessary explicit setting of binlog format.
mysql-test/suite/rpl/r/rpl_stm_found_rows.result:
Split rpl_found_rows.test into rpl_mix_found_rows.test (a new file) and
rpl_stm_found_rows.test (renamed rpl_found_rows.test). Changes in
this file:
- minor formatting changes
- warning is now issued for unsafe statements inside procedures
(since BUG#45824 is fixed)
- second half of file is moved to rpl_mix_found_rows.result
mysql-test/suite/rpl/r/rpl_stm_insert_delayed.result:
Moved out code operating in mixed mode from rpl_stm_insert_delayed
(into rpl_mix_insert_delayed) and got rid of explicit setting of
binlog format.
mysql-test/suite/rpl/r/rpl_stm_loadfile.result:
error message changed
mysql-test/suite/rpl/r/rpl_temporary_errors.result:
updated result file
mysql-test/suite/rpl/r/rpl_udf.result:
Remove explicit set of binlog format (and triplicate test execution)
and rely on test system executing the test in all binlog formats.
mysql-test/suite/rpl/t/rpl_bug31076.test:
Test is only valid in mixed or row mode since it generates row events.
mysql-test/suite/rpl/t/rpl_events.test:
Removed explicit set of binlog_format and removed duplicate testing.
Instead, we rely on the test system to try all binlog formats.
mysql-test/suite/rpl/t/rpl_extraColmaster_innodb.test:
Removed triplicate testing and instead relying on test system.
Test is only relevant for row format since statement-based replication
cannot handle extra columns on master.
mysql-test/suite/rpl/t/rpl_extraColmaster_myisam.test:
Removed triplicate testing and instead relying on test system.
Test is only relevant for row format since statement-based replication
cannot handle extra columns on master.
mysql-test/suite/rpl/t/rpl_idempotency-slave.opt:
Removed .opt file to avoid server restarts.
mysql-test/suite/rpl/t/rpl_idempotency.test:
- Moved out row-only tests to a new test file, rpl_row_idempotency.test.
rpl_idempotency now only contains tests that execute in all
binlog_formats.
- While I was here, also removed .opt file to avoid server restarts.
The slave_exec_mode is now set inside the test instead.
mysql-test/suite/rpl/t/rpl_mix_found_rows.test:
Split rpl_found_rows.test into rpl_mix_found_rows.test (a new file) and
rpl_stm_found_rows.test (renamed rpl_found_rows.test). This file
contains the second half of the original rpl_found_rows.test with the
follwing changes:
- initialization
- removed SET_BINLOG_FORMAT and added have_binlog_format_mixed.inc
- minor formatting changes
mysql-test/suite/rpl/t/rpl_mix_insert_delayed.test:
Moved out code operating in mixed mode from rpl_stm_insert_delayed
(into rpl_mix_insert_delayed) and got rid of explicit setting of
binlog format.
mysql-test/suite/rpl/t/rpl_rbr_to_sbr.test:
Test cannot execute in statement mode, since we no longer
switch to row format when binlog_format=statement.
Enforced mixed mode throughout the test.
mysql-test/suite/rpl/t/rpl_row_idempotency.test:
Moved the second half of rpl_idempotency.test, which only
executed in row mode, to this new file. We now rely on the
test system to set binlog format.
mysql-test/suite/rpl/t/rpl_row_insert_delayed.test:
- Got rid of unnecessary explicit setting of binlog format.
- extra/rpl_tests/rpl_insert_delayed.test does not need the
$binlog_format_statement variable any more, so that was
removed.
mysql-test/suite/rpl/t/rpl_slave_skip.test:
The test switches binlog_format internally and master generates both
row and statement events. Hence, the slave must be able to log in both
statement and row format. Hence test was changed to only execute in
mixed mode.
mysql-test/suite/rpl/t/rpl_stm_found_rows.test:
Split rpl_found_rows.test into rpl_mix_found_rows.test (a new file) and
rpl_stm_found_rows.test (renamed rpl_found_rows.test). Changes in
this file:
- minor formatting changes
- added have_binlog_format_statement and removed SET BINLOG_FORMAT.
- second half of file is moved to rpl_mix_found_rows.test
- added cleanup code
mysql-test/suite/rpl/t/rpl_stm_insert_delayed.test:
Moved out code operating in mixed mode from rpl_stm_insert_delayed
(into rpl_mix_insert_delayed) and got rid of explicit setting of
binlog format.
mysql-test/suite/rpl/t/rpl_switch_stm_row_mixed.test:
The test switches binlog_format internally and master generates both
row and statement events. Hence, the slave must be able to log in both
statement and row format. Hence test was changed to only execute in
mixed mode on slave.
mysql-test/suite/rpl/t/rpl_temporary_errors.test:
Removed explicit set of binlog format. Instead, the test now only
executes in row mode.
mysql-test/suite/rpl/t/rpl_udf.test:
Remove explicit set of binlog format (and triplicate test execution)
and rely on test system executing the test in all binlog formats.
mysql-test/suite/rpl_ndb/combinations:
Added combinations file for rpl_ndb.
mysql-test/suite/rpl_ndb/r/rpl_ndb_binlog_format_errors.result:
new result file
mysql-test/suite/rpl_ndb/r/rpl_ndb_circular_simplex.result:
updated result file
mysql-test/suite/rpl_ndb/t/rpl_ndb_2innodb.test:
The test needs slave to be able to switch to row mode, so the
test was changed to only execute in mixed and row mode.
mysql-test/suite/rpl_ndb/t/rpl_ndb_2myisam.test:
The test needs slave to be able to switch to row mode, so the
test was changed to only execute in mixed and row mode.
mysql-test/suite/rpl_ndb/t/rpl_ndb_basic.test:
The test needs slave to be able to switch to row mode, so the
test was changed to only execute in mixed and row mode.
mysql-test/suite/rpl_ndb/t/rpl_ndb_binlog_format_errors-master.opt:
new option file
mysql-test/suite/rpl_ndb/t/rpl_ndb_binlog_format_errors-slave.opt:
new option file
mysql-test/suite/rpl_ndb/t/rpl_ndb_binlog_format_errors.test:
New test case to verify all errors and warnings generated by
decide_logging_format.
mysql-test/suite/rpl_ndb/t/rpl_ndb_blob.test:
The test needs slave to be able to switch to row mode, so the
test was changed to only execute in mixed and row mode.
mysql-test/suite/rpl_ndb/t/rpl_ndb_blob2.test:
The test needs slave to be able to switch to row mode, so the
test was changed to only execute in mixed and row mode.
mysql-test/suite/rpl_ndb/t/rpl_ndb_circular.test:
The test needs slave to be able to switch to row mode, so the
test was changed to only execute in mixed and row mode.
mysql-test/suite/rpl_ndb/t/rpl_ndb_circular_simplex.test:
The test needs slave to be able to switch to row mode, so the
test was changed to only execute in mixed and row mode.
While I was here, also made the test clean up after itself.
mysql-test/suite/rpl_ndb/t/rpl_ndb_commit_afterflush.test:
The test needs slave to be able to switch to row mode, so the
test was changed to only execute in mixed and row mode.
mysql-test/suite/rpl_ndb/t/rpl_ndb_ctype_ucs2_def.test:
The test needs slave to be able to switch to row mode, so the
test was changed to only execute in mixed and row mode.
mysql-test/suite/rpl_ndb/t/rpl_ndb_delete_nowhere.test:
The test needs slave to be able to switch to row mode, so the
test was changed to only execute in mixed and row mode.
mysql-test/suite/rpl_ndb/t/rpl_ndb_do_db.test:
The test needs slave to be able to switch to row mode, so the
test was changed to only execute in mixed and row mode.
mysql-test/suite/rpl_ndb/t/rpl_ndb_do_table.test:
The test needs slave to be able to switch to row mode, so the
test was changed to only execute in mixed and row mode.
mysql-test/suite/rpl_ndb/t/rpl_ndb_func003.test:
The test needs slave to be able to switch to row mode, so the
test was changed to only execute in mixed and row mode.
mysql-test/suite/rpl_ndb/t/rpl_ndb_innodb_trans.test:
The test needs slave to be able to switch to row mode, so the
test was changed to only execute in mixed and row mode.
mysql-test/suite/rpl_ndb/t/rpl_ndb_insert_ignore.test:
The test needs slave to be able to switch to row mode, so the
test was changed to only execute in mixed and row mode.
mysql-test/suite/rpl_ndb/t/rpl_ndb_mixed_engines_transactions.test:
The test needs slave to be able to switch to row mode, so the
test was changed to only execute in mixed and row mode.
mysql-test/suite/rpl_ndb/t/rpl_ndb_multi_update3.test:
The test needs slave to be able to switch to row mode, so the
test was changed to only execute in mixed and row mode.
mysql-test/suite/rpl_ndb/t/rpl_ndb_rep_ignore.test:
The test needs slave to be able to switch to row mode, so the
test was changed to only execute in mixed and row mode.
mysql-test/suite/rpl_ndb/t/rpl_ndb_row_001.test:
The test needs slave to be able to switch to row mode, so the
test was changed to only execute in mixed and row mode.
mysql-test/suite/rpl_ndb/t/rpl_ndb_sp003.test:
The test needs slave to be able to switch to row mode, so the
test was changed to only execute in mixed and row mode.
mysql-test/suite/rpl_ndb/t/rpl_ndb_sp006.test:
The test needs slave to be able to switch to row mode, so the
test was changed to only execute in mixed and row mode.
mysql-test/suite/rpl_ndb/t/rpl_ndb_trig004.test:
The test needs slave to be able to switch to row mode, so the
test was changed to only execute in mixed and row mode.
mysql-test/t/partition_innodb_stmt.test:
Changed error code for innodb updates with READ COMMITTED or
READ UNCOMMITTED transaction isolation level and
binlog_format=statement.
sql/event_db_repository.cc:
Use member function to read current_stmt_binlog_row_based.
sql/events.cc:
Use member function to read current_stmt_binlog_row_based.
sql/ha_ndbcluster_binlog.cc:
reset_current_stmt_binlog_row_based() is not a no-op for the ndb_binlog
thread any more. Instead, the ndb_binlog thread now forces row mode both
initially and just after calling mysql_parse. (mysql_parse() is the only
place where reset_current_stmt_binlog_row_based() may be called from
the ndb_binlog thread, so these are the only two places that need to
change.)
sql/ha_partition.cc:
Use member function to read current_stmt_binlog_row_based.
sql/handler.cc:
Use member function to read current_stmt_binlog_row_based.
sql/item_create.cc:
Added DBUG_ENTER to some functions, to be able to trace when
set_stmt_unsafe is called.
sql/log.cc:
Use member function to read current_stmt_binlog_row_based.
sql/log_event.cc:
- Moved logic for changing to row format out of do_apply_event (and into
decide_logging_format).
- Added @todo comment for post-push cleanup.
sql/log_event_old.cc:
Move logic for changing to row format out of do_apply_event (and into
decide_logging_format).
sql/mysql_priv.h:
Make decide_logging_format() a member of the THD class, for two reasons:
- It is natural from an object-oriented perspective.
- decide_logging_format() needs to access private members of THD
(specifically, the new binlog_warning_flags field).
sql/rpl_injector.cc:
Removed call to set_current_stmt_binlog_row_based().
From now on, only decide_logging_fromat is allowed to modify
current_stmt_binlog_row_based. This call is from the ndb_binlog
thread, mostly executing code in ha_ndbcluster_binlog.cc.
This call can be safely removed, because:
- current_stmt_binlog_row_based is initialized for the ndb_binlog
thread's THD object when the THD object is created. So we're
not going to read uninitialized memory.
- The behavior of ndb_binlog thread does not use the state of the
current_stmt_binlog_row_based. It is conceivable that the
ndb_binlog thread would rely on the current_stmt_binlog_format
in two situations:
(1) when it calls mysql_parse;
(2) when it calls THD::binlog_query.
In case (1), it always clears THD::options&OPTION_BIN_LOG (because
run_query() in ha_ndbcluster_binlog.cc is only called with
disable_binlogging = TRUE).
In case (2), it always uses qtype=STMT_QUERY_TYPE.
sql/set_var.cc:
Added @todo comment for post-push cleanup.
sql/share/errmsg.txt:
Added new error messages and clarified ER_BINLOG_UNSAFE_STATEMENT.
sql/sp.cc:
Added DBUG_ENTER, to be able to trace when set_stmt_unsafe is called.
Got rid of MYSQL_QUERY_TYPE: it was equivalent to STMT_QUERY_TYPE.
sql/sp_head.cc:
Use member function to read current_stmt_binlog_row_based.
sql/sp_head.h:
Added DBUG_ENTER, to be able to trace when set_stmt_unsafe is called.
sql/sql_acl.cc:
Got rid of MYSQL_QUERY_TYPE: it was equivalent to STMT_QUERY_TYPE.
sql/sql_base.cc:
- Made decide_logging_format take care of all logic for deciding the
logging format, and for determining the related warnings and errors.
See comment above decide_logging_format for details.
- Made decide_logging_format a member function of THD, since it needs
to access private members of THD and since its purpose is to update
the state of a THD object.
- Added DBUG_ENTER, to be able to trace when set_stmt_unsafe is called.
sql/sql_class.cc:
- Moved logic for determining unsafe warnings away from THD::binlog_query
(and into decide_logging_format()). Now, it works like this:
1. decide_logging_format detects that the current statement shall
produce a warning, if it ever makes it to the binlog
2. decide_logging_format sets a flag of THD::binlog_warning_flags.
3. THD::binlog_query reads the flag. If the flag is set, it generates
a warning.
- Use member function to read current_stmt_binlog_row_based.
sql/sql_class.h:
- Added THD::binlog_warning_flags (see sql_class.cc for explanation).
- Made decide_logging_format() and reset_for_next_command() member
functions of THD (instead of standalone functions). This was needed
for two reasons: (1) the functions need to access the private member
THD::binlog_warning_flags; (2) the purpose of these functions is to
update the staet of a THD object, so from an object-oriented point
of view they should be member functions.
- Encapsulated current_stmt_binlog_row_based, so it is now private and
can only be accessed from a member function. Also changed the
data type to an enumeration instead of a bool.
- Removed MYSQL_QUERY_TYPE, because it was equivalent to
STMT_QUERY_TYPE anyways.
- When reset_current_stmt_binlog_row_based was called from the
ndb_binlog thread, it would behave as a no-op. This special
case has been removed, and the behavior of
reset_current_stmt_binlog_row_based does not depend on which thread
calls it any more. The special case did not serve any purpose,
since the ndb binlog thread did not take the
current_stmt_binlog_row_based flag into account anyways.
sql/sql_delete.cc:
- Moved logic for setting row format for DELETE..LIMIT away from
mysql_prepare_delete.
(Instead, we mark the statement as unsafe at parse time (sql_yacc.yy)
and rely on decide_logging_format() (sql_class.cc) to set row format.)
This is part of the fix for BUG#45831.
- Use member function to read current_stmt_binlog_row_based.
sql/sql_insert.cc:
- Removed unnecessary calls to thd->lex->set_stmt_unsafe() and
thd->set_current_stmt_binlog_row_based_if_mixed() from
handle_delayed_insert(). The calls are unnecessary because they
have already been made; they were made in the constructor of
the `di' object.
- Since decide_logging_format() is now a member function of THD, code
that calls decide_logging_format() had to be updated.
- Added DBUG_ENTER call, to be able to trace when set_stmt_unsafe is
called.
- Moved call to set_stmt_unsafe() for INSERT..SELECT..LIMIT away from
mysql_insert_select_prepare() (and into decide_logging_format).
This is part of the fix for BUG#45831.
- Use member function to read current_stmt_binlog_row_based.
sql/sql_lex.h:
- Added the flag BINLOG_STMT_FLAG_ROW_INJECTION to enum_binlog_stmt_flag.
This was necessary so that a statement can identify itself as a row
injection.
- Added appropriate setter and getter functions for the new flag.
- Added or clarified some comments.
- Added DBUG_ENTER()
sql/sql_load.cc:
Use member function to read current_stmt_binlog_row_based.
sql/sql_parse.cc:
- Made mysql_reset_thd_for_next_command() clear thd->binlog_warning_flags.
- Since thd->binlog_warning_flags is private, it must be set in a
member function of THD. Hence, moved the body of
mysql_reset_thd_for_next_command() to the new member function
THD::reset_thd_for_next_command(), and made
mysql_reset_thd_for_next_command() call
THD::reset_thd_for_next_command().
- Removed confusing comment.
- Use member function to read current_stmt_binlog_row_based.
sql/sql_repl.cc:
Use member function to read current_stmt_binlog_row_based.
sql/sql_table.cc:
Use member function to read current_stmt_binlog_row_based.
sql/sql_udf.cc:
Use member function to read current_stmt_binlog_row_based.
sql/sql_update.cc:
Moved logic for setting row format for UPDATE..LIMIT away from
mysql_prepare_update.
(Instead, we mark the statement as unsafe at parse time (sql_yacc.yy)
and rely on decide_logging_format() (sql_class.cc) to set row format.)
This is part of the fix for BUG#45831.
sql/sql_yacc.yy:
Made INSERT DELAYED, INSERT..SELECT..LIMIT, UPDATE..LIMIT, and
DELETE..LIMIT mark themselves as unsafe at parse time (instead
of at execution time).
This is part of the fixes BUG#45831 and BUG#45825.
storage/example/ha_example.cc:
Made exampledb accept inserts. This was needed by the new test case
rpl_ndb_binlog_format_errors, because it needs an engine that
is statement-only (and accepts inserts).
storage/example/ha_example.h:
Made exampledb a statement-only engine instead of a row-only engine.
No existing test relied exampledb's row-only capabilities. The new
test case rpl_ndb_binlog_format_errors needs an engine that is
statement-only.
storage/innobase/handler/ha_innodb.cc:
- Changed error error code and message given by innodb when
binlog_format=STATEMENT and transaction isolation level is
READ COMMITTED or READ UNCOMMITTED.
- While I was here, also simplified the condition for
checking when to give the error.
There is an inconsistency with DROP DATABASE|TABLE|EVENT IF EXISTS and
CREATE DATABASE|TABLE|EVENT IF NOT EXISTS. DROP IF EXISTS statements are
binlogged even if either the DB, TABLE or EVENT does not exist. In
contrast, Only the CREATE EVENT IF NOT EXISTS is binlogged when the EVENT
exists.
This patch fixes the following cases for all the replication formats:
CREATE DATABASE IF NOT EXISTS.
CREATE TABLE IF NOT EXISTS,
CREATE TABLE IF NOT EXISTS ... LIKE,
CREAET TABLE IF NOT EXISTS ... SELECT.
sql/sql_insert.cc:
Part of the code was moved from the create_table_from_items to select_create::prepare.
When replication is row based, CREATE TABLE IF NOT EXISTS.. SELECT is binlogged if the table exists.
timeout
In STMT and MIXED modes, a statement that changes both non-transactional and
transactional tables must be written to the binary log whenever there are
changes to non-transactional tables. This means that the statement gets into the
binary log even when the changes to the transactional tables fail. In particular
, in the presence of a failure such statement is annotated with the error number
and wrapped in a begin/rollback. On the slave, while applying the statement, it
is expected the same failure and the rollback prevents the transactional changes
to be persisted.
Unfortunately, statements that fail due to concurrency issues (e.g. deadlocks,
timeouts) are logged in the same way causing the slave to stop as the statements
are applied sequentially by the SQL Thread. To fix this bug, we automatically
ignore concurrency failures on the slave. Specifically, the following failures
are ignored: ER_LOCK_WAIT_TIMEOUT, ER_LOCK_DEADLOCK and ER_XA_RBDEADLOCK.
Large transactions and statements may corrupt the binary log if the size of the
cache, which is set by the max_binlog_cache_size, is not enough to store the
the changes.
In a nutshell, to fix the bug, we save the position of the next character in the
cache before starting processing a statement. If there is a problem, we simply
restore the position thus removing any effect of the statement from the cache.
Unfortunately, to avoid corrupting the binary log, we may end up loosing changes
on non-transactional tables if they do not fit in the cache. In such cases, we
store an Incident_log_event in order to stop the slave and alert users that some
changes were not logged.
Precisely, for every non-transactional changes that do not fit into the cache,
we do the following:
a) the statement is *not* logged
b) an incident event is logged after committing/rolling back the transaction,
if any. Note that if a failure happens before writing the incident event to
the binary log, the slave will not stop and the master will not have reported
any error.
c) its respective statement gives an error
For transactional changes that do not fit into the cache, we do the following:
a) the statement is *not* logged
b) its respective statement gives an error
To work properly, this patch requires two additional things. Firstly, callers to
MYSQL_BIN_LOG::write and THD::binlog_query must handle any error returned and
take the appropriate actions such as undoing the effects of a statement. We
already changed some calls in the sql_insert.cc, sql_update.cc and sql_insert.cc
modules but the remaining calls spread all over the code should be handled in
BUG#37148. Secondly, statements must be either classified as DDL or DML because
DDLs that do not get into the cache must generate an incident event since they
cannot be rolled back.
The server was not cleaning the last IO error and error number when
resetting slave.
This patch addresses this issue by backporting into 5.1 part of the
patch in BUG 34654. A fix for this issue had already been pushed into
6.0 as part of the aforementioned bug, however the patch also included
some refactoring. The fix for 5.1 does not take into account the
refactoring part.
mysql-test/extra/rpl_tests/rpl_reset_slave.test:
Backported the test case and improved with deploying include/start_slave.inc
in relevant spots.
sql/slave.cc:
Backported part of patch from 6.0 that includes cleaning
mi->clear_error() at:
1. beginning of handle_slave_io
2. on successful connection
Also, backported the assertion added in the original patch.
sql/sql_repl.cc:
Backported the call to mi->clear_error() on reset_slave().
Certain multi-updates gave different results on InnoDB from
to MyISAM, due to on-the-fly updates being used on the former and
the update order matters.
Fixed by turning off on-the-fly updates when update order
dependencies are present.
mysql-test/r/innodb_mysql.result:
Bug#43580: Test result.
mysql-test/suite/rpl/r/rpl_slave_skip.result:
Bug#43580: Changed test result. The InnoDB result is now what it would have been on MyISAM.
mysql-test/t/innodb_mysql.test:
Bug#43580: Test case.
sql/sql_base.cc:
Bug#43580: Added a word of caution about using tmp_set here.
sql/sql_update.cc:
Bug#43580: Fix.
Calls to TABLE::mark_columns_needed_for_update() are moved
from mysql_multi_update_prepare() and right before the decison
to do on-the-fly updates to the place where we do (or don't do)
on-the-fly updates.
In order to define the --slave-load-tmpdir, the init_relay_log_file()
was calling fn_format(MY_PACK_FILENAME) which internally was indirectly
calling strmov_overlapp() (through pack_dirname) and the following
warning message was being printed out while running in Valgrind:
"source and destination overlap in strcpy".
We fixed the issue by removing the flag MY_PACK_FILENAME as it was not
necessary. In a nutshell, with this flag the function fn_format() tried
to replace a directory by either "~", "." or "..". However, we wanted
exactly to remove such strings.
In this patch, we also refactored the functions init_relay_log_file()
and check_temp_dir(). The former was refactored to call the fn_format()
with the flag MY_SAFE_PATH along with the MY_RETURN_REAL_PATH, in order
to avoid issues with long directories and return an absolute path,
respectively. The flag MY_SAFE_UNPACK_FILENAME was removed too as it was
responsible for removing "~", "." or ".." only from the file parameter
and we wanted to remove such strings from the directory parameter in
the fn_format(). This result is stored in an rli variable, which is then
processed by the other function in order to verify if the directory exists
and if we are able to create files in it.
mysql-test/suite/rpl/t/rpl_slave_load_tmpdir_not_exist.test:
Changed the output to make it consistent among different runs.
mysys/mf_format.c:
Replaced a return for DBUG_RETURN.
Bug#319 if while a non-transactional slave is replicating a transaction possible problem
only testing related: addressing reviewers' comments.
mysql-test/suite/bugs/r/rpl_bug38205.result:
new results.
mysql-test/suite/bugs/t/rpl_bug38205.test:
changing output per reviewer's offering.
mysql-test/suite/rpl/r/rpl_start_stop_slave.result:
new results.
mysql-test/suite/rpl/t/rpl_start_stop_slave.test:
refining tests for Bug #38205, Bug#319 to start slave with the necessary synchronization;
changing output per reviewer's offering;
removing redundant wait for stop.
The result set for multi-row statements is not the same between STMT and
RBR and among different versions. Thus to avoid test failures, we are not
printing out such result sets. Note, however, that this does not have
impact on coverage and accuracy since the execution is able to continue
without further issues when an error is found on the master and such error
is set to be skipped.
RBR was not considering the option --slave-skip-errors.
To fix the problem, we are reporting the ignored ERROR(s) as warnings thus avoiding
stopping the SQL Thread. Besides, it fixes the output of "SHOW VARIABLES LIKE
'slave_skip_errors'" which was showing nothing when the value "all" was assigned
to --slave-skip-errors.
@sql/log_event.cc
skipped rbr errors when the option skip-slave-errors is set.
@sql/slave.cc
fixed the output of for SHOW VARIABLES LIKE 'slave_skip_errors'"
@test-cases
fixed the output of rpl.rpl_idempotency
updated the test case rpl_skip_error
1. Test case was rewritten completely.
2. Test covers 3 cases:
a) do deadlock on slave, wait retries of transaction, unlock slave before lock
timeout;
b) do deadlock on slave and wait error 'lock timeout exceed' on slave;
c) same as b) but if of max relay log size = 0;
3. Added comments inline.
4. Updated result file.
Mysql server crashes because unsafe statements warning is wrongly elevated to error,
which is set the error status of Diagnostics_area of the thread in THD::binlog_query().
Yet the caller believes that binary logging shouldn't touch the status, so it will
set the status also later by my_ok(), my_error() or my_message() seperately
according to the execution result of the statement or transaction.
But the status of Diagnostics_area of the thread is allowed to set only once.
Fixed to clear the error wrongly set by binary logging, but keep the warning message.
mysql-test/suite/binlog/r/binlog_stm_ps.result:
Change unsafe warning to NOTE level
mysql-test/suite/binlog/r/binlog_unsafe.result:
Test case result for unsafe statements to ensure mysql sever don't crash
mysql-test/suite/binlog/t/binlog_unsafe.test:
Test case for unsafe statements to ensure mysql sever don't crash
mysql-test/suite/rpl/r/rpl_skip_error.result:
Change unsafe warning to NOTE level
mysql-test/suite/rpl/r/rpl_stm_loadfile.result:
Change unsafe warning to NOTE level
mysql-test/suite/rpl/r/rpl_udf.result:
Change unsafe warning to NOTE level
sql/sql_class.cc:
the error status of the thread is cleared When a warning is elevated to an error
because of unsafe warning of binary log.
Test was flakey on some machines and showed spurious
reds for races.
New-and-improved test makes do with fewer statements,
no mysqltest-variables, and no backticks. Should hope-
fully be more robust. Heck, it's debatable whether we
should have a test for this, anyway.
mysql-test/suite/rpl/r/rpl_temporary.result:
streamlined
mysql-test/suite/rpl/t/rpl_temporary.test:
streamlined
Bug#319 if while a non-transactional slave is replicating a transaction possible problem
It is impossible to roll back a mixed engines transaction when one of the engine is
non-transaction. In replication that fact is crucial because the slave can not safely
re-apply a transction that was interrupted with STOP SLAVE.
Fixed with making STOP SLAVE not be effective immediately in the case the current
group of replication events has modified a non-transaction table. In order for slave to leave
either the group needs finishing or the user issues KILL QUERY|CONNECTION slave_thread_id.
mysql-test/suite/bugs/r/rpl_bug38205.result:
bug#38205 non-deterministic part of tests results.
mysql-test/suite/bugs/t/rpl_bug38205.test:
bug#38205 non-deterministic part of tests.
mysql-test/suite/rpl/r/rpl_start_stop_slave.result:
bug#38205 deterministic part of tests results.
mysql-test/suite/rpl/t/rpl_start_stop_slave-slave.opt:
increasing `innodb_lock_wait_timeout' to make the test pass on slow env w/o
timeout expired issue.
mysql-test/suite/rpl/t/rpl_start_stop_slave.test:
bug#38205 deterministic part of tests.
sql/log_event.cc:
Augmenting row-based events applying with the notion of
thd->transaction.{all,stmt}.modified_non_trans_table.
The pair is set and reset according to its specification
for the mixed transaction processing.
Particualry, once `modified_non_trans_table' is set in the row-events
processing loop, it will remain till the commit of the transaction.
sql/slave.cc:
Consulting `thd->transaction.all.modified_non_trans_table' to decide
whether to terminate by the sql thread or to continue even though
the sql thread might have been STOP-ed (rli->abort_slave).
manual merge. also adds test specific to 5.1+
mysql-test/suite/rpl/r/rpl_temporary.result:
show that a non-privileged user trying to
kill system-threads no longer crashes the
server. test in 5.1+ only.
mysql-test/suite/rpl/t/rpl_temporary.test:
show that a non-privileged user trying to
kill system-threads no longer crashes the
server. test in 5.1+ only.
sql/sql_class.cc:
manual merge
sql/sql_class.h:
manual merge
sql/sql_parse.cc:
manual merge
The issue happened to be two-fold.
The table map event was recorded into binlog having
an incorrect size when number of columns exceeded 251.
The Row-based event had incorrect recording and restoring m_width member within
the same as above conditions.
Fixed with correcting m_data_size and m_width.
mysql-test/suite/rpl/r/rpl_row_wide_table.result:
the new test results.
mysql-test/suite/rpl/t/rpl_row_wide_table.test:
regression test for bug#42977.
sql/log_event.cc:
0. all buffers that used in net_store_length() are augmented with 1 for safety
to be able to contain the magic and the content of ulonglong as well;
1. Rows_log_event::get_data_size() yieled incorrect size |m_width/8| whereas
it should be m_width;
2. Table_map_log_event::Table_map_log_event yieled incorrect value for
`m_data_size' probably presuming 1-byte integer max for the column number;
sql/rpl_utility.h:
DBUG_PRINT_BITSET() macro is left 256-cols limited but has made safe and commented.
Reset master before next test.
mysql-test/suite/rpl/r/rpl_timezone.result:
Test result.
mysql-test/suite/rpl/t/rpl_timezone.test:
Reset master before next test.
LOAD_FILE
LOAD_FILE is not safe to replicate in STATEMENT mode, because it
depends on a file (which is loaded on master and may not exist in
slave(s)). This leads to scenarios on which the slave replicates the
statement with 'load_file' and it will try to load the file from local
file system. Given that the file may not exist in the slave filesystem
the operation will not succeed (probably returning NULL), causing
master and slave(s) to diverge. However, when using MIXED mode
replication, this can be made to work, if the statement including
LOAD_FILE is marked as unsafe, triggering a switch to ROW mode,
meaning that the contents of the file are written to binlog as row
events. Consequently, the contents from the file in the master will
reach the slave via the binlog.
This patch addresses this bug by marking the load_file function as
unsafe. When in mixed mode and when LOAD_FILE is issued, there will be
a switch to row mode. Furthermore, when in statement mode, the
LOAD_FILE will raise a warning that the statement is unsafe in that
mode.
mysql-test/extra/rpl_tests/rpl_loadfile.test:
Extra file that is "sourced" on both rpl_loadfile and rpl_stm_loadfile
test files.
mysql-test/suite/rpl/r/rpl_loadfile.result:
Updated with the results from the test case added to this file.
mysql-test/suite/rpl/r/rpl_stm_loadfile.result:
Result file for rpl_loadfile test split with the warnings in statement
mode.
mysql-test/suite/rpl/t/rpl_loadfile.test:
After splitting the original rpl_loadfile file, this one is only
required to be executed in mixed or row format.
Appended the test for 39701 to this file.
mysql-test/suite/rpl/t/rpl_stm_loadfile.test:
Split the original rpl_loadfile test because load_file now raises
a warning when in statement mode. The goal of this split is
two-fold: i) make the test case more resilient; ii) assert that
warnings are indeed raised when in statement mode.
sql/item_create.cc:
Added the set_stmt_unsafe call to lex.
Compiling with debug and assigning an invalid directory to --slave-load-tmpdir
was crashing the slave due to the following assertion DBUG_ASSERT(! is_set() ||
can_overwrite_status). This assertion assumes that a thread can change its
state once (i.e. ok,error, etc) before aborting, cleaning/resuming or completing
its execution unless the overwrite flag (i.e. can_overwrite_status) is true.
The Append_block_log_event::do_apply_event which is responsible for creating
temporary file(s) was not cleaning the thread state. Thus a failure while
trying to create a file in an invalid temporary directory was causing the crash.
To fix the problem we check if the temporary directory is valid before starting
the SQL Thread and reset the thread state before creating a file in
Append_block_log_event::do_apply_event.
mysql.procs_priv table itself does not get replicated.
Inserting routine privilege record into mysql.procs_priv table
is triggered by creating function/procedure statements
according to current user's privileges.
Because the current user of SQL thread has GLOBAL_ACL,
which doesn't need any check mysql.procs_priv privilege
when create/alter/execute routines.
Corresponding GLOBAL_ACL privilege user
doesn't insert routine privilege record into
mysql.procs_priv when creating a routine.
Fixed by switching the current user of SQL thread to definer user if
the definer user exists on slave.
That populates procs_priv, otherwise to keep the SQL thread
user and procs_priv remains unchanged.
mysql-test/suite/rpl/r/rpl_do_grant.result:
Test case result for routine privilege when definer user exist or not on slave
mysql-test/suite/rpl/t/rpl_do_grant.test:
Test case result for routine privilege when definer user exist or not on slave
sql/sql_parse.cc:
Switch current user of SQL thread to definer user if the definer user
existes on slave when checking whether the routine privilege is
needed to insert mysql.procs_priv table or not.
The problem is issued because we set wrong start position and stop position of query string into binlog.
That two values are stored as part of head info of query string.
When we parse binlog, we first get position values then get the query string according position values.
But seems that two values are not calculated correctly after the parse of Yacc.
We don't want to touch so much of yacc because it may influence other codes.
So just add one space after 'INTO' key word when parsing.
This can easily resolve the problem.
mysql-test/suite/rpl/r/rpl_loaddatalocal.result:
Test result
mysql-test/suite/rpl/t/rpl_loaddatalocal.test:
Test case
sql/log_event.cc:
Add space after 'INTO'.
The method to purge binary log files produces different results in some platforms.
The reason is that the purge time is calculated based on table modified time and
that can't guarantee to purge master-bin.000002 in all platforms.(eg. windows)
Use a new way that sets the time to purge binlog file 1 second after the last
modified time of master-bin.000002.
That can be sure that the file is always deleted in any platform.
mysql-test/suite/rpl/r/rpl_rotate_logs.result:
Test result
mysql-test/suite/rpl/t/rpl_rotate_logs.test:
Test file
The method to purge binary log files produces different results in some platforms.
The reason is that the purge time is calculated based on table modified time and
that can't guarantee to purge master-bin.000002 in all platforms.(eg. windows)
Use a new way that sets the time to purge binlog file 1 second after the last modified time of master-bin.000002.
That can be sure that the file is always deleted in any platform.
mysql-test/suite/rpl/r/rpl_rotate_logs.result:
Test result
mysql-test/suite/rpl/t/rpl_rotate_logs.test:
Change 'purge time' 1 sencond after the last modified time of master-bin.000002.
When using mixed mode the record values stored inside the storage
engine differed from the ones computed from the row event. This
happened because the prepare_record function was calling
empty_record macro causing some don't care bits to be left set.
Replacing the empty_record plus explicitly setting defaults with
restore_record to restore the record default values fixes this.
slave.
In mixed mode, if we create a temporary table and do some update which switch to ROW format,
the format will keep in ROW format until the session ends or the table is dropped explicitly.
When the session ends, the temp table is dropped automaticly at cleanup time.
but it checks only current binlog format and so skip insertion of DROP TABLE instructions into binlog.
So the temp table can't be dropped correctly at slave.
Our solution is that when closing temp tables at cleanup time we check both binlog format and binlog mode,
and we could write DROP TABLE instructions into binlog if current binlog format is ROW but in MIX mode.
mysql-test/suite/rpl/r/rpl_temp_table_mix_row.result:
Test result file.
mysql-test/suite/rpl/t/rpl_temp_table_mix_row.test:
Test file.
sql/sql_base.cc:
Didn't do binloging when both current format and default format are ROW.
If secure-file-priv was set on slave, it became unable to execute
LOAD DATA INFILE statements sent from master using mixed or
statement-based replication.
This patch fixes the issue by ignoring this security restriction
and checking if the files are created and read by the slave in the
--slave-load-tmpdir while executing the SQL Thread.
