Major replication test framework cleanup. This does the following:
- Ensure that all tests clean up the replication state when they
finish, by making check-testcase check the output of SHOW SLAVE STATUS.
This implies:
- Slave must not be running after test finished. This is good
because it removes the risk for sporadic errors in subsequent
tests when a test forgets to sync correctly.
- Slave SQL and IO errors must be cleared when test ends. This is
good because we will notice if a test gets an unexpected error in
the slave threads near the end.
- We no longer have to clean up before a test starts.
- Ensure that all tests that wait for an error in one of the slave
threads waits for a specific error. It is no longer possible to
source wait_for_slave_[sql|io]_to_stop.inc when there is an error
in one of the slave threads. This is good because:
- If a test expects an error but there is a bug that causes
another error to happen, or if it stops the slave thread without
an error, then we will notice.
- When developing tests, wait_for_*_to_[start|stop].inc will fail
immediately if there is an error in the relevant slave thread.
Before this patch, we had to wait for the timeout.
- Remove duplicated and repeated code for setting up unusual replication
topologies. Now, there is a single file that is capable of setting
up arbitrary topologies (include/rpl_init.inc, but
include/master-slave.inc is still available for the most common
topology). Tests can now end with include/rpl_end.inc, which will clean
up correctly no matter what topology is used. The topology can be
changed with include/rpl_change_topology.inc.
- Improved debug information when tests fail. This includes:
- debug info is printed on all servers configured by include/rpl_init.inc
- User can set $rpl_debug=1, which makes auxiliary replication files
print relevant debug info.
- Improved documentation for all auxiliary replication files. Now they
describe purpose, usage, parameters, and side effects.
- Many small code cleanups:
- Made have_innodb.inc output a sensible error message.
- Moved contents of rpl000017-slave.sh into rpl000017.test
- Added mysqltest variables that expose the current state of
disable_warnings/enable_warnings and friends.
- Too many to list here: see per-file comments for details.
Row-based replication requires the types of columns on the
master and slave to be approximately the same (some safe
conversions between strings are allowed), but does not
allow safe conversions between fields of similar types such
as TINYINT and INT.
This patch implement type conversions between similar fields
on the master and slave.
The conversions are controlled using a new variable
SLAVE_TYPE_CONVERSIONS of type SET('ALL_LOSSY','ALL_NON_LOSSY').
Non-lossy conversions are any conversions that do not run the
risk of losing any information, while lossy conversions can
potentially truncate the value. The column definitions are
checked to decide if the conversion is acceptable.
If neither conversion is enabled, it is required that the
definitions of the columns are identical on master and slave.
Conversion is done by creating an internal conversion table,
unpacking the master data into it, and then copy the data to
the real table on the slave.
General overview:
The logic for switching to row format when binlog_format=MIXED had
numerous flaws. The underlying problem was the lack of a consistent
architecture.
General purpose of this changeset:
This changeset introduces an architecture for switching to row format
when binlog_format=MIXED. It enforces the architecture where it has
to. It leaves some bugs to be fixed later. It adds extensive tests to
verify that unsafe statements work as expected and that appropriate
errors are produced by problems with the selection of binlog format.
It was not practical to split this into smaller pieces of work.
Problem 1:
To determine the logging mode, the code has to take several parameters
into account (namely: (1) the value of binlog_format; (2) the
capabilities of the engines; (3) the type of the current statement:
normal, unsafe, or row injection). These parameters may conflict in
several ways, namely:
- binlog_format=STATEMENT for a row injection
- binlog_format=STATEMENT for an unsafe statement
- binlog_format=STATEMENT for an engine only supporting row logging
- binlog_format=ROW for an engine only supporting statement logging
- statement is unsafe and engine does not support row logging
- row injection in a table that does not support statement logging
- statement modifies one table that does not support row logging and
one that does not support statement logging
Several of these conflicts were not detected, or were detected with
an inappropriate error message. The problem of BUG#39934 was that no
appropriate error message was written for the case when an engine
only supporting row logging executed a row injection with
binlog_format=ROW. However, all above cases must be handled.
Fix 1:
Introduce new error codes (sql/share/errmsg.txt). Ensure that all
conditions are detected and handled in decide_logging_format()
Problem 2:
The binlog format shall be determined once per statement, in
decide_logging_format(). It shall not be changed before or after that.
Before decide_logging_format() is called, all information necessary to
determine the logging format must be available. This principle ensures
that all unsafe statements are handled in a consistent way.
However, this principle is not followed:
thd->set_current_stmt_binlog_row_based_if_mixed() is called in several
places, including from code executing UPDATE..LIMIT,
INSERT..SELECT..LIMIT, DELETE..LIMIT, INSERT DELAYED, and
SET @@binlog_format. After Problem 1 was fixed, that caused
inconsistencies where these unsafe statements would not print the
appropriate warnings or errors for some of the conflicts.
Fix 2:
Remove calls to THD::set_current_stmt_binlog_row_based_if_mixed() from
code executed after decide_logging_format(). Compensate by calling the
set_current_stmt_unsafe() at parse time. This way, all unsafe statements
are detected by decide_logging_format().
Problem 3:
INSERT DELAYED is not unsafe: it is logged in statement format even if
binlog_format=MIXED, and no warning is printed even if
binlog_format=STATEMENT. This is BUG#45825.
Fix 3:
Made INSERT DELAYED set itself to unsafe at parse time. This allows
decide_logging_format() to detect that a warning should be printed or
the binlog_format changed.
Problem 4:
LIMIT clause were not marked as unsafe when executed inside stored
functions/triggers/views/prepared statements. This is
BUG#45785.
Fix 4:
Make statements containing the LIMIT clause marked as unsafe at
parse time, instead of at execution time. This allows propagating
unsafe-ness to the view.
Problem: it is unsafe to read base64-printed events without first
reading the Format_description_log_event (FD). Currently, mysqlbinlog
cannot print the FD.
As a side effect, another bug has also been fixed: When mysqlbinlog
--start-position=X was specified, no ROLLBACK was printed. I changed
this, so that ROLLBACK is always printed.
This patch does several things:
- Format_description_log_event (FD) now print themselves in base64
format.
- mysqlbinlog is now able to print FD events. It has three modes:
--base64-output=auto Print row events in base64 output, and print
FD event. The FD event is printed even if
it is outside the range specified with
--start-position, because it would not be
safe to read row events otherwise. This is
the default.
--base64-output=always Like --base64-output=auto, but also print
base64 output for query events. This is
like the old --base64-output flag, which
is also a shorthand for
--base64-output=always
--base64-output=never Never print base64 output, generate error if
row events occur in binlog. This is
useful to suppress the FD event in binlogs
known not to contain row events (e.g.,
because BINLOG statement is unsafe,
requires root privileges, is not SQL, etc)
- the BINLOG statement now handles FD events correctly, by setting
the thread's rli's relay log's description_event_for_exec to the
loaded event.
In fact, executing a BINLOG statement is almost the same as reading
an event from a relay log. Before my patch, the code for this was
separated (exec_relay_log_event in slave.cc executes events from
the relay log, mysql_client_binlog_statement in sql_binlog.cc
executes BINLOG statements). I needed to augment
mysql_client_binlog_statement to do parts of what
exec_relay_log_event does. Hence, I did a small refactoring and
moved parts of exec_relay_log_event to a new function, which I
named apply_event_and_update_pos. apply_event_and_update_pos is
called both from exec_relay_log_event and from
mysql_client_binlog_statement.
- When a non-FD event is executed in a BINLOG statement, without
previously executing a FD event in a BINLOG statement, it generates
an error, because that's unsafe. I took a new error code for that:
ER_NO_FORMAT_DESCRIPTION_EVENT_BEFORE_BINLOG_STATEMENTS.
In order to get a decent error message containing the name of the
event, I added the class method char*
Log_event::get_type_str(Log_event_type type), which returns a
string name for the given Log_event_type. This is just like the
existing char* Log_event::get_type_str(), except it is a class
method that takes the log event type as parameter.
I also added PRE_GA_*_ROWS_LOG_EVENT to Log_event::get_type_str(),
so that names of old rows event are properly printed.
- When reading an event, I added a check that the event type is known
by the current Format_description_log_event. Without this, it may
crash on bad input (and I was struck by this several times).
- I patched the following test cases, which all contain BINLOG
statements for row events which must be preceded by BINLOG
statements for FD events:
- rpl_bug31076
While I was here, I fixed some small things in log_event.cc:
- replaced hard-coded 4 by EVENT_TYPE_OFFSET in 3 places
- replaced return by DBUG_VOID_RETURN in one place
- The name of the logfile can be '-' to indicate stdin. Before my
patch, the code just checked if the first character is '-'; now it
does a full strcmp(). Probably, all arguments that begin with a -
are already handled somewhere else as flags, but I still think it
is better that the code reflects what it is supposed to do, with as
little dependencies as possible on other parts of the code. If we
one day implement that all command line arguments after -- are
files (as most unix tools do), then we need this.
I also fixed the following in slave.cc:
- next_event() was declared twice, and queue_event was not static but
should be static (not used outside the file).
