NOTE: this is the backport to next-mr.
When using replication, the slave will not log any slow query logs queries
replicated from the master, even if the option "--log-slow-slave-statements"
is set and these take more than "log_query_time" to execute.
In order to log slow queries in replicated thread one needs to set the
--log-slow-slave-statements, so that the SQL thread is initialized with the
correct switch. Although setting this flag correctly configures the slave
thread option to log slow queries, there is an issue with the condition that
is used to check whether to log the slow query or not. When replaying binlog
events the statement contains the SET TIMESTAMP clause which will force the
slow logging condition check to fail. Consequently, the slow query logging will
not take place.
This patch addresses this issue by removing the second condition from the
log_slow_statements as it prevents slow queries to be binlogged and seems
to be deprecated.
Problem: LOGGER::general_log_write() relied on valid "thd" parameter passed
but had inconsistent "if (thd)" check.
Fix: as we always pass a valid "thd" parameter to the method,
redundant check removed.
sql/log.cc:
Fix for bug#47130: misplaced or redundant check for null pointer?
- code clean-up, as we rely on the "thd" parameter in the
LOGGER::general_log_write(), redundant "if (thd)" check removed,
added assert(thd) instead.
with gcc 4.3.2
This patch fixes a number of GCC warnings about variables used
before initialized. A new macro UNINIT_VAR() is introduced for
use in the variable declaration, and LINT_INIT() usage will be
gradually deprecated. (A workaround is used for g++, pending a
patch for a g++ bug.)
GCC warnings for unused results (attribute warn_unused_result)
for a number of system calls (present at least in later
Ubuntus, where the usual void cast trick doesn't work) are
also fixed.
client/mysqlmanager-pwgen.c:
A fix for warn_unused_result, adding fallback to use of
srand()/rand() if /dev/random cannot be used. Also actually
adds calls to rand() in the second branch so that it actually
creates a random password.
binlog
Mixing transactional (T) and non-transactional (N) tables on behalf of a
transaction may lead to inconsistencies among masters and slaves in STATEMENT
mode. The problem stems from the fact that although modifications done to
non-transactional tables on behalf of a transaction become immediately visible
to other connections they do not immediately get to the binary log and therefore
consistency is broken. Although there may be issues in mixing T and M tables in
STATEMENT mode, there are safe combinations that clients find useful.
In this bug, we fix the following issue. Mixing N and T tables in multi-level
(e.g. a statement that fires a trigger) or multi-table table statements (e.g.
update t1, t2...) were not handled correctly. In such cases, it was not possible
to distinguish when a T table was updated if the sequence of changes was N and T.
In a nutshell, just the flag "modified_non_trans_table" was not enough to reflect
that both a N and T tables were changed. To circumvent this issue, we check if an
engine is registered in the handler's list and changed something which means that
a T table was modified.
Check WL 2687 for a full-fledged patch that will make the use of either the MIXED or
ROW modes completely safe.
mysql-test/suite/binlog/r/binlog_row_mix_innodb_myisam.result:
Truncate statement is wrapped in BEGIN/COMMIT.
mysql-test/suite/binlog/r/binlog_stm_mix_innodb_myisam.result:
Truncate statement is wrapped in BEGIN/COMMIT.
binlog
The fix for BUG 43929 introduced a regression issue. In a nutshell, when a
statement that changes a non-transactional table fails, it is written to the
binary log with the error code appended. Unfortunately, after BUG 43929, this
failure was flushing the transactional chace causing mismatch between execution
and logging histories. To fix this issue, we avoid flushing the transactional
cache when a commit or rollback is not issued.
NOTE: This undoes changes by BUG#42829 in sql_class.cc:binlog_query().
I will revert the change in a post-push fix (the binlog filter should
be checked in sql_base.cc:decide_logging_format()).
General overview:
The logic for switching to row format when binlog_format=MIXED had
numerous flaws. The underlying problem was the lack of a consistent
architecture.
General purpose of this changeset:
This changeset introduces an architecture for switching to row format
when binlog_format=MIXED. It enforces the architecture where it has
to. It leaves some bugs to be fixed later. It adds extensive tests to
verify that unsafe statements work as expected and that appropriate
errors are produced by problems with the selection of binlog format.
It was not practical to split this into smaller pieces of work.
Problem 1:
To determine the logging mode, the code has to take several parameters
into account (namely: (1) the value of binlog_format; (2) the
capabilities of the engines; (3) the type of the current statement:
normal, unsafe, or row injection). These parameters may conflict in
several ways, namely:
- binlog_format=STATEMENT for a row injection
- binlog_format=STATEMENT for an unsafe statement
- binlog_format=STATEMENT for an engine only supporting row logging
- binlog_format=ROW for an engine only supporting statement logging
- statement is unsafe and engine does not support row logging
- row injection in a table that does not support statement logging
- statement modifies one table that does not support row logging and
one that does not support statement logging
Several of these conflicts were not detected, or were detected with
an inappropriate error message. The problem of BUG#39934 was that no
appropriate error message was written for the case when an engine
only supporting row logging executed a row injection with
binlog_format=ROW. However, all above cases must be handled.
Fix 1:
Introduce new error codes (sql/share/errmsg.txt). Ensure that all
conditions are detected and handled in decide_logging_format()
Problem 2:
The binlog format shall be determined once per statement, in
decide_logging_format(). It shall not be changed before or after that.
Before decide_logging_format() is called, all information necessary to
determine the logging format must be available. This principle ensures
that all unsafe statements are handled in a consistent way.
However, this principle is not followed:
thd->set_current_stmt_binlog_row_based_if_mixed() is called in several
places, including from code executing UPDATE..LIMIT,
INSERT..SELECT..LIMIT, DELETE..LIMIT, INSERT DELAYED, and
SET @@binlog_format. After Problem 1 was fixed, that caused
inconsistencies where these unsafe statements would not print the
appropriate warnings or errors for some of the conflicts.
Fix 2:
Remove calls to THD::set_current_stmt_binlog_row_based_if_mixed() from
code executed after decide_logging_format(). Compensate by calling the
set_current_stmt_unsafe() at parse time. This way, all unsafe statements
are detected by decide_logging_format().
Problem 3:
INSERT DELAYED is not unsafe: it is logged in statement format even if
binlog_format=MIXED, and no warning is printed even if
binlog_format=STATEMENT. This is BUG#45825.
Fix 3:
Made INSERT DELAYED set itself to unsafe at parse time. This allows
decide_logging_format() to detect that a warning should be printed or
the binlog_format changed.
Problem 4:
LIMIT clause were not marked as unsafe when executed inside stored
functions/triggers/views/prepared statements. This is
BUG#45785.
Fix 4:
Make statements containing the LIMIT clause marked as unsafe at
parse time, instead of at execution time. This allows propagating
unsafe-ness to the view.
mysql-test/extra/rpl_tests/create_recursive_construct.inc:
Added auxiliary file used by binlog_unsafe.test to create and
execute recursive constructs
(functions/procedures/triggers/views/prepared statements).
mysql-test/extra/rpl_tests/rpl_foreign_key.test:
removed unnecessary set @@session.binlog_format
mysql-test/extra/rpl_tests/rpl_insert_delayed.test:
Filter out table id from table map events in binlog listing.
Got rid of $binlog_format_statement.
mysql-test/extra/rpl_tests/rpl_ndb_apply_status.test:
disable warnings around call to unsafe procedure
mysql-test/include/rpl_udf.inc:
Disabled warnings for code that generates warnings
for some binlog formats. That would otherwise cause
inconsistencies in the result file.
mysql-test/r/mysqldump.result:
Views are now unsafe if they contain a LIMIT clause.
That fixed BUG#45831. Due to BUG#45832, a warning is
printed for the CREATE VIEW statement.
mysql-test/r/sp_trans.result:
Unsafe statements in stored procedures did not give a warning if
binlog_format=statement. This is BUG#45824. Now they do, so this
result file gets a new warning.
mysql-test/suite/binlog/r/binlog_multi_engine.result:
Error message changed.
mysql-test/suite/binlog/r/binlog_statement_insert_delayed.result:
INSERT DELAYED didn't generate a warning when binlog_format=STATEMENT.
That was BUG#45825. Now there is a warning, so result file needs to be
updated.
mysql-test/suite/binlog/r/binlog_stm_ps.result:
Changed error message.
mysql-test/suite/binlog/r/binlog_unsafe.result:
updated result file:
- error message changed
- added test for most combinations of unsafe constructs invoked
from recursive constructs
- INSERT DELAYED now gives a warning (because BUG#45826 is fixed)
- INSERT..SELECT..LIMIT now gives a warning from inside recursive
constructs (because BUG#45785 was fixed)
- When a recursive construct (e.g., stored proc or function)
contains more than one statement, at least one of which is
unsafe, then all statements in the recursive construct give
warnings. This is a new bug introduced by this changeset.
It will be addressed in a post-push fix.
mysql-test/suite/binlog/t/binlog_innodb.test:
Changed error code for innodb updates with READ COMMITTED or
READ UNCOMMITTED transaction isolation level and
binlog_format=statement.
mysql-test/suite/binlog/t/binlog_multi_engine.test:
The error code has changed for statements where more than one
engine is involved and one of them is self-logging.
mysql-test/suite/binlog/t/binlog_unsafe-master.opt:
Since binlog_unsafe now tests unsafe-ness of UDF's, we need an extra
flag in the .opt file.
mysql-test/suite/binlog/t/binlog_unsafe.test:
- Clarified comment.
- Rewrote first part of test. Now it tests not only unsafe variables
and functions, but also unsafe-ness due to INSERT..SELECT..LIMIT,
INSERT DELAYED, insert into two autoinc columns, use of UDF's, and
access to log tables in the mysql database.
Also, in addition to functions, procedures, triggers, and prepared
statements, it now also tests views; and it constructs recursive
calls in two levels by combining these recursive constructs.
Part of the logic is in extra/rpl_tests/create_recursive_construct.inc.
- added tests for all special system variables that should not be unsafe.
- added specific tests for BUG#45785 and BUG#45825
mysql-test/suite/rpl/r/rpl_events.result:
updated result file
mysql-test/suite/rpl/r/rpl_extraColmaster_innodb.result:
updated result file
mysql-test/suite/rpl/r/rpl_extraColmaster_myisam.result:
updated result file
mysql-test/suite/rpl/r/rpl_foreign_key_innodb.result:
updated result file
mysql-test/suite/rpl/r/rpl_idempotency.result:
updated result file
mysql-test/suite/rpl/r/rpl_mix_found_rows.result:
Split rpl_found_rows.test into rpl_mix_found_rows.test (a new file) and
rpl_stm_found_rows.test (renamed rpl_found_rows.test). This file equals
the second half of the old rpl_found_rows.result, with the following
modifications:
- minor formatting changes
- additional initialization
mysql-test/suite/rpl/r/rpl_mix_insert_delayed.result:
Moved out code operating in mixed mode from rpl_stm_insert_delayed
(into rpl_mix_insert_delayed) and got rid of explicit setting of
binlog format.
mysql-test/suite/rpl/r/rpl_rbr_to_sbr.result:
updated result file
mysql-test/suite/rpl/r/rpl_row_idempotency.result:
Moved the second half of rpl_idempotency.test, which only
executed in row mode, to rpl_row_idempotency.test. This is
the new result file.
mysql-test/suite/rpl/r/rpl_row_insert_delayed.result:
Got rid of unnecessary explicit setting of binlog format.
mysql-test/suite/rpl/r/rpl_stm_found_rows.result:
Split rpl_found_rows.test into rpl_mix_found_rows.test (a new file) and
rpl_stm_found_rows.test (renamed rpl_found_rows.test). Changes in
this file:
- minor formatting changes
- warning is now issued for unsafe statements inside procedures
(since BUG#45824 is fixed)
- second half of file is moved to rpl_mix_found_rows.result
mysql-test/suite/rpl/r/rpl_stm_insert_delayed.result:
Moved out code operating in mixed mode from rpl_stm_insert_delayed
(into rpl_mix_insert_delayed) and got rid of explicit setting of
binlog format.
mysql-test/suite/rpl/r/rpl_stm_loadfile.result:
error message changed
mysql-test/suite/rpl/r/rpl_temporary_errors.result:
updated result file
mysql-test/suite/rpl/r/rpl_udf.result:
Remove explicit set of binlog format (and triplicate test execution)
and rely on test system executing the test in all binlog formats.
mysql-test/suite/rpl/t/rpl_bug31076.test:
Test is only valid in mixed or row mode since it generates row events.
mysql-test/suite/rpl/t/rpl_events.test:
Removed explicit set of binlog_format and removed duplicate testing.
Instead, we rely on the test system to try all binlog formats.
mysql-test/suite/rpl/t/rpl_extraColmaster_innodb.test:
Removed triplicate testing and instead relying on test system.
Test is only relevant for row format since statement-based replication
cannot handle extra columns on master.
mysql-test/suite/rpl/t/rpl_extraColmaster_myisam.test:
Removed triplicate testing and instead relying on test system.
Test is only relevant for row format since statement-based replication
cannot handle extra columns on master.
mysql-test/suite/rpl/t/rpl_idempotency-slave.opt:
Removed .opt file to avoid server restarts.
mysql-test/suite/rpl/t/rpl_idempotency.test:
- Moved out row-only tests to a new test file, rpl_row_idempotency.test.
rpl_idempotency now only contains tests that execute in all
binlog_formats.
- While I was here, also removed .opt file to avoid server restarts.
The slave_exec_mode is now set inside the test instead.
mysql-test/suite/rpl/t/rpl_mix_found_rows.test:
Split rpl_found_rows.test into rpl_mix_found_rows.test (a new file) and
rpl_stm_found_rows.test (renamed rpl_found_rows.test). This file
contains the second half of the original rpl_found_rows.test with the
follwing changes:
- initialization
- removed SET_BINLOG_FORMAT and added have_binlog_format_mixed.inc
- minor formatting changes
mysql-test/suite/rpl/t/rpl_mix_insert_delayed.test:
Moved out code operating in mixed mode from rpl_stm_insert_delayed
(into rpl_mix_insert_delayed) and got rid of explicit setting of
binlog format.
mysql-test/suite/rpl/t/rpl_rbr_to_sbr.test:
Test cannot execute in statement mode, since we no longer
switch to row format when binlog_format=statement.
Enforced mixed mode throughout the test.
mysql-test/suite/rpl/t/rpl_row_idempotency.test:
Moved the second half of rpl_idempotency.test, which only
executed in row mode, to this new file. We now rely on the
test system to set binlog format.
mysql-test/suite/rpl/t/rpl_row_insert_delayed.test:
- Got rid of unnecessary explicit setting of binlog format.
- extra/rpl_tests/rpl_insert_delayed.test does not need the
$binlog_format_statement variable any more, so that was
removed.
mysql-test/suite/rpl/t/rpl_slave_skip.test:
The test switches binlog_format internally and master generates both
row and statement events. Hence, the slave must be able to log in both
statement and row format. Hence test was changed to only execute in
mixed mode.
mysql-test/suite/rpl/t/rpl_stm_found_rows.test:
Split rpl_found_rows.test into rpl_mix_found_rows.test (a new file) and
rpl_stm_found_rows.test (renamed rpl_found_rows.test). Changes in
this file:
- minor formatting changes
- added have_binlog_format_statement and removed SET BINLOG_FORMAT.
- second half of file is moved to rpl_mix_found_rows.test
- added cleanup code
mysql-test/suite/rpl/t/rpl_stm_insert_delayed.test:
Moved out code operating in mixed mode from rpl_stm_insert_delayed
(into rpl_mix_insert_delayed) and got rid of explicit setting of
binlog format.
mysql-test/suite/rpl/t/rpl_switch_stm_row_mixed.test:
The test switches binlog_format internally and master generates both
row and statement events. Hence, the slave must be able to log in both
statement and row format. Hence test was changed to only execute in
mixed mode on slave.
mysql-test/suite/rpl/t/rpl_temporary_errors.test:
Removed explicit set of binlog format. Instead, the test now only
executes in row mode.
mysql-test/suite/rpl/t/rpl_udf.test:
Remove explicit set of binlog format (and triplicate test execution)
and rely on test system executing the test in all binlog formats.
mysql-test/suite/rpl_ndb/combinations:
Added combinations file for rpl_ndb.
mysql-test/suite/rpl_ndb/r/rpl_ndb_binlog_format_errors.result:
new result file
mysql-test/suite/rpl_ndb/r/rpl_ndb_circular_simplex.result:
updated result file
mysql-test/suite/rpl_ndb/t/rpl_ndb_2innodb.test:
The test needs slave to be able to switch to row mode, so the
test was changed to only execute in mixed and row mode.
mysql-test/suite/rpl_ndb/t/rpl_ndb_2myisam.test:
The test needs slave to be able to switch to row mode, so the
test was changed to only execute in mixed and row mode.
mysql-test/suite/rpl_ndb/t/rpl_ndb_basic.test:
The test needs slave to be able to switch to row mode, so the
test was changed to only execute in mixed and row mode.
mysql-test/suite/rpl_ndb/t/rpl_ndb_binlog_format_errors-master.opt:
new option file
mysql-test/suite/rpl_ndb/t/rpl_ndb_binlog_format_errors-slave.opt:
new option file
mysql-test/suite/rpl_ndb/t/rpl_ndb_binlog_format_errors.test:
New test case to verify all errors and warnings generated by
decide_logging_format.
mysql-test/suite/rpl_ndb/t/rpl_ndb_blob.test:
The test needs slave to be able to switch to row mode, so the
test was changed to only execute in mixed and row mode.
mysql-test/suite/rpl_ndb/t/rpl_ndb_blob2.test:
The test needs slave to be able to switch to row mode, so the
test was changed to only execute in mixed and row mode.
mysql-test/suite/rpl_ndb/t/rpl_ndb_circular.test:
The test needs slave to be able to switch to row mode, so the
test was changed to only execute in mixed and row mode.
mysql-test/suite/rpl_ndb/t/rpl_ndb_circular_simplex.test:
The test needs slave to be able to switch to row mode, so the
test was changed to only execute in mixed and row mode.
While I was here, also made the test clean up after itself.
mysql-test/suite/rpl_ndb/t/rpl_ndb_commit_afterflush.test:
The test needs slave to be able to switch to row mode, so the
test was changed to only execute in mixed and row mode.
mysql-test/suite/rpl_ndb/t/rpl_ndb_ctype_ucs2_def.test:
The test needs slave to be able to switch to row mode, so the
test was changed to only execute in mixed and row mode.
mysql-test/suite/rpl_ndb/t/rpl_ndb_delete_nowhere.test:
The test needs slave to be able to switch to row mode, so the
test was changed to only execute in mixed and row mode.
mysql-test/suite/rpl_ndb/t/rpl_ndb_do_db.test:
The test needs slave to be able to switch to row mode, so the
test was changed to only execute in mixed and row mode.
mysql-test/suite/rpl_ndb/t/rpl_ndb_do_table.test:
The test needs slave to be able to switch to row mode, so the
test was changed to only execute in mixed and row mode.
mysql-test/suite/rpl_ndb/t/rpl_ndb_func003.test:
The test needs slave to be able to switch to row mode, so the
test was changed to only execute in mixed and row mode.
mysql-test/suite/rpl_ndb/t/rpl_ndb_innodb_trans.test:
The test needs slave to be able to switch to row mode, so the
test was changed to only execute in mixed and row mode.
mysql-test/suite/rpl_ndb/t/rpl_ndb_insert_ignore.test:
The test needs slave to be able to switch to row mode, so the
test was changed to only execute in mixed and row mode.
mysql-test/suite/rpl_ndb/t/rpl_ndb_mixed_engines_transactions.test:
The test needs slave to be able to switch to row mode, so the
test was changed to only execute in mixed and row mode.
mysql-test/suite/rpl_ndb/t/rpl_ndb_multi_update3.test:
The test needs slave to be able to switch to row mode, so the
test was changed to only execute in mixed and row mode.
mysql-test/suite/rpl_ndb/t/rpl_ndb_rep_ignore.test:
The test needs slave to be able to switch to row mode, so the
test was changed to only execute in mixed and row mode.
mysql-test/suite/rpl_ndb/t/rpl_ndb_row_001.test:
The test needs slave to be able to switch to row mode, so the
test was changed to only execute in mixed and row mode.
mysql-test/suite/rpl_ndb/t/rpl_ndb_sp003.test:
The test needs slave to be able to switch to row mode, so the
test was changed to only execute in mixed and row mode.
mysql-test/suite/rpl_ndb/t/rpl_ndb_sp006.test:
The test needs slave to be able to switch to row mode, so the
test was changed to only execute in mixed and row mode.
mysql-test/suite/rpl_ndb/t/rpl_ndb_trig004.test:
The test needs slave to be able to switch to row mode, so the
test was changed to only execute in mixed and row mode.
mysql-test/t/partition_innodb_stmt.test:
Changed error code for innodb updates with READ COMMITTED or
READ UNCOMMITTED transaction isolation level and
binlog_format=statement.
sql/event_db_repository.cc:
Use member function to read current_stmt_binlog_row_based.
sql/events.cc:
Use member function to read current_stmt_binlog_row_based.
sql/ha_ndbcluster_binlog.cc:
reset_current_stmt_binlog_row_based() is not a no-op for the ndb_binlog
thread any more. Instead, the ndb_binlog thread now forces row mode both
initially and just after calling mysql_parse. (mysql_parse() is the only
place where reset_current_stmt_binlog_row_based() may be called from
the ndb_binlog thread, so these are the only two places that need to
change.)
sql/ha_partition.cc:
Use member function to read current_stmt_binlog_row_based.
sql/handler.cc:
Use member function to read current_stmt_binlog_row_based.
sql/item_create.cc:
Added DBUG_ENTER to some functions, to be able to trace when
set_stmt_unsafe is called.
sql/log.cc:
Use member function to read current_stmt_binlog_row_based.
sql/log_event.cc:
- Moved logic for changing to row format out of do_apply_event (and into
decide_logging_format).
- Added @todo comment for post-push cleanup.
sql/log_event_old.cc:
Move logic for changing to row format out of do_apply_event (and into
decide_logging_format).
sql/mysql_priv.h:
Make decide_logging_format() a member of the THD class, for two reasons:
- It is natural from an object-oriented perspective.
- decide_logging_format() needs to access private members of THD
(specifically, the new binlog_warning_flags field).
sql/rpl_injector.cc:
Removed call to set_current_stmt_binlog_row_based().
From now on, only decide_logging_fromat is allowed to modify
current_stmt_binlog_row_based. This call is from the ndb_binlog
thread, mostly executing code in ha_ndbcluster_binlog.cc.
This call can be safely removed, because:
- current_stmt_binlog_row_based is initialized for the ndb_binlog
thread's THD object when the THD object is created. So we're
not going to read uninitialized memory.
- The behavior of ndb_binlog thread does not use the state of the
current_stmt_binlog_row_based. It is conceivable that the
ndb_binlog thread would rely on the current_stmt_binlog_format
in two situations:
(1) when it calls mysql_parse;
(2) when it calls THD::binlog_query.
In case (1), it always clears THD::options&OPTION_BIN_LOG (because
run_query() in ha_ndbcluster_binlog.cc is only called with
disable_binlogging = TRUE).
In case (2), it always uses qtype=STMT_QUERY_TYPE.
sql/set_var.cc:
Added @todo comment for post-push cleanup.
sql/share/errmsg.txt:
Added new error messages and clarified ER_BINLOG_UNSAFE_STATEMENT.
sql/sp.cc:
Added DBUG_ENTER, to be able to trace when set_stmt_unsafe is called.
Got rid of MYSQL_QUERY_TYPE: it was equivalent to STMT_QUERY_TYPE.
sql/sp_head.cc:
Use member function to read current_stmt_binlog_row_based.
sql/sp_head.h:
Added DBUG_ENTER, to be able to trace when set_stmt_unsafe is called.
sql/sql_acl.cc:
Got rid of MYSQL_QUERY_TYPE: it was equivalent to STMT_QUERY_TYPE.
sql/sql_base.cc:
- Made decide_logging_format take care of all logic for deciding the
logging format, and for determining the related warnings and errors.
See comment above decide_logging_format for details.
- Made decide_logging_format a member function of THD, since it needs
to access private members of THD and since its purpose is to update
the state of a THD object.
- Added DBUG_ENTER, to be able to trace when set_stmt_unsafe is called.
sql/sql_class.cc:
- Moved logic for determining unsafe warnings away from THD::binlog_query
(and into decide_logging_format()). Now, it works like this:
1. decide_logging_format detects that the current statement shall
produce a warning, if it ever makes it to the binlog
2. decide_logging_format sets a flag of THD::binlog_warning_flags.
3. THD::binlog_query reads the flag. If the flag is set, it generates
a warning.
- Use member function to read current_stmt_binlog_row_based.
sql/sql_class.h:
- Added THD::binlog_warning_flags (see sql_class.cc for explanation).
- Made decide_logging_format() and reset_for_next_command() member
functions of THD (instead of standalone functions). This was needed
for two reasons: (1) the functions need to access the private member
THD::binlog_warning_flags; (2) the purpose of these functions is to
update the staet of a THD object, so from an object-oriented point
of view they should be member functions.
- Encapsulated current_stmt_binlog_row_based, so it is now private and
can only be accessed from a member function. Also changed the
data type to an enumeration instead of a bool.
- Removed MYSQL_QUERY_TYPE, because it was equivalent to
STMT_QUERY_TYPE anyways.
- When reset_current_stmt_binlog_row_based was called from the
ndb_binlog thread, it would behave as a no-op. This special
case has been removed, and the behavior of
reset_current_stmt_binlog_row_based does not depend on which thread
calls it any more. The special case did not serve any purpose,
since the ndb binlog thread did not take the
current_stmt_binlog_row_based flag into account anyways.
sql/sql_delete.cc:
- Moved logic for setting row format for DELETE..LIMIT away from
mysql_prepare_delete.
(Instead, we mark the statement as unsafe at parse time (sql_yacc.yy)
and rely on decide_logging_format() (sql_class.cc) to set row format.)
This is part of the fix for BUG#45831.
- Use member function to read current_stmt_binlog_row_based.
sql/sql_insert.cc:
- Removed unnecessary calls to thd->lex->set_stmt_unsafe() and
thd->set_current_stmt_binlog_row_based_if_mixed() from
handle_delayed_insert(). The calls are unnecessary because they
have already been made; they were made in the constructor of
the `di' object.
- Since decide_logging_format() is now a member function of THD, code
that calls decide_logging_format() had to be updated.
- Added DBUG_ENTER call, to be able to trace when set_stmt_unsafe is
called.
- Moved call to set_stmt_unsafe() for INSERT..SELECT..LIMIT away from
mysql_insert_select_prepare() (and into decide_logging_format).
This is part of the fix for BUG#45831.
- Use member function to read current_stmt_binlog_row_based.
sql/sql_lex.h:
- Added the flag BINLOG_STMT_FLAG_ROW_INJECTION to enum_binlog_stmt_flag.
This was necessary so that a statement can identify itself as a row
injection.
- Added appropriate setter and getter functions for the new flag.
- Added or clarified some comments.
- Added DBUG_ENTER()
sql/sql_load.cc:
Use member function to read current_stmt_binlog_row_based.
sql/sql_parse.cc:
- Made mysql_reset_thd_for_next_command() clear thd->binlog_warning_flags.
- Since thd->binlog_warning_flags is private, it must be set in a
member function of THD. Hence, moved the body of
mysql_reset_thd_for_next_command() to the new member function
THD::reset_thd_for_next_command(), and made
mysql_reset_thd_for_next_command() call
THD::reset_thd_for_next_command().
- Removed confusing comment.
- Use member function to read current_stmt_binlog_row_based.
sql/sql_repl.cc:
Use member function to read current_stmt_binlog_row_based.
sql/sql_table.cc:
Use member function to read current_stmt_binlog_row_based.
sql/sql_udf.cc:
Use member function to read current_stmt_binlog_row_based.
sql/sql_update.cc:
Moved logic for setting row format for UPDATE..LIMIT away from
mysql_prepare_update.
(Instead, we mark the statement as unsafe at parse time (sql_yacc.yy)
and rely on decide_logging_format() (sql_class.cc) to set row format.)
This is part of the fix for BUG#45831.
sql/sql_yacc.yy:
Made INSERT DELAYED, INSERT..SELECT..LIMIT, UPDATE..LIMIT, and
DELETE..LIMIT mark themselves as unsafe at parse time (instead
of at execution time).
This is part of the fixes BUG#45831 and BUG#45825.
storage/example/ha_example.cc:
Made exampledb accept inserts. This was needed by the new test case
rpl_ndb_binlog_format_errors, because it needs an engine that
is statement-only (and accepts inserts).
storage/example/ha_example.h:
Made exampledb a statement-only engine instead of a row-only engine.
No existing test relied exampledb's row-only capabilities. The new
test case rpl_ndb_binlog_format_errors needs an engine that is
statement-only.
storage/innobase/handler/ha_innodb.cc:
- Changed error error code and message given by innodb when
binlog_format=STATEMENT and transaction isolation level is
READ COMMITTED or READ UNCOMMITTED.
- While I was here, also simplified the condition for
checking when to give the error.
The problem: described in the bug report.
The fix:
--increase buffers where it's necessary
(buffers which are used in stxnmov)
--decrease buffer lengths which are used
client/mysql.cc:
--increase buffers where it's necessary
(buffers which are used in stxnmov)
--decrease buffer lengths which are used
as argument for strxnmov function
sql/ha_ndbcluster.cc:
--increase buffers where it's necessary
(buffers which are used in stxnmov)
--decrease buffer lengths which are used
as argument for strxnmov function
sql/ha_ndbcluster_binlog.cc:
--increase buffers where it's necessary
(buffers which are used in stxnmov)
--decrease buffer lengths which are used
as argument for strxnmov function
sql/handler.cc:
--increase buffers where it's necessary
(buffers which are used in stxnmov)
--decrease buffer lengths which are used
as argument for strxnmov function
sql/log.cc:
--increase buffers where it's necessary
(buffers which are used in stxnmov)
--decrease buffer lengths which are used
as argument for strxnmov function
sql/mysqld.cc:
removed unnecessary line
sql/parse_file.cc:
--increase buffers where it's necessary
(buffers which are used in stxnmov)
--decrease buffer lengths which are used
as argument for strxnmov function
sql/sql_acl.cc:
--increase buffers where it's necessary
(buffers which are used in stxnmov)
--decrease buffer lengths which are used
as argument for strxnmov function
sql/sql_base.cc:
--increase buffers where it's necessary
(buffers which are used in stxnmov)
--decrease buffer lengths which are used
as argument for strxnmov function
sql/sql_db.cc:
--increase buffers where it's necessary
(buffers which are used in stxnmov)
--decrease buffer lengths which are used
as argument for strxnmov function
sql/sql_delete.cc:
--increase buffers where it's necessary
(buffers which are used in stxnmov)
--decrease buffer lengths which are used
as argument for strxnmov function
sql/sql_partition.cc:
--increase buffers where it's necessary
(buffers which are used in stxnmov)
--decrease buffer lengths which are used
as argument for strxnmov function
sql/sql_rename.cc:
--increase buffers where it's necessary
(buffers which are used in stxnmov)
--decrease buffer lengths which are used
as argument for strxnmov function
sql/sql_show.cc:
--increase buffers where it's necessary
(buffers which are used in stxnmov)
--decrease buffer lengths which are used
as argument for strxnmov function
sql/sql_table.cc:
--increase buffers where it's necessary
(buffers which are used in stxnmov)
--decrease buffer lengths which are used
as argument for strxnmov function
sql/sql_view.cc:
--increase buffers where it's necessary
(buffers which are used in stxnmov)
--decrease buffer lengths which are used
as argument for strxnmov function
Large transactions and statements may corrupt the binary log if the size of the
cache, which is set by the max_binlog_cache_size, is not enough to store the
the changes.
In a nutshell, to fix the bug, we save the position of the next character in the
cache before starting processing a statement. If there is a problem, we simply
restore the position thus removing any effect of the statement from the cache.
Unfortunately, to avoid corrupting the binary log, we may end up loosing changes
on non-transactional tables if they do not fit in the cache. In such cases, we
store an Incident_log_event in order to stop the slave and alert users that some
changes were not logged.
Precisely, for every non-transactional changes that do not fit into the cache,
we do the following:
a) the statement is *not* logged
b) an incident event is logged after committing/rolling back the transaction,
if any. Note that if a failure happens before writing the incident event to
the binary log, the slave will not stop and the master will not have reported
any error.
c) its respective statement gives an error
For transactional changes that do not fit into the cache, we do the following:
a) the statement is *not* logged
b) its respective statement gives an error
To work properly, this patch requires two additional things. Firstly, callers to
MYSQL_BIN_LOG::write and THD::binlog_query must handle any error returned and
take the appropriate actions such as undoing the effects of a statement. We
already changed some calls in the sql_insert.cc, sql_update.cc and sql_insert.cc
modules but the remaining calls spread all over the code should be handled in
BUG#37148. Secondly, statements must be either classified as DDL or DML because
DDLs that do not get into the cache must generate an incident event since they
cannot be rolled back.
statements missed from general log
A refinement of the test in the previous patch to avoid
using sleep as a means to ensure that timestamps are
added to the log entries.
mysql-test/t/log_tables_debug.test:
New test file. A debug feature is used to ensure that
log entries are prefixed with a timestamp.
sql/log.cc:
A debug feature is implemented to ensure that
log entries are prefixed with a timestamp.
BEGIN/COMMIT/ROLLBACK was subject to replication db rules, and
caused the boundary of a transaction not recognized correctly
when these queries were ignored by the rules.
Fixed the problem by skipping replication db rules for these
statements.
sql/log_event.cc:
Skip checking replication db rules for BEGIN/COMMIT/ROLLBACK statements
Make the caller of Query_log_event, Execute_load_log_event
constructors and THD::binlog_query to provide the error code
instead of having the constructors to figure out the error code.
sql/log_event.cc:
Changed constructors of Query_log_event and Execute_load_log_event to accept the error code argument instead of figuring it out by itself
sql/log_event.h:
Changed constructors of Query_log_event and Execute_load_log_event to accept the error code argument
Bug#44091: libmysqld gets stuck waiting on mutex on initialization
The problem was that libmysqld wasn't enforcing a certain
initialization and deinitialization order for the mysys
library. Another problem was that the global object used
for management of log event handlers (aka LOGGER) wasn't
being prepared for a possible reutilization.
What leads to the hang/crash reported is that a failure
to load the language file triggers a double call of the
cleanup functions, causing an already destroyed mutex to
be used.
The solution is enforce a order on the initialization and
deinitialization of the mysys library within the libmysqld
library and to ensure that the global LOGGER object reset
it's internal state during cleanup.
mysys/my_init.c:
Deinitialize only if initialized already.
sql/log.cc:
Reset state.
When the thread executing a DDL was killed after finished its
execution but before writing the binlog event, the error code in
the binlog event could be set wrongly to ER_SERVER_SHUTDOWN or
ER_QUERY_INTERRUPTED.
This patch fixed the problem by ignoring the kill status when
constructing the event for DDL statements.
This patch also included the following changes in order to
provide the test case.
1) modified mysqltest to support variable for connection command
2) modified mysql-test-run.pl, add new variable MYSQL_SLAVE to
run mysql client against the slave mysqld.
The problem here seem to be that when mysql
is redirecting stderr to a file, stderr becomes
buffered, whereas it is unbuffered by definition.
The solution is to unbuffer it by setting buffer
to null.
sql/log.cc:
use setbuf(stderr, NULL) to set the buffer to null.
sql/mysqld.cc:
use setbuf(stderr, NULL) to set the buffer to null.
- Remove bothersome warning messages. This change focuses on the warnings
that are covered by the ignore file: support-files/compiler_warnings.supp.
- Strings are guaranteed to be max uint in length
conflicts:
Text conflict in client/mysqltest.cc
Text conflict in mysql-test/include/wait_until_connected_again.inc
Text conflict in mysql-test/lib/mtr_report.pm
Text conflict in mysql-test/mysql-test-run.pl
Text conflict in mysql-test/r/events_bugs.result
Text conflict in mysql-test/r/log_state.result
Text conflict in mysql-test/r/myisam_data_pointer_size_func.result
Text conflict in mysql-test/r/mysqlcheck.result
Text conflict in mysql-test/r/query_cache.result
Text conflict in mysql-test/r/status.result
Text conflict in mysql-test/suite/binlog/r/binlog_index.result
Text conflict in mysql-test/suite/binlog/r/binlog_innodb.result
Text conflict in mysql-test/suite/rpl/r/rpl_packet.result
Text conflict in mysql-test/suite/rpl/t/rpl_packet.test
Text conflict in mysql-test/t/disabled.def
Text conflict in mysql-test/t/events_bugs.test
Text conflict in mysql-test/t/log_state.test
Text conflict in mysql-test/t/myisam_data_pointer_size_func.test
Text conflict in mysql-test/t/mysqlcheck.test
Text conflict in mysql-test/t/query_cache.test
Text conflict in mysql-test/t/rpl_init_slave_func.test
Text conflict in mysql-test/t/status.test
BUG#39325 Server crash inside MYSQL_LOG::purge_first_log halts replicaiton
The patch reverses the order of the purging and updating events for log and
relay-log.info/index files respectively.
This solves the problem of having holes caused by crashes happening between updating
info/index files and purging logs.
This patch also contains an aditional test case for testing the crashing before purge logs.
NOTE1: This is a combined patch for BUG#38826 and BUG#39325. This patch is based on
bugteam tree and takes into account reviewers suggestions.
NOTE2: Merge from 5.0-bugteam
BUG#39325 Server crash inside MYSQL_LOG::purge_first_log halts replicaiton
The patch reverses the order of the purging and updating events for log and relay-log.info/index files respectively.
This solves the problem of having holes caused by crashes happening between updating info/index files and purging logs.
NOTE: This is a combined patch for BUG#38826 and BUG#39325. This patch is based on bugteam tree and takes into account reviewers suggestions.
after rollback on master
When starting a transaction with a statement containing changes
to both transactional tables and non-transactional tables, the
statement is considered as non-transactional and is therefore
written directly to the binary log. This behaviour was present
in 5.0, and has propagated to 5.1.
If a trigger containing a change of a non-transactional table is
added to a transactional table, any changes to the transactional
table is "tainted" as non-transactional.
This patch solves the problem by removing the existing "hack" that
allows non-transactional statements appearing first in a transaction
to be written directly to the binary log. Instead, anything inside
a transaction is treaded as part of the transaction and not written
to the binary log until the transaction is committed.
mysql-test/suite/rpl/t/rpl_row_create_table.test:
Removing positions from SHOW BINLOG EVENTS and using
reset_master_and_slave to start on a fresh binary log each time.
mysql-test/suite/rpl/t/rpl_slave_skip.test:
Adding explicit commit in AUTOCOMMIT=0 to make test work correctly.
mysql-test/suite/rpl/t/rpl_trigger.test:
Adding test case for BUG#40116.
sql/log.cc:
Changing commit logic in binlog_commit() to only commit when
committing a real transaction or committing a punch transaction.
A transaction could result in having an extra event after a query that
errored e.g because of a dup key. Such a query is rolled back in
innodb, as specified, but has not been in binlog.
It appeares that the binlog engine did not always register for a query
(statement) because the previous query had not reset at its statement
commit time. Because of that fact there was no roll-back to the
trx_data->before_stmt_pos position and a the pending event of the
errorred query could become flushed to the binlog file.
Fixed with deploying the reset of trx_data->before_stmt_pos at the end
of the query processing.
mysql-test/suite/binlog/r/binlog_innodb_row.result:
a new test file to contain tests dealing with binlogging innodb with
the row-based format.
mysql-test/suite/binlog/t/binlog_innodb_row.test:
a new test file to contain tests dealing with binlogging innodb with
the row-based format.
sql/log.cc:
Flushing the pending row event at binlog_end_trans() is moved down to the commit
branch.
Resetting of trx_data->before_stmt_pos to the uninitialized value at the end of the
statement is implemented in binlog_commit() and binlog_rollback().
Asserting emptiness the transaction cache after reset() and the uninitilized value for the statement's savepoint binlog position.
Debug builds of MySQL 5.1, 6.0 with Sun Studio 12 broke because of
use of gcc specific feature.
The fix is to replace __FUNCTION__ with the corresponding character string
Debug builds of MySQL 5.1, 6.0 with Sun Studio 12 broke because of
use of gcc specific feature.
The fix is to replace __FUNCTION__ with the corresponding character
string
The failure was caused by executing a CREATE-SELECT statement that creates a
table in another database than the current one. In row-based logging, the
CREATE statement was written to the binary log without the database, hence
creating the table in the wrong database, causing the following inserts to
fail since the table didn't exist in the given database.
Fixed the bug by adding a parameter to store_create_info() that will make
the function print the database name before the table name and used that
in the calls that write the CREATE statement to the binary log. The database
name is only printed if it is different than the currently selected database.
The output of SHOW CREATE TABLE has not changed and is still printed without
the database name.
mysql-test/suite/rpl/t/rpl_row_create_table.test:
Added test to check that CREATE-SELECT into another database than the
current one replicates.
sql/sql_insert.cc:
Adding parameter to calls to store_create_info().
sql/sql_show.cc:
Adding parameter to calls to store_create_info().
Extending store_create_info() with parameter 'show_database' that will cause
the database to be written before the table name.
sql/sql_show.h:
Adding parameter to call to store_create_info() to tell if the database should be shown or not.
sql/sql_table.cc:
Adding parameter to calls to store_create_info().
"Trigger fired multiple times leads to gaps in auto_increment sequence".
The bug was that if a trigger fired multiple times inside a top
statement (for example top-statement is a multi-row INSERT,
and trigger is ON INSERT), and that trigger inserted into an auto_increment
column, then gaps could be observed in the auto_increment sequence,
even if there were no other users of the database (no concurrency).
It was wrong usage of THD::auto_inc_intervals_in_cur_stmt_for_binlog.
Note that the fix changes "class handler", I'll tell the Storage Engine API team.
mysql-test/r/trigger-trans.result:
result; before the bugfix, the sequence was 1,2,4,6,8,10,12...
mysql-test/t/trigger-trans.test:
test for BUG#31612
sql/handler.cc:
See revision comment of handler.h.
As THD::auto_inc_intervals_in_cur_stmt_for_binlog is cumulative
over all trigger invokations by the top statement, the
second invokation of the trigger arrived in handler::update_auto_increment()
with already one interval in
THD::auto_inc_intervals_in_cur_stmt_for_binlog. The method thus
believed it had already reserved one interval for that invokation,
thus reserved a twice larger interval (heuristic when we don't know
how large the interval should be: we grow by powers of two). InnoDB
thus increased its internal per-table auto_increment counter by 2
while only one row was to be inserted. Hence a gap in the sequence.
The fix is to use the new handler::auto_inc_intervals_count.
Note that the trigger's statement knows how many rows it is going
to insert, but provides estimation_rows_to_insert == 0 (see comments
in sql_insert.cc why triggers don't call handler::ha_start_bulk_insert()).
* removing white space at end of line
* we don't need to maintain THD::auto_inc_intervals_in_cur_stmt_for_binlog
if no binlogging or if row-based binlogging. Using auto_inc_intervals_count in
the heuristic makes the heuristic independent of binary logging, which is good.
sql/handler.h:
THD::auto_inc_intervals_in_cur_stmt_for_binlog served
- for binlogging
- as a heuristic when we have no estimation of how many records the
statement will insert.
But the first goal needs to be cumulative over all statements which
form a binlog event, while the second one needs to be attached to each
statement. THD::auto_inc_intervals_in_cur_stmt_for_binlog is cumulative,
leading to BUG#31612. So we introduce handler::auto_inc_intervals_count
for the second goal. See the revision comment of handler.cc.
A smaller issue was that, even when the binlog event was only one
statement (no triggers, no stored functions),
THD::auto_inc_intervals_in_cur_stmt.nb_elements() could be lower than
the number of reserved intervals (fooling the heuristic), because its
append() method collapses two contiguous intervals in one.
Note that as auto_inc_intervals_count is in class 'handler' and not
in class 'THD', it does not need to be handled in
THD::reset|restore_sub_statement_state().
sql/log.cc:
Comment is wrong: if auto_increment is second, in handler::update_auto_increment()
'append' is false and so auto_inc_intervals_in_cur_stmt_for_binlog
is empty, we do not come here.
sql/sql_class.h:
comment
Rotate event is automatically generated and written when rotating binary
log or relay log. Rotate events for relay logs are usually ignored by slave
SQL thread becuase they have the same server id as that of the slave.
However, if --replicate-same-server-id is enabled, rotate event
for relay log would be treated as if it's a rotate event from master, and
would be executed by slave to update the rli->group_master_log_name and
rli->group_master_log_pos to a wrong value and cause the MASTER_POS_WAIT
function to fail and return NULL.
This patch fixed this problem by setting a flag bit (LOG_EVENT_RELAY_LOG_F)
in the event to tell the SQL thread to ignore these Rotate events generated
for relay logs.
This patch also added another binlog event flag bit (LOG_EVENT_ARTIFICIAL_F)
to distinquish faked events, the method used before this was by checking if
log_pos was zero.
sql/log.h:
Add a member to MYSQL_BIN_LOG to distinguish binary log from relay log.
sql/log_event.cc:
Change artificial_event member to LOG_EVENT_ARTIFICIAL_F flag
If LOG_EVENT_RELAY_LOG_F is set in the event flags for a rotate event, ignore it when updating position
Refactored the code in Rotate_log_event::do_update_pos
sql/log_event.h:
Add LOG_EVENT_RELAY_LOG_F flag to Log_event flags
Add RELAY_LOG flag to Rotate_log_event flags
sql/sql_repl.cc:
Set LOG_EVENT_ARTIFICIAL_F for fake rotate events
The assertion indicates that some data was left in the transaction
cache when the server was shut down, which means that a previous
statement did not commit or rollback correctly.
What happened was that a bug in the rollback of a transactional
table caused the transaction cache to be emptied, but not reset.
The error can be triggered by having a failing UPDATE or INSERT,
on a transactional table, causing an implicit rollback.
Fixed by always flushing the pending event to reset the state
properly.
mysql-test/extra/rpl_tests/rpl_row_basic.test:
Testing that a failed update (that writes some rows to the
transaction cache) does not cause the transaction cache to
hold on to the data or forget to reset the transaction cache.
sql/log.cc:
Added call to remove pending event when the transaction cache
is emptied instead of written to binary log. The call will also
clear the outstanding table map count so that the cache is not
left it in a state of "empty but not reset".
Added function MYSQL_BIN_LOG::remove_pending_rows_event().
sql/log.h:
Added function MYSQL_BIN_LOG::remove_pending_rows_event().
sql/sql_class.cc:
Adding function THD::binlog_remove_pending_rows_event().
sql/sql_class.h:
Adding function THD::binlog_remove_pending_rows_event().
include/my_global.h:
Moving YESNO() macro here from log.cc (it prints either "yes" or "no" depending on a boolean value).
sql/log.cc:
Moving YESNO() function to my_global.h.
sql/sql_class.cc:
Adding default case to printout function to avoid warning.
Only defining function for debug builds since it isn't used
in non-debug build (hence produce a warning).
sql/sql_class.h:
Printing yes/no answer instead of memory address since the
case produces an error/warning on valgrind platform.
The failure was caused by executing a CREATE-SELECT statement that creates a
table in another database than the current one. In row-based logging, the
CREATE statement was written to the binary log without the database, hence
creating the table in the wrong database, causing the following inserts to
fail since the table didn't exist in the given database.
Fixed the bug by adding a parameter to store_create_info() that will make
the function print the database name before the table name and used that
in the calls that write the CREATE statement to the binary log. The database
name is only printed if it is different than the currently selected database.
The output of SHOW CREATE TABLE has not changed and is still printed without
the database name.
mysql-test/suite/rpl/r/rpl_row_create_table.result:
Result file change.
mysql-test/suite/rpl/t/rpl_row_create_table.test:
Added test to check that CREATE-SELECT into another database than the
current one replicates.
sql/sql_insert.cc:
Adding parameter to calls to store_create_info().
sql/sql_show.cc:
Adding parameter to calls to store_create_info().
Extending store_create_info() with parameter 'show_database' that will cause
the database to be written before the table name.
sql/sql_show.h:
Adding parameter to call to store_create_info() to tell if the database should be shown or not.
sql/sql_table.cc:
Adding parameter to calls to store_create_info().
If the server failed to expired log files during start up it could crash.
sql/log.cc:
Added predicate to protect againt cases when current_thd might be NULL.
