BUG#31665 sync_binlog should cause relay logs to be synchronized
NOTE: Backporting the patch to next-mr.
Add sync_relay_log option to server, this option works for relay log
the same as option sync_binlog for binlog. This option also synchronize
master info to disk when set to non-zero value.
Original patches from Sinisa and Mark, with some modifications
BUG#31665 sync_binlog should cause relay logs to be synchronized
NOTE: Backporting the patch to next-mr.
Add sync_relay_log option to server, this option works for relay log
the same as option sync_binlog for binlog. This option also synchronize
master info to disk when set to non-zero value.
Original patches from Sinisa and Mark, with some modifications
beyond unsigned long.
BUG#44779: binlog.binlog_max_extension may be causing failure on
next test in PB
NOTE1: this is the backport to next-mr.
NOTE2: already includes patch for BUG#44779.
Binlog file extensions would turn into negative numbers once the
variable used to hold the value reached maximum for signed
long. Consequently, incrementing value to the next (negative) number
would lead to .000000 extension, causing the server to fail.
This patch addresses this issue by not allowing negative extensions
and by returning an error on find_uniq_filename, when the limit is
reached. Additionally, warnings are printed to the error log when the
limit is approaching. FLUSH LOGS will also report warnings to the
user, if the extension number has reached the limit. The limit has been
set to 0x7FFFFFFF as the maximum.
mysql-test/suite/binlog/t/binlog_max_extension.test:
Test case added that checks the maximum available number for
binlog extensions.
sql/log.cc:
Changes to find_uniq_filename and test_if_number.
sql/log.h:
Added macros with values for MAX_LOG_UNIQUE_FN_EXT and
LOG_WARN_UNIQUE_FN_EXT_LEFT, as suggested in review.
beyond unsigned long.
BUG#44779: binlog.binlog_max_extension may be causing failure on
next test in PB
NOTE1: this is the backport to next-mr.
NOTE2: already includes patch for BUG#44779.
Binlog file extensions would turn into negative numbers once the
variable used to hold the value reached maximum for signed
long. Consequently, incrementing value to the next (negative) number
would lead to .000000 extension, causing the server to fail.
This patch addresses this issue by not allowing negative extensions
and by returning an error on find_uniq_filename, when the limit is
reached. Additionally, warnings are printed to the error log when the
limit is approaching. FLUSH LOGS will also report warnings to the
user, if the extension number has reached the limit. The limit has been
set to 0x7FFFFFFF as the maximum.
Large transactions and statements may corrupt the binary log if the size of the
cache, which is set by the max_binlog_cache_size, is not enough to store the
the changes.
In a nutshell, to fix the bug, we save the position of the next character in the
cache before starting processing a statement. If there is a problem, we simply
restore the position thus removing any effect of the statement from the cache.
Unfortunately, to avoid corrupting the binary log, we may end up loosing changes
on non-transactional tables if they do not fit in the cache. In such cases, we
store an Incident_log_event in order to stop the slave and alert users that some
changes were not logged.
Precisely, for every non-transactional changes that do not fit into the cache,
we do the following:
a) the statement is *not* logged
b) an incident event is logged after committing/rolling back the transaction,
if any. Note that if a failure happens before writing the incident event to
the binary log, the slave will not stop and the master will not have reported
any error.
c) its respective statement gives an error
For transactional changes that do not fit into the cache, we do the following:
a) the statement is *not* logged
b) its respective statement gives an error
To work properly, this patch requires two additional things. Firstly, callers to
MYSQL_BIN_LOG::write and THD::binlog_query must handle any error returned and
take the appropriate actions such as undoing the effects of a statement. We
already changed some calls in the sql_insert.cc, sql_update.cc and sql_insert.cc
modules but the remaining calls spread all over the code should be handled in
BUG#37148. Secondly, statements must be either classified as DDL or DML because
DDLs that do not get into the cache must generate an incident event since they
cannot be rolled back.
Large transactions and statements may corrupt the binary log if the size of the
cache, which is set by the max_binlog_cache_size, is not enough to store the
the changes.
In a nutshell, to fix the bug, we save the position of the next character in the
cache before starting processing a statement. If there is a problem, we simply
restore the position thus removing any effect of the statement from the cache.
Unfortunately, to avoid corrupting the binary log, we may end up loosing changes
on non-transactional tables if they do not fit in the cache. In such cases, we
store an Incident_log_event in order to stop the slave and alert users that some
changes were not logged.
Precisely, for every non-transactional changes that do not fit into the cache,
we do the following:
a) the statement is *not* logged
b) an incident event is logged after committing/rolling back the transaction,
if any. Note that if a failure happens before writing the incident event to
the binary log, the slave will not stop and the master will not have reported
any error.
c) its respective statement gives an error
For transactional changes that do not fit into the cache, we do the following:
a) the statement is *not* logged
b) its respective statement gives an error
To work properly, this patch requires two additional things. Firstly, callers to
MYSQL_BIN_LOG::write and THD::binlog_query must handle any error returned and
take the appropriate actions such as undoing the effects of a statement. We
already changed some calls in the sql_insert.cc, sql_update.cc and sql_insert.cc
modules but the remaining calls spread all over the code should be handled in
BUG#37148. Secondly, statements must be either classified as DDL or DML because
DDLs that do not get into the cache must generate an incident event since they
cannot be rolled back.
Make the caller of Query_log_event, Execute_load_log_event
constructors and THD::binlog_query to provide the error code
instead of having the constructors to figure out the error code.
sql/log_event.cc:
Changed constructors of Query_log_event and Execute_load_log_event to accept the error code argument instead of figuring it out by itself
sql/log_event.h:
Changed constructors of Query_log_event and Execute_load_log_event to accept the error code argument
Make the caller of Query_log_event, Execute_load_log_event
constructors and THD::binlog_query to provide the error code
instead of having the constructors to figure out the error code.
conflicts:
Text conflict in client/mysqltest.cc
Text conflict in mysql-test/include/wait_until_connected_again.inc
Text conflict in mysql-test/lib/mtr_report.pm
Text conflict in mysql-test/mysql-test-run.pl
Text conflict in mysql-test/r/events_bugs.result
Text conflict in mysql-test/r/log_state.result
Text conflict in mysql-test/r/myisam_data_pointer_size_func.result
Text conflict in mysql-test/r/mysqlcheck.result
Text conflict in mysql-test/r/query_cache.result
Text conflict in mysql-test/r/status.result
Text conflict in mysql-test/suite/binlog/r/binlog_index.result
Text conflict in mysql-test/suite/binlog/r/binlog_innodb.result
Text conflict in mysql-test/suite/rpl/r/rpl_packet.result
Text conflict in mysql-test/suite/rpl/t/rpl_packet.test
Text conflict in mysql-test/t/disabled.def
Text conflict in mysql-test/t/events_bugs.test
Text conflict in mysql-test/t/log_state.test
Text conflict in mysql-test/t/myisam_data_pointer_size_func.test
Text conflict in mysql-test/t/mysqlcheck.test
Text conflict in mysql-test/t/query_cache.test
Text conflict in mysql-test/t/rpl_init_slave_func.test
Text conflict in mysql-test/t/status.test
conflicts:
Text conflict in client/mysqltest.cc
Text conflict in mysql-test/include/wait_until_connected_again.inc
Text conflict in mysql-test/lib/mtr_report.pm
Text conflict in mysql-test/mysql-test-run.pl
Text conflict in mysql-test/r/events_bugs.result
Text conflict in mysql-test/r/log_state.result
Text conflict in mysql-test/r/myisam_data_pointer_size_func.result
Text conflict in mysql-test/r/mysqlcheck.result
Text conflict in mysql-test/r/query_cache.result
Text conflict in mysql-test/r/status.result
Text conflict in mysql-test/suite/binlog/r/binlog_index.result
Text conflict in mysql-test/suite/binlog/r/binlog_innodb.result
Text conflict in mysql-test/suite/rpl/r/rpl_packet.result
Text conflict in mysql-test/suite/rpl/t/rpl_packet.test
Text conflict in mysql-test/t/disabled.def
Text conflict in mysql-test/t/events_bugs.test
Text conflict in mysql-test/t/log_state.test
Text conflict in mysql-test/t/myisam_data_pointer_size_func.test
Text conflict in mysql-test/t/mysqlcheck.test
Text conflict in mysql-test/t/query_cache.test
Text conflict in mysql-test/t/rpl_init_slave_func.test
Text conflict in mysql-test/t/status.test
BUG#39325 Server crash inside MYSQL_LOG::purge_first_log halts replicaiton
The patch reverses the order of the purging and updating events for log and
relay-log.info/index files respectively.
This solves the problem of having holes caused by crashes happening between updating
info/index files and purging logs.
This patch also contains an aditional test case for testing the crashing before purge logs.
NOTE1: This is a combined patch for BUG#38826 and BUG#39325. This patch is based on
bugteam tree and takes into account reviewers suggestions.
NOTE2: Merge from 5.0-bugteam
BUG#39325 Server crash inside MYSQL_LOG::purge_first_log halts replicaiton
The patch reverses the order of the purging and updating events for log and
relay-log.info/index files respectively.
This solves the problem of having holes caused by crashes happening between updating
info/index files and purging logs.
This patch also contains an aditional test case for testing the crashing before purge logs.
NOTE1: This is a combined patch for BUG#38826 and BUG#39325. This patch is based on
bugteam tree and takes into account reviewers suggestions.
NOTE2: Merge from 5.0-bugteam
Rotate event is automatically generated and written when rotating binary
log or relay log. Rotate events for relay logs are usually ignored by slave
SQL thread becuase they have the same server id as that of the slave.
However, if --replicate-same-server-id is enabled, rotate event
for relay log would be treated as if it's a rotate event from master, and
would be executed by slave to update the rli->group_master_log_name and
rli->group_master_log_pos to a wrong value and cause the MASTER_POS_WAIT
function to fail and return NULL.
This patch fixed this problem by setting a flag bit (LOG_EVENT_RELAY_LOG_F)
in the event to tell the SQL thread to ignore these Rotate events generated
for relay logs.
This patch also added another binlog event flag bit (LOG_EVENT_ARTIFICIAL_F)
to distinquish faked events, the method used before this was by checking if
log_pos was zero.
sql/log.h:
Add a member to MYSQL_BIN_LOG to distinguish binary log from relay log.
sql/log_event.cc:
Change artificial_event member to LOG_EVENT_ARTIFICIAL_F flag
If LOG_EVENT_RELAY_LOG_F is set in the event flags for a rotate event, ignore it when updating position
Refactored the code in Rotate_log_event::do_update_pos
sql/log_event.h:
Add LOG_EVENT_RELAY_LOG_F flag to Log_event flags
Add RELAY_LOG flag to Rotate_log_event flags
sql/sql_repl.cc:
Set LOG_EVENT_ARTIFICIAL_F for fake rotate events
Rotate event is automatically generated and written when rotating binary
log or relay log. Rotate events for relay logs are usually ignored by slave
SQL thread becuase they have the same server id as that of the slave.
However, if --replicate-same-server-id is enabled, rotate event
for relay log would be treated as if it's a rotate event from master, and
would be executed by slave to update the rli->group_master_log_name and
rli->group_master_log_pos to a wrong value and cause the MASTER_POS_WAIT
function to fail and return NULL.
This patch fixed this problem by setting a flag bit (LOG_EVENT_RELAY_LOG_F)
in the event to tell the SQL thread to ignore these Rotate events generated
for relay logs.
This patch also added another binlog event flag bit (LOG_EVENT_ARTIFICIAL_F)
to distinquish faked events, the method used before this was by checking if
log_pos was zero.
The assertion indicates that some data was left in the transaction
cache when the server was shut down, which means that a previous
statement did not commit or rollback correctly.
What happened was that a bug in the rollback of a transactional
table caused the transaction cache to be emptied, but not reset.
The error can be triggered by having a failing UPDATE or INSERT,
on a transactional table, causing an implicit rollback.
Fixed by always flushing the pending event to reset the state
properly.
mysql-test/extra/rpl_tests/rpl_row_basic.test:
Testing that a failed update (that writes some rows to the
transaction cache) does not cause the transaction cache to
hold on to the data or forget to reset the transaction cache.
sql/log.cc:
Added call to remove pending event when the transaction cache
is emptied instead of written to binary log. The call will also
clear the outstanding table map count so that the cache is not
left it in a state of "empty but not reset".
Added function MYSQL_BIN_LOG::remove_pending_rows_event().
sql/log.h:
Added function MYSQL_BIN_LOG::remove_pending_rows_event().
sql/sql_class.cc:
Adding function THD::binlog_remove_pending_rows_event().
sql/sql_class.h:
Adding function THD::binlog_remove_pending_rows_event().
The assertion indicates that some data was left in the transaction
cache when the server was shut down, which means that a previous
statement did not commit or rollback correctly.
What happened was that a bug in the rollback of a transactional
table caused the transaction cache to be emptied, but not reset.
The error can be triggered by having a failing UPDATE or INSERT,
on a transactional table, causing an implicit rollback.
Fixed by always flushing the pending event to reset the state
properly.
into mysql.com:/home/bar/mysql-work/mysql-5.1-new-rpl-merge
client/mysql.cc:
Auto merged
mysql-test/r/ctype_euckr.result:
Auto merged
mysql-test/r/ctype_uca.result:
Auto merged
mysql-test/r/ctype_utf8.result:
Auto merged
mysql-test/r/func_regexp.result:
Auto merged
mysql-test/suite/rpl/r/rpl_bug31076.result:
Auto merged
mysql-test/t/ctype_utf8.test:
Auto merged
mysql-test/t/func_regexp.test:
Auto merged
mysql-test/t/partition.test:
Auto merged
sql/field.cc:
Auto merged
sql/field.h:
Auto merged
sql/handler.cc:
Auto merged
sql/item_cmpfunc.cc:
Auto merged
sql/item_cmpfunc.h:
Auto merged
sql/item_xmlfunc.cc:
Auto merged
sql/log.cc:
Auto merged
sql/log.h:
Auto merged
sql/log_event.cc:
Auto merged
sql/mysqld.cc:
Auto merged
sql/sp_head.cc:
Auto merged
sql/sql_class.cc:
Auto merged
sql/sql_class.h:
Auto merged
sql/sql_delete.cc:
Auto merged
sql/sql_insert.cc:
Auto merged
sql/sql_parse.cc:
Auto merged
sql/sql_repl.cc:
Auto merged
sql/sql_select.cc:
Auto merged
sql/sql_show.cc:
Auto merged
sql/sql_table.cc:
Auto merged
sql/sql_update.cc:
Auto merged
strings/ctype-euc_kr.c:
Auto merged
sql/sql_yacc.yy:
Reverting Rafal's changes: TRANSACTIONAL_SYM was removed in a mistake.
mysql-test/r/ctype_ucs.result:
After merge fix
mysql-test/suite/rpl/t/rpl_bug31076.test:
After merge fix.
mysql-test/t/ctype_ucs.test:
After megre fix
The general log write function (general_log_print) uses printf style
arguments which need to be pre-processed, meaning that the all arguments
are copied to a single buffer and the problem is that the buffer size is
constant (1022 characters) but queries can be much larger then this.
The solution is to introduce a new log write function that accepts a
buffer and it's length as arguments. The function is to be used when
a formatted output is not required, which is the case for almost all
query write-to-log calls.
This is a incompatible change with respect to the log format of prepared
statements.
mysql-test/r/log_tables.result:
Add test case result for Bug#21557
mysql-test/t/log_tables.test:
Add test case for Bug#21557
sql/log.cc:
Introduce the logger function general_log_write which is similar to
general_log_print but accepts a single buffer and the buffer length.
The function doesn't perform any formatting and sends the buffer
directly to the underlying log handlers.
sql/log.h:
Introduce the logger function general_log_write.
sql/log_event.cc:
Pass the query buffer directly to the logger function, formatting
is not required on this case.
sql/mysql_priv.h:
Prototype for the logger function general_log_write.
sql/sp_head.cc:
Pass the query buffer directly to the logger function, formatting
is not required on this case.
sql/sql_parse.cc:
Pass the buffer directly to the logger function when formatting
is not required.
sql/sql_prepare.cc:
Don't log the statement id, it avoids making a extra copy of the query
and the query is not truncated anymore if it exceeds the limit.
The general log write function (general_log_print) uses printf style
arguments which need to be pre-processed, meaning that the all arguments
are copied to a single buffer and the problem is that the buffer size is
constant (1022 characters) but queries can be much larger then this.
The solution is to introduce a new log write function that accepts a
buffer and it's length as arguments. The function is to be used when
a formatted output is not required, which is the case for almost all
query write-to-log calls.
This is a incompatible change with respect to the log format of prepared
statements.
into kindahl-laptop.dnsalias.net:/home/bk/fix-mysql-5.1-rpl
sql/ha_ndbcluster.cc:
Auto merged
sql/log.cc:
Auto merged
sql/log.h:
Auto merged
sql/slave.cc:
Auto merged
sql/sql_class.h:
Auto merged
sql/log_event.cc:
Manual merge.
(and be more friendly to Doxygen by removing unnecessary typedefs).
sql/log.cc:
Renaming struct st_relay_log_info to class Relay_log_info.
sql/log.h:
Renaming struct st_relay_log_info to class Relay_log_info.
sql/log_event.cc:
Renaming RELAY_LOG_INFO to Relay_log_info.
sql/log_event.h:
Renaming struct st_relay_log_info to class Relay_log_info.
Renaming RELAY_LOG_INFO to Relay_log_info.
Removing typedef RELAY_LOG_INFO.
sql/log_event_old.cc:
Renaming RELAY_LOG_INFO to Relay_log_info.
sql/log_event_old.h:
Renaming RELAY_LOG_INFO to Relay_log_info.
sql/rpl_mi.h:
Renaming RELAY_LOG_INFO to Relay_log_info.
sql/rpl_record.cc:
Renaming RELAY_LOG_INFO to Relay_log_info.
sql/rpl_record.h:
Renaming RELAY_LOG_INFO to Relay_log_info.
sql/rpl_record_old.cc:
Renaming RELAY_LOG_INFO to Relay_log_info.
sql/rpl_record_old.h:
Renaming RELAY_LOG_INFO to Relay_log_info.
sql/rpl_rli.cc:
Renaming struct st_relay_log_info to class Relay_log_info.
Renaming RELAY_LOG_INFO to Relay_log_info.
sql/rpl_rli.h:
Renaming struct st_relay_log_info to class Relay_log_info.
Renaming RELAY_LOG_INFO to Relay_log_info.
Removing typedef RELAY_LOG_INFO.
sql/rpl_utility.cc:
Renaming RELAY_LOG_INFO to Relay_log_info.
sql/rpl_utility.h:
Renaming struct st_relay_log_info to class Relay_log_info.
Renaming RELAY_LOG_INFO to Relay_log_info.
Removing typedef RELAY_LOG_INFO.
sql/slave.cc:
Renaming RELAY_LOG_INFO to Relay_log_info.
sql/slave.h:
Renaming struct st_relay_log_info to class Relay_log_info.
Renaming RELAY_LOG_INFO to Relay_log_info.
Removing typedef RELAY_LOG_INFO.
sql/sql_binlog.cc:
Renaming RELAY_LOG_INFO to Relay_log_info.
sql/sql_class.h:
Renaming struct st_relay_log_info to class Relay_log_info.
Renaming RELAY_LOG_INFO to Relay_log_info.
Removing typedef RELAY_LOG_INFO.
sql/sql_repl.cc:
Renaming RELAY_LOG_INFO to Relay_log_info.
into mysql.com:/home/my/mysql-5.1
client/client_priv.h:
Auto merged
client/mysqldump.c:
Auto merged
client/mysqlslap.c:
Auto merged
mysql-test/mysql-test-run.pl:
Auto merged
mysql-test/lib/mtr_report.pl:
Auto merged
mysql-test/t/disabled.def:
Auto merged
sql/event_data_objects.cc:
Auto merged
sql/event_queue.cc:
Auto merged
sql/ha_partition.cc:
Auto merged
sql/lock.cc:
Auto merged
sql/log.cc:
Auto merged
sql/log.h:
Auto merged
sql/mysql_priv.h:
Auto merged
sql/set_var.cc:
Auto merged
sql/slave.cc:
Auto merged
sql/sql_class.cc:
Auto merged
sql/sql_class.h:
Auto merged
sql/sql_insert.cc:
Auto merged
sql/sql_parse.cc:
Auto merged
sql/sql_plugin.cc:
Auto merged
sql/sql_select.cc:
Auto merged
sql-common/client.c:
Auto merged
sql/sql_show.cc:
Auto merged
sql/sql_table.cc:
Auto merged
sql/sql_cache.cc:
Manual merge
Removed comment about bug in old code (not relevant)
--long-query-time is now given in seconds with microseconds as decimals
--min_examined_row_limit added for slow query log
long_query_time user variable is now double with 6 decimals
Added functions to get time in microseconds
Added faster time() functions for system that has gethrtime() (Solaris)
We now do less time() calls.
Added field->in_read_set() and field->in_write_set() for easier field manipulation by handlers
set_var.cc and my_getopt() can now handle DOUBLE variables.
All time() calls changed to my_time()
my_time() now does retry's if time() call fails.
Added debug function for stopping in mysql_admin_table() when tables are locked
Some trivial function and struct variable renames to avoid merge errors.
Fixed compiler warnings
Initialization of some time variables on windows moved to my_init()
include/my_getopt.h:
Added support for double arguments
include/my_sys.h:
Fixed wrong type to packfrm()
Added new my_time functions
include/mysql/plugin.h:
Added support for DOUBLE
libmysql/CMakeLists.txt:
Added new time functions
libmysql/Makefile.shared:
Added new time functions
mysql-test/r/variables.result:
Testing of long_query_time
mysql-test/t/variables.test:
Testing of long_query_time
mysys/charset.c:
Fixed compiler warnings
mysys/default_modify.c:
Fixed compiler warnings
mysys/hash.c:
Fixed compiler warnings
mysys/mf_getdate.c:
Use my_time()
mysys/mf_iocache2.c:
Fixed compiler warnings
mysys/mf_pack.c:
Fixed compiler warnings
mysys/mf_path.c:
Fixed compiler warnings
mysys/my_append.c:
Fixed compiler warnings
mysys/my_compress.c:
Fixed compiler warnings
mysys/my_copy.c:
Fixed compiler warnings
mysys/my_gethwaddr.c:
Fixed compiler warnings
mysys/my_getopt.c:
Added support for double arguments
mysys/my_getsystime.c:
Added functions to get time in microseconds.
Added faster time() functions for system that has gethrtime() (Solaris)
Moved windows initialization code to my_init()
mysys/my_init.c:
Added initializing of variables needed for windows time functions
mysys/my_static.c:
Added variables needed for windows time functions
mysys/my_static.h:
Added variables needed for windows time functions
mysys/my_thr_init.c:
Added THR_LOCK_time, used for faster my_time()
mysys/mysys_priv.h:
Added THR_LOCK_time, used for faster my_time()
mysys/thr_alarm.c:
time() -> my_time()
sql/event_data_objects.cc:
end_time() -> set_current_time()
sql/event_queue.cc:
end_time() -> set_current_time()
sql/event_scheduler.cc:
Fixed compiler warnings
sql/field.h:
Added field->in_read_set() and field->in_write_set() for easier field manipulation by handlers
sql/item.h:
Added decimal to Item_float(double)
sql/item_cmpfunc.h:
Added decimal to Item_float(double)
sql/item_timefunc.cc:
time() -> my_time()
sql/item_xmlfunc.cc:
Fixed compiler warning
sql/lock.cc:
lock_time() -> set_time_after_lock()
sql/log.cc:
Timing in slow query log to file is now done in microseconds
Changed some while() loops to for() loops.
Fixed indentation
time() -> my_time()
sql/log.h:
Slow query logging is now done based on microseconds
sql/log_event.cc:
time() -> my_time()
Fixed arguments to new Item_float()
sql/mysql_priv.h:
Fixed compiler warnings
Added opt_log_slow_slave_statements
sql/mysqld.cc:
Added --log_slow_slave_statements and --min_examined_row_limit
--long-query-time now takes a double argument with microsecond resolution
Don't write shutdown message when using --help
Removed not needed \n
Thread create time and connect time is now done in microseconds
time() -> my_time()
Avoid some time() calls
sql/net_serv.cc:
Fixed compiler warnings
sql/parse_file.cc:
time() -> my_time()
sql/set_var.cc:
Added support for DOUBLE variables
Added support for variables that are given in seconds with microsecond resolution
sql/set_var.h:
Added support for variables that are given in seconds with microsecond resolution
sql/slave.cc:
Allow logging of slave queries to slow query log if 'opt_log_slow_slave_statements' is given
time() -> my_time()
sql/sql_cache.h:
Fixed compiler warning()
sql/sql_class.cc:
Initialize new THD variables
sql/sql_class.h:
long_query_time is now in microseconds
Added min_examined_row_limit
Reordered some THD elements for higher efficency
Added timers in microseconds (connect_utime, thr_create_utime, start_utime and utime_after_lock)
Start of query is now recorded both in seconds and in microseconds.
Following renames was made for more clarity and avoid merge problems from earlier versions:
connect_time -> connect_utime
thr_create_time -> thr_create_utime
end_time() -> set_current_time()
lock_time() -> set_time_after_lock()
Added THD::start_utime, which is start of query in microseconds from some arbitary time
Added function THD::current_utime()
Removed safe_time() as retry's are handled in my_time()
sql/sql_connect.cc:
User resources are now using microsecond resolution
sql/sql_insert.cc:
end_time() -> set_current_time()
sql-common/client.c:
time() -> my_time()
sql/sql_parse.cc:
Testing if we should print to slow_query_log() is now done with microsecond precission.
If min_examined_row_limit is given, only log queries to slow query log that has examined more rows than this.
sql/sql_select.cc:
Simplify code now that Item_float() takes decimals as argument
sql/sql_show.cc:
time() -> my_time()
Added support for SYS_DOUBLE
sql/sql_table.cc:
Added debug function for stopping in mysql_admin_table() when tables are locked
sql/structs.h:
intime -> reset_utime
--long-query-time is now given in seconds with microseconds as decimals
--min_examined_row_limit added for slow query log
long_query_time user variable is now double with 6 decimals
Added functions to get time in microseconds
Added faster time() functions for system that has gethrtime() (Solaris)
We now do less time() calls.
Added field->in_read_set() and field->in_write_set() for easier field manipulation by handlers
set_var.cc and my_getopt() can now handle DOUBLE variables.
All time() calls changed to my_time()
my_time() now does retry's if time() call fails.
Added debug function for stopping in mysql_admin_table() when tables are locked
Some trivial function and struct variable renames to avoid merge errors.
Fixed compiler warnings
Initialization of some time variables on windows moved to my_init()
Bug#25422 (Hang with log tables)
Bug 17876 (Truncating mysql.slow_log in a SP after using cursor locks the
thread)
Bug 23044 (Warnings on flush of a log table)
Bug 29129 (Resetting general_log while the GLOBAL READ LOCK is set causes
a deadlock)
Prior to this fix, the server would hang when performing concurrent
ALTER TABLE or TRUNCATE TABLE statements against the LOG TABLES,
which are mysql.general_log and mysql.slow_log.
The root cause traces to the following code:
in sql_base.cc, open_table()
if (table->in_use != thd)
{
/* wait_for_condition will unlock LOCK_open for us */
wait_for_condition(thd, &LOCK_open, &COND_refresh);
}
The problem with this code is that the current implementation of the
LOGGER creates 'fake' THD objects, like
- Log_to_csv_event_handler::general_log_thd
- Log_to_csv_event_handler::slow_log_thd
which are not associated to a real thread running in the server,
so that waiting for these non-existing threads to release table locks
cause the dead lock.
In general, the design of Log_to_csv_event_handler does not fit into the
general architecture of the server, so that the concept of general_log_thd
and slow_log_thd has to be abandoned:
- this implementation does not work with table locking
- it will not work with commands like SHOW PROCESSLIST
- having the log tables always opened does not integrate well with DDL
operations / FLUSH TABLES / SET GLOBAL READ_ONLY
With this patch, the fundamental design of the LOGGER has been changed to:
- always open and close a log table when writing a log
- remove totally the usage of fake THD objects
- clarify how locking of log tables is implemented in general.
See WL#3984 for details related to the new locking design.
Additional changes (misc bugs exposed and fixed):
1)
mysqldump which would ignore some tables in dump_all_tables_in_db(),
but forget to ignore the same in dump_all_views_in_db().
2)
mysqldump would also issue an empty "LOCK TABLE" command when all the tables
to lock are to be ignored (numrows == 0), instead of not issuing the query.
3)
Internal errors handlers could intercept errors but not warnings
(see sql_error.cc).
4)
Implementing a nested call to open tables, for the performance schema tables,
exposed an existing bug in remove_table_from_cache(), which would perform:
in_use->some_tables_deleted=1;
against another thread, without any consideration about thread locking.
This call inside remove_table_from_cache() was not required anyway,
since calling mysql_lock_abort() takes care of aborting -- cleanly -- threads
that might hold a lock on a table.
This line (in_use->some_tables_deleted=1) has been removed.
sql/handler.cc:
Moved logic for system / log tables in the SQL layer.
sql/handler.h:
Moved logic for system / log tables in the SQL layer.
sql/lock.cc:
Revised locking of log tables
sql/log.cc:
Major cleanup: changed how log tables are locked / written to.
sql/log.h:
Major cleanup: changed how log tables are locked / written to.
sql/mysql_priv.h:
performance schema helpers
sql/slave.cc:
open_ltable() lock flags
sql/sp.cc:
open_ltable() lock flags
sql/sql_acl.cc:
open_ltable() lock flags
sql/sql_class.h:
performance schema helpers
sql/sql_delete.cc:
log tables cleanup in TRUNCATE
sql/sql_error.cc:
Internal handlers can also intercept warnings
sql/sql_insert.cc:
open_ltable() lock flags
sql/sql_parse.cc:
performance schema helpers
sql/sql_plugin.cc:
open_ltable() lock flags
sql/sql_rename.cc:
log tables cleanup in RENAME
sql/sql_servers.cc:
open_ltable() lock flags
sql/sql_show.cc:
Move INFORMATION_SCHEMA_NAME to table.cc
sql/sql_table.cc:
log tables cleanup (admin operations, ALTER TABLE)
sql/sql_udf.cc:
open_ltable() lock flags
sql/table.cc:
Implemented TABLE_CATEGORY.
sql/share/errmsg.txt:
Changed the wording and name of ER_CANT_READ_LOCK_LOG_TABLE
sql/table.h:
Implemented TABLE_CATEGORY.
storage/csv/ha_tina.cc:
Moved logic for system / log tables in the SQL layer.
storage/csv/ha_tina.h:
Moved logic for system / log tables in the SQL layer.
storage/myisam/ha_myisam.cc:
Moved logic for system / log tables in the SQL layer.
storage/myisam/ha_myisam.h:
Moved logic for system / log tables in the SQL layer.
client/mysqldump.c:
Don't lock tables in the ignore list.
Don't issue empty LOCK TABLES queries.
sql/sql_base.cc:
log tables cleanup
performance schema helpers
mysql-test/r/ps.result:
Adjust test results
mysql-test/r/show_check.result:
Adjust test results
mysql-test/r/status.result:
Adjust test results
mysql-test/t/log_state.test:
Added tests for Bug#29129
mysql-test/t/ps.test:
Make the test output deterministic
mysql-test/t/show_check.test:
Make the test output deterministic
mysql-test/r/log_state.result:
Changed the default location of the log output to LOG_FILE,
for backward compatibility with MySQL 5.0
---
Adjust test results
mysql-test/r/log_tables.result:
cleanup for -ps-protocol
mysql-test/t/log_tables.test:
cleanup for -ps-protocol
sql/set_var.cc:
Changed the default location of the log output to LOG_FILE,
for backward compatibility with MySQL 5.0
---
log tables cleanup
Bug#25422 (Hang with log tables)
Bug 17876 (Truncating mysql.slow_log in a SP after using cursor locks the
thread)
Bug 23044 (Warnings on flush of a log table)
Bug 29129 (Resetting general_log while the GLOBAL READ LOCK is set causes
a deadlock)
Prior to this fix, the server would hang when performing concurrent
ALTER TABLE or TRUNCATE TABLE statements against the LOG TABLES,
which are mysql.general_log and mysql.slow_log.
The root cause traces to the following code:
in sql_base.cc, open_table()
if (table->in_use != thd)
{
/* wait_for_condition will unlock LOCK_open for us */
wait_for_condition(thd, &LOCK_open, &COND_refresh);
}
The problem with this code is that the current implementation of the
LOGGER creates 'fake' THD objects, like
- Log_to_csv_event_handler::general_log_thd
- Log_to_csv_event_handler::slow_log_thd
which are not associated to a real thread running in the server,
so that waiting for these non-existing threads to release table locks
cause the dead lock.
In general, the design of Log_to_csv_event_handler does not fit into the
general architecture of the server, so that the concept of general_log_thd
and slow_log_thd has to be abandoned:
- this implementation does not work with table locking
- it will not work with commands like SHOW PROCESSLIST
- having the log tables always opened does not integrate well with DDL
operations / FLUSH TABLES / SET GLOBAL READ_ONLY
With this patch, the fundamental design of the LOGGER has been changed to:
- always open and close a log table when writing a log
- remove totally the usage of fake THD objects
- clarify how locking of log tables is implemented in general.
See WL#3984 for details related to the new locking design.
Additional changes (misc bugs exposed and fixed):
1)
mysqldump which would ignore some tables in dump_all_tables_in_db(),
but forget to ignore the same in dump_all_views_in_db().
2)
mysqldump would also issue an empty "LOCK TABLE" command when all the tables
to lock are to be ignored (numrows == 0), instead of not issuing the query.
3)
Internal errors handlers could intercept errors but not warnings
(see sql_error.cc).
4)
Implementing a nested call to open tables, for the performance schema tables,
exposed an existing bug in remove_table_from_cache(), which would perform:
in_use->some_tables_deleted=1;
against another thread, without any consideration about thread locking.
This call inside remove_table_from_cache() was not required anyway,
since calling mysql_lock_abort() takes care of aborting -- cleanly -- threads
that might hold a lock on a table.
This line (in_use->some_tables_deleted=1) has been removed.
The log tables are by nature PERFORMANCE_SCHEMA tables,
which should not be affected by SET GLOBAL READ_ONLY or FLUSH TABLES
WITH READ LOCK.
The implementation of FLUSH TABLES WITH READ LOCK already ignored log tables.
Now with this patch, the implementation of LOCK TABLE also ignore a
global read lock for log tables, which was the missing symmetry.
mysql-test/r/flush.result:
Fix for bug #26380: LOCK TABLES + FLUSH LOGS causes deadlock
- test result.
mysql-test/t/flush.test:
Fix for bug #26380: LOCK TABLES + FLUSH LOGS causes deadlock
- test case.
sql/lock.cc:
Fix for bug #26380: LOCK TABLES + FLUSH LOGS causes deadlock
- logger.is_privileged_thread() used.
sql/log.h:
Fix for bug #26380: LOCK TABLES + FLUSH LOGS causes deadlock
- LOGGER::is_privileged_thread() introduced that returns TRUE if a given thread
is either a general_log or a slow_log or a privileged thread.
sql/sql_base.cc:
Fix for bug #26380: LOCK TABLES + FLUSH LOGS causes deadlock
- pass MYSQL_LOCK_IGNORE_GLOBAL_READ_LOCK to the mysql_lock_tables() in case of
call from a logger in order not to honor the GLOBAL READ LOCK
and to avoid possible deadlocks.
The log tables are by nature PERFORMANCE_SCHEMA tables,
which should not be affected by SET GLOBAL READ_ONLY or FLUSH TABLES
WITH READ LOCK.
The implementation of FLUSH TABLES WITH READ LOCK already ignored log tables.
Now with this patch, the implementation of LOCK TABLE also ignore a
global read lock for log tables, which was the missing symmetry.
provide the writable datadir for mysqld --help (for lower_case test)
sql/log.h, sql/mysqld.cc
make default binlog_format value to be 0 (for a hack of using
GET_STR for enum to continue to work)
mysql-test/mysql-test-run.pl:
provide the writable datadir for mysqld --help (for lower_case test)
sql/log.h:
make default binlog_format value to be 0 (for a hack of using
GET_STR for enum to continue to work)
sql/mysqld.cc:
make default binlog_format value to be 0 (for a hack of using
GET_STR for enum to continue to work)
provide the writable datadir for mysqld --help (for lower_case test)
sql/log.h, sql/mysqld.cc
make default binlog_format value to be 0 (for a hack of using
GET_STR for enum to continue to work)
There was hanging at binlog_commit by a thread executing autocommit query.
The hang appeared to be due to an overly condtion for early return
from binlog_commit introduced by bug#20265 fix.
Fixed with reverting the logic back to 5.0 version.
mysql-test/extra/binlog_tests/binlog.test:
added a regression test
mysql-test/r/binlog_row_binlog.result:
results changed
mysql-test/r/binlog_stm_binlog.result:
results changed
sql/log.cc:
Removing `all' conjuction arg from early return condition.
There is nothing to execute by transaction if trx_data is empty.
The work for rotate_and_purge is delayed till TC_LOG::unlog
(same as in 5.0 code)
sql/log.h:
singed because there are assert on positiveness
There was hanging at binlog_commit by a thread executing autocommit query.
The hang appeared to be due to an overly condtion for early return
from binlog_commit introduced by bug#20265 fix.
Fixed with reverting the logic back to 5.0 version.
into mysql_cab_desk.:C:/source/C++/mysql-5.1-new-rpl
sql/item_func.cc:
Auto merged
sql/log.cc:
Auto merged
sql/log.h:
Auto merged
sql/sp_head.cc:
Auto merged
sql/sql_class.cc:
Auto merged
SF/Triggers in SBR mode."
BUG#14914 "SP: Uses of session variables in routines are not always replicated"
BUG#25167 "Dupl. usage of user-variables in trigger/function is not replicated
correctly"
User-defined variables used inside of stored functions/triggers in
statements which did not update tables directly were not replicated.
We also had problems with replication of user-defined variables which
were used in triggers (or stored functions called from table-updating
statements) more than once.
This patch addresses the first issue by enabling logging of all
references to user-defined variables in triggers/stored functions
and not only references from table-updating statements.
The second issue stemmed from the fact that for user-defined
variables used from triggers or stored functions called from
table-updating statements we were writing binlog events for each
reference instead of only one event for the first reference.
This problem is already solved for stored functions called from
non-updating statements with help of "event unioning" mechanism.
So the patch simply extends this mechanism to the case affected.
It also fixes small problem in this mechanism which caused wrong
logging of references to user-variables in cases when non-updating
statement called several stored functions which used the same
variable and some of these function calls were omitted from binlog
as they were not updating any tables.
mysql-test/r/rpl_user_variables.result:
BUG#20141 - User-defined variables are not replicated properly for
SF/Triggers in SBR mode.
This patch adds the correct results for execution of the added test
procedures to the rpl_user_variables test.
mysql-test/t/rpl_user_variables.test:
BUG#20141 - User-defined variables are not replicated properly for
SF/Triggers in SBR mode.
This patch adds additional tests to the rpl_user_variables test that test
many of the different ways user-defined variables can be required to be
replicated.
sql/item_func.cc:
BUG#20141 - User-defined variables are not replicated properly for SF/Triggers
in SBR mode.
To properly log accesses to user-defined variables from stored
functions/triggers, the get_var_with_binlog() method needs to log references
to such variables even from non-table-updating statements within them.
sql/log.cc:
BUG#20141 - User-defined variables are not replicated properly for SF/Triggers
in SBR mode.
This patch modifies the start_union_events method to accept the query id from
a parameter. This allows callers to set the query_id to the id of the sub
statement such as a trigger or stored function. Which permits the code to
identify when a user defined variable has been used by the statement and this
already present in THD::user_var_event.
Note:
The changes to sql_class.cc, sp_head.cc, and log.cc are designed to allow the
proper replication of access to user-defined variables under a special test
case (the last case shown in rpl_user_variables.test).
sql/log.h:
BUG#20141 - User-defined variables are not replicated properly for
SF/Triggers in SBR mode.
This patch adds the query_id parameter to the calls to
mysql_bin_log.start_union_events().
sql/sp_head.cc:
BUG#20141 - User-defined variables are not replicated properly for
SF/Triggers in SBR mode.
This patch modifies the code to allow for cases where events for function calls
have a separate union for each event and thus cannot use the query_id of the
caller as the start of the union. Thus, we use an artifically created query_id
to set the start of the events.
Note:
The changes to sql_class.cc, sp_head.cc, and log.cc are designed to allow the
proper replication of access to user-defined variables under a special test
case (the last case shown in rpl_user_variables.test).
sql/sql_class.cc:
BUG#20141 - User-defined variables are not replicated properly for
SF/Triggers in SBR mode.
This patch adds the query_id parameter to the calls to
mysql_bin_log.start_union_events().
Note:
The changes to sql_class.cc, sp_head.cc, and log.cc are designed to allow
the proper replication of access to user-defined variables under a special
test case (the last case shown in rpl_user_variables.test).
SF/Triggers in SBR mode."
BUG#14914 "SP: Uses of session variables in routines are not always replicated"
BUG#25167 "Dupl. usage of user-variables in trigger/function is not replicated
correctly"
User-defined variables used inside of stored functions/triggers in
statements which did not update tables directly were not replicated.
We also had problems with replication of user-defined variables which
were used in triggers (or stored functions called from table-updating
statements) more than once.
This patch addresses the first issue by enabling logging of all
references to user-defined variables in triggers/stored functions
and not only references from table-updating statements.
The second issue stemmed from the fact that for user-defined
variables used from triggers or stored functions called from
table-updating statements we were writing binlog events for each
reference instead of only one event for the first reference.
This problem is already solved for stored functions called from
non-updating statements with help of "event unioning" mechanism.
So the patch simply extends this mechanism to the case affected.
It also fixes small problem in this mechanism which caused wrong
logging of references to user-variables in cases when non-updating
statement called several stored functions which used the same
variable and some of these function calls were omitted from binlog
as they were not updating any tables.
