There were some places where insufficient locking between
parallel threads could cause invalid memory accesses and
possibly other grief.
This patch adds the missing locking, and moves the locking
into the struct rpl_binlog_state methods to make it easier
to see that proper locking is in place everywhere.
Fixed the following compilation errors and test failures:
- maria SE: "stage_waiting_for_a_resource" wasn't declared w/o PFS
- sql_repl.h: PSI_mutex_key is not available in non-PFS builds
- mysqld.cc: pfs_param is not available in non-PFS builds
- mysqld.cc: init_show_explain_psi_keys() is not available in non-PFS builds
- mysqld.cc: call net_before_header_psi, net_after_header_psi even if PFS
is not available so that thread enters stage_init at proper time. Fixes
sp-threads and a few tests in funcs_1.
- myisam_file_io.opt: added missing loose prefix
The merge is still missing a few hunks related to temporary tables and
InnoDB log file size. The associated code did not seem to exist in
10.0, so the merge of that needs more work. Until this is fixed, there
are a number of test failures as a result.
than an empty host '' is the same as any-host wildcard '%'.
Replace '' with '%' in the parser (for GRANT ... foo@'') and when loading grant tables.
Side effect: one cannot have foo@'' and foo@'%' both at the same time
(but one can have foo@'%' and foo@'%%')
- Made slaves temporary table multi-thread slave safe by adding mutex around save_temporary_table usage.
- rli->save_temporary_tables is the active list of all used temporary tables
- This is copied to THD->temporary_tables when temporary tables are opened and updated when temporary tables are closed
- Added THD->lock_temporary_tables() and THD->unlock_temporary_tables() to simplify this.
- Relay_log_info->sql_thd renamed to Relay_log_info->sql_driver_thd to avoid wrong usage for merged code.
- Added is_part_of_group() to mark functions that are part of the next function. This replaces setting IN_STMT when events are executed.
- Added is_begin(), is_commit() and is_rollback() functions to Query_log_event to simplify code.
- If slave_skip_counter is set run things in single threaded mode. This simplifies code for skipping events.
- Updating state of relay log (IN_STMT and IN_TRANSACTION) is moved to one single function: update_state_of_relay_log()
We can't use OPTION_BEGIN to check for the state anymore as the sql_driver and sql execution threads may be different.
Clear IN_STMT and IN_TRANSACTION in init_relay_log_pos() and Relay_log_info::cleanup_context() to ensure the flags doesn't survive slave restarts
is_in_group() is now independent of state of executed transaction.
- Reset thd->transaction.all.modified_non_trans_table() if we did set it for single table row events.
This was mainly for keeping the flag as documented.
- Changed slave_open_temp_tables to uint32 to be able to use atomic operators on it.
- Relay_log_info::sleep_lock -> rpl_group_info::sleep_lock
- Relay_log_info::sleep_cond -> rpl_group_info::sleep_cond
- Changed some functions to take rpl_group_info instead of Relay_log_info to make them multi-slave safe and to simplify usage
- do_shall_skip()
- continue_group()
- sql_slave_killed()
- next_event()
- Simplifed arguments to io_salve_killed(), check_io_slave_killed() and sql_slave_killed(); No reason to supply THD as this is part of the given structure.
- set_thd_in_use_temporary_tables() removed as in_use is set on usage
- Added information to thd_proc_info() which thread is waiting for slave mutex to exit.
- In open_table() reuse code from find_temporary_table()
Other things:
- More DBUG statements
- Fixed the rpl_incident.test can be run with --debug
- More comments
- Disabled not used function rpl_connect_master()
mysql-test/suite/perfschema/r/all_instances.result:
Moved sleep_lock and sleep_cond to rpl_group_info
mysql-test/suite/rpl/r/rpl_incident.result:
Updated result
mysql-test/suite/rpl/t/rpl_incident-master.opt:
Not needed anymore
mysql-test/suite/rpl/t/rpl_incident.test:
Fixed that test can be run with --debug
sql/handler.cc:
More DBUG_PRINT
sql/log.cc:
More comments
sql/log_event.cc:
Added DBUG statements
do_shall_skip(), continue_group() now takes rpl_group_info param
Use is_begin(), is_commit() and is_rollback() functions instead of inspecting query string
We don't have set slaves temporary tables 'in_use' as this is now done when tables are opened.
Removed IN_STMT flag setting. This is now done in update_state_of_relay_log()
Use IN_TRANSACTION flag to test state of relay log.
In rows_event_stmt_cleanup() reset thd->transaction.all.modified_non_trans_table if we had set this before.
sql/log_event.h:
do_shall_skip(), continue_group() now takes rpl_group_info param
Added is_part_of_group() to mark events that are part of the next event. This replaces setting IN_STMT when events are executed.
Added is_begin(), is_commit() and is_rollback() functions to Query_log_event to simplify code.
sql/log_event_old.cc:
Removed IN_STMT flag setting. This is now done in update_state_of_relay_log()
do_shall_skip(), continue_group() now takes rpl_group_info param
sql/log_event_old.h:
Added is_part_of_group() to mark events that are part of the next event.
do_shall_skip(), continue_group() now takes rpl_group_info param
sql/mysqld.cc:
Changed slave_open_temp_tables to uint32 to be able to use atomic operators on it.
Relay_log_info::sleep_lock -> Rpl_group_info::sleep_lock
Relay_log_info::sleep_cond -> Rpl_group_info::sleep_cond
sql/mysqld.h:
Updated types and names
sql/rpl_gtid.cc:
More DBUG
sql/rpl_parallel.cc:
Updated TODO section
Set thd for event that is execution
Use new is_begin(), is_commit() and is_rollback() functions.
More comments
sql/rpl_rli.cc:
sql_thd -> sql_driver_thd
Relay_log_info::sleep_lock -> rpl_group_info::sleep_lock
Relay_log_info::sleep_cond -> rpl_group_info::sleep_cond
Clear IN_STMT and IN_TRANSACTION in init_relay_log_pos() and Relay_log_info::cleanup_context() to ensure the flags doesn't survive slave restarts.
Reset table->in_use for temporary tables as the table may have been used by another THD.
Use IN_TRANSACTION instead of OPTION_BEGIN to check state of relay log.
Removed IN_STMT flag setting. This is now done in update_state_of_relay_log()
sql/rpl_rli.h:
Changed relay log state flags to bit masks instead of bit positions (most other code we have uses bit masks)
Added IN_TRANSACTION to mark if we are in a BEGIN ... COMMIT section.
save_temporary_tables is now thread safe
Relay_log_info::sleep_lock -> rpl_group_info::sleep_lock
Relay_log_info::sleep_cond -> rpl_group_info::sleep_cond
Relay_log_info->sql_thd renamed to Relay_log_info->sql_driver_thd to avoid wrong usage for merged code
is_in_group() is now independent of state of executed transaction.
sql/slave.cc:
Simplifed arguments to io_salve_killed(), sql_slave_killed() and check_io_slave_killed(); No reason to supply THD as this is part of the given structure.
set_thd_in_use_temporary_tables() removed as in_use is set on usage in sql_base.cc
sql_thd -> sql_driver_thd
More DBUG
Added update_state_of_relay_log() which will calculate the IN_STMT and IN_TRANSACTION state of the relay log after the current element is executed.
If slave_skip_counter is set run things in single threaded mode.
Simplifed arguments to io_salve_killed(), check_io_slave_killed() and sql_slave_killed(); No reason to supply THD as this is part of the given structure.
Added information to thd_proc_info() which thread is waiting for slave mutex to exit.
Disabled not used function rpl_connect_master()
Updated argument to next_event()
sql/sql_base.cc:
Added mutex around usage of slave's temporary tables. The active list is always kept up to date in sql->rgi_slave->save_temporary_tables.
Clear thd->temporary_tables after query (safety)
More DBUG
When using temporary table, set table->in_use to current thd as the THD may be different for slave threads.
Some code is ifdef:ed with REMOVE_AFTER_MERGE_WITH_10 as the given code in 10.0 is not yet in this tree.
In open_table() reuse code from find_temporary_table()
sql/sql_binlog.cc:
rli->sql_thd -> rli->sql_driver_thd
Remove duplicate setting of rgi->rli
sql/sql_class.cc:
Added helper functions rgi_lock_temporary_tables() and rgi_unlock_temporary_tables()
Would have been nicer to have these inline, but there was no easy way to do that
sql/sql_class.h:
Added functions to protect slaves temporary tables
sql/sql_parse.cc:
Added DBUG_PRINT
sql/transaction.cc:
Added comment
Following variables do not require LOCK_open protection anymore:
- table_def_cache (renamed to tdc_hash) is protected by rw-lock
LOCK_tdc_hash;
- table_def_shutdown_in_progress doesn't need LOCK_open protection;
- last_table_id use atomics;
- TABLE_SHARE::ref_count (renamed to TABLE_SHARE::tdc.ref_count)
is protected by TABLE_SHARE::tdc.LOCK_table_share;
- TABLE_SHARE::next, ::prev (renamed to tdc.next and tdc.prev),
oldest_unused_share, end_of_unused_share are protected by
LOCK_unused_shares;
- TABLE_SHARE::m_flush_tickets (renamed to tdc.m_flush_tickets)
is protected by TABLE_SHARE::tdc.LOCK_table_share;
- refresh_version (renamed to tdc_version) use atomics.
* update results
* don't force HA_CREATE_DELAY_KEY_WRITE on all temp tables,
(bad for CREATE ... LIKE) instead imply it in myisam/aria
* restore HA_ERR_TABLE_DEF_CHANGED in archive
* increase the default number of rwlock classes in P_S to fit all our rwlocks
includes:
* remove some remnants of "Bug#14521864: MYSQL 5.1 TO 5.5 BUGS PARTITIONING"
* introduce LOCK_share, now LOCK_ha_data is strictly for engines
* rea_create_table() always creates .par file (even in "frm-only" mode)
* fix a 5.6 bug, temp file leak on dummy ALTER TABLE
revno: 4559
committer: Marc Alff <marc.alff@oracle.com>
branch nick: mysql-5.6-bug14741537-v4
timestamp: Thu 2012-11-08 22:40:31 +0100
message:
Bug#14741537 - MYSQL 5.6, GTID AND PERFORMANCE_SCHEMA
Before this fix, statements using performance_schema tables:
- were marked as unsafe for replication,
- did cause warnings during execution,
- were written to the binlog, either in STATEMENT or ROW format.
When using replication with the new GTID feature,
unsafe warnings are elevated to errors,
which prevents to use both the performance_schema and GTID together.
The root cause of the problem is not related to raising warnings/errors
in some special cases, but deeper: statements involving the performance
schema should not even be written to the binary log in the first place,
because the content of the performance schema tables is 'local' to a server
instance, and may differ greatly between nodes in a replication
topology.
In particular, the DBA should be able to configure (INSERT, UPDATE, DELETE)
or flush (TRUNCATE) performance schema tables on one node,
without affecting other nodes.
This fix introduces the concept of a 'non-replicated' or 'local' table,
and adjusts the replication logic to ignore tables that are not replicated
when deciding if or how to log a statement to the binlog.
Note that while this issue was detected using the performance_schema,
other tables are also affected by the same problem.
This fix define as 'local' the following tables, which are then never
replicated:
- performance_schema.*
- mysql.general_log
- mysql.slow_log
- mysql.slave_relay_log_info
- mysql.slave_master_info
- mysql.slave_worker_info
Existing behavior for information_schema.* is unchanged by this fix,
to limit the scope of changes.
Coding wise, this fix implements the following changes:
1)
Performance schema tables are not using any replication flags,
since performance schema tables are not replicated.
2)
In open_table_from_share(),
tables with no replication capabilities (performance_schema.*),
tables with TABLE_CATEGORY_LOG (logs)
and tables with TABLE_CATEGORY_RPL_INFO (replication)
are marked as non replicated, with TABLE::no_replicate
3)
A new THD member, THD::m_binlog_filter_state,
indicate if the current statement is written to the binlog
(normal cases for most statements), or is to be discarded
(because the statements affects non replicated tables).
4)
In THD::decide_logging_format(), the replication logic
is changed to take into account non replicated tables.
Statements that affect only non replicated tables are
executed normally (no warning or errors), but not written
to the binlog.
Statements that affect (i.e., write to) a replicated table
while also using (i.e., reading from or writing to) a non replicated table
are executed normally in MIXED and ROW binlog format,
and cause a new error in STATEMENT binlog format.
THD::decide_logging_format() uses THD::m_binlog_filter_state
to indicate if a statement is to be ignored, when writing to
the binlog.
5)
In THD::binlog_query(), statements marked as ignored
are not written to the binary log.
6)
For row based replication, the existing test for 'table->no_replicate',
has been moved from binlog_log_row() to check_table_binlog_row_based().
MySQL 5.6.10 performance schema: merge of the test files for the host_cache table
With this patch, all tests perfschema.hostcache_* pass.
There are still some additional unmerged minor changes to these tests
in mysql 5.6.10.
When @@GLOBAL.gtid_strict_mode=1, then certain operations result
in error that would otherwise result in out-of-order binlog files
between servers.
GTID sequence numbers are now allocated independently per domain;
this results in less/no holes in GTID sequences, increasing the
likelyhood that diverging binlogs will be caught by the slave when
GTID strict mode is enabled.
Merge of 10.0-mdev26 feature tree into 10.0-base.
Global transaction ID is prepended to each event group in the binlog.
Slave connect can request to start from GTID position instead of specifying
file name/offset of master binlog. This facilitates easy switch to a new
master.
Slave GTID state is stored in a table mysql.rpl_slave_state, which can be
InnoDB to get crash-safe slave state.
GTID includes a replication domain ID, allowing to keep track of distinct
positions for each of multiple masters.
Adjust full test suite to work with GTID.
Huge patch, mainly due to having to update .result file for all SHOW BINLOG
EVENTS and mysqlbinlog outputs, where the new GTID events pop up.
Everything was painstakingly checked to be still correct and valid .result
file updates.
-Change my_rnd() slightly to make it safer if two threads use it at the same time.
-Avoid some sprintf and strmov in vio.
-Changed thread_count to be automaticly incremented (instead of under LOCK_thread_count).
-Thread cache now uses LOCK_thread_cache instead of LOCK_thread_count.
-Moved delete thd out from LOCK_thread_count.
-Save some mysql_cond_broadcast(&COND_thread_count) calls.
-Removed call to getsockname() during connect.
-Initialize random generator without locks.
Other things:
-Fixed test cases that depends on changes for LOCK_grant
-Added thread_safe_decrement32() and thread_safe_increment32()
-Removed sql_rnd_with_mutex() and get_thread_running()
-In check_table_access() don't lock LOCK_grant if we can resolve the grant with user or db level grants (the normal case).
-Don't use a lock for setting THD->query_id.
-Fixed bug where thd->set_query_id() could be set to same value by multiple threads.
Thanks to Yoshinori Matsunobu for the benchmark of connection speed and to
Domas Mituzas for the inspiration for many of the fixes.
include/violite.h:
Change desc to a string pointer
mysql-test/suite/perfschema/r/all_instances.result:
Added new mutex
mysql-test/suite/perfschema/t/func_mutex.test:
Test for LOCK_system_variables_hash instead of LOCK_grant, as LOCK_grant is not anymore always taken for SELECT's.
mysys/my_gethwaddr.c:
More DBUG
mysys/my_rnd.c:
Change my_rnd() slightly to make it safer if two threads use it at the same time.
sql/event_scheduler.cc:
Changed thread_count to be automically incremented
Moved some safe things out from LOCK_thread_count.
Simplify deleting of THD for running thread.
sql/mysqld.cc:
Changed thread_count to be automically incremented
Thread cache now uses LOCK_thread_cache instead of LOCK_thread_count
Added delete_running_thd()
Moved delete thd out from LOCK_thread_count
More DBUG
Only call mysql_cond_broadcast(&COND_thread_count) if thread_count is 0
Removed call to getsockname() (old not anymore needed check)
sql/mysqld.h:
Removed sql_rnd_with_mutex() (not needed anymore)
Removed not used function get_thread_running()
Added thread_safe_decrement32() and thread_safe_increment32()
Simplified dec_thread_running() and inc_thread_running()
next_query_id() should return the original value for global_query_id, not the next one.
(Bug introduced with MySQL 5.5 merge).
sql/sql_acl.cc:
In check_table_access() don't lock LOCK_grant if we can resolve the grant with user or db level grants (the normal case).
sql/sql_class.cc:
Removed thd_lock_thread_count() and thd_unlock_thread_count()
Initialize random generator without locks
Don't use a lock for setting THD->query_id.
(This is only accessed by thread owning the THD)
sql/sql_class.h:
Don't use a lock for setting THD->query_id.
sql/sql_insert.cc:
Changed thread_count to be automically incremented
sql/sql_parse.cc:
Changed thread_count to be automically incremented
Fixed bug where thd->set_query_id() could be set to same value by multiple threads.
vio/vio.c:
Don't generate 'desc' with sprintf/strmov. Assign a pointer instead.
(Good enough as this is just for debugging)
- Added --verbose to BUILD scripts to get make to write out compile commands.
- Detect if AM_EXTRA_MAKEFLAGS=VERBOSE=1 was used with build scripts.
- Don't write warnings about replication variables when doing bootstrap.
- Fixed that mysql_cond_wait() and mysql_cond_timedwait() will report original source file in case of errors.
- Ignore some compiler warnings
BUILD/FINISH.sh:
Detect if AM_EXTRA_MAKEFLAGS=VERBOSE=1 or --verbose was used
BUILD/SETUP.sh:
Added --verbose to print out the full compile lines
Updated help message
client/mysqltest.cc:
Fixed that one can use 'replace' with cat_file
cmake/configure.pl:
If --verbose is used, get make to write out compile commands
debian/dist/Debian/rules:
Added $AM_EXTRA_MAKEFLAGS to get VERBOSE=1 on buildbot builds
debian/dist/Ubuntu/rules:
Added $AM_EXTRA_MAKEFLAGS to get VERBOSE=1 on buildbot builds
include/my_pthread.h:
Made set_timespec_time_nsec() more portable.
include/mysql/psi/mysql_thread.h:
Fixed that mysql_cond_wait() and mysql_cond_timedwait() will report original source file in case of errors.
mysql-test/suite/innodb/r/auto_increment_dup.result:
Fixed wrong DBUG_SYNC
mysql-test/suite/innodb/t/auto_increment_dup.test:
Fixed wrong DBUG_SYNC
mysql-test/suite/perfschema/include/upgrade_check.inc:
Make test more portable for changes in *.sql files
mysql-test/suite/perfschema/r/pfs_upgrade.result:
Updated test results
mysql-test/valgrind.supp:
Ignore running Aria checkpoint thread
scripts/mysqlaccess.sh:
Changed reference of bugs database
Ensure that also client-server group is read.
sql/handler.cc:
Added missing syncpoint
sql/mysqld.cc:
Don't write warnings about replication variables when doing bootstrap
sql/mysqld.h:
Don't write warnings about replication variables when doing bootstrap
sql/rpl_rli.cc:
Don't write warnings about replication variables when doing bootstrap
sql/sql_insert.cc:
Don't mask SERVER_SHUTDOWN in insert_delayed
This is done to be able to distingush between shutdown and interrupt errors
support-files/compiler_warnings.supp:
Ignore some compiler warnings in xtradb,innobase, oqgraph, yassl, string3.h
Before this fix, configuring the server with:
- performance_schema_events_waits_history_size=0
- performance_schema_events_waits_history_long_size=0
could cause a crash in the performance schema.
These settings to 0 are intended to be valid and supported,
and are in fact working properly in mysql 5.6 and up already.
This fix backports the code fix and test cases from mysql 5.6
to the mysql 5.5 release.
Before this fix, configuring the server with:
- performance_schema_events_waits_history_size=0
- performance_schema_events_waits_history_long_size=0
could cause a crash in the performance schema.
These settings to 0 are intended to be valid and supported,
and are in fact working properly in mysql 5.6 and up already.
This fix backports the code fix and test cases from mysql 5.6
to the mysql 5.5 release.
