Made allocation of memory for statistical data in a table share to be thread safe.
This memory is now allocated in a special MEM_ROOT that is created for each
table share.
The problem is a shift operation that is not 64-bit safe.
The consequence is that used tables information for a join with 32 tables
or more will be incorrect.
Fixed by adding a type cast in Item_sum::update_used_tables().
Also used the opportunity to fix some other potential bugs by adding an
explicit type-cast to an integer in a left-shift operation.
Some of them were quite harmless, but was fixed in order to get the same
signed-ness as the other operand of the operation it was used in.
sql/item_cmpfunc.cc
Adjusted signed-ness for some integers in left-shift.
sql/item_subselect.cc
Added type-cast to nesting_map (which is a 32/64 bit type, so
potential bug for deeply nested queries).
sql/item_sum.cc
Added type-cast to nesting_map (32/64-bit type) and table_map
(64-bit type).
sql/opt_range.cc
Added type-cast to ulonglong (which is a 64-bit type).
sql/sql_base.cc
Added type-cast to nesting_map (which is a 32/64-bit type).
sql/sql_select.cc
Added type-cast to nesting_map (32/64-bit type) and key_part_map
(64-bit type).
sql/strfunc.cc
Changed type-cast from longlong to ulonglong, to preserve signed-ness.
that introduced engine independent persistent statistics.
In particular:
- added an enumeration type for possible values of the system
variable use_stat_tables
- renamed KEY::real_rec_per_key to KEY::actual_rec_per_key
- optimized the collection of statistical data for any primary
key defined only on one column.
If a query referenced some system statistical tables, but not all of them,
then executing an ANALYZE command simultaneously with this query could
lead to a deadlock.
The fix prohibited reading statistics from system statistical tables
for such queries.
Removed the function unlock_tables_n_open_system_tables_for_write()
as not used anymore.
Performed some minor refactoring of the code in sql_statistics.cc.
ROBUST AGAINST BUGS IN CALLERS".
Both MDL subsystems and Table Definition Cache code assume
that callers ensure that names of objects passed to them are
not longer than NAME_LEN bytes. Unfortunately due to bugs in
callers this assumption might be broken in some cases. As
result we get nasty bugs causing buffer overruns when we
construct MDL key or TDC key from object names.
This patch makes TDC code more robust against such bugs by
ensuring that we always checking size of result buffer when
constructing TDC keys. This doesn't free its callers from
ensuring that both db and table names are shorter than
NAME_LEN bytes. But at least this steps prevents buffer
overruns in case of bug in caller, replacing them with less
harmful behavior.
This is 5.1-only version of patch.
This patch introduces new version of create_table_def_key()
helper function which constructs TDC key without risk of
result buffer overrun. Places in code that construct TDC keys
were changed to use this function.
Also changed rm_temporary_table() and open_new_frm() functions
to avoid use of "unsafe" strmov() and strxmov() functions and
use safer strnxmov() instead.
ROBUST AGAINST BUGS IN CALLERS".
Both MDL subsystems and Table Definition Cache code assume
that callers ensure that names of objects passed to them are
not longer than NAME_LEN bytes. Unfortunately due to bugs in
callers this assumption might be broken in some cases. As
result we get nasty bugs causing buffer overruns when we
construct MDL key or TDC key from object names.
This patch makes MDL and TDC code more robust against such
bugs by ensuring that we always checking size of result
buffer when constructing MDL and TDC keys. This doesn't
free its callers from ensuring that both db and table names
are shorter than NAME_LEN bytes. But at least these steps
prevents buffer overruns in case of bug in caller, replacing
them with less harmful behavior.
This is 5.5-only version of patch.
Changed code of MDL_key::mdl_key_init() to take into account
size of buffer for the key.
Introduced new version of create_table_def_key() helper function
which constructs TDC key without risk of result buffer overrun.
Places in code that construct TDC keys were changed to use this
function.
Also changed rm_temporary_table() and open_new_frm() functions
to avoid use of "unsafe" strmov() and strxmov() functions and
use safer strnxmov() instead.
mysql-test/r/create.result:
Updated test results
mysql-test/t/create.test:
Updated test
sql/sql_base.cc:
Use push_internal_handler/pop_internal_handler to avoid errors & warnings instead of clear_error
Give a warnings instead of an error for CREATE TABLE IF EXISTS
sql/sql_parse.cc:
Check if we failed because of table exists (can only happen from create)
sql/sql_table.cc:
Check if we failed because of table exists (can only happen from create)
mysql-test/r/create.result:
Added test case to show that CREATE TABLE also is not waiting if table exists.
mysql-test/t/create.test:
Added test case to show that CREATE TABLE also is not waiting if table exists.
sql/sql_base.cc:
Clear also warnings from acquire_locks if we retry.
- Added option to check_if_table_exists() to quickly check if table exists (either SHARE or .FRM)
- Extended lock_table_names() to not wait for meta data locks if CREATE IF NOT EXISTS is used.
mysql-test/r/create.result:
New test case
mysql-test/t/create.test:
New test case
sql/sql_base.cc:
Added option to check_if_table_exists() to quickly check if table exists (either SHARE or .FRM)
Extended lock_table_names() to not wait for meta data locks if CREATE IF NOT EXISTS is used.
sql/sql_base.h:
Updated prototype
sql/sql_db.cc:
Added extra argument to call to check_if_table_exists()
If triggers are used for an insert/update/delete statement than the values of
all virtual columns must be computed as any of them may be used by the triggers.
MDEV-26: Global transaction id, partial commit
Change server_id to be a session variable.
User with SUPER can set it to binlog with different server_id.
Implement backward-compatible ::server_id mirror for plugins.
Generalized support for auto-updated and/or auto-initialized timestamp
and datetime columns. This patch is a reimplementation of MySQL's
"WL#5874: CURRENT_TIMESTAMP as DEFAULT for DATETIME columns". In order to
ease future merges, this implementation reused few function and variable
names from MySQL's patch, however the implementation is quite different.
TODO:
The only unresolved problem in this patch is the semantics of LOAD DATA for
TIMESTAMP and DATETIME columns in the cases when there are missing or NULL
columns. I couldn't fully comprehend the logic behind MySQL's behavior and
its relationship with their own documentation, so I left the results to be
more consistent with all other LOAD cases.
The problematic test cases can be seen by running the test file function_defaults,
and observing the test case differences. Those were left on purpose for discussion.
Documentation of the feature can be found at: http://kb.askmonty.org/en/multi-source-replication/
This code is based on code from Taobao, developed by Plinux
BUILD/SETUP.sh:
Added -Wno-invalid-offsetof to get rid of warning of offsetof() on C++ class (safe in the contex we use it)
client/mysqltest.cc:
Added support for error names starting with 'W'
Added connection_name support to --sync_with_master
cmake/maintainer.cmake:
Added -Wno-invalid-offsetof to get rid of warning of offsetof() on C++ class (safe in the contex we use it)
mysql-test/r/mysqltest.result:
Updated results
mysql-test/r/parser.result:
Updated results
mysql-test/suite/multi_source/my.cnf:
Setup of multi-master tests
mysql-test/suite/multi_source/simple.result:
Simple basic test of multi-source functionality
mysql-test/suite/multi_source/simple.test:
Simple basic test of multi-source functionality
mysql-test/suite/multi_source/syntax.result:
Test of multi-source syntax
mysql-test/suite/multi_source/syntax.test:
Test of multi-source syntax
mysql-test/suite/rpl/r/rpl_rotate_logs.result:
Updated results because of new error messages
mysql-test/t/parser.test:
Updated test as master_pos_wait() now takes more arguments than before
sql/event_scheduler.cc:
No reason to initialize slave_thread (it's guaranteed to be zero here)
sql/item_create.cc:
Added connection_name argument to master_pos_wait()
Simplified code
sql/item_func.cc:
Added connection_name argument to master_pos_wait()
sql/item_func.h:
Added connection_name argument to master_pos_wait()
sql/log.cc:
Added tag "Master 'connection_name'" to slave errors that has a connection name.
sql/mysqld.cc:
Added variable mysqld_server_initialized so that other functions can test if server is fully initialized.
Free all slave data in one place (fewer ifdef's)
Removed not needed call to close_active_mi()
Initialize slaves() later in startup to ensure that everthing is really initialized when slaves start.
Made status variable slave_running multi-source safe
sql/mysqld.h:
Added mysqld_server_initialized
sql/rpl_mi.cc:
Store connection name and cmp_connection_name (only used for show full slave status) in Master_info
Added code for Master_info_index, which handles storage of multi-master information
Don't write the empty "" connection_name to multi-master.info file. This is handled by the original code.
sql/rpl_mi.h:
Added connection_name and Master_info_index
sql/rpl_rli.cc:
Added connection_name to relay log files.
sql/rpl_rli.h:
Fixed type of slave_skip_counter as we now access it directly in sys_vars.cc, so it must be uint
sql/share/errmsg-utf8.txt:
Added new error messages needed for multi-source
Added multi-source name to error ER_MASTER_INFO and WARN_NO_MASTER_INFO
sql/slave.cc:
Moved things a bit around to make it easier to handle error conditions.
Create a global master_info_index and add the "" connection to it
Ensure that new Master_info doesn't fail.
Don't call terminate_slave_threads(active_mi..) on end_slave() as this is now done automaticly when deleting master_info_index.
Delete not needed function close_active_mi(). One can achive same thing by calling end_slave().
Added support for SHOW FULL SLAVE STATUS (show status for all master connections with connection_name as first column)
sql/slave.h:
Added new prototypes
sql/sql_base.cc:
More DBUG_PRINT
sql/sql_class.cc:
Reset thd->connection_name and thd-->default_master_connection
sql/sql_class.h:
Added thd->connection_name and thd-->default_master_connection
Added slave_skip_count to variables to make changing the @@sql_slave_skip_count variable thread safe
sql/sql_const.h:
Added MAX_CONNECTION_NAME
sql/sql_lex.cc:
Reset 'lex->verbose' (to simplify some sql_yacc.yy code)
sql/sql_lex.h:
Added connection_name
sql/sql_parse.cc:
Added support for connection_name to all SLAVE commands.
- Instead of using active_mi, we now get the current Master_info from master_info_index.
- Create new replication threads with CHANGE MASTER
- Added support for show_all_master_info()
sql/sql_reload.cc:
Made reset/full slave use master_info_index->get_master_info() instead of active_mi.
If one uses 'RESET SLAVE "connection_name" all' the connection is removed from master_info_index.
sql/sql_repl.cc:
sql_slave_skip_counter is moved to thd->variables to make it thread safe and fix some bugs with it
Add connection name to relay log files.
Added connection name to errors.
Added some logging for multi-master if log_warnings > 1
stop_slave():
- Don't check if thd is set. It's guaranteed to always be set.
change_master():
- Check for duplicate connection names in change_master()
- Check for wrong arguments first in file (to simplify error handling)
- Register new connections in master_info_index
sql/sql_yacc.yy:
Added optional connection_name to a all relevant master/slave commands
sql/strfunc.cc:
my_global.h shoud always be included first.
sql/sys_vars.cc:
Added variable default_master_connection
Made variable sql_slave_skip_counter multi-source safe
sql/sys_vars.h:
Added Sys_var_session_lexstring (needed for default_master_connection)
Added Sys_var_multi_source_uint (needed for sql_slave_skip_counter).
QUOTING IN REPLICATION
Problem: Misquoting or unquoted identifiers may lead to
incorrect statements to be logged to the binary log.
Fix: we use specialized functions to append quoted identifiers in
the statements generated by the server.
feature_dynamic_columns,feature_fulltext,feature_gis,feature_locale,feature_subquery,feature_timezone,feature_trigger,feature_xml
Opened_views, Executed_triggers, Executed_events
Added new process status 'updating status' as part of 'freeing items'
mysql-test/r/features.result:
Test of feature_xxx status variables
mysql-test/r/mysqld--help.result:
Removed duplicated 'language' variable.
mysql-test/r/view.result:
Test of opened_views
mysql-test/suite/rpl/t/rpl_start_stop_slave.test:
Write more information on failure
mysql-test/t/features.test:
Test of feature_xxx status variables
mysql-test/t/view.test:
Test of opened_views
sql/event_scheduler.cc:
Increment executed_events status variable
sql/field.cc:
Increment status variable
sql/item_func.cc:
Increment status variable
sql/item_strfunc.cc:
Increment status variable
sql/item_subselect.cc:
Increment status variable
sql/item_xmlfunc.cc:
Increment status variable
sql/mysqld.cc:
Add new status variables to 'show status'
sql/mysqld.h:
Added executed_events
sql/sql_base.cc:
Increment status variable
sql/sql_class.h:
Add new status variables
sql/sql_parse.cc:
Added new process status 'updating status' as part of 'freeing items'
sql/sql_trigger.cc:
Increment status variable
sql/sys_vars.cc:
Increment status variable
sql/tztime.cc:
Increment status variable
Opening system statistical tables and reading statistical data from
them for a regular table should be done after opening and locking
this regular table.
No test case is provided with this patch.
If a table is already in the table cache but without data from persistent
statistical tables then the function open_and_process_table should not
only allocate memory for this statistical data in the corresponding
TABLE_SHARE object, but also should copy the references to the data into
certain fields of the TABLE data structure: for each key of the table
KEY::read_stats should be copied, and for each column of the table
Field::read_stats should be copied.
FAILED IN CHECK_LOCK_AND_ST
Problem:
--------
lock_tables() is supposed to invoke check_lock_and_start_stmt()
for TABLE_LIST which are directly used by top level statement.
TABLE_LIST->prelocking_placeholder is set only for TABLE_LIST
which are used indirectly by stored programs invoked by top
level statement. Hence check_lock_and_start_stmt() should have
TABLE_LIST->prelocking_placeholder==false always, but it is
observed that this assert fails.
The failure is found during RQG test rqg_signal_resignal.
Analysis:
---------
open_tables() invokes open_and_process_routines() where it
finds all the TABLE_LIST that belong to the routine and
adds it to thd->lex->query_tables. During this process if
the open_and_process_routines() fail for some reason,
we are supposed to chop-off all the TABLE_LIST found during
calls to open_and_process_routines(). But, in practice this
is not happening.
thd->lex->query_tables_own_last is supposed to point to a
node in thd->lex->query_tables, which would be a first
TABLE_LIST used indirectly by stored programs invoked by
top level statement. This is found to be not-set correctly
when we plan to chop-off TABLE_LIST's, when
open_and_process_routines() failed.
close_tables_for_reopen() does chop-off all the TABLE_LIST
added after thd->lex->query_table_own_last. This is invoked
upon error in open_and_process_routines(). This call would
not work as expected as thd->lex->query_tables_own_last
is not set, or is not set to correctly.
Further, when open_tables() restarts the process of finding
TABLE_LIST belonging to stored programs, and as the
thd->lex->query_tables_own_last points to in-correct node,
there is possibility of new iteration setting the
thd->lex->query_tables_own_last past some old nodes that
belong to stored programs, added earlier and not removed.
Later when open_tables() completes, lock_tables() ends up
invoking check_lock_and_start_stmt() for TABLE_LIST which
belong to stored programs, which is not expected behavior
and hence we hit the assert
TABLE_LIST->prelocking_placeholder==false.
Due to above behavior, if a user application tries to
execute a SQL statement which invokes some stored function
and if the lock grant on stored function fails due to a
deadlock, then mysqld crashes.
Fix:
----
open_tables() remembers save_query_tables_last which points
to thd-lex->query_tables_last before calls to
open_and_process_routines(). If there is no known
thd->lex->query_tables_own_last set, we are now setting
thd->lex->query_tables_own_last to save_query_tables_last.
This will make sure that the call to close_tables_for_reopen()
will chop-off the list correctly, in other words we now
remove all the nodes added to thd->lex->query_tables, by
previous calls to open_and_process_routines().
Further, it is found that the problem exists starting
from 5.5, due to a code refactoring effort related to
open_tables(). Hence, the fix will be pushed in 5.5, 5.6
and trunk.
With the new code of mysql-5.5 for metadata locking the function
unlock_tables_n_open_system_tables_for_write should not explicitly
unlock tables for which external locks have been set and should not
explicitly reset thd->lock to 0.
two tests still fail:
main.innodb_icp and main.range_vs_index_merge_innodb
call records_in_range() with both range ends being open
(which triggers an assert)
- Moved the definitions of the classes to store data from persistent
statistical tables into statistics.h, leaving in other internal
data structures only references to the corresponding objects.
- Defined class Column_statistics_collected derived from the class
Column_statistics. This is a helper class to collect statistics
on columns.
- Moved references to read statistics to TABLE SHARE, leaving the
the reference to the collected statistics in TABLE.
- Added a new clone method for the class Field allowing to clone
fields attached to table shares. It was was used to create
fields for min/max values in the memory of the table share.
A lso:
- Added procedures to allocate memory for statistical data in
the table share memory and in table memory.
Also:
- Added a test case demonstrating how ANALYZE could work in parallel
to collect statistics on different indexes of the same table.
- Added a test two demonstrate how two connections working
simultaneously could allocate memory for statistical data in the
table share memory.
Now when a table is dropped the statistics on the table is removed
from the statistical tables. If the table is altered in such a way
that a column is dropped or the type of the column is changed then
statistics on the column is removed from the table column_stat.
It also triggers removal of the statistics on the indexes who use
this column as its component.
Added procedures that changes the names of the tables or columns
in the statistical tables for.
These procedures are used when tables/columns are renamed.
Also partly re-factored the code that introduced the persistent
statistical tables.
Added test cases into statistics.test to cover the new code.
