bug #57006 "Deadlock between HANDLER and FLUSH TABLES WITH READ
LOCK" and bug #54673 "It takes too long to get readlock for
'FLUSH TABLES WITH READ LOCK'".
The first bug manifested itself as a deadlock which occurred
when a connection, which had some table open through HANDLER
statement, tried to update some data through DML statement
while another connection tried to execute FLUSH TABLES WITH
READ LOCK concurrently.
What happened was that FTWRL in the second connection managed
to perform first step of GRL acquisition and thus blocked all
upcoming DML. After that it started to wait for table open
through HANDLER statement to be flushed. When the first connection
tried to execute DML it has started to wait for GRL/the second
connection creating deadlock.
The second bug manifested itself as starvation of FLUSH TABLES
WITH READ LOCK statements in cases when there was a constant
stream of concurrent DML statements (in two or more
connections).
This has happened because requests for protection against GRL
which were acquired by DML statements were ignoring presence of
pending GRL and thus the latter was starved.
This patch solves both these problems by re-implementing GRL
using metadata locks.
Similar to the old implementation acquisition of GRL in new
implementation is two-step. During the first step we block
all concurrent DML and DDL statements by acquiring global S
metadata lock (each DML and DDL statement acquires global IX
lock for its duration). During the second step we block commits
by acquiring global S lock in COMMIT namespace (commit code
acquires global IX lock in this namespace).
Note that unlike in old implementation acquisition of
protection against GRL in DML and DDL is semi-automatic.
We assume that any statement which should be blocked by GRL
will either open and acquires write-lock on tables or acquires
metadata locks on objects it is going to modify. For any such
statement global IX metadata lock is automatically acquired
for its duration.
The first problem is solved because waits for GRL become
visible to deadlock detector in metadata locking subsystem
and thus deadlocks like one in the first bug become impossible.
The second problem is solved because global S locks which
are used for GRL implementation are given preference over
IX locks which are acquired by concurrent DML (and we can
switch to fair scheduling in future if needed).
Important change:
FTWRL/GRL no longer blocks DML and DDL on temporary tables.
Before this patch behavior was not consistent in this respect:
in some cases DML/DDL statements on temporary tables were
blocked while in others they were not. Since the main use cases
for FTWRL are various forms of backups and temporary tables are
not preserved during backups we have opted for consistently
allowing DML/DDL on temporary tables during FTWRL/GRL.
Important change:
This patch changes thread state names which are used when
DML/DDL of FTWRL is waiting for global read lock. It is now
either "Waiting for global read lock" or "Waiting for commit
lock" depending on the stage on which FTWRL is.
Incompatible change:
To solve deadlock in events code which was exposed by this
patch we have to replace LOCK_event_metadata mutex with
metadata locks on events. As result we have to prohibit
DDL on events under LOCK TABLES.
This patch also adds extensive test coverage for interaction
of DML/DDL and FTWRL.
Performance of new and old global read lock implementations
in sysbench tests were compared. There were no significant
difference between new and old implementations.
The thing is that the following attributes are fixed (remembered) when a trigger
is created:
- character_set_client
- character_set_results
- collation_connection
There are two triggers created in mysql-test/include/mtr_warnings.sql.
They were created using "current default" character set / collation.
is_triggers.test shows definition of these triggers including recorded
character set information.
The problem was that if "current default" changed, the recorded character
set information was not accurate.
There might be two ways to fix that:
a) update is_triggers.test so that it does not put character-set information
into result-file;
b) update mtr_warnings.sql so that the triggers are created using
hard-coded character sets.
This patch implements option b).
This is the 5.5 version of the fix. The 5.1 version was too complicated to
merge and was null merged.
This is a regression from the fix for bug no 38999. A storage engine capable
of reading only a subset of a table's columns updates corresponding bits in
the read buffer to signal that it has read NULL values for the corresponding
columns. It cannot, and should not, update any other bits. Bug no 38999
occurred because the implementation of UPDATE statements compare the NULL bits
using memcmp, inadvertently comparing bits that were never requested from the
storage engine. The regression was caused by the storage engine trying to
alleviate the situation by writing to all NULL bits, even those that it had no
knowledge of. This has devastating effects for the index merge algorithm,
which relies on all NULL bits, except those explicitly requested, being left
unchanged.
The fix reverts the fix for bug no 38999 in both InnoDB and InnoDB plugin and
changes the server's method of comparing records. For engines that always read
entire rows, we proceed as usual. For engines capable of reading only select
columns, the record buffers are now compared on a column by column basis. An
assertion was also added so that non comparable buffers are never read. Some
relevant copy-pasted code was also consolidated in a new function.
This is a regression from the fix for bug no 38999. A storage engine capable
of reading only a subset of a table's columns updates corresponding bits in
the read buffer to signal that it has read NULL values for the corresponding
columns. It cannot, and should not, update any other bits. Bug no 38999
occurred because the implementation of UPDATE statements compare the NULL bits
using memcmp, inadvertently comparing bits that were never requested from the
storage engine. The regression was caused by the storage engine trying to
alleviate the situation by writing to all NULL bits, even those that it had no
knowledge of. This has devastating effects for the index merge algorithm,
which relies on all NULL bits, except those explicitly requested, being left
unchanged.
The fix reverts the fix for bug no 38999 in both InnoDB and InnoDB plugin and
changes the server's method of comparing records. For engines that always read
entire rows, we proceed as usual. For engines capable of reading only select
columns, the record buffers are now compared on a column by column basis. An
assertion was also added so that non comparable buffers are never read. Some
relevant copy-pasted code was also consolidated in a new function.
Problem: CASE didn't work with a mixture of different character
sets in THEN/ELSE in some cases.
This happened because after character set aggregation
newly created Item_func_conv_charset items corresponding
to THEN/ELSE arguments were not put back to args[] array.
Fix:
put all Item_func_conv_charset back to args[].
@ mysql-test/include/ctype_numconv.inc
@ mysql-test/r/ctype_ucs.result
Adding tests
@ sql/item_cmpfunc.cc
Put "agg" back to args[] after character set aggregation.
but broken.
Before this patch, it was allowed to use stored functions in
HANDLER ... READ statements. The problem was that this functionality
was not really supported by the code. Proper locking would for example
not be performed, and it was also possible to break replication by
having stored functions that performed updates.
This patch disallows the use of stored functions in HANDLER ... READ.
Any such statement will now give an ER_NOT_SUPPORTED_YET error.
This is an incompatible change and should be reflected in the
documentation.
Test case added to handler_myisam/handler_innodb.test.
Subselect executes twice, at JOIN::optimize stage
and at JOIN::execute stage. At optimize stage
Innodb prebuilt struct which is used for the
retrieval of column values is initialized in.
ha_innobase::index_read(), prebuilt->sql_stat_start is true.
After QUICK_ROR_INTERSECT_SELECT finished his job it
restores read_set/write_set bitmaps with initial values
and deactivates one of the handlers used by
QUICK_ROR_INTERSECT_SELECT in JOIN::cleanup
(it's the case when we reuse original handler as one of
handlers required by QUICK_ROR_INTERSECT_SELECT object).
On second subselect execution inactive handler is activated
in QUICK_RANGE_SELECT::reset, file->ha_index_init().
In ha_index_init Innodb prebuilt struct is reinitialized
with inappropriate read_set/write_set bitmaps. Further
reinitialization in ha_innobase::index_read() does not
happen as prebuilt->sql_stat_start is false.
It leads to partial retrieval of required field values
and we get a mix of field values from different records
in the record buffer.
The fix is to reset
read_set/write_set bitmaps as these values
are required for proper intialization of
internal InnoDB struct which is used for
the retrieval of column values
(see build_template(), ha_innodb.cc)
With recent changes in the performance schema default sizing parameters,
the memory used by a mysqld binary increased accordingly.
This negatively affects the MTR test suite,
because running several tests in parallel now consumes more ressources.
The fix is to leave the default production values unchanged,
and to configure the MTR environment to limit memory
used when running tests in the test suite, which is ok
because only a few objects are typically used within a test script.
This fix:
- changed the default configuration in MTR to use less memory
- adjusted the performance schema tests accordingly
Note that 1,000 mutex instances was too short and caused test failures
in the past in team trees, so the default used is now 10,000 in MTR.
The amount of memory used by the performance schema itself
can be observed with the statement SHOW ENGINE PERFORMANCE_SCHEMA STATUS
Original changeset:
------------------------------------------------------------
revno: 3197
revision-id: alik@sun.com-20100831135426-h5a4s2w6ih1d8q2x
parent: magnus.blaudd@sun.com-20100830120632-u3xzy002mdwueli8
committer: Alexander Nozdrin <alik@sun.com>
branch nick: mysql-5.5-bugfixing
timestamp: Tue 2010-08-31 17:54:26 +0400
message:
Bug#55980 Character sets: supplementary character _bin ordering is wrong
Problem:
- ORDER BY for utf8mb4_bin, utf16_bin and utf32_bin returned
results in a wrong order, because old functions
(supporting only BMP range) were used to handle these collations.
- Additionally, utf16_bin did not sort supplementary characters
between U+D700 and U+E000, as WL#1213 specification specified.
------------------------------------------------------------
Problem:
- ORDER BY for utf8mb4_bin, utf16_bin and utf32_bin returned
results in a wrong order, because old functions
(supporting only BMP range) were used to handle these collations.
- Additionally, utf16_bin did not sort supplementary characters
between U+D700 and U+E000, as WL#1213 specification specified.
The include/mysqlhotcopy.inc had an error in the 'if' condition, so it failed
if the mysqlhotcopy tool was found.
per-file comments:
mysql-test/include/mysqlhotcopy.inc
test should proceed exactly if the mysqlhotcopy was set.
mysql-test/mysql-test-run.pl
don't set the MYSQL_HOTCOPY variable if no mysqlhotcopy was found.
Problem: a few functions did not calculate their max_length correctly.
This is an after-fix for WL#2649 Number-to-string conversions".
Fix: changing the buggy functions to calculate max_length
using fix_char_length() introduced in WL#2649,
instead of setting max_length directly
mysql-test/include/ctype_numconv.inc
Adding new tests
mysql-test/r/ctype_binary.result
Adding new tests
mysql-test/r/ctype_cp1251.result
Adding new tests
mysql-test/r/ctype_latin1.result
Adding new tests
mysql-test/r/ctype_ucs.result
Adding new tests
mysql-test/r/ctype_utf8.result
Adding new tests
mysql-test/t/ctype_utf8.test
Including ctype_numconv
sql/item.h
- Introducing new method fix_char_length_ulonglong(),
for the cases when length is potentially greater
than UINT_MAX32. This method removes a few
instances of duplicate code, e.g. in item_strfunc.cc.
- Setting collation in Item_copy properly. This change
fixes wrong metadata on client side in some cases, when
"binary" instead of the real character set was reported.
sql/item_cmpfunc.cc
- Using fix_char_length() and max_char_length() methods,
instead of direct access to max_length, to calculate
item length properly.
- Moving count_only_length() in COALESCE after
agg_arg_charsets_for_string_result(). The old
order was incorrect and led to wrong length
calucation in case of multi-byte character sets.
sql/item_func.cc
Fixing that count_only_length() didn't work
properly for multi-byte character sets.
Using fix_char_length() and max_char_length()
instead of direct access to max_length.
sql/item_strfunc.cc
- Using fix_char_length(), fix_char_length_ulonglong(),
max_char_length() instead of direct access to max_length.
- Removing wierd condition: "if (collation.collation->mbmaxlen > 0)",
which is never FALSE.
'CREATE TABLE IF NOT EXISTS ... SELECT' behaviour
BUG#47132, BUG#47442, BUG49494, BUG#23992 and BUG#48814 will disappear
automatically after the this patch.
BUG#55617 is fixed by this patch too.
This is the 5.5 part.
It implements:
- 'CREATE TABLE IF NOT EXISTS ... SELECT' statement will not insert
anything and binlog anything if the table already exists.
It only generate a warning that table already exists.
- A couple of test cases for the behavior changing.
'CREATE TABLE IF NOT EXISTS ... SELECT' behaviour
BUG#55474, BUG#55499, BUG#55598, BUG#55616 and BUG#55777 are fixed
in this patch too.
This is the 5.1 part.
It implements:
- if the table exists, binlog two events: CREATE TABLE IF NOT EXISTS
and INSERT ... SELECT
- Insert nothing and binlog nothing on master if the existing object
is a view. It only generates a warning that table already exists.
example files)
The system variable 'thread_concurrency' has been
(re-)enabled on all platforms, to prevent startup
errors.
'thread_concurrency' is unused and has no effect,
on any platform, in MySQL 5.1 and later versions. It
will be deprecated, and removed, in context of
worklog WL#5265
FLUSH TABLES <list> WITH READ LOCK are incompatible" to
be pushed as separate patch.
Replaced thread state name "Waiting for table", which was
used by threads waiting for a metadata lock or table flush,
with a set of names which better reflect types of resources
being waited for.
Also replaced "Table lock" thread state name, which was used
by threads waiting on thr_lock.c table level lock, with more
elaborate "Waiting for table level lock", to make it
more consistent with other thread state names.
Updated test cases and their results according to these
changes.
Fixed sys_vars.query_cache_wlock_invalidate_func test to not
to wait for timeout of wait_condition.inc script.
