AS A INNODB PARTITTION.
PROBLEM
-------
The correct engine_type was not being set during
rebuild of the partition due to which the handler
was always created with the default engine,
which is innodb for 5.5+ ,therefore even if the
table was myisam, after rebuilding the partitions
ended up as innodb partitions.
FIX
---
Set the correct engine type during rebuild.
[Approved by mattiasj #rb3599]
Inside a loop in a stored procedure, we create a partitioned
table. The CREATE statement is thus treated as a prepared statement:
it is prepared once, and then executed by each iteration. Thus its Lex
is reused many times. This Lex contains a part_info member, which
describes how the partitions should be laid out, including the
partitioning function. Each execution of the CREATE does this, in
open_table_from_share ():
tmp= mysql_unpack_partition(thd, share->partition_info_str,
share->partition_info_str_len,
outparam, is_create_table,
share->default_part_db_type,
&work_part_info_used);
...
tmp= fix_partition_func(thd, outparam, is_create_table);
The first line calls init_lex_with_single_table() which creates
a TABLE_LIST, necessary for the "field fixing" which will be
done by the second line; this is how it is created:
if ((!(table_ident= new Table_ident(thd,
table->s->db,
table->s->table_name, TRUE))) ||
(!(table_list= select_lex->add_table_to_list(thd,
table_ident,
NULL,
0))))
return TRUE;
it is allocated in the execution memory root.
Then the partitioning function ("id", stored in Lex -> part_info)
is fixed, which calls Item_ident:: fix_fields (), which resolves
"id" to the table_list above, and stores in the item's
cached_table a pointer to this table_list.
The table is created, later it is dropped by another statement,
then we execute again the prepared CREATE. This reuses the Lex,
thus also its part_info, thus also the item representing the
partitioning function (part_info is cloned but it's a shallow
cloning); CREATE wants to fix the item again (which is
normal, every execution fixes items again), fix_fields ()
sees that the cached_table pointer is set and picks up the
pointed table_list. But this last object does not exist
anymore (it was allocated in the execution memory root of
the previous execution, so it has been freed), so we access
invalid memory.
The solution: when creating the table_list, mark that it
cannot be cached.
The problem was in get_partition_id_cols_range_for_endpoint
and cmp_rec_and_tuple_prune, which stepped one partition too long.
Solution was to move a small portion of logic to cmp_rec_and_tuple_prune,
to simplify both get_partition_id_cols_range_for_endpoint and
get_partition_id_cols_list_for_endpoint.
DOWNGRADED FROM 5.6.11 TO 5.6.10
Problem was new syntax not accepted by previous version.
Fixed by adding version comment of /*!50531 around the
new syntax.
Like this in the .frm file:
'PARTITION BY KEY /*!50611 ALGORITHM = 2 */ () PARTITIONS 3'
and also changing the output from SHOW CREATE TABLE to:
CREATE TABLE t1 (a INT)
/*!50100 PARTITION BY KEY */ /*!50611 ALGORITHM = 1 */ /*!50100 ()
PARTITIONS 3 */
It will always add the ALGORITHM into the .frm for KEY [sub]partitioned
tables, but for SHOW CREATE TABLE it will only add it in case it is the non
default ALGORITHM = 1.
Also notice that for 5.5, it will say /*!50531 instead of /*!50611, which
will make upgrade from 5.5 > 5.5.31 to 5.6 < 5.6.11 fail!
If one downgrades an fixed version to the same major version (5.5 or 5.6) the
bug 14521864 will be visible again, but unless the .frm is updated, it will
work again when upgrading again.
Also fixed so that the .frm does not get updated version
if a single partition check passes.
Due to an internal change in the server code in between 5.1 and 5.5
(wl#2649) the hash function used in KEY partitioning changed
for numeric and date/time columns (from binary hash calculation
to character based hash calculation).
Also enum/set changed from latin1 ci based hash calculation to
binary hash between 5.1 and 5.5. (bug#11759782).
These changes makes KEY [sub]partitioned tables on any of
the affected column types incompatible with 5.5 and above,
since the calculation of partition id differs.
Also since InnoDB asserts that a deleted row was previously
read (positioned), the server asserts on delete of a row that
is in the wrong partition.
The solution for this situation is:
1) The partitioning engine will check that delete/update will go to the
partition the row was read from and give an error otherwise, consisting
of the rows partitioning fields. This will avoid asserts in InnoDB and
also alert the user that there is a misplaced row. A detailed error
message will be given, including an entry to the error log consisting
of both table name, partition and row content (PK if exists, otherwise
all partitioning columns).
2) A new optional syntax for KEY () partitioning in 5.5 is allowed:
[SUB]PARTITION BY KEY [ALGORITHM = N] (list_of_cols)
Where N = 1 uses the same hashing as 5.1 (Numeric/date/time fields uses
binary hashing, ENUM/SET uses charset hashing) N = 2 uses the same
hashing as 5.5 (Numeric/date/time fields uses charset hashing,
ENUM/SET uses binary hashing). If not set on CREATE/ALTER it will
default to 2.
This new syntax should probably be ignored by NDB.
3) Since there is a demand for avoiding scanning through the full
table, during upgrade the ALTER TABLE t PARTITION BY ... command is
considered a no-op (only .frm change) if everything except ALGORITHM
is the same and ALGORITHM was not set before, which allows manually
upgrading such table by something like:
ALTER TABLE t PARTITION BY KEY ALGORITHM = 1 () or
ALTER TABLE t PARTITION BY KEY ALGORITHM = 2 ()
4) Enhanced partitioning with CHECK/REPAIR to also check for/repair
misplaced rows. (Also works for ALTER TABLE t CHECK/REPAIR PARTITION)
CHECK FOR UPGRADE:
If the .frm version is < 5.5.3
and uses KEY [sub]partitioning
and an affected column type
then it will fail with an message:
KEY () partitioning changed, please run:
ALTER TABLE `test`.`t1` PARTITION BY KEY ALGORITHM = 1 (a)
PARTITIONS 12
(i.e. current partitioning clause, with the addition of
ALGORITHM = 1)
CHECK without FOR UPGRADE:
if MEDIUM (default) or EXTENDED options are given:
Scan all rows and verify that it is in the correct partition.
Fail for the first misplaced row.
REPAIR:
if default or EXTENDED (i.e. not QUICK/USE_FRM):
Scan all rows and every misplaced row is moved into its correct
partitions.
5) Updated mysqlcheck (called by mysql_upgrade) to handle the
new output from CHECK FOR UPGRADE, to run the ALTER statement
instead of running REPAIR.
This will allow mysql_upgrade (or CHECK TABLE t FOR UPGRADE) to upgrade
a KEY [sub]partitioned table that has any affected field type
and a .frm version < 5.5.3 to ALGORITHM = 1 without rebuild.
Also notice that if the .frm has a version of >= 5.5.3 and ALGORITHM
is not set, it is not possible to know if it consists of rows from
5.1 or 5.5! In these cases I suggest that the user does:
(optional)
LOCK TABLE t WRITE;
SHOW CREATE TABLE t;
(verify that it has no ALGORITHM = N, and to be safe, I would suggest
backing up the .frm file, to be used if one need to change to another
ALGORITHM = N, without needing to rebuild/repair)
ALTER TABLE t <old partitioning clause, but with ALGORITHM = N>;
which should set the ALGORITHM to N (if the table has rows from
5.1 I would suggest N = 1, otherwise N = 2)
CHECK TABLE t;
(here one could use the backed up .frm instead and change to a new N
and run CHECK again and see if it passes)
and if there are misplaced rows:
REPAIR TABLE t;
(optional)
UNLOCK TABLES;
Analysis:
---------
When the server is out of memory, an error is raised
to indicate the same. Handling the error requires
more memory to be allocated which fails, hence the
error handling loops in a recursion and causes the
server to crash.
Fix:
---
a) Prevents pushing the 'out of memory' error condition
to the diagnostic area as it requires memory allocation.
GET DIAGNOSTICS, SHOW WARNINGS and SHOW ERRORS statements
will not show information about this error. However the
'out of memory' error is returned to the client.
b) It sets the ME_FATALERROR flag when 'out of memory' errors
are reported (for places where the flag is not already set).
This flag prevents activation of SP error handlers which also
require memory allocation and therefore are likely to fail.
INCLUDES FIRST PARTITION WHEN PRUNING
PROBLEM
-------
TO_DAYS()/TO_SECONDS() can return NULL for invalid dates which
was stored in the first partition ,therefore the first partition
was always included for the scan when range was specified.
FIX
---
The fix is a small optimization which we have included ,which will
prune the scanning of NULL/first partition if the dates specified
in the range are valid and in the same year and month . TO_SECONDS()
function is not supported in 5.1 so removed it from the fix and test
scripts for mysql-5.1 version.
ALTER TABLE AFTER DROP PARTITION
Bug#13608188 - 64038: CRASH IN HANDLER::HA_THD ON ALTER TABLE AFTER
REPAIR NON-EXISTING PARTITION
Backport of bug#13357766 from -trunk to -5.5.
The state of some partitions was not reset on failure, leading
to invalid states of partitions in consequent statements.
Fixed by reverting back to original state for all partitions
if not all partition names was resolved.
Also adding extra security by forcing tables to be reopened
in case of error in mysql_alter_table.
(There is also removal of \r at the end of some lines.)
In sql_class.cc, 'row_count', of type 'ha_rows', was used as last argument for
ER_TRUNCATED_WRONG_VALUE_FOR_FIELD which is
"Incorrect %-.32s value: '%-.128s' for column '%.192s' at row %ld".
So 'ha_rows' was used as 'long'.
On SPARC32 Solaris builds, 'long' is 4 bytes and 'ha_rows' is 'longlong' i.e. 8 bytes.
So the printf-like code was reading only the first 4 bytes.
Because the CPU is big-endian, 1LL is 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x01
so the first four bytes yield 0. So the warning message had "row 0" instead of
"row 1" in test outfile_loaddata.test:
-Warning 1366 Incorrect string value: '\xE1\xE2\xF7' for column 'b' at row 1
+Warning 1366 Incorrect string value: '\xE1\xE2\xF7' for column 'b' at row 0
All error-messaging functions which internally invoke some printf-life function
are potential candidate for such mistakes.
One apparently easy way to catch such mistakes is to use
ATTRIBUTE_FORMAT (from my_attribute.h).
But this works only when call site has both:
a) the format as a string literal
b) the types of arguments.
So:
func(ER(ER_BLAH), 10);
will silently not be checked, because ER(ER_BLAH) is not known at
compile time (it is known at run-time, and depends on the chosen
language).
And
func("%s", a va_list argument);
has the same problem, as the *real* type of arguments is not
known at this site at compile time (it's known in some caller).
Moreover,
func(ER(ER_BLAH));
though possibly correct (if ER(ER_BLAH) has no '%' markers), will not
compile (gcc says "error: format not a string literal and no format
arguments").
Consequences:
1) ATTRIBUTE_FORMAT is here added only to functions which in practice
take "string literal" formats: "my_error_reporter" and "print_admin_msg".
2) it cannot be added to the other functions: my_error(),
push_warning_printf(), Table_check_intact::report_error(),
general_log_print().
To do a one-time check of functions listed in (2), the following
"static code analysis" has been done:
1) replace
my_error(ER_xxx, arguments for substitution in format)
with the equivalent
my_printf_error(ER_xxx,ER(ER_xxx), arguments for substitution in
format),
so that we have ER(ER_xxx) and the arguments *in the same call site*
2) add ATTRIBUTE_FORMAT to push_warning_printf(),
Table_check_intact::report_error(), general_log_print()
3) replace ER(xxx) with the hard-coded English text found in
errmsg.txt (like: ER(ER_UNKNOWN_ERROR) is replaced with
"Unknown error"), so that a call site has the format as string literal
4) this way, ATTRIBUTE_FORMAT can effectively do its job
5) compile, fix errors detected by ATTRIBUTE_FORMAT
6) revert steps 1-2-3.
The present patch has no compiler error when submitted again to the
static code analysis above.
It cannot catch all problems though: see Field::set_warning(), in
which a call to push_warning_printf() has a variable error
(thus, not replacable by a string literal); I checked set_warning() calls
by hand though.
See also WL 5883 for one proposal to avoid such bugs from appearing
again in the future.
The issues fixed in the patch are:
a) mismatch in types (like 'int' passed to '%ld')
b) more arguments passed than specified in the format.
This patch resolves mismatches by changing the type/number of arguments,
not by changing error messages of sql/share/errmsg.txt. The latter would be wrong,
per the following old rule: errmsg.txt must be as stable as possible; no insertions
or deletions of messages, no changes of type or number of printf-like format specifiers,
are allowed, as long as the change impacts a message already released in a GA version.
If this rule is not followed:
- Connectors, which use error message numbers, will be confused (by insertions/deletions
of messages)
- using errmsg.sys of MySQL 5.1.n with mysqld of MySQL 5.1.(n+1)
could produce wrong messages or crash; such usage can easily happen if
installing 5.1.(n+1) while /etc/my.cnf still has --language=/path/to/5.1.n/xxx;
or if copying mysqld from 5.1.(n+1) into a 5.1.n installation.
When fixing b), I have verified that the superfluous arguments were not used in the format
in the first 5.1 GA (5.1.30 'bteam@astra04-20081114162938-z8mctjp6st27uobm').
Had they been used, then passing them today, even if the message doesn't use them
anymore, would have been necessary, as explained above.
include/my_getopt.h:
this function pointer is used only with "string literal" formats, so we can add
ATTRIBUTE_FORMAT.
mysql-test/collections/default.experimental:
test should pass now
sql/derror.cc:
by having a format as string literal, ATTRIBUTE_FORMAT check becomes effective.
sql/events.cc:
Change justified by the following excerpt from sql/share/errmsg.txt:
ER_EVENT_SAME_NAME
eng "Same old and new event name"
ER_EVENT_SET_VAR_ERROR
eng "Error during starting/stopping of the scheduler. Error code %u"
sql/field.cc:
ER_TOO_BIG_SCALE 42000 S1009
eng "Too big scale %d specified for column '%-.192s'. Maximum is %lu."
ER_TOO_BIG_PRECISION 42000 S1009
eng "Too big precision %d specified for column '%-.192s'. Maximum is %lu."
ER_TOO_BIG_DISPLAYWIDTH 42000 S1009
eng "Display width out of range for column '%-.192s' (max = %lu)"
sql/ha_ndbcluster.cc:
ER_OUTOFMEMORY HY001 S1001
eng "Out of memory; restart server and try again (needed %d bytes)"
(sizeof() returns size_t)
sql/ha_ndbcluster_binlog.cc:
Too many arguments for:
ER_GET_ERRMSG
eng "Got error %d '%-.100s' from %s"
Patch by Jonas Oreland.
sql/ha_partition.cc:
print_admin_msg() is used only with a literal as format, so ATTRIBUTE_FORMAT
works.
sql/handler.cc:
ER_OUTOFMEMORY HY001 S1001
eng "Out of memory; restart server and try again (needed %d bytes)"
(sizeof() returns size_t)
sql/item_create.cc:
ER_TOO_BIG_SCALE 42000 S1009
eng "Too big scale %d specified for column '%-.192s'. Maximum is %lu."
ER_TOO_BIG_PRECISION 42000 S1009
eng "Too big precision %d specified for column '%-.192s'. Maximum is %lu."
'c_len' and 'c_dec' are char*, passed as %d !! We don't know their value
(as strtoul() failed), but they are likely big, so we use INT_MAX.
'len' is ulong.
sql/item_func.cc:
ER_WARN_DATA_OUT_OF_RANGE 22003
eng "Out of range value for column '%s' at row %ld"
ER_CANT_FIND_UDF
eng "Can't load function '%-.192s'"
sql/item_strfunc.cc:
ER_TOO_BIG_FOR_UNCOMPRESS
eng "Uncompressed data size too large; the maximum size is %d (probably, length of uncompressed data was corrupted)"
max_allowed_packet is ulong.
sql/mysql_priv.h:
sql_print_message_func is a function _pointer_.
sql/sp_head.cc:
ER_SP_RECURSION_LIMIT
eng "Recursive limit %d (as set by the max_sp_recursion_depth variable) was exceeded for routine %.192s"
max_sp_recursion_depth is ulong
sql/sql_acl.cc:
ER_PASSWORD_NO_MATCH 42000
eng "Can't find any matching row in the user table"
ER_CANT_CREATE_USER_WITH_GRANT 42000
eng "You are not allowed to create a user with GRANT"
sql/sql_base.cc:
ER_NOT_KEYFILE
eng "Incorrect key file for table '%-.200s'; try to repair it"
ER_TOO_MANY_TABLES
eng "Too many tables; MySQL can only use %d tables in a join"
MAX_TABLES is size_t.
sql/sql_binlog.cc:
ER_UNKNOWN_ERROR
eng "Unknown error"
sql/sql_class.cc:
ER_TRUNCATED_WRONG_VALUE_FOR_FIELD
eng "Incorrect %-.32s value: '%-.128s' for column '%.192s' at row %ld"
WARN_DATA_TRUNCATED 01000
eng "Data truncated for column '%s' at row %ld"
sql/sql_connect.cc:
ER_HANDSHAKE_ERROR 08S01
eng "Bad handshake"
ER_BAD_HOST_ERROR 08S01
eng "Can't get hostname for your address"
sql/sql_insert.cc:
ER_WRONG_VALUE_COUNT_ON_ROW 21S01
eng "Column count doesn't match value count at row %ld"
sql/sql_parse.cc:
ER_WARN_HOSTNAME_WONT_WORK
eng "MySQL is started in --skip-name-resolve mode; you must restart it without this switch for this grant to work"
ER_TOO_HIGH_LEVEL_OF_NESTING_FOR_SELECT
eng "Too high level of nesting for select"
ER_UNKNOWN_ERROR
eng "Unknown error"
sql/sql_partition.cc:
ER_OUTOFMEMORY HY001 S1001
eng "Out of memory; restart server and try again (needed %d bytes)"
sql/sql_plugin.cc:
ER_OUTOFMEMORY HY001 S1001
eng "Out of memory; restart server and try again (needed %d bytes)"
sql/sql_prepare.cc:
ER_OUTOFMEMORY HY001 S1001
eng "Out of memory; restart server and try again (needed %d bytes)"
ER_UNKNOWN_STMT_HANDLER
eng "Unknown prepared statement handler (%.*s) given to %s"
length value (for '%.*s') must be 'int', per the doc of printf()
and the code of my_vsnprintf().
sql/sql_show.cc:
ER_OUTOFMEMORY HY001 S1001
eng "Out of memory; restart server and try again (needed %d bytes)"
sql/sql_table.cc:
ER_TOO_BIG_FIELDLENGTH 42000 S1009
eng "Column length too big for column '%-.192s' (max = %lu); use BLOB or TEXT instead"
sql/table.cc:
ER_NOT_FORM_FILE
eng "Incorrect information in file: '%-.200s'"
ER_COL_COUNT_DOESNT_MATCH_PLEASE_UPDATE
eng "Column count of mysql.%s is wrong. Expected %d, found %d. Created with MySQL %d, now running %d. Please use mysql_upgrade to fix this error."
table->s->mysql_version is ulong.
sql/unireg.cc:
ER_TOO_LONG_TABLE_COMMENT
eng "Comment for table '%-.64s' is too long (max = %lu)"
ER_TOO_LONG_FIELD_COMMENT
eng "Comment for field '%-.64s' is too long (max = %lu)"
ER_TOO_BIG_ROWSIZE 42000
eng "Row size too large. The maximum row size for the used table type, not counting BLOBs, is %ld. You have to change some columns to TEXT or BLOBs"
multiple columns in the partition key
ndb crash if duplicate columns in the partitioning key.
Backport from mysql-5.1-telco-7.0, see bug#53354.
Changed from case sensitive field name comparision
to non case sensitive too.
mysql-test/r/partition_error.result:
updated result
mysql-test/t/partition_error.test:
Added test for the error in non-ndb partitioned table.
sql/sql_partition.cc:
Added check for duplicated field names in the
partitioning key.
multiline inserts into partition
Bug#57071: EXTRACT(WEEK from date_col) cannot be
allowed as partitioning function
Renamed function according to reviewers comments.
sql/item.h:
better name of processor function
sql/item_func.h:
better name of processor function
sql/item_timefunc.h:
better name of processor function
sql/sql_partition.cc:
better name of processor function
Updated comment.
Bug#57071: EXTRACT(WEEK from date_col) cannot be allowed as partitioning function
There were functions allowed as partitioning functions
that implicit allowed cast. That could result in unacceptable
behaviour.
Solution was to check that the arguments of date and time functions
have allowed types (field and date/datetime/time depending on function).
mysql-test/r/partition.result:
Updated result
mysql-test/r/partition_error.result:
Updated result
mysql-test/suite/parts/inc/part_supported_sql_funcs_main.inc:
disabled test with not allowed arguments.
mysql-test/suite/parts/r/part_supported_sql_func_innodb.result:
Updated result
mysql-test/suite/parts/r/part_supported_sql_func_myisam.result:
Updated result
mysql-test/t/partition.test:
Fixed typo in bug number and removed non allowed function (bad argument)
mysql-test/t/partition_error.test:
Added tests to verify correct type of argument.
sql/item.h:
Renamed processor since it is no longer only for timezone
sql/item_func.h:
Added help functions for checking date/time/datetime arguments.
sql/item_timefunc.h:
Added processors for argument correctness
sql/sql_partition.cc:
renamed the processor for checking arguments.
Bug#57995: Compiler flag change build error on OSX 10.4: my_getncpus.c
Bug#57996: Compiler flag change build error on OSX 10.5 : bind.c
Bug#57994: Compiler flag change build error : my_redel.c
Bug#57993: Compiler flag change build error on FreeBsd 7.0 : regexec.c
Bug#57992: Compiler flag change build error on FreeBsd : mf_keycache.c
Bug#57997: Compiler flag change build error on OSX 10.6: debug_sync.cc
Fix assorted compiler generated warnings.
cmd-line-utils/readline/bind.c:
Bug#57996: Compiler flag change build error on OSX 10.5 : bind.c
Initialize variable to work around a false positive warning.
include/m_string.h:
Bug#57994: Compiler flag change build error : my_redel.c
The expansion of stpcpy (in glibc) causes warnings if the
return value of strmov is not being used. Since stpcpy is
a GNU extension and the expansion ends up using a built-in
provided by GCC, use the compiler provided built-in directly
when possible.
include/my_compiler.h:
Define a dummy MY_GNUC_PREREQ when not compiling with GCC.
libmysql/libmysql.c:
Bug#58057: 5.1 libmysql/libmysql.c unused variable/compile failure
Variable might not be used in some cases. So, tag it as unused.
mysys/mf_keycache.c:
Bug#57992: Compiler flag change build error on FreeBsd : mf_keycache.c
Use UNINIT_VAR to work around a false positive warning.
mysys/my_getncpus.c:
Bug#57995: Compiler flag change build error on OSX 10.4: my_getncpus.c
Declare variable in the same block where it is used.
regex/regexec.c:
Bug#57993: Compiler flag change build error on FreeBsd 7.0 : regexec.c
Work around a compiler bug which causes the cast to not be enforced.
sql/debug_sync.cc:
Bug#57997: Compiler flag change build error on OSX 10.6: debug_sync.cc
Use UNINIT_VAR to work around a false positive warning.
sql/handler.cc:
Use UNINIT_VAR to work around a false positive warning.
sql/slave.cc:
Use UNINIT_VAR to work around a false positive warning.
sql/sql_partition.cc:
Use UNINIT_VAR to work around a false positive warning.
storage/myisam/ft_nlq_search.c:
Use UNINIT_VAR to work around a false positive warning.
storage/myisam/mi_create.c:
Use UNINIT_VAR to work around a false positive warning.
storage/myisammrg/myrg_open.c:
Use UNINIT_VAR to work around a false positive warning.
tests/mysql_client_test.c:
Change function to take a pointer to const, no need for a cast.
add boolean command-line option --autocommit.
mysql-test/mysql-test-run.pl:
do in --gdb like in --ddd: to let the developer debug the startup
phase (like command-line options parsing), don't "run".
It's the third time I do this change, it was previously lost
by merges, port of 6.0 to next-mr...
mysql-test/r/mysqld--help-notwin.result:
new command-line option
mysql-test/r/mysqld--help-win.result:
a Linux user's best guess at what the Windows result should be
mysql-test/suite/sys_vars/inc/autocommit_func2.inc:
new test
mysql-test/suite/sys_vars/t/autocommit_func2-master.opt:
test new option
mysql-test/suite/sys_vars/t/autocommit_func3-master.opt:
test new option
sql/mysqld.cc:
new --autocommit
sql/mysqld.h:
new --autocommit
sql/sql_partition.cc:
What matters to this partitioning quote is to have
the OPTION_QUOTE_SHOW_CREATE flag down, it's all
that append_identifier() uses. So we make it explicit.
Fix assorted warnings that are generated in optimized builds.
Most of it is silencing variables that are set but unused.
This patch also introduces the MY_ASSERT_UNREACHABLE macro
which helps the compiler to deduce that a certain piece of
code is unreachable.
include/my_compiler.h:
Use GCC's __builtin_unreachable if available. It allows
GCC to deduce the unreachability of certain code paths,
thus avoiding warnings that, for example, accused that a
variable could be used without being initialized (due to
unreachable code paths).
In case of failure in ALTER ... PARTITION under LOCK TABLE
the server could crash, due to it had modified the locked
table object, which was not reverted in case of failure,
resulting in a bad table definition used after the failed
command.
Solved by always closing the LOCKED TABLE, even in case
of error.
Note: this is a 5.1-only fix, bug#56172 fixed it in 5.5+
mysql-test/r/partition_innodb_plugin.result:
updated result
mysql-test/t/disabled.def:
Only disabled valgrind instead.
mysql-test/t/partition_innodb_plugin.test:
Added test
sql/sql_partition.cc:
close_thread_tables do not close LOCKED TABLEs
and destroys the table object (including part_info),
so to avoid it to be reused, always close the table
regardless of any previous failure.
REBUILD PARTITION under LOCK TABLE
Collapsed patch including updates from the reviews.
In case of failure in ALTER ... PARTITION under LOCK TABLE
the server could crash, due to it had modified the locked
table object, which was not reverted in case of failure,
resulting in a bad table definition used after the failed
command.
Solved by instead of altering the locked table object and
its partition_info struct, creating an internal temporary
intermediate table object used for altering,
just like the non partitioned mysql_alter_table.
So if an error occur before the alter operation is complete,
the original table is not modified at all.
But if the alter operation have succeeded so far that it
must be completed as whole,
the table is properly closed and reopened.
(The completion on failure is done by the ddl_log.)
mysql-test/suite/parts/inc/partition_fail.inc:
Added tests under LOCK TABLE
mysql-test/suite/parts/r/partition_debug_innodb.result:
Updated results
mysql-test/suite/parts/r/partition_debug_myisam.result:
Updated results
mysql-test/suite/parts/r/partition_special_innodb.result:
updated result
mysql-test/suite/parts/t/partition_special_innodb.test:
changing comment, since this patch also fixes this.
sql/sql_partition.cc:
Added TODO, to use DBUG_SUICIDE() instead of abort()
to avoid core-files on expected crashes.
Removed unused arguments to fast_end_partition.
Opening a intermediate table in prep_alter_part_table, instead of altering
(a possible locked) normally opened table.
That way we do not have to do anything more than close
the intermediate table on error,
leaving the ordinary table opened and locked.
Also making sure that the intermediate table are
closed/destroyed on failure. If no error occur
it is later destroyed in the end of fast_alter_partition_table.
Added ha_external_lock to make sure MyISAM flushed the index file
after copying the partitions.
This also leads to removal of the special close and removal from
the table cache for other instances of the table.
sql/sql_partition.h:
Changed the arguments for prep_alter_part_table and
fast_alter_partition_table to use an intermediate table
instead of altering a (possibly locked) normal table.
sql/sql_table.cc:
Using an intermediate table created in prep_alter_part_table
to be used in fast_alter_partition_table, also closing/destroying
it on failure.
It was hard to understand what the error really meant.
The error checking in partitioning is done in several different
parts during the execution of a query which can make it
hard to return useful errors.
Added a new error for bad VALUES part in the per PARTITION clause.
Using the more verbose error that a column is not allowed in
the partitioning function instead of just that the function is
not allowed.
mysql-test/r/partition.result:
changed error to be more specific
mysql-test/r/partition_error.result:
updated result
mysql-test/std_data/parts/t1TIMESTAMP.frm:
.frm file of CREATE TABLE t1 (a TIMESTAMP) PARTITION BY HASH(TO_DAYS(a));
mysql-test/t/partition.test:
changed error to be more specific
mysql-test/t/partition_error.test:
Added test (also for verifying behaviour of previously
created tables which is no longer allowed).
Updated expected errors in other places
sql/partition_info.cc:
Added function report_part_expr_error to
be able to return a more specific error.
Renamed fix_func_partition to fix_partition_values
since the function really fixes/checks the VALUES clause.
sql/partition_info.h:
removed part_result_type, since it was unused.
renamed fix_funk_partition->fix_partition_values
added report_part_expr_error
sql/share/errmsg-utf8.txt:
Added a more specific error.
sql/sql_partition.cc:
made use of report_part_expr_error to get a more specific error.
sql/sql_yacc.yy:
Changed error message to be more specific. And return an other error code.
The ALTER PARTITION and SELECT seemed to be deadlocked
when having innodb_thread_concurrency = 1.
Problem was that there was unreleased latches
in the ALTER PARTITION thread which was needed
by the SELECT thread to be able to continue.
Solution was to release the latches by commit
before requesting upgrade to exclusive MDL lock.
Updated according to reviewers comments (3).
mysql-test/r/partition_innodb.result:
updated test result
mysql-test/t/partition_innodb.test:
added test
sql/sql_partition.cc:
Moved implicit commit into mysql_change_partition
so that if latches are taken, they are always released
before waiting on exclusive lock.
sql/sql_table.cc:
refactored the code to prepare and commit
around copy_data_between_tables, to be able
to reuse it in mysql_change_partitions
sql/sql_table.h:
exporting mysql_trans_prepare/commit_alter_copy_data
corruption on ADD PARTITION and LOCK TABLE
Bug#53770: Server crash at handler.cc:2076 on
LOAD DATA after timed out COALESCE PARTITION
5.5 fix for:
Bug#51042: REORGANIZE PARTITION can leave table in an
inconsistent state in case of crash
Needs to be back-ported to 5.1
5.5 fix for:
Bug#50418: DROP PARTITION does not interact with
transactions
Main problem was non-persistent operations done
before meta-data lock was taken (53770+53676).
And 53676 needed to keep the table/partitions opened and locked
while copying the data to the new partitions.
Also added thorough tests to spot some additional bugs
in the ddl_log code, which could result in bad state
between the .frm and partitions.
Collapsed patch, includes all fixes required from the reviewers.
mysql-test/r/partition_innodb.result:
updated result with new test
mysql-test/suite/parts/inc/partition_crash.inc:
crash test include file
mysql-test/suite/parts/inc/partition_crash_add.inc:
test all states in fast_alter_partition_table
ADD PARTITION branch
mysql-test/suite/parts/inc/partition_crash_change.inc:
test all states in fast_alter_partition_table
CHANGE PARTITION branch
mysql-test/suite/parts/inc/partition_crash_drop.inc:
test all states in fast_alter_partition_table
DROP PARTITION branch
mysql-test/suite/parts/inc/partition_fail.inc:
recovery test including injecting errors
mysql-test/suite/parts/inc/partition_fail_add.inc:
test all states in fast_alter_partition_table
ADD PARTITION branch
mysql-test/suite/parts/inc/partition_fail_change.inc:
test all states in fast_alter_partition_table
CHANGE PARTITION branch
mysql-test/suite/parts/inc/partition_fail_drop.inc:
test all states in fast_alter_partition_table
DROP PARTITION branch
mysql-test/suite/parts/inc/partition_mgm_crash.inc:
include file that runs all crash and failure injection tests.
mysql-test/suite/parts/r/partition_debug_innodb.result:
new test result file
mysql-test/suite/parts/r/partition_debug_myisam.result:
new test result file
mysql-test/suite/parts/r/partition_special_innodb.result:
updated result
mysql-test/suite/parts/r/partition_special_myisam.result:
updated result
mysql-test/suite/parts/t/partition_debug_innodb-master.opt:
opt file for using with crashing tests of partitioned innodb
mysql-test/suite/parts/t/partition_debug_innodb.test:
partitioned innodb test that require debug builds
mysql-test/suite/parts/t/partition_debug_myisam-master.opt:
opt file for using with crashing tests of partitioned myisam
mysql-test/suite/parts/t/partition_debug_myisam.test:
partitioned myisam test that require debug builds
mysql-test/suite/parts/t/partition_special_innodb-master.opt:
added innodb-file-per-table to easier verify partition status on disk
mysql-test/suite/parts/t/partition_special_innodb.test:
added test case
mysql-test/suite/parts/t/partition_special_myisam.test:
added test case
mysql-test/t/partition_innodb.test:
added test case
sql/sql_base.cc:
Moved alter_close_tables to sql_partition.cc
sql/sql_base.h:
removed some non existing and duplicated functions.
sql/sql_partition.cc:
fast_alter_partition_table:
Spletted abort_and_upgrad_lock_and_close_table
to its parts (wait_while_table_is_used and
alter_close_tables) and always have
wait_while_table_is_used before any persistent
operations (including logs, which will be executed
on failure) and alter_close_tables after
create/read/write operations and before
drop operations.
moved alter_close_tables here from sql_base.cc
Added error injections for better test coverage.
write_log_final_change_partition:
fixed a log_entry linking bug (delete_frm was not
linked to change/drop partition)
and drop partition must be executed before
change partition (change partition can rename a
partition to an old name, like REORG p1 INTO (p1,p2).
write_log_add_change_partition:
need to use drop_frm first, and relinking that entry
and reusing its execute entry.
sql/sql_table.cc:
added initialization of next_active_log_entry.
sql/table.h:
removed a duplicate declaration.
/*![:version:] Query Code */, where [:version:] is a sequence of 5
digits representing the mysql server version(e.g /*!50200 ... */),
is a special comment that the query in it can be executed on those
servers whose versions are larger than the version appearing in the
comment. It leads to a security issue when slave's version is larger
than master's. A malicious user can improve his privileges on slaves.
Because slave SQL thread is running with SUPER privileges, so it can
execute queries that he/she does not have privileges on master.
This bug is fixed with the logic below:
- To replace '!' with ' ' in the magic comments which are not applied on
master. So they become common comments and will not be applied on slave.
- Example:
'INSERT INTO t1 VALUES (1) /*!10000, (2)*/ /*!99999 ,(3)*/
will be binlogged as
'INSERT INTO t1 VALUES (1) /*!10000, (2)*/ /* 99999 ,(3)*/
mysql-test/suite/rpl/t/rpl_conditional_comments.test:
Test the patch for this bug.
sql/mysql_priv.h:
Rename inBuf as rawBuf and remove the const limitation.
sql/sql_lex.cc:
To replace '!' with ' ' in the magic comments which are not applied on
master.
sql/sql_lex.h:
Remove the const limitation on parameter buff, as it can be modified in the function since
this patch.
Add member function yyUnput for Lex_input_stream. It set a character back the query buff.
sql/sql_parse.cc:
Rename inBuf as rawBuf and remove the const limitation.
sql/sql_partition.cc:
Remove the const limitation on parameter part_buff, as it can be modified in the function since
this patch.
sql/sql_partition.h:
Remove the const limitation on parameter part_buff, as it can be modified in the function since
this patch.
sql/table.h:
Remove the const limitation on variable partition_info, as it can be modified since
this patch.
This patch also fixes Bug#55452 "SET PASSWORD is
replicated twice in RBR mode".
The goal of this patch is to remove the release of
metadata locks from close_thread_tables().
This is necessary to not mistakenly release
the locks in the course of a multi-step
operation that involves multiple close_thread_tables()
or close_tables_for_reopen().
On the same token, move statement commit outside
close_thread_tables().
Other cleanups:
Cleanup COM_FIELD_LIST.
Don't call close_thread_tables() in COM_SHUTDOWN -- there
are no open tables there that can be closed (we leave
the locked tables mode in THD destructor, and this
close_thread_tables() won't leave it anyway).
Make open_and_lock_tables() and open_and_lock_tables_derived()
call close_thread_tables() upon failure.
Remove the calls to close_thread_tables() that are now
unnecessary.
Simplify the back off condition in Open_table_context.
Streamline metadata lock handling in LOCK TABLES
implementation.
Add asserts to ensure correct life cycle of
statement transaction in a session.
Remove a piece of dead code that has also become redundant
after the fix for Bug 37521.
mysql-test/r/variables.result:
Update results: set @@autocommit and statement transaction/
prelocked mode.
mysql-test/r/view.result:
A harmless change in CHECK TABLE <view> status for a broken view.
If previously a failure to prelock all functions used in a view
would leave the connection in LTM_PRELOCKED mode, now we call
close_thread_tables() from open_and_lock_tables()
and leave prelocked mode, thus some check in mysql_admin_table() that
works only in prelocked/locked tables mode is no longer activated.
mysql-test/suite/rpl/r/rpl_row_implicit_commit_binlog.result:
Fixed Bug#55452 "SET PASSWORD is replicated twice in
RBR mode": extra binlog events are gone from the
binary log.
mysql-test/t/variables.test:
Add a test case: set autocommit and statement transaction/prelocked
mode.
sql/event_data_objects.cc:
Simplify code in Event_job_data::execute().
Move sp_head memory management to lex_end().
sql/event_db_repository.cc:
Move the release of metadata locks outside
close_thread_tables().
Make sure we call close_thread_tables() when
open_and_lock_tables() fails and remove extra
code from the events data dictionary.
Use close_mysql_tables(), a new internal
function to properly close mysql.* tables
in the data dictionary.
Contract Event_db_repository::drop_events_by_field,
drop_schema_events into one function.
When dropping all events in a schema,
make sure we don't mistakenly release all
locks acquired by DROP DATABASE. These
include locks on the database name
and the global intention exclusive
metadata lock.
sql/event_db_repository.h:
Function open_event_table() does not require an instance
of Event_db_repository.
sql/events.cc:
Use close_mysql_tables() instead of close_thread_tables()
to bootstrap events, since the latter no longer
releases metadata locks.
sql/ha_ndbcluster.cc:
- mysql_rm_table_part2 no longer releases
acquired metadata locks. Do it in the caller.
sql/ha_ndbcluster_binlog.cc:
Deploy the new protocol for closing thread
tables in run_query() and ndb_binlog_index
code.
sql/handler.cc:
Assert that we never call ha_commit_trans/
ha_rollback_trans in sub-statement, which
is now the case.
sql/handler.h:
Add an accessor to check whether THD_TRANS object
is empty (has no transaction started).
sql/log.cc:
Update a comment.
sql/log_event.cc:
Since now we commit/rollback statement transaction in
mysql_execute_command(), we need a mechanism to communicate
from Query_log_event::do_apply_event() to mysql_execute_command()
that the statement transaction should be rolled back, not committed.
Ideally it would be a virtual method of THD. I hesitate
to make THD a virtual base class in this already large patch.
Use a thd->variables.option_bits for now.
Remove a call to close_thread_tables() from the slave IO
thread. It doesn't open any tables, and the protocol
for closing thread tables is more complicated now.
Make sure we properly close thread tables, however,
in Load_data_log_event, which doesn't
follow the standard server execution procedure
with mysql_execute_command().
@todo: this piece should use Server_runnable
framework instead.
Remove an unnecessary call to mysql_unlock_tables().
sql/rpl_rli.cc:
Update Relay_log_info::slave_close_thread_tables()
to follow the new close protocol.
sql/set_var.cc:
Remove an unused header.
sql/slave.cc:
Remove an unnecessary call to
close_thread_tables().
sql/sp.cc:
Remove unnecessary calls to close_thread_tables()
from SP DDL implementation. The tables will
be closed by the caller, in mysql_execute_command().
When dropping all routines in a database, make sure
to not mistakenly drop all metadata locks acquired
so far, they include the scoped lock on the schema.
sql/sp_head.cc:
Correct the protocol that closes thread tables
in an SP instruction.
Clear lex->sphead before cleaning up lex
with lex_end to make sure that we don't
delete the sphead twice. It's considered
to be "cleaner" and more in line with
future changes than calling delete lex->sphead
in other places that cleanup the lex.
sql/sp_head.h:
When destroying m_lex_keeper of an instruction,
don't delete the sphead that all lex objects
share.
@todo: don't store a reference to routine's sp_head
instance in instruction's lex.
sql/sql_acl.cc:
Don't call close_thread_tables() where the caller will
do that for us.
Fix Bug#55452 "SET PASSWORD is replicated twice in RBR
mode" by disabling RBR replication in change_password()
function.
Use close_mysql_tables() in bootstrap and ACL reload
code to make sure we release all metadata locks.
sql/sql_base.cc:
This is the main part of the patch:
- remove manipulation with thd->transaction
and thd->mdl_context from close_thread_tables().
Now this function is only responsible for closing
tables, nothing else.
This is necessary to be able to easily use
close_thread_tables() in procedures, that
involve multiple open/close tables, which all
need to be protected continuously by metadata
locks.
Add asserts ensuring that TABLE object
is only used when is protected by a metadata lock.
Simplify the back off condition of Open_table_context,
we no longer need to look at the autocommit mode.
Make open_and_lock_tables() and open_normal_and_derived_tables()
close thread tables and release metadata locks acquired so-far
upon failure. This simplifies their usage.
Implement close_mysql_tables().
sql/sql_base.h:
Add declaration for close_mysql_tables().
sql/sql_class.cc:
Remove a piece of dead code that has also become redundant
after the fix for Bug 37521.
The code became dead when my_eof() was made a non-protocol method,
but a method that merely modifies the diagnostics area.
The code became redundant with the fix for Bug#37521, when
we started to cal close_thread_tables() before
Protocol::end_statement().
sql/sql_do.cc:
Do nothing in DO if inside a substatement
(the assert moved out of trans_rollback_stmt).
sql/sql_handler.cc:
Add comments.
sql/sql_insert.cc:
Remove dead code.
Release metadata locks explicitly at the
end of the delayed insert thread.
sql/sql_lex.cc:
Add destruction of lex->sphead to lex_end(),
lex "reset" method called at the end of each statement.
sql/sql_parse.cc:
Move close_thread_tables() and other related
cleanups to mysql_execute_command()
from dispatch_command(). This has become
possible after the fix for Bug#37521.
Mark federated SERVER statements as DDL.
Next step: make sure that we don't store
eof packet in the query cache, and move
the query cache code outside mysql_parse.
Brush up the code of COM_FIELD_LIST.
Remove unnecessary calls to close_thread_tables().
When killing a query, don't report "OK"
if it was a suicide.
sql/sql_parse.h:
Remove declaration of a function that is now static.
sql/sql_partition.cc:
Remove an unnecessary call to close_thread_tables().
sql/sql_plugin.cc:
open_and_lock_tables() will clean up
after itself after a failure.
Move close_thread_tables() above
end: label, and replace with close_mysql_tables(),
which will also release the metadata lock
on mysql.plugin.
sql/sql_prepare.cc:
Now that we no longer release locks in close_thread_tables()
statement prepare code has become more straightforward.
Remove the now redundant check for thd->killed() (used
only by the backup project) from Execute_server_runnable.
Reorder code to take into account that now mysql_execute_command()
performs lex->unit.cleanup() and close_thread_tables().
sql/sql_priv.h:
Add a new option to server options to interact
between the slave SQL thread and execution
framework (hack). @todo: use a virtual
method of class THD instead.
sql/sql_servers.cc:
Due to Bug 25705 replication of
DROP/CREATE/ALTER SERVER is broken.
Make sure at least we do not attempt to
replicate these statements using RBR,
as this violates the assert in close_mysql_tables().
sql/sql_table.cc:
Do not release metadata locks in mysql_rm_table_part2,
this is done by the caller.
Do not call close_thread_tables() in mysql_create_table(),
this is done by the caller.
Fix a bug in DROP TABLE under LOCK TABLES when,
upon error in wait_while_table_is_used() we would mistakenly
release the metadata lock on a non-dropped table.
Explicitly release metadata locks when doing an implicit
commit.
sql/sql_trigger.cc:
Now that we delete lex->sphead in lex_end(),
zero the trigger's sphead in lex after loading
the trigger, to avoid double deletion.
sql/sql_udf.cc:
Use close_mysql_tables() instead of close_thread_tables().
sql/sys_vars.cc:
Remove code added in scope of WL#4284 which would
break when we perform set @@session.autocommit along
with setting other variables and using tables or functions.
A test case added to variables.test.
sql/transaction.cc:
Add asserts.
sql/tztime.cc:
Use close_mysql_tables() rather than close_thread_tables().
Essentially, the problem is that safemalloc is excruciatingly
slow as it checks all allocated blocks for overrun at each
memory management primitive, yielding a almost exponential
slowdown for the memory management functions (malloc, realloc,
free). The overrun check basically consists of verifying some
bytes of a block for certain magic keys, which catches some
simple forms of overrun. Another minor problem is violation
of aliasing rules and that its own internal list of blocks
is prone to corruption.
Another issue with safemalloc is rather the maintenance cost
as the tool has a significant impact on the server code.
Given the magnitude of memory debuggers available nowadays,
especially those that are provided with the platform malloc
implementation, maintenance of a in-house and largely obsolete
memory debugger becomes a burden that is not worth the effort
due to its slowness and lack of support for detecting more
common forms of heap corruption.
Since there are third-party tools that can provide the same
functionality at a lower or comparable performance cost, the
solution is to simply remove safemalloc. Third-party tools
can provide the same functionality at a lower or comparable
performance cost.
The removal of safemalloc also allows a simplification of the
malloc wrappers, removing quite a bit of kludge: redefinition
of my_malloc, my_free and the removal of the unused second
argument of my_free. Since free() always check whether the
supplied pointer is null, redudant checks are also removed.
Also, this patch adds unit testing for my_malloc and moves
my_realloc implementation into the same file as the other
memory allocation primitives.
client/mysqldump.c:
Pass my_free directly as its signature is compatible with the
callback type -- which wasn't the case for free_table_ent.
