This is port of fix for MySQL BUG#17647863.
revno: 5572
revision-id: jon.hauglid@oracle.com-20131030232243-b0pw98oy72uka2sj
committer: Jon Olav Hauglid <jon.hauglid@oracle.com>
timestamp: Thu 2013-10-31 00:22:43 +0100
message:
Bug#17647863: MYSQL DOES NOT COMPILE ON OSX 10.9 GM
Rename test() macro to MY_TEST() to avoid conflict with libc++.
- Added MALLOC_LIBRARY variable to hold name of malloc library
- Back ported valgrind related fixes from jemalloc 3.4.1 to the included jemalloc 3.3.1
- Renamed bitmap_init() and bitmap_free() to my_bitmap_init() and my_bitmap_free() to avoid clash with jemalloc 3.4.1
- Use option --soname-synonyms=somalloc=NON to valgrind when using jemalloc
- Show version related variables in mysqld --help
-- Added SHOW_VALUE_IN_HELP marker
Increased back_log to 150 as the original value was a bit too small
CMakeLists.txt:
Added MALLOC_LIBRARY variable to hold name of malloc library
cmake/jemalloc.cmake:
Added MALLOC_LIBRARY variable to hold name of malloc library
config.h.cmake:
Added MALLOC_LIBRARY variable to hold name of malloc library
extra/jemalloc/ChangeLog:
Updates changelog
extra/jemalloc/include/jemalloc/internal/arena.h:
Backported valgrind fixes from jemalloc 3.4.1
extra/jemalloc/include/jemalloc/internal/jemalloc_internal.h.in:
Backported valgrind fixes from jemalloc 3.4.1
extra/jemalloc/include/jemalloc/internal/private_namespace.h:
Backported valgrind fixes from jemalloc 3.4.1
extra/jemalloc/include/jemalloc/internal/tcache.h:
Backported valgrind fixes from jemalloc 3.4.1
extra/jemalloc/src/arena.c:
Backported valgrind fixes from jemalloc 3.4.1
include/my_bitmap.h:
Renamed bitmap_init() and bitmap_free() to my_bitmap_init() and my_bitmap_free() to avoid clash with jemalloc 3.4.1
mysql-test/mysql-test-run.pl:
Use option --soname-synonyms=somalloc=NON to valgrind when using jemalloc
mysql-test/valgrind.supp:
Supression of memory leak in OpenSuse 12.3
mysys/my_bitmap.c:
Renamed bitmap_init() and bitmap_free() to my_bitmap_init() and my_bitmap_free()
sql/ha_ndbcluster_binlog.cc:
Renames
sql/ha_ndbcluster_cond.h:
Renames
sql/ha_partition.cc:
Renames
sql/handler.cc:
Renames
sql/item_subselect.cc:
Renames
sql/log_event.cc:
Renames
sql/log_event_old.cc:
Renames
sql/mysqld.cc:
Renames
Show version related variables in mysqld --help
sql/opt_range.cc:
Renames
sql/opt_table_elimination.cc:
Renames
sql/partition_info.cc:
Renames
sql/rpl_injector.h:
Renames
sql/set_var.h:
Renames
sql/slave.cc:
Renames
sql/sql_bitmap.h:
Renames
sql/sql_insert.cc:
Renames
sql/sql_lex.h:
Renames
sql/sql_parse.cc:
Renames
sql/sql_partition.cc:
Renames
sql/sql_select.cc:
Renames
sql/sql_show.cc:
Renames
sql/sql_update.cc:
Renames
sql/sys_vars.cc:
Show version related variables in mysqld --help
sql/sys_vars.h:
Added SHOW_VALUE_IN_HELP marker for variables that should be shown in --help
sql/table.cc:
Renames
sql/table.h:
Removed not used bitmap_init_value
storage/connect/ha_connect.cc:
Removed compiler warning
storage/maria/ma_open.c:
Renames
unittest/mysys/bitmap-t.c:
Renames
merged from 5.6:
Bug#14521864: MYSQL 5.1 TO 5.5 BUGS PARTITIONING
Bug#16589511: MYSQL_UPGRADE FAILS TO WRITE OUT ENTIRE ALTER TABLE ... ALGORITHM= ... STATEMENT
Bug#16274455: CAN NOT ACESS PARTITIONED TABLES WHEN DOWNGRADED FROM 5.6.11 TO 5.6.10
plus minor changes from 5.6, mainly comments
AS A INNODB PARTITTION.
PROBLEM
-------
The correct engine_type was not being set during
rebuild of the partition due to which the handler
was always created with the default engine,
which is innodb for 5.5+ ,therefore even if the
table was myisam, after rebuilding the partitions
ended up as innodb partitions.
FIX
---
Set the correct engine type during rebuild.
[Approved by mattiasj #rb3599]
Inside a loop in a stored procedure, we create a partitioned
table. The CREATE statement is thus treated as a prepared statement:
it is prepared once, and then executed by each iteration. Thus its Lex
is reused many times. This Lex contains a part_info member, which
describes how the partitions should be laid out, including the
partitioning function. Each execution of the CREATE does this, in
open_table_from_share ():
tmp= mysql_unpack_partition(thd, share->partition_info_str,
share->partition_info_str_len,
outparam, is_create_table,
share->default_part_db_type,
&work_part_info_used);
...
tmp= fix_partition_func(thd, outparam, is_create_table);
The first line calls init_lex_with_single_table() which creates
a TABLE_LIST, necessary for the "field fixing" which will be
done by the second line; this is how it is created:
if ((!(table_ident= new Table_ident(thd,
table->s->db,
table->s->table_name, TRUE))) ||
(!(table_list= select_lex->add_table_to_list(thd,
table_ident,
NULL,
0))))
return TRUE;
it is allocated in the execution memory root.
Then the partitioning function ("id", stored in Lex -> part_info)
is fixed, which calls Item_ident:: fix_fields (), which resolves
"id" to the table_list above, and stores in the item's
cached_table a pointer to this table_list.
The table is created, later it is dropped by another statement,
then we execute again the prepared CREATE. This reuses the Lex,
thus also its part_info, thus also the item representing the
partitioning function (part_info is cloned but it's a shallow
cloning); CREATE wants to fix the item again (which is
normal, every execution fixes items again), fix_fields ()
sees that the cached_table pointer is set and picks up the
pointed table_list. But this last object does not exist
anymore (it was allocated in the execution memory root of
the previous execution, so it has been freed), so we access
invalid memory.
The solution: when creating the table_list, mark that it
cannot be cached.
includes:
* remove some remnants of "Bug#14521864: MYSQL 5.1 TO 5.5 BUGS PARTITIONING"
* introduce LOCK_share, now LOCK_ha_data is strictly for engines
* rea_create_table() always creates .par file (even in "frm-only" mode)
* fix a 5.6 bug, temp file leak on dummy ALTER TABLE
Merged all ddl_logging code.
Merged sql_partition.cc
innodb_mysql_lock2.test and partition_cache.test now works.
Changed interface to strconvert() to make it easier to use with not \0 terminated strings.
sql/sql_partition.cc:
Full merge with 5.6
sql/sql_table.cc:
Merged all ddl_logging code
sql/strfunc.cc:
Added from_length argument to strconvert() to make it possible to use without end terminated strings.
sql/strfunc.h:
Added from_length argument to strconvert() to make it possible to use without end terminated strings.
The problem was in get_partition_id_cols_range_for_endpoint
and cmp_rec_and_tuple_prune, which stepped one partition too long.
Solution was to move a small portion of logic to cmp_rec_and_tuple_prune,
to simplify both get_partition_id_cols_range_for_endpoint and
get_partition_id_cols_list_for_endpoint.
DOWNGRADED FROM 5.6.11 TO 5.6.10
Problem was new syntax not accepted by previous version.
Fixed by adding version comment of /*!50531 around the
new syntax.
Like this in the .frm file:
'PARTITION BY KEY /*!50611 ALGORITHM = 2 */ () PARTITIONS 3'
and also changing the output from SHOW CREATE TABLE to:
CREATE TABLE t1 (a INT)
/*!50100 PARTITION BY KEY */ /*!50611 ALGORITHM = 1 */ /*!50100 ()
PARTITIONS 3 */
It will always add the ALGORITHM into the .frm for KEY [sub]partitioned
tables, but for SHOW CREATE TABLE it will only add it in case it is the non
default ALGORITHM = 1.
Also notice that for 5.5, it will say /*!50531 instead of /*!50611, which
will make upgrade from 5.5 > 5.5.31 to 5.6 < 5.6.11 fail!
If one downgrades an fixed version to the same major version (5.5 or 5.6) the
bug 14521864 will be visible again, but unless the .frm is updated, it will
work again when upgrading again.
Also fixed so that the .frm does not get updated version
if a single partition check passes.
Due to an internal change in the server code in between 5.1 and 5.5
(wl#2649) the hash function used in KEY partitioning changed
for numeric and date/time columns (from binary hash calculation
to character based hash calculation).
Also enum/set changed from latin1 ci based hash calculation to
binary hash between 5.1 and 5.5. (bug#11759782).
These changes makes KEY [sub]partitioned tables on any of
the affected column types incompatible with 5.5 and above,
since the calculation of partition id differs.
Also since InnoDB asserts that a deleted row was previously
read (positioned), the server asserts on delete of a row that
is in the wrong partition.
The solution for this situation is:
1) The partitioning engine will check that delete/update will go to the
partition the row was read from and give an error otherwise, consisting
of the rows partitioning fields. This will avoid asserts in InnoDB and
also alert the user that there is a misplaced row. A detailed error
message will be given, including an entry to the error log consisting
of both table name, partition and row content (PK if exists, otherwise
all partitioning columns).
2) A new optional syntax for KEY () partitioning in 5.5 is allowed:
[SUB]PARTITION BY KEY [ALGORITHM = N] (list_of_cols)
Where N = 1 uses the same hashing as 5.1 (Numeric/date/time fields uses
binary hashing, ENUM/SET uses charset hashing) N = 2 uses the same
hashing as 5.5 (Numeric/date/time fields uses charset hashing,
ENUM/SET uses binary hashing). If not set on CREATE/ALTER it will
default to 2.
This new syntax should probably be ignored by NDB.
3) Since there is a demand for avoiding scanning through the full
table, during upgrade the ALTER TABLE t PARTITION BY ... command is
considered a no-op (only .frm change) if everything except ALGORITHM
is the same and ALGORITHM was not set before, which allows manually
upgrading such table by something like:
ALTER TABLE t PARTITION BY KEY ALGORITHM = 1 () or
ALTER TABLE t PARTITION BY KEY ALGORITHM = 2 ()
4) Enhanced partitioning with CHECK/REPAIR to also check for/repair
misplaced rows. (Also works for ALTER TABLE t CHECK/REPAIR PARTITION)
CHECK FOR UPGRADE:
If the .frm version is < 5.5.3
and uses KEY [sub]partitioning
and an affected column type
then it will fail with an message:
KEY () partitioning changed, please run:
ALTER TABLE `test`.`t1` PARTITION BY KEY ALGORITHM = 1 (a)
PARTITIONS 12
(i.e. current partitioning clause, with the addition of
ALGORITHM = 1)
CHECK without FOR UPGRADE:
if MEDIUM (default) or EXTENDED options are given:
Scan all rows and verify that it is in the correct partition.
Fail for the first misplaced row.
REPAIR:
if default or EXTENDED (i.e. not QUICK/USE_FRM):
Scan all rows and every misplaced row is moved into its correct
partitions.
5) Updated mysqlcheck (called by mysql_upgrade) to handle the
new output from CHECK FOR UPGRADE, to run the ALTER statement
instead of running REPAIR.
This will allow mysql_upgrade (or CHECK TABLE t FOR UPGRADE) to upgrade
a KEY [sub]partitioned table that has any affected field type
and a .frm version < 5.5.3 to ALGORITHM = 1 without rebuild.
Also notice that if the .frm has a version of >= 5.5.3 and ALGORITHM
is not set, it is not possible to know if it consists of rows from
5.1 or 5.5! In these cases I suggest that the user does:
(optional)
LOCK TABLE t WRITE;
SHOW CREATE TABLE t;
(verify that it has no ALGORITHM = N, and to be safe, I would suggest
backing up the .frm file, to be used if one need to change to another
ALGORITHM = N, without needing to rebuild/repair)
ALTER TABLE t <old partitioning clause, but with ALGORITHM = N>;
which should set the ALGORITHM to N (if the table has rows from
5.1 I would suggest N = 1, otherwise N = 2)
CHECK TABLE t;
(here one could use the backed up .frm instead and change to a new N
and run CHECK again and see if it passes)
and if there are misplaced rows:
REPAIR TABLE t;
(optional)
UNLOCK TABLES;
Analysis:
---------
When the server is out of memory, an error is raised
to indicate the same. Handling the error requires
more memory to be allocated which fails, hence the
error handling loops in a recursion and causes the
server to crash.
Fix:
---
a) Prevents pushing the 'out of memory' error condition
to the diagnostic area as it requires memory allocation.
GET DIAGNOSTICS, SHOW WARNINGS and SHOW ERRORS statements
will not show information about this error. However the
'out of memory' error is returned to the client.
b) It sets the ME_FATALERROR flag when 'out of memory' errors
are reported (for places where the flag is not already set).
This flag prevents activation of SP error handlers which also
require memory allocation and therefore are likely to fail.
INCLUDES FIRST PARTITION WHEN PRUNING
PROBLEM
-------
TO_DAYS()/TO_SECONDS() can return NULL for invalid dates which
was stored in the first partition ,therefore the first partition
was always included for the scan when range was specified.
FIX
---
The fix is a small optimization which we have included ,which will
prune the scanning of NULL/first partition if the dates specified
in the range are valid and in the same year and month . TO_SECONDS()
function is not supported in 5.1 so removed it from the fix and test
scripts for mysql-5.1 version.
Generalized support for auto-updated and/or auto-initialized timestamp
and datetime columns. This patch is a reimplementation of MySQL's
"WL#5874: CURRENT_TIMESTAMP as DEFAULT for DATETIME columns". In order to
ease future merges, this implementation reused few function and variable
names from MySQL's patch, however the implementation is quite different.
TODO:
The only unresolved problem in this patch is the semantics of LOAD DATA for
TIMESTAMP and DATETIME columns in the cases when there are missing or NULL
columns. I couldn't fully comprehend the logic behind MySQL's behavior and
its relationship with their own documentation, so I left the results to be
more consistent with all other LOAD cases.
The problematic test cases can be seen by running the test file function_defaults,
and observing the test case differences. Those were left on purpose for discussion.
mysql-test/r/partition.result:
Added test case
mysql-test/t/partition.test:
Added test case
sql/sql_partition.cc:
Do mysql_trans_prepare_alter_copy_data() after all original tables are locked.
(We don't want to disable transactions for the original tables, that still may be in the cache)
sql/sql_table.cc:
Fixed two wrong DBUG_ENTER
Changed HA_EXTRA_NORMAL to HA_EXTRA_NOT_USED (more clean)
mysql-test/suite/maria/lock.result:
More extensive tests of LOCK TABLE with FLUSH and REPAIR
mysql-test/suite/maria/lock.test:
More extensive tests of LOCK TABLE with FLUSH and REPAIR
sql/sql_admin.cc:
Fix that REPAIR TABLE ... USE_FRM works with LOCK TABLES
sql/sql_base.cc:
Ensure that transactions are closed in ARIA when doing flush
HA_EXTRA_NORMAL -> HA_EXTRA_NOT_USED
Don't call extra many times for a table in close_all_tables_for_name()
Added test if table_list->table as this can happen in error situations
sql/sql_partition.cc:
HA_EXTRA_NORMAL -> HA_EXTRA_NOT_USED
sql/sql_reload.cc:
Fixed comment
sql/sql_table.cc:
HA_EXTRA_NORMAL -> HA_EXTRA_NOT_USED
sql/sql_trigger.cc:
HA_EXTRA_NORMAL -> HA_EXTRA_NOT_USED
sql/sql_truncate.cc:
HA_EXTRA_FORCE_REOPEN -> HA_EXTRA_PREPARE_FOR_DROP for truncate, as this speeds up truncate by not having to flush the cache to disk.
mysql-test/suite/maria/maria-partitioning.result:
New test case
mysql-test/suite/maria/maria-partitioning.test:
New test case
sql/sql_base.cc:
Ignore HA_EXTRA_NORMAL for wait_while_table_is_used()
More DBUG
sql/sql_partition.cc:
Don't use HA_EXTRA_FORCE_REOPEN for wait_while_table_is_used() as the table is opened multiple times (in prep_alter_part_table)
This fixes the assert in Aria where we check if table is opened multiple times if HA_EXTRA_FORCE_REOPEN is issued
- 5.5 was missing calls to ha_extra(HA_PREPARE_FOR_DROP | HA_PREPARE_FOR_RENAME); Lost in merge 5.3 -> 5.5
sql/sql_admin.cc:
Updated arguments for close_all_tables_for_name
sql/sql_base.h:
Updated arguments for close_all_tables_for_name
sql/sql_partition.cc:
Updated arguments for close_all_tables_for_name
sql/sql_table.cc:
Updated arguments for close_all_tables_for_name
Removed test of kill, as we have already called 'ha_extra(HA_PREPARE_FOR_DROP)' and the table may be inconsistent.
sql/sql_trigger.cc:
Updated arguments for close_all_tables_for_name
sql/sql_truncate.cc:
For truncate that is done with drop + recreate, signal that the table will be dropped.
mysql-test/suite/innodb/t/group_commit_crash.test:
remove autoincrement to avoid rbr being used for insert ... select
mysql-test/suite/innodb/t/group_commit_crash_no_optimize_thread.test:
remove autoincrement to avoid rbr being used for insert ... select
mysys/my_addr_resolve.c:
a pointer to a buffer is returned to the caller -> the buffer cannot be on the stack
mysys/stacktrace.c:
my_vsnprintf() is ok here, in 5.5
ALTER TABLE AFTER DROP PARTITION
Bug#13608188 - 64038: CRASH IN HANDLER::HA_THD ON ALTER TABLE AFTER
REPAIR NON-EXISTING PARTITION
Backport of bug#13357766 from -trunk to -5.5.
The state of some partitions was not reset on failure, leading
to invalid states of partitions in consequent statements.
Fixed by reverting back to original state for all partitions
if not all partition names was resolved.
Also adding extra security by forcing tables to be reopened
in case of error in mysql_alter_table.
(There is also removal of \r at the end of some lines.)
client/CMakeLists.txt:
1. -rdynamic is a linker flag.
2. it should be used in all builds, not debug only
libmysql/get_password.c:
prefer a standard function, when possible
(otherwise a plugin will need to load it from the client)
- ensure that mtr supressions table is flushed before doing controlled crash and restart
- use DBUG_SUICIDE() rather than abort() in partition tests - avoids a crash message/warning
- disable perfschema all_instances test on Windows- there are legitimate reasons for output to be different on Unix (some different threads, some different locks), the differences are expected to grow in the future, e.g with threadpool.
sql/sql_insert.cc:
CREATE ... IF NOT EXISTS may do nothing, but
it is still not a failure. don't forget to my_ok it.
******
CREATE ... IF NOT EXISTS may do nothing, but
it is still not a failure. don't forget to my_ok it.
sql/sql_table.cc:
small cleanup
******
small cleanup
