Since log_throttle is not available in 5.5. Logging of
error message for failure of thread to create new connection
in "create_thread_to_handle_connection" is not backported.
Since, function "my_plugin_log_message" is not available in
5.5 version and since there is incompatibility between
sql_print_XXX function compiled with g++ and alog files with
gcc to use sql_print_error, changes related to audit log
plugin is not backported.
Backport to 5.5
(external Bug#69407 Build warnings with mysql)
support-files/build-tags:
Run etags on sql_yacc.yy, ignore other .yy files
unittest/mysys/explain_filename-t.cc:
NO_PLAN seems to fail on some platforms, use the actual number instead.
Analysis:
---------
When the server is out of memory, an error is raised
to indicate the same. Handling the error requires
more memory to be allocated which fails, hence the
error handling loops in a recursion and causes the
server to crash.
Fix:
---
a) Prevents pushing the 'out of memory' error condition
to the diagnostic area as it requires memory allocation.
GET DIAGNOSTICS, SHOW WARNINGS and SHOW ERRORS statements
will not show information about this error. However the
'out of memory' error is returned to the client.
b) It sets the ME_FATALERROR flag when 'out of memory' errors
are reported (for places where the flag is not already set).
This flag prevents activation of SP error handlers which also
require memory allocation and therefore are likely to fail.
Problem:-
In case of blob data field, UNION ALL doesn't give correct result.
Analysis:-
In MyISAM table, when we dont want to check for the distinct for particular
key, we set the key_map to zero.
While writing record in MyISAM table, we check the distinct with the help
of keys, by checking whether that key is active in key_map and then writing
the record.
In case of blob field, we are checking for distinct by unique constraint,
where we are not checking whether that unique key is active or not in key_map.
Solution:
Before checking for distinct, check whether any key is active in key_map.
storage/myisam/mi_write.c:
check whether key_map is active before checking distinct.
Problem:-
In case of blob data field, UNION ALL doesn't give correct result.
Analysis:-
In MyISAM table, when we dont want to check for the distinct for particular
key, we set the key_map to zero.
While writing record in MyISAM table, we check the distinct with the help
of keys, by checking whether that key is active in key_map and then writing
the record.
In case of blob field, we are checking for distinct by unique constraint,
where we are not checking whether that unique key is active or not in key_map.
Solution:-
Before checking for distinct, check whether any key is active in key_map.
storage/myisam/mi_write.c:
check whether key_map is active before checking distinct.
Problem description: Incorrect key file. Key file is corrupted,
while reading the keys from the file. The problem here is that
keyseg->start (which should point to the beginning of a field)
is pointing beyond total record length.
Fix: If keyseg->start is greater than total record length then
return error.
Problem description: Corrupt key file for the table. Size of the
key is greater than the maximum specified size. This results in
the overflow of the key buffer while reading the key from key
file.
Fix: If size of key is greater than the maximum size it returns
an error before writing it into the key buffer. Gives error as
corrupt file but no stack overflow.
Problem description:
mysql server crashes when we run repair table on currupted table.
Analysis:
The problem with this bug seem to be key_reflength out of bounds
(186 according to debugger). We read this value from meta-data
segment of .MYI file while doing mi_open().
If you look into _mi_kpointer() you can see that the upper limit
for key_reflength is 7.
Solution:
In mi_open() there is a line like:
if (share->base.keystart > 65535 || share->base.rec_reflength > 8)
we should verify key_reflength here as well.
TO 'MYISAM_SORT_BUFFER_SIZE'
Problem: 'myisam_sort_buffer_size' is a parameter used by
mysqld program only whereas 'sort_buffer_size' is used by
mysqld and myisamchk programs. But the error message printed
when myisamchk program is run with insufficient buffer size
is myisam_sort_buffer_size is too small which may mislead to the
server parameter myisam_sort_buffer_size.
SOLUTION: A parameter 'myisam_sort_buffer_size' is added as an
alias for 'sort_buffer_size' and the 'sort_buffer_size' parameter
is marked as deprecated. So myisamchk also has both the parameters
with the same role.
TO 'MYISAM_SORT_BUFFER_SIZE'
Problem: 'myisam_sort_buffer_size' is a parameter used by
mysqld program only whereas 'sort_buffer_size' is used by
mysqld and myisamchk programs. But the error message printed
when myisamchk program is run with insufficient buffer size
is myisam_sort_buffer_size is too small which may mislead to the
server parameter myisam_sort_buffer_size.
SOLUTION: A parameter 'myisam_sort_buffer_size' is added as an
alias for 'sort_buffer_size' and the 'sort_buffer_size' parameter
is marked as deprecated. So myisamchk also has both the parameters
with the same role.
ISSUE: Incorrect key file. Key file is corrupted,
Reading incorrect key information (keyseg)
from index file. Key definition in .MYI
and .FRM file differs. Starting pointer
to read the keyseg information is changed
to a value greater than the pack_reclength.
Memcpy tries to read keyseg information from
unallocated memory which causes the crash.
SOLUTION: One more check added to compare the
the key definition in .MYI and .FRM
file. If the definition differ, server
produces an error.
FULLTEXT INDEX AND CONCURRENT DML.
Problem Statement:
------------------
1) Create a table with FT index.
2) Enable concurrent inserts.
3) In multiple threads do below operations repeatedly
a) truncate table
b) insert into table ....
c) select ... match .. against .. non-boolean/boolean mode
After some time we could observe two different assert core dumps
Analysis:
--------
1)assert core dump at key_read_cache():
Two select threads operating in-parallel on same key
root block.
1st select thread block->status is set to BLOCK_ERROR
because the my_pread() in read_block() is returning '0'.
Truncate table made the index file size as 1024 and pread
was asked to get the block of count bytes(1024 bytes)
from offset of 1024 which it cannot read since its
"end of file" and retuning '0' setting
"my_errno= HA_ERR_FILE_TOO_SHORT" and the key_file_length,
key_root[0] is same i.e. 1024. Since block status has BLOCK_ERROR
the 1st select thread enter into the free_block() and will
be under wait on conditional mutex by making status as
BLOCK_REASSIGNED and goes for wait_on_readers(). Other select
thread will also work on the same block and sees the status as
BLOCK_ERROR and enters into free_block(), checks for BLOCK_REASSIGNED
and asserting the server.
2)assert core dump at key_write_cache():
One select thread and One insert thread.
Select thread gets the unlocks the 'keycache->cache_lock',
which allows other threads to continue and gets the pread()
return value as'0'(please see the explanation above) and
tries to get the lock on 'keycache->cache_lock' and waits
there for the lock.
Insert thread requests for the block, block will be assigned
from the hash list and makes the page_status as
'PAGE_WAIT_TO_BE_READ' and goes for the read_block(), waits
in the queue since there are some other threads performing
reads on the same block.
Select thread which was waiting for the 'keycache->cache_lock'
mutex in the read_block() will continue after getting the my_pread()
value as '0' and sets the block status as BLOCK_ERROR and goes to
the free_block() and go to the wait_for_readers().
Now the insert thread will awake and continues. and checks
block->status as not BLOCK_READ and it asserts.
Fix:
---
In the full text code, multiple readers of index file is not guarded.
Hence added below below code in _ft2_search() and walk_and_match().
to lock the key_root I have used below code in _ft2_search()
if (info->s->concurrent_insert)
mysql_rwlock_rdlock(&share->key_root_lock[0]);
and to unlock
if (info->s->concurrent_insert)
mysql_rwlock_unlock(&share->key_root_lock[0]);
storage/myisam/ft_boolean_search.c:
Since its a recursion function, to avoid confusion in taking and
releasing the locks, renamed _ft2_search() to _ft2_search_internal()
function. And _ft2_search() will take the lock, call
_ft2_search_internal() and release the lock in case of concurrent
inserts.
storage/myisam/ft_nlq_search.c:
Added read locks code in walk_and_match()
FULLTEXT INDEX AND CONCURRENT DML.
Problem Statement:
------------------
1) Create a table with FT index.
2) Enable concurrent inserts.
3) In multiple threads do below operations repeatedly
a) truncate table
b) insert into table ....
c) select ... match .. against .. non-boolean/boolean mode
After some time we could observe two different assert core dumps
Analysis:
--------
1)assert core dump at key_read_cache():
Two select threads operating in-parallel on same key
root block.
1st select thread block->status is set to BLOCK_ERROR
because the my_pread() in read_block() is returning '0'.
Truncate table made the index file size as 1024 and pread
was asked to get the block of count bytes(1024 bytes)
from offset of 1024 which it cannot read since its
"end of file" and retuning '0' setting
"my_errno= HA_ERR_FILE_TOO_SHORT" and the key_file_length,
key_root[0] is same i.e. 1024. Since block status has BLOCK_ERROR
the 1st select thread enter into the free_block() and will
be under wait on conditional mutex by making status as
BLOCK_REASSIGNED and goes for wait_on_readers(). Other select
thread will also work on the same block and sees the status as
BLOCK_ERROR and enters into free_block(), checks for BLOCK_REASSIGNED
and asserting the server.
2)assert core dump at key_write_cache():
One select thread and One insert thread.
Select thread gets the unlocks the 'keycache->cache_lock',
which allows other threads to continue and gets the pread()
return value as'0'(please see the explanation above) and
tries to get the lock on 'keycache->cache_lock' and waits
there for the lock.
Insert thread requests for the block, block will be assigned
from the hash list and makes the page_status as
'PAGE_WAIT_TO_BE_READ' and goes for the read_block(), waits
in the queue since there are some other threads performing
reads on the same block.
Select thread which was waiting for the 'keycache->cache_lock'
mutex in the read_block() will continue after getting the my_pread()
value as '0' and sets the block status as BLOCK_ERROR and goes to
the free_block() and go to the wait_for_readers().
Now the insert thread will awake and continues. and checks
block->status as not BLOCK_READ and it asserts.
Fix:
---
In the full text code, multiple readers of index file is not guarded.
Hence added below below code in _ft2_search() and walk_and_match().
to lock the key_root I have used below code in _ft2_search()
if (info->s->concurrent_insert)
mysql_rwlock_rdlock(&share->key_root_lock[0]);
and to unlock
if (info->s->concurrent_insert)
mysql_rwlock_unlock(&share->key_root_lock[0]);
storage/myisam/ft_boolean_search.c:
Since its a recursion function, to avoid confusion in taking and
releasing the locks, renamed _ft2_search() to _ft2_search_internal()
function. And _ft2_search() will take the lock, call
_ft2_search_internal() and release the lock in case of concurrent
inserts.
storage/myisam/ft_nlq_search.c:
Added read locks code in walk_and_match()
TABLES IN INCORRECT ENGINE
PROBLEM:
CREATE/ALTER TABLE currently can move system tables like
mysql.db, user, host etc, to engines other than MyISAM. This is not
completely supported as of now, by mysqld. When some of system tables
like plugin, servers, event, func, *_priv, time_zone* are moved
to innodb, mysqld restart crashes. Currently system tables
can be moved to BLACKHOLE also!!!.
ANALYSIS:
The problem is that there is no check before creating or moving
a system table to some particular engine.
System tables are suppose to be residing in MyISAM. We can think
of restricting system tables to exist only in MyISAM. But, there could
be future needs of these system tables to be part of other engines
by design. For eg, NDB cluster expects some tables to be on innodb
or ndb engine. This calls for a solution, by which system
tables can be supported by any desired engine, with minimal effort.
FIX:
The solution provides a handlerton interface using which,
mysqld server can query particular storage engine handlerton for
system tables that it supports. This way each storage engine
layer can define their own system database and system tables.
The check_engine() function uses the new handlerton function
ha_check_if_supported_system_table() to check if db.tablename
provided in the DDL is supported by the SE.
Note: This fix has modified a test in help.test, which was moving
mysql.help_* to innodb. The primary intention of the test was not
to move them between engines.
storage/innobase/include/sync0rw.ic:
Prerequisite for compiling with gcc4 on solaris: ignore result from
os_compare_and_swap_ulint
storage/myisam/mi_dynrec.c:
Prerequisite for compiling with gcc4 on solaris: cast to void*
ON 64 BIT MACHINES
PROBLEM: When sorting index during repair of
myisam tables, due to improper casting
of buffer size variables value of myisam_
sort_buffer_size is not set greater than
4GB.
SOLUTION: Proper casting of buffer size variable.
myisam_buffer_size changed to unsigned
long long to handle size > 4GB on
linux as well as windows.
CASES RESETS DATA POINTER TO SMAL
ISSUE: Myisamchk doing sort recover
on a table reduces data_file_length.
Maximum size of data file decreases,
lesser number of rows are stored.
SOLUTION: Size of data_file_length is
fixed to the original length.
CASES RESETS DATA POINTER TO SMAL
ISSUE: Myisamchk doing sort recover
on a table reduces data_file_length.
Maximum size of data file decreases,
lesser number of rows are stored.
SOLUTION: Size of data_file_length is
fixed to the original length.
WITH MYISAM_USE_MMAP ENABLED
MySQL server can crash due to segmentation fault when
started with myisam_use_mmap.
The reason behind this being, while making a request to
unmap (munmap) the previously mapped memory (mmap), the
size passed was 7 bytes larger than the size requested at
the time of mapping. This can eventually unmap the adjacent
memory mapped block, belonging to some other memory-map pool.
Hence the subsequent call to mmap can map a region which was
still a valid memory mapped area.
Fixed by removing the extra 7-byte margin which was erroneously
added to the size, used for unmappping.
storage/myisam/mi_close.c:
Bug#11756764 48726: MYSQLD KEEPS CRASHING WITH SIGSEGV
WITH MYISAM_USE_MMAP ENABLED
Added a condition to call _mi_unmap_file() in case
of compressed records. mi_munmap_file() is called
otherwise.
storage/myisam/mi_packrec.c:
Bug#11756764 48726: MYSQLD KEEPS CRASHING WITH SIGSEGV
WITH MYISAM_USE_MMAP ENABLED
mi_dynmap_file() function, after successfully executing
mmap, stores the total size in info->s->mapped_length
variable. Now, if mi_dynmap_file() is invoked with a size
with an extra 7-byte margin (MEMMAP_EXTRA_MARGIN),
the margin will eventually also get stored in mapped_length.
So, un-mapping function can simply use the value stored in
mapped_length in order to unmap the previously mapped
region.
USING MYISAM_USE_MMAP ON WINDOWS
When OPTIMIZE/REPAIR TABLE is switching to new data file,
old data file is removed while memory mapping is still
active.
With 5.1 implementation of nt_share_delete() it is not
permitted to remove mmaped file.
This fix disables memory mapping for mi_repair() operations.
mysql-test/r/myisam.result:
A test case for BUG#11757032.
mysql-test/t/myisam.test:
A test case for BUG#11757032.
storage/myisam/ha_myisam.cc:
mi_repair*() functions family use file I/O even if memory
mapping is available.
Since mixing mmap I/O and file I/O may cause various artifacts,
memory mapping must be disabled.
storage/myisam/mi_delete_all.c:
Clean-up: do not attempt to remap file after truncate, since
there is nothing to map.
FULLTEXT INDEXES
myisamchk may create incorrect fulltext index for compressed
tables. Incorrect data pointer size was used while creating
fulltext index.
mysql-test/r/myisampack.result:
A test case for BUG#11761180.
mysql-test/t/myisampack.test:
A test case for BUG#11761180.
storage/myisam/ft_boolean_search.c:
rec_reflength on share may have adjustments required for
compressed tables and must be used instead of rec_reflength
on base info.
storage/myisam/ft_nlq_search.c:
rec_reflength on share may have adjustments required for
compressed tables and must be used instead of rec_reflength
on base info.
storage/myisam/mi_check.c:
rec_reflength on share may have adjustments required for
compressed tables and must be used instead of rec_reflength
on base info.
storage/myisam/mi_write.c:
rec_reflength on share may have adjustments required for
compressed tables and must be used instead of rec_reflength
on base info.
Problem: in case of wrong data insert into indexed GEOMETRY fields
(e.g. NULL value for a not NULL field) MyISAM reported
"ERROR 126 (HY000): Incorrect key file for table; try to repair it"
due to misuse of the key deletion function.
Fix: always use R-tree key functions for R-tree based indexes
and B-tree key functions for B-tree based indexes.
mysql-test/r/gis-rtree.result:
Bug#11764487: myisam corruption with insert ignore and invalid spatial data
- test result.
mysql-test/t/gis-rtree.test:
Bug#11764487: myisam corruption with insert ignore and invalid spatial data
- test case.
storage/myisam/mi_update.c:
Bug#11764487: myisam corruption with insert ignore and invalid spatial data
- handling update errors check for HA_ERR_NULL_IN_SPATIAL as well to be
consistent with mi_write();
- always use keyinfo->ck_delete()/ck_insert() instead of _mi_ck_delete()/_mi_ck_write()
to handle index properly, as it may be of B-tree or R-tree type.
storage/myisam/mi_write.c:
Bug#11764487: myisam corruption with insert ignore and invalid spatial data
- always use keyinfo->ck_delete() instead of _mi_ck_delete() to handle
index properly, as it may be of B-tree or R-tree type.
The LGPL license is used in some legacy code, and to
adhere to current licensing polity, we remove those
files that are no longer used, and reorganize the
remaining LGPL code so it will be GPL licensed from
now on.
Note: This patch only removed LGPL licensed files
in MySQL 5.5 and later, and is the third of a
set of patches to remove LGPL from all trees.
(See Bug# 11840513 for details)
The LGPL license is used in some legacy code, and to
adhere to current licensing polity, we remove those
files that are no longer used, and reorganize the
remaining LGPL code so it will be GPL licensed from
now on.
Note: This patch only removed LGPL licensed files
in MySQL 5.1, and is the second of a set of
patches to remove LGPL from all trees.
(See Bug# 11840513 for details)
When executing row-ordered-retrieval index merge,
the handler was cloned, but it used the wrong
memory root, so instead of allocating memory
on the thread/query's mem_root, it used the table's
mem_root, resulting in non released memory in the
table object, and was not freed until the table was
closed.
Solution was to ensure that memory used during cloning
of a handler was allocated from the correct memory root.
This was implemented by fixing handler::clone() to also
take a name argument, so it can be used with partitioning.
And in ha_partition only allocate the ha_partition's ref, and
call the original ha_partition partitions clone() and set at cloned
partitions.
Fix of .bzrignore on Windows with VS 2010
MAP 'REPAIR TABLE' TO RECREATE +ANALYZE FOR ENGINES NOT
SUPPORTING NATIVE REPAIR
Executing 'mysqlcheck --check-upgrade --auto-repair ...' will first issue
'CHECK TABLE FOR UPGRADE' for all tables in the database in order to check if the
tables are compatible with the current version of MySQL. Any tables that are
found incompatible are then upgraded using 'REPAIR TABLE'.
The problem was that some engines (e.g. InnoDB) do not support 'REPAIR TABLE'.
This caused any such tables to be left incompatible. As a result such tables were
not properly fixed by the mysql_upgrade tool.
This patch fixes the problem by first changing 'CHECK TABLE FOR UPGRADE' to return
a different error message if the engine does not support REPAIR. Instead of
"Table upgrade required. Please do "REPAIR TABLE ..." it will report
"Table rebuild required. Please do "ALTER TABLE ... FORCE ..."
Second, the patch changes mysqlcheck to do 'ALTER TABLE ... FORCE' instead of
'REPAIR TABLE' in these cases.
This patch also fixes 'ALTER TABLE ... FORCE' to actually rebuild the table.
This change should be reflected in the documentation. Before this patch,
'ALTER TABLE ... FORCE' was unused (See Bug#11746162)
Test case added to mysqlcheck.test
client/mysqlcheck.c:
Changed mysqlcheck to do 'ALTER TABLE ... FORCE' if
'CHECK TABLE FOR UPGRADE' reports ER_TABLE_NEEDS_REBUILD
and not ER_TABLE_NEEDS_UPGRADE.
mysql-test/r/mysqlcheck.result:
Added regression test.
mysql-test/std_data/bug47205.frm:
InnoDB 5.0 FRM which contains a varchar primary key using
utf8_general_ci. This is an incompatible FRM for 5.5.
mysql-test/t/mysqlcheck.test:
Added regression test.
sql/handler.h:
Added new HA_CAN_REPAIR flag.
sql/share/errmsg-utf8.txt:
Added new error message ER_TABLE_NEEDS_REBUILD
sql/sql_admin.cc:
Changed 'CHECK TABLE FOR UPDATE' to give ER_TABLE_NEEDS_REBUILD
instead of ER_TABLE_NEEDS_UPGRADE if the engine does not support
REPAIR (as indicated by the new HA_CAN_REPAIR flag).
sql/sql_lex.h:
Remove unused ALTER_FORCE flag.
sql/sql_yacc.yy:
Make sure ALTER TABLE ... FORCE recreates the table
by setting the ALTER_RECREATE flag as the ALTER_FORCE
flag was unused.
storage/archive/ha_archive.h:
Added new HA_CAN_REPAIR flag to Archive
storage/csv/ha_tina.h:
Added new HA_CAN_REPAIR flag to CSV
storage/federated/ha_federated.h:
Added new HA_CAN_REPAIR flag to Federated
storage/myisam/ha_myisam.cc:
Added new HA_CAN_REPAIR flag to MyISAM
