database/mariadb
1
0
Fork 0
mirror of https://github.com/MariaDB/server.git synced 2025-07-30 16:24:05 +03:00
Code Activity

Added some fixes that should make MyISAM & Aria REPAIR work with more than 4G records

Browse Source 
- If one specifies --force twice to myisamchk and aria_chk, then we will try to finnish the repair even if sort_buffer would be too small.
  This was done by dynamically allocate buffer handler objects as long as memory lasts.
- New option for myisamchk and aria_chk: create-missing-keys
- Changed default size of myisam_sort_buffer_size from 8M to 128M.
- Changed default size of sort_buffer_size in aria_chk from 128M to 256M.
- Increased information in error message about 'sort_buffer_size' beeing to small.
- Print also to 'show warnings' if repair was retried.
- Increased size of internal sort-buffer-readers from 16K to 128K
- Changed printing of 'number of records' to use %ll instead of casting to long
- Changed buffer sizes for myisam and aria to use MY_ALIGN_DOWN() to get same number of bytes allocated on different machines.





include/my_global.h:
  Added MY_ALIGN_DOWN() to get previous alignment (for big memory areas)
include/myisam.h:
  Increased size of types to be able to handle more records
include/myisamchk.h:
  Increased size of types to be able to handle more records
  Added T_FORCE_SORT_MEMORY to force repair to work even if sort_buffer would not be big enough
mysql-test/r/myisam.result:
  Updated result
mysql-test/r/mysqld--help.result:
  Updated result
mysql-test/r/repair.result:
  Updated result
mysql-test/suite/maria/maria.result:
  Added test cases for some fixed bugs in MyISAM to verify that Aria doesn't have them too.
mysql-test/suite/maria/maria.test:
  Added test cases for some fixed bugs in MyISAM to verify that Aria doesn't have them too.
mysql-test/suite/maria/maria3.result:
  Updated result after sort buffer size increase
mysql-test/suite/sys_vars/r/aria_sort_buffer_size_basic.result:
  Updated result after sort buffer size increase
mysql-test/suite/sys_vars/r/myisam_sort_buffer_size_basic.result:
  Updated result after sort buffer size increase
mysql-test/suite/sys_vars/t/myisam_sort_buffer_size_basic.test:
  Updated result after sort buffer size increase
mysql-test/t/myisam.test:
  Fixed error messages to not print system specific data
mysql-test/t/repair.test:
  Fixed error messages to not print system specific data
storage/maria/ha_maria.cc:
  Print also to 'show warnings' if repair was retried
  Changed default size of sort_buffer_size from 128M to 256M (same as in mysqld)
storage/maria/ma_check.c:
  Renamed USE_BUFFER_INIT -> PAGE_BUFFER_INIT
storage/maria/ma_sort.c:
  Increased size of internal sort-buffer-readers from 16K to 128K
  Increased size of types to be able to handle more records
  Added support for T_FORCE_SORT_MEMORY
  Don't allocate too many extra BUFFPEK at a time (they are probably not needed)
  Improved error message for "sort_buffer_size is too small"
  Changed printing of 'number of records' to use %ll instead of casting to long
  Fixed bug where maria_update_key_parts() was called too early.
  Fixed bug in detecting result from read_to_buffer().
  Added 'out of memory' checking when calling 'alloc_dynamic()'.
storage/maria/maria_chk.c:
  Added --create-missing-keys
  If one specifies --force twice then we will try to finnish the repair even if sort_buffer would be too small.
  check_param.sort_buffer_length varialble was used with wrong type.
storage/maria/maria_def.h:
  Increased size of types to be able to handle more records
  Use MY_ALIGN_DOWN() to get same number of bytes allocated on different machines
  Renamed USE_BUFFER_INIT -> PAGE_BUFFER_INIT
storage/maria/maria_ftdump.c:
  Renamed USE_BUFFER_INIT -> PAGE_BUFFER_INIT
storage/maria/maria_read_log.c:
  Use PAGE_BUFFER_INIT for page cache
storage/myisam/ha_myisam.cc:
  Changed default size of myisam_sort_buffer_size from 8M to 128M
storage/myisam/mi_check.c:
  Renamed USE_BUFFER_INIT -> KEY_BUFFER_INIT
storage/myisam/myisam_ftdump.c:
  Renamed USE_BUFFER_INIT -> KEY_BUFFER_INIT
storage/myisam/myisamchk.c:
  Added --create-missing-keys
  If one specifies --force twice then we will try to finnish the repair even if sort_buffer would be too small.
  check_param.sort_buffer_length varialble was used with wrong type.
  Renamed USE_BUFFER_INIT -> KEY_BUFFER_INIT
storage/myisam/myisamdef.h:
  Increased SORT_BUFFER_INIT to 64M
  (speeds up repair a lot and most machines have nowadays a lot of memory)
  Use MY_ALIGN_DOWN() to get same number of bytes allocated on different machines
  Renamed USE_BUFFER_INIT -> KEY_BUFFER_INIT
storage/myisam/sort.c:
  Increased size of internal sort-buffer-readers from 16K to 128K
  Increased size of types to be able to handle more records
  Added support for T_FORCE_SORT_MEMORY
  Don't allocate too many extra BUFFPEK at a time (they are probably not needed)
  Improved error message for "sort_buffer_size is too small"
  Changed printing of 'number of records' to use %ll instead of casting to long
  Fixed bug in detecting result from read_to_buffer().
  Added 'out of memory' checking when calling 'alloc_dynamic()'.
This commit is contained in:
Michael Widenius
2013-05-10 12:32:34 +03:00
parent e5a0daae5a
commit 53d44ad18b
27 changed files with 651 additions and 326 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
mysql-test/r/myisam.result
View File

@ -2355,7 +2355,7 @@ Warnings:
Warning 1292 Truncated incorrect myisam_sort_buffer_size value: '4'
REPAIR TABLE t1;
Table Op Msg_type Msg_text
test.t1 repair error myisam_sort_buffer_size is too small
test.t1 repair error myisam_sort_buffer_size is too small. X
test.t1 repair warning Number of rows changed from 0 to 7168
test.t1 repair status OK
SET myisam_repair_threads=2;
@ -2424,7 +2424,7 @@ INSERT INTO t1 VALUES
('0'),('0'),('0'),('0'),('0'),('0'),('0'),('0'),('0'),('0'),
('0'),('0'),('0'),('0'),('0'),('0'),('0');
Warnings:
Error 1034 myisam_sort_buffer_size is too small
Error 1034 myisam_sort_buffer_size is too small. X
Error 1034 Number of rows changed from 0 to 157
SET myisam_sort_buffer_size=@@global.myisam_sort_buffer_size;
INSERT INTO t1 VALUES('1');

2
mysql-test/r/mysqld--help.result
View File

@ -1045,7 +1045,7 @@ myisam-max-sort-file-size 9223372036853727232
myisam-mmap-size 18446744073709551615
myisam-recover-options DEFAULT
myisam-repair-threads 1
myisam-sort-buffer-size 8388608
myisam-sort-buffer-size 134216704
myisam-stats-method nulls_unequal
myisam-use-mmap FALSE
net-buffer-length 16384

4
mysql-test/r/repair.result
View File

@ -77,12 +77,12 @@ INSERT INTO t1 VALUES
('0'),('0'),('0'),('0'),('0'),('0'),('0'),('0'),('0'),('0'),
('0'),('0'),('0'),('0'),('0'),('0'),('0');
Warnings:
Error 1034 myisam_sort_buffer_size is too small
Error 1034 myisam_sort_buffer_size is too small. X
Error 1034 Number of rows changed from 0 to 157
SET myisam_repair_threads=2;
REPAIR TABLE t1;
Table Op Msg_type Msg_text
test.t1 repair error myisam_sort_buffer_size is too small
test.t1 repair error myisam_sort_buffer_size is too small. X
test.t1 repair warning Number of rows changed from 0 to 157
test.t1 repair status OK
SET myisam_repair_threads=@@global.myisam_repair_threads;

62
mysql-test/suite/maria/maria.result
View File

@ -2679,3 +2679,65 @@ select count(*) from t1;
count(*)
13
drop table t1;
#
# BUG#47444 - --myisam_repair_threads > 1 can result in all index
# cardinalities=1
#
SET aria_repair_threads=2;
SET aria_sort_buffer_size=8192;
CREATE TABLE t1(a CHAR(255), KEY(a), KEY(a), KEY(a));
INSERT INTO t1 VALUES (0),(1),(2),(3),(4),(5),(6),(7),(8),(9),(0),(1),(2),(3);
REPAIR TABLE t1;
Table Op Msg_type Msg_text
test.t1 repair status OK
SELECT CARDINALITY FROM INFORMATION_SCHEMA.STATISTICS WHERE TABLE_SCHEMA='test' AND TABLE_NAME='t1';
CARDINALITY
14
14
14
CHECK TABLE t1;
Table Op Msg_type Msg_text
test.t1 check status OK
DROP TABLE t1;
SET aria_sort_buffer_size=@@global.aria_sort_buffer_size;
SET aria_repair_threads=@@global.aria_repair_threads;
#
# BUG#47073 - valgrind errs, corruption,failed repair of partition,
# low myisam_sort_buffer_size
#
CREATE TABLE t1(a INT, b CHAR(10), KEY(a), KEY(b));
INSERT INTO t1 VALUES(1,'0'),(2,'0'),(3,'0'),(4,'0'),(5,'0'),
(6,'0'),(7,'0');
INSERT INTO t1 SELECT a+10,b FROM t1;
INSERT INTO t1 SELECT a+20,b FROM t1;
INSERT INTO t1 SELECT a+40,b FROM t1;
INSERT INTO t1 SELECT a+80,b FROM t1;
INSERT INTO t1 SELECT a+160,b FROM t1;
INSERT INTO t1 SELECT a+320,b FROM t1;
INSERT INTO t1 SELECT a+640,b FROM t1;
INSERT INTO t1 SELECT a+1280,b FROM t1;
INSERT INTO t1 SELECT a+2560,b FROM t1;
INSERT INTO t1 SELECT a+5120,b FROM t1;
SET aria_sort_buffer_size=4096;
REPAIR TABLE t1;
Table Op Msg_type Msg_text
test.t1 repair error aria_sort_buffer_size is too small. X
test.t1 repair error Create index by sort failed
test.t1 repair info Retrying repair with keycache
test.t1 repair status OK
CHECK TABLE t1;
Table Op Msg_type Msg_text
test.t1 check status OK
SET aria_repair_threads=2;
REPAIR TABLE t1;
Table Op Msg_type Msg_text
test.t1 repair error aria_sort_buffer_size is too small. X
test.t1 repair error Create index by sort failed
test.t1 repair info Retrying repair with keycache
test.t1 repair status OK
CHECK TABLE t1;
Table Op Msg_type Msg_text
test.t1 check status OK
SET aria_repair_threads=@@global.aria_repair_threads;
SET aria_sort_buffer_size=@@global.aria_sort_buffer_size;
DROP TABLE t1;

46
mysql-test/suite/maria/maria.test
View File

@ -1963,6 +1963,52 @@ unlock tables;
select count(*) from t1;
drop table t1;
--echo #
--echo # BUG#47444 - --myisam_repair_threads > 1 can result in all index
--echo # cardinalities=1
--echo #
SET aria_repair_threads=2;
SET aria_sort_buffer_size=8192;
CREATE TABLE t1(a CHAR(255), KEY(a), KEY(a), KEY(a));
INSERT INTO t1 VALUES (0),(1),(2),(3),(4),(5),(6),(7),(8),(9),(0),(1),(2),(3);
--replace_regex /Current aria_sort_buffer_size.*/X/
REPAIR TABLE t1;
SELECT CARDINALITY FROM INFORMATION_SCHEMA.STATISTICS WHERE TABLE_SCHEMA='test' AND TABLE_NAME='t1';
CHECK TABLE t1;
DROP TABLE t1;
SET aria_sort_buffer_size=@@global.aria_sort_buffer_size;
SET aria_repair_threads=@@global.aria_repair_threads;
--echo #
--echo # BUG#47073 - valgrind errs, corruption,failed repair of partition,
--echo # low myisam_sort_buffer_size
--echo #
CREATE TABLE t1(a INT, b CHAR(10), KEY(a), KEY(b));
INSERT INTO t1 VALUES(1,'0'),(2,'0'),(3,'0'),(4,'0'),(5,'0'),
(6,'0'),(7,'0');
INSERT INTO t1 SELECT a+10,b FROM t1;
INSERT INTO t1 SELECT a+20,b FROM t1;
INSERT INTO t1 SELECT a+40,b FROM t1;
INSERT INTO t1 SELECT a+80,b FROM t1;
INSERT INTO t1 SELECT a+160,b FROM t1;
INSERT INTO t1 SELECT a+320,b FROM t1;
INSERT INTO t1 SELECT a+640,b FROM t1;
INSERT INTO t1 SELECT a+1280,b FROM t1;
INSERT INTO t1 SELECT a+2560,b FROM t1;
INSERT INTO t1 SELECT a+5120,b FROM t1;
SET aria_sort_buffer_size=4096;
--replace_regex /Current aria_sort_buffer_size.*/X/
REPAIR TABLE t1;
CHECK TABLE t1;
SET aria_repair_threads=2;
# May report different values depending on threads activity.
--replace_regex /Current aria_sort_buffer_size.*/X/
REPAIR TABLE t1;
CHECK TABLE t1;
SET aria_repair_threads=@@global.aria_repair_threads;
SET aria_sort_buffer_size=@@global.aria_sort_buffer_size;
DROP TABLE t1;
#
# End of test
#

2
mysql-test/suite/maria/maria3.result
View File

@ -317,7 +317,7 @@ aria_pagecache_division_limit 100
aria_page_checksum OFF
aria_recover NORMAL
aria_repair_threads 1
aria_sort_buffer_size 134217728
aria_sort_buffer_size 268434432
aria_stats_method nulls_unequal
aria_sync_log_dir NEWFILE
show status like 'aria%';

22
mysql-test/suite/sys_vars/r/aria_sort_buffer_size_basic.result
View File

@ -1,30 +1,34 @@
SET @start_global_value = @@global.aria_sort_buffer_size;
select @@global.aria_sort_buffer_size;
@@global.aria_sort_buffer_size
134217728
268434432
select @@session.aria_sort_buffer_size;
@@session.aria_sort_buffer_size
134217728
268434432
show global variables like 'aria_sort_buffer_size';
Variable_name Value
aria_sort_buffer_size 134217728
aria_sort_buffer_size 268434432
show session variables like 'aria_sort_buffer_size';
Variable_name Value
aria_sort_buffer_size 134217728
aria_sort_buffer_size 268434432
select * from information_schema.global_variables where variable_name='aria_sort_buffer_size';
VARIABLE_NAME VARIABLE_VALUE
ARIA_SORT_BUFFER_SIZE 134217728
ARIA_SORT_BUFFER_SIZE 268434432
select * from information_schema.session_variables where variable_name='aria_sort_buffer_size';
VARIABLE_NAME VARIABLE_VALUE
ARIA_SORT_BUFFER_SIZE 134217728
ARIA_SORT_BUFFER_SIZE 268434432
set global aria_sort_buffer_size=10;
Warnings:
Warning 1292 Truncated incorrect aria_sort_buffer_size value: '10'
select @@global.aria_sort_buffer_size;
@@global.aria_sort_buffer_size
10
4096
set session aria_sort_buffer_size=10;
Warnings:
Warning 1292 Truncated incorrect aria_sort_buffer_size value: '10'
select @@session.aria_sort_buffer_size;
@@session.aria_sort_buffer_size
10
4096
set global aria_sort_buffer_size=1.1;
ERROR 42000: Incorrect argument type to variable 'aria_sort_buffer_size'
set session aria_sort_buffer_size=1e1;
@ -36,7 +40,7 @@ Warnings:
Warning 1292 Truncated incorrect aria_sort_buffer_size value: '0'
select @@global.aria_sort_buffer_size;
@@global.aria_sort_buffer_size
4
4096
set session aria_sort_buffer_size=cast(-1 as unsigned int);
select @@session.aria_sort_buffer_size;
@@session.aria_sort_buffer_size

20
mysql-test/suite/sys_vars/r/myisam_sort_buffer_size_basic.result
View File

@ -1,11 +1,11 @@
SET @start_global_value = @@global.myisam_sort_buffer_size ;
SELECT @start_global_value;
@start_global_value
8388608
134216704
SET @start_session_value = @@session.myisam_sort_buffer_size ;
SELECT @start_session_value;
@start_session_value
8388608
134216704
'#--------------------FN_DYNVARS_005_01-------------------------#'
SET @@global.myisam_sort_buffer_size = 100;
Warnings:
@ -13,22 +13,22 @@ Warning 1292 Truncated incorrect myisam_sort_buffer_size value: '100'
SET @@global.myisam_sort_buffer_size = DEFAULT;
SELECT @@global.myisam_sort_buffer_size ;
@@global.myisam_sort_buffer_size
8388608
134216704
SET @@session.myisam_sort_buffer_size = 200;
Warnings:
Warning 1292 Truncated incorrect myisam_sort_buffer_size value: '200'
SET @@session.myisam_sort_buffer_size = DEFAULT;
SELECT @@session.myisam_sort_buffer_size ;
@@session.myisam_sort_buffer_size
8388608
134216704
'#--------------------FN_DYNVARS_005_02-------------------------#'
SET @@global.myisam_sort_buffer_size = DEFAULT;
SELECT @@global.myisam_sort_buffer_size = 8388608;
@@global.myisam_sort_buffer_size = 8388608
SELECT @@global.myisam_sort_buffer_size = 134216704;
@@global.myisam_sort_buffer_size = 134216704
1
SET @@session.myisam_sort_buffer_size = DEFAULT;
SELECT @@session.myisam_sort_buffer_size = 8388608;
@@session.myisam_sort_buffer_size = 8388608
SELECT @@session.myisam_sort_buffer_size = 134216704;
@@session.myisam_sort_buffer_size = 134216704
1
'#--------------------FN_DYNVARS_005_03-------------------------#'
SET @@global.myisam_sort_buffer_size = 4;
@ -187,8 +187,8 @@ ERROR 42S22: Unknown column 'myisam_sort_buffer_size' in 'field list'
SET @@global.myisam_sort_buffer_size = @start_global_value;
SELECT @@global.myisam_sort_buffer_size ;
@@global.myisam_sort_buffer_size
8388608
134216704
SET @@session.myisam_sort_buffer_size = @start_session_value;
SELECT @@session.myisam_sort_buffer_size ;
@@session.myisam_sort_buffer_size
8388608
134216704

4
mysql-test/suite/sys_vars/t/myisam_sort_buffer_size_basic.test
View File

@ -60,10 +60,10 @@ SELECT @@session.myisam_sort_buffer_size ;
########################################################################
SET @@global.myisam_sort_buffer_size = DEFAULT;
SELECT @@global.myisam_sort_buffer_size = 8388608;
SELECT @@global.myisam_sort_buffer_size = 134216704;
SET @@session.myisam_sort_buffer_size = DEFAULT;
SELECT @@session.myisam_sort_buffer_size = 8388608;
SELECT @@session.myisam_sort_buffer_size = 134216704;
--echo '#--------------------FN_DYNVARS_005_03-------------------------#'

2
mysql-test/t/myisam.test
View File

@ -1600,6 +1600,7 @@ INSERT INTO t1 SELECT a+1280,b FROM t1;
INSERT INTO t1 SELECT a+2560,b FROM t1;
INSERT INTO t1 SELECT a+5120,b FROM t1;
SET myisam_sort_buffer_size=4;
--replace_regex /Current myisam_sort_buffer_size.*/X/
REPAIR TABLE t1;
SET myisam_repair_threads=2;
@ -1648,6 +1649,7 @@ DROP TABLE t1, t2, t3;
CREATE TABLE t1(a CHAR(255), KEY(a));
SELECT * FROM t1, t1 AS a1;
SET myisam_sort_buffer_size=4;
--replace_regex /Current myisam_sort_buffer_size.*/X/
INSERT INTO t1 VALUES
('0'),('0'),('0'),('0'),('0'),('0'),('0'),('0'),('0'),('0'),
('0'),('0'),('0'),('0'),('0'),('0'),('0'),('0'),('0'),('0'),

2
mysql-test/t/repair.test
View File

@ -61,6 +61,7 @@ DROP TABLE t1;
#
CREATE TABLE t1(a CHAR(255), KEY(a));
SET myisam_sort_buffer_size=4096;
--replace_regex /Current myisam_sort_buffer_size.*/X/
INSERT INTO t1 VALUES
('0'),('0'),('0'),('0'),('0'),('0'),('0'),('0'),('0'),('0'),
('0'),('0'),('0'),('0'),('0'),('0'),('0'),('0'),('0'),('0'),
@ -79,6 +80,7 @@ INSERT INTO t1 VALUES
('0'),('0'),('0'),('0'),('0'),('0'),('0'),('0'),('0'),('0'),
('0'),('0'),('0'),('0'),('0'),('0'),('0');
SET myisam_repair_threads=2;
--replace_regex /Current myisam_sort_buffer_size.*/X/
REPAIR TABLE t1;
SET myisam_repair_threads=@@global.myisam_repair_threads;
SET myisam_sort_buffer_size=@@global.myisam_sort_buffer_size;