innodb_stats_sample_pages
Analysis: If you set the number of analyzed pages
to very low number compared to actual pages on
that table/index it randomly pics those pages
(default 8 pages), this leads to fact that query
after analyze table returns different results. If
the index tree is small, smaller than 10 *
n_sample_pages + total_external_size, then the
estimate is ok. For bigger index trees it is
common that we do not see any borders between
key values in the few pages we pick. But still
there may be n_sample_pages different key values,
or even more. And it just tries to
approximate to n_sample_pages (8).
Fix: (1) Introduced new dynamic configuration variable
innodb_stats_sample_traditional that retains
the current design. Default false.
(2) If traditional sample is not used we use
n_sample_pages = max(min(srv_stats_sample_pages,
index->stat_index_size),
log2(index->stat_index_size)*
srv_stats_sample_pages);
(3) Introduced new dynamic configuration variable
stat_modified_counter (default = 0) if set
sets lower bound for row updates when statistics is re-estimated.
If user has provided upper bound for how many rows needs to be updated
before we calculate new statistics we use minimum of provided value
and 1/16 of table every 16th round. If no upper bound is provided
(srv_stats_modified_counter = 0, default) then calculate new statistics
if 1 / 16 of table has been modified
since the last time a statistics batch was run.
We calculate statistics at most every 16th round, since we may have
a counter table which is very small and updated very often.
@param t table
@return true if the table has changed too much and stats need to be
recalculated
*/
#define DICT_TABLE_CHANGED_TOO_MUCH(t) \
((ib_int64_t) (t)->stat_modified_counter > (srv_stats_modified_counter ? \
ut_min(srv_stats_modified_counter, (16 + (t)->stat_n_rows / 16)) : \
16 + (t)->stat_n_rows / 16))
Merge Facebook commit cd063ab930
authored by Peng Tian from https://github.com/facebook/mysql-5.6
Introduced a new configuration variable innodb_fatal_semaphore_wait_threshold,
it makes the fatal semaphore timeout configurable. Modified original commit
so that no MariaDB server files are changed, instead introduced a new
InnoDB/XtraDB configuration variable.
Its default/min/max vlaues are 600/1/2^32-1 in seconds (it was hardcoded
as 600, now its default value is 600, so the default behavior of this diff
should be no change).
setting of innodb_io_capacity_max
(a) Changed the behaviour so that if you set innodb_io_capacity to a
value > innodb_io_capacity_max that the value is accepted AND
that innodb_io_capacity_max = innodb_io_capacity * 2.
(b) If someone wants to reduce innodb_io_capacity_max and
reduce it below innodb_io_capacity then innodb_io_capacity
should be reduced to the same level as innodb_io_capacity_max.
In both cases give a warning to user.
Merge Facebook commit 154c579b828a60722a7d9477fc61868c07453d08
and e8f0052f9b112dc786bf9b957ed5b16a5749f7fd authored
by Steaphan Greene from https://github.com/facebook/mysql-5.6
Optimize prefix index queries to skip cluster index lookup when possible.
Currently InnoDB will always fetch the clustered index (primary key
index) for all prefix columns in an index, even when the value of a
particular record is smaller than the prefix length. This change
optimizes that case to use the record from the secondary index and avoid
the extra lookup.
Also adds two status vars that track how effective this is:
innodb_secondary_index_triggered_cluster_reads:
Times secondary index lookup triggered cluster lookup.
innodb_secondary_index_triggered_cluster_reads_avoided:
Times prefix optimization avoided triggering cluster lookup.
Merge Facebook commit 4f3e0343fd2ac3fc7311d0ec9739a8f668274f0d
authored by Steaphan Greene from https://github.com/facebook/mysql-5.6
Adds innodb_idle_flush_pct to enable tuning of the page flushing rate
when the system is relatively idle. We care about this, since doing
extra unnecessary flash writes shortens the lifespan of the flash.
Merged Facebook commit ecff018632c6db49bad73d9233c3cdc9f41430e9
authored by Steaphan Greene from https://github.com/facebook/mysql-5.6
This change is to fix: http://bugs.mysql.com/62534
This makes innodb_max_dirty_pages_pct a double with min,default,max values
0.001, 75, 99.999.
This also makes innodb_max_dirty_pages_pct_lwm and adaptive_flushing_lwm
doubles, as these sysvars are inter-dependent.
Added more to the BUFFER POOL AND MEMORY section of SHOW INNODB STATUS:
Percent pages dirty: X.X
This is all n_dirty_pages / used_pages
Percent all pages dirty: X.X
This is all n_dirty_pages / all-pages
Max dirty pages percent: X.X
This is innodb_max_dirty_pages_pct
Also changed all of buf from 2 to 3 digits of precision (%.2f -> %.3f).
Added MAX_STATEMENT_TIME user variable to automaticly kill queries after a given time limit has expired.
- Added timer functions based on pthread_cond_timedwait
- Added kill_handlerton() to signal storage engines about kill/timeout
- Added support for GRANT ... MAX_STATEMENT_TIME=#
- Copy max_statement_time to current user, if stored in mysql.user
- Added status variable max_statement_time_exceeded
- Added KILL_TIMEOUT
- Removed digest hash from performance schema tests as they change all the time.
- Updated test results that changed because of the new user variables or new fields in mysql.user
This functionallity is inspired by work done by Davi Arnaut at twitter.
Test case is copied from Davi's work.
Documentation can be found at
https://kb.askmonty.org/en/how-to-limittimeout-queries/
mysql-test/r/mysqld--help.result:
Updated for new help message
mysql-test/suite/perfschema/r/all_instances.result:
Added new mutex
mysql-test/suite/sys_vars/r/max_statement_time_basic.result:
Added testing of max_statement_time
mysql-test/suite/sys_vars/t/max_statement_time_basic.test:
Added testing of max_statement_time
mysql-test/t/max_statement_time.test:
Added testing of max_statement_time
mysys/CMakeLists.txt:
Added thr_timer
mysys/my_init.c:
mysys/mysys_priv.h:
Added new mutex and condition variables
Added new mutex and condition variables
mysys/thr_timer.c:
Added timer functions based on pthread_cond_timedwait()
This can be compiled with HAVE_TIMER_CREATE to benchmark agains timer_create()/timer_settime()
sql/lex.h:
Added MAX_STATEMENT_TIME
sql/log_event.cc:
Safety fix (timeout should be threated as an interrupted query)
sql/mysqld.cc:
Added support for timers
Added status variable max_statement_time_exceeded
sql/share/errmsg-utf8.txt:
Added ER_QUERY_TIMEOUT
sql/signal_handler.cc:
Added support for KILL_TIMEOUT
sql/sql_acl.cc:
Added support for GRANT ... MAX_STATEMENT_TIME=#
Copy max_statement_time to current user
sql/sql_class.cc:
Added timer functionality to THD.
Added thd_kill_timeout()
sql/sql_class.h:
Added timer functionality to THD.
Added KILL_TIMEOUT
Added max_statement_time variable in similar manner as long_query_time was done.
sql/sql_connect.cc:
Added handling of max_statement_time_exceeded
sql/sql_parse.cc:
Added starting and stopping timers for queries.
sql/sql_show.cc:
Added max_statement_time_exceeded for user/connects status in MariaDB 10.0
sql/sql_yacc.yy:
Added support for GRANT ... MAX_STATEMENT_TIME=# syntax, to be enabled in 10.0
sql/structs.h:
Added max_statement_time user resource
sql/sys_vars.cc:
Added max_statement_time variables
mysql-test/suite/roles/create_and_drop_role_invalid_user_table.test
Removed test as we require all fields in mysql.user table.
scripts/mysql_system_tables.sql
scripts/mysql_system_tables_data.sql
scripts/mysql_system_tables_fix.sql
Updated mysql.user with new max_statement_time field
Merged lp:maria/maria-10.0-galera up to revision 3880.
Added a new functions to handler API to forcefully abort_transaction,
producing fake_trx_id, get_checkpoint and set_checkpoint for XA. These
were added for future possiblity to add more storage engines that
could use galera replication.
Merged lp:maria/maria-10.0-galera up to revision 3879.
Added a new functions to handler API to forcefully abort_transaction,
producing fake_trx_id, get_checkpoint and set_checkpoint for XA. These
were added for future possiblity to add more storage engines that
could use galera replication.
mysqldump:
- Added --log-queries to allow one to disable logging for the dump
sql/log_event.cc:
- Removed setting of enable_slow_log as it's not required anymore.
sql/sql_parse.cc:
- Set enable_slow_log to value of thd->variables.sql_log_slow as this will speed up tests if slow log is disabled.
- opt_log_slow_admin_statements can now only disable slow log, not enable it.
sql/sql_explain.cc:
- Minor cleanup
Other things:
- Added sql_log_slow to system variables.
- Changed opt_slow_log to global_system_variables.sql_log_slow in all files
- Updated tests to reflect changes
Reason for the problem was that the hash of changed files in the key cache was too small (was 128). Fixed by making the hash size larger and changeable.
- Introduced key-cache-file-hash-size (default 512) for MyISAM and aria_pagecache_file_hash_size (default 512) for Aria.
- Added new status variable "Feature_delay_key_write" which counts number of tables opened that are using delay_key_write
mysql-test/r/features.result:
Added test of Feature_delay_key_write
mysql-test/r/key_cache.result:
Updated tests as the number of blocks has changed
mysql-test/r/mysqld--help.result:
Updated result
mysql-test/suite/maria/maria3.result:
Updated result
mysql-test/suite/sys_vars/r/key_cache_file_hash_size_basic.result:
Test new variable
mysql-test/suite/sys_vars/t/aria_pagecache_file_hash_size_basic.test:
Test new variable
mysql-test/suite/sys_vars/t/key_cache_file_hash_size_basic.test:
Test new variable
mysql-test/t/features.test:
Added test of Feature_delay_key_write
mysql-test/t/key_cache.test:
Updated tests as the number of blocks has changed
mysys/mf_keycache.c:
Made CHANGED_BLOCKS_HASH dynamic
sql/handler.cc:
Updated call to init_key_cache()
sql/mysqld.cc:
Added "Feature_delay_key_write"
Added support for key-cache-file-hash-size
sql/mysqld.h:
Added support for key-cache-file-hash-size
sql/sql_class.h:
Added feature_files_opened_with_delayed_keys
sql/sys_vars.cc:
Added key_cache_file_hash_size
storage/maria/ha_maria.cc:
Added pagecache_file_hash_size
Added counting of files with delay_key_write
storage/maria/ma_checkpoint.c:
Fixed compiler warning
storage/maria/ma_pagecache.c:
Made PAGECACHE_CHANGED_BLOCKS_HASH into a variable
storage/maria/ma_pagecache.h:
Made PAGECACHE_CHANGED_BLOCKS_HASH into a variable
storage/maria/ma_rt_test.c:
Updated parameters for init_pagecache()
storage/maria/ma_test1.c:
Updated parameters for init_pagecache()
storage/maria/ma_test2.c:
Updated parameters for init_pagecache()
storage/maria/ma_test3.c:
Updated parameters for init_pagecache()
storage/maria/maria_chk.c:
Updated parameters for init_pagecache()
storage/maria/maria_ftdump.c:
Updated parameters for init_pagecache()
storage/maria/maria_pack.c:
Updated parameters for init_pagecache()
storage/maria/maria_read_log.c:
Updated parameters for init_pagecache()
storage/maria/unittest/ma_pagecache_consist.c:
Updated parameters for init_pagecache()
storage/maria/unittest/ma_pagecache_rwconsist.c:
Updated parameters for init_pagecache()
storage/maria/unittest/ma_pagecache_rwconsist2.c:
Updated parameters for init_pagecache()
storage/maria/unittest/ma_pagecache_single.c:
Updated parameters for init_pagecache()
storage/maria/unittest/ma_test_loghandler-t.c:
Updated parameters for init_pagecache()
storage/maria/unittest/ma_test_loghandler_first_lsn-t.c:
Updated parameters for init_pagecache()
storage/maria/unittest/ma_test_loghandler_max_lsn-t.c:
Updated parameters for init_pagecache()
storage/maria/unittest/ma_test_loghandler_multigroup-t.c:
Updated parameters for init_pagecache()
storage/maria/unittest/ma_test_loghandler_multithread-t.c:
Updated parameters for init_pagecache()
storage/maria/unittest/ma_test_loghandler_noflush-t.c:
Updated parameters for init_pagecache()
storage/maria/unittest/ma_test_loghandler_nologs-t.c:
Updated parameters for init_pagecache()
storage/maria/unittest/ma_test_loghandler_pagecache-t.c:
Updated parameters for init_pagecache()
storage/maria/unittest/ma_test_loghandler_purge-t.c:
Updated parameters for init_pagecache()
storage/myisam/ha_myisam.cc:
Added counting of files with delay_key_write
storage/myisam/mi_check.c:
Updated call to init_key_cache()
storage/myisam/mi_test1.c:
Updated call to init_key_cache()
storage/myisam/mi_test2.c:
Updated call to init_key_cache()
storage/myisam/mi_test3.c:
Updated call to init_key_cache()
storage/myisam/mi_test_all.sh:
Fixed broken test
storage/myisam/myisam_ftdump.c:
Updated call to init_key_cache()
storage/myisam/myisamchk.c:
Updated call to init_key_cache()
storage/myisam/myisamlog.c:
Updated call to init_key_cache()
it checked that the default is lz4. Which only worked on systems that
had lz4 and did not have lzo. Now it checks for the default to be zlib,
which works on systems that has neither lz4 or lzo. Like our package
builders in buildbot. This is intentional, we don't want introduce
additional dependencies (lz4, lzo) for our packages just yet.
This can (and will) be reconsidered, and this test can (and will)
be updated again.
Adapt default_tmp_storage_engine implementation from mysql-5.6
New feature (as compared to 5.6), default_tmp_storage_engine=NULL
means that temporary tables will use default_storage_engine value.
This makes the behavior backward compatible.
(Inadequate background LRU flushing for write workloads with InnoDB compression).
If InnoDB compression is used and the workload has writes, the
following situation is possible. The LRU flusher issues an LRU flush
request for an instance. buf_do_LRU_batch decides to perform
unzip_LRU eviction and this eviction might fully satisfy the
request. Then buf_flush_LRU_tail checks the number of flushed pages in
the last iteration, finds it to be zero, and wrongly decides not to
flush that instance anymore.
Fixed by maintaining unzip_LRU eviction counter in struct
flush_counter_t variables, and checking it in buf_flush_LRU_tail when
deciding whether to stop flushing the current instance.
Added test cases for new configuration files to get mysql-test-run suite sys_vars
to pass. Fix some small errors.
innodb_force_primary_key default off. If option is true, create table without
primary key or unique key where all keyparts are NOT NULL is not
accepted. Instead an error message is printed. Variable value can
be changed with set global innodb_force_primary_key = <value>.
Clean up and improve the parallel implementation code, mainly related to
scheduling of work to threads and handling of stop and errors.
Fix a lot of bugs in various corner cases that could lead to crashes or
corruption.
Fix that a single replication domain could easily grab all worker threads and
stall all other domains; now a configuration variable
--slave-domain-parallel-threads allows to limit the number of
workers.
Allow next event group to start as soon as previous group begins the commit
phase (as opposed to when it ends it); this allows multiple event groups on
the slave to participate in group commit, even when no other opportunities for
parallelism are available.
Various fixes:
- Fix some races in the rpl.rpl_parallel test case.
- Fix an old incorrect assertion in Log_event iocache read.
- Fix repeated malloc/free of wait_for_commit and rpl_group_info objects.
- Simplify wait_for_commit wakeup logic.
- Fix one case in queue_for_group_commit() where killing one thread would
fail to correctly signal the error to the next, causing loss of the
transaction after slave restart.
- Fix leaking of pthreads (and their allocated stack) due to missing
PTHREAD_CREATE_DETACHED attribute.
- Fix how one batch of group-committed transactions wait for the previous
batch before starting to execute themselves. The old code had a very
complex scheduling where the first transaction was handled differently,
with subtle bugs in corner cases. Now each event group is always scheduled
for a new worker (in a round-robin fashion amongst available workers).
Keep a count of how many transactions have started to commit, and wait for
that counter to reach the appropriate value.
- Fix slave stop to wait for all workers to actually complete processing;
before, the wait was for update of last_committed_sub_id, which happens a
bit earlier, and could leave worker threads potentially accessing bits of
the replication state that is no longer valid after slave stop.
- Fix a couple of places where the test suite would kill a thread waiting
inside enter_cond() in connection with debug_sync; debug_sync + kill can
crash in rare cases due to a race with mysys_var_current_mutex in this
case.
- Fix some corner cases where we had enter_cond() but no exit_cond().
- Fix that we could get failure in wait_for_prior_commit() but forget to flag
the error with my_error().
- Fix slave stop (both for normal stop and stop due to error). Now, at stop
we pick a specific safe point (in terms of event groups executed) and make
sure that all event groups before that point are executed to completion,
and that no event group after start executing; this ensures a safe place to
restart replication, even for non-transactional stuff/DDL. In error stop,
make sure that all prior event groups are allowed to execute to completion,
and that any later event groups that have started are rolled back, if
possible. The old code could leave eg. T1 and T3 committed but T2 not, or
it could even leave half a transaction not rolled back in some random
worker, which would cause big problems when that worker was later reused
after slave restart.
- Fix the accounting of amount of events queued for one worker. Before, the
amount was reduced immediately as soon as the events were dequeued (which
happens all at once); this allowed twice the amount of events to be queued
in memory for each single worker, which is not what users would expect.
- Fix that an error set during execution of one event was sometimes not
cleared before executing the next, causing problems with the error
reporting.
- Fix incorrect handling of thd->killed in worker threads.
