1
0
mirror of https://github.com/MariaDB/server.git synced 2025-12-03 05:41:09 +03:00
Commit Graph

2165 Commits

Author SHA1 Message Date
Sachin Setiya
3cecb1bab3 Merge tag 'mariadb-10.0.33' into bb-10.0-galera 2017-11-03 12:34:05 +05:30
Alexander Barkov
835cbbcc7b Merge remote-tracking branch 'origin/bb-10.2-ext' into 10.3
TODO: enable MDEV-13049 optimization for 10.3
2017-10-30 20:47:39 +04:00
Alexander Barkov
003cb2f424 Merge remote-tracking branch 'origin/10.2' into bb-10.2-ext 2017-10-30 16:42:46 +04:00
Marko Mäkelä
771305b21d MDEV-12569 InnoDB suggests filing bugs at MySQL bug tracker
Replace all references in InnoDB error log messages
to bugs.mysql.com with references to https://jira.mariadb.org/.
2017-10-26 14:24:03 +03:00
Vicențiu Ciorbaru
4ef64e01a7 5.6.38 2017-10-25 21:35:33 +03:00
Jan Lindström
b23a109695 MDEV-11025: Make number of page cleaner threads variable dynamic
New test cases
        innodb-page-cleaners

Modified test cases
        innodb_page_cleaners_basic

New function buf_flush_set_page_cleaner_thread_cnt
    Increase or decrease the amount of page cleaner worker threads.
    In case of increase this function creates based on current
    abount and requested amount how many new threads should be
    created. In case of decrease this function sets up the
    requested amount of threads and uses is_requested event
    to signal workers. Then we wait until all new treads
    are started, old threads that should exit signal
    is_finished or shutdown has marked that page cleaner
    should finish.

buf_flush_page_cleaner_worker
        Store current thread id and thread_no and then signal
        event is_finished. If number of used page cleaner threads
        decrease we shut down those threads that have thread_no
        greater or equal than number of page configured page
        cleaners - 1 (note that there will be always page cleaner
        coordinator). Before exiting we signal is_finished.

New function innodb_page_cleaners_threads_update
        Update function for innodb-page-cleaners system variable.

innobase_start_or_create_for_mysql
        If more than one page cleaner threads is configured
        we use new function buf_flush_set_page_cleaner_thread_cnt
        to set up the requested threads (-1 coordinator).
2017-10-24 19:12:59 +03:00
Sergei Golubchik
e0a1c745ec Merge branch '10.1' into 10.2 2017-10-24 14:53:18 +02:00
Sergei Golubchik
9d2e2d7533 Merge branch '10.0' into 10.1 2017-10-22 13:03:41 +02:00
Jan Lindström
8da6b4ef52 Merge tag 'mariadb-5.5.58' into 5.5-galera 2017-10-19 09:06:17 +03:00
Alexander Barkov
30e7d6709f Merge remote-tracking branch 'origin/10.2' into bb-10.2-ext 2017-10-18 14:11:55 +04:00
Marko Mäkelä
3bc094d32a Remove dead code for pushing down LIMIT to InnoDB FULLTEXT INDEX queries
MySQL 5.7 added code to push down the LIMIT to fulltext search
in InnoDB:

commit 2cd0ebf97e1b265e2282d7ceb5d8dfb663ffc48f
Author: Thirunarayanan Balathandayuthapani <thirunarayanan.balathandayuth@oracle.com>
Date:   Fri May 27 13:49:28 2016 +0530

    Bug #22709692   FTS QUERY EXCEEDS RESULT CACHE LIMIT

The code was disabled when MySQL 5.7.9 was merged to MariaDB 10.2.2.

We shall remove the disabled code and unnecessary variables.
2017-10-18 09:07:53 +03:00
Marko Mäkelä
dfd010ef90 MDEV-14086 Setting innodb_buffer_pool_load_now or innodb_buffer_load_abort will crash if innodb_read_only
When MySQL 5.6.10 introduced innodb_read_only mode, it skipped the
creation of the InnoDB buffer pool dump/restore subsystem in that mode.
Attempts to set the variable innodb_buf_pool_dump_now would have
no effect in innodb_read_only mode, but the corresponding condition
was forgotten in from the other two update functions.

MySQL 5.7.20 would fix the innodb_buffer_pool_load_now,
but not innodb_buffer_pool_load_abort. Let us fix both in MariaDB.
2017-10-18 06:35:11 +03:00
Sergei Golubchik
d76f5774fe MDEV-13459 Warnings, when compiling with gcc-7.x
mostly caused by -Wimplicit-fallthrough
2017-10-17 07:37:39 +02:00
Jan Lindström
a4fa940bad MDEV-11336: Enable defragmentation on 10.2 when tests pass
Problem was that we could take page latches on different
order than wat is entitled with SX-lock. To follow the
latching order defined in WL#6326, acquire index->lock X-latch.
This entitles us to acquire page latches in any order for the index.

btr0btr.cc
	Document latch rules before and after MariaDB 10.2.2

sync0rw.cc
	Document latch compatibility rules better.

btr_defragment_merge_pages
	Fix parameter value.

btr_defragment_thread
	Acquire X-lock to dict_index_t::lock before restoring
	cursor position and continuing defragmentation.

ha_innobase::optimize
	Restore defragment feature.

Testing
	Add GIS-index and FT-index to table being defragmented.

	Defragmentation is not done to GIS-indexes and FT auxiliary
	tables.
2017-10-12 12:56:20 +03:00
Jan Lindström
fc9ff69578 MDEV-13838: Wrong result after altering a partitioned table
Reverted incorrect changes done on MDEV-7367 and MDEV-9469. Fixes properly
also related bugs:

MDEV-13668: InnoDB unnecessarily rebuilds table when renaming a column and adding index
MDEV-9469: 'Incorrect key file' on ALTER TABLE
MDEV-9548: Alter table (renaming and adding index) fails with "Incorrect key file for table"
MDEV-10535: ALTER TABLE causes standalone/wsrep cluster crash
MDEV-13640: ALTER TABLE CHANGE and ADD INDEX on auto_increment column fails with "Incorrect key file for table..."

Root cause for all these bugs is the fact that MariaDB .frm file
can contain virtual columns but InnoDB dictionary does not and
previous fixes were incorrect or unnecessarily forced table
rebuilt. In index creation key_part->fieldnr can be bigger than
number of columns in InnoDB data dictionary. We need to skip not
stored fields when calculating correct column number for InnoDB
data dictionary.

dict_table_get_col_name_for_mysql
        Remove

innobase_match_index_columns
        Revert incorrect change done on MDEV-7367

innobase_need_rebuild
        Remove unnecessary rebuild force when column is renamed.

innobase_create_index_field_def
        Calculate InnoDB column number correctly and remove
        unnecessary column name set.

innobase_create_index_def, innobase_create_key_defs
        Remove unneeded fields parameter. Revert unneeded memset.

prepare_inplace_alter_table_dict
        Remove unneeded col_names parameter

index_field_t
        Remove unneeded col_name member.

row_merge_create_index
        Remove unneeded col_names parameter and resolution.

Effected tests:
         innodb-alter-table : Add test case for MDEV-13668
         innodb-alter : Remove MDEV-13668, MDEV-9469 FIXMEs
                        and restore original tests
         innodb-wl5980-alter : Remove MDEV-13668,  MDEV-9469 FIXMEs
                        and restore original tests
2017-10-10 17:03:40 +03:00
Marko Mäkelä
e1d9a23797 MDEV-14023 10.1 InnoDB tables with virtual columns cannot be accessed in 10.2
MariaDB 10.1 introduced non-indexed virtual columns for InnoDB tables.
When MySQL 5.7 introduced virtual columns in InnoDB tables, it also
introduced the table SYS_VIRTUAL that stores metadata on virtual
columns. This table does not initially exist in data files that were
imported from 10.1. So, we do not always have virtual column metadata
inside InnoDB.

dict_index_contains_col_or_prefix(): In the clustered index records,
all non-virtual columns are present and no virtual columns are present.

ha_innobase::build_template(): In the clustered index, do not
include virtual columns in the query template. The SQL layer is
supposed to compute the virtual column values when needed.
2017-10-09 15:05:17 +03:00
Marko Mäkelä
a4948dafcd MDEV-11369 Instant ADD COLUMN for InnoDB
For InnoDB tables, adding, dropping and reordering columns has
required a rebuild of the table and all its indexes. Since MySQL 5.6
(and MariaDB 10.0) this has been supported online (LOCK=NONE), allowing
concurrent modification of the tables.

This work revises the InnoDB ROW_FORMAT=REDUNDANT, ROW_FORMAT=COMPACT
and ROW_FORMAT=DYNAMIC so that columns can be appended instantaneously,
with only minor changes performed to the table structure. The counter
innodb_instant_alter_column in INFORMATION_SCHEMA.GLOBAL_STATUS
is incremented whenever a table rebuild operation is converted into
an instant ADD COLUMN operation.

ROW_FORMAT=COMPRESSED tables will not support instant ADD COLUMN.

Some usability limitations will be addressed in subsequent work:

MDEV-13134 Introduce ALTER TABLE attributes ALGORITHM=NOCOPY
and ALGORITHM=INSTANT
MDEV-14016 Allow instant ADD COLUMN, ADD INDEX, LOCK=NONE

The format of the clustered index (PRIMARY KEY) is changed as follows:

(1) The FIL_PAGE_TYPE of the root page will be FIL_PAGE_TYPE_INSTANT,
and a new field PAGE_INSTANT will contain the original number of fields
in the clustered index ('core' fields).
If instant ADD COLUMN has not been used or the table becomes empty,
or the very first instant ADD COLUMN operation is rolled back,
the fields PAGE_INSTANT and FIL_PAGE_TYPE will be reset
to 0 and FIL_PAGE_INDEX.

(2) A special 'default row' record is inserted into the leftmost leaf,
between the page infimum and the first user record. This record is
distinguished by the REC_INFO_MIN_REC_FLAG, and it is otherwise in the
same format as records that contain values for the instantly added
columns. This 'default row' always has the same number of fields as
the clustered index according to the table definition. The values of
'core' fields are to be ignored. For other fields, the 'default row'
will contain the default values as they were during the ALTER TABLE
statement. (If the column default values are changed later, those
values will only be stored in the .frm file. The 'default row' will
contain the original evaluated values, which must be the same for
every row.) The 'default row' must be completely hidden from
higher-level access routines. Assertions have been added to ensure
that no 'default row' is ever present in the adaptive hash index
or in locked records. The 'default row' is never delete-marked.

(3) In clustered index leaf page records, the number of fields must
reside between the number of 'core' fields (dict_index_t::n_core_fields
introduced in this work) and dict_index_t::n_fields. If the number
of fields is less than dict_index_t::n_fields, the missing fields
are replaced with the column value of the 'default row'.
Note: The number of fields in the record may shrink if some of the
last instantly added columns are updated to the value that is
in the 'default row'. The function btr_cur_trim() implements this
'compression' on update and rollback; dtuple::trim() implements it
on insert.

(4) In ROW_FORMAT=COMPACT and ROW_FORMAT=DYNAMIC records, the new
status value REC_STATUS_COLUMNS_ADDED will indicate the presence of
a new record header that will encode n_fields-n_core_fields-1 in
1 or 2 bytes. (In ROW_FORMAT=REDUNDANT records, the record header
always explicitly encodes the number of fields.)

We introduce the undo log record type TRX_UNDO_INSERT_DEFAULT for
covering the insert of the 'default row' record when instant ADD COLUMN
is used for the first time. Subsequent instant ADD COLUMN can use
TRX_UNDO_UPD_EXIST_REC.

This is joint work with Vin Chen (陈福荣) from Tencent. The design
that was discussed in April 2017 would not have allowed import or
export of data files, because instead of the 'default row' it would
have introduced a data dictionary table. The test
rpl.rpl_alter_instant is exactly as contributed in pull request #408.
The test innodb.instant_alter is based on a contributed test.

The redo log record format changes for ROW_FORMAT=DYNAMIC and
ROW_FORMAT=COMPACT are as contributed. (With this change present,
crash recovery from MariaDB 10.3.1 will fail in spectacular ways!)
Also the semantics of higher-level redo log records that modify the
PAGE_INSTANT field is changed. The redo log format version identifier
was already changed to LOG_HEADER_FORMAT_CURRENT=103 in MariaDB 10.3.1.

Everything else has been rewritten by me. Thanks to Elena Stepanova,
the code has been tested extensively.

When rolling back an instant ADD COLUMN operation, we must empty the
PAGE_FREE list after deleting or shortening the 'default row' record,
by calling either btr_page_empty() or btr_page_reorganize(). We must
know the size of each entry in the PAGE_FREE list. If rollback left a
freed copy of the 'default row' in the PAGE_FREE list, we would be
unable to determine its size (if it is in ROW_FORMAT=COMPACT or
ROW_FORMAT=DYNAMIC) because it would contain more fields than the
rolled-back definition of the clustered index.

UNIV_SQL_DEFAULT: A new special constant that designates an instantly
added column that is not present in the clustered index record.

len_is_stored(): Check if a length is an actual length. There are
two magic length values: UNIV_SQL_DEFAULT, UNIV_SQL_NULL.

dict_col_t::def_val: The 'default row' value of the column.  If the
column is not added instantly, def_val.len will be UNIV_SQL_DEFAULT.

dict_col_t: Add the accessors is_virtual(), is_nullable(), is_instant(),
instant_value().

dict_col_t::remove_instant(): Remove the 'instant ADD' status of
a column.

dict_col_t::name(const dict_table_t& table): Replaces
dict_table_get_col_name().

dict_index_t::n_core_fields: The original number of fields.
For secondary indexes and if instant ADD COLUMN has not been used,
this will be equal to dict_index_t::n_fields.

dict_index_t::n_core_null_bytes: Number of bytes needed to
represent the null flags; usually equal to UT_BITS_IN_BYTES(n_nullable).

dict_index_t::NO_CORE_NULL_BYTES: Magic value signalling that
n_core_null_bytes was not initialized yet from the clustered index
root page.

dict_index_t: Add the accessors is_instant(), is_clust(),
get_n_nullable(), instant_field_value().

dict_index_t::instant_add_field(): Adjust clustered index metadata
for instant ADD COLUMN.

dict_index_t::remove_instant(): Remove the 'instant ADD' status
of a clustered index when the table becomes empty, or the very first
instant ADD COLUMN operation is rolled back.

dict_table_t: Add the accessors is_instant(), is_temporary(),
supports_instant().

dict_table_t::instant_add_column(): Adjust metadata for
instant ADD COLUMN.

dict_table_t::rollback_instant(): Adjust metadata on the rollback
of instant ADD COLUMN.

prepare_inplace_alter_table_dict(): First create the ctx->new_table,
and only then decide if the table really needs to be rebuilt.
We must split the creation of table or index metadata from the
creation of the dictionary table records and the creation of
the data. In this way, we can transform a table-rebuilding operation
into an instant ADD COLUMN operation. Dictionary objects will only
be added to cache when table rebuilding or index creation is needed.
The ctx->instant_table will never be added to cache.

dict_table_t::add_to_cache(): Modified and renamed from
dict_table_add_to_cache(). Do not modify the table metadata.
Let the callers invoke dict_table_add_system_columns() and if needed,
set can_be_evicted.

dict_create_sys_tables_tuple(), dict_create_table_step(): Omit the
system columns (which will now exist in the dict_table_t object
already at this point).

dict_create_table_step(): Expect the callers to invoke
dict_table_add_system_columns().

pars_create_table(): Before creating the table creation execution
graph, invoke dict_table_add_system_columns().

row_create_table_for_mysql(): Expect all callers to invoke
dict_table_add_system_columns().

create_index_dict(): Replaces row_merge_create_index_graph().

innodb_update_n_cols(): Renamed from innobase_update_n_virtual().
Call my_error() if an error occurs.

btr_cur_instant_init(), btr_cur_instant_init_low(),
btr_cur_instant_root_init():
Load additional metadata from the clustered index and set
dict_index_t::n_core_null_bytes. This is invoked
when table metadata is first loaded into the data dictionary.

dict_boot(): Initialize n_core_null_bytes for the four hard-coded
dictionary tables.

dict_create_index_step(): Initialize n_core_null_bytes. This is
executed as part of CREATE TABLE.

dict_index_build_internal_clust(): Initialize n_core_null_bytes to
NO_CORE_NULL_BYTES if table->supports_instant().

row_create_index_for_mysql(): Initialize n_core_null_bytes for
CREATE TEMPORARY TABLE.

commit_cache_norebuild(): Call the code to rename or enlarge columns
in the cache only if instant ADD COLUMN is not being used.
(Instant ADD COLUMN would copy all column metadata from
instant_table to old_table, including the names and lengths.)

PAGE_INSTANT: A new 13-bit field for storing dict_index_t::n_core_fields.
This is repurposing the 16-bit field PAGE_DIRECTION, of which only the
least significant 3 bits were used. The original byte containing
PAGE_DIRECTION will be accessible via the new constant PAGE_DIRECTION_B.

page_get_instant(), page_set_instant(): Accessors for the PAGE_INSTANT.

page_ptr_get_direction(), page_get_direction(),
page_ptr_set_direction(): Accessors for PAGE_DIRECTION.

page_direction_reset(): Reset PAGE_DIRECTION, PAGE_N_DIRECTION.

page_direction_increment(): Increment PAGE_N_DIRECTION
and set PAGE_DIRECTION.

rec_get_offsets(): Use the 'leaf' parameter for non-debug purposes,
and assume that heap_no is always set.
Initialize all dict_index_t::n_fields for ROW_FORMAT=REDUNDANT records,
even if the record contains fewer fields.

rec_offs_make_valid(): Add the parameter 'leaf'.

rec_copy_prefix_to_dtuple(): Assert that the tuple is only built
on the core fields. Instant ADD COLUMN only applies to the
clustered index, and we should never build a search key that has
more than the PRIMARY KEY and possibly DB_TRX_ID,DB_ROLL_PTR.
All these columns are always present.

dict_index_build_data_tuple(): Remove assertions that would be
duplicated in rec_copy_prefix_to_dtuple().

rec_init_offsets(): Support ROW_FORMAT=REDUNDANT records whose
number of fields is between n_core_fields and n_fields.

cmp_rec_rec_with_match(): Implement the comparison between two
MIN_REC_FLAG records.

trx_t::in_rollback: Make the field available in non-debug builds.

trx_start_for_ddl_low(): Remove dangerous error-tolerance.
A dictionary transaction must be flagged as such before it has generated
any undo log records. This is because trx_undo_assign_undo() will mark
the transaction as a dictionary transaction in the undo log header
right before the very first undo log record is being written.

btr_index_rec_validate(): Account for instant ADD COLUMN

row_undo_ins_remove_clust_rec(): On the rollback of an insert into
SYS_COLUMNS, revert instant ADD COLUMN in the cache by removing the
last column from the table and the clustered index.

row_search_on_row_ref(), row_undo_mod_parse_undo_rec(), row_undo_mod(),
trx_undo_update_rec_get_update(): Handle the 'default row'
as a special case.

dtuple_t::trim(index): Omit a redundant suffix of an index tuple right
before insert or update. After instant ADD COLUMN, if the last fields
of a clustered index tuple match the 'default row', there is no
need to store them. While trimming the entry, we must hold a page latch,
so that the table cannot be emptied and the 'default row' be deleted.

btr_cur_optimistic_update(), btr_cur_pessimistic_update(),
row_upd_clust_rec_by_insert(), row_ins_clust_index_entry_low():
Invoke dtuple_t::trim() if needed.

row_ins_clust_index_entry(): Restore dtuple_t::n_fields after calling
row_ins_clust_index_entry_low().

rec_get_converted_size(), rec_get_converted_size_comp(): Allow the number
of fields to be between n_core_fields and n_fields. Do not support
infimum,supremum. They are never supposed to be stored in dtuple_t,
because page creation nowadays uses a lower-level method for initializing
them.

rec_convert_dtuple_to_rec_comp(): Assign the status bits based on the
number of fields.

btr_cur_trim(): In an update, trim the index entry as needed. For the
'default row', handle rollback specially. For user records, omit
fields that match the 'default row'.

btr_cur_optimistic_delete_func(), btr_cur_pessimistic_delete():
Skip locking and adaptive hash index for the 'default row'.

row_log_table_apply_convert_mrec(): Replace 'default row' values if needed.
In the temporary file that is applied by row_log_table_apply(),
we must identify whether the records contain the extra header for
instantly added columns. For now, we will allocate an additional byte
for this for ROW_T_INSERT and ROW_T_UPDATE records when the source table
has been subject to instant ADD COLUMN. The ROW_T_DELETE records are
fine, as they will be converted and will only contain 'core' columns
(PRIMARY KEY and some system columns) that are converted from dtuple_t.

rec_get_converted_size_temp(), rec_init_offsets_temp(),
rec_convert_dtuple_to_temp(): Add the parameter 'status'.

REC_INFO_DEFAULT_ROW = REC_INFO_MIN_REC_FLAG | REC_STATUS_COLUMNS_ADDED:
An info_bits constant for distinguishing the 'default row' record.

rec_comp_status_t: An enum of the status bit values.

rec_leaf_format: An enum that replaces the bool parameter of
rec_init_offsets_comp_ordinary().
2017-10-06 09:50:10 +03:00
Marko Mäkelä
41e581b30b ha_innobase::open(): Simplify a consistency check 2017-10-06 06:52:30 +03:00
Marko Mäkelä
2c1067166d Merge bb-10.2-ext into 10.3 2017-10-04 08:24:06 +03:00
Alexander Barkov
8ae8cd6348 Merge remote-tracking branch 'origin/10.2' into bb-10.2-ext 2017-10-02 22:35:13 +04:00
Marko Mäkelä
387bdf07ae Remove MYSQL_REPLACE_TRX_IN_THD
The method handlerton::replace_native_transaction_in_thd was added
into MySQL 5.7 in WL#6860, Binlogging XA-prepared transaction.
In MariaDB we do not have that change yet.
2017-10-02 12:51:01 +03:00
Marko Mäkelä
d6f857ddbc Remove a constant parameter commit=false 2017-10-02 11:27:53 +03:00
Marko Mäkelä
3c4cff3357 Merge 10.1 into 10.2 2017-10-02 11:16:53 +03:00
Marko Mäkelä
76953c0e45 Remove remaining InnoDB references to the TABLESPACE keyword
MySQL 5.7 allows the creation of InnoDB tables in a user-created
tablespace, but MariaDB does not. Remove the remaining references.

This is a non-functional change.
2017-10-02 10:03:47 +03:00
Marko Mäkelä
358ab5d6b1 Remove remaining references to InnoDB native partitioning
The InnoDB native partitioning (ha_innopart) is a MySQL 5.7
code refactoring exercise that was not merged to MariaDB.
2017-09-29 15:45:58 +03:00
Vladislav Vaintroub
7354dc6773 MDEV-13384 - misc Windows warnings fixed 2017-09-28 17:20:46 +00:00
Vladislav Vaintroub
509928718d MDEV-13384 Fix Windows warnings. thd_alloc functions now use size_t parameters
Fixes some warnings in popular headers.
2017-09-28 17:20:46 +00:00
Sachin Setiya
0627929f62 MDEV-13787 Crash in persistent stats wsrep_on (thd=0x0)
Problem:- This crash happens because of thd = NULL , and while checking
for wsrep_on , we no longer check for thd != NULL (MDEV-7955). So this
problem is regression of MDEV-7955. However this patch not only solves
this regression , It solves all regression caused by MDEV-7955 patch.

To get all possible cases when thd can be null , assert(thd)/
assert(trx->mysql_thd) is place just before all wsrep_on and innodb test
suite is run. And the assert which caused failure are removed with a physical
check for thd != NULL. Rest assert are removed. Hopefully this method will
remove all current/potential regression of MDEV-7955.
2017-09-27 10:15:08 +05:30
Marko Mäkelä
e17a282da9 Merge bb-10.2-ext into 10.3 2017-09-18 11:38:07 +03:00
Marko Mäkelä
fe949504f0 Merge 10.2 into bb-10.2-ext 2017-09-17 14:03:51 +03:00
Marko Mäkelä
d9277732d7 Merge 10.1 into 10.2
This should also fix the MariaDB 10.2.2 bug
MDEV-13826 CREATE FULLTEXT INDEX on encrypted table fails.

MDEV-12634 FIXME: Modify innodb-index-online, innodb-table-online
so that they will write and read merge sort files. InnoDB 5.7
introduced some optimizations to avoid using the files for small tables.

Many collation test results have been adjusted for MDEV-10191.
2017-09-17 11:05:33 +03:00
Marko Mäkelä
a449f72a4c Remove the redundant function row_table_got_default_clust_index()
Use dict_index_is_auto_gen_clust() instead.
2017-09-15 20:06:26 +03:00
Jan Lindström
fa2701c6f7 MDEV-12634: Uninitialised ROW_MERGE_RESERVE_SIZE bytes written to tem…
…porary file

Fixed by removing writing key version to start of every block that
was encrypted. Instead we will use single key version from log_sys
crypt info.

After this MDEV also blocks writen to row log are encrypted and blocks
read from row log aren decrypted if encryption is configured for the
table.

innodb_status_variables[], struct srv_stats_t
	Added status variables for merge block and row log block
	encryption and decryption amounts.

Removed ROW_MERGE_RESERVE_SIZE define.

row_merge_fts_doc_tokenize
	Remove ROW_MERGE_RESERVE_SIZE

row_log_t
	Add index, crypt_tail, crypt_head to be used in case of
	encryption.

row_log_online_op, row_log_table_close_func
	Before writing a block encrypt it if encryption is enabled

row_log_table_apply_ops, row_log_apply_ops
	After reading a block decrypt it if encryption is enabled

row_log_allocate
	Allocate temporary buffers crypt_head and crypt_tail
	if needed.

row_log_free
	Free temporary buffers crypt_head and crypt_tail if they
	exist.

row_merge_encrypt_buf, row_merge_decrypt_buf
	Removed.

row_merge_buf_create, row_merge_buf_write
	Remove ROW_MERGE_RESERVE_SIZE

row_merge_build_indexes
	Allocate temporary buffer used in decryption and encryption
	if needed.

log_tmp_blocks_crypt, log_tmp_block_encrypt, log_temp_block_decrypt
	New functions used in block encryption and decryption

log_tmp_is_encrypted
	New function to check is encryption enabled.

Added test case innodb-rowlog to force creating a row log and
verify that operations are done using introduced status
variables.
2017-09-14 09:23:20 +03:00
Marko Mäkelä
348eaf4252 Merge bb-10.2-ext into 10.3 2017-09-14 09:12:47 +03:00
Marko Mäkelä
c97e504a5f Merge 10.2 into bb-10.2-ext 2017-09-14 08:09:44 +03:00
Marko Mäkelä
24062fed70 Remove the debug variables innodb_purge_stop_now, innodb_purge_run_now
The InnoDB purge subsystem can be best stopped by opening a read view,
for example by START TRANSACTION WITH CONSISTENT SNAPSHOT.

To ensure that everything is purged, use wait_all_purged.inc,
which waits for the History list length in SHOW ENGINE INNODB STATUS
to reach 0. Setting innodb_purge_run_now never guaranteed this.
2017-09-13 16:02:32 +03:00
Aleksey Midenkov
7e764ae188 SQL: 1-row partition rotation fix [fixes #260] 2017-09-13 10:57:46 +03:00
Jan Lindström
016c35a7f2 MDEV-13690: Remove unnecessary innodb_use_mtflush, innodb_mtflush_threads parameters and related code
Users can use innodb-page-cleaners instead.
2017-09-01 18:33:46 +03:00
Marko Mäkelä
4e1fa7f63d Merge bb-10.2-ext into 10.3 2017-09-01 11:33:45 +03:00
Marko Mäkelä
6749d39a95 Merge 10.2 into bb-10.2-ext 2017-09-01 08:47:55 +03:00
Jan Lindström
e23de9f2e0 MDEV-13674: Deprecate innodb_use_mtflush and innodb_mtflush_threads
These parameters and associated code is to be removed in 10.3.
Users can use innodb-page-cleaners > 1 instead.
2017-08-31 13:36:36 +03:00
Marko Mäkelä
a36c369bda Merge 10.1 into 10.2
For running the Galera tests, the variable my_disable_leak_check
was set to true in order to avoid assertions due to memory leaks
at shutdown.

Some adjustments due to MDEV-13625 (merge InnoDB tests from MySQL 5.6)
were performed. The most notable behaviour changes from 10.0 and 10.1
are the following:

* innodb.innodb-table-online: adjustments for the DROP COLUMN
behaviour change (MDEV-11114, MDEV-13613)

* innodb.innodb-index-online-fk: the removal of a (1,NULL) record
from the result; originally removed in MySQL 5.7 in the
Oracle Bug #16244691 fix
377774689b

* innodb.create-index-debug: disabled due to MDEV-13680
(the MySQL Bug #77497 fix was not merged from 5.6 to 5.7.10)

* innodb.innodb-alter-autoinc: MariaDB 10.2 behaves like MySQL 5.6/5.7,
while MariaDB 10.0 and 10.1 assign different values when
auto_increment_increment or auto_increment_offset are used.
Also MySQL 5.6/5.7 exhibit different behaviour between
LGORITHM=INPLACE and ALGORITHM=COPY, so something needs to be tested
and fixed in both MariaDB 10.0 and 10.2.

* innodb.innodb-wl5980-alter: disabled because it would trigger an
InnoDB assertion failure (MDEV-13668 may need additional effort in 10.2)
2017-08-31 09:30:40 +03:00
Marko Mäkelä
829752973b Merge branch '10.0' into 10.1 2017-08-30 13:06:13 +03:00
Andrei Elkin
888a8b69bd MDEV-13437 InnoDB fails to return error for XA COMMIT or XA ROLLBACK in read-only mode
Assertions failed due to incorrect handling of the --tc-heuristic-recover
option when InnoDB is in read-only mode either due to innodb_read_only=1
or innodb_force_recovery>3. InnoDB failed to refuse a XA COMMIT or
XA ROLLBACK operation, and there were errors in the error handling in
the upper layer.

This was fixed by making InnoDB XA operations respect the
high_level_read_only flag. The InnoDB part of the fix and
parts of the test main.tc_heuristic_recover were provided
by Marko Mäkelä.

LOCK_log mutex lock/unlock had to be added to fix MDEV-13438.
The measure is confirmed by mysql sources as well.

For testing of the conflicting option combination, mysql-test-run is
made to export a new $MYSQLD_LAST_CMD. It holds the very last value
generated by mtr.mysqld_start().  Even though the options have been
also always stored in $mysqld->{'started_opts'} there were no access
to them beyond the automatic server restart by mtr through the expect
file interface.

Effectively therefore $MYSQLD_LAST_CMD represents a more general
interface to $mysqld->{'started_opts'} which can be used in wider
scopes including server launch with incompatible options.

Notice another existing method to restart the server with incompatible
options relying on $MYSQLD_CMD is is aware of $mysqld->{'started_opts'}
(the actual options that the server is launched by mtr). In order to use
this method they would have to be provided manually.

NOTE: When merging to 10.2, the file search_pattern_in_file++.inc
should be replaced with the pre-existing search_pattern_in_file.inc.
2017-08-29 11:59:59 +03:00
Sergei Golubchik
bb8e99fdc3 Merge branch 'bb-10.2-ext' into 10.3 2017-08-26 00:34:43 +02:00
Sergei Golubchik
27412877db Merge branch '10.2' into bb-10.2-ext 2017-08-25 10:25:48 +02:00
Michael Widenius
cc77f9882d Changed KEY names to use LEX_CSTRING 2017-08-24 01:05:53 +02:00
Monty
536215e32f Added DBUG_ASSERT_AS_PRINTF compile flag
If compiling a non DBUG binary with
-DDBUG_ASSERT_AS_PRINTF asserts will be
changed to printf + stack trace (of stack
trace are enabled).

- Changed #ifndef DBUG_OFF to
  #ifdef DBUG_ASSERT_EXISTS
  for those DBUG_OFF that was just used to enable
  assert
- Assert checking that could greatly impact
  performance where changed to DBUG_ASSERT_SLOW which
  is not affected by DBUG_ASSERT_AS_PRINTF
- Added one extra option to my_print_stacktrace() to
  get more silent in case of stack trace printing as
  part of assert.
2017-08-24 01:05:50 +02:00
Marko Mäkelä
59caf2c3c1 MDEV-13485 MTR tests fail massively with --innodb-sync-debug
The parameter --innodb-sync-debug, which is disabled by default,
aims to find potential deadlocks in InnoDB.

When the parameter is enabled, lots of tests failed. Most of these
failures were due to bogus diagnostics. But, as part of this fix,
we are also fixing a bug in error handling code and removing dead
code, and fixing cases where an uninitialized mutex was being
locked and unlocked.

dict_create_foreign_constraints_low(): Remove an extraneous
mutex_exit() call that could cause corruption in an error handling
path. Also, do not unnecessarily acquire dict_foreign_err_mutex.
Its only purpose is to control concurrent access to
dict_foreign_err_file.

row_ins_foreign_trx_print(): Replace a redundant condition with a
debug assertion.

srv_dict_tmpfile, srv_dict_tmpfile_mutex: Remove. The
temporary file is never being written to or read from.

log_free_check(): Allow SYNC_FTS_CACHE (fts_cache_t::lock)
to be held.

ha_innobase::inplace_alter_table(), row_merge_insert_index_tuples():
Assert that no unexpected latches are being held.

sync_latch_meta_init(): Properly initialize dict_operation_lock_key
at SYNC_DICT_OPERATION. dict_sys->mutex is SYNC_DICT, and
the now-removed SRV_DICT_TMPFILE was wrongly registered at
SYNC_DICT_OPERATION.

buf_block_init(): Correctly register buf_block_t::debug_latch.
It was previously misleadingly reported as LATCH_ID_DICT_FOREIGN_ERR.

latch_level_t: Correct the relative latching order of
SYNC_IBUF_PESS_INSERT_MUTEX,SYNC_INDEX_TREE and
SYNC_FILE_FORMAT_TAG,SYNC_DICT_OPERATION to avoid bogus failures.

row_drop_table_for_mysql(): Avoid accessing btr_defragment_mutex
if the defragmentation thread has not been started. This is the
case during fts_drop_orphaned_tables() in recv_recovery_rollback_active().

fil_space_destroy_crypt_data(): Avoid acquiring fil_crypt_threads_mutex
when it is uninitialized. We may have created crypt_data before the
mutex was created, and the mutex creation would be skipped if
InnoDB startup failed or --innodb-read-only was specified.
2017-08-23 08:44:11 +03:00
Jan Lindström
c23efc7d50 Merge remote-tracking branch 'origin/10.0-galera' into 10.1 2017-08-21 13:35:00 +03:00