1
0
mirror of https://github.com/MariaDB/server.git synced 2025-05-05 16:59:35 +03:00

111 Commits

Author SHA1 Message Date
Marko Mäkelä
a78476d342 Merge 10.1 into 10.2 2017-06-12 17:43:07 +03:00
Jan Lindström
58c56dd7f8 MDEV-12610: MariaDB start is slow
Problem appears to be that the function fsp_flags_try_adjust()
is being unconditionally invoked on every .ibd file on startup.
Based on performance investigation also the top function
fsp_header_get_crypt_offset() needs to addressed.

Ported implementation of fsp_header_get_encryption_offset()
function from 10.2 to fsp_header_get_crypt_offset().

Introduced a new function fil_crypt_read_crypt_data()
to read page 0 if it is not yet read.

fil_crypt_find_space_to_rotate(): Now that page 0 for every .ibd
file is not read on startup we need to check has page 0 read
from space that we investigate for key rotation, if it is not read
we read it.

fil_space_crypt_get_status(): Now that page 0 for every .ibd
file is not read on startup here also we need to read page 0
if it is not yet read it. This is needed
as tests use IS query to wait until background encryption
or decryption has finished and this function is used to
produce results.

fil_crypt_thread(): Add is_stopping condition for tablespace
so that we do not rotate pages if usage of tablespace should
be stopped. This was needed for failure seen on regression
testing.

fil_space_create: Remove page_0_crypt_read and extra
unnecessary info output.

fil_open_single_table_tablespace(): We call fsp_flags_try_adjust
only when when no errors has happened and server was not started
on read only mode and tablespace validation was requested or
flags contain other table options except low order bits to
FSP_FLAGS_POS_PAGE_SSIZE position.

fil_space_t::page_0_crypt_read removed.

Added test case innodb-first-page-read to test startup when
encryption is on and when encryption is off to check that not
for all tables page 0 is read on startup.
2017-06-09 13:15:39 +03:00
Marko Mäkelä
2d8fdfbde5 Merge 10.1 into 10.2
Replace have_innodb_zip.inc with innodb_page_size_small.inc.
2017-06-08 12:45:08 +03:00
Marko Mäkelä
30df297c2f Merge 10.0 into 10.1
Rewrite the test encryption.innodb-checksum-algorithm not to
require any restarts or re-bootstrapping, and to cover all
innodb_page_size combinations.

Test innodb.101_compatibility with all innodb_page_size combinations.
2017-06-06 10:59:54 +03:00
Jan Lindström
6b6987154a MDEV-12114: install_db shows corruption for rest encryption and innodb_checksum_algorithm=strict_none
Problem was that checksum check resulted false positives that page is
both not encrypted and encryted when checksum_algorithm was
strict_none.

Encrypton checksum will use only crc32 regardless of setting.

buf_zip_decompress: If compression fails report a error message
containing the space name if available (not available during import).
And note if space could be encrypted.

buf_page_get_gen: Do not assert if decompression fails,
instead unfix the page and return NULL to upper layer.

fil_crypt_calculate_checksum: Use only crc32 method.

fil_space_verify_crypt_checksum: Here we need to check
crc32, innodb and none method for old datafiles.

fil_space_release_for_io: Allow null space.

encryption.innodb-compressed-blob is now run with crc32 and none
combinations.

Note that with none and strict_none method there is not really
a way to detect page corruptions and page corruptions after
decrypting the page with incorrect key.

New test innodb-checksum-algorithm to test different checksum
algorithms with encrypted, row compressed and page compressed
tables.
2017-06-01 14:07:48 +03:00
Marko Mäkelä
8f643e2063 Merge 10.1 into 10.2 2017-05-23 11:09:47 +03:00
Marko Mäkelä
45fe62b8d6 Clean up a test
Changed global parameters will be reset by server restart.
There is no need to save and restore the original values.
2017-05-20 08:41:34 +03:00
Marko Mäkelä
069d0472b3 MDEV-12762 Some files in current 10.2 tree seem to be reverted to an old state
The issue was a bad merge of MDEV-12253 from 10.1 to 10.2
in commit f9cc391863ab962e8c8a8a8a676d730498eb8d6f.
In that merge, I wrongly assumed that all test file conflicts
for mysql-test/suite/encryption had been properly resolved in
bb-10.2-MDEV-12253 (commit 76aa6be77635c7017459ce33b41c837c9acb606d)
while in fact, some files there had been copied from the 10.1 branch.

This commit is based on a manually done conflict resolution of
the mysql-test/suite/encryption on the same merge, applied to
the current 10.2 branch.

As part of this commit, the test encryption.innodb-bad-key-change4
which was shortly disabled due to MDEV-11336 will be re-enabled again.
(While the test enables innodb_defragment, it does not fail even though
enabling innodb_defragment currently has no effect.)
2017-05-10 09:06:57 +03:00
Marko Mäkelä
588a6a186a MDEV-12750 Fix crash recovery of key rotation
When MySQL 5.7.9 was merged to MariaDB 10.2.2, an important
debug assertion was omitted from mlog_write_initial_log_record_low().

mlog_write_initial_log_record_low(): Put back the assertion
mtr_t::is_named_space().

fil_crypt_start_encrypting_space(), fil_crypt_rotate_page():
Call mtr_t::set_named_space() before modifying any pages.

fsp_flags_try_adjust(): Call mtr_t::set_named_space(). This additional
breakage was introduced in the merge of MDEV-11623 from 10.1. It was
not caught because of the missing debug assertion in
mlog_write_initial_log_record_low().

Remove some suppressions from the encryption.innodb-redo-badkey test.
2017-05-09 21:03:27 +03:00
Marko Mäkelä
f9cc391863 Merge 10.1 into 10.2
This only merges MDEV-12253, adapting it to MDEV-12602 which is already
present in 10.2 but not yet in the 10.1 revision that is being merged.

TODO: Error handling in crash recovery needs to be improved.
If a page cannot be decrypted (or read), we should cleanly abort
the startup. If innodb_force_recovery is specified, we should
ignore the problematic page and apply redo log to other pages.
Currently, the test encryption.innodb-redo-badkey randomly fails
like this (the last messages are from cmake -DWITH_ASAN):

2017-05-05 10:19:40 140037071685504 [Note] InnoDB: Starting crash recovery from checkpoint LSN=1635994
2017-05-05 10:19:40 140037071685504 [ERROR] InnoDB: Missing MLOG_FILE_NAME or MLOG_FILE_DELETE before MLOG_CHECKPOINT for tablespace 1
2017-05-05 10:19:40 140037071685504 [ERROR] InnoDB: Plugin initialization aborted at srv0start.cc[2201] with error Data structure corruption
2017-05-05 10:19:41 140037071685504 [Note] InnoDB: Starting shutdown...
i=================================================================
==5226==ERROR: AddressSanitizer: attempting free on address which was not malloc()-ed: 0x612000018588 in thread T0
    #0 0x736750 in operator delete(void*) (/mariadb/server/build/sql/mysqld+0x736750)
    #1 0x1e4833f in LatchCounter::~LatchCounter() /mariadb/server/storage/innobase/include/sync0types.h:599:4
    #2 0x1e480b8 in LatchMeta<LatchCounter>::~LatchMeta() /mariadb/server/storage/innobase/include/sync0types.h:786:17
    #3 0x1e35509 in sync_latch_meta_destroy() /mariadb/server/storage/innobase/sync/sync0debug.cc:1622:3
    #4 0x1e35314 in sync_check_close() /mariadb/server/storage/innobase/sync/sync0debug.cc:1839:2
    #5 0x1dfdc18 in innodb_shutdown() /mariadb/server/storage/innobase/srv/srv0start.cc:2888:2
    #6 0x197e5e6 in innobase_init(void*) /mariadb/server/storage/innobase/handler/ha_innodb.cc:4475:3
2017-05-05 10:38:53 +03:00
Jan Lindström
765a43605a MDEV-12253: Buffer pool blocks are accessed after they have been freed
Problem was that bpage was referenced after it was already freed
from LRU. Fixed by adding a new variable encrypted that is
passed down to buf_page_check_corrupt() and used in
buf_page_get_gen() to stop processing page read.

This patch should also address following test failures and
bugs:

MDEV-12419: IMPORT should not look up tablespace in
PageConverter::validate(). This is now removed.

MDEV-10099: encryption.innodb_onlinealter_encryption fails
sporadically in buildbot

MDEV-11420: encryption.innodb_encryption-page-compression
failed in buildbot

MDEV-11222: encryption.encrypt_and_grep failed in buildbot on P8

Removed dict_table_t::is_encrypted and dict_table_t::ibd_file_missing
and replaced these with dict_table_t::file_unreadable. Table
ibd file is missing if fil_get_space(space_id) returns NULL
and encrypted if not. Removed dict_table_t::is_corrupted field.

Ported FilSpace class from 10.2 and using that on buf_page_check_corrupt(),
buf_page_decrypt_after_read(), buf_page_encrypt_before_write(),
buf_dblwr_process(), buf_read_page(), dict_stats_save_defrag_stats().

Added test cases when enrypted page could be read while doing
redo log crash recovery. Also added test case for row compressed
blobs.

btr_cur_open_at_index_side_func(),
btr_cur_open_at_rnd_pos_func(): Avoid referencing block that is
NULL.

buf_page_get_zip(): Issue error if page read fails.

buf_page_get_gen(): Use dberr_t for error detection and
do not reference bpage after we hare freed it.

buf_mark_space_corrupt(): remove bpage from LRU also when
it is encrypted.

buf_page_check_corrupt(): @return DB_SUCCESS if page has
been read and is not corrupted,
DB_PAGE_CORRUPTED if page based on checksum check is corrupted,
DB_DECRYPTION_FAILED if page post encryption checksum matches but
after decryption normal page checksum does not match. In read
case only DB_SUCCESS is possible.

buf_page_io_complete(): use dberr_t for error handling.

buf_flush_write_block_low(),
buf_read_ahead_random(),
buf_read_page_async(),
buf_read_ahead_linear(),
buf_read_ibuf_merge_pages(),
buf_read_recv_pages(),
fil_aio_wait():
        Issue error if page read fails.

btr_pcur_move_to_next_page(): Do not reference page if it is
NULL.

Introduced dict_table_t::is_readable() and dict_index_t::is_readable()
that will return true if tablespace exists and pages read from
tablespace are not corrupted or page decryption failed.
Removed buf_page_t::key_version. After page decryption the
key version is not removed from page frame. For unencrypted
pages, old key_version is removed at buf_page_encrypt_before_write()

dict_stats_update_transient_for_index(),
dict_stats_update_transient()
        Do not continue if table decryption failed or table
        is corrupted.

dict0stats.cc: Introduced a dict_stats_report_error function
to avoid code duplication.

fil_parse_write_crypt_data():
        Check that key read from redo log entry is found from
        encryption plugin and if it is not, refuse to start.

PageConverter::validate(): Removed access to fil_space_t as
tablespace is not available during import.

Fixed error code on innodb.innodb test.

Merged test cased innodb-bad-key-change5 and innodb-bad-key-shutdown
to innodb-bad-key-change2.  Removed innodb-bad-key-change5 test.
Decreased unnecessary complexity on some long lasting tests.

Removed fil_inc_pending_ops(), fil_decr_pending_ops(),
fil_get_first_space(), fil_get_next_space(),
fil_get_first_space_safe(), fil_get_next_space_safe()
functions.

fil_space_verify_crypt_checksum(): Fixed bug found using ASAN
where FIL_PAGE_END_LSN_OLD_CHECKSUM field was incorrectly
accessed from row compressed tables. Fixed out of page frame
bug for row compressed tables in
fil_space_verify_crypt_checksum() found using ASAN. Incorrect
function was called for compressed table.

Added new tests for discard, rename table and drop (we should allow them
even when page decryption fails). Alter table rename is not allowed.
Added test for restart with innodb-force-recovery=1 when page read on
redo-recovery cant be decrypted. Added test for corrupted table where
both page data and FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION is corrupted.

Adjusted the test case innodb_bug14147491 so that it does not anymore
expect crash. Instead table is just mostly not usable.

fil0fil.h: fil_space_acquire_low is not visible function
and fil_space_acquire and fil_space_acquire_silent are
inline functions. FilSpace class uses fil_space_acquire_low
directly.

recv_apply_hashed_log_recs() does not return anything.
2017-04-26 15:19:16 +03:00
Sergei Golubchik
b2865a437f search_pattern_in_file.inc changes
1. Special mode to search in error logs: if SEARCH_RANGE is not set,
   the file is considered an error log and the search is performed
   since the last CURRENT_TEST: line
2. Number of matches is printed too. "FOUND 5 /foo/ in bar".
   Use greedy .* at the end of the pattern if number of matches
   isn't stable. If nothing is found it's still "NOT FOUND",
   not "FOUND 0".
3. SEARCH_ABORT specifies the prefix of the output.
   Can be "NOT FOUND" or "FOUND" as before,
   but also "FOUND 5 " if needed.
2017-03-31 19:28:58 +02:00
Sergei Golubchik
da4d71d10d Merge branch '10.1' into 10.2 2017-03-30 12:48:42 +02:00
Jan Lindström
c56b896c17 Fix test failure on debug_key_management test. 2017-03-28 12:28:09 +03:00
Marko Mäkelä
7668a79a88 MDEV-12269 Port Bug#22996442 INNODB: MAKE UNIV_DEBUG DEPEND ON DBUG_OFF
This is a partial port of my patch in MySQL 8.0.
In MySQL 8.0, all InnoDB references to DBUG_OFF were replaced
with UNIV_DEBUG. We will not do that in MariaDB.

InnoDB used two independent compile-time flags that distinguish
debug and non-debug builds, which is confusing.

Also, make ut_ad() and alias of DBUG_ASSERT().
2017-03-16 10:24:53 +02:00
Marko Mäkelä
b5285bd7e2 MDEV-11873 Unnecessary InnoDB warnings upon bootstrap
dict_create_or_check_foreign_constraint_tables(): Change the warning
about the foreign key metadata table creation to a note.

Remove messages after metadata table creation. If the creation fails,
startup will abort with a message. Normally the creation succeeds on
bootstrap, and the messages would only be noise.

Remove the related suppressions from the tests.
2017-03-14 17:11:46 +02:00
Jan Lindström
50eb40a2a8 MDEV-11738: Mariadb uses 100% of several of my 8 cpus doing nothing
MDEV-11581: Mariadb starts InnoDB encryption threads
when key has not changed or data scrubbing turned off

Background: Key rotation is based on background threads
(innodb-encryption-threads) periodically going through
all tablespaces on fil_system. For each tablespace
current used key version is compared to max key age
(innodb-encryption-rotate-key-age). This process
naturally takes CPU. Similarly, in same time need for
scrubbing is investigated. Currently, key rotation
is fully supported on Amazon AWS key management plugin
only but InnoDB does not have knowledge what key
management plugin is used.

This patch re-purposes innodb-encryption-rotate-key-age=0
to disable key rotation and background data scrubbing.
All new tables are added to special list for key rotation
and key rotation is based on sending a event to
background encryption threads instead of using periodic
checking (i.e. timeout).

fil0fil.cc: Added functions fil_space_acquire_low()
to acquire a tablespace when it could be dropped concurrently.
This function is used from fil_space_acquire() or
fil_space_acquire_silent() that will not print
any messages if we try to acquire space that does not exist.
fil_space_release() to release a acquired tablespace.
fil_space_next() to iterate tablespaces in fil_system
using fil_space_acquire() and fil_space_release().
Similarly, fil_space_keyrotation_next() to iterate new
list fil_system->rotation_list where new tables.
are added if key rotation is disabled.
Removed unnecessary functions fil_get_first_space_safe()
fil_get_next_space_safe()

fil_node_open_file(): After page 0 is read read also
crypt_info if it is not yet read.

btr_scrub_lock_dict_func()
buf_page_check_corrupt()
buf_page_encrypt_before_write()
buf_merge_or_delete_for_page()
lock_print_info_all_transactions()
row_fts_psort_info_init()
row_truncate_table_for_mysql()
row_drop_table_for_mysql()
    Use fil_space_acquire()/release() to access fil_space_t.

buf_page_decrypt_after_read():
    Use fil_space_get_crypt_data() because at this point
    we might not yet have read page 0.

fil0crypt.cc/fil0fil.h: Lot of changes. Pass fil_space_t* directly
to functions needing it and store fil_space_t* to rotation state.
Use fil_space_acquire()/release() when iterating tablespaces
and removed unnecessary is_closing from fil_crypt_t. Use
fil_space_t::is_stopping() to detect when access to
tablespace should be stopped. Removed unnecessary
fil_space_get_crypt_data().

fil_space_create(): Inform key rotation that there could
be something to do if key rotation is disabled and new
table with encryption enabled is created.
Remove unnecessary functions fil_get_first_space_safe()
and fil_get_next_space_safe(). fil_space_acquire()
and fil_space_release() are used instead. Moved
fil_space_get_crypt_data() and fil_space_set_crypt_data()
to fil0crypt.cc.

fsp_header_init(): Acquire fil_space_t*, write crypt_data
and release space.

check_table_options()
	Renamed FIL_SPACE_ENCRYPTION_* TO FIL_ENCRYPTION_*

i_s.cc: Added ROTATING_OR_FLUSHING field to
information_schema.innodb_tablespace_encryption
to show current status of key rotation.
2017-03-14 16:23:10 +02:00
Marko Mäkelä
2af28a363c MDEV-11782: Redefine the innodb_encrypt_log format
Write only one encryption key to the checkpoint page.
Use 4 bytes of nonce. Encrypt more of each redo log block,
only skipping the 4-byte field LOG_BLOCK_HDR_NO which the
initialization vector is derived from.

Issue notes, not warning messages for rewriting the redo log files.

recv_recovery_from_checkpoint_finish(): Do not generate any redo log,
because we must avoid that before rewriting the redo log files, or
otherwise a crash during a redo log rewrite (removing or adding
encryption) may end up making the database unrecoverable.
Instead, do these tasks in innobase_start_or_create_for_mysql().

Issue a firm "Missing MLOG_CHECKPOINT" error message. Remove some
unreachable code and duplicated error messages for log corruption.

LOG_HEADER_FORMAT_ENCRYPTED: A flag for identifying an encrypted redo
log format.

log_group_t::is_encrypted(), log_t::is_encrypted(): Determine
if the redo log is in encrypted format.

recv_find_max_checkpoint(): Interpret LOG_HEADER_FORMAT_ENCRYPTED.

srv_prepare_to_delete_redo_log_files(): Display NOTE messages about
adding or removing encryption. Do not issue warnings for redo log
resizing any more.

innobase_start_or_create_for_mysql(): Rebuild the redo logs also when
the encryption changes.

innodb_log_checksums_func_update(): Always use the CRC-32C checksum
if innodb_encrypt_log. If needed, issue a warning
that innodb_encrypt_log implies innodb_log_checksums.

log_group_write_buf(): Compute the checksum on the encrypted
block contents, so that transmission errors or incomplete blocks can be
detected without decrypting.

Rewrite most of the redo log encryption code. Only remember one
encryption key at a time (but remember up to 5 when upgrading from the
MariaDB 10.1 format.)
2017-02-15 08:07:20 +02:00
Sergei Golubchik
2195bb4e41 Merge branch '10.1' into 10.2 2017-02-10 17:01:45 +01:00
Jan Lindström
7b27465e10 MDEV-11974: MariaDB 10.2 encryption does not support spatial indexes
Encryption stores used key_version to
FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION (offset 26)
field. Spatial indexes store RTREE Split Sequence Number
(FIL_RTREE_SPLIT_SEQ_NUM) in the same field. Both values
can't be stored in same field. Thus, current encryption
implementation does not support encrypting spatial indexes.

fil_space_encrypt(): Do not encrypt page if page type is
FIL_PAGE_RTREE (this is required for background
encryption innodb-encrypt-tables=ON).

create_table_info_t::check_table_options() Do not allow creating
table with ENCRYPTED=YES if table contains spatial index.
2017-02-08 09:05:15 +02:00
Marko Mäkelä
92bbf4ad04 MDEV-11782 WIP: Support upgrade from MariaDB 10.1.
recv_log_format_0_recover(): Invoke log_decrypt_after_read() after
reading the old-format redo log buffer.

With this change, we will upgrade to an encrypted redo log that
is misleadingly carrying a MySQL 5.7.9 compatible format tag while
the log blocks (other than the header and the checkpoint blocks)
are in an incompatible, encrypted format.
That needs to be fixed by introducing a new redo log format tag that
indicates that the entire redo log is encrypted.
2017-02-07 11:55:16 +02:00
Marko Mäkelä
b40a1fbc93 MDEV-11782 WIP: Clean up the code, and add a test.
LOG_CHECKPOINT_ARRAY_END, LOG_CHECKPOINT_SIZE: Remove.

Change some error messages to refer to MariaDB 10.2.2 instead of
MySQL 5.7.9.

recv_find_max_checkpoint_0(): Do not abort when decrypting one of the
checkpoint pages fails.
2017-02-07 11:55:16 +02:00
Jan Lindström
ddf2fac733 MDEV-11759: Encryption code in MariaDB 10.1/10.2 causes
compatibility problems

Pages that are encrypted contain post encryption checksum on
different location that normal checksum fields. Therefore,
we should before decryption check this checksum to avoid
unencrypting corrupted pages. After decryption we can use
traditional checksum check to detect if page is corrupted
or unencryption was done using incorrect key.

Pages that are page compressed do not contain any checksum,
here we need to fist unencrypt, decompress and finally
use tradional checksum check to detect page corruption
or that we used incorrect key in unencryption.

buf0buf.cc: buf_page_is_corrupted() mofified so that
compressed pages are skipped.

buf0buf.h, buf_block_init(), buf_page_init_low():
removed unnecessary page_encrypted, page_compressed,
stored_checksum, valculated_checksum fields from
buf_page_t

buf_page_get_gen(): use new buf_page_check_corrupt() function
to detect corrupted pages.

buf_page_check_corrupt(): If page was not yet decrypted
check if post encryption checksum still matches.
If page is not anymore encrypted, use buf_page_is_corrupted()
traditional checksum method.

If page is detected as corrupted and it is not encrypted
we print corruption message to error log.
If page is still encrypted or it was encrypted and now
corrupted, we will print message that page is
encrypted to error log.

buf_page_io_complete(): use new buf_page_check_corrupt()
function to detect corrupted pages.

buf_page_decrypt_after_read(): Verify post encryption
checksum before tring to decrypt.

fil0crypt.cc: fil_encrypt_buf() verify post encryption
checksum and ind fil_space_decrypt() return true
if we really decrypted the page.

fil_space_verify_crypt_checksum(): rewrite to use
the method used when calculating post encryption
checksum. We also check if post encryption checksum
matches that traditional checksum check does not
match.

fil0fil.ic: Add missed page type encrypted and page
compressed to fil_get_page_type_name()

Note that this change does not yet fix innochecksum tool,
that will be done in separate MDEV.

Fix test failures caused by buf page corruption injection.
2017-02-06 15:40:16 +02:00
Jan Lindström
17430a802b MDEV-11905: encryption.innodb-discard-import test fails
Problem was that on import with buffered input/output
trim is not possible as we write several pages in
one write.
2017-01-25 08:54:02 +02:00
Jan Lindström
6495806e59 MDEV-11254: innodb-use-trim has no effect in 10.2
Problem was that implementation merged from 10.1 was incompatible
with InnoDB 5.7.

buf0buf.cc: Add functions to return should we punch hole and
how big.

buf0flu.cc: Add written page to IORequest

fil0fil.cc: Remove unneeded status call and add test is
sparse files and punch hole supported by file system when
tablespace is created. Add call to get file system
block size. Used file node is added to IORequest. Added
functions to check is punch hole supported and setting
punch hole.

ha_innodb.cc: Remove unneeded status variables (trim512-32768)
and trim_op_saved. Deprecate innodb_use_trim and
set it ON by default. Add function to set innodb-use-trim
dynamically.

dberr.h: Add error code DB_IO_NO_PUNCH_HOLE
if punch hole operation fails.

fil0fil.h: Add punch_hole variable to fil_space_t and
block size to fil_node_t.

os0api.h: Header to helper functions on buf0buf.cc and
fil0fil.cc for os0file.h

os0file.h: Remove unneeded m_block_size from IORequest
and add bpage to IORequest to know actual size of
the block and m_fil_node to know tablespace file
system block size and does it support punch hole.

os0file.cc: Add function punch_hole() to IORequest
to do punch_hole operation,
get the file system block size and determine
does file system support sparse files (for punch hole).

page0size.h: remove implicit copy disable and
use this implicit copy to implement copy_from()
function.

buf0dblwr.cc, buf0flu.cc, buf0rea.cc, fil0fil.cc, fil0fil.h,
os0file.h, os0file.cc, log0log.cc, log0recv.cc:
Remove unneeded write_size parameter from fil_io
calls.

srv0mon.h, srv0srv.h, srv0mon.cc: Remove unneeded
trim512-trim32678 status variables. Removed
these from monitor tests.
2017-01-24 14:40:58 +02:00
Jan Lindström
dc557ca817 MDEV-11835: InnoDB: Failing assertion: free_slot != NULL on
restarting server with encryption and read-only

buf0buf.cc: Temporary slots used in encryption was calculated
by read_threads * write_threads. However, in read-only mode
write_threads is zero. Correct way is to calculate
(read_threads + write_threads) * max pending IO requests.
2017-01-19 08:19:08 +02:00
Marko Mäkelä
08413254b7 Remove references to innodb_file_format.
innodb_file_format=Barracuda is the default in MariaDB 10.2.
Do not set it, because the option will be removed in MariaDB 10.3.

Also, do not set innodb_file_per_table=1 because it is the default.

Note that MDEV-11828 should fix the test innodb.innodb-64k
already in 10.1.
2017-01-18 08:43:11 +02:00
Marko Mäkelä
7c81f15ec3 Merge 10.1 into bb-10.2-mdev-11782 2017-01-12 13:00:11 +02:00
Marko Mäkelä
2dc5d8bb7e Improve an MDEV-9011 test of innodb_encrypt_log.
Test crash recovery from an encrypted redo log with innodb_encrypt_log=0.
Previously, we did a clean shutdown, so only the log checkpoint
information would have been read from the redo log. With this change,
we will be reading and applying encrypted redo log records.

include/start_mysqld.inc: Observe $restart_parameters.

encryption.innodb-log-encrypt: Remove some unnecessary statements,
and instead of restarting the server and concurrently accessing
the files while the server is running, kill the server, check the
files, and finally start up the server.

innodb.log_data_file_size: Use start_mysqld.inc with $restart_parameters.
2017-01-12 12:33:46 +02:00
Marko Mäkelä
3d46768da2 Merge 10.1 into 10.2 2017-01-09 09:47:12 +02:00
Marko Mäkelä
4b05d60e62 Make encryption.innodb_lotoftables more robust.
Perform a slow shutdown at the start of the test, and create all
InnoDB tables with STATS_PERSISTENT=0, so that any I/O related to
background tasks (change buffer merge, purge, persistent statistics)
should be eliminated.
2017-01-09 09:32:44 +02:00
Marko Mäkelä
4e7b382d31 Merge 10.1 to 10.2
Most conflicts are related to the MDEV-11638 InnoDB shutdown refactoring.
2017-01-05 10:48:03 +02:00
Marko Mäkelä
ffb38c9771 MDEV-8139 Fix scrubbing tests
encryption.innodb_scrub: Clean up. Make it also cover ROW_FORMAT=COMPRESSED,
removing the need for encryption.innodb_scrub_compressed.
Add a FIXME comment saying that we should create a secondary index, to
demonstrate that also undo log pages get scrubbed. Currently that is
not working!

Also clean up encryption.innodb_scrub_background, but keep it disabled,
because the background scrubbing does not work reliably.

Fix both tests so that if something is not scrubbed, the test will be
aborted, so that the data files will be preserved. Allow the tests to
run on Windows as well.
2017-01-05 00:20:17 +02:00
Jan Lindström
403f6e9607 MDEV-11705: InnoDB: Failing assertion: (&log_sys->mutex)->is_owned() if server started with innodb-scrub-log
Problem was that log_scrub function did not take required log_sys mutex.

Background: Unused space in log blocks are padded with MLOG_DUMMY_RECORD if innodb-scrub-log
is enabled. As log files are written on circular fashion old log blocks can be reused
later for new redo-log entries. Scrubbing pads unused space in log blocks to avoid visibility
of the possible old redo-log contents.

log_scrub(): Take log_sys mutex

log_pad_current_log_block(): Increase srv_stats.n_log_scrubs if padding is done.

srv0srv.cc: Set srv_stats.n_log_scrubs to export vars innodb_scrub_log

ha_innodb.cc: Export innodb_scrub_log to global status.
2017-01-03 11:22:49 +02:00
Marko Mäkelä
970f17cbfc Merge 10.1 into 10.2 2016-12-30 08:56:13 +02:00
Marko Mäkelä
341c375d4b Merge 10.1 into 10.2 2016-12-30 08:53:54 +02:00
Sergei Golubchik
ab89359dde enable tests that were skipped because of have_xtradb
* some of these tests run just fine with InnoDB:
   -> s/have_xtradb/have_innodb/
* sys_var tests did basic tests for xtradb only variables
   -> remove them, they're useless anyway (sysvar_innodb does it better)
* multi_update had innodb specific tests
   -> move to multi_update_innodb.test
2016-12-29 13:23:49 +01:00
Sergei Golubchik
4a5d25c338 Merge branch '10.1' into 10.2 2016-12-29 13:23:18 +01:00
Jan Lindström
283e9cf4cb MDEV-11656: 'Data structure corruption' IMPORT TABLESPACE doesn't work for encrypted InnoDB tables if space_id changed
Problem was that for encryption we use temporary scratch area for
reading and writing tablespace pages. But if page was not really
decrypted the correct updated page was not moved to scratch area
that was then written. This can happen e.g. for page 0 as it is
newer encrypted even if encryption is enabled and as we write
the contents of old page 0 to tablespace it contained naturally
incorrect space_id that is then later noted and error message
was written. Updated page with correct space_id was lost.

If tablespace is encrypted we use additional
temporary scratch area where pages are read
for decrypting readptr == crypt_io_buffer != io_buffer.

Destination for decryption is a buffer pool block
block->frame == dst == io_buffer that is updated.
Pages that did not require decryption even when
tablespace is marked as encrypted are not copied
instead block->frame is set to src == readptr.

If tablespace was encrypted we copy updated page to
writeptr != io_buffer. This fixes above bug.

For encryption we again use temporary scratch area
writeptr != io_buffer == dst
that is then written to the tablespace

(1) For normal tables src == dst ==  writeptr
ut_ad(!encrypted && !page_compressed ?
	src == dst && dst == writeptr + (i * size):1);
(2) For page compressed tables src == dst == writeptr
ut_ad(page_compressed && !encrypted ?
	src == dst && dst == writeptr + (i * size):1);
(3) For encrypted tables src != dst != writeptr
ut_ad(encrypted ?
	src != dst && dst != writeptr + (i * size):1);
2016-12-28 16:32:45 +02:00
Jan Lindström
55eb7120a0 MDEV-11218: encryption.innodb_encryption_discard_import failed in buildbot
Try to stabilize test cases. These test behave badly when run in certain order.
2016-12-22 14:02:51 +02:00
Marko Mäkelä
8375a2c1ce MDEV-11585 Hard-code the shared InnoDB temporary tablespace ID at -1
MySQL 5.7 supports only one shared temporary tablespace.

MariaDB 10.2 does not support any other shared InnoDB tablespaces than
the two predefined tablespaces: the persistent InnoDB system tablespace
(default file name ibdata1) and the temporary tablespace
(default file name ibtmp1).

InnoDB is unnecessarily allocating a tablespace ID for the predefined
temporary tablespace on every startup, and it is in several places
testing whether a tablespace ID matches this dynamically generated ID.

We should use a compile-time constant to reduce code size and to avoid
unnecessary updates to the DICT_HDR page at every startup.

Using a hard-coded tablespace ID will should make it easier to remove the
TEMPORARY flag from FSP_SPACE_FLAGS in MDEV-11202.
2016-12-19 16:24:10 +02:00
Jan Lindström
9f31949b64 MDEV-10739: encryption.innodb-page_encryption_compression fails with timeout on valgrind
Test moved to big_test and not run on valgrind. Test heavy especially on debug builds.
2016-11-29 08:35:51 +02:00
Jan Lindström
bb4b8c7470 MDEV-9099: Test encryption.innodb_encryption_discard_import fails on buildbot 2016-10-29 10:09:06 +03:00
Jan Lindström
885577fb10 MDEV-11004: Unable to start (Segfault or os error 2) when encryption key missing
Two problems:

(1) When pushing warning to sql-layer we need to check that thd != NULL
to avoid NULL-pointer reference.

(2) At tablespace key rotation if used key_id is not found from
encryption plugin tablespace should not be rotated.
2016-10-29 10:09:06 +03:00
Monty
14b1c8c80d After merge and bug fixes
- Fixed compiler warnings
- Removed have_debug.inc from innochecksum_3
- Fixed race condition in innodb_buffer_pool_load
- Fixed merge issue in innodb-bad-key-change.test
- Fixed missing array allocation that could cause
  function_defaults_notembedded to fail
- Fixed thread_cache_size_func
2016-10-05 01:11:08 +03:00
Monty
af7490f95d Remove end . from error messages to get them consistent
Fixed a few failing tests
2016-10-05 01:11:08 +03:00
Jan Lindström
2e7baca97d Replace tablespace numbers to make repeatable. 2016-10-03 08:23:30 +03:00
Jan Lindström
737295c752 MDEV-10727: Merge 5.7 Innochecksum with 5.6 2016-09-30 08:08:19 +03:00
Jan Lindström
4e2a0c34b0 MDEV-10888: encryption.filekeys_emptyfile fails in buildbot with valgrind
Problem was that length of the filekeys file was not checked and if
length is less than OpenSSL_prefix_len uninitialized memory was
accessed.
2016-09-27 07:54:27 +03:00
Jan Lindström
e136aa1ba7 Fix test failure. Need to mask more tablespace numbers as they
are not consistent on parallel mtr runs.
2016-09-23 09:11:11 +03:00