This commit rewrites the rpl_master_has_bug() mechanism to solve a problem
with invalid memory access. The rpl_master_has_bug() mechanism detects
certain bugs depending on the master version, and uses that to enable
specific work-arounds on the slave. The problem was that
rpl_master_has_bug() accessed Relay_log_info::description_event_for_exec
that is not valid to access from concurrent parallel replication worker
threads, only from the SQL driver thread. Thus it could use the wrong event
or access invalid/freed memory.
This patch instead computes a bitmask of detected bugs when the SQL driver
thread processes the format description event, and reads that bitmask with
an atomic load from the worker threads. The bitmask of bugs can only change
when the master restarts with a new version, and we do not replicate events
concurrently across a format description event from a master restart. Thus,
the bitmask is safe to read concurrently from the Relay_log_info object
without locking.
This also avoids an expensive match of each entry in the bug list against
the master server version done for every single call to
rpl_master_has_bug(), which could be quite expensive when done eg. per field
in row events as in Field_string::compatible_field_size().
Also remove redundant conditional in table_def::compatible_with().
Thanks to Andrei Elkin for the idea to safely read the bitmask
concurrently from the Relay_log_info.
Reviewed-by: Andrei Elkin <andrei.elkin@pp.inet.fi>
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
Problem was that check for partitioned tables was missing
because then partition implementing handlerton should be used in
condition instead.
Thanks to Roel Van de Paar <roel.vandepaar@mariadb.com> for
providing test case and fix candidate.
In MDEV-25644 vers_check_update() sets vers_write to false in case it
returns false. It is ok for UPDATE but is not correct for ODKU is bulk
insert requires vers_write on next tuple.
The fix return vers_write value back when vers_check_update() and
related vers_insert_history_row() are done in ODKU.
Problem:
REPLACE DELAYED into a system-versioned table read an uninitialized
lex->vers_conditions.delete_history in TABLE::delete_row().
Delayed_insert has its own THD, and hence its own LEX, of which the
constructor copies only a few fields from the parser LEX.
vers_conditions was not copied, so it stayed uninitialized. It is read
when a REPLACE resolves a duplicate on a versioned table.
Fix:
Copy vers_conditions from the parser LEX in the Delayed_insert
constructor.
Remove the unscoped #pragma GCC diagnostic ignored
"-Wdeprecated-declarations" from include/violite.h and replace all
sprintf/vsprintf calls with snprintf/vsnprintf across the codebase.
Where possible, pass actual buffer sizes as function parameters
instead of using hardcoded constants or recomputing strlen().
Notable API changes:
- get_date(): added to_len parameter for buffer size
- nice_time(), end_timer() in client/mysql.cc: added buff_size
- generate_new_name(): added name_size parameter
- calc_md5(): added buffer_size parameter (fixes off-by-one)
- MD5_HASH_TO_STRING macro: added _size parameter
- Added MYSQL_UDF_MAX_RESULT_LENGTH constant in mysql_com.h
Also fixed a pre-existing bug in sql/rpl_mi.cc where the ellipsis
"..." was written to the wrong buffer (dbuff instead of buff).
Vendored code is excluded: extra/readline, extra/wolfssl,
wsrep-lib, libmariadb, storage/mroonga/vendor, zlib.
Prior to this change, AFTER UPDATE did not fire, if data was unchanged.
Now this is fixed. Also added a test to verify that trigger behavior for
on-duplicate-update is the same as normal update, in case data does not
change.
When prepare_for_replace() fails, the code returned early via
DBUG_RETURN(1), bypassing the abort label which resets
thd->abort_on_warning to 0. This left the THD in a dirty state,
triggering the dispatch_command assertion at end of command dispatch.
Signed-off-by: Amodh Dhakal <amodhakal@gmail.com>
Problem:
=========
An assertion failure occurs in InnoDB during consecutive
ALTER TABLE operations using the COPY algorithm. The crash
happens during the table rename phase because the
source table (dict_table_t::n_ref_count) is non-zero, despite
the thread holding an exclusive Metadata Lock (MDL).
Reason:
========
When ALTER IGNORE TABLE is executed via the COPY algorithm,
it generates undo logs for every row inserted into the
intermediate table (e.g., #sql-alter-...). The background Purge
Thread, responsible for cleaning up these undo logs, attempts
to take an MDL on the table to prevent the table from
being dropped while in use.
Race condition:
==================
First ALTER: Creates #sql-alter-, copies data, and renames it to t1.
Purge Activation: The Purge thread picks up the undo logs from step 1.
It takes an MDL on the temporary name (#sql-alter-) and increments
the table's n_ref_count.
Identity Shift: InnoDB renames the physical table object to t1, but
the Purge thread still holds a reference to this object.
Second ALTER: Starts a new copy process. When it attempts to rename
the "new" t1 to a backup name, it checks if n_ref_count == 0.
Because the Purge thread is still "pinning" the object to
clean up logs from the first ALTER, the count is > 0,
triggering the assertion failure.
Solution:
========
ALTER IGNORE TABLE needs row-level undo logging to easily
roll back the last inserted row in case of duplicate key errors.
By discarding the last undo log record after inserting each row,
purge will not process any log records generated by
ALTER IGNORE TABLE, preventing unexpected access from the purge
subsystem during subsequent DDL operations.
Make skip_alter_undo (1-bit) to (2-bit enum)
to support different ALTER operation modes:
- NO_UNDO (0): Normal mode with standard undo logging
- SKIP_UNDO (1): ALTER mode that skips undo logging
- IGNORE_UNDO (2): ALTER IGNORE mode that rewrites undo blocks
trx_undo_report_row_operation(): Add ALTER IGNORE undo
rewriting logic. Store old undo record info before writing
new records for IGNORE_UNDO mode. Reset undo top_offset
to maintain only latest insert undo
row_ins_clust_index_entry_low(): Prevent ALTER IGNORE from
using bulk insert optimization. since it requires to write
undo log for each row operation that's incompatible with
existing bulk operation.
Reason:
======
During InnoDB DDL, statistics updation fails due to lock wait
timeout and calls push_warning_printf() to generate warnings
but then returns success, causing the SQL layer
to attempt calling set_ok_status() when the diagnostics area
is already set.
Solution:
=========
By temporarily setting abort_on_warning to false around operations
that prevents warning to error escalation and restore the original
setting after calling HA_EXTRA_END_ALTER_COPY for alter operation.
don't copy field default values and check constraints in
CREATE ... SELECT.
CREATE ... SELECT means a table is created from a *result set*
not from some other table.
For backward compatibility, though, let's keep copying constant
default values and the "compressed" attribute.
Caused by optimization done in 2e2b2a0469.
Cannot use lookup_handler in default branch of locate_dup_record() as
InnoDB update depends on positioned record and update is done in table
main handler.
The patch reverts some non-pure changes done by 2e2b2a0469 to
original logic from 72429cad. There was no long_unique_table condition
to init search on table->file, so we get into default branch with long
unique and table->file search uninitialized.
ha_rnd_init_with_error() on demand for HA_DUPLICATE_POS branch was
original logic as well.
More info: 2e2b2a0469 reverts 5e345281e3, but it seems to be OK as
MDEV-3888 test case passes. mysql-5.6.13 has the original code with
HA_WHOLE_KEY as well.
Timestamp-versioned row deletion was exposed to a collisional problem: if
current timestamp wasn't changed, then a sequence of row delete+insert could
get a duplication error. A row delete would find another conflicting history row
and return an error.
This is true both for REPLACE and DELETE statements, however in REPLACE, the
"optimized" path is usually taken, especially in the tests. There, delete+insert
is substituted for a single versioned row update. In the end, both paths end up
as ha_update_row + ha_write_row.
The solution is to handle a history collision somehow.
From the design perspective, the user shouldn't experience history rows loss,
unless there's a technical limitation.
To the contrary, trxid-based changes should never generate history for the same
transaction, see MDEV-15427.
If two operations on the same row happened too quickly, so that they happen at
the same timestamp, the history row shouldn't be lost. We can still write a
history row, though it'll have row_start == row_end.
We cannot store more than one such historical row, as this will violate the
unique constraint on row_end. So we will have to phisically delete the row if
the history row is already available.
In this commit:
1. Improve TABLE::delete_row to handle the history collision: if an update
results with a duplicate error, delete a row for real.
2. use TABLE::delete_row in a non-optimistic path of REPLACE, where the
system-versioned case now belongs entirely.
We had a protection against it, by allowing versioned delete if:
trx->id != table->vers_start_id()
For replace this check fails: replace calls ha_delete_row(record[2]), but
table->vers_start_id() returns the value from record[0], which is irrelevant.
The same problem hits Field::is_max, which may have checked the wrong record.
Fix:
* Refactor Field::is_max to optionally accept a pointer as an argument.
* Refactor vers_start_id and vers_end_id to always accept a pointer to the
record. there is a difference with is_max is that is_max accepts the pointer to
the
field data, rather than to the record.
Method val_int() would be too effortful to refactor to accept the argument, so
instead the value in record is fetched directly, like it is done in
Field_longlong.
See also MDEV-30046.
Idempotent write_row works same as REPLACE: if there is a duplicating
record in the table, then it will be deleted and re-inserted, with the
same update optimization.
The code in Rows:log_event::write_row was basically copy-pasted from
write_record.
What's done:
REPLACE operation was unified across replication and sql. It is now
representred as a Write_record class, that holds the whole state, and allows
re-using some resources in between the row writes.
Replace, IODKU and single insert implementations are split across different
methods, reluting in a much cleaner code.
The entry point is preserved as a single Write_record::write_record() call.
The implementation to call is chosen on the constructor stage.
This allowed several optimizations to be done:
1. The table key list is not iterated for every row. We find last unique key in
the order of checking once and preserve it across the rows. See last_uniq_key().
2. ib_handler::referenced_by_foreign_key acquires a global lock. This call was
done per row as well. Not all the table config that allows optimized replace is
folded into a single boolean field can_optimize. All the fields to check are
even stored in a single register on a 64-bit platform.
3. DUP_REPLACE and DUP_UPDATE cases now have one less level of indirection
4. modified_non_trans_tables is checked and set only when it's really needed.
5. Obsolete bitmap manipulations are removed.
Also:
* Unify replace initialization step across implementations:
add prepare_for_replace and finalize_replace
* alloca is removed in favor of mem_root allocation. This memory is reused
across the rows.
* An rpl-related callback is added to the replace branch, meaning that an extra
check is made per row replace even for the common case. It can be avoided with
templates if considered a problem.
Timestamp-versioned row deletion was exposed to a collisional problem: if
current timestamp wasn't changed, then a sequence of row delete+insert could
get a duplication error. A row delete would find another conflicting history row
and return an error.
This is true both for REPLACE and DELETE statements, however in REPLACE, the
"optimized" path is usually taken, especially in the tests. There, delete+insert
is substituted for a single versioned row update. In the end, both paths end up
as ha_update_row + ha_write_row.
The solution is to handle a history collision somehow.
From the design perspective, the user shouldn't experience history rows loss,
unless there's a technical limitation.
To the contrary, trxid-based changes should never generate history for the same
transaction, see MDEV-15427.
If two operations on the same row happened too quickly, so that they happen at
the same timestamp, the history row shouldn't be lost. We can still write a
history row, though it'll have row_start == row_end.
We cannot store more than one such historical row, as this will violate the
unique constraint on row_end. So we will have to phisically delete the row if
the history row is already available.
In this commit:
1. Improve TABLE::delete_row to handle the history collision: if an update
results with a duplicate error, delete a row for real.
2. use TABLE::delete_row in a non-optimistic path of REPLACE, where the
system-versioned case now belongs entirely.
We had a protection against it, by allowing versioned delete if:
trx->id != table->vers_start_id()
For replace this check fails: replace calls ha_delete_row(record[2]), but
table->vers_start_id() returns the value from record[0], which is irrelevant.
The same problem hits Field::is_max, which may have checked the wrong record.
Fix:
* Refactor Field::is_max to optionally accept a pointer as an argument.
* Refactor vers_start_id and vers_end_id to always accept a pointer to the
record. there is a difference with is_max is that is_max accepts the pointer to
the
field data, rather than to the record.
Method val_int() would be too effortful to refactor to accept the argument, so
instead the value in record is fetched directly, like it is done in
Field_longlong.
See also MDEV-30046.
Idempotent write_row works same as REPLACE: if there is a duplicating
record in the table, then it will be deleted and re-inserted, with the
same update optimization.
The code in Rows:log_event::write_row was basically copy-pasted from
write_record.
What's done:
REPLACE operation was unified across replication and sql. It is now
representred as a Write_record class, that holds the whole state, and allows
re-using some resources in between the row writes.
Replace, IODKU and single insert implementations are split across different
methods, reluting in a much cleaner code.
The entry point is preserved as a single Write_record::write_record() call.
The implementation to call is chosen on the constructor stage.
This allowed several optimizations to be done:
1. The table key list is not iterated for every row. We find last unique key in
the order of checking once and preserve it across the rows. See last_uniq_key().
2. ib_handler::referenced_by_foreign_key acquires a global lock. This call was
done per row as well. Not all the table config that allows optimized replace is
folded into a single boolean field can_optimize. All the fields to check are
even stored in a single register on a 64-bit platform.
3. DUP_REPLACE and DUP_UPDATE cases now have one less level of indirection
4. modified_non_trans_tables is checked and set only when it's really needed.
5. Obsolete bitmap manipulations are removed.
Also:
* Unify replace initialization step across implementations:
add prepare_for_replace and finalize_replace
* alloca is removed in favor of mem_root allocation. This memory is reused
across the rows.
* An rpl-related callback is added to the replace branch, meaning that an extra
check is made per row replace even for the common case. It can be avoided with
templates if considered a problem.
Lots of different cases, SELECT, SELECT DEFAULT(),
UPDATE t SET x=DEFAULT, prepares statements,
opening of a table for the I_S, prelocking (so TL_WRITE),
insert with subquery (so SQLCOM_SELECT), etc.
Don't check NEXTVAL privileges in fix_fields() anymore, it cannot
possibly handle all the cases correctly. Make a special method
Item_func_nextval::check_access() for that and invoke it from
* fix_fields on explicit SELECT NEXTVAL()
(but not if NEXTVAL() is used in a DEFAULT clause)
* when DEFAULT bareword in used in, say, UPDATE t SET x=DEFAULT
(but not if DEFAULT() itself is used in a DEFAULT clause)
* in CREATE TABLE
* in ALTER TABLE ALGORITHM=INPLACE (that doesn't go CREATE TABLE path)
* on INSERT
helpers
* Virtual_column_info::check_access() to walk the item tree and invoke
Item::check_access()
* TABLE::check_sequence_privileges() to iterate default expressions
and invoke Virtual_column_info::check_access()
also, single-table UPDATE in prepared statements now associates
value items with fields just as multi-update already did, fixes the
case of PREPARE s "UPDATE t SET x=?"; EXECUTE s USING DEFAULT.
When the SELECT sub-statement executes a stored function that is defined
to modify a non-transactional table, like
delimiter |;
create function f_ia(arg int)
returns integer
begin
insert into ti_pk set a=1;
insert into ta set a=1;
insert into ti_pk set a=arg;
return 1;
end |
delimiter ;|
any modified records that the function has succeeded
on must be binlogged as a "side effect" of CREATE-SELECT.
It is expected that a failing CREATE-SELECT like
--error ER_DUP_ENTRY
set statement binlog_format = ROW for create table t_y (a int) engine=aria select f_ia(1 /* err in Innodb after Aria stmt is done */) as a;
leaves upon itself the following state:
include/show_binlog_events.inc
Log_name Pos Event_type Server_id End_log_pos Info
master-bin.000001 # Gtid # # BEGIN GTID #-#-#
master-bin.000001 # Table_map # #
table_id: # (test. ta)
master-bin.000001 # Write_rows_v1 # #
table_id: # flags: STMT_END_F
master-bin.000001 # Query # # COMMIT
select * from ta;
a
1
select count(*) = 0 from ti_pk;
true
However it's not so for the binlog part.
The reason is that prior to MDEV-34150 fixes the CREATE-SELECT's
errored phase leaves the binlog caches intact (the file:pos from 10.11 c06c36218a)
to defer their reset to the rollback phase of the top-level
/* the statement cache gets binlogged */
where the side-effect changes gets binlogged.
MDEV-34150 fixes harmed (+#4 line) the statement cache in particular
in the error phase (file:pos are from 395db6f1d5 the current 11.8 )
/* The caches incl the statement cache are gone */
/* 'cos of MDEV-34150 */
+#4 0x00005d75f9b6a92e in THD::binlog_remove_rows_events (this=0x52c000240288) at log.cc:579
Apparently it should not have been there, as proper emptying (either
with reset for the transactional cache or flush and then reset for the
statement cache) is (must be) always done via binlog_rollback of the
top-level statement.
To observe the above requirement the case is fixed with the removal of
thd->binlog_remove_rows_events() and its definition.
Tested with rpl.rpl_create_select_row.
Reviewed-by Brandon Nesterenko.
Get rid of need of matherialization for usual INSERT (cache results in
Item_cache* if needed)
- subqueries in VALUE do not see new records in the table we are
inserting to
- subqueries in RETIRNING prohibited to use the table we are inserting to
Problem:
========
- After commit cc8eefb0dc (MDEV-33087),
InnoDB does use bulk insert operation for ALTER TABLE.. ALGORITHM=COPY
and CREATE TABLE..SELECT as well. InnoDB fails to clear the bulk
buffer when it encounters error during CREATE..SELECT. Problem
is that while transaction cleanup, InnoDB fails to identify
the bulk insert for DDL operation.
Fix:
====
- Represent bulk_insert in trx by 2 bits. By doing that, InnoDB
can distinguish between TRX_DML_BULK, TRX_DDL_BULK. During DDL,
set bulk insert value for transaction to TRX_DDL_BULK.
- Introduce a parameter HA_EXTRA_ABORT_ALTER_COPY which rollbacks
only TRX_DDL_BULK transaction.
- bulk_insert_apply() happens for TRX_DDL_BULK transaction happens
only during HA_EXTRA_END_ALTER_COPY extra() call.
