Optimizer Context code changed format_and_store_row() to check both
table->read_set and table->write_set (when required). It used to
use one of those depending on the lock level.
But we don't set the table->read_set bit so we can get an assertion
failure when dumping the column value.
This is fairly rare as Optimizer Context now reads all columns, and the
only other user is statements like "DBUG_PRINT("dml", dbug_format_row(..."
There are 2 problems: -
1. The REPLACE statement that is recorded doesn't store the
value of geometry type field correctly.
2. The table definition that got recorded has fields with non-null constraint,
and no default value is specified.
Also, the "REPLACE INTO" statement that gets stored in the context,
doesn't have any value specified for these non-null fields.
Solution is to: -
1. When using REPLACE INTO statement, store all the non-numeric values in HEX,
whenever conversion from field's charset to output's charset is lossy.
2. Instead of storing only the column values that were projected in the
query, store all the non-virtual column values into the recorded
REPLACE INTO statement.
Implementation details: -
1. Introduce a new method is_charset_conversion_lossless() in filesort.cc,
to check if the output charset to which field's data is being written to,
results in a lossless conversion. If so, non-numeric values being witten
using REPLACE INTO statement are stored in string representation,
else they are converted to HEX.
2. Modify join_read_const(), and join_read_system() methods in sql_select.cc,
and opt_sum_query() method in opt_sum.cc the following way: -
a. Extend the read_set to make sure, we read all the non-virtual column
using Optimizer_context_recorder::prepare_captured_row_read().
This method also saves the original read_set.
b. Read the row.
c. Dump the row into the context when no error is noticed while
reading. Irrespective of the error, restore back the read_set state to
the original using Optimizer_context_recorder::finish_captured_row_read()
The problem was with multi-table DELETE deleting from a const table.
const table was saved into the context but no columns were printed,
despite that the query's WHERE had references to them.
The cause was format_and_store_row() which found lock_type to be
TL_WRITE and so checked table->write_set, which was empty.
Fixed by making to check both read and write set.
The only other user of row printing seems to be DBUG_PRINT messages.
For constant tables, the rows are stored in the context using the
charset my_charset_utf8mb4_bin. However, that is not correct, as
every field in the row could use its own charset.
This leads to a replay failure, as the data that gets read from the
sql_script file is different from the original row data.
This PR addresses uses field's own charset instead of
my_charset_utf8mb4_bin.
REPLACE INTO statements would now use my_charset_utf8mb4_bin conversion
Also, the sql_script file structure would now look like
SET NAMES utf8mb4;
SET var1=value1;
.
.
CREATE DATABASE 'name' if NOT..;
USE DATABASE 'name';
CREATE TABLE ...;
.
REPLACE INTO ..;
.
.
set @opt_context='...';
.
.
Additionally, made few changes to the documentation comments
1. Add escaping in strings if they contain quotes in the names
2. Record CONST insert statement for MYISAM system table
3. Add tests to oad the sql_script file that is produced with
record_context=ON and then use it to run same queries on empty tables.
This task loads the stats of the tables that are used in a query,
from the trace into the optimizer. This feature is controlled
by optimizer_replay_context, and points to an user defined variable name,
that has the context information.
The stats such as num_of_records present in the table, indexes if present,
their names, along with the average number of records_per_key
with in each index, cost of reading an index are loaded from the trace.
Additionally, stats from range analysis i.e. ranges, and the
corresponding number of records, along with their costs are also
loaded from the trace.
The trace context which is in JSON format is firstly set into the
user defined session variable, and this variable name is set to
optimizer_replay_context system variable.
Later, when a user issues a query, the contents of the json context is
parsed and an in memory representation is built using the class
Optimizer_context_replay. This class is then used by the optimizer
to update and save original values of the tables, indexes, and range stats
in the methods "set_statistics_for_table()" of sql_statistics.cc, and
"check_quick_select()" of opt_range.cc.
After the query gets finished, the statistics that were updated
in the optimizer are restored back to the saved original values.
The entry point for parsing the json structure is in
"mysql_execute_command()" of sql_parse.cc, and similarly exit point
i.e. to restore the saved stats of the optimizer is at the end
of the same method.
Two new warning/error messages are introduced:
ER_JSON_OPTIMIZER_REPLAY_CONTEXT_PARSE_FAILED, and
ER_JSON_OPTIMIZER_REPLAY_CONTEXT_MATCH_FAILED
Reason:
=======
Following the changes in MDEV-38734, the server no longer marks
all indexed virtual columns during an UPDATE operation.
Consequently, ha_innobase::update() populates the upd_t vector
with old_vrow but omits the actual data for these virtual columns.
Despite this omission, trx_undo_page_report_modify() continues to
write metadata for indexed virtual columns into the undo log. Because
the actual values are missing from the update vector, the undo log
entry is recorded without the historical data for these columns.
When the purge thread processes the undo log to reconstruct a
previous record state for MVCC, it identifies an indexed virtual
column but finds no associated data.
The purge thread incorrectly interprets this missing data as a NULL
value, rather than a "missing/unrecorded" value. The historical
record is reconstructed with an incorrect NULL for the virtual column.
This causes the purge thread to incorrectly identify and purge
records that are not actually delete-marked, leading to abort
of server.
Solution:
=========
ha_innobase::column_bitmaps_signal(): Revert the column-marking
logic to the state prior to commit a4e4a56720, ensuring all
indexed virtual columns are unconditionally marked during an UPDATE.
The previous "optimization" attempted to manually detect indexed
column changes before marking virtual columns. The manual check
for indexed column modifications is redundant. InnoDB already
provides the UPD_NODE_NO_ORD_CHANGE flag within row_upd_step().
This flag is being used in trx_undo_page_report_modify() and
trx_undo_read_undo_rec() should log or read virtual column values.
Refactored column_bitmaps_signal() to accept a mark_for_update
parameter that controls when indexed virtual columns are marked.
TABLE::mark_columns_needed_for_update() is the only place that needs
mark_for_update=true because only UPDATE operations need to mark
indexed virtual columns for InnoDB's undo logging mechanism.
INSERT operation is already handled by
TABLE::mark_virtual_columns_for_write(insert_fl=true).
Even the commit a4e4a56720 changes are
going to affect TABLE::mark_virtual_column_for_write(false) and
It is called during UPDATE operation, and that's when
column_bitmaps_signal() needs to mark indexed virtual columns.
Online DDL has separate code path which is handled by
row_log_mark_virtual_cols() for all DML operations
- Removed duplicate words, like "the the" and "to to"
- Removed duplicate lines (one double sort line found in mysql.cc)
- Fixed some typos found while searching for duplicate words.
Command used to find duplicate words:
egrep -rI "\s([a-zA-Z]+)\s+\1\s" | grep -v param
Thanks to Artjoms Rimdjonoks for the command and pointing out the
spelling errors.
7544fd4cae had to make use of a static array to avoid memory
use-after-free or leak.
Instead, let us make a function returning String, this is the only way
to automatically manage the memory after the function returned.
To make it all correct, move constructor is added. Normally, it is
expected, that the constructor will be elided upon return of an object
by value, but if something goes different, or -fno-elide-constructors is
used, we can have a problem. So this was a move constructor avoids
copy elision-related UB.
dbug_print_row returning char* is still there for convenient use in a
debugger.
* rpl.rpl_system_versioning_partitions updated for MDEV-32188
* innodb.row_size_error_log_warnings_3 changed error for MDEV-33658
(checks are done in a different order)
Partial commit of the greater MDEV-34348 scope.
MDEV-34348: MariaDB is violating clang-16 -Wcast-function-type-strict
The functions queue_compare, qsort2_cmp, and qsort_cmp2
all had similar interfaces, and were used interchangable
and unsafely cast to one another.
This patch consolidates the functions all into the
qsort_cmp2 interface.
Reviewed By:
============
Marko Mäkelä <marko.makela@mariadb.com>
create templates
thd->alloc<X>(n) to use instead of (X*)thd->alloc(sizeof(X)*n)
and the same for thd->calloc(). By the default the type is char,
so old usage of thd->alloc(size) works too.
Bounded_queue<> pretended to be a typesafe C++ wrapper
on top of pure C queues.h.
But it wasn't, it was tightly bounded to filesort and only useful there.
* implement Queue<> - a typesafe C++ wrapper on top of QUEUE
* move Bounded_queue to filesort.cc, remove pointless "generalizations"
change it to use Queue.
* remove bounded_queue.h
* change subselect_rowid_merge_engine to use Queue, not QUEUE
The TIMESTAMP related code did not handle AUTO_SEC_PART_DIGITS.
FROM_UNIXTIME() sets its member 'decimals' to AUTO_SEC_PART_DIGITS.
So some scripts involving FROM_UNIXTIME() crashed on assert in debug
builds and returned unexpected results in release builds.
During a query execution some sorting and grouping operations
on strings may be involved. System variable max_sort_length defines
the maximum number of bytes to use when comparing strings during
sorting/grouping. Thus, the comparable parts of strings may be less
than their actual size, so the results of the query may be not
sorted/grouped properly.
To indicate that some comparisons were done on a truncated lengths,
a new warning has been introduced with this commit.
Step#1: fixing the return type of strnxfrm() from size_t to this structure:
typedef struct
{
size_t m_output_length;
size_t m_source_length_used;
uint m_warnings;
} my_strnxfrm_ret_t;
