Applying HEX encoding write writting an ENUM/SET TYPELIB to FRM
if the TYPELIB has 0x00 bytes in the value.
This HEX encoding was earlier used only to write UCS2/UTF16/UTF32 TYPELIBs.
A new flag FIELDFLAG_FRM_HEX_ENCODED_TYPELIB was added to indicate
that the TYPELIB is hex encoded. It's used only inside FRM.
Note, it's mangled with FIELDFLAG_TREAT_BIT_AS_CHAR.
This should not be harmful:
- BIT and ENUM/SET columns are handled by two separate code branches
when opening an FRM
- The flag is unset immediately after decoding TYPELIB, so the rest
of the code does not se an unexpected flag combination.
numerous checks that the frm is valid, no OOB reads,
values make sense (number of keyparts not less than number of keys,
no keys means no keyparts, number of long unique fields is not larger than
number of fields, fields values in the record don't overlap and don't
go over record ends, and so on). most asserts were changed to if()'s
Introduce CAN_BE_NULL annotation into Table_check_intact checks.
By default columns are considered NOT NULL, nullable
columns must be explicitly marked with CAN_BE_NULL.
All TABLE_FIELD_TYPE arrays are fixed as above.
Assisted-By: Claude:claude-4.6-sonnet
1.
Only allow #mysql50# prefix on table names that
can not be decoded from a file name to a table name.
It is an error to put this prefix on a name that can be accessed
as a valid table name without a prefix. Use `t1` not `#mysql50#t1`,
use `a-b` not `#mysql50#a@002db`.
2.
Don't allow creation of new tables with the #mysql50# prefix.
Except when in SQLCOM_ALTER_DB_UPGRADE - main.upgrade test runs
ALTER DATABASE `#mysql50#mysqltest-1` UPGRADE DATA DIRECTORY NAME;
This command moves tables from the old db name to the new db name,
and some of these tables also have old #mysql50# names.
3.
Also don't allow to create such databases, views, partitions,
subpartitions, sequences, triggers.
Three problems:
1. Option value specified as `foo bar` was not quoted,
printed as foo bar, breaking CREATE TABLE.
2. Option value that included */ was printed as is, in unknown
options when the whole option was commented out, this broke the
comment: /* opt='val*/ue' */
3. Option names can include */ too, `opt*/ion`=1
Fix: simplify value printing, ignore the original quoting, always
print the value quoted as a string. Break comment markers with an
escape (as *\/) if inside a comment. As \-escapes don't work inside,
backticks, we cannot use `-quoting for option values at all.
For the same reason we cannot allow */ inside option names, thus
let's only allow keywords for option names and reject fancy names
even if IGNORE_BAD_TABLE_OPTIONS is in force.
Analysis:
When OLD_VALUE() is used on a view, field resolution creates an
Item_direct_view_ref instead of an Item_ref where real_item() points to the
expression. But Item_old_field::fix_fields() continues assuming that a Field
object was available, but view references do not have a field pointer,
which resulted in a crash.
Fix:
Store the real referenced item in Item_old_field::expr and use it when
the OLD_VALUE() reference does not have a Field object. This allows
OLD_VALUE() to work with view fields and avoids dereferencing a NULL
field pointer. Fix relevant methods accordingly.
The prefix database_name was getting dropped in the explain extended plan note
for certain non-view blocks of queries.
alias_name_used field in the table is not initialized correctly in all
the query blocks.
This PR makes the database prefix to be added before the [alias]
table name consistently when they are part of the non-view query blocks.
A couple of things interact in this scenario.
1. A virtual column whose value depends on session state, such as a
VCOL calling DATE_FORMAT, has fix_fields (via fix_expr) run on it
again before each INSERT. For a table reused from the cache this runs in
Vcol_expr_context before setup_tables sets the table map. The test
first opens t1 with a SELECT that fails before setup_tables, leaving
it cached with map zero.
2. When the connection character set needs conversion, CONCAT (more
generally, any charset aggregation) wraps that column in a charset
converter. The converter evaluates any argument reported as
constant, so it calls val_str() on the column as part of
fix_fields, before a row exists and without the column in the read
set. On DEBUG builds this fails the marked_for_read assertion;
otherwise it reads an unread column.
Consequently:
When a table instance is opened, fix_fields runs with table->map
already nonzero, but the fix_fields (via fix_expr) call from vcol_fix_expr
did not have it nonzero; so set table->map=1 during Vcol_expr_context::init.
The original Vcol_expr_context::init() (MDEV-24176) called
init_lex_with_single_table, which set table->map= 1 as a side effect.
A later merge of 10.2 into 10.3 (6f6c74b0d1) replaced that call with
a backup of table->expr_arena, dropping the side effect.
The destructor of the Vcol_expr_context will reset the map back to its
original value.
The pushdown from HAVING into WHERE optimization cleans up and refixes
every condition to be pushed.
The virtual column (vcol) index substitution optimization replaces
vcol expressions in GROUP BY (and WHERE and ORDER BY) with vcol
fields.
The refixing requires the correct name resolution context to find the
vcol fields.
The commit 0316c6e4f2 assigns context
from the select_lex that the GROUP BY belongs to, but that may not
work when there are derived table subqueries.
In this commit we assign the correct context during substitution for
the newly constructed vcol Item_field in the substitution.
Also make the walk of vcol expressions with
intersect_field_part_of_key not descend to subquery since the
generated columns can't have subqueries.
Alternative considered:
1. Assign the context when constructing vcol_info, in
unpack_vcol_info_from_frm. This does not work because the
current_context() in parsing is not the correct context, not to
mention that unpack_vcol_info_from_frm is not always called from a
SELECT statement.
2. Get the correct context for vcol_info after its construction and
before the substitution. Debugger with watch -a vcol_info shows that
there are no common functions accessing vcol_info before the
substitution.
3. Cache the name resolution context value to a new vcol_info->context
field and assign value to it in an Item_field constructor as well as
during substitution. The problem with this is that
vcol_info->context->select_lex may be cleared at the end of the one
SELECT statement, which can cause crash in the next SELECT. It looks
like vcol_info is meant for "static"/ddl information, not
statement-specific info, so it is the wrong place to cache the
context.
1. `TABLE_LIST::init_derived` is called twice during derived tables
processing: first time from `mysql_derived_init` and the second
time from `mysql_derived_prepare`. Both times there is a check
for whether an optimizer hint or switch setting allows a derived
table to be merged.
However, it is not necessary to restrict merging during the
initialization call, it is enough to apply hints/switch setting
during the preparation phase.
2. `open_normal_and_derived_tables()` runs processing of all phases
of derived tables handling with a single call. But for future
commits it is required to separate DT initialization from other
phases. This commit implements the separation.
Remove the unscoped #pragma GCC diagnostic ignored
"-Wdeprecated-declarations" from include/violite.h and replace all
sprintf/vsprintf calls with snprintf/vsnprintf across the codebase.
Where possible, pass actual buffer sizes as function parameters
instead of using hardcoded constants or recomputing strlen().
Notable API changes:
- get_date(): added to_len parameter for buffer size
- nice_time(), end_timer() in client/mysql.cc: added buff_size
- generate_new_name(): added name_size parameter
- calc_md5(): added buffer_size parameter (fixes off-by-one)
- MD5_HASH_TO_STRING macro: added _size parameter
- Added MYSQL_UDF_MAX_RESULT_LENGTH constant in mysql_com.h
Also fixed a pre-existing bug in sql/rpl_mi.cc where the ellipsis
"..." was written to the wrong buffer (dbuff instead of buff).
Vendored code is excluded: extra/readline, extra/wolfssl,
wsrep-lib, libmariadb, storage/mroonga/vendor, zlib.
Reason:
=======
Following the changes in MDEV-38734, the server no longer marks
all indexed virtual columns during an UPDATE operation.
Consequently, ha_innobase::update() populates the upd_t vector
with old_vrow but omits the actual data for these virtual columns.
Despite this omission, trx_undo_page_report_modify() continues to
write metadata for indexed virtual columns into the undo log. Because
the actual values are missing from the update vector, the undo log
entry is recorded without the historical data for these columns.
When the purge thread processes the undo log to reconstruct a
previous record state for MVCC, it identifies an indexed virtual
column but finds no associated data.
The purge thread incorrectly interprets this missing data as a NULL
value, rather than a "missing/unrecorded" value. The historical
record is reconstructed with an incorrect NULL for the virtual column.
This causes the purge thread to incorrectly identify and purge
records that are not actually delete-marked, leading to abort
of server.
Solution:
=========
ha_innobase::column_bitmaps_signal(): Revert the column-marking
logic to the state prior to commit a4e4a56720, ensuring all
indexed virtual columns are unconditionally marked during an UPDATE.
The previous "optimization" attempted to manually detect indexed
column changes before marking virtual columns. The manual check
for indexed column modifications is redundant. InnoDB already
provides the UPD_NODE_NO_ORD_CHANGE flag within row_upd_step().
This flag is being used in trx_undo_page_report_modify() and
trx_undo_read_undo_rec() should log or read virtual column values.
Refactored column_bitmaps_signal() to accept a mark_for_update
parameter that controls when indexed virtual columns are marked.
TABLE::mark_columns_needed_for_update() is the only place that needs
mark_for_update=true because only UPDATE operations need to mark
indexed virtual columns for InnoDB's undo logging mechanism.
INSERT operation is already handled by
TABLE::mark_virtual_columns_for_write(insert_fl=true).
Even the commit a4e4a56720 changes are
going to affect TABLE::mark_virtual_column_for_write(false) and
It is called during UPDATE operation, and that's when
column_bitmaps_signal() needs to mark indexed virtual columns.
Online DDL has separate code path which is handled by
row_log_mark_virtual_cols() for all DML operations
Adding support for the strict cursor data types:
Example 1a:
TYPE rec0_t IS RECORD (a INT, VARCHAR(10));
TYPE cur0_t IS REF CURSOR RETURN rec0_t;
Example 1b:
TYPE rec0_t IS RECORD (a t1.a%TYPE, b t1.b%TYPE);
TYPE cur0_t IS REF CURSOR RETURN rec0_t;
Example 1c:
TYPE rec0_t IS RECORD (a INT, VARCHAR(10));
r0 rec0_t;
TYPE cur0_t IS REF CURSOR RETURN r0%TYPE;
Example 1d:
TYPE rec0_t IS RECORD (a t1.a%TYPE, b t1.b%TYPE);
r0 rec0_t;
TYPE cur0_t IS REF CURSOR RETURN r0%TYPE;
Example2a:
TYPE cur0_t IS REF CURSOR RETURN t1%ROWTYPE; -- t1 is a table
Example 2b:
r0 t1%ROWTYPE;
TYPE cur0_t IS REF CURSOR RETURN r0%TYPE;
Example3a:
CURSOR cursor_sample IS SELECT a,b FROM t1;
TYPE cur0_t IS REF CURSOR RETURN cursor_sample%ROWTYPE;
Example3b:
CURSOR cursor_sample IS SELECT a,b FROM t1;
r0 cursor_sample%ROWTYPE;
TYPE cur0_t IS REF CURSOR RETURN r0%TYPE;
If a cursor variable is declared with a RETURN clause then:
1. At OPEN type the data type of the SELECT list row is compared
for compatibility with the cursor RETURN data type.
The SELECT list row must be assignable to the RETURN type row.
If case if assignability is not meet, an error is raised
Assignability means:
- The arity of the SELECT list must be equal to the arity
of the RETURN clause
- Every n-th field of the SELECT list must be assignable to the
n-th field of the RETURN Clause
2. At FETCH time, the data is fetched in two steps:
a. On the first step the data is fetched into a virtual table
with the row type described in the RETURN clause
b. On the second step the data is copied from the virtual table
to the target fetch list. Data type conversion can happen
on this step.
Change details:
Adding new methods:
- sp_cursor::check_assignability_to
- Virtual_tmp_table::check_assignability_from
- Virtual_tmp_table::sp_set_from_select_list
- Virtual_tmp_table::sp_save_in_vtable
- Virtual_tmp_table::sp_save_in_target_list
- LEX::check_ref_cursor_components
- LEX::make_sp_instr_copy_struct_for_last_context_variables
- LEX::declare_type_ref_cursor
- sp_cursor::Select_fetch_into_spvars::send_data_with_return_type
Adding new members:
- sp_instr_copen_by_ref::m_cursor_name
- Select_fetch_into_spvars::m_return_type
- Select_materialize::m_cursor_name
- Select_materialize::m_return_type
Adding new virtual methods:
- Item::resolve_spvar_cursor_rowtype
- Type_handler::Spvar_definition_resolve_type_refs
- Server_side_cursor::check_assignability_to
- Overriding Select_materialize::prepare to raise an error when the cursor
returned data type is not compatible with the RETURN clause
Making these methods virtual:
- Field::check_assignability_from
Adding new classes:
- sp_type_def_ref
- RowTypeBuffer
Adding new constructors to:
- Spvar_definition
Adding new helper methods (e.g. to reuse the code)
- Field::store_field_maybe_null
- ChanBuffer::append_ulonglong
- sp_pcontext::set_type_for_last_context_variables
Minor changes:
- Making TABLE::export_structure const
- Overriding Item_splocal::type_extra_attributes. It was forgotten in earlier changes.
Adding new error messages
- ER_CANNOT_CAST_ON_IDENT1_ASSIGNMENT_FOR_OPERATION
- ER_CANNOT_CAST_ON_IDENT2_ASSIGNMENT_FOR_OPERATION
The first key point of this commit is changing Type_extra_attributes in the way
that the pointer to a generic attribute is now "Type_generic_attributes*" -
a pointer to a new class with vitrual method type_handler().
Before this change the pointer to a generic attribute was
"void *m_attr_const_void_ptr", which was error prone, as allowed
to pass a pointer to a wrong structure without control.
Another key point is that the SET/ENUM data type related structures
are now attributed by the changed class Type_typelib_attributes,
which now looks as follows:
class Type_typelib_attributes: public Sql_alloc,
public Type_generic_attributes,
public TYPELIB
(with virtual method type_handler()), instead of being attributed by
TYPELIB directly.
Details:
- Using Column_definition's method typelib() where the member m_typelib
was used directly. The related lines change anyway, because the member
is renamed m_typelib_attr.
- Adding a new class Type_generic_attributes.
Deriving the following classes from it:
* Type_typelib_attributes
* sp_type_def
- sp_type_def does not derive from Type_handler_hybrid_field_type any more,
because the method type_handler() is now implemented virtually.
- Removing Type_extra_attributes::m_attr_const_void_ptr and
adding Type_extra_attributes::m_attr_const_generic_attributes_ptr instead -
a pointer to the new class Type_generic_attributes.
Renaming methods to set and read it according to the new data type name.
- Type_typelib_attributes now derives from TYPELIB instead of having
a pointer to TYPELIB.
- Adding a new class Type_typelib_ptr_attributes. It's a replacement for
the old implementation of Type_typelib_attributes.
Instead of deriving from Type_typelib_attributes, Field_enum now derives
from Type_typelib_ptr_attributes. The latter can store/read itself into/from
Type_extra_attributes. It's a bridge between the new Type_typelib_attributes
and Type_extra_attributes.
- Changing parameter data type in a few methods in Field_enum from
a pointer to TYPELIB to a pointer to Type_typelib_attributes.
- Removing typelib related methods from Type_extra_attributes.
Moving this functionality into Type_typelib_ptr_attributes and
Column_definition_attributes.
This turns Type_extra_attributes into a data structure independent
from any data type specific structures/methods.
- Adding methods:
Column_definition_attributes::typelib_attr() - Column_definition derives it
Column_definition_attributes::typelib() - Column_definition derives it
Type_typelib_ptr_attributes::typelib_attr() - Field_enum derives it
Type_typelib_ptr_attributes::typelib() - Field_enum derives it
- Renaming the member Create_field::save_interval into
Create_field::save_typelib_attr.
Changing its data type from a pointer to TYPELIB into a pointer
to Type_typelib_attributes pointer.
- Removing the method Type_typelib_attributes::store().
Adding Type_typelib_ptr_attributes::save_in_type_extra_attributes() instead.
The new method name makes the code more readable.
SEQUENCE is a pseudo-engine intended to produce sequential numbers.
This engine has HTON_HIDDEN flag, and optimizer costs for such engines
are zeroed and marked with MEM_UNDEFINED() macro.
However, some code paths still examine optimizer costs during
execution of queries involving SEQUENCE tables. For example,
deletion from a sequence table, despite being invalid, still evaluates
costs of access to the table before producing an error. This makes
MSAN trigger on the error "use-of-uninitialized-value".
Not to overcomplicate the logic for handling such scenarios,
this patch simply removes MEM_UNDEFINED() macro for engines with
HTON_HIDDEN flag.
The original symptom of this was an assertion 'marked_for_read()' failing in
RBR with unique blobs and binlog_row_image=MINIMAL. The problem was that the
hidden DB_ROW_HASH_1 virtual column was included in the after-image of the
update, but the underlying blob column was not being updated, so it was not
in the read_set/write_set.
It seems clearly wrong to include the DB_ROW_HASH_1 in the after-image when
the underlying blob isn't even being updated. The cause of this is the
following commit:
Author: Monty <monty@mariadb.org>
Date: Wed May 23 22:42:29 2018 +0300
MDEV-15243 Crash with virtual fields and row based binary logging
That patch removed a check for if the underlying fields of a virtual column
were being updated, and just added them unconditionally. This seems wrong.
So revert that part of the commit, restoring the logic to only add a virtual
column if any underlying field is actually in the write_set.
Also fix a typo in that commit where a code reformat accidentally reversed a
condition.
Also fix an assertion when InnoDB goes to update secondary indexes: If any
part of the primary key is being updated, then add all virtual columns that
are part of secondary indexes to the read_set.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
When evaluating a CHECK constraint during INSERT/UPDATE IGNORE, an evaluation error (like ER_DATA_OUT_OF_RANGE) can be set in the Diagnostics Area.
If the failure is ignored (downgraded to a warning), verify_constraints() was not clearing the error state from the DA before returning.
This led to a crash in my_ok() because it asserts that no error is set in the DA.
Fixed by calling clear_error() inside verify_constraints() when ignore_failure is true, ensuring the DA is ready for the subsequent OK packet.
Outdated table_list->derived_result after first trigger run. It was
allocated on statment mem_root and freed when trigger procedure
finishes.
Next INSERT trigger on new view in mysql_derived_prepare() jumps to
exit on error before it had a chance to allocate new derived_result.
It jumps here:
797 if ((res= sl->handle_derived(lex, DT_PREPARE)))
798 goto exit;
and the allocation is later:
837 if ((!derived->is_with_table_recursive_reference() ||
838 !derived->derived_result) &&
839 !(derived->derived_result= new (thd->mem_root) select_unit(thd)))
840 DBUG_RETURN(TRUE); // out of memory
on exit it does destruction of already freed object from previous statement:
901 exit:
...
921 if (res)
922 {
923 if (!derived->is_with_table_recursive_reference())
924 {
...
927 delete derived->derived_result;
The most obvious fix is to cleanup derived_result when the TABLE_LIST
is reinited for new use. Also as a precaution the patch does the
cleanup after delete operator.
let's simplify the code by removing copying of part_elem->connect_string
to table->s->connect_string. Move CONNECTION support from he server into
engines, engines define TOPTION("CONNECTION") if they want to support it,
it's stored in the option_struct and handled by the option_struct fix.
Problems:
* Mroonga used CONNECTION for something but there were no tests for it.
Removed Mroonga support for CONNECTION as requested by the maintainer
in MDEV-38530
* DROP/ALTER SERVER command used to close all tables using the server
in question, avoiding the need for FLUSH TABLES. This functionality
never worked for partitioned tables anyway and is now removed. Will be
done properly in MDEV-37827.
remove copying of part_elem->option_struct to table->s->option_struct.
It meant that most of the time a handler had no access to its options
(that's why some handlers resorted to a complex process of guessing the
correct partition) and couldn't work concurrenty. Instead
* introduce handler::option_struct which will point to
table->s->option_struct or part_elem->option_struct as appropriate,
the handler can use it any time as needed. A handler should not use
table->s->option_struct unless it really knows what it's doing and has
verified that it doesn't break partitioning
* rename option_struct in table->s to option_struct_table and in
part_elem to option_struct_part to emphasize it's semantics and
prevent incorrect usage.
* fix engines accordingly.
Lex::save_list contents remain from a previous query that has invalid
syntax and isn't reset when processing a new query.
We initialize this structure along with it's peer in LEX::start.
MDEV-38295 Recursive CTE usage leaks memory when not used in a top level select
During st_select_lex::cleanup() we assume that leaf tables and any
associated recursive table references can only be present when a join
structure is present. After MDEV-37220 this is no longer true.
We shift our cleanup routine out of the test for a join.
We add Multidelete_prelocking_strategy::handle_end() in the same manner
as Multiupdate_prelocking_strategy::handle_end() (but with minimal
processing as yet) to populate these leaf_tables in the outermost
select when it is a delete statement.
We add in a call to free_underlaid_joins() to the error processing section
of Sql_cmd_dml::execute to free up temporary tables that were allocated
in prepare, but not freed up when execute is bypassed because an error
occured during prepare.
Sergei Petrunia: Better comments for MDEV-38272, MDEV-38272
Approved by: Sergei Petrunia (sergey@mariadb.com)
CTE's are read only i.e It cannot have their columns updated in updates
set clause. Attempting to do so , should throw error
ERROR 1288 (HY000): The target table cte of the UPDATE is not updatable
with cte as (select * from t1 where c < 5)
update cte set cte.a =(select a from cte);
Here, conversion from a single table to a multi table update bypasses
the normal check by setting the TABLE_LIST object representing cte
to be merge derived. This is incorrect, so we prohibit this conversion
in TABLE_LIST::init_derived.
Approved by: Sanja Byelkin (sanja@mariadb.com)
