Remove the HA_NO_BLOBS restriction from the HEAP engine, allowing
the optimizer to keep temporary tables with BLOB/TEXT columns in
memory when they fit within max_heap_table_size / tmp_memory_table_size
limits. Additionally, advertise HA_CAN_GEOMETRY so explicit
CREATE TABLE ... ENGINE=MEMORY with GEOMETRY columns works.
Unlike other HEAP blob implementations (e.g. Percona), this patch
provides full HASH index support on blob columns, enabling efficient
lookups, GROUP BY, and DISTINCT operations directly in HEAP without
falling back to disk.
Architecture
------------
BLOB data is stored using continuation records -- additional fixed-size
records allocated from the same HP_BLOCK that holds regular rows. This
reuses existing allocation, free list, and size accounting with minimal
structural change, and avoids per-blob my_malloc() calls.
The existing single-byte visibility flag is extended into a flags byte
with bits for HP_ROW_HAS_CONT, HP_ROW_IS_CONT, HP_ROW_CONT_ZEROCOPY,
HP_ROW_SINGLE_REC, and HP_ROW_MULTIPLE_REC. Continuation records are
grouped into variable-length runs -- contiguous sequences within a leaf
block. Only the first record of each run carries a 10-byte header
(next_cont pointer + run_rec_count); inner records are pure payload.
Three storage formats, detected by flag bits via inline predicates:
Case A (HP_ROW_SINGLE_REC): single record, no header, data at
offset 0. Zero-copy read.
Case B (HP_ROW_CONT_ZEROCOPY): single run, multiple records.
Header in rec 0, data contiguous in rec 1..N-1. Zero-copy read
via chain + recbuffer.
Case C (HP_ROW_MULTIPLE_REC): one or more runs linked via
next_cont. Reassembly into blob_buff required.
Run allocation uses a two-phase strategy: (1) peek-then-unlink walk
of the free list detecting contiguous groups, (2) tail allocation
from HP_BLOCK for remaining data. A Step 3 scavenge fallback
walks the entire free list when tail allocation fails.
HP_SHARE::total_records tracks all physical records (primary +
continuation), while HP_SHARE::records remains the logical count
used by hash bucket mapping.
Reassembly buffer (HP_INFO::blob_buff) follows the same pattern as
InnoDB's blob_heap -- allocated once, grown via my_realloc, freed
on heap_reset()/close. Zero-copy cases (A/B) return pointers
directly into HP_BLOCK with no copy.
Full HASH index key handling for BLOB columns: hp_rec_hashnr(),
hp_rec_key_cmp(), hp_key_cmp(), hp_make_key(), hp_hashnr() are
extended for HA_BLOB_PART segments. Hash pre-check optimization
skips expensive blob materialization when hashes differ. PAD SPACE
collation semantics are preserved for blob key comparisons.
Field_blob_key (Monty) produces HEAP-native key format (4-byte length
+ 8-byte data pointer) directly, eliminating key buffer translation
between the SQL layer and HEAP engine.
SQL layer changes
-----------------
pick_engine() (new, extracted from choose_engine()): replaced the
blob_fields check with a reclength > HA_MAX_REC_LENGTH guard.
choose_engine() calls pick_engine() with the real reclength.
pick_engine() is also called early in finalize() with reclength=0
to predict whether the engine will be HEAP, enabling blob-aware
GROUP BY key setup that avoids unnecessary m_using_unique_constraint.
finalize(): HEAP+blob uses fixed-width rows; GROUP BY key setup sets
key_part_flag from field, uses item max_length for blob key sizing.
store_length initialized for all GROUP BY key parts. key_type uses
field->binary() to determine FIELDFLAG_BINARY vs text collation.
DISTINCT key setup skips null-bits helper for HEAP.
remove_duplicates(): blob check moved before HEAP check to fall
through to remove_dup_with_compare().
Aggregator_distinct::add(): overflow-to-disk conversion via
create_internal_tmp_table_from_heap() for non-dup write errors.
Expression cache disabled for HEAP+blob (key format incompatibility).
FULLTEXT early detection in mysql_derived_prepare(): forces disk
engine via TMP_TABLE_FORCE_MYISAM when outer query uses MATCH.
Deferred blob chain free (MDEV-39732): heap_delete() saves chain
pointers to pending_blob_chains, flushed on next mutation or
heap_reset()/close. Prevents dangling zero-copy pointers during
binlog_log_row().
REPLACE safety (MDEV-39825): HP_SHARE::write_can_replace flag
forces copy mode in hp_read_blobs(), preventing blob data corruption
from freed-then-reused continuation records during REPLACE.
Geometry GROUP_CONCAT fix (MDEV-39761): downgrade Field_geom to
Field_blob for GROUP_CONCAT temp tables in both expression creation
paths. Type_handler_geometry::type_handler_for_tmp_table() added.
Geometry GROUP BY key fix (MDEV-39871): detect when new_key_field()
produced non-blob Field_varstring for a blob column, replace with
Field_blob_key.
Performance
-----------
Non-blob tables: zero regression. Every blob-specific code path is
guarded by if (share->blob_count). No new allocations, no format
changes, no hash function changes for non-blob keys.
Blob tables: eliminates file creation/deletion overhead and page cache
management. For single-run blobs (common case), the read path is
entirely zero-copy.
Limitations
-----------
1. No BTREE indexes on blob columns (HASH only)
2. No partial-key prefix indexing for blobs
3. 2x memory for Case C reads only (A/B are 1x)
4. No blob compression
5. 65,535 records per run (uint16 cap, auto-split)
6. max_heap_table_size applies to continuation records
7. Expression cache disabled for HEAP+blob
8. FULLTEXT forces disk engine
Linked bugs fixed:
- MDEV-39703: mroonga fulltext test ordering
- MDEV-39723: ER_DUP_ENTRY on GROUP BY with blob column
- MDEV-39724: crash in hp_is_single_rec with GROUP BY
- MDEV-39732: slave crash in hp_free_run_chain on blob replication
- MDEV-39761: Field_geom::store() assertion in GROUP_CONCAT
- MDEV-39782: RBR ER_KEY_NOT_FOUND on HEAP blob UPDATE
- MDEV-39825: blob data corruption on REPLACE into HEAP table
- MDEV-39871: crash in my_hash_sort_bin on GROUP BY with geometry
Reviewed by: Michael Widenius <monty@mariadb.org>
Monty reviewed the entire patch. Areas where he suggested changes
or contributed code:
- Field_blob_key class (HEAP-native blob key format, 4-byte length +
data pointer)
- Duplicate key fix on HEAP-to-Aria conversion
- hp_blob_key_length() uint32 fix
- hp_rec_hashnr_stored removal
- type_handler_for_tmp_table() param cleanup
- Type_handler_geometry::type_handler_for_tmp_table() virtual
- blob pointer bzero()
- find_unique_row() double-materialization fix
- Tail reclaim review
- Batch tail allocation review
- hp_update.c cleanup
- Field_blob_compressed temp table fix
- row_pack_length() dedup
- pack_length_no_ptr() removal
- Race condition fix in HEAP
- MDEV-39703 mroonga test fix
- MDEV-39825 write_can_replace optimization
- is_text_key_segment removal (field->binary() simplification)
- Documentation (Docs/internal-temporary-tables.txt)
Contribution by: Alexander Barkov <bar@mariadb.com>
Type_handler::make_and_init_table_field_ex() -- refactored temp table
field creation from inline code in sql_select.cc into type handler
virtual methods (sql_type.cc, sql_type_geom.cc), enabling clean
per-type-handler field creation for HEAP blob promotion.
Applying HEX encoding write writting an ENUM/SET TYPELIB to FRM
if the TYPELIB has 0x00 bytes in the value.
This HEX encoding was earlier used only to write UCS2/UTF16/UTF32 TYPELIBs.
A new flag FIELDFLAG_FRM_HEX_ENCODED_TYPELIB was added to indicate
that the TYPELIB is hex encoded. It's used only inside FRM.
Note, it's mangled with FIELDFLAG_TREAT_BIT_AS_CHAR.
This should not be harmful:
- BIT and ENUM/SET columns are handled by two separate code branches
when opening an FRM
- The flag is unset immediately after decoding TYPELIB, so the rest
of the code does not se an unexpected flag combination.
A 0xHHHH / b'..' literal keeps its hex_hybrid type through COALESCE/IF/CASE/...,
but was evaluated, sized and stored as a string, so it behaved differently from
a bare literal (10+COALESCE(0x31) gave 11, not 59).
Type_handler_hex_hybrid now implements the numeric side to match the bare
literal: val_int/val_real/val_decimal read bytes as a hybrid, Item_decimal_precision
returns the integer precision, Item_save_in_field stores like the bare literal.
The traditional type merge keeps the hybrid through a typeless NULL (as for BIT),
and Item_hybrid_func_fix_attributes attributes it binary/unsigned.
Problem:
SELECT on a table with a TIMESTAMP column and a virtual column
expression containing a field reference crashes with assertion
marked_for_read() in Field_timestamp0::get_timestamp.
Cause:
convert_item_for_comparison() physically evaluates the subject item
via Datetime dt(thd, subject, ...) to attempt a TIMESTAMP vs DATETIME
optimization. During vcol_fix_expr() at open_table() time,
used_tables_cache is not yet propagated up the item tree, causing
const_item() to return true incorrectly for a non-constant expression
containing a field reference. The evaluation then reads Field_timestamp0
before read bitmaps are initialized, violating marked_for_read().
Fix:
Add find_field_processor() to the Item base class, overridden in
Item_field to return true. Guard convert_item_for_comparison() with
a walk() check to skip physical evaluation if the subject subtree
contains any field reference.
Field::get_date and Item::get_date can return true and not
have their ltime pointer structure initialized.
This was the case with Field_vers_trx_id::get_date.
Type_handler_time_common::stored_field_cmp_to_item rather
than doing a my_time_compare on uninitialized values can
return a non-equality (!=0) value for when the fetching of
Field or Item fails and avoid the call to my_time_compare.
Added checks for other invalid results before being
passed to my_time_compare.
Add nonnull as an attribute on my_time_compare and the
unpack_time function it used. Also pack_time since related
and it obviously requires a non-null argument.
Reviewed by: Alexey Midenkov
This patch to mysqldump is only merged partially:
commit 666ff1db36
Author: Monty <monty@mariadb.org>
Date: Fri May 15 16:39:35 2026 +0300
MDEV-39519 Change mariadb-dump to use SHOW REPLICA STATUS instead of SHOW SLAVE STATUS
The functionality for --dump-slave cannot be merged as the underlying
functionality was removed in this commit:
commit 21c940d599
Author: ParadoxV5 <paradox.ver5@gmail.com>
Date: Thu Dec 4 21:22:20 2025 -0700
MDEV-37146 mariadb-dump: Replace SHOW SLAVE STATUS usage with information_schema.SLAVE_STATUS
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
When setup_copy_fields() needs to copy a non-aggregate function value,
it doesn't construct an Item_copy directly. Instead, it calls
Type_handler::create_item_copy, which is a kind of factory. The base
Type_handler::create_item_copy returns Item_copy_string. Some type
handlers override it, like timestamp and fixed binary. However, the
numeric type handlers (e.g., float, double, int, decimal) did not, so
they fell through to that base and got Item_copy_string.
A SELECT that aggregates will copy each non aggregate function value
into a temporary table through an Item_copy object, whose concrete
type is chosen by the create_item_copy method on the value's type
handler. For numeric types that method returned Item_copy_string,
which stores the value as text. A FLOAT keeps only FLT_DIG
significant digits as text, too few to reproduce its 24 bit mantissa,
so the copied value differed from the original. With one row per
group, CAST(c1 AS FLOAT) - MIN(CAST(c1 AS FLOAT)) returned a large
number instead of zero.
Add Item_copy_real with Item_copy_float and Item_copy_double variants
that keep the value as a double, the same way Item_cache_real does, and
let the float and double type handlers create them. This mirrors the
existing copy items for timestamp and fixed binary types.
JSON is utf8mb4. Change INFORMATION_SCHEMA.OPTIMIZER_TRACE.TRACE from
using utf8mb3 to mb4.
In order to do that:
- Add m_charset into Show::Type (NULL means "use the default")
- Make Type_handler_varchar::make_schema_field() check that field
and use the specified charset/collation.
- Add a constructor to Show::Longtext that allows to specify the charset
- Make optimizer_trace use my_charset_utf8mb4_general_ci.
- In fill_optimizer_trace_info(), indicate that the trace we've
collected is utf8mb4
Problem:
When a stored function returning `ROW` type is used as a `ROW` field
and comapred against a non-ROW field, server crashes trying to
aggregate types for comaprison in-
`Type_collection_row::aggregate_for_comparison`.
for example:
CREATE FUNCTION f() RETURNS ROW TYPE OF t RETURN 1;
SELECT ROW(f(),1)=ROW(1,1) AS eq;
Fix:
Convert assertions in `Type_collection_row::aggregate_for_comparison`
into a condition and return `NULL` if an invalid type combination is
detected.
Adding support for the strict cursor data types:
Example 1a:
TYPE rec0_t IS RECORD (a INT, VARCHAR(10));
TYPE cur0_t IS REF CURSOR RETURN rec0_t;
Example 1b:
TYPE rec0_t IS RECORD (a t1.a%TYPE, b t1.b%TYPE);
TYPE cur0_t IS REF CURSOR RETURN rec0_t;
Example 1c:
TYPE rec0_t IS RECORD (a INT, VARCHAR(10));
r0 rec0_t;
TYPE cur0_t IS REF CURSOR RETURN r0%TYPE;
Example 1d:
TYPE rec0_t IS RECORD (a t1.a%TYPE, b t1.b%TYPE);
r0 rec0_t;
TYPE cur0_t IS REF CURSOR RETURN r0%TYPE;
Example2a:
TYPE cur0_t IS REF CURSOR RETURN t1%ROWTYPE; -- t1 is a table
Example 2b:
r0 t1%ROWTYPE;
TYPE cur0_t IS REF CURSOR RETURN r0%TYPE;
Example3a:
CURSOR cursor_sample IS SELECT a,b FROM t1;
TYPE cur0_t IS REF CURSOR RETURN cursor_sample%ROWTYPE;
Example3b:
CURSOR cursor_sample IS SELECT a,b FROM t1;
r0 cursor_sample%ROWTYPE;
TYPE cur0_t IS REF CURSOR RETURN r0%TYPE;
If a cursor variable is declared with a RETURN clause then:
1. At OPEN type the data type of the SELECT list row is compared
for compatibility with the cursor RETURN data type.
The SELECT list row must be assignable to the RETURN type row.
If case if assignability is not meet, an error is raised
Assignability means:
- The arity of the SELECT list must be equal to the arity
of the RETURN clause
- Every n-th field of the SELECT list must be assignable to the
n-th field of the RETURN Clause
2. At FETCH time, the data is fetched in two steps:
a. On the first step the data is fetched into a virtual table
with the row type described in the RETURN clause
b. On the second step the data is copied from the virtual table
to the target fetch list. Data type conversion can happen
on this step.
Change details:
Adding new methods:
- sp_cursor::check_assignability_to
- Virtual_tmp_table::check_assignability_from
- Virtual_tmp_table::sp_set_from_select_list
- Virtual_tmp_table::sp_save_in_vtable
- Virtual_tmp_table::sp_save_in_target_list
- LEX::check_ref_cursor_components
- LEX::make_sp_instr_copy_struct_for_last_context_variables
- LEX::declare_type_ref_cursor
- sp_cursor::Select_fetch_into_spvars::send_data_with_return_type
Adding new members:
- sp_instr_copen_by_ref::m_cursor_name
- Select_fetch_into_spvars::m_return_type
- Select_materialize::m_cursor_name
- Select_materialize::m_return_type
Adding new virtual methods:
- Item::resolve_spvar_cursor_rowtype
- Type_handler::Spvar_definition_resolve_type_refs
- Server_side_cursor::check_assignability_to
- Overriding Select_materialize::prepare to raise an error when the cursor
returned data type is not compatible with the RETURN clause
Making these methods virtual:
- Field::check_assignability_from
Adding new classes:
- sp_type_def_ref
- RowTypeBuffer
Adding new constructors to:
- Spvar_definition
Adding new helper methods (e.g. to reuse the code)
- Field::store_field_maybe_null
- ChanBuffer::append_ulonglong
- sp_pcontext::set_type_for_last_context_variables
Minor changes:
- Making TABLE::export_structure const
- Overriding Item_splocal::type_extra_attributes. It was forgotten in earlier changes.
Adding new error messages
- ER_CANNOT_CAST_ON_IDENT1_ASSIGNMENT_FOR_OPERATION
- ER_CANNOT_CAST_ON_IDENT2_ASSIGNMENT_FOR_OPERATION
Some data types:
- RECORD
- assoc arrays
- SYS_REFCURSOR
erroneously accepted non-relevant attributes:
- length
- scale
- character set / collation
- REF_SYSTEM_ID
Fixing to raise an error.
Details:
- Overriding get_column_attributes() in Type_handler_row,
Type_handler_sys_refcursor, Type_handler_assoc_array.
- Moving a piece of the old code into a new method
Type_handler::check_data_type_attributes(), to reuse it
for testing attributes for both plugin types and TYPE-def types.
- Reusing check_data_type_attributes() in methods
* LEX::set_field_type_udt_or_typedef()
* LEX::set_field_type_udt()
- Adding opt_binary into the rule field_type_all_with_typedefs,
into the grammar branch for the plugin types, as XMLTYPE needs it.
The method Type_handler::Column_definition_fix_attributes() was an
addition for Type_handler::Column_definition_set_attributes().
The code in virtual implementations of Column_definition_fix_attributes()
has now been joined to corresponding virtual implementations of
Column_definition_set_attributes().
After this change everything is done in Column_definition_set_attributes().
This makes the code clearer. Previous implementation was rather confusing.
For example, for temporal data types:
(1) After Column_definition_set_attributes():
- Column_definition::length meant the fractional precision
- Column_definition::decimals was set to 0
(2) After Column_definition_fix_attributes():
- Column_definition::length means the full character length
- Column_definition::decimals means the fractional precision
Now everything gets set to (2) right in Column_definition_set_attributes().
The first key point of this commit is changing Type_extra_attributes in the way
that the pointer to a generic attribute is now "Type_generic_attributes*" -
a pointer to a new class with vitrual method type_handler().
Before this change the pointer to a generic attribute was
"void *m_attr_const_void_ptr", which was error prone, as allowed
to pass a pointer to a wrong structure without control.
Another key point is that the SET/ENUM data type related structures
are now attributed by the changed class Type_typelib_attributes,
which now looks as follows:
class Type_typelib_attributes: public Sql_alloc,
public Type_generic_attributes,
public TYPELIB
(with virtual method type_handler()), instead of being attributed by
TYPELIB directly.
Details:
- Using Column_definition's method typelib() where the member m_typelib
was used directly. The related lines change anyway, because the member
is renamed m_typelib_attr.
- Adding a new class Type_generic_attributes.
Deriving the following classes from it:
* Type_typelib_attributes
* sp_type_def
- sp_type_def does not derive from Type_handler_hybrid_field_type any more,
because the method type_handler() is now implemented virtually.
- Removing Type_extra_attributes::m_attr_const_void_ptr and
adding Type_extra_attributes::m_attr_const_generic_attributes_ptr instead -
a pointer to the new class Type_generic_attributes.
Renaming methods to set and read it according to the new data type name.
- Type_typelib_attributes now derives from TYPELIB instead of having
a pointer to TYPELIB.
- Adding a new class Type_typelib_ptr_attributes. It's a replacement for
the old implementation of Type_typelib_attributes.
Instead of deriving from Type_typelib_attributes, Field_enum now derives
from Type_typelib_ptr_attributes. The latter can store/read itself into/from
Type_extra_attributes. It's a bridge between the new Type_typelib_attributes
and Type_extra_attributes.
- Changing parameter data type in a few methods in Field_enum from
a pointer to TYPELIB to a pointer to Type_typelib_attributes.
- Removing typelib related methods from Type_extra_attributes.
Moving this functionality into Type_typelib_ptr_attributes and
Column_definition_attributes.
This turns Type_extra_attributes into a data structure independent
from any data type specific structures/methods.
- Adding methods:
Column_definition_attributes::typelib_attr() - Column_definition derives it
Column_definition_attributes::typelib() - Column_definition derives it
Type_typelib_ptr_attributes::typelib_attr() - Field_enum derives it
Type_typelib_ptr_attributes::typelib() - Field_enum derives it
- Renaming the member Create_field::save_interval into
Create_field::save_typelib_attr.
Changing its data type from a pointer to TYPELIB into a pointer
to Type_typelib_attributes pointer.
- Removing the method Type_typelib_attributes::store().
Adding Type_typelib_ptr_attributes::save_in_type_extra_attributes() instead.
The new method name makes the code more readable.
Year::year_precision(): Make static, so that VYear::VYear
and VYear_op::VYear_op() will avoid invoking this function with
an unnecessary "this" pointer to an uninitialized object.
removed:
* DES encryption
* --secure-auth
* --old
* spider table options: bfz, btt, cmd, ctp, cwg, isa, ilm, ios, smd, stc, stl
extended under old-mode:
* YEAR(2), still available when old-mode=2_DIGIT_YEAR
un-deprecated:
* keep_files_on_create, originally (MDEV-23570) the idea was to make it
TRUE and deprecate. It cannot be removed when FALSE, but TRUE breaks
mariabackup.aria_backup where a table is altered from Aria to InnoDB
during a backup, so both t.MAD/t.MAI and t.ibd gets into a backup.
Fixing the return type of Type_handler::Item_save_in_value() from
bool to void.
Adding a new method st_value::is_null().
This makes the code less confusing.
Item_singerow_subselect::fix_length_and_dec() incorrectly calculated
its Item::max_length when the underlying expression was Item_int.
The reason of the problem:
Item_int has an optimized max_length to make CONCAT(1) create a VARCHAR(1)
column rather than a VARCHAR(2) column. Its max_length does not include one
extra character for the sign in case the value is positive but the value
is not marked as Item::unsigned==true.
So copying max_length from the underlying Item_int (with value==9)
in cases like this:
SELECT CONCAT((SELECT 9 FROM t0));
was not correct.
Implementing a new virtual method
Type_handler::Item_type_std_attributes_generic(const Item *item)
- The default implementation just copies attributes from "item" as is.
- In case of Type_handler_int_result it evaluates max_length
using item->decimal_precision() rather than item->max_length.
This works correctly for both "optimized" items like Item_int and
non-"optimized" Items whose max_length includes +1 for the sign
in case of signed expressions.
all params are initialized to use Type_handler_null.
NULL params do not have a value to read from the packet buffer,
so their Item_param_set_param_func() should do nothing.
Type_handler_null inherits from Type_handler_string_result,
that is, Type_handler_null *must* override parent's
Item_param_set_param_func().
In commit fa7d4abf16
when the 16-bit type alias decimal_digits_t was introduced,
some incompatible use of the 32-bit uint data type was overlooked.
Only for converting Item_result_field::max_length (uint), we will use
explicit conversion.
Fixing a confusing error message:
Unknown column 'assoc_array_var' in 'unknown_method'
to a clearer:
FUNCTION/PROCEDURE assoc_array_var.unknown_method does not exist
