SET sql_mode=ORACLE;
DELIMITER $$
CREATE OR REPLACE PACKAGE pkg AS
-- Declare a package public data type
TYPE varchar_array IS TABLE OF VARCHAR(2000) INDEX BY INTEGER;
END;
$$
DELIMITER ;
DELIMITER $$
CREATE OR REPLACE PROCEDURE p1 AS
v pkg.varchar_array; -- Use the package public data type
BEGIN
v(0):='test';
SELECT v(0);
END;
$$
DELIMITER ;
Note, the change is done only for sql_mode=ORACLE, because the TYPE
declaration is not available for the default mode.
Where package-wide types are available
--------------------------------------
- Variabe list type:
DECLARE var pkg1.type1;
- RETURN type for a package routine:
CREATE FUNCTION .. RETURN pkg1.type1 ...
- Parameter type for a package routine:
PROCEDURE p1(param1 pkg1.type1);
- Assoc array element type:
TYPE assoc1_t IS TABLE OF pkg1.type1 ...
- REF CURSOR RETURN type:
TYPE cur1_t IS REF CURSOR RETURN pkg1.type1;
Change details
--------------
- Adding a member Lex_length_and_dec_st::m_foreign_module_type
It's set to true when the data type was initialized from a TYPE
in foreign routine (e.g. in PACKAGE spec).
It's needed to prevent use of qualified identifiers in public contexts,
i.e. in schema public routine parameter types and schema publuc function
RETURN types.
Adding a helper method sp_head::check_applicability() which prevents
use of qualified types in public context.
- Adding a helper method sp_head::raise_unknown_data_type().
- Adding methods LEX::set_field_type_typedef_package_spec() for
2-step and 3-step qualified indentifiers.
It's used in field_type_all_with_typedefs which covers cases:
- Variabe list type : DECLARE var pkg1.type1;
- RETURN type : CREATE FUNCTION .. RETURN pkg1.type1 ...
- Parameter type : PROCEDURE p1(param1 pkg1.type1);
- Assoc array element type : TYPE assoc1_t IS TABLE OF pkg1.type1 ...
- Adding a method LEX::declare_type_ref_cursor_return_typedef().
It handles cases when a new TYPE REF CURSOR RETURN is declared,
for both for qualified RETURN types and non-qualified RETURN types:
- TYPE cur0_t IS REF CURSOR RETURN rec1_t;
- TYPE cur0_t IS REF CURSOR RETURN pkg1.rec1_t;
- TYPE cur0_t IS REF CURSOR RETURN db1.pkg1.rec1_t;
The code was moved from LEX::declare_type_ref_cursor() into
LEX::declare_type_ref_cursor_return_typedef() and extended
to cover qualified RETURN types.
- Adding a method Sql_path::find_package_spec_type().
It iterates through all schemas specified in @@path and searches
for the given type in the given package.
- Adding a helper method sp_pcontext::type_defs_add_ref_cursor()
to reuse the code.
- Adding a new method sp_package::get_typedef() to search
for TYPE definitions in PACKAGE specifications.
- Adding a new method sp_head::get_typedef_package_spec()
to search for TYPE definitions used by a PROCEDURE or FUNCTION.
- Adding a helper method
Sp_handler::sp_cache_routine_reentrant_suppress_errors
Adding a method Sp_handler::find_package_spec().
This task loads the stats of the tables that are used in a query,
from the trace into the optimizer. This feature is controlled
by optimizer_replay_context, and points to an user defined variable name,
that has the context information.
The stats such as num_of_records present in the table, indexes if present,
their names, along with the average number of records_per_key
with in each index, cost of reading an index are loaded from the trace.
Additionally, stats from range analysis i.e. ranges, and the
corresponding number of records, along with their costs are also
loaded from the trace.
The trace context which is in JSON format is firstly set into the
user defined session variable, and this variable name is set to
optimizer_replay_context system variable.
Later, when a user issues a query, the contents of the json context is
parsed and an in memory representation is built using the class
Optimizer_context_replay. This class is then used by the optimizer
to update and save original values of the tables, indexes, and range stats
in the methods "set_statistics_for_table()" of sql_statistics.cc, and
"check_quick_select()" of opt_range.cc.
After the query gets finished, the statistics that were updated
in the optimizer are restored back to the saved original values.
The entry point for parsing the json structure is in
"mysql_execute_command()" of sql_parse.cc, and similarly exit point
i.e. to restore the saved stats of the optimizer is at the end
of the same method.
Two new warning/error messages are introduced:
ER_JSON_OPTIMIZER_REPLAY_CONTEXT_PARSE_FAILED, and
ER_JSON_OPTIMIZER_REPLAY_CONTEXT_MATCH_FAILED
Allow auto partitioning by interval in PARTITION BY RANGE COLUMNS
PARTITION BY RANGE COLUMNS (col_name)
INTERVAL interval [AUTO]
(
PARTITION partition_name VALUES LESS THAN (value)
[, PARTITION partition_name VALUES LESS THAN (value) ... ]
)
where
- col_name is the name of one column of type DATE or DATETIME or
TIMESTAMP
- at least one partition is supplied, and the highest partition cannot
have MAXVALUE range
- INTERVAL interval is a positive time interval. it can be mariadb
format or oracle NUMTODSINTERVAL/NUMTOYMINTERVAL format. Like
versioning, the smallest unit is second, i.e. no subsecond like
microsecond.
- DATE column cannot have interval with values less than a day
- Subpartitions are allowed, but restricted to existing subpartition
types, i.e. [LINEAR] (KEY|HASH)
When performing one of the following DML statements on such a table,
it will first add partitions by the specified interval until the
partition covers the current time:
- INSERT
- INSERT ... SELECT
- LOAD
- UPDATE
- REPLACE
- REPLACE ... SELECT
Partition addition will not cause an implicit commit like DDL normally
does.
The partitions are named pN.
Otherwise the table behaves exactly the same as a normal RANGE COLUMNS
partitioned table.
Note that TIMESTAMP is not allowed as a type for PARTITION BY RANGE
COLUMNS otherwise.
Including the following fixes:
1.
MDEV-40088 Disallow subqueries in INTERVAL clause in range interval auto partitioning
This is consistent with system time (versioning) partitioning. Also
consistent is that both allow expressions otherwise.
2.
MDEV-40048 Allow trigger and LOCK TABLES to work with range interval auto partitioning
When a range interval auto partitioned table is the target of a
trigger, the triggering statement is not necessarily one that would
cause the auto-creation of new partitions, so we need to account for
that.
Also added support for LOCK TABLES ... WRITE.
Improved tests coverage by adapting tests from versioning.partition.
3.
MDEV-40467 Improve Y2038/2106 problem error message
When nearing the end times TIMESTAMP_MAX_VALUE, auto-partition
creation may encounter overflow when calculating new partition range.
This patch improves the message of such an overflow
This patch to mysqldump is only merged partially:
commit 666ff1db36
Author: Monty <monty@mariadb.org>
Date: Fri May 15 16:39:35 2026 +0300
MDEV-39519 Change mariadb-dump to use SHOW REPLICA STATUS instead of SHOW SLAVE STATUS
The functionality for --dump-slave cannot be merged as the underlying
functionality was removed in this commit:
commit 21c940d599
Author: ParadoxV5 <paradox.ver5@gmail.com>
Date: Thu Dec 4 21:22:20 2025 -0700
MDEV-37146 mariadb-dump: Replace SHOW SLAVE STATUS usage with information_schema.SLAVE_STATUS
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
If the first Partial_rows_log_event in a group provides an
original_event_size that is smaller than the actual size of the
underlying Rows_log_event, the slave will not allocate enough memory
to hold the actual Rows_log_event content, but still try to copy
that much memory, resulting in the slave crashing.
This patch adds validation checking for this condition, and will error
with code ER_PARTIAL_ROWS_LOG_EVENT_BAD_STREAM when detected.
Additionally:
* A new validation is added to ensure that the slave supports
re-constructing the size of the original event on the master (i.e. if
repliating from a 64-bit to a 32-bit system). A new test is added to
show the error message is correct. Note that the error code used here is
ER_OUTOFMEMORY, and that isn't exactly correct; but the actual error
isn't practical enough to warrant its own new error code.
* The error message for ER_PARTIAL_ROWS_LOG_EVENT_BAD_STREAM
is generalized to better support various types of broken streams. The
previous message was hard-coded to be about detecting out-of-sequence
fragments. The new message is updated to allow general "found %s" and
"expected %s" conditions.
Signed-off-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
When ALTER TABLE uses the copy algorithm, ha_create_table() is
invoked to build the shadow table. A failure there was previously
reported as "Can't create table", which is misleading because the
user never issued a CREATE.
Fix (per review by @midenok): keep errno 1005 / SQLSTATE HY000
unchanged and only swap the message text when HA_CREATE_TMP_ALTER
is set, via a new translatable template ER_CANT_CREATE_TMP_ALTER_TABLE
in errmsg-utf8.txt (no SQLSTATE clause -- it is a message template
rather than a wire error code).
The user-visible error becomes:
ERROR 1005 (HY000): Can't create temporary table for ALTER TABLE \
`db`.`tbl` (errno: N "...")
No existing error handler, stored procedure, tool, or application
that matches on errno 1005 or SQLSTATE HY000 is affected.
Notes on the 12.3 rebase:
* In 12.3 the ER_CANT_CREATE_TABLE reporting logic has been moved
from ha_create_table() into its new helper ha_create_table_from_share().
The HA_CREATE_TMP_ALTER branch is added in the helper so that both
the main shadow table and any high-level (vector) index sub-tables
go through the same path. Vector index sub-tables use a zero-
initialised index_cinfo, so HA_CREATE_TMP_ALTER is never set for
them and they still get the plain "Can't create table" message.
Review follow-ups (Round 6, @midenok):
* Moved the test content from suite/innodb/t/alter_table_error_message.test
into main/alter_table.test (no duplicate --include directives);
dropped the extra .test/.result files accordingly.
* Added the standard "End of 12.3 tests" section marker at the end.
* Removed the unnecessary DROP TABLE IF EXISTS cluster_t; in the
cleanup step (the test is deterministic, the table no longer
exists at that point).
Review follow-ups (Round 7, test re-record + errmsg formatting):
* Fixed ER_CANT_CREATE_TMP_ALTER_TABLE template to use the exact
ER_CANT_CREATE_TABLE format specifiers, so error frames produced
by my_error() are byte-identical to the existing error (only the
fixed prefix changes).
* Re-recorded innodb.innodb-index.result and innodb.foreign_key.result
which now observe the new "Can't create temporary table for ALTER
TABLE" message in ALTER TABLE error paths that used to print
"Can't create table".
* No code changes outside of the intentional message swap; all other
.result files remain untouched.
Review follow-ups (Round 8, additional test re-record from CI):
* Re-recorded innodb.table_flags.result for the MDEV-38079 scenario
(ALTER TABLE t1 PAGE_COMPRESSED=1, ALGORITHM=COPY under
INNODB_DEFAULT_ROW_FORMAT=REDUNDANT) -- this ALTER path now reports
the new "Can't create temporary table for ALTER TABLE" message.
Verified on all 10 page size / checksum combinations (4k/8k/16k/
32k/64k x strict_crc32/strict_full_crc32).
* Re-recorded parts.engine_defined_part_attributes.result for the
ALTER TABLE `t4` PARTITION BY RANGE(...) ENCRYPTED path. The four
remaining "Can't create table" occurrences in the same file refer
to pure CREATE TABLE failures and correctly retain the original
message.
* No code or test-case logic changes; only .result re-recordings.
Signed-off-by: a18792721831 <735809899@qq.com>
CLA: a18792721831 <735809899@qq.com>
Extended QB_NAME hint to support path-based addressing of query blocks
nested within views, derived tables, and CTEs, following TiDB's syntax.
New syntax:
QB_NAME(name, query_block_path), where
query_block_path ::= query_block_path_element
[ {. query_block_path_element }... ]
query_block_path_element ::= @ qb_path_element_select_num |
qb_path_element_view_sel
qb_path_element_view_sel ::= qb_path_element_view_name
[ @ qb_path_element_select_num ]
For example,
`SELECT /*+ qb_name(qb_v1, v1) */* FROM v1`
The name `qb_v1` is assigned to the inner query block of the view `v1`.
`SELECT /*+ qb_name(qb_v1, v1@sel_1) */* FROM v1`
Means the same but specifies that `v1` is present in SELECT#1 of the current
query block.
`SELECT /*+ qb_name(qb_v1, v1@sel_1 .@sel_2) */* FROM v1`
This means SELECT#2 of view `v1`, which is present in SELECT#1 of
the current query block, gets the name `qb_v1`.
It is possible to specify not only view names but also derived tables
and CTE's in the path.
Views and derived tables may be nested on multiple levels, for example:
`SELECT /*+ qb_name(dt2_dt1_v1_1, dt1 .dt2 .v2 .@SEL_2)
no_index(t1@dt2_dt1_v1_1)*/ v1.*
FROM v1 JOIN (SELECT v1.* FROM v1 JOIN (SELECT * FROM v2) dt2) dt1`
Limitations:
- Only SELECT statements support QB names with path. DML operations
(UPDATE, DELETE, INSERT) only support explicitly defined QB names
This patch implements support for implicit query block (QB) names in
optimizer hints, allowing hints to reference query blocks and tables
within derived tables, views and CTEs without requiring explicit
QB_NAME hints.
Examples.
-- Addressing a table inside a derived table using implicit QB name
select /*+ no_index(t1@dt) */ *
from (select * from t1 where a > 10) as DT;
-- this is an equivalent to:
select /*+ no_index(t1@dt) */ * from
(select /*+ qb_name(dt)*/ * from t1 where a > 10) as DT;
-- Addressing a query block corresponding to the derived table
select /*+ no_bnl(@dt) */ *
from (select * from t1, t2 where t.1.a > t2.a) as DT;
-- View
create view v1 as select * from t1 where a > 10 and b > 100;
-- referencing a table inside a view by implicit QB name:
select /*+ index_merge(t1@v1 idx_a, idx_b) */ *
from v1, t2 where v1.a = t2.a;
-- equivalent to:
create view v1 as select /*+ qb_name(qb_v1) */ *
from t1 where a > 10 and b > 100;
select /*+ index_merge(t1@qb_v1 idx_a, idx_b) */ *
from v1, t2 where v1.a = t2.a;
-- CTE
with aless100 as (select a from t1 where b <100)
select /*+ index(t1@aless100) */ * from aless100;
-- equivalent to:
with aless100 as (select /*+ qb_name(aless100) */ a from t1 where b <100)
select /*+ index(t1@aless100) */ * from aless100;
Limitations:
- Only SELECT statements support implicit QB names. DML operations
(UPDATE, DELETE, INSERT) only support explicit QB names
Similar to ER_LOCK_DEADLOCK and ER_LOCK_WAIT_TIMEOUT, let the ER_CHECKREAD
message suggest that the transaction needs to be restarted. When this error
is reported by InnoDB (only if innodb_snapshot_isolation=ON),
the transaction will be rolled back.
Reviewed by: Sergei Golubchik
Adding support for the strict cursor data types:
Example 1a:
TYPE rec0_t IS RECORD (a INT, VARCHAR(10));
TYPE cur0_t IS REF CURSOR RETURN rec0_t;
Example 1b:
TYPE rec0_t IS RECORD (a t1.a%TYPE, b t1.b%TYPE);
TYPE cur0_t IS REF CURSOR RETURN rec0_t;
Example 1c:
TYPE rec0_t IS RECORD (a INT, VARCHAR(10));
r0 rec0_t;
TYPE cur0_t IS REF CURSOR RETURN r0%TYPE;
Example 1d:
TYPE rec0_t IS RECORD (a t1.a%TYPE, b t1.b%TYPE);
r0 rec0_t;
TYPE cur0_t IS REF CURSOR RETURN r0%TYPE;
Example2a:
TYPE cur0_t IS REF CURSOR RETURN t1%ROWTYPE; -- t1 is a table
Example 2b:
r0 t1%ROWTYPE;
TYPE cur0_t IS REF CURSOR RETURN r0%TYPE;
Example3a:
CURSOR cursor_sample IS SELECT a,b FROM t1;
TYPE cur0_t IS REF CURSOR RETURN cursor_sample%ROWTYPE;
Example3b:
CURSOR cursor_sample IS SELECT a,b FROM t1;
r0 cursor_sample%ROWTYPE;
TYPE cur0_t IS REF CURSOR RETURN r0%TYPE;
If a cursor variable is declared with a RETURN clause then:
1. At OPEN type the data type of the SELECT list row is compared
for compatibility with the cursor RETURN data type.
The SELECT list row must be assignable to the RETURN type row.
If case if assignability is not meet, an error is raised
Assignability means:
- The arity of the SELECT list must be equal to the arity
of the RETURN clause
- Every n-th field of the SELECT list must be assignable to the
n-th field of the RETURN Clause
2. At FETCH time, the data is fetched in two steps:
a. On the first step the data is fetched into a virtual table
with the row type described in the RETURN clause
b. On the second step the data is copied from the virtual table
to the target fetch list. Data type conversion can happen
on this step.
Change details:
Adding new methods:
- sp_cursor::check_assignability_to
- Virtual_tmp_table::check_assignability_from
- Virtual_tmp_table::sp_set_from_select_list
- Virtual_tmp_table::sp_save_in_vtable
- Virtual_tmp_table::sp_save_in_target_list
- LEX::check_ref_cursor_components
- LEX::make_sp_instr_copy_struct_for_last_context_variables
- LEX::declare_type_ref_cursor
- sp_cursor::Select_fetch_into_spvars::send_data_with_return_type
Adding new members:
- sp_instr_copen_by_ref::m_cursor_name
- Select_fetch_into_spvars::m_return_type
- Select_materialize::m_cursor_name
- Select_materialize::m_return_type
Adding new virtual methods:
- Item::resolve_spvar_cursor_rowtype
- Type_handler::Spvar_definition_resolve_type_refs
- Server_side_cursor::check_assignability_to
- Overriding Select_materialize::prepare to raise an error when the cursor
returned data type is not compatible with the RETURN clause
Making these methods virtual:
- Field::check_assignability_from
Adding new classes:
- sp_type_def_ref
- RowTypeBuffer
Adding new constructors to:
- Spvar_definition
Adding new helper methods (e.g. to reuse the code)
- Field::store_field_maybe_null
- ChanBuffer::append_ulonglong
- sp_pcontext::set_type_for_last_context_variables
Minor changes:
- Making TABLE::export_structure const
- Overriding Item_splocal::type_extra_attributes. It was forgotten in earlier changes.
Adding new error messages
- ER_CANNOT_CAST_ON_IDENT1_ASSIGNMENT_FOR_OPERATION
- ER_CANNOT_CAST_ON_IDENT2_ASSIGNMENT_FOR_OPERATION
Summary:
The charset definition files sql/share/charsets/Index.xml and
mysql-test/std_data/ldml/Index.xml contained duplicate "flag" attributes
on single <collation> elements, violating XML well-formedness rules.
Standard XML parsers (xmllint, libxml2, etc.) reject duplicate attributes,
making these files unparseable by any spec-compliant tool.
Root Cause:
When nopad_bin collations were added, their flags were specified as
XML attributes: flag="binary" flag="nopad". The XML specification
(Section 3.1, Well-Formedness Constraint: Unique Att Spec) prohibits
duplicate attribute names on a single element. MariaDB's custom XML
parser in strings/xml.c happened to process both duplicates because
it handles attributes sequentially in a while loop, but this is
non-standard behavior that breaks interoperability with standard
XML tooling.
What the patch does:
Converts all 24 occurrences of duplicate flag attributes from
self-closing elements with duplicate attributes to elements with
child <flag> nodes. This follows the existing pattern already used
by many collations in the same file (e.g., big5_chinese_ci,
latin1_swedish_ci, utf8mb3_general_ci).
Before (invalid XML):
<collation name="latin2_nopad_bin" id="1101" flag="binary" flag="nopad"/>
After (valid XML):
<collation name="latin2_nopad_bin" id="1101">
<flag>binary</flag>
<flag>nopad</flag>
</collation>
No C code changes are required. The _CS_FLAG handler in
strings/ctype.c (around line 621) already processes <flag> child
elements using bitwise OR (|=) to accumulate flags, so both "binary"
(MY_CS_BINSORT) and "nopad" (MY_CS_NOPAD) flags are correctly applied.
Files modified:
- sql/share/charsets/Index.xml (23 collations fixed)
- mysql-test/std_data/ldml/Index.xml (1 collation fixed)
Complete list of 24 collations fixed:
sql/share/charsets/Index.xml:
1. latin2_nopad_bin (id=1101)
2. dec8_nopad_bin (id=1093)
3. cp850_nopad_bin (id=1104)
4. hp8_nopad_bin (id=1096)
5. koi8r_nopad_bin (id=1098)
6. swe7_nopad_bin (id=1106)
7. ascii_nopad_bin (id=1089)
8. cp1251_nopad_bin (id=1074)
9. hebrew_nopad_bin (id=1095)
10. latin7_nopad_bin (id=1103)
11. koi8u_nopad_bin (id=1099)
12. greek_nopad_bin (id=1094)
13. cp1250_nopad_bin (id=1090)
14. cp1257_nopad_bin (id=1082)
15. latin5_nopad_bin (id=1102)
16. armscii8_nopad_bin (id=1088)
17. cp866_nopad_bin (id=1092)
18. keybcs2_nopad_bin (id=1097)
19. macce_nopad_bin (id=1067)
20. macroman_nopad_bin (id=1077)
21. cp852_nopad_bin (id=1105)
22. cp1256_nopad_bin (id=1091)
23. geostd8_nopad_bin (id=1117)
mysql-test/std_data/ldml/Index.xml:
24. ascii2_nopad_bin (id=325)
Validation:
- xmllint --noout passes cleanly on both files after the fix
- Zero duplicate flag attributes remain (verified with grep)
- The fix is consistent with the existing pattern used by other
collations in the same files
Co-Authored-By: Claude AI <noreply@anthropic.com>
removed:
* DES encryption
* --secure-auth
* --old
* spider table options: bfz, btt, cmd, ctp, cwg, isa, ilm, ios, smd, stc, stl
extended under old-mode:
* YEAR(2), still available when old-mode=2_DIGIT_YEAR
un-deprecated:
* keep_files_on_create, originally (MDEV-23570) the idea was to make it
TRUE and deprecate. It cannot be removed when FALSE, but TRUE breaks
mariabackup.aria_backup where a table is altered from Aria to InnoDB
during a backup, so both t.MAD/t.MAI and t.ibd gets into a backup.
The persistence code of CHANGE MASTER values needs to match file
lines with the formatting for the corresponding field’s type.
This is unstructured, repetitive (not DRY), and makes feature expansions
(such as MDEV-28302 `CHANGE MASTER …=DEFAULT`)
error-prone if not difficult.
This commit moves these functions and global constants from
`slave.cc`/`.h`, as well as the Master and Relay Log Info File
entries from `Rpl_mi` and `Rpl_rli`, to dedicated `rpl_*info_file.h`
files and under corresponding structs to facilitate organization.
Namely, this commit wraps those file entries with transparent
structs that inherit from a `Persistent` interface,
or shared helper structs that themselves inherit from `Persistent`.
By moving the file read/write helper functions to or behind
implementations of `Persistent`’s virtual methods,
reading or writing the file only takes a type-agnostic loop
over the (wrapped) CHANGE MASTER values.
* This commit also includes preemptive support
for preserving MDEV-28302’s `=DEFAULT`.
As such, unset fields (namely `master_connect_retry`)
now remember their `DEFAULT` states
rather than whatever the default is at CHANGE MASTER time.
* For consistency’s sake, `master_heartbeat_period` is
now reset at RESET REPLICA instead of CHANGE MASTER.
As this refactor will disconnect it from fixes for
some open bugs in prior versions, this commit also:
* Reimplements the value reader functions to be strict with their input
* Fixes MDEV-38010
number parsing ignores trailing garbage and overflows
* Supercedes MDEV-38020 integer overflow
* Changes master_heartbeat_period from a `float` of seconds
to a `uint32_t` of milliseconds (adding `/1000.0`s as needed)
* Fixes MDEV-35879 `Slave_heartbeat_period` is imprecise
* The maximum of `master_heartbeat_period` has been
increased to 4294967.295, i.e., (2³²-1)÷1000.
* `master_heartbeat_period` now rounds
instead of truncates (rounds down).
* Prepares to make `master_retry_count` 64-bit on
LLP64 (e.g., Windows) to match LP64 (e.g., Linux)
Reviewed-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
One can have data loss in multi-master setups when 1) both masters
update the same table, 2) ALTER TABLE is run on one master which
re-arranges the column ordering, and 3) transactions are binlogged
in ROW binlog_format.
This is because the slave assumes that all columns are in the same
order on the master and slave and all columns on the master also
exists on the slave. This happens even if binlog_row_metadata=FULL is
used. If this is not the case, this will lead to silent data loss.
A new option for slave_type_conversions bit field,
ERROR_IF_MISSING_FIELD, has been added, along with a new error,
ER_SLAVE_INCOMPATIBLE_TABLE_DEF. This allows the user to define if
the slave should abort replication if it is missing some field that
existed on the master. The option is off by default to keep things
compatible with earlier versions.
If a field is missing on the slave and log_warnings >= 1, a warning
will be logged to the error log.
This patch fixes this, when binlog_row_metadata=FULL is used on the
master, by mapping fields with identical names on the master and slave.
If slave has fields that does not exist in the row event, these will
be set to their default value.
The main idea is that we added two conversion tables:
m_tabledef.master_to_slave_map[master_column_index] -> slave_column_index
and m_tabledef.master_to_slave_error[master_column_index] which contains
an error number if the master_column does not exist on the slave or
it is not possible to convert the master data to the slave column.
master_to_slave_error[#] contains 0 if the column exists and is compatible.
General code changes:
- Instead of looping over row fields in the order of slave table
we are instead looping over fields in the order of the binary log.
- We are using table->write_set to know which fields should be updated
on the slave. This is reflected in unpack_row
- We are calling TABLE::mark_columns_per_binlog_row_image() to ensure
that rpl_write_set is properly set. This is needed if the slave also
is doing binary logging.
- Before replication aborted if the master and slave tables were too
different. Now replication is only aborted if the row actually uses
columns that does not exists on the slave (and ERROR_IF_MISSING_FIELD
is used) or uses columns that cannot be converted.
- Instead of giving errors in compatible_with(), used when table is
accessed by first the row event, we are instead giving errors
when we examine a row event and notice that it is accessing
a not existing or not compatible field.
Other code changes:
- Removed conv_table argument from compatible_with() and store it
directly in RPL_TABLE_LIST->m_conv_table
- table_def::compatible_with() returns now 1 on error (not 0).
- Remove m_width and skip arguments from prepare_record() as we are
now using table->write_set() to check which elements need a default
value.
- Moved DBUG_ENTER() to it's proper place (after variable
declarations) in a few functions.
- Some changes in unpack_row():
- Replaced null_mask and null_ptr with an indexed bit check for
simplicity.
- Removed check of rgi == null and table_found which never worked.
- Updated comments to reflect current code.
- Indentation changes as the code now uses 'continue' instead of
'if-else' in the main loop.
- The code to throw away 'extra master fields' is not needed as we
are now looping over fields in binary log, not over fields in
slave table.
- Simplified get_table_data(TABLE *table_arg) by returning found
table_list.
- Errors for row events are now initialized in compatible_with(),
checked in check_wrong_column_usage() and reported in
give_compatibility_error().
Co-authored-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
This patch solves two problems:
1. Rows log events cannot be transmitted to the slave if their
size exceeds slave_max_packet_size (max 1GB at the time of
writing this patch, i.e. MariaDB 12.3)
2. Rows log events cannot be binlogged if they are larger than
4GB because the common binlog event header field event_len is
32-bits.
This patch adds support for fragmenting large Rows_log_events
through a new event type, Partial_rows_log_event. When any given
instantiation of a Rows_log_event (e.g. Write_rows_log_event, etc)
is too large to be sent to a replica (i.e. larger than the value
slave_max_allowed_packet, as configured on a replica), then the rows
event must be fragmented into sub-events (i.e.
Partial_rows_log_events), so the event can be transmitted to the
replica. The replica will then take the content of each of these
Partial_rows_log_events, and join them together into a large
Rows_log_event to be executed as normal. Partial_rows_log_events
are written to the binary log sequentially, and the replica assembles
the events in the order they are binlogged.
To control the size of each Partial_rows_log_event, a new system
variable is added: binlog_row_event_fragment_threshold. All
Partial_rows_log_events within the same group will have this size,
except for the last, which will take up the remaining length of the
underlying Rows_log_event.
Each Partial_rows_log_event stores its sequence number (seq_no) in the
overall series of fragments, the total number of fragments needed to
re-assemble the Rows_log_event (total_fragments), a uchar for flags for
embedding extra data, and any additional data as specified by the
flags. Currently, only the first event in a grouping will have
additional data: it will set the first bit in the flags
(FL_ORIG_EVENT_SIZE) to indicate it will be storing the total size of
the underlying Rows_log_event.
The cached Rows_log_event data is fragmented into
Partial_rows_log_events as follows. The primary will still generate a
Rows_log_event to write to the binlog; however, during the actual
writing process, the raw data of the rows event is split into
fragments, each covering some continuous section of the rows data. A
Partial_rows_log_event is created for each continuous section, and the
Partial_rows_log_events are written sequentially in-place of the
too-large Rows_log_event. The original data to be fragmented will
include a header and data header; however, will not include a checksum,
as each Partial_rows_log_event will have a checksum for validation, as
well as a sequence_number and total number of fragments to ensure all
fragments are present.
The re-assembly and execution of the original Rows_log_event on the
replica happens in Partial_rows_log_event::do_apply_event(). The rgi
is extended with a memory buffer that holds all data for the
original Rows_log_event. As each Partial_rows_log_event is
ingested, its Rows_log_event content is appended to this memory
buffer. Once the last fragment has added its content, a new
Rows_log_event is created using that buffer, and executed.
A new error message is added to indicate that the slave has received an
invalid stream of Partial_rows_log_events:
ER_PARTIAL_ROWS_LOG_EVENT_BAD_STREAM.
Note this commit only adds the server logic for fragmentic and
assembling events, the client logic (mysqlbinlog) is in the next
commit.
Alternative designs considered were:
1. Alternative 1: Change the master-slave communication protocol
such that the master would send events in chunks of size
slave_max_allowed_packet. Though this is still a valid idea,
and would solve the first problem described in this commit
message, this would still leave the limitation that
Rows_log_events could not exceed 4GB. Eventually, this change
should still be addressed (e.g. in MDEV-37853), and for users
of binlog-in-engine (MDEV-34705) which supports out-of-band
binlogging of events, this MDEV-32570 work will be superseded.
2. Alternative 2: Create a generic “Container_log_event” with the
intention to embed various other types of event data for
various purposes, with flags that describe the purpose of a
given container. This seemed overboard, as there is already a
generic Log_event framework that provides the necessary
abstractions to fragment/reassemble events without adding in
extra abstractions.
3. Alternative 3: Add a flag to Rows_log_event with semantics to
overwrite/correct the event_len field of the common event
header to use a 64-bit field stored in the data_header of the
Rows_log_event; and also do alternative 1, so the master would
send the large (> 4GB) rows event in chunks. This approach
would add too much complexity (changing both the binlogging
and transport layer); as well as introduce inconsistency to
the event definition (event_len and next_event_position would
no longer have consistent meanings).
Reviewed-by: Hemant Dangi <hemant.dangi@mariadb.com>
Acked-by: Kristian Nielsen <knielsen@knielsen-hq.org>
Signed-off-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
Implement an improved binlog implementation that is integrated into
the storage engine. The new implementation is enabled with the
--binlog-storage-engine option. Initially the InnoDB storage engine
implements the binlog.
Integrating the binlog in the storage engine improves performance,
since it makes the InnoDB redo log the single source of truth and
avoids the need for expensive two-phase commit between binlog and
engine. It also makes it possible to disable durability (set
--innodb-flush-log-at-trx-commit=0) to further improve performance,
while still preserving the ability to recover the binlog and database
into a consistent state after a crash.
The new binlog implementation also greatly improves the internal
design and implementation of the binlog, and enables future
enhancements for replication.
This is a squash of the original 11.4-based patch series.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
The error message shown when MAX_EXECUTION_TIME hint causes a timeout
incorrectly referred to "max_statement_time", which is a different
variable with different units (seconds vs milliseconds). This could
confuse users.
Before: "Query execution was interrupted (max_statement_time exceeded)"
After: "Query was interrupted: execution time limit 1.5 sec exceeded"
The new message is neutral and accurate for both max_statement_time
system variable and MAX_EXECUTION_TIME optimizer hint, displaying the
actual timeout value that was exceeded.
Query blocks have implicit names, such as `select#1`, formulated
by appending their select number to the string `select#`. This patch
allows hints to scope their applicability by implicit query block
name. For example,
SELECT /*+ JOIN_ORDER(@`select#2` t1, t2) */ ...
@`select#2` is an implicit query block name. Users can control hint
applicability per query block without first naming the blocks with
QB_NAME().
Hints may now be specified within VIEWs during their creation and
they are applied locally within that VIEW's scope. For example,
CREATE VIEW v1 AS
SELECT /*+ IGNORE_INDEX(t1 idx1) */ FROM t1 ... GROUP BY ... HAVING ...
In many cases and for some parts of the VIEW, the query plan
doesn't really depend on how the VIEW is used, so it makes sense
to control a part of the query plan from the VIEW definition.
Implicit names are not yet supported in VIEWs. Attempting to create
a VIEW with an implicit name reference will cause the server to create
the VIEW, but it will emit a warning and exclude that hint from the query.
cannot add new warnings in 11.8 anymore:
* remove ER_WARN_DEFAULT_SYNTAX
* use ER_VARIABLE_IGNORED instead
* change the wording in it to be more generic
* simplified new_mode warning-printing code
(cherry picked from commit 399edc7c62)
@@new_mode is a set of flags to control introduced features.
Flags are by default off. Setting a flag in @@new_mode will introduce
a new different server behaviour and/or set of features.
We also introduce a new option 'hidden_values' into some system variable
types to hide options that we do not wish to show as options.
- Don't print hidden values in mysqld --help output.
- Make get_options() use the same logic as check_new_mode_value() does.
- Setting @@new_mode=ALL shouldn't give warnings.
(cherry picked from commit f7387cb13d)
cannot add new warnings in 11.8 anymore:
* remove ER_WARN_DEFAULT_SYNTAX
* use ER_VARIABLE_IGNORED instead
* change the wording in it to be more generic
* simplified new_mode warning-printing code
@@new_mode is a set of flags to control introduced features.
Flags are by default off. Setting a flag in @@new_mode will introduce
a new different server behaviour and/or set of features.
We also introduce a new option 'hidden_values' into some system variable
types to hide options that we do not wish to show as options.
- Don't print hidden values in mysqld --help output.
- Make get_options() use the same logic as check_new_mode_value() does.
- Setting @@new_mode=ALL shouldn't give warnings.
- Removed duplicate words, like "the the" and "to to"
- Removed duplicate lines (one double sort line found in mysql.cc)
- Fixed some typos found while searching for duplicate words.
Command used to find duplicate words:
egrep -rI "\s([a-zA-Z]+)\s+\1\s" | grep -v param
Thanks to Artjoms Rimdjonoks for the command and pointing out the
spelling errors.
Part of an umbrella task MDEV-33281 for implementing optimizer hints.
This commit introduces hints affecting the use of indexes:
- JOIN_INDEX, NO_JOIN_INDEX
- GROUP_INDEX, NO_GROUP_INDEX
- ORDER_INDEX, NO_ORDER_INDEX
- INDEX, NO_INDEX
Syntax of index hints:
hint_name([@query_block_name] tbl_name [index_name [, index_name] ...])
hint_name(tbl_name@query_block_name [index_name [, index_name] ...])
JOIN_INDEX, NO_JOIN_INDEX: Forces the server to use or ignore the specified
index or indexes for any access method, such as ref, range, index_merge,
and so on. Equivalent to FORCE INDEX FOR JOIN, IGNORE INDEX FOR JOIN.
GROUP_INDEX, NO_GROUP_INDEX: Enable or disable the specified index or indexes
for index scans for GROUP BY operations. Equivalent to the index hints
FORCE INDEX FOR GROUP BY, IGNORE INDEX FOR GROUP BY.
ORDER_INDEX, NO_ORDER_INDEX: Causes the server to use or to ignore
the specified index or indexes for sorting rows. Equivalent to
FORCE INDEX FOR ORDER BY, IGNORE INDEX FOR ORDER BY.
INDEX, NO_INDEX: Acts as the combination of JOIN_INDEX, GROUP_INDEX
and ORDER_INDEX, forcing the server to use the specified index or indexes
for any and all scopes, or as the combination of NO_JOIN_INDEX, NO_GROUP_INDEX
and NO_ORDER_INDEX, which causes the server to ignore the specified index
or indexes for any and all scopes. Equivalent to FORCE INDEX, IGNORE INDEX.
Two kinds of index hints were introduced during implementation:
the global kind for [NO_]INDEX hint, and the non-global kind for all others.
Possible conflicts which will generate warnings:
- for a table level hint
- a hint of the same type or the opposite kind has already been specified
for the same table
- for a index level hint
- the same type of hint has already been specified for the same
table or for the same index, OR
- the opposite kind of hint has already been specified for the
same index
- For a multi index hint like JOIN_INDEX(t1 i1, i2, i3), it conflicts
with a previous hint if any of the JOIN_INDEX(t1 i1), JOIN_INDEX(t1 i2),
JOIN_INDEX(t1 i3) conflicts with a previous hint
Parser changes made by Alexander Barkov <bar@mariadb.com>.
Part of the tests made by Iqbal Hassan <iqbal@hasprime.com>.
Initially marking with ORA_JOIN flag made also by
Iqbal Hassan <iqbal@hasprime.com>.
Main idea is that parser mark fields with (+) with a flag
(ORA_JOIN).
During Prepare the flag bring to the new created items if needed.
Later after preparing (fix_firlds()) WHERE confition the
relations betweel the tables analyzed and tables reordered
so to make JOIN/LEFT JOIN operators in chain equivalent to
the query with oracle outer join operator (+).
Then the flags of (+) removed.
This patch adds support for associative arrays in stored procedures
for sql_mode=ORACLE.
The syntax follows Oracle's PL/SQL syntax for associative arrays -
TYPE assoc_array_t IS TABLE OF VARCHAR2(100) INDEX BY INTEGER;
or
TYPE assoc_array_t IS TABLE OF record_t INDEX BY VARCHAR2(100);
where record_t is a record type.
The following functions were added for associative arrays:
- COUNT - Retrieve the number of elements within the arra
- EXISTS - Check whether given key exists in the array
- FIRST - Retrieve the first key in the array
- LAST - Retrieve the last key in the array
- PRIOR - Retrieve the key before the given key
- NEXT - Retrieve the key after the given key
- DELETE - Remove the element with the given key or remove all elements
if no key is given
The arrays/elements can be initialized with the following methods:
- Constructor
i.e. array:= assoc_array_t('key1'=>1, 'key2'=>2, 'key3'=>3)
- Assignment
i.e. array(key):= record_t(1, 2)
- SELECT INTO
i.e. SELECT x INTO array(key)
TODOs:
- Nested tables are not supported yet.
i.e. TYPE assoc_array_t IS TABLE OF other_assoc_array_t INDEX BY INTEGER;
- Associative arrays comparisons are not supported yet.
DROP USER looks for sessions by the do-be-dropped user and if found:
* fails with ER_CANNOT_USER in Oracle mode
* continues with ER_ACTIVE_CONNECTIONS_FOR_USER_TO_DROP warning otherwise
Every user being dropped is marked with flag that disallow establishing
a new connections on behalf this user.
Currently, there are multiple error codes reported for the issue
'Can't execute init_slave query'. Those error codes are the underlying
reason the init_slave query cannot be executed, but this makes it
difficult to detect the issue in an automated way.
This patch introduces a new error code, ER_INIT_SLAVE_ERROR, to unify
all the errors related to the init_slave query. The ER_INIT_SLAVE_ERROR
error is raised for any issue related to the init_slave query, and the
underlying error code and message are included in the Last_SQL_Error
field.
Reviewed by:
Jimmy Hu <jimmy.hu@mariadb.com>
Brandon Nesterenko <brandon.nesterenko@mariadb.com>
Two new error codes ER_SEQUENCE_TABLE_HAS_TOO_FEW_ROWS and
ER_SEQUENCE_TABLE_HAS_TOO_MANY_ROWS were introduced in MDEV-36032 in
both 10.11 and, as part of MDEV-22491, 12.0. Here we remove them from
10.11, but they should remain in 12.0.
Due to complications while parsing of INSERT..SELECT statements,
optimizer hints placed at the INSERT part are ignored.
At the same time, hints placed at the SELECT part of
INSERT..SELECT statements are fully supported.
This commit implements optimizer hints allowing to affect the order
of joining tables:
- JOIN_FIXED_ORDER similar to existing STRAIGHT_JOIN hint;
- JOIN_ORDER to apply the specified table order;
- JOIN_PREFIX to hint what tables should be first in the join;
- JOIN_SUFFIX to hint what tables should be last in the join.
Forbid adding optimizer hints to view definitions.
In the case when optimizer hints are added to the view definition
at a `CREATE (OR REPLACE) VIEW`/`ALTER VIEW` statement, a warning is
generated and the hints are ignored.
This commit also disables ps-protocol for test cases where
`Unresolved table/index name` warnings are generated. The reason
for this is such warnings are generated during both PREPARE
and EXECUTE stages. Since opt_hints.test has `--enable_prepare_warnings`,
running it with `--ps-protocol` causes duplication of warning messages
