This merges the patch series that fixes a lot of missing validation
checks for read_log_event(). The merge includes a few added fixes that
are needed in new code introduced in 12.3.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
cherry-pick commits 9ecec1f730, 24fd8c7856, 3e97105975
* make it easier to add more old-mode values
* update deprecation.h to match changes in versioning
When querying replication filter system variables without specifying a
connection name qualifier (e.g., SELECT @@GLOBAL.replicate_do_db),
Sys_var_rpl_filter::global_value_ptr() previously passed the unspecified
base_name directly to get_master_info(), which resolved to the default
connection (empty_clex_str) rather than respecting
@@SESSION.default_master_connection.
Fix by falling back to thd->variables.default_master_connection when
base_name->str is NULL in Sys_var_rpl_filter::global_value_ptr().
Add regression test in suite/multi_source/rpl_filter_default_master.test
covering explicit named connection, unspecified connection falling back
to default, and explicit unnamed connection with empty backticks.
`@@slave_connections_needed_for_purge`’s default of `1` ensures binary
log availability on replication masters, but is not a sensible default
suitable for all scenarios, especially for long-term slave servers and
standalone (not in a replication setup) servers.
The outcome was that standalone server users were confused why automatic
binlog purging does not work.
This commit changes this default to `0`, which is suitable for both
standalone and (when backed by prompt failure recovery)
replication setups.
`0` also more closely matches the behaviour before MDEV-31404,
which added this variable, out of the box.
This commit also adds a one-time replication warning when registering a
slave, but `@@slave_connections_needed_for_purge` is left unchanged.
Rather than enforcing a defence with an unsensible default, this
reminder will bring awareness of the risk of automatic binlog purging.
This commit also cleans up Galera and MTR workarounds to the
introduction of the `@@slave_connections_needed_for_purge=1` default.
OpenSSL 3 escapes '/' and '+' in X509_NAME_oneline() output; OpenSSL 1.1
and WolfSSL don't. A REQUIRE ISSUER/SUBJECT grant from one library can
stop matching after switching to another.
Default comparison stays strcmp(). old_mode=X509_LENIENT_COMPARE opts
into falling back to an escape-aware compare, applied to whichever side
the currently-linked library's own escaping affects, at the cost of
reopening the ambiguity a crafted certificate could exploit to
impersonate another identity.
Adds regression tests against a real certificate with an ambiguous CN
Assisted-by: Claude Sonnet 5 <noreply@anthropic.com>
don't allow to access /proc if secure-file-priv="",
set secure-file-priv=/ to access everything and disable the exclude list
remove test for a conditon that can no longer happen
Store all READ_ONLY sysvar values in the read_only_root
Even though READ_ONLY sysvars are protected, for string variables
it usually means that the pointer cannot be changed. The value it
points to - the string itself - still can be. Let's store all
values of string READ_ONLY sysvars in the read_only_root.
sysvars that point directly into argv are copied to read_only_root.
sysvars that have their values calculated and allocated now
must be explicitly marked with PREALLOCATED to let it know they
have to be free()-d.
Assisted-By: Claude:claude-4.8-opus
Protect all READ_ONLY sysvars from run-time changes.
Put them into a separate section and use my_virtual_mem_protect()
to make this section read-only before the server starts accepting
connections.
Verify that they're all protected in the sys_var constructor.
One exception: opt_noacl (--skip-grant-tables) can be changed
from 1 to 0 on FLUSH PRIVILEGES. Let's briefly drop the
protection for this 1->0 change. It can be needed only once
in a server lifetime and only if it was started with --skip-grant-tables
On shutdown the protection is removed, making variables writable
again because shutdown resets some of them during the cleanup
Assisted-By: Claude:claude-4.8-opus
GROUP_CONCAT is limited by group_concat_max_len,
but also, as a string function it must respect max_allowed_packet.
Let's introduce THD::gconcat_max_len() helper to simplify checks.
Also:
* make max group_concat_max_len value the same as max max_allowed_packet
* use the same MY_MIN((ulonglong) ..., UINT_MAX32) in
Item_func_json_objectagg as in Item_func_group_concat
* use overflow-safe type for lengths in Item_func_quote
(MAX_MAX_ALLOWED_PACKET is 1G so uint cannot overflow yet, but it's
a fragile assumption)
originally (MDEV-23570) the idea was to make it TRUE and deprecate.
It cannot be deprecated when it's FALSE, but TRUE breaks
mariabackup.aria_backup where a table is altered from Aria to InnoDB
during a backup, so both t.MAD/t.MAI and t.ibd gets into a backup.
This is MDEV-38866, when it's fixed, keep_files_on_create can be
enabled by default and deprecated at last.
Since ATM wsrep does not replicate any cascaded changes (because they are
not binlogged in the upstream code) we have to rely on foreign checks on
replicas to reproduce the cascade. Thus setting wsrep_slave_fk_checks=OFF
will result in inconsistency in any setup where there are cascading relations.
Mark this option as deprecated and make it a NOOP.
Lock LOCK_plugin only when the variable is a plugin variable.
Lock LOCK_global_system_variables only when the variable is a
global-only variable or a plugin variable.
Also fix lockless session value for Sys_var_multi_source_ulonglong.
This saves 54 calls to pthread_mutex_lock when constructing the
initial OK packet.
The added testcases do not show the lock savings. They are there to
improve coverage for the MDEV-31609 feature, in light of bugs like
MDEV-39207 and MDEV-39217.
(Variant 3, more comments)
When multiple indexes produce ordering that matches the
ORDER BY ... LIMIT clause, like
INDEX idx1(kp1, kp2)
INDEX idx2(kp2)
and the query with
WHERE kp1=const ORDER BY kp2 LIMIT 2
then test_if_cheaper_ordering() in pre-11.0 versions will choose
the index with the smallest KEY::user_defined_key_parts (idx2 in
this example). This can produce a much worse query plan.
The fix is to do what MariaDB 11.0 would do: use the index for which
we have the cheapest access method. The cost of access method here
takes into account that we will stop after producing #LIMIT rows.
The fix is controlled by the
@@optimizer_adjust_secondary_key_costs=fix_order_by_index_choice
flag and is OFF by default.
Problem was that in multi-table update that contains tables that are not
transactional and tables that are transactional error handling does not
work correctly if write set size is limited. Because, Galera
replication for MyISAM is experimental it is better to avoid
multi-table updates in this case.
Fixed by not allowing multi-table updates if statement contains
non transactional and transactional tables and write set size
is limited.
Since ATM wsrep does not replicate any cascaded changes (because they are
not binlogged in the upstream code) we have to rely on foreign checks on
replicas to reproduce the cascade. Thus setting wsrep_slave_fk_checks=OFF
will result in inconsistency in any setup where there are cascading relations.
Mark this option as deprecated and make it a NOOP.
Definition
==========
Implement the SET PATH statement.
```
SET PATH 'schema1, schema2, schema3';
```
Function
--------
In MariaDB, the SQL-path will also be used to lookup packages in
package routine invocations, so the following script:
SET PATH 'sys';
SELECT UTL_ENCODE.BASE64_DECODE('data');
invokes SYS.UTL_ENCODE.BASE64_DECODE('data'), i.e. the function
BASE64_DECODE() in the packge UTL_ENCODE in the database SYS
(if such package and package routine exist).
MariaDB implementation details
------------------------------
The ticket's requirement stated the grammar should support delimited
identifiers with backticks or double quotes within the string quote,
ie: the single tick. A simple parser was implemented to handle this
requirement in Sql_path::from_text.
Only 16 schemas are supported per session - we do not want too many
schemas in the path anyway since it will impact performance. This also
means that the path cache is a simple static array for easier memory
management.
When adding in variable type binlog_row_event_fragment_threshold, it
was made a ulong to mimick max_allowed_packet and
slave_max_allowed_packet variables (as they are intertwined). However,
using ulong is bad practice as it has a different type depending on the
architecture. This is shown by test sys_vars.sysvars_server_notembedded
where the variable shows as `BIGINT UNSIGNED` on 64-bit systems, and
`INT UNSIGNED` on 32-bit systems.
This patch updates the variable to be uint (32-bit), as there is a
fundamental limitation of 4GB on event size, so the new
Partial_rows_log_event event couldn't exceed that size anyway.
Signed-off-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
(Based on a patch by Yuchen Pei)
When computing table dependencies in simplify_joins(), do not make
LEFT JOIN-ed table (or join nest) to be dependent on all preceding tables.
For queries in form
t1
LEFT JOIN t2 ON t2.col=t1.col
LEFT JOIN t3 ON t3.col=t1.col
this allows the optimizer to construct join order of t1-t3-t2. Before
this patch, t1-t2-t3 was the only possible order.
Note that queries that use Oracle's Outer Join syntax (col1(+)=col2) are
converted into chained independent LEFT JOINs like the above.
The optimization is controlled by optimizer_switch='reorder_outer_joins=ON'
It is NOT enabled by default as it is known to expose problems with
join pruning. It is advised to set optimizer_prune_level=0 when setting
reorder_outer_joins=on.
Co-authored-by: Yuchen Pei <ycp@mariadb.com>
One can have data loss in multi-master setups when 1) both masters
update the same table, 2) ALTER TABLE is run on one master which
re-arranges the column ordering, and 3) transactions are binlogged
in ROW binlog_format.
This is because the slave assumes that all columns are in the same
order on the master and slave and all columns on the master also
exists on the slave. This happens even if binlog_row_metadata=FULL is
used. If this is not the case, this will lead to silent data loss.
A new option for slave_type_conversions bit field,
ERROR_IF_MISSING_FIELD, has been added, along with a new error,
ER_SLAVE_INCOMPATIBLE_TABLE_DEF. This allows the user to define if
the slave should abort replication if it is missing some field that
existed on the master. The option is off by default to keep things
compatible with earlier versions.
If a field is missing on the slave and log_warnings >= 1, a warning
will be logged to the error log.
This patch fixes this, when binlog_row_metadata=FULL is used on the
master, by mapping fields with identical names on the master and slave.
If slave has fields that does not exist in the row event, these will
be set to their default value.
The main idea is that we added two conversion tables:
m_tabledef.master_to_slave_map[master_column_index] -> slave_column_index
and m_tabledef.master_to_slave_error[master_column_index] which contains
an error number if the master_column does not exist on the slave or
it is not possible to convert the master data to the slave column.
master_to_slave_error[#] contains 0 if the column exists and is compatible.
General code changes:
- Instead of looping over row fields in the order of slave table
we are instead looping over fields in the order of the binary log.
- We are using table->write_set to know which fields should be updated
on the slave. This is reflected in unpack_row
- We are calling TABLE::mark_columns_per_binlog_row_image() to ensure
that rpl_write_set is properly set. This is needed if the slave also
is doing binary logging.
- Before replication aborted if the master and slave tables were too
different. Now replication is only aborted if the row actually uses
columns that does not exists on the slave (and ERROR_IF_MISSING_FIELD
is used) or uses columns that cannot be converted.
- Instead of giving errors in compatible_with(), used when table is
accessed by first the row event, we are instead giving errors
when we examine a row event and notice that it is accessing
a not existing or not compatible field.
Other code changes:
- Removed conv_table argument from compatible_with() and store it
directly in RPL_TABLE_LIST->m_conv_table
- table_def::compatible_with() returns now 1 on error (not 0).
- Remove m_width and skip arguments from prepare_record() as we are
now using table->write_set() to check which elements need a default
value.
- Moved DBUG_ENTER() to it's proper place (after variable
declarations) in a few functions.
- Some changes in unpack_row():
- Replaced null_mask and null_ptr with an indexed bit check for
simplicity.
- Removed check of rgi == null and table_found which never worked.
- Updated comments to reflect current code.
- Indentation changes as the code now uses 'continue' instead of
'if-else' in the main loop.
- The code to throw away 'extra master fields' is not needed as we
are now looping over fields in binary log, not over fields in
slave table.
- Simplified get_table_data(TABLE *table_arg) by returning found
table_list.
- Errors for row events are now initialized in compatible_with(),
checked in check_wrong_column_usage() and reported in
give_compatibility_error().
Co-authored-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
This patch solves two problems:
1. Rows log events cannot be transmitted to the slave if their
size exceeds slave_max_packet_size (max 1GB at the time of
writing this patch, i.e. MariaDB 12.3)
2. Rows log events cannot be binlogged if they are larger than
4GB because the common binlog event header field event_len is
32-bits.
This patch adds support for fragmenting large Rows_log_events
through a new event type, Partial_rows_log_event. When any given
instantiation of a Rows_log_event (e.g. Write_rows_log_event, etc)
is too large to be sent to a replica (i.e. larger than the value
slave_max_allowed_packet, as configured on a replica), then the rows
event must be fragmented into sub-events (i.e.
Partial_rows_log_events), so the event can be transmitted to the
replica. The replica will then take the content of each of these
Partial_rows_log_events, and join them together into a large
Rows_log_event to be executed as normal. Partial_rows_log_events
are written to the binary log sequentially, and the replica assembles
the events in the order they are binlogged.
To control the size of each Partial_rows_log_event, a new system
variable is added: binlog_row_event_fragment_threshold. All
Partial_rows_log_events within the same group will have this size,
except for the last, which will take up the remaining length of the
underlying Rows_log_event.
Each Partial_rows_log_event stores its sequence number (seq_no) in the
overall series of fragments, the total number of fragments needed to
re-assemble the Rows_log_event (total_fragments), a uchar for flags for
embedding extra data, and any additional data as specified by the
flags. Currently, only the first event in a grouping will have
additional data: it will set the first bit in the flags
(FL_ORIG_EVENT_SIZE) to indicate it will be storing the total size of
the underlying Rows_log_event.
The cached Rows_log_event data is fragmented into
Partial_rows_log_events as follows. The primary will still generate a
Rows_log_event to write to the binlog; however, during the actual
writing process, the raw data of the rows event is split into
fragments, each covering some continuous section of the rows data. A
Partial_rows_log_event is created for each continuous section, and the
Partial_rows_log_events are written sequentially in-place of the
too-large Rows_log_event. The original data to be fragmented will
include a header and data header; however, will not include a checksum,
as each Partial_rows_log_event will have a checksum for validation, as
well as a sequence_number and total number of fragments to ensure all
fragments are present.
The re-assembly and execution of the original Rows_log_event on the
replica happens in Partial_rows_log_event::do_apply_event(). The rgi
is extended with a memory buffer that holds all data for the
original Rows_log_event. As each Partial_rows_log_event is
ingested, its Rows_log_event content is appended to this memory
buffer. Once the last fragment has added its content, a new
Rows_log_event is created using that buffer, and executed.
A new error message is added to indicate that the slave has received an
invalid stream of Partial_rows_log_events:
ER_PARTIAL_ROWS_LOG_EVENT_BAD_STREAM.
Note this commit only adds the server logic for fragmentic and
assembling events, the client logic (mysqlbinlog) is in the next
commit.
Alternative designs considered were:
1. Alternative 1: Change the master-slave communication protocol
such that the master would send events in chunks of size
slave_max_allowed_packet. Though this is still a valid idea,
and would solve the first problem described in this commit
message, this would still leave the limitation that
Rows_log_events could not exceed 4GB. Eventually, this change
should still be addressed (e.g. in MDEV-37853), and for users
of binlog-in-engine (MDEV-34705) which supports out-of-band
binlogging of events, this MDEV-32570 work will be superseded.
2. Alternative 2: Create a generic “Container_log_event” with the
intention to embed various other types of event data for
various purposes, with flags that describe the purpose of a
given container. This seemed overboard, as there is already a
generic Log_event framework that provides the necessary
abstractions to fragment/reassemble events without adding in
extra abstractions.
3. Alternative 3: Add a flag to Rows_log_event with semantics to
overwrite/correct the event_len field of the common event
header to use a 64-bit field stored in the data_header of the
Rows_log_event; and also do alternative 1, so the master would
send the large (> 4GB) rows event in chunks. This approach
would add too much complexity (changing both the binlogging
and transport layer); as well as introduce inconsistency to
the event definition (event_len and next_event_position would
no longer have consistent meanings).
Reviewed-by: Hemant Dangi <hemant.dangi@mariadb.com>
Acked-by: Kristian Nielsen <knielsen@knielsen-hq.org>
Signed-off-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
Implement an improved binlog implementation that is integrated into
the storage engine. The new implementation is enabled with the
--binlog-storage-engine option. Initially the InnoDB storage engine
implements the binlog.
Integrating the binlog in the storage engine improves performance,
since it makes the InnoDB redo log the single source of truth and
avoids the need for expensive two-phase commit between binlog and
engine. It also makes it possible to disable durability (set
--innodb-flush-log-at-trx-commit=0) to further improve performance,
while still preserving the ability to recover the binlog and database
into a consistent state after a crash.
The new binlog implementation also greatly improves the internal
design and implementation of the binlog, and enables future
enhancements for replication.
This is a squash of the original 11.4-based patch series.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
This is a combined patch of various spelling fixes originally done in
Debian.
* Fix misc typos in MariaDB Server
* Fix spelling of 'allows one to'
Fix the following Lintian nags introduced in commit
c8d040938a:
I: mariadb-backup: spelling-error-in-binary "allows to" "allows one to" [usr/bin/mariadb-backup]
I: mariadb-server-core: spelling-error-in-binary "allows to" "allows one to" [usr/sbin/mariadbd]
I: mariadb-test: spelling-error-in-binary "allows to" "allows one to" [usr/bin/mariadb-client-test-embedded]
I: mariadb-test: spelling-error-in-binary "allows to" "allows one to" [usr/bin/mariadb-test-embedded]
I: mariadb-test: spelling-error-in-binary "allows to" "allows one to" [usr/bin/test-connect-t]
(Cherry-pick into 12.2: the fix is enabled by default,
control via @@new_mode=FIX_INDEX_LOOKUP_COST is removed)
They were based on the maximum possible length key length, which
can be much larger than the actual data.
The return value is used by handler::keyread_time(), which is used
to estimate the cost of range access.
This could cause range access not to be picked, even if it uses
the clustered PK and reads about 8% of the table.
The fix is to add KEY::stat_storage_length (next to KEY::rec_per_key) and
have the storage engine fill it in handler::info(HA_STATUS_CONST).
Currently, only InnoDB fills this based on its internal statistics:
index->stat_index_size and ib_table->stat_n_rows.
Also changed:
- In handler::calculate_costs(), use ha_keyread_clustered_time() when
computing clustered PK read cost, not ha_keyread_time().
They were based on the maximum possible key tuple length, which can be
much larger than the real data size.
The return value is used by handler::keyread_time(), which is used
to estimate the cost of range access.
This could cause range access not to be picked, even if it uses
the clustered PK and reads about 8% of the table.
The fix is to add KEY::stat_storage_length (next to KEY::rec_per_key) and
have the storage engine fill it in handler::info(HA_STATUS_CONST).
Currently, only InnoDB fills this based on its internal statistics:
index->stat_index_size and ib_table->stat_n_rows.
Also changed:
- In handler::calculate_costs(), use ha_keyread_clustered_time() when
computing clustered PK read cost, not ha_keyread_time().
The fix is OFF by default and enabled by setting FIX_INDEX_LOOKUP_COST flag
in @@new_mode.
