This is a patch posted by Marek Stipek's review, with following comment:
The row images of a reported cascade action lived in two plain THD
members, on the assumption that at most one cascade action is in flight
per THD. InnoDB performs the next level of an FK chain from inside the
outer cascade action, though (row_update_cascade_for_mysql() recurses
into row_ins_foreign_check_on_constraint()). For a chain like
gp -> p -> c:
- the nested action for c overwrote the before-image captured for p,
reported c and freed the images, so thd_fk_cascade_row() for p found
no image and silently dropped the row. Only the deepest level reached
the binary log, and since its events carry FK_CASCADE_EVENTS_F the
replica did not re-run the cascade either: it kept the rows of p,
silently diverging from the primary;
- the nested capture reused the buffer allocated for p's row length, so
a child with a longer row than its parent overran it (debug builds
report "Freeing overrun buffer" and crash).
Keep the images on a stack instead: capturing a before-image opens a
frame with buffers sized for its own table, and thd_fk_cascade_row() or
thd_fk_cascade_abort() closes it. InnoDB now closes the frame also when
capturing the before-image failed, as the service contract already
asked for, so every frame is closed exactly once.
Self-referencing foreign keys: the child of such a cascade is the table
the statement itself changes, through the instance of it opened for the
foreign key. That handler has not read anything, so it has no row
template yet, and ha_innobase::fk_cascade_fetch_row() refused to convert
the row, although it rebuilds the template anyway, allocating it if
needed. The cascaded rows were not logged; when the statement also
cascaded into another table, whose rows were, the replica did not re-run
the cascade either and kept the self-referencing rows. Drop the check.
The same check made capture fail on a replica, for the handler of the
child opened for the foreign key there has not read anything either: the
pull request documents the feature as inert on the applier and pins it
in rpl_fk_cascade_binlog_row_slave_option, but the child is open via FK
prelocking there too. With the check gone, a replica with the feature
ON (origin OFF) captures the cascades it re-runs and logs them in its own
binary log, as any other session does; the test and the design document
now say so.
Order of the cascaded rows: the queue was emitted at statement end, after
all rows of the statement itself. When a cascade changes a row that the
statement changes after it - with the self-referencing cascades now
captured, e.g. DELETE FROM s WHERE id IN (1,2) with s.parent_id -> s.id
ON DELETE SET NULL - the binlog had DELETE 2 before the cascaded UPDATE
of 2, and the replica stopped with ERROR 1032. handler::binlog_log_row()
now emits the queue before each row of the statement is logged: the
cascades that row change ran precede it, as they did in execution.
Derived rows and the statement's own go to separate row events, so that
FK_CASCADE_DERIVED_F keeps marking exactly the derived ones.
- rpl_fk_cascade_binlog_row_mixed_eligibility: a statement with an
ineligible child table is not captured at all (an eligible and an
ineligible sibling, an ineligible grandchild, an all-eligible control
case), checked on the replica and in the binlog. The design document
refers to it, but it is not in the pull request.
- rpl_fk_cascade_binlog_row_multilevel: cascades through several levels
up to the InnoDB maximum of 14 below the parent, and one level deeper
(the statement fails, nothing is logged); UPDATE and SET NULL chains,
a diamond, savepoints, self-referencing foreign keys with and without
another child, and the order of the cascaded rows relative to the
statement's own. Each case is checked on the replica.
- galera_fk_cascade_binlog_events, ..._log_bin: the same in Galera,
without and with a binary log, with parallel applying.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
1. Overview
This commit implements a feature which changes the handling of cascading foreign
key operations to write the changes of cascading operations into binlog.
The applying of such transaction, in the slave node, will apply just the binlog
events, and does not execute the actual foreign key cascade operation.
Having cascaded FK changes in binlog simplifies the slave side applying and makes
it more predictable in terms of potential interference with other parallel applying
happening in the node.
This feature can be turned ON/OFF by new variable:
rpl_use_binlog_events_for_fk_cascade, with default value OFF
2. Design
The engine reports cascade actions through a new generic service,
include/mysql/service_thd_fk_cascade.h, and carries no binlog knowledge of its
own. It asks whether the server wants to be notified of cascade FK, lets the
server capture the before and after row images at the points where cursor is
positioned, and reports the completed action. The server owns the row
images, the column bitmaps that define them, and every decision about what is
done with them. The engine's only contribution is handler::fk_cascade_fetch_row(),
implemented by ha_innobase, which converts the record the cascade cursor sits on
into MySQL row format.
Server side, the reported rows are queued on the THD in execution order and
written to the binary log at statement end or commit, after the originating
statement's own events; they are discarded on rollback and on rollback to
savepoint. Logging them inline would place every cascade delete ahead of every
deferred update within a statement.
The cascaded FK binlog events have new event flag: FK_CASCADE_EVENTS_F, which
tells the slave applier to skip FK cascading.
Binlog events have also flags to mark both original and derived (cascaded FK) events.
This will make it possible for the slave to autonomously choose whether to use
the derived events in applying or to execute the cascade operation.
Binlogged cascade FK events are also correctly handled in slaves with old MariaDB
version. This is because the events logged in cascade operation are additionally
flagged with the NO_FOREIGN_KEY_CHECKS_F flag, so replica that does not understand
the new FK_CASCADE_EVENTS_F flag, still disables foreign key checks in applying
and does not re-execute the cascade
Conplete design is presented in a design document MDEV-38243-design.md, which can
be removed if PR is merged.
3. Trigger support
The SE/server API delegates the actual work to be done in server side. SE (innodb)
has to call 4 handler functions and prepare row images for the arguments.
With this API, server is notified of cascade FK action to happen, and can react
accordingly.
This commit has implemented the binlogging of cascade FK actions, but
there are two other possible uses as well:
- checking FK constraints after FK child table delete
- calling trigger for the child table delete/update
In the design, these handlers are called consumers.
For supporting triggers, it turned out that BEFORE row type of triggers are
too hard to support. This is because the the handler is called from inside
innodb, having a number of innodb resouces locked and execution residing
inside a innodb mtr. Note that the cascade operation may dive deeper in the
FK constraint chain, and callig BEFORE row trigger would require all this
work to be rolled back to isolate innodb for the trigger call.
AFTER row type of triggers, otoh can be supported with this API. For these,
the trigger call requests can be queued and called after the statement execution
is complete.
4. Testing
The commit has mtr tests for testing the feature:
rpl.rpl_fk_cascade_binlog_row - feature OFF vs ON
rpl.rpl_fk_cascade_binlog_row_ordering - event ordering
rpl.rpl_fk_cascade_binlog_row_rollback - discard on rollback
rpl.rpl_fk_set_null_binlog_row - SET NULL capture
rpl.rpl_fk_cascade_binlog_row_slave_option - origin OFF / replica ON
rpl.rpl_fk_cascade_binlog_row_old_slave - old-replica compatibility
rpl.rpl_fk_cascade_binlog_row_mixed_eligibility - mixed child eligibility
The server_audit_timestamp_format %f specifier always rendered zero
microseconds. The server downcast the precise time to seconds before
passing it to audit plugins, and the plugin then re-fetched the time
itself at write time.
Pass the server's high-resolution time through the audit API instead:
- extend mysql_event_general with general_time_microseconds (added in
MYSQL_AUDIT_INTERFACE_VERSION 0x0304), keeping general_time in
seconds for backward compatibility
- the server_audit plugin uses event->general_time_microseconds for
query log entries instead of re-fetching the time at write time
Connection and table events carry no timestamp in the audit API, so the
plugin keeps taking the time at event time for those entries.
WolfSSL code path already operates without fixed DH parameters. OpenSSL code path still sets fixed static precomputed DH params, which is now prohibited by IETF.
Also OPENSSL_init_ssl is not required since OpenSSL 1.1.0, for over 10 years now. Also cleaned up at the same time.
https://www.rfc-editor.org/rfc/rfc10015.html#section-2:
> Clients MUST NOT offer and servers MUST NOT select non-ephemeral FFDH cipher suites in (D)TLS 1.2 connections.
https://www.rfc-editor.org/rfc/rfc10015.html#section-3:
> Clients MUST NOT offer and servers MUST NOT select FFDHE cipher suites in (D)TLS 1.2 connections.
And the depreciated tables include all ciphersuites that can use SSL_CTX_set_tmp_dh as part of the connection.
Also for a very long time OpenSSL was handling these automatically anyway, back when DHE was still recommended.
Create a GitHub action into the MariaDB server tree to build the Markdown
documentation using the cmake target provided by MDEV-39718.
This workflow generates API documentation for the plugin using a
Docker container packed with all the extra doxygen/moxygen tools.
Fixed some more doxygen problems.
Moved the repo to ghcr.io/mariadb/mariadb-doc-gen:latest.
Generated the plugin API headers using a shell script.
Fixed some doxygen comment mistakes in the headers.
Added a cmake conveninence target to generate the docs into $BUILD_DIR/docs
Added a main page for the API docs.
Included all of the existing group .md files into the CMake target
Leveraged moxygen 2.1.11's fixes to produce the full API docs in a single go
Removed the list of output .md files from the CMake target and switched to a
stamp file to avoid unnecessary rebuilds of the docs when the list of .md
files changes.
Addressed various review comments.
When starting to do a full table/index scan without a WHERE or JOIN
condition, tell the storage engine so and the corresponding
ulong-truncated LIMIT.
Include an innodb implementation: added an innodb switch
table_lock_on_full_scan, so that when the switch is on, on receiving
the full scan signal from the sql layer, if the truncated LIMIT is
ULONG_MAX (likely no LIMIT), attempt to acquire a table lock.
Updated tests that have different results with the switch on.
The three deadlock_*_race tests cannot reach their DEBUG_SYNC race
under a table lock (it degenerates to a timeout), and the three I_S
tests only restate a lock-mode change already covered
by innodb_full_scan.test.
(Comment and code edited by Sergei Petrunia <sergey@mariadb.com> and
Thirunarayanan Balathandayuthapani <thiru@mariadb.com>)
Across a range of JSON functions taking a path argument
there are SQL expressions that dont' have a character
set. If these expressions don't have a character set
fall back to the character set of the argument of the
json function that represent the document being operated
on. If this doesn't have a character set fall back to
my_charset_utf8mb4_bin.
This covers the 11.4 JSON_KEY_VALUE function also as it
reuses the Json_path_extractor::extract method.
Add nonnull and warn_unused_result to the json path
functions to facilitate compiler and UBSAN catching of the
problem early.
As null values of s_p are incompatible with report_path_error,
jump directly to a null return which is consistent with
the defination of the JSON sql funciton.
Protect all READ_ONLY sysvars from run-time changes.
Put them into a separate section and use my_virtual_mem_protect()
to make this section read-only before the server starts accepting
connections.
Verify that they're all protected in the sys_var constructor.
One exception: opt_noacl (--skip-grant-tables) can be changed
from 1 to 0 on FLUSH PRIVILEGES. Let's briefly drop the
protection for this 1->0 change. It can be needed only once
in a server lifetime and only if it was started with --skip-grant-tables
On shutdown the protection is removed, making variables writable
again because shutdown resets some of them during the cleanup
Assisted-By: Claude:claude-4.8-opus
* it's called ROOT_FLAG_VMEM
* it does not imply any protection, ROOT_FLAG_MPROTECT was a misnomer
* it means that the memory is allocated using my_virtual_mem_commit,
not my_malloc, so not on heap
* memory allocated this way can be protected with my_virtual_mem_protect()
* thus root using ROOT_FLAG_VMEM can be protected with protect_root()
Assisted-By: Claude:claude-5-sonnet
make my_virtual_mem_reserve() work on Windows and on Linux
(falling back to my_large_virtual_alloc() - that's what InnoDB
did instead). This makes the usage pattern portable without #ifdef:
- reserve a memory range
- commit (allocate) memory from a reserved range
- decommit memory, keeping it reserved
- release reserved memory back to OS
invalid pointer pairs are when the length/memory of one string are
intermixed with another.
For comp_err, the end null pointer was compared to soffset within
my_strtoll10. As we didn't need the end position a NULL arg option
was compatble.
For uca-dump, Address Sanitizer raises invalid pointer pair because
argv options (opt) don't have an obvious correlation of having a start
at opt_X.length, even though the lstrncmp makes this true.
The DBUG_ASSERTS of strmov (added MDEV-11752) where incompatible with
pointer-compare. Replaced strmov with static inline version in
m_string.h using memmove that allows overlaps, and being inline allows
the uneeded parts of the implementation to be optimized away.
* restore result=NULL after free in mysqlshow.c (to prevent double-free)
* remove DBUG_EVALUATE_IF again
* remove useless 'stop slave' from a test
* restore charset encoding in mysqltest.{test,result}
* correct error for unparseable BINLOG data is ER_SYNTAX_ERROR now
* restore old libfmt on rhel-7, gcc 4.8.5 cannot compile anything newer
With proxy protocol thd_set_peer_addr() runs twice (proxy host, then the
real client from the proxy header). Connect errors are accounted against
the real client, but check_connection() incorrectly uses condition on the
proxy host's count, rather than real client's address.
Fix: reset both the proxy host, and real client's connect errors on
successful connection.
Added tests for incomplete handshake, and reset behavior,
under proxy protocol, for both real client errors, and proxy host errors.
a90779098b changed the
interface to the json_valid to take a json_engine_t
so that it could be interupted and so that warnings
could become visible.
On merge from that 10.11 to 11.4 it was discovered
that the ColumnStore engine uses this interface.
Restored the original json_valid function exactly
how ColumnStore expects this and added renamed
the function with json_engine_t to be json_valid_engine.
This follows the same convention as in the commit
756fc6fd9a that was
for json_normalize, that ColumnStore also used.
a90779098b changed the
interface to the json_valid to take a json_engine_t
so that it could be interupted and so that warnings
could become visible.
On merge from that 10.11 to 11.4 it was discovered
that the ColumnStore engine uses this interface.
Restored the original json_valid function exactly
how ColumnStore expects this and added renamed
the function with json_engine_t to be json_valid_engine.
This follows the same convention as in the commit
756fc6fd9a that was
for json_normalize, that ColumnStore also used.
14c16e02b2 changed the
interface for json_normalize to take a json_engine_t so that
there could be some error checking.
Unfortunately ColumnStore depended on this interface.
Restored original interface of json_normalize() and
added a json_normalize_engine() that take as engine as the
argument.
14c16e02b2 changed the
interface for json_normalize to take a json_engine_t so that
there could be some error checking.
Unfortunately ColumnStore depended on this interface.
Restored original interface of json_normalize() and
added a json_normalize_engine() that take as engine as the
argument.
JSON_CONTAINS, JSON_EQUALS, and JSON_OVERLAPS used raw byte-level
comparison (memcmp) for JSON string values, which meant semantically
equivalent strings like "A" and "\u0041" were incorrectly treated
as different.
Fix: add json_string_compare() that decodes Unicode escape sequences
before comparing, and fix json_normalize to produce a canonical form
for strings with escapes so JSON_EQUALS works correctly.
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.
Or max_statement_time limit.
Change the json_valid to take a json_engine_t argument.
Adjust the Item_json_valid to have a json_engine_t and
to report an error.
Note this means that invalid json syntax now has a
note on any json error. Raising this to a warning can
cause check constraints of JSON_VALID() to result
in ER_JSON_SYNTAX rather than ER_CONSTRAINT_FAILED
(valided in main.type_json test).
Consequential changes in debug build to json acl
testing validation and unit test having a dummy json_engine_t.
json_normalize also uses the same engine that was
passed as its argument to test json_valid. Because
the character set may have changed, if there's an error
adjust the je->s.c_str pointer. Its only the offset that
is currently used in the error messages anyway so there
may be offset errors depending on the original character
set.
Or max_statement_time limit.
JSON_EQUAL/JSON_NORMALIZE also didn't report warnings
on invalid JSON inputs.
Change the json_normalize function to take a json_engine_t
as a argument and implement the sql/item_jsonfunc.cc calls
to have a json_engine_t.
To the JSON_EQUALS and JSON_NORMALIZE, add error handling
to produce an error when there was one. compare_nested_object,
part of JSON_OVERLAPS, uses a new json_engine and
copies the error/position to the returning function.
json_normalize expects the incoming je structure to have
a valid, or nullptr, killed_ptr, as json_start will reset this.
Removed type aliases and macros defined in my_global.h from my_bit.h,
making it possible to include it in plugins or external code without
pulling in my_global.h
Or max_statement_time limit or MAX_DEPTH.
Correct interuptability and depth checking of:
* JSON_CONTAINS
* JSON_CONTAINS_PATH
* JSON_EXISTS
* JSON_EXTRACT
* JSON_KEYS
* ST_GeomFromGeoJSON
Populate in the json library the killed as a specific
error message. As the json library has a killed_ptr
is approprate it handles it.
As json_report_error now processes the killed there's
no need of thd->check_killed().
Verified that the path after every json_scan_next, the
location of checking the killed_ptr, reports errors
correctly.
Backport to 10.6 without JSON_OVERLAPS
The json_engine killed_ptr was of type uchar however the enum
in the server is dependant on the architecture. On big-endian
architectures like IBM Z, retreiving a uchar* retreived the
unmodifed part of the THD->kill enum location and was always 0.
As C++11 allows enums to inherit a type, used uint32_t as the base
class of killed_state type in the server and used uint32_t in the
json library.
reload_acl_and_cache required an expression change to avoid
the compile error:
sql/sql_reload.cc:472:24: error: enumerated and non-enumerated
type in conditional expression [-Werror=enum-conversion]
472 | return result || (thd ? thd->killed : 0);
Added the kill_ptr assignment the following functions can be
killed:
* Item_func_json_depth::val_int
* Item_func_json_type::val_str
* Item_func_json_length::val_int
Item_func_json_array_append::val_str, check_killed() only applied
to json_error: label and not return_null.
Item_func_json_format::val_str(), corresponding to the SQL,
json_compact, json_detailed, json_loose - add debug instrumentation
for the func_json_notembedded test.
MDEV-26726 (fcd345de48) added a json_pause_execution sync point
however the test case used debug_max_statement_time, which doesn't
exist. Fix the sync point name in the test.
