New read-only system variables (set at startup, need --log-error):
* log_error_file_buffer_size: error log buffer size, default 0 (off)
* log_error_file_rotate_size: rotate the error log above this size,
default 1073741824
* log_error_file_rotations: number of rotated files kept, default 0
Changes:
* Write the error log through the logger service, which batches writes
in an in-memory buffer, when log_error_file_buffer_size is set.
* print_buffer_to_file() uses one static buffer, allocated on first use
(MAX_LOG_BUFFER_SIZE+1), grown as needed and freed at shutdown.
* vprint_msg_to_log() writes lines longer than 1024 bytes in full, up
to 1 MB.
* Galera messages use the same error log path. Debug messages are
written only if wsrep_debug is set.
* Add mtr tests for the three variables, buffered logging, rotation and
long lines.
Assisted-by: https://mariadb.org/governance/governance-ai-policy/
Replace inet_aton/inet_addr/gethostbyname/gethostbyaddr with
their modern POSIX equivalents inet_pton/getaddrinfo/getnameinfo.
This adds IPv6 support to resolveip and removes dependency on the
deprecated h_errno global.
Handle IPv6 unspecified address (::) as Null-IP-Addr, consistent
with existing IPv4 INADDR_ANY handling for 0.0.0.0.
Add MTR test covering IPv4/IPv6 reverse lookups, special
addresses, multiple arguments, mixed address types, and
invalid hostname handling.
Co-Authored-By: Claude AI <noreply@anthropic.com>
The prefix database_name was getting dropped in the explain extended plan note
for certain non-view blocks of queries.
alias_name_used field in the table is not initialized correctly in all
the query blocks.
This PR makes the database prefix to be added before the [alias]
table name consistently when they are part of the non-view query blocks.
The attached_condition field of the produced json text had
"String with illegal unicode symbol" instead of the proper unicode
value.
The reason is that, the item field when being written to the json
writer in write_item(), used a String with charset my_charset_bin,
and that prevented the string from being escaped.
Solution is to change: -
1. charset to my_charset_utf8mb4_bin in sql_explain.cc.
2. Item::print_value(), handle string type specially if
the charset is my_charset_bin by appending hex value.
This patch adds the following syntax:
OPEN c0 FOR LOCAL spvar_with_ps_name;
PREPARE LOCAL spvar_with_ps_name FROM 'dynamic sql';
EXECUTE LOCAL spvar_with_ps_name;
DEALLOCATE PREPARE LOCAL spvar_with_ps_name;
OPEN c0 FOR PREPARE stmt;
* reduced PARSEC_ITERATIONS_MAX because PKCS5_PBKDF2_HMAC() takes
a signed int
* instead of checking for '0'..'L' range (which includes punctuation,
check for the valid result of base62_to_uchar()
* add a connection test with non-default parsec_iterations
* make parsec_iterations a normal command line option
* update C/C to allow larger number of iterations in the client parsec
This patch adds a global and session level plugin variable
parsec_iterations to define the number of iterations to be
used when generating the key corresponding to the password.
It has a default value, lower and upper bounds.
Removed type aliases and macros defined in my_global.h from my_bit.h,
making it possible to include it in plugins or external code without
pulling in my_global.h
Under --view-protocol, SET NAMES utf8mb4 isn't copied
to the connection the view protocol is running on. As
a result the SELECT statements, from which the view are
created, have string literals are interpreted as latin1.
Cast the string literals explicitly to utf8mb4 for
consistent results. Also rename the column on select
queries to ensure result name is compatible with latin1.
Reviewed-by: Yuchen Pei
While performing loose scan semi-join optimization in
Loose_scan_opt::check_ref_access_part1() [opt_subselect.h],
the filtered value in the explain plan was in-consistent for a table
with index when same data was inserted using "insert into t1 values()...",
and "insert into t1 select * from t2". This is noticed when
rec_per_key[] was used to get the number of records during optimization.
However, when actual_rec_per_key() is used to get the number of records,
we don't notice this discrepancy.
This PR changes the rec_per_key[] usage to actual_rec_per_key() in the
method Loose_scan_opt::check_ref_access_part1() [opt_subselect.h]
MySQL 5.7+ supports shorthand JSON path operators as syntactic sugar:
col->'$.path' is equivalent to JSON_EXTRACT(col, '$.path')
col->>'$.path' is equivalent to JSON_UNQUOTE(JSON_EXTRACT(col, '$.path'))
This patch adds support for both operators in MariaDB.
Changes:
- sql/sql_lex.cc: Added lookahead on '-' to tokenise '->' and '->>'
as distinct tokens (JSON_ARROW and JSON_LONGVARROW).
- sql/sql_yacc.yy: Added grammar rules under primary_expr. The LHS is
restricted to primary_base_expr to avoid shift/reduce ambiguity.
Operators are left-associative with higher precedence than arithmetic.
Bison conflict counts updated from 70/71 to 72/73 accordingly.
- mysql-test/suite/json/t/mdev13594.test,
mysql-test/suite/json/r/mdev13594.result: New MTR test suite
verifying both operators across basic extraction, unquoting, nested
paths, and NULL handling. Passes with an ASAN-enabled build.
Tested by: Saahil Alam
MDEV-39516 corrected the interface with curl particularly around
passing options.
After this the s3.debug actually enabled curl debugging like is
was meant to. With this enabled, there are addition s3_test_ pattens
in the result file, from the error log where curl correctly provide
debug message.
ps-protocol test also failed under MDEV-33532 where a differnet
number of entries was recored.
Correct variations by using the $database/$table rather than just the
s3_test_ pattern.
As the purpose of this test is to check if s3_debug is dynamic, the
second part of the test searches for a previously, under s3_debug=1,
table t2 interaction, where the results should be NOT_FOUND.
Remove count from the result file to allow for different curl versions
and the possibiliy of being run under ps-protocol.
wsrep_causal_reads was deprecated in 10.1.3 and removed in 11.3.1.
Put in place the equivalent (and default) wsrep_sync_wait=0.
Amend a few early comments to show MariaDB as our product.
Geometry::create_from_json reset the killedptr and the stack
depth (stack_p) every time json_scan_start was called. We save/
restore these values so that the ST_GeomFrom_GeoJSON can be killed
and incur timeout, and also so that its depth measurement is kept.
Test case an bug report thanks to byteoverride.
Or max_statement_time limit or MAX_DEPTH.
Correct interuptability and depth checking of:
* JSON_CONTAINS
* JSON_CONTAINS_PATH
* JSON_EXISTS
* JSON_EXTRACT
* JSON_KEYS
* ST_GeomFromGeoJSON
Populate in the json library the killed as a specific
error message. As the json library has a killed_ptr
is approprate it handles it.
As json_report_error now processes the killed there's
no need of thd->check_killed().
Verified that the path after every json_scan_next, the
location of checking the killed_ptr, reports errors
correctly.
Backport to 10.6 without JSON_OVERLAPS
The json_engine killed_ptr was of type uchar however the enum
in the server is dependant on the architecture. On big-endian
architectures like IBM Z, retreiving a uchar* retreived the
unmodifed part of the THD->kill enum location and was always 0.
As C++11 allows enums to inherit a type, used uint32_t as the base
class of killed_state type in the server and used uint32_t in the
json library.
reload_acl_and_cache required an expression change to avoid
the compile error:
sql/sql_reload.cc:472:24: error: enumerated and non-enumerated
type in conditional expression [-Werror=enum-conversion]
472 | return result || (thd ? thd->killed : 0);
Added the kill_ptr assignment the following functions can be
killed:
* Item_func_json_depth::val_int
* Item_func_json_type::val_str
* Item_func_json_length::val_int
Item_func_json_array_append::val_str, check_killed() only applied
to json_error: label and not return_null.
Item_func_json_format::val_str(), corresponding to the SQL,
json_compact, json_detailed, json_loose - add debug instrumentation
for the func_json_notembedded test.
MDEV-26726 (fcd345de48) added a json_pause_execution sync point
however the test case used debug_max_statement_time, which doesn't
exist. Fix the sync point name in the test.
(10.6 backport)
Json test about max statement time fails with freebsd because on some
architectures the test might execute faster and the statement may not fail.
To simulate failure regardless of architecture, introduce a wait of seconds
longer than the max_statement_time.
Clang upstream bug https://github.com/llvm/llvm-project/issues/173210
shows that a "int x=x" construct will in the intermediate representation
have code to read from x. With this generated sanitizer=undefined and
static analyzers will all see the uninitialized read and write.
Because clang has a stronger implementation of following paths to
uninitialized variables, "#define UNINIT_VAR(x) x" is the which
is what our release binaries use is the ideal path for this compiler.
Clang based compilers with error during compilation if any
uninitialized behaviour is detected at compile time because of
0c80ddb519.
Corrects MDEV-36542 - 6fd57f478f.
Fill in the anomaly shutdown paths of InnoDB to include call to
buf_mem_pressure_shutdown now that MDEV-39585 provides some
proper calls shutdown InnoDB and other plugins during the shutdown
under --bootstrap.
Alternate: destructor attribute on buf_mem_pressure_shutdown would
acheive the same thing and given Linux compilers are capabile of
this. This is possible as mem_pressure is currently implemented
in Linux only.
Don't perform mysqld_win_initiate_shutdown under --bootstrap when
triggered by SHUTDOWN. With this we don't perform any service
interactions.
Then the shutdown can proceeded without then hard process termination
in mysqld_win_initiate_shutdown. This previously occurred because the
handle_connections_win() was never called in --bootstrap and therefore
startup_complete() was false.
Thanks Vladislav Vaintroub for investigation and providing
implementation guidance.
mariadbd under --bootstrap failed to preform plugin deinitialization.
The sleep(2);exit is removed and replaced to a goto termination label
to perform the same shutdown procedure of the server after all the
connection closing.
To prevent a compile error about char *user being uninitialized
this sql_print_information(ER_DEFAULT(ER_NORMAL_SHUTDOWN)) is moved to
its own block. The memory free did need to occur in the bootstrap mode
too to avoid memory leak errors.
wait_for_signal_thread_to_end(), was previously in close_connections()
however its required too for --bootstrap.
When a NATURAL JOIN operates on a derived table (or view with
derived_merge=off) having more than 64 columns, the optimizer's
generate_derived_keys_for_table() would:
1) Overflow a 32-bit shift: (key_part_map)(1 << parts) when parts >= 32
2) Create a derived key with more parts than Bitmap<64>/key_part_map can hold
3) Crash on DBUG_ASSERT(prefix_size <= width) in Bitmap<64>::is_prefix(65)
Fix: cap the number of hash-join KEYUSE entries for a materialized derived
table at the handler's max_key_parts(). Excess columns are excluded from the
key, so generate_derived_keys_for_table() never builds a key with more parts
than key_part_map (64 bits) or Bitmap<64> can represent. Also fix the shift
to cast before shifting: (key_part_map) 1 << parts.
Problem:
The following functions-
- `Item_func_last_day::get_date`
- `Item_date_typecast::get_date`
- `Item_datetime_typecast::get_date` forwarded the TIME_INTERVAL_DAY /
TIME_INTERVAL_hhmmssff flags, set by an enclosing EXTRACT, into the
DATE/DATETIME conversion of their argument.This made, for ex:
LAST_DAY(200012010000) and CAST(200012010000 AS DATETIME) parse number
as TIME interval instead of a DATETIME. Under ZERO_DATE_TIME_CAST
that hour became an out-of-range month, overflowing days_in_month in
`check_date` and invalidating the Datetime.
Fix:
A DATE / DATETIME can never be an interval, so suppress the
TIME_INTERVAL_hhmmssff / TIME_INTERVAL_DAY flags when `LAST_DAY` and
`CAST` converts its argument. Also add a debug assert in `check_date`
to catch any future caller that violates its "month is assumed valid"
contract.
- Make Json_writer::add_str() do the quoting with json_escape_to_string().
= All other string writing methods are handled as they call this one.
- Move json_escape_to_string() from opt_histogram_json.cc to
my_json_writer.cc. This is needed because JSON writer unit tests do not
link with opt_histogram_json.
- Histogram code does its own escaping (with handling for special cases).
Add Json_writer::add_escaped_str() to accept escaped strings.
This is not intended to be commonly used.
- opt_trace_print_expanded_query() did its own escaping for
"expanded_query" string. It was done in MDEV-30354. Remove escaping
there to avoid double escaping.
- dump_record_to_trace() also did own escaping when writing table DDLs
into the optimizer trace. Remove double escaping.
- Adjust my_json_writer unit test.
prepare_inplace_alter_table() maps a field position to an InnoDB
cols[] position by skipping columns absent from that array. It used
Field::vcol_info, which is also set for STORED generated columns, so
such a column shifted the mapping by one and the FK-membership check
examined the wrong column, raising a false ER_FK_COLUMN_CANNOT_CHANGE.
Skip only truly virtual columns: use !stored_in_db() instead of
vcol_info.
A prepared statement runs preparation again on every execution, and
preparation merges any VIEWs named in the statement. A VIEW that
appears only inside a subquery is meant to be skipped by the merge for
insert pass during mysql_handle_derived, since it is not the target of
the DELETE.
The first execution prepares with the subquery still nested, so the
view is correctly skipped, and the DELETE succeeds. Optimization then
converts the IN subquery into a semi-join and adds its tables into the
first SELECT_LEX's table list (reassigning them to that SELECT_LEX).
This conversion is meant to persist for later executions.
Then the PS's second execution prepares against the already flattened
VIEW. mysql_handle_derived now finds the subquery's VIEW among the
first SELECT_LEX's tables and tries to merge it. mysql_handle_derived
already had some existing skip logic to recognize a VIEW, but the
semijoin conversion has already merged the VIEW into the first
SELECT_LEX, so the skip doesn't correctly apply. So a nested VIEW is
merged for insert with no table, causing a crash on the NULL table
pointer.
Solution is to skip a table during mysql_handle_derived when it is the
inner side of a semijoin, recognized by its embedding semijoin nest.
The plugins.unix_socket test showed the previous command process.
This occurs as after MDEV-38019 the early result to the client
results in the server side thread lingering for its cleanup.
The subsequent of DROP USER can observer the connection
still in progress and hence outputing a warning.
Used --ping in the peer_cred_test so that on completion, there
is no connection active for the user.
Use modern segmentHeap heap manager on Windows.
The problem it is solving is severily reduced scalability on concurrent
OLTP benchmarks, due to inherent scalability problems in default NT Heap
implementation.
The benchmarks (see the corresponding ticket) show improvements across
the board, with about 5-10% on low concurrency up to several orders of
magnitude on high concurrency.
Also addressed Gemini and Copilot concerns about correct namespace in
the app manifest XML file - make it the same as in Microsoft examples.
A recursive reference to a WITH RECURSIVE table that is reachable only
through a scalar subquery (in particular through a WITH clause nested
inside such a subquery) was not recognized as a subquery reference. As a
result the recursive CTE was wrongly accepted as standard-compliant instead
of being rejected with ER_NOT_STANDARD_COMPLIANT_RECURSIVE, and execution
later dereferenced an uninitialized join_tab for that reference in
st_select_lex_unit::exec_recursive(), crashing the server.
The cause was in With_element::check_dependencies_in_unit(). The flag
marking that we are inside a subquery (in_subq) was updated only after the
unit's own WITH clause had already been analyzed. Hence dependencies
discovered while descending into a WITH clause attached to a scalar
subquery were recorded in top_level_dep_map instead of sq_dep_map, so
contains_sq_with_recursive_reference() failed to report the violation.
The fix: set in_subq from unit->item before processing the unit's WITH
clause, so that dependencies found in nested WITH clauses of a scalar
subquery are correctly attributed to the subquery context.
The expected error for the MDEV-32299 cases in cte_recursive.test changes
accordingly from ER_BAD_FIELD_ERROR to ER_NOT_STANDARD_COMPLIANT_RECURSIVE:
the query is now rejected at dependency analysis for its non-compliant
recursive reference rather than failing later during name resolution. The
test still verifies that no crash occurs.
Replace the one line comment on join_type in sql_select.h with a
comment on each value. Each comment explains what the access method
is, the conditions under which the optimizer chooses it, and what
EXPLAIN will show for it.
Problem:
Values shorter than `column_compression_threshold` (default 100) are
stored uncompressed. Reading such a column returns a pointer into that
buffer rather than a copy.
When the new value is a substring of the column that aliases this
buffer at a non-zero offset, e.g. `RIGHT(c,n)` or `SUBSTRING(c,n)`,
the source and destination overlap, and the `memcpy()` in the "store
uncompressed" path copies overlapping regions, which is undefined
behaviour.
Fix:
Replace `memcpy` with `memmove` to avoid copying between overlapping
memory regions.
