`json_normalize`/`json_equals` unnecessary
double parsing JSON document
Removed the redundant `json_valid_engine` pre-check to eliminate
double-parsing, and integrated error handling directly into the
normalization engine:
* Catch Empty Strings: Return an error directly in
`json_normalize_engine` if the root type is `JSON_VALUE_UNINITIALIZED`.
* Catch Trailing Garbage on Scalars: Updated `json_norm_build`
to enforce a full scan to the end of the document for scalar values.
This prevents edge cases (e.g., raw date strings like 2026-07-17...)
from being falsely accepted as a valid JSON number without checking
the remainder of the string for syntax errors.
* Propagate Syntax Errors: Updated `json_normalize_engine` to
explicitly check the engine's error flag after the build phase.
* Add Edge Case Tests: Added tests in `json_normalize.test` for
empty strings, whitespace, and trailing garbage on scalars to
ensure correct error generation.
Testing:
Verified locally using MTR
(`main.json_normalize` and `main.json_equals`).
Signed-off-by: VasuBhakt <cpswastik31@gmail.com>
invalid pointer pairs are when the length/memory of one string are
intermixed with another.
For comp_err, the end null pointer was compared to soffset within
my_strtoll10. As we didn't need the end position a NULL arg option
was compatble.
For uca-dump, Address Sanitizer raises invalid pointer pair because
argv options (opt) don't have an obvious correlation of having a start
at opt_X.length, even though the lstrncmp makes this true.
The DBUG_ASSERTS of strmov (added MDEV-11752) where incompatible with
pointer-compare. Replaced strmov with static inline version in
m_string.h using memmove that allows overlaps, and being inline allows
the uneeded parts of the implementation to be optimized away.
Despite the JSON spec rfc8259/#section-4
"The names within an object SHOULD be unique"[1], it does happen.
As our sort algorithm has access to both key and value,
if the key matches, sort by the value. The type, and length of
objects, strings, arrays and numbers is easy, so use those as a
first comparision point.
[1] https://www.rfc-editor.org/info/rfc8259/#section-4
a90779098b changed the
interface to the json_valid to take a json_engine_t
so that it could be interupted and so that warnings
could become visible.
On merge from that 10.11 to 11.4 it was discovered
that the ColumnStore engine uses this interface.
Restored the original json_valid function exactly
how ColumnStore expects this and added renamed
the function with json_engine_t to be json_valid_engine.
This follows the same convention as in the commit
756fc6fd9a that was
for json_normalize, that ColumnStore also used.
14c16e02b2 changed the
interface for json_normalize to take a json_engine_t so that
there could be some error checking.
Unfortunately ColumnStore depended on this interface.
Restored original interface of json_normalize() and
added a json_normalize_engine() that take as engine as the
argument.
Or max_statement_time limit.
Change the json_valid to take a json_engine_t argument.
Adjust the Item_json_valid to have a json_engine_t and
to report an error.
Note this means that invalid json syntax now has a
note on any json error. Raising this to a warning can
cause check constraints of JSON_VALID() to result
in ER_JSON_SYNTAX rather than ER_CONSTRAINT_FAILED
(valided in main.type_json test).
Consequential changes in debug build to json acl
testing validation and unit test having a dummy json_engine_t.
json_normalize also uses the same engine that was
passed as its argument to test json_valid. Because
the character set may have changed, if there's an error
adjust the je->s.c_str pointer. Its only the offset that
is currently used in the error messages anyway so there
may be offset errors depending on the original character
set.
Or max_statement_time limit.
JSON_EQUAL/JSON_NORMALIZE also didn't report warnings
on invalid JSON inputs.
Change the json_normalize function to take a json_engine_t
as a argument and implement the sql/item_jsonfunc.cc calls
to have a json_engine_t.
To the JSON_EQUALS and JSON_NORMALIZE, add error handling
to produce an error when there was one. compare_nested_object,
part of JSON_OVERLAPS, uses a new json_engine and
copies the error/position to the returning function.
json_normalize expects the incoming je structure to have
a valid, or nullptr, killed_ptr, as json_start will reset this.
Issue:
For unknown error numbers, BSD-derived libc (macOS, FreeBSD, ...) formats
"Unknown error: <n>" with a colon, while glibc uses "Unknown error <n>".
my_strerror()'s colon-stripping (MDEV-35578) was gated on __APPLE__ only,
so FreeBSD output diverged from the .result.
Solution:
Normalize by behavior instead of by platform: always strip the colon after
"Unknown error". glibc never emits it, so it is a no-op there, and
macOS/FreeBSD/other BSD libc are all covered without enumeration. This is
what MDEV-35578 already intended ("consistent across the platforms ...
when present"); the __APPLE__ guard was just narrower than that intent.
Runs only in the unknown-error path.
Problem:
my_mb_wc_filename() decodes the my_charset_filename '@HHHH' escape, which
is 5 bytes, but the length guard only required 4 (s[0..3]) before the hex
branch read the 4th hex digit at s[4]. A truncated escape whose end
pointer sits at s+4 (reachable through well_formed_length()/charpos() on a
my_charset_filename string with a tight, non-terminated bound) read one
byte past e.
Fix/Solution:
A 4-byte escape is not valid, so require all 5 bytes up front: s + 5 > e
now returns MY_CS_TOOSMALL5. The read of s[4] keeps the s[3] guard,
because 'e' may be fake (see the strconvert() note above the function):
a NUL-terminated input such as "@00\0" with an over-long end must still
stop at the terminator instead of reading s[4]. Added a strings-t case
covering both the tight-end (MY_CS_TOOSMALL5) and the NUL-terminated
(MY_CS_ILSEQ) truncations.
Problem:
With `div_precision_increment=0`, a division whose quotient rounds
away to zero (e.g. -299/450) produced a decimal with no digit words
(intg=0 and frac=0) while still keeping the sign bit set. That is a
malformed value.
Multiplying such a malformed zero value by another zero reached the
negative-zero check in `decimal_mul`, where there were no digit words
to scan, leading to assertion failure.
Fix:
Fix `do_div_mod`, which produced the malformed zero. `do_div_mod`
already normalizes this case to a canonical zero on the modulo path;
do the same on the division path: when the quotient has no digits
(intg0 = 0 && frac0 = 0) return a correct decimal zero instead of a
digit-less, signed value.
Problem:
When converting a string like '-9223372036854775808' to an integer,
the parsed magnitude (2^63) equals `(ulonglong) LONGLONG_MIN` and is
accepted as valid. The return expression then cast it to signed
(LONGLONG_MIN) and negated it. Negating LONGLONG_MIN is signed integer
overflow, i.e. undefined behaviour.
Fix:
Handle the boundary value explicitly. When the parsed magnitude equals
`(ulonglong) LONGLONG_MIN`, return LONGLONG_MIN directly. Any smaller
magnitude fits in a positive longlong, so the existing cast and
negation stay well defined.
Backport from 11.4 commit - f552febe43
Problem:
When `my_charpos_mb()` is called with pos = end = NULL and the string
has fewer than `length` characters, the `end + 2 - start` return
expression evaluates `end+2`, forming the pointer NULL+2. Offsetting
a null pointer is undefined behavior.
Fix:
Compute the integer difference before adding the offset. The result is
identical but no invalid pointer is ever formed.
Or max_statement_time limit or MAX_DEPTH.
Correct interuptability and depth checking of:
* JSON_CONTAINS
* JSON_CONTAINS_PATH
* JSON_EXISTS
* JSON_EXTRACT
* JSON_KEYS
* ST_GeomFromGeoJSON
Populate in the json library the killed as a specific
error message. As the json library has a killed_ptr
is approprate it handles it.
As json_report_error now processes the killed there's
no need of thd->check_killed().
Verified that the path after every json_scan_next, the
location of checking the killed_ptr, reports errors
correctly.
Backport to 10.6 without JSON_OVERLAPS
The json_engine killed_ptr was of type uchar however the enum
in the server is dependant on the architecture. On big-endian
architectures like IBM Z, retreiving a uchar* retreived the
unmodifed part of the THD->kill enum location and was always 0.
As C++11 allows enums to inherit a type, used uint32_t as the base
class of killed_state type in the server and used uint32_t in the
json library.
reload_acl_and_cache required an expression change to avoid
the compile error:
sql/sql_reload.cc:472:24: error: enumerated and non-enumerated
type in conditional expression [-Werror=enum-conversion]
472 | return result || (thd ? thd->killed : 0);
Added the kill_ptr assignment the following functions can be
killed:
* Item_func_json_depth::val_int
* Item_func_json_type::val_str
* Item_func_json_length::val_int
Item_func_json_array_append::val_str, check_killed() only applied
to json_error: label and not return_null.
Item_func_json_format::val_str(), corresponding to the SQL,
json_compact, json_detailed, json_loose - add debug instrumentation
for the func_json_notembedded test.
MDEV-26726 (fcd345de48) added a json_pause_execution sync point
however the test case used debug_max_statement_time, which doesn't
exist. Fix the sync point name in the test.
The json_engine killed_ptr was of type uchar however the enum
in the server is dependant on the architecture. On big-endian
architectures like IBM Z, retreiving a uchar* retreived the
unmodifed part of the THD->kill enum location and was always 0.
As C++11 allows enums to inherit a type, used uint32_t as the base
class of killed_state type in the server and used uint32_t in the
json library.
reload_acl_and_cache required an expression change to avoid
the compile error:
sql/sql_reload.cc:472:24: error: enumerated and non-enumerated
type in conditional expression [-Werror=enum-conversion]
472 | return result || (thd ? thd->killed : 0);
Added the kill_ptr assignment the following functions can be
killed:
* Item_func_json_depth::val_int
* Item_func_json_type::val_str
* Item_func_json_length::val_int
Item_func_json_array_append::val_str, check_killed() only applied
to json_error: label and not return_null.
Item_func_json_overlaps::val_bool ensure check_killed() is called.
Item_func_json_format::val_str(), corresponding to the SQL,
json_compact, json_detailed, json_loose - add debug instrumentation
for the func_json_notembedded test.
MDEV-26726 (fcd345de48) added a json_pause_execution sync point
however the test case used debug_max_statement_time, which doesn't
exist. Fix the sync point name in the test.
Analysis:
When json is being parsed, the step decreases without a out-of-bound check
resulting in failure.
Fix:
Before decreasing the step, check if it will result into out of bound.
json_get_path_start() set p->last_step to p->steps - 1, creating a
pointer before the beginning of the steps[] array. This is undefined
behavior flagged by UBSAN as "index -1 out of bounds for type
json_path_step_t[32]".
Use NULL as the sentinel value instead, and check for NULL in
json_get_path_next() rather than comparing against p->steps.
Backport to 10.6 of commit 4f546897b8.
Additional test case from Rucha Deodhar
Signed-off-by: Varun Deep Saini <varun.23bcs10048@ms.sst.scaler.com>
Signed-off-by: Varun Deep Saini <deepsainivarun@gmail.com>
json_get_path_start() set p->last_step to p->steps - 1, creating a
pointer before the beginning of the steps[] array. This is undefined
behavior flagged by UBSAN as "index -1 out of bounds for type
json_path_step_t[32]".
Use NULL as the sentinel value instead, and check for NULL in
json_get_path_next() rather than comparing against p->steps.
Signed-off-by: Varun Deep Saini <varun.23bcs10048@ms.sst.scaler.com>
Signed-off-by: Varun Deep Saini <deepsainivarun@gmail.com>
Fix multiplication correctly - by truncating long factors before
multiplication. Not both equally, but in a way that minimizes the
error.
Add more multiplication tests to verify that now multiplication
works correctly.
Revert "MDEV-17256 Decimal field multiplication bug." (57898316b6)
It removes zero truncation from decimal_mul() and fixes
reported symptoms.
But reintroduces multiplication bug.
Remove the unscoped #pragma GCC diagnostic ignored
"-Wdeprecated-declarations" from include/violite.h and replace all
sprintf/vsprintf calls with snprintf/vsnprintf across the codebase.
Where possible, pass actual buffer sizes as function parameters
instead of using hardcoded constants or recomputing strlen().
Notable API changes:
- get_date(): added to_len parameter for buffer size
- nice_time(), end_timer() in client/mysql.cc: added buff_size
- generate_new_name(): added name_size parameter
- calc_md5(): added buffer_size parameter (fixes off-by-one)
- MD5_HASH_TO_STRING macro: added _size parameter
- Added MYSQL_UDF_MAX_RESULT_LENGTH constant in mysql_com.h
Also fixed a pre-existing bug in sql/rpl_mi.cc where the ellipsis
"..." was written to the wrong buffer (dbuff instead of buff).
Vendored code is excluded: extra/readline, extra/wolfssl,
wsrep-lib, libmariadb, storage/mroonga/vendor, zlib.
Normalize -0.0 to +0.0 in Item_func_neg::real_op() to match
binary subtraction behavior. Add test case in type_newdecimal.
add result for tests
fix tests
Revert whitespace changes in type_newdecimal.result
Revert whitespace changes in type_newdecimal.test
normalize all real_op functions in item_func.cc and item_cmpfunc.cc to convert -0.0 to 0.0
normalize all real_op functions in item_func.cc and item_cmpfunc.cc to convert -0.0 to 0.0
normalize all real_op functions in item_func.cc and item_cmpfunc.cc to convert -0.0 to 0.0
normalize all real_op functions in item_func.cc and item_cmpfunc.cc to convert -0.0 to 0.0
normalize -0.0 to 0.0
normalize -0.0 to 0.0
Revert unintended changes in item_func.h
Revert unintended changes in item_func.h (10.6)
fix dtoa.cc and feild.cc and remove past edits
fix buildbot/amd64-ubuntu-2204-debug-ps failed test
fix result which print (-0,0)
rm
fix typo
fix typo
The function json_skip_array_and_count() was trapping syntax errors
(e.g., trailing commas) in a local engine copy. Because the error state
wasn't propagated back to the main engine, json_find_path() would
proceed with an inconsistent state, eventually triggering an assertion
failure.
This patch ensures that any error encountered during the lookahead
scan is propagated to the primary engine. This allows the parser to
fail gracefully with a syntax error instead of crashing.
my_utf16le_uni(), my_lengthsp_utf16le(): Instead of wrongly claiming
aligned access by invoking uint2korr(), inform the compiler of
unaligned access by invoking memcpy(), which will be optimized away.
my_convert(): Correctly identify unaligned access by invoking memcpy(),
which will be translated to a single x86 MOV instruction already by
GCC 4.8.5. Also, process the data 64 bits at a time when possible.
The use of memcpy() prevents GCC from emitting a SIMD instruction that
expects aligned memory (MDEV-37148, MDEV-37786, MDEV-38398) and
allows us to enable the fast path across all ISA.
Reviewed by: Alexander Barkov
Reviewed by: Vladislav Vaintroub
Caused by MDEV-30577 when the CHARSET_INFO structure changed.
As hinted by clang:
error: no member named 'csname' in 'struct charset_info_st'; did you mean 'cs_name'?
Because the target conf_to_src has the properties EXCLUDE_FROM_DEFAULT_BUILD
it hasn't been noticed.
Also correct:
warning: field precision should have type 'int', but argument has type 'size_t'
5 years ago e843033d02 worked around a
MSAN bug when retrieving a double from a va_list.
The construct {{__msan_check_mem_is_initialized(ap,size)}} where ap
is a va_list is undefined as {{__msan_check_mem_is_initialized}}
expects a pointer. The implementation detail of va_list is architecture
dependant and on aarch64 this isn't a pointer.
The need to any msan action is no unnecessary since this has been
corrected in recent clang versions.
As such the additions from e843033d02
have been reverted.
Tested with clang-20.1 and the test cases from MDEV-22690 and
MDEV-22691.
When json_escape changed[1] to return a -1 in the case of
a character that didn't match the character set, json_unescape_to_string
assumed the -1 meant out of memory and just looped with more
memory.
Problem 1 - json_escape needs to return a different code
so that the different between charset incompatibility and out
of memory needs to occur. This enables json_escape_to_string
to handle the it correctly (ignore and fail seems the best
option).
Problem 2 - JSON histograms need to support character with
where the column json min/maximum value aren't a character
set represented by a single byte.
Problem 2 was previously hidden as ? was a result of the conversion.
As JSON histograms can relate to columns when have an explict
character set, use that and fall back to bin which was the
previous default for non-string columns.
Replaces -1/-2 constants and handling with JSON_ERROR_ILLEGAL_SYMBOL /
JSON_ERROR_OUT_OF_SPACE defines.
[1] regression from: f699010c0f
New warnings come from 3 places
1. Warning C5287: Warning comes from json_lib.c from code like
compile_time_assert((int) JSON_VALUE_NULL == (int) JSV_NULL);
2. Warning C5287: Similar warning come from wc_static_assert() from code
in wolfSSL's header file
3. Warning C5286 in WolfSSL code, -enum_value
(i.e multiplying enum with -1)is used
To fix:
- Disable warnings in WolfSSL code, using /wd<num> flag.
- workaround warning for users of WolfSSL, disable
wc_static_assert() with -DWC_NO_STATIC_ASSERT compile flag
- Rewrite some compile_time_assert in json_lib.c to avoid warning.
- add target_link_libraries(vio ${SSL_LIBRARIES}) so that
vio picks up -DWC_NO_STATIC_ASSERT
in _ma_unique_hash, skip_trailing_space, my_hash_sort_mb_nopad_bin and my_strnncollsp_utf8mb4_bin
UBSAN detected the nullptr-with-offset in a few places
when handling empty blobs.
Fix:
- Adding DBUG_ASSERT(source_string) into all hash_sort() implementations
to catch this problem in non-UBSAN debug builds.
- Fixing mi_unique_hash(), mi_unique_comp(),
_ma_unique_hash(), _ma_unique_comp() to replace NULL pointer to
an empty string ponter..
Note, we should also add DBUG_ASSERT(source_string != NULL) into
all implementations of strnncoll*(). But I'm afraid the patch
is going to be too long and too dangerous for 10.5.
