`json_normalize`/`json_equals` unnecessary
double parsing JSON document
Removed the redundant `json_valid_engine` pre-check to eliminate
double-parsing, and integrated error handling directly into the
normalization engine:
* Catch Empty Strings: Return an error directly in
`json_normalize_engine` if the root type is `JSON_VALUE_UNINITIALIZED`.
* Catch Trailing Garbage on Scalars: Updated `json_norm_build`
to enforce a full scan to the end of the document for scalar values.
This prevents edge cases (e.g., raw date strings like 2026-07-17...)
from being falsely accepted as a valid JSON number without checking
the remainder of the string for syntax errors.
* Propagate Syntax Errors: Updated `json_normalize_engine` to
explicitly check the engine's error flag after the build phase.
* Add Edge Case Tests: Added tests in `json_normalize.test` for
empty strings, whitespace, and trailing garbage on scalars to
ensure correct error generation.
Testing:
Verified locally using MTR
(`main.json_normalize` and `main.json_equals`).
Signed-off-by: VasuBhakt <cpswastik31@gmail.com>
Despite the JSON spec rfc8259/#section-4
"The names within an object SHOULD be unique"[1], it does happen.
As our sort algorithm has access to both key and value,
if the key matches, sort by the value. The type, and length of
objects, strings, arrays and numbers is easy, so use those as a
first comparision point.
[1] https://www.rfc-editor.org/info/rfc8259/#section-4
a90779098b changed the
interface to the json_valid to take a json_engine_t
so that it could be interupted and so that warnings
could become visible.
On merge from that 10.11 to 11.4 it was discovered
that the ColumnStore engine uses this interface.
Restored the original json_valid function exactly
how ColumnStore expects this and added renamed
the function with json_engine_t to be json_valid_engine.
This follows the same convention as in the commit
756fc6fd9a that was
for json_normalize, that ColumnStore also used.
14c16e02b2 changed the
interface for json_normalize to take a json_engine_t so that
there could be some error checking.
Unfortunately ColumnStore depended on this interface.
Restored original interface of json_normalize() and
added a json_normalize_engine() that take as engine as the
argument.
Or max_statement_time limit.
Change the json_valid to take a json_engine_t argument.
Adjust the Item_json_valid to have a json_engine_t and
to report an error.
Note this means that invalid json syntax now has a
note on any json error. Raising this to a warning can
cause check constraints of JSON_VALID() to result
in ER_JSON_SYNTAX rather than ER_CONSTRAINT_FAILED
(valided in main.type_json test).
Consequential changes in debug build to json acl
testing validation and unit test having a dummy json_engine_t.
json_normalize also uses the same engine that was
passed as its argument to test json_valid. Because
the character set may have changed, if there's an error
adjust the je->s.c_str pointer. Its only the offset that
is currently used in the error messages anyway so there
may be offset errors depending on the original character
set.
Or max_statement_time limit.
JSON_EQUAL/JSON_NORMALIZE also didn't report warnings
on invalid JSON inputs.
Change the json_normalize function to take a json_engine_t
as a argument and implement the sql/item_jsonfunc.cc calls
to have a json_engine_t.
To the JSON_EQUALS and JSON_NORMALIZE, add error handling
to produce an error when there was one. compare_nested_object,
part of JSON_OVERLAPS, uses a new json_engine and
copies the error/position to the returning function.
json_normalize expects the incoming je structure to have
a valid, or nullptr, killed_ptr, as json_start will reset this.
json_normalize_number(): Avoid accessing str past str_len.
The function would seem to work incorrectly when some digits are
not followed by a decimal point (.) or an exponent (E or e).
https://jira.mariadb.org/browse/MDEV-26221
my_sys DYNAMIC_ARRAY and DYNAMIC_STRING inconsistancy
The DYNAMIC_STRING uses size_t for sizes, but DYNAMIC_ARRAY used uint.
This patch adjusts DYNAMIC_ARRAY to use size_t like DYNAMIC_STRING.
As the MY_DIR member number_of_files is copied from a DYNAMIC_ARRAY,
this is changed to be size_t.
As MY_TMPDIR members 'cur' and 'max' are copied from a DYNAMIC_ARRAY,
these are also changed to be size_t.
The lists of plugins and stored procedures use DYNAMIC_ARRAY,
but their APIs assume a size of 'uint'; these are unchanged.
This patch implements a library for normalizing json documents.
The algorithm is:
* Recursively sort json keys according to utf8mb4_bin collation.
* Normalize numbers to be of the form [-]<digit>.<frac>E<exponent>
* All unneeded whitespace and line endings are removed.
* Arrays are not sorted.
Co-authored-by: Vicențiu Ciorbaru <vicentiu@mariadb.org>
