Analysis:
Parsing json path happens only once. When paring, we set types of path
(types_used) to use later. If the path type has range or wild card, only
then multiple values get added to the result set.
However for each row in the table, types_used still gets
overwritten to default (no multiple values) and is also not set again
(because path is already parsed). Since multiple values depend on the
type of path, they dont get added to result set either.
Fix:
set default for types_used only if path is not parsed already.
procedures
MDEV-22224 caused the parsing of keys with hyphens to break by setting
the state transitions for parsing keys to JE_SYN (syntax error) when
they encounter a hyphen. However json key names may contain hyphens and
still be considered valid json.
This patch changes the state transition table so that key names with
hyphens remain valid. Note that unquoted key names in paths like
$.key-name are also valid again. This restores the previous behaviour
when hyphens were considered part of the P_ETC character class.
Tests with checking metadata or that cannot be run with
the view-protocol are excluded from --view-protocol.
For tests that do not allow the use of an additional connection,
the util connection is disabled with "--disable_service_connection".
Also cases with bugs for --view-protocol are disabled.
Analysis: The JSON functions(JSON_ARRAY[OBJECT|ARRAY_APPEND|ARRAY_INSERT|INSERT|SET|REPLACE]) result is truncated when the function is called based on LONGTEXT field. The overflow occurs when computing the result length due to the LONGTEXT max length is same as uint32 max length. It lead to wrong result length.
Fix: Add static_cast<ulonglong> to avoid uint32 overflow and fix the arguments used.
Analysis: JSON_VALUE() returns "null" string instead of NULL pointer.
Fix: When the type is JSON_VALUE_NULL (which is also a scalar) set
null_value to true and return 0 instead of returning string.
Analysis: JSON_OVERLAPS() does not check nested key-value pair completely.
If there is nested object, then it only scans and validates if two json values
overlap until one of the value (which is of type object) is exhausted.
This does not really check if the two values of keys are exacly the same, instead
it only checks if key-value pair of one is present in key-value pair of the
other
Fix: Normalize the values (which are of type object) and compare
using string compare. This will validate if two values
are exactly the same.
Analysis: When trying to compare json paths, the array_sizes variable is
NULL when beginning. But trying to access address by adding to the NULL
pointer while recursive calling json_path_parts_compare() for handling
double wildcard, it causes undefined behaviour and the array_sizes
variable eventually becomes non-null (has some address).
This eventually results in crash.
Fix: If array_sizes variable is NULL then pass NULL recursively as well.
path (when range is used)
Analysis: When 0 comes after space, then the json path parser changes the
state to JE_SYN instead of PS_Z (meaning parse zero). Hence the warning.
Fix: Make the state PS_Z instead of JE_SYN.
json path
Analysis: When searching for the given path in json string, if the current
step is of array range type, then path was considered reached which meant
path exists. So output was always true. The end indexes of range were not
evaluated.
Fix: If the current step type for a path is array range, then check if the
value array_counter[] is in range of n_item and n_item_end. If it is, then
path exists. Only then return true. If the range criteria is never met
then return false.
Range can be thought about in similar manner as wildcard (*) where
more than one elements are processed. To implement range notation, extended
json parser to parse the 'to' keyword and added JSON_PATH_ARRAY_RANGE for
path type. If there is 'to' keyword then use JSON_PATH_ARRAY range for
path type along with existing type.
This new integer to store the end index of range is n_item_end.
When there is 'to' keyword, store the integer in n_item_end else store in
n_item.
JSON Path
Analysis: When we have '-' followed by 0, then the state is
changed to JE_SYN, meaning syntax error.
Fix: Change the state to PS_INT instead, because we are
reading '0' next (integer) and it is not a syntax error.
This patch can be viewed as combination of two parts:
1) Enabling '-' in the path so that the parser does not give out a warning.
2) Setting the negative index to a correct value and returning the
appropriate value.
1) To enable using the negative index in the path:
To make the parser not return warning when negative index is used in path
'-' needs to be allowed in json path characters. P_NEG is added
to enable this and is made recognizable by setting the 45th index of
json_path_chr_map[] to P_NEG (instead of previous P_ETC)
because 45 corresponds to '-' in unicode.
When the path is being parsed and '-' is encountered, the parser should
recognize it as parsing '-' sign, so a new json state PS_NEG is required.
When the state is PS_NEG, it means that a negative integer is
going to be parsed so set is_negative_index of current step to 1 and
n_item is set accordingly when integer is encountered after '-'.
Next proceed with parsing rest of the path and get the correct path.
Next thing is parsing the json and returning correct value.
2) Setting the negative index to a correct value and returning the value:
While parsing json if we encounter array and the path step for the array
is a negative index (n_item < 0), then we can count the number of elements
in the array and set n_item to correct corresponding value. This is done in
json_skip_array_and_count.
Analysis: In case of error while processing json document, we goto
error label which eventually return 1 instead of 0.
Fix: Return 0 in case of error instead of 1.
1) When at least one of the two json documents is of scalar type:
1.a) If value and json document both are scalar, then return true
if they have same type and value.
1.b) If json document is scalar but other is array (or vice versa),
then return true if array has at least one element of same type
and value as scalar.
1.c) If one is scalar and other is object, then return false because
it can't be compared.
2) When both arguments are of non-scalar type and below conditons
are satisfied then return true:
2.a) When both arguments are arrays:
Iterate over the value and json document. If there exists at
least one element in other array of same type and value as
that of element in value.
2.b) If both arguments are objects:
Iterate over value and json document and if there exists at least
one key-value pair common between two objects.
2.c) If either of json document or value is array and other is object:
Iterate over the array, if an element of type object is found,
then compare it with the object (which is the other arguemnt).
If the entire object matches i.e all they key value pairs match.
Hybrid functions (IF, COALESCE, etc) did not preserve the JSON property
from their arguments. The same problem was repeatable for single row subselects.
The problem happened because the method Item::is_json_type() was inconsistently
implemented across the Item hierarchy. For example, Item_hybrid_func
and Item_singlerow_subselect did not override is_json_type().
Solution:
- Removing Item::is_json_type()
- Implementing specific JSON type handlers:
Type_handler_string_json
Type_handler_varchar_json
Type_handler_tiny_blob_json
Type_handler_blob_json
Type_handler_medium_blob_json
Type_handler_long_blob_json
- Reusing the existing data type infrastructure to pass JSON
type handlers across all item types, including classes Item_hybrid_func
and Item_singlerow_subselect. Note, these two classes themselves do not
need any changes!
- Extending the data type infrastructure so data types can inherit
their properties (e.g. aggregation rules) from their base data types.
E.g. VARCHAR/JSON acts as VARCHAR, LONGTEXT/JSON acts as LONGTEXT
when mixed to a non-JSON data type. This is done by:
- adding virtual method Type_handler::type_handler_base()
- adding a helper class Type_handler_pair
- refactoring Type_handler_hybrid_field_type methods
aggregate_for_result(), aggregate_for_min_max(),
aggregate_for_num_op() to use Type_handler_pair.
This change also fixes:
MDEV-27361 Hybrid functions with JSON arguments do not send format metadata
Also, adding mtr tests for JSON replication. It was not covered yet.
And the current patch changes the replication code slightly.
