Per ISO/IEC 9075-2:2023, 7.11 <JSON table>, Syntax Rules 1)e)iv) and
1)f)xi), a regular or formatted JSON_TABLE column that does not specify
its own ON ERROR clause takes its default error behavior from the
table-level ON ERROR clause: with ERROR ON ERROR on the table, the
column behaves as ERROR ON ERROR; otherwise the column defaults to
NULL ON ERROR.
PostgreSQL instead always defaulted such columns to NULL ON ERROR, so
SELECT * FROM JSON_TABLE(jsonb '"err"', '$'
COLUMNS (a int PATH '$') ERROR ON ERROR) jt;
returned a NULL row where the standard requires an error. The
documentation stated the divergence as if it were a rule, saying that
the table-level clause "does not affect the errors that occur when
evaluating columns".
Implement the standard behavior in the JSON_TABLE syntactic
transformation: when a column lacks its own ON ERROR clause and the
table-level behavior is ERROR ON ERROR, synthesize an implicit ERROR ON
ERROR for the column before it is transformed into a JsonExpr. A column
with its own ON ERROR clause is unaffected. EXISTS columns are not
covered by those syntax rules, so they keep their FALSE ON ERROR
default. A JSON_TABLE stored in a view is now deparsed with the
previously implicit ERROR ON ERROR shown explicitly on the affected
columns, which is a semantically equivalent, round-trip-stable form.
86ab7f4c72 introduced this cascade as part of the PLAN clause and
af6fad879f reverted it, on the grounds that it should be a deliberate
and separately documented change rather than a side effect of an
unrelated feature. This is that change.
Do not back-patch. A query that specifies ERROR ON ERROR at the table
level and relies on its columns still yielding NULL now raises an error
instead of returning rows. Nothing is silently wrong in the released
branches: the table-level clause is ignored for columns consistently,
and that is what the documentation has promised since PostgreSQL 17, so
users could reasonably have depended on it.
Discussion: https://postgr.es/m/CAPpHfdtDXseGhrL14a2asOkwGnFTVrDk8SQc3iWZgWEEhXXMGw%40mail.gmail.com
Reviewed-by: Nikita Malakhov <hukutoc@gmail.com>
generateJsonTablePathName() produced names of the form
"json_table_path_N" in the same namespace as user-supplied path and
column names, without checking whether the name was already in use.
When an unnamed NESTED path's generated name happened to match a
user-supplied path name that a PLAN clause referenced, two sibling paths
matched the same plan entry and one of them, together with its columns,
was silently dropped from the output.
Bump the counter until the generated name is unused, so a generated name
can no longer coincide with a user-supplied one. The still-uncovered
path is then correctly reported as not found in the plan.
The row pattern (root) path is named before the user-supplied column and
path names are collected, so when it is left unnamed its generated name
could not avoid them either, and a user column or path named like a
generated name, e.g.
SELECT * FROM JSON_TABLE(jsonb '1', '$'
COLUMNS (json_table_path_0 int PATH '$')) jt;
was rejected with a bogus "duplicate JSON_TABLE column or path name"
error. Collect the user-supplied names first and generate the row
pattern path's name afterwards, so that it avoids all of them. An
explicit root path name is still seeded into the namespace, so a column
duplicating it is still correctly rejected.
Reported-by: Thom Brown <thom@linux.com>
Discussion: https://postgr.es/m/CAA-aLv7aZGSExnbjJRw8eKkoXbu34TdoKLLA2gPye3aHjO5OSA@mail.gmail.com
Discussion: https://postgr.es/m/CAA-aLv5U94KD4C%2BLhAPYcCeGvs1xBMngcS5oEkZHN9YWwXUHsA%40mail.gmail.com
get_json_table_plan() parenthesized the child of a parent/child
(OUTER/INNER) plan only when that child was a sibling (UNION/CROSS)
join, not when it was itself a parent/child join. A plan such as
PLAN (p0 OUTER (p1 INNER p11)) was therefore deparsed as
PLAN (p0 OUTER p1 INNER p11), which does not parse back to the same
plan tree -- a dump/restore hazard. Parenthesize the child whenever it
is not a bare path name, matching the logic already used for the
operands of sibling joins.
Reported-by: Thom Brown <thom@linux.com>
Discussion: https://postgr.es/m/CAA-aLv7aZGSExnbjJRw8eKkoXbu34TdoKLLA2gPye3aHjO5OSA@mail.gmail.com
86ab7f4c72 commit made the table-level ON ERROR clause serve as the default
ON ERROR for columns lacking their own, so that a top-level ERROR ON ERROR
turned per-column evaluation errors into hard errors.
The SQL standard does mandate this cascade, but introducing it should be
a deliberate, separately-documented change, so restore the previous
behavior for now. This also reverts the paired ruleutils.c logic that
deparsed a column's behavior against an ERROR default: that dropped an
explicit ERROR ON EMPTY from a dumped view, and otherwise emitted a
redundant NULL ON EMPTY.
Reported-by: Thom Brown <thom@linux.com>
Discussion: https://postgr.es/m/CAA-aLv7aZGSExnbjJRw8eKkoXbu34TdoKLLA2gPye3aHjO5OSA@mail.gmail.com
transformJsonBehavior() coerced an ON EMPTY / ON ERROR DEFAULT
expression only when its type differed from the RETURNING type's OID.
When the base type matched but the RETURNING type carried a type
modifier (e.g. numeric(4,1) or varchar(3)), the coercion that enforces
the typmod was skipped, so the DEFAULT value could violate the
declared type:
SELECT JSON_VALUE(jsonb '{}', '$.a'
RETURNING numeric(4,1) DEFAULT 99999.999 ON EMPTY);
returned 99999.999, which 99999.999::numeric(4,1) would reject; the
value could even be stored into a numeric(4,1) column, as later
coercions trust its already-correct type label.
Fix by also coercing when the RETURNING type has a typmod, except for
a NULL constant. coerce_to_target_type() is a no-op when the typmod
already matches. The matching-OID short-circuit dates to 74c96699be.
Reported-by: Ewan Young <kdbase.hack@gmail.com>
Author: Ewan Young <kdbase.hack@gmail.com>
Discussion: https://postgr.es/m/CAON2xHPO9f4cAmyGn1mQ=VqoS7wN5rz4yOiqudxX78zninZpCw@mail.gmail.com
Backpatch-through: 17
The deparsing code in get_json_expr_options() unnecessarily emitted
the default column-specific ON ERROR / EMPTY behavior when the
top-level ON ERROR behavior in JSON_TABLE was set to ERROR. Fix that
by not overriding the column-specific default, determined based on
the column's JsonExprOp in get_json_table_columns(), with
JSON_BEHAVIOR_ERROR when that is the top-level ON ERROR behavior.
Note that this only removes redundancy; the current deparsing output
is not incorrect, just redundant.
Reviewed-by: Jian He <jian.universality@gmail.com>
Discussion: https://postgr.es/m/CACJufxEo4sUjKCYtda0_qt9tazqqKPmF1cqhW9KBOUeJFqQd2g@mail.gmail.com
Backpatch-through: 17
The deparsing code in get_json_expr_options() unnecessarily emitted
the default column-specific ON ERROR / EMPTY behavior when the
top-level ON ERROR behavior in JSON_TABLE was set to ERROR. Fix that
by not overriding the column-specific default, determined based on
the column's JsonExprOp in get_json_table_columns(), with
JSON_BEHAVIOR_ERROR when that is the top-level ON ERROR behavior.
Note that this only removes redundancy; the current deparsing output
is not incorrect, just redundant.
Reviewed-by: Jian He <jian.universality@gmail.com>
Discussion: https://postgr.es/m/CACJufxEo4sUjKCYtda0_qt9tazqqKPmF1cqhW9KBOUeJFqQd2g@mail.gmail.com
Backpatch-through: 17
The current method of coercing the boolean result value of
JsonPathExists() to the target type specified for an EXISTS column,
which is to call the type's input function via json_populate_type(),
leads to an error when the target type is integer, because the
integer input function doesn't recognize boolean literal values as
valid.
Instead use the boolean-to-integer cast function for coercion in that
case so that using integer or domains thereof as type for EXISTS
columns works. Note that coercion for ON ERROR values TRUE and FALSE
already works like that because the parser creates a cast expression
including the cast function, but the coercion of the actual result
value is not handled by the parser.
Tests by Jian He.
Reported-by: Jian He <jian.universality@gmail.com>
Author: Jian He <jian.universality@gmail.com>
Author: Amit Langote <amitlangote09@gmail.com>
Discussion: https://postgr.es/m/CACJufxEo4sUjKCYtda0_qt9tazqqKPmF1cqhW9KBOUeJFqQd2g@mail.gmail.com
Backpatch-through: 17
1. Remove the special case handling when casting the JsonBehavior
expressions to types with typmod, like 86d33987 did for the casting
of SQL/JSON constructor functions.
2. Fix casting for fixed-length character and bit string types by
using assignment-level casts. This is again similar to what
86d33987 did, but for ON ERROR / EMPTY expressions.
3. Use runtime coercion for the boolean ON ERROR constants so that
using fixed-length character string types, for example, for an
EXISTS column doesn't cause a "value too long for type
character(n)" when the parser tries to coerce the default ON ERROR
value "false" to that type, that is, even when clause is not
specified.
4. Simplify the conditions of when to use runtime coercion vs
creating the cast expression in the parser itself. jsonb-valued
expressions are now always coerced at runtime and boolean
expressions too if the target type is a string type for the
reasons mentioned above.
Tests are taken from a patch that Jian He posted.
Reported-by: Jian He <jian.universality@gmail.com>
Author: Jian He <jian.universality@gmail.com>
Author: Amit Langote <amitlangote09@gmail.com>
Discussion: https://postgr.es/m/CACJufxEo4sUjKCYtda0_qt9tazqqKPmF1cqhW9KBOUeJFqQd2g@mail.gmail.com
Backpatch-through: 17
Instead of looking up casts at parse time for converting the result
of JsonPath* query functions to the specified or the default
RETURNING type, always perform the conversion at runtime using either
the target type's input function or the function
json_populate_type().
There are two motivations for this change:
1. json_populate_type() coerces to types with typmod such that any
string values that exceed length limit cause an error instead of
silent truncation, which is necessary to be standard-conforming.
2. It was possible to end up with a cast expression that doesn't
support soft handling of errors causing bugs in the of handling
ON ERROR clause.
JsonExpr.coercion_expr which would store the cast expression is no
longer necessary, so remove.
Bump catversion because stored rules change because of the above
removal.
Reported-by: Alvaro Herrera <alvherre@alvh.no-ip.org>
Reviewed-by: Jian He <jian.universality@gmail.com>
Discussion: Discussion: https://postgr.es/m/202405271326.5a5rprki64aw%40alvherre.pgsql
Currently, the grammar allows any supported values in the ON ERROR
and ON EMPTY clauses for SQL/JSON functions, regardless of whether
the values are appropriate for the function. This commit ensures
that during parse analysis, the provided value is checked for
validity for the given function and throws a syntax error if it is
not.
While at it, this fixes some omissions in the documentation of the
ON ERROR/EMPTY clauses for JSON_TABLE().
Reported-by: Jian He <jian.universality@gmail.com>
Reviewed-by: Jian He <jian.universality@gmail.com>
Discussion: https://postgr.es/m/CACJufxFgWGqpESSYzyJ6tSurr3vFYBSNEmCfkGyB_dMdptFnZQ%40mail.gmail.com
Currently, when the ON EMPTY clause is not present, the ON ERROR
clause (implicit or explicit) dictates the behavior when jsonpath
evaluation in ExecEvalJsonExprPath() results in an empty sequence.
That is an oversight in the commit 6185c9737c.
This commit fixes things so that a NULL is returned instead in that
case which is the default behavior when the ON EMPTY clause is not
present.
Reported-by: Markus Winand
Discussion: https://postgr.es/m/F7DD1442-265C-4220-A603-CB0DEB77E91D%40winand.at
SQL/JSON query functions allow specifying an expression to return
when either of ON ERROR or ON EMPTY condition occurs when evaluating
the JSON path expression. The parser (transformJsonBehavior()) checks
that the specified expression is one of the supported expressions, but
there are two issues with how the check is done that are fixed in this
commit:
* No check for some expressions related to coercion, such as
CoerceViaIO, that may appear in the transformed user-specified
expressions that include cast(s)
* An unsupported expression may be masked by a coercion-related
expression, which must be flagged by checking the latter's
argument expression recursively
Author: Jian He <jian.universality@gmail.com>
Author: Amit Langote <amitlangote09@gmail.com>
Reported-by: Jian He <jian.universality@gmail.com>
Discussion: https://postgr.es/m/CACJufxEqhqsfrg_p7EMyo5zak3d767iFDL8vz_4%3DZBHpOtrghw@mail.gmail.com
Discussion: https://postgr.es/m/CACJufxGOerH1QJknm1noh-Kz5FqU4p7QfeZSeVT2tN_4SLXYNg@mail.gmail.com
This improves some error messages emitted by SQL/JSON query functions
by mentioning column name when available, such as when they are
invoked as part of evaluating JSON_TABLE() columns. To do so, a new
field column_name is added to both JsonFuncExpr and JsonExpr that is
only populated when creating those nodes for transformed JSON_TABLE()
columns.
While at it, relevant error messages are reworded for clarity.
Reported-by: Jian He <jian.universality@gmail.com>
Suggested-by: Jian He <jian.universality@gmail.com>
Discussion: https://postgr.es/m/CACJufxG_e0QLCgaELrr2ZNz7AxPeGCNKAORe3fHtFCQLsH4J4Q@mail.gmail.com
Currently, transformJsonFuncExpr() enforces some restrictions on
the combinations of QUOTES and WRAPPER clauses that can be specified
in JSON_QUERY(). The intent was to only prevent the useless
combination WITH WRAPPER OMIT QUOTES, but the coding prevented KEEP
QUOTES too, which is not helpful. Fix that.
JSON_TABLE() allows JSON data to be converted into a relational view
and thus used, for example, in a FROM clause, like other tabular
data. Data to show in the view is selected from a source JSON object
using a JSON path expression to get a sequence of JSON objects that's
called a "row pattern", which becomes the source to compute the
SQL/JSON values that populate the view's output columns. Column
values themselves are computed using JSON path expressions applied to
each of the JSON objects comprising the "row pattern", for which the
SQL/JSON query functions added in 6185c9737c are used.
To implement JSON_TABLE() as a table function, this augments the
TableFunc and TableFuncScanState nodes that are currently used to
support XMLTABLE() with some JSON_TABLE()-specific fields.
Note that the JSON_TABLE() spec includes NESTED COLUMNS and PLAN
clauses, which are required to provide more flexibility to extract
data out of nested JSON objects, but they are not implemented here
to keep this commit of manageable size.
Author: Nikita Glukhov <n.gluhov@postgrespro.ru>
Author: Teodor Sigaev <teodor@sigaev.ru>
Author: Oleg Bartunov <obartunov@gmail.com>
Author: Alexander Korotkov <aekorotkov@gmail.com>
Author: Andrew Dunstan <andrew@dunslane.net>
Author: Amit Langote <amitlangote09@gmail.com>
Author: Jian He <jian.universality@gmail.com>
Reviewers have included (in no particular order):
Andres Freund, Alexander Korotkov, Pavel Stehule, Andrew Alsup,
Erik Rijkers, Zihong Yu, Himanshu Upadhyaya, Daniel Gustafsson,
Justin Pryzby, Álvaro Herrera, Jian He
Discussion: https://postgr.es/m/cd0bb935-0158-78a7-08b5-904886deac4b@postgrespro.ru
Discussion: https://postgr.es/m/20220616233130.rparivafipt6doj3@alap3.anarazel.de
Discussion: https://postgr.es/m/abd9b83b-aa66-f230-3d6d-734817f0995d%40postgresql.org
Discussion: https://postgr.es/m/CA+HiwqE4XTdfb1nW=Ojoy_tQSRhYt-q_kb6i5d4xcKyrLC1Nbg@mail.gmail.com
