database/postgres
1
0
Fork 0
mirror of https://github.com/postgres/postgres.git synced 2025-07-28 23:42:10 +03:00
Code Activity

Allow Unicode escapes in any server encoding, not only UTF-8.

Browse Source 
SQL includes provisions for numeric Unicode escapes in string
literals and identifiers.  Previously we only accepted those
if they represented ASCII characters or the server encoding
was UTF-8, making the conversion to internal form trivial.
This patch adjusts things so that we'll call the appropriate
encoding conversion function in less-trivial cases, allowing
the escape sequence to be accepted so long as it corresponds
to some character available in the server encoding.

This also applies to processing of Unicode escapes in JSONB.
However, the old restriction still applies to client-side
JSON processing, since that hasn't got access to the server's
encoding conversion infrastructure.

This patch includes some lexer infrastructure that simplifies
throwing errors with error cursors pointing into the middle of
a string (or other complex token).  For the moment I only used
it for errors relating to Unicode escapes, but we might later
expand the usage to some other cases.

Patch by me, reviewed by John Naylor.

Discussion: https://postgr.es/m/2393.1578958316@sss.pgh.pa.us
This commit is contained in:
Tom Lane
2020-03-06 14:17:43 -05:00
parent fe30e7ebfa
commit a6525588b7
20 changed files with 613 additions and 227 deletions
Download Patch File Download Diff File Expand all files Collapse all files

100
doc/src/sgml/syntax.sgml
View File

@ -189,6 +189,23 @@ UPDATE "my_table" SET "a" = 5;
ampersands. The length limitation still applies.
</para>
<para>
Quoting an identifier also makes it case-sensitive, whereas
unquoted names are always folded to lower case. For example, the
identifiers <literal>FOO</literal>, <literal>foo</literal>, and
<literal>"foo"</literal> are considered the same by
<productname>PostgreSQL</productname>, but
<literal>"Foo"</literal> and <literal>"FOO"</literal> are
different from these three and each other. (The folding of
unquoted names to lower case in <productname>PostgreSQL</productname> is
incompatible with the SQL standard, which says that unquoted names
should be folded to upper case. Thus, <literal>foo</literal>
should be equivalent to <literal>"FOO"</literal> not
<literal>"foo"</literal> according to the standard. If you want
to write portable applications you are advised to always quote a
particular name or never quote it.)
</para>
<indexterm>
<primary>Unicode escape</primary>
<secondary>in identifiers</secondary>
@ -230,7 +247,8 @@ U&amp;"d!0061t!+000061" UESCAPE '!'
The escape character can be any single character other than a
hexadecimal digit, the plus sign, a single quote, a double quote,
or a whitespace character. Note that the escape character is
written in single quotes, not double quotes.
written in single quotes, not double quotes,
after <literal>UESCAPE</literal>.
</para>
<para>
@ -239,32 +257,18 @@ U&amp;"d!0061t!+000061" UESCAPE '!'
</para>
<para>
The Unicode escape syntax works only when the server encoding is
<literal>UTF8</literal>. When other server encodings are used, only code
points in the ASCII range (up to <literal>\007F</literal>) can be
specified. Both the 4-digit and the 6-digit form can be used to
Either the 4-digit or the 6-digit escape form can be used to
specify UTF-16 surrogate pairs to compose characters with code
points larger than U+FFFF, although the availability of the
6-digit form technically makes this unnecessary. (Surrogate
pairs are not stored directly, but combined into a single
code point that is then encoded in UTF-8.)
pairs are not stored directly, but are combined into a single
code point.)
</para>
<para>
Quoting an identifier also makes it case-sensitive, whereas
unquoted names are always folded to lower case. For example, the
identifiers <literal>FOO</literal>, <literal>foo</literal>, and
<literal>"foo"</literal> are considered the same by
<productname>PostgreSQL</productname>, but
<literal>"Foo"</literal> and <literal>"FOO"</literal> are
different from these three and each other. (The folding of
unquoted names to lower case in <productname>PostgreSQL</productname> is
incompatible with the SQL standard, which says that unquoted names
should be folded to upper case. Thus, <literal>foo</literal>
should be equivalent to <literal>"FOO"</literal> not
<literal>"foo"</literal> according to the standard. If you want
to write portable applications you are advised to always quote a
particular name or never quote it.)
If the server encoding is not UTF-8, the Unicode code point identified
by one of these escape sequences is converted to the actual server
encoding; an error is reported if that's not possible.
</para>
</sect2>
@ -427,25 +431,11 @@ SELECT 'foo' 'bar';
<para>
It is your responsibility that the byte sequences you create,
especially when using the octal or hexadecimal escapes, compose
valid characters in the server character set encoding. When the
server encoding is UTF-8, then the Unicode escapes or the
valid characters in the server character set encoding.
A useful alternative is to use Unicode escapes or the
alternative Unicode escape syntax, explained
in <xref linkend="sql-syntax-strings-uescape"/>, should be used
instead. (The alternative would be doing the UTF-8 encoding by
hand and writing out the bytes, which would be very cumbersome.)
</para>
<para>
The Unicode escape syntax works fully only when the server
encoding is <literal>UTF8</literal>. When other server encodings are
used, only code points in the ASCII range (up
to <literal>\u007F</literal>) can be specified. Both the 4-digit and
the 8-digit form can be used to specify UTF-16 surrogate pairs to
compose characters with code points larger than U+FFFF, although
the availability of the 8-digit form technically makes this
unnecessary. (When surrogate pairs are used when the server
encoding is <literal>UTF8</literal>, they are first combined into a
single code point that is then encoded in UTF-8.)
in <xref linkend="sql-syntax-strings-uescape"/>; then the server
will check that the character conversion is possible.
</para>
<caution>
@ -524,16 +514,23 @@ U&amp;'d!0061t!+000061' UESCAPE '!'
</para>
<para>
The Unicode escape syntax works only when the server encoding is
<literal>UTF8</literal>. When other server encodings are used, only
code points in the ASCII range (up to <literal>\007F</literal>)
can be specified. Both the 4-digit and the 6-digit form can be
used to specify UTF-16 surrogate pairs to compose characters with
code points larger than U+FFFF, although the availability of the
6-digit form technically makes this unnecessary. (When surrogate
pairs are used when the server encoding is <literal>UTF8</literal>, they
are first combined into a single code point that is then encoded
in UTF-8.)
To include the escape character in the string literally, write
it twice.
</para>
<para>
Either the 4-digit or the 6-digit escape form can be used to
specify UTF-16 surrogate pairs to compose characters with code
points larger than U+FFFF, although the availability of the
6-digit form technically makes this unnecessary. (Surrogate
pairs are not stored directly, but are combined into a single
code point.)
</para>
<para>
If the server encoding is not UTF-8, the Unicode code point identified
by one of these escape sequences is converted to the actual server
encoding; an error is reported if that's not possible.
</para>
<para>
@ -546,11 +543,6 @@ U&amp;'d!0061t!+000061' UESCAPE '!'
parameter is set to off, this syntax will be rejected with an
error message.
</para>
<para>
To include the escape character in the string literally, write it
twice.
</para>
</sect3>
<sect3 id="sql-syntax-dollar-quoting">