database/mariadb
1
0
Fork 0
mirror of https://github.com/MariaDB/server.git synced 2026-10-23 18:49:29 +03:00
Code Activity
Files
12.3
mariadb/strings/conf_to_src.c
Alexander Barkov 8c4943d208 MDEV-41246 "Illegal mix of collations" on the mysql.user view 
This SQL script failed:

SET NAMES latin1 COLLATE latin1_swedish_ci;
CREATE OR REPLACE VIEW v1 AS SELECT 'Y' AS c1;
SET NAMES big5 COLLATE big5_chinese_ci;
SELECT * FROM v1 WHERE c1='y';

with the following error:

ERROR 1267 (HY000): Illegal mix of collations
(latin1_swedish_ci,COERCIBLE) and (big5_chinese_ci,COERCIBLE)
for operation '='

Note, latin1_swedish_ci and big5_chinese_ci are used here as examples.
The error also happened with different collation combinations.

Fix main idea:

If two collations have equal comparison rules (known as "tailoring")
on a given character repertoire,
like latin1_swedish_ci and big5_chinese_ci on ASCII letters,
then the "Illegal mix of collations" error can be avoided in
a comparison operator. We can choose any of the sides as the operation
effective collation - the result will be equal.

This optimization is not applied when at least one side has an explicit
COLLATE clause. Two explicit COLLATE clauses in one comparison are
already illegal when the character sets are the same, so for
consistency this stays illegal when the character sets differ too.

Most important details:

- Splitting enum_repertoire_t into smaller subsets,
  for better repertoire granularity.
  A variable holding a repertoire value can now have multiple
  MY_REPERTOIRE_XXX flags set.

  This patch implements detecting tailoring equality
  on these repertoires:
  * MY_REPERTOIRE_ASCII_DIGITS - [0..9]
  * MY_REPERTOIRE_ASCII_MINUS_DOT_DIGITS - [-.0..9], e.g. '-10.5'.
    It consists of the new bits MY_REPERTOIRE_ASCII_MINUS and
    MY_REPERTOIRE_ASCII_DOT. They were a part of
    MY_REPERTOIRE_ASCII_NOT_IDENT before. As a result, the numeric values
    of MY_REPERTOIRE_ASCII and MY_REPERTOIRE_UNICODE30 changed
    (223 and 255).
  * MY_REPERTOIRE_ASCII_ALNUM - [A..Z,a..z,0..9].
  * MY_REPERTOIRE_ASCII_IDENT - ALNUM + underscore
  * MY_REPERTOIRE_ASCII       - the entire range U+0000..U+007F

- Adding a new virtual function "tailoring" in my_collation_handler_st.

  It returns the tailoring on the given repertoire for the given
  collation.

  If cs1->coll->tailoring(cs1, some_repertoire) returns {0,0},
  it means the illegal mix optimization cannot be used
  for this collation on the given repertoire.

  If these calls:
     tr1= cs1->coll->tailoring(cs1, some_repertoire);
     tr2= cs2->coll->tailoring(cs2, some_repertoire);
  return both non-NULL results and tr1.str==tr2.str,
  then these collations are equal on the given repertoire
  and are mutually replaceable for a comparison operator,
  so "Illegal mix of collations" can be avoided.

- Tailoring strings are shared constants my_tailoring_str_*
  (strings/strings_def.h). Tailorings are compared by the string
  pointer, so tailoring() implementations return strings only from
  these constants. The strings do not depend on PAD/NOPAD.

- my_string_repertoire() for ucs2, utf16, utf32 and NONASCII 8bit
  character sets no longer stops at the first bad byte sequence with
  the repertoire found so far, which could be MY_REPERTOIRE_NONE
  (compatible with everything). It returns MY_REPERTOIRE_EXTENDED
  for such strings. A unit test was added into unittest/strings.

- Adding CHARSET_INFO::is_ascii_superset(). It tells if a character set
  can store all ASCII characters (it is false for swe7).
  It is used in Item_func_conv_charset to calculate "safe", and in
  left_is_algorithmically_simpler() to prefer other character sets
  to those where ASCII strings cannot be converted safely.

- DBUG_ASSERTs were added into mysys/charset.c, to check that
  the "tailoring" method is set in collation handlers when a collation
  is added: compiled-in, from ctype-extra.c, or loaded from Index.xml.

- Adding a new function my_collations_equal_on_repertoire(cs1, cs2,
  repertoire) in strings/. It tells if two collations are equal on
  the repertoire, taking tailoring() results and PAD/NOPAD and the UCA
  version into account. Callers do not compare tailoring() results
  directly, so the way tailorings are compared (currently by the pointer
  to the shared constant string) is private to strings/ and can be
  extended later, e.g. for UCA collations on MY_REPERTOIRE_UNICODE30.

- Adding a new method DTCollation::aggregate_by_tailoring().
  Collations with different MY_CS_NOPAD are never compatible.
  UCA collations of different versions are not compatible on
  the repertoire with ASCII punctuation, as the UCA version
  affects its order (e.g. UCA-6.2.0 moved GRAVE ACCENT
  and CIRCUMFLEX ACCENT after PERCENT SIGN).

- DTCollation::aggregate() now makes the repertoire of the result cover
  the repertoires of both sides in all successful branches.
  Before, set(dt) replaced the repertoire with the repertoire of the
  winner side only, so the result could have a too narrow repertoire,
  e.g. IF(1, 1.5, HEX(255)) had MY_REPERTOIRE_ASCII_ALNUM although the
  value '1.5' has a punctuation character. With the more granular
  repertoires this could wrongly allow mixing collations which are
  equal on ALNUM but different on punctuation.

- Adding a new flag MY_COLL_ALLOW_BY_TAILORING.
  It indicates to DTCollation::aggregate() that the illegal
  mix optimization by repertoire can be used in the given context.

  MY_COLL_CMP_CONV now includes MY_COLL_ALLOW_BY_TAILORING.
  Note, only comparison operators pass this flag.
  Functions returning a string result do not pass this flag,
  because in operations like CONCAT(a,b) we still need to evaluate
  precisely the collation of the result - we cannot just choose
  a collation of one of the sides (even if they are compatible
  on the given repertoire). This also applies to ExtractValue()
  and UpdateXML(), which now aggregate their arguments without
  this flag.

- As in my_repertoire_t the value MY_REPERTOIRE_ASCII is now a set of
  bits rather than a single bit, the way to detect "is only ASCII"
  repertoires has changed in the code.

  For example:
    // repertoire *IS* ascii
    if (repertoire == MY_REPERTOIRE_ASCII)

  has changed in multiple places in the code to

    // repertoire *HAS* only ascii characters
    if (my_repertoire_is_subset_of(repertoire, MY_REPERTOIRE_ASCII))

  The new function my_repertoire_is_subset_of() in m_ctype.h
  is used for this purpose in C code. In C++ code
  DTCollation::repertoire_is_subset_of() is used, e.g.:

    if (collation.repertoire_is_subset_of(MY_REPERTOIRE_ASCII))

- Repertoire of some expressions was adjusted to the new meaning:
  * Item_null now has MY_REPERTOIRE_NONE (was ASCII).
  * MY_LOCALE::repertoire() now returns MY_REPERTOIRE_UNICODE30
    (was EXTENDED).
  * Lex_string_with_metadata_st::repertoire(cs) now scans the string
    contents to detect the actual repertoire.
  * HEX() now has MY_REPERTOIRE_ASCII_ALNUM.
  * DATE_FORMAT() decides on MY_REPERTOIRE_EXTENDED using the locale
    which is actually used: the explicit third argument, or
    @@lc_time_names. It was always @@lc_time_names before.
    A non-constant locale argument is assumed to be non-ASCII.
  * QUOTE, MAKE_SET, EXPORT_SET, LPAD, RPAD, GROUP_CONCAT, JSON_ARRAY,
    JSON_OBJECT and JSON_OBJECTAGG add the repertoire of the extra
    characters they put into the result (quotes, separators, padding,
    brackets).
  * LOWER() and UPPER() add the letters of both cases to the repertoire,
    because the result can contain letters of the opposite case.
    In Turkish collations they also add MY_REPERTOIRE_EXTENDED, as an
    ASCII letter can be converted to a non-ASCII one (I -> dotless i).
    Unicode collations are detected by the new member
    casefold_info_st::can_convert_to_non_ascii_on_casefolding,
    simple 8bit collations by to_lower['I'] and to_upper['i'].

- Lex_string_with_metadata_st::repertoire(cs) now scans a string literal
  for any byte oriented character set (mbminlen==1), not only for
  ASCII based ones. It was assumed to have the full repertoire for swe7,
  which is not ASCII based. The scan is done by Unicode code points, so
  swe7 strings with digits and letters get a narrow repertoire, while
  e.g. 0x5B (LATIN CAPITAL LETTER A WITH DIAERESIS in swe7) is non-ASCII.
  Therefore swe7_swedish_ci can be mixed with other collations now,
  in the same way as other collations with MY_CS_IDENT_CASEUP_CI.
  The test func_time was adjusted: a swe7 literal ' ull' with ASCII
  characters is now safe to convert, the error is still tested for '[ull'.

- cp1250_czech_cs sorts -, . and 0..9 in the code point order, so it gets
  the MINUS_DOT_DIGITS tailoring. latin2_czech_cs gets only DIGITS,
  as it sorts the minus and the dot differently. Both were verified
  with ctype_digits_verify.

- The tis620 collations now set CHARSET_INFO::tab_to_uni (was NULL),
  so my_charset_is_ascii_based() is true for tis620.
  It was false for tis620, although tis620 is ASCII compatible.
  Therefore tis620 string literals were always considered to have
  a non-ASCII repertoire, and the repertoire optimizations did not work
  for them. Now ASCII-only tis620 literals are detected as ASCII.
  This changes the result of subselect_extra_no_semijoin
  from an error to success.

- tis620_thai_ci and tis620_thai_nopad_ci fold letters to lower case
  and then compare by the code point on the entire ASCII range
  (the Thai specific rules affect only non-ASCII characters).
  A new function my_tailoring_ascii_casedn_ci() returns tailorings
  for such collations on ALNUM, IDENT and ASCII. For example, the
  underscore sorts before the letters, unlike in the caseup tailorings.

- New flags were added for CHARSET_INFO::state
  * MY_CS_ASCII_CASEUP_CI - for simple 8bit case insensitive collations.
    It means that this collation has no irregularities on the ASCII
    repertoire, and converts lower case letters to upper case ones,
    so the underscore sorts after the letters (with an upper to lower
    case conversion it would sort before the letters).

  * MY_CS_IDENT_CASEUP_CI - for simple 8bit case insensitive collations.
    It means that this collation has no irregularities on the IDENT
    repertoire (but can have irregularities say on punctuation).

  * MY_CS_ASCII_STD_UCA - for UCA collations.
    It means that a UCA collation does not reorder ASCII characters.

  * MY_CS_DIGITS_STD - for any collation. It means that the ASCII digits
    0..9 are sorted in the code point order and are not tailored, and,
    for PAD collations, the space sorts before the digits. Strings of
    digits are compared in the same way in all such collations, even if
    they have irregularities on letters (latin7, cp866, czech, danish,
    turkish UCA, etc.), or differ by case sensitivity. It is detected
    from sort_order for 8bit collations, and from the tailoring rules
    for UCA collations. Collations with MY_CS_DIGITS_STD still must have
    the same PAD/NOPAD attribute.

  * MY_CS_ASCII_MINUS_DOT_DIGITS - for any collation. Means
    MY_CS_DIGITS_STD, and additionally the minus sorts before the dot,
    which sorts before the digits (and the space sorts before the minus
    in PAD collations). Strings like '-10.5' are compared in the same
    way in all such collations. It is detected from sort_order for simple
    8bit collations (my_coll_init_simple()), and from the tailoring rules
    for UCA collations. For the multi-byte collations big5_chinese_ci,
    gbk_chinese_ci, gb2312_chinese_ci and sjis_japanese_ci it is declared
    by my_tailoring_ident_caseup_ci_generic(). The flags MY_CS_DIGITS_STD
    and MY_CS_ASCII_MINUS_DOT_DIGITS are stored in ctype-extra.c for
    compiled collations (conf_to_src.c prints them), and the detection is
    skipped for a collation which already has any of the repertoire flags
    (MY_CS_REPERTOIRE_FLAGS).

- strings/conf_to_src.c was modified to detect and print
  MY_CS_ASCII_CASEUP_CI and MY_CS_IDENT_CASEUP_CI flags.

- strings/ctype-extra.c was regenerated with new flags.

- 8bit collations are not given MY_CS_ASCII_CASEUP_CI and
  MY_CS_IDENT_CASEUP_CI if they are PAD and the space does not sort
  before the digits. A shorter string is padded with spaces for
  comparison, so the position of the space matters even for strings
  without spaces ("ab" is compared to "abc" as "ab " to "abc").
  A test collation latin1_space_after_letters_ci was added to
  mysql-test/std_data/ldml/ for this. ctype-extra.c does not change.

- The initializer of my_collation_cs_handler in ctype-utf8.c
  (utf8mb3_general_cs, under HAVE_UTF8_GENERAL_CS) was fixed to match
  my_collation_handler_st: the missing get_id and get_collation_name
  members were added, so that eq_collation was not initialized in their
  place, and the new member "tailoring" is now set to my_tailoring_none.

- Results of the existing tests func_str, func_test, ps, view
  and subselect_sj changed from "Illegal mix of collations" to success.
  In these tests ASCII-only strings were compared using collations
  with equal tailoring on ASCII, e.g. latin1_swedish_ci with
  latin2_general_ci (func_str, ps), koi8r_general_ci with
  latin1_swedish_ci (func_test), latin1_general_ci with
  latin1_swedish_ci on letters and digits (view), cp932_japanese_ci
  with latin1_swedish_ci (subselect_sj).
  Such comparisons are now allowed.
  Cases with collations that are still incompatible on ASCII,
  e.g. latin7_general_ci, were added to the tests and still fail.

- Adding a number of MTR tests in
  plugin/func_test/mysql-test/func_test/.
  They display a tailoring by collation name and repertoire as
  returned by:
    cs->coll->tailoring(cs, some_repertoire)
  A dynamically linked plugin function collation_tailoring() was added
  for the purpose of these tests. Tailorings do not depend on PAD/NOPAD
  and on the UCA version, so the function shows "[nopad]" for NOPAD
  collations, and "[version X.Y.Z]" for UCA collations on the ASCII
  repertoire. The version is formatted by a new class UCAVersion,
  which uses a new method CharBuffer::append_uint8().

- Adding the test plugin/func_test/mysql-test/func_test/
  ctype_caseup_ci_verify.test. It checks the order of ASCII characters
  for all collations declared "caseup CI" on the ASCII or IDENT range.

- Adding the test plugin/func_test/mysql-test/func_test/
  collation_tailoring_args.test. It checks the arguments of the plugin
  function collation_tailoring(): the repertoire passed by name or by
  number, and the NULL results for unknown names.

- Adding the test plugin/func_test/mysql-test/func_test/ctype_ldml.test.
  It displays the tailorings of collations loaded at runtime from
  Index.xml: simple 8bit collations (flags detected from the weights)
  and Unicode collations (LDML rules which do or do not reorder ASCII).

- Adding a number of MTR tests mysql-test/main/ctype_xxx_tailoring.test
  They display two-dimensional charts showing which collations
  are compatible on which repertoires.
  Tests for DATE_FORMAT() with an explicit locale were added to
  ctype_big5_tailoring.test, ctype_cp932_tailoring.test and
  ctype_latin5_tailoring.test, which also has the chart for latin5.

- Adding a number of MTR tests in the form of the originally
  reported script for various collations:

    SELECT Insert_priv FROM mysql.user WHERE Insert_priv='...';

  The repeated blocks of queries are shared through
  mysql-test/include/
  ctype_tailoring_0{1,2,3}_mysql_user_Insert_priv*.inc.

- Adding tests to ctype_cp932.test checking that ExtractValue() and
  UpdateXML() still raise "Illegal mix of collations".
2026-10-15 15:39:04 +04:00

550 lines
17 KiB
C
Raw Permalink Blame History

/* Copyright (c) 2000-2003, 2005-2007 MySQL AB, 2009 Sun Microsystems, Inc.
Copyright (c) 2009-2011, Monty Program Ab
Use is subject to license terms.
Copyright (c) 2009-2011, Monty Program Ab
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; version 2 of the License.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1335 USA */
#include "strings_def.h"
#include <m_ctype.h>
#include <fcntl.h>
#include <my_xml.h>
#define ROW_LEN 16
#define ROW16_LEN 8
#define MAX_BUF (64*1024)
#define MY_ALL_CHARSETS_SIZE 2048
static struct charset_info_st all_charsets[MY_ALL_CHARSETS_SIZE];
static uint refids[MY_ALL_CHARSETS_SIZE];
static CHARSET_INFO *inheritance_source(uint id)
{
return &all_charsets[refids[id]];
}
void
print_array(FILE *f, const char *set, const char *name, const uchar *a, int n)
{
int i;
fprintf(f,"static const uchar %s_%s[] = {\n", name, set);
for (i=0 ;i<n ; i++)
{
fprintf(f,"0x%02X",a[i]);
fprintf(f, (i+1<n) ? "," :"" );
fprintf(f, ((i+1) % ROW_LEN == n % ROW_LEN) ? "\n" : "" );
}
fprintf(f,"};\n\n");
}
void
print_array16(FILE *f, const char *set, const char *name, const uint16 *a, int n)
{
int i;
fprintf(f,"static const uint16 %s_%s[] = {\n", name, set);
for (i=0 ;i<n ; i++)
{
fprintf(f,"0x%04X",a[i]);
fprintf(f, (i+1<n) ? "," :"" );
fprintf(f, ((i+1) % ROW16_LEN == n % ROW16_LEN) ? "\n" : "" );
}
fprintf(f,"};\n\n");
}
static uint get_collation_number(const char *name)
{
CHARSET_INFO *cs;
for (cs= all_charsets;
cs < all_charsets + array_elements(all_charsets);
cs++)
{
if (cs->coll_name.str && !strcmp(cs->coll_name.str, name))
return cs->number;
}
return 0;
}
static uint
get_charset_number_internal(const char *charset_name, uint cs_flags)
{
CHARSET_INFO *cs;
for (cs= all_charsets;
cs < all_charsets + array_elements(all_charsets);
cs++)
{
if (cs->cs_name.str && (cs->state & cs_flags) &&
!strcmp(cs->cs_name.str, charset_name))
return cs->number;
}
return 0;
}
char *mdup(const char *src, uint len)
{
char *dst=(char*)malloc(len);
if (!dst)
exit(1);
memcpy(dst,src,len);
return dst;
}
static void simple_cs_copy_data(struct charset_info_st *to, CHARSET_INFO *from)
{
to->number= from->number ? from->number : to->number;
to->state|= from->state;
if (from->cs_name.str)
{
to->cs_name.str= strndup(from->cs_name.str, from->cs_name.length);
to->cs_name.length= from->cs_name.length;
}
if (from->coll_name.str)
{
to->coll_name.str= strndup(from->coll_name.str, from->coll_name.length);
to->coll_name.length= from->coll_name.length;
}
if (from->tailoring)
to->tailoring= strdup(from->tailoring);
if (from->m_ctype)
to->m_ctype= (uchar*) mdup((char*) from->m_ctype, MY_CS_CTYPE_TABLE_SIZE);
if (from->to_lower)
to->to_lower= (uchar*) mdup((char*) from->to_lower, MY_CS_TO_LOWER_TABLE_SIZE);
if (from->to_upper)
to->to_upper= (uchar*) mdup((char*) from->to_upper, MY_CS_TO_UPPER_TABLE_SIZE);
if (from->sort_order)
{
to->sort_order= (uchar*) mdup((char*) from->sort_order, MY_CS_SORT_ORDER_TABLE_SIZE);
/*
set_max_sort_char(to);
*/
}
if (from->tab_to_uni)
{
uint sz= MY_CS_TO_UNI_TABLE_SIZE*sizeof(uint16);
to->tab_to_uni= (uint16*) mdup((char*)from->tab_to_uni, sz);
/*
create_fromuni(to);
*/
}
}
/*
cs->xxx arrays can be NULL in case when a collation has an entry only
in Index.xml and has no entry in csname.xml (e.g. in case of a binary
collation or a collation using <import> command).
refcs->xxx arrays can be NULL if <import> refers to a collation
which is not defined in csname.xml, e.g. an always compiled collation
such as latin1_swedish_ci.
*/
static void inherit_charset_data(struct charset_info_st *cs,
CHARSET_INFO *refcs)
{
cs->state|= (refcs->state & (MY_CS_PUREASCII|MY_CS_NONASCII));
if (refcs->m_ctype && cs->m_ctype &&
!memcmp(cs->m_ctype, refcs->m_ctype, MY_CS_CTYPE_TABLE_SIZE))
cs->m_ctype= NULL;
if (refcs->to_lower && cs->to_lower &&
!memcmp(cs->to_lower, refcs->to_lower, MY_CS_TO_LOWER_TABLE_SIZE))
cs->to_lower= NULL;
if (refcs->to_upper && cs->to_upper &&
!memcmp(cs->to_upper, refcs->to_upper, MY_CS_TO_LOWER_TABLE_SIZE))
cs->to_upper= NULL;
if (refcs->tab_to_uni && cs->tab_to_uni &&
!memcmp(cs->tab_to_uni, refcs->tab_to_uni,
MY_CS_TO_UNI_TABLE_SIZE * sizeof(uint16)))
cs->tab_to_uni= NULL;
}
static CHARSET_INFO *find_charset_data_inheritance_source(CHARSET_INFO *cs)
{
CHARSET_INFO *refcs;
uint refid= get_charset_number_internal(cs->cs_name.str, MY_CS_PRIMARY);
return refid && refid != cs->number &&
(refcs= &all_charsets[refid]) &&
(refcs->state & MY_CS_LOADED) ? refcs : NULL;
}
/**
Detect if "cs" needs further loading from csname.xml
@param cs - the character set pointer
@retval FALSE - if the current data (e.g. loaded from from Index.xml)
is not enough to dump the character set and requires
further reading from the csname.xml file.
@retval TRUE - if the current data is enough to dump,
no reading of csname.xml is needed.
*/
static my_bool simple_cs_is_full(CHARSET_INFO *cs)
{
return ((cs->cs_name.str && cs->tab_to_uni && cs->m_ctype && cs->to_upper &&
cs->to_lower) &&
(cs->number && cs->coll_name.str &&
(cs->sort_order || cs->tailoring || (cs->state & MY_CS_BINSORT))));
}
static int add_collation(struct charset_info_st *cs)
{
if (cs->coll_name.str &&
(cs->number || (cs->number= get_collation_number(cs->coll_name.str))))
{
if (!(all_charsets[cs->number].state & MY_CS_COMPILED))
{
simple_cs_copy_data(&all_charsets[cs->number],cs);
}
cs->number= 0;
cs->coll_name.str= 0;
cs->coll_name.length= 0;
cs->tailoring= NULL;
cs->state= 0;
cs->sort_order= NULL;
cs->state= 0;
}
return MY_XML_OK;
}
static void
default_reporter(enum loglevel level __attribute__ ((unused)),
const char *format __attribute__ ((unused)),
...)
{
}
static void
my_charset_loader_init(MY_CHARSET_LOADER *loader)
{
loader->error[0]= '\0';
loader->once_alloc= malloc;
loader->malloc= malloc;
loader->realloc= realloc;
loader->free= free;
loader->reporter= default_reporter;
loader->add_collation= add_collation;
}
static int my_read_charset_file(const char *filename)
{
char buf[MAX_BUF];
int fd;
uint len;
MY_CHARSET_LOADER loader;
my_charset_loader_init(&loader);
if ((fd=open(filename,O_RDONLY)) < 0)
{
fprintf(stderr,"Can't open '%s'\n",filename);
return 1;
}
len=read(fd,buf,MAX_BUF);
DBUG_ASSERT(len < MAX_BUF);
close(fd);
if (my_parse_charset_xml(&loader, buf, len))
{
fprintf(stderr, "Error while parsing '%s': %s\n", filename, loader.error);
exit(1);
}
return FALSE;
}
void print_arrays(FILE *f, CHARSET_INFO *cs)
{
if (cs->m_ctype)
print_array(f, cs->coll_name.str, "ctype", cs->m_ctype, MY_CS_CTYPE_TABLE_SIZE);
if (cs->to_lower)
print_array(f, cs->coll_name.str, "to_lower", cs->to_lower, MY_CS_TO_LOWER_TABLE_SIZE);
if (cs->to_upper)
print_array(f, cs->coll_name.str, "to_upper", cs->to_upper, MY_CS_TO_UPPER_TABLE_SIZE);
if (cs->sort_order)
print_array(f, cs->coll_name.str, "sort_order", cs->sort_order, MY_CS_SORT_ORDER_TABLE_SIZE);
if (cs->tab_to_uni)
print_array16(f, cs->coll_name.str, "to_uni", cs->tab_to_uni, MY_CS_TO_UNI_TABLE_SIZE);
}
/**
Print an array member of a CHARSET_INFO.
@param f - the file to print into
@param cs0 - reference to the CHARSET_INFO to print
@param array0 - pointer to the array data (can be NULL)
@param cs1 - reference to the CHARSET_INFO that the data
can be inherited from (e.g. primary collation)
@param array1 - pointer to the array data in cs1 (can be NULL)
@param name - name of the member
If array0 is not null, then the CHARSET_INFO being dumped has its
own array (e.g. the default collation for the character set).
We print the name of this array using cs0->name and return.
If array1 is not null, then the CHARSET_INFO being dumpled reuses
the array from another collation. We print the name of the array of
the referenced collation using cs1->name and return.
Otherwise (if both array0 and array1 are NULL), we have a collation
of a character set whose primary collation is not available now,
and which does not have its own entry in csname.xml file.
For example, Index.xml has this entry:
<collation name="latin1_swedish_ci_copy">
<rules>
<import source="latin1_swedish_ci"/>
</rules>
</collation>
and latin1.xml does not have entries for latin1_swedish_ci_copy.
In such cases we print NULL as a pointer to the array.
It will be set to a not-null data during the first initialization
by the inherit_charset_data() call (see mysys/charset.c for details).
*/
static void
print_array_ref(FILE *f,
CHARSET_INFO *cs0, const void *array0,
CHARSET_INFO *cs1, const void *array1,
const char *name)
{
CHARSET_INFO *cs= array0 ? cs0 : array1 ? cs1 : NULL;
if (cs)
fprintf(f," %s_%s, /* %s */\n",
name, cs->coll_name.str, name);
else
fprintf(f," NULL, /* %s */\n", name);
}
static const char *nopad_infix(CHARSET_INFO *cs)
{
return (cs->state & MY_CS_NOPAD) ? "_nopad" : "";
}
void fprintf_lex_str_member(FILE *f, const LEX_CSTRING str, const char *comment)
{
fprintf(f," { STRING_WITH_LEN(\"%s\") }, %s\n", str.str, comment);
}
void dispcset(FILE *f,CHARSET_INFO *cs)
{
fprintf(f,"{\n");
fprintf(f," %d,%d,%d,\n",cs->number,0,0);
fprintf(f," MY_CS_COMPILED%s%s%s%s%s%s%s%s%s%s,\n",
cs->state & MY_CS_BINSORT ? "|MY_CS_BINSORT" : "",
cs->state & MY_CS_PRIMARY ? "|MY_CS_PRIMARY" : "",
cs->state & MY_CS_CSSORT ? "|MY_CS_CSSORT" : "",
cs->state & MY_CS_PUREASCII ? "|MY_CS_PUREASCII" : "",
cs->state & MY_CS_NONASCII ? "|MY_CS_NONASCII" : "",
cs->state & MY_CS_NOPAD ? "|MY_CS_NOPAD" : "",
cs->state & MY_CS_ASCII_CASEUP_CI ? "|MY_CS_ASCII_CASEUP_CI" : "",
cs->state & MY_CS_IDENT_CASEUP_CI ? "|MY_CS_IDENT_CASEUP_CI" : "",
cs->state & MY_CS_DIGITS_STD ? "|MY_CS_DIGITS_STD" : "",
cs->state & MY_CS_ASCII_MINUS_DOT_DIGITS ?
"|MY_CS_ASCII_MINUS_DOT_DIGITS" : "");
if (cs->coll_name.str)
{
CHARSET_INFO *srccs= inheritance_source(cs->number);
fprintf_lex_str_member(f, cs->cs_name, "/* cset name */");
fprintf_lex_str_member(f, cs->coll_name, "/* coll name */");
fprintf(f," \"\", /* comment */\n");
if (cs->tailoring)
fprintf(f, " \"%s\", /* tailoring */\n", cs->tailoring);
else
fprintf(f," NULL, /* tailoring */\n");
print_array_ref(f, cs, cs->m_ctype, srccs, srccs->m_ctype, "ctype");
print_array_ref(f, cs, cs->to_lower, srccs, srccs->to_lower, "to_lower");
print_array_ref(f, cs, cs->to_upper, srccs, srccs->to_upper, "to_upper");
if (cs->sort_order)
fprintf(f," sort_order_%s, /* sort_order */\n", cs->coll_name.str);
else
fprintf(f," NULL, /* sort_order */\n");
fprintf(f," NULL, /* uca */\n");
print_array_ref(f, cs, cs->tab_to_uni, srccs, srccs->tab_to_uni, "to_uni");
}
else
{
fprintf(f," {NULL,0}, /* cset name */\n");
fprintf(f," {NULL,0}, /* coll name */\n");
fprintf(f," NULL, /* comment */\n");
fprintf(f," NULL, /* tailoging */\n");
fprintf(f," NULL, /* ctype */\n");
fprintf(f," NULL, /* lower */\n");
fprintf(f," NULL, /* upper */\n");
fprintf(f," NULL, /* sort order */\n");
fprintf(f," NULL, /* uca */\n");
fprintf(f," NULL, /* to_uni */\n");
}
fprintf(f," NULL, /* from_uni */\n");
fprintf(f," NULL, /* casefold */\n");
fprintf(f," NULL, /* state map */\n");
fprintf(f," NULL, /* ident map */\n");
fprintf(f," 1, /* strxfrm_multiply*/\n");
fprintf(f," 1, /* mbminlen */\n");
fprintf(f," 1, /* mbmaxlen */\n");
fprintf(f," 0, /* min_sort_char */\n");
fprintf(f," 255, /* max_sort_char */\n");
fprintf(f," ' ', /* pad_char */\n");
fprintf(f," 0, /* escape_with_backslash_is_dangerous */\n");
fprintf(f," MY_CS_COLL_LEVELS_S1,\n");
fprintf(f," &my_charset_8bit_handler,\n");
if (cs->state & MY_CS_BINSORT)
fprintf(f," &my_collation_8bit%s_bin_handler,\n", nopad_infix(cs));
else
fprintf(f," &my_collation_8bit_simple%s_ci_handler,\n", nopad_infix(cs));
fprintf(f,"}\n");
}
static void
fprint_copyright(FILE *file)
{
fprintf(file,
"/* Copyright 2000-2008 MySQL AB, 2008 Sun Microsystems, Inc.\n"
" Copyright (c) 2000, 2011, Oracle and/or its affiliates.\n"
" Copyright 2008-2023 MariaDB Corporation\n"
"\n"
" This program is free software; you can redistribute it and/or modify\n"
" it under the terms of the GNU General Public License as published by\n"
" the Free Software Foundation; version 2 of the License.\n"
"\n"
" This program is distributed in the hope that it will be useful,\n"
" but WITHOUT ANY WARRANTY; without even the implied warranty of\n"
" MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n"
" GNU General Public License for more details.\n"
"\n"
" You should have received a copy of the GNU General Public License\n"
" along with this program; if not, write to the Free Software\n"
" Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1335 USA */\n"
"\n");
}
int
main(int argc, char **argv __attribute__((unused)))
{
struct charset_info_st ncs, *cs;
char filename[256];
FILE *f= stdout;
if (argc < 2)
{
fprintf(stderr, "usage: %s source-dir\n", argv[0]);
exit(EXIT_FAILURE);
}
bzero((void*)&ncs,sizeof(ncs));
bzero((void*)&all_charsets,sizeof(all_charsets));
bzero((void*) refids, sizeof(refids));
snprintf(filename,sizeof(filename),"%s/%s",argv[1],"Index.xml");
my_read_charset_file(filename);
for (cs= all_charsets;
cs < all_charsets + array_elements(all_charsets);
cs++)
{
if (cs->number && !(cs->state & MY_CS_COMPILED))
{
if ( (!simple_cs_is_full(cs)) && (cs->cs_name.str))
{
snprintf(filename, sizeof filename, "%s/%.*s.xml",
argv[1], (int) cs->cs_name.length, cs->cs_name.str);
my_read_charset_file(filename);
}
cs->state|= MY_CS_LOADED;
}
}
fprintf(f, "/*\n");
fprintf(f, " This file was generated by the conf_to_src utility. "
"Do not edit it directly,\n");
fprintf(f, " edit the XML definitions in sql/share/charsets/ instead.\n\n");
fprintf(f, " To re-generate, run the following in the strings/ "
"directory:\n");
fprintf(f, " ./conf_to_src ../sql/share/charsets/ > FILE\n");
fprintf(f, "*/\n\n");
fprint_copyright(f);
fprintf(f,"#include \"strings_def.h\"\n");
fprintf(f,"#include <m_ctype.h>\n\n");
for (cs= all_charsets;
cs < all_charsets + array_elements(all_charsets);
cs++)
{
if (cs->state & MY_CS_LOADED)
{
CHARSET_INFO *refcs= find_charset_data_inheritance_source(cs);
cs->state|= my_8bit_charset_flags_from_data(cs) |
my_8bit_collation_flags_from_data(cs);
if (refcs)
{
refids[cs->number]= refcs->number;
inherit_charset_data(cs, refcs);
}
fprintf(f,"#ifdef HAVE_CHARSET_%s\n", cs->cs_name.str);
print_arrays(f, cs);
fprintf(f,"#endif\n");
fprintf(f,"\n");
}
}
fprintf(f,"struct charset_info_st compiled_charsets[] = {\n");
for (cs= all_charsets;
cs < all_charsets + array_elements(all_charsets);
cs++)
{
if (cs->state & MY_CS_LOADED)
{
fprintf(f,"#ifdef HAVE_CHARSET_%s\n", cs->cs_name.str);
dispcset(f,cs);
fprintf(f,",\n");
fprintf(f,"#endif\n");
}
}
dispcset(f,&ncs);
fprintf(f,"};\n");
return 0;
}
View Git Blame Copy Permalink