database/mariadb
1
0
Fork 0
mirror of https://github.com/MariaDB/server.git synced 2025-08-08 11:22:35 +03:00
Code Activity

MDEV-36213 Doubled memory usage (11.4.4 <-> 11.4.5)

Browse Source 
Fixing the code adding MySQL _0900_ collations as _uca1400_ aliases
not to perform deep initialization of the corresponding _uca1400_
collations.

Only basic initialization is now performed which allows to watch
these collations (both _0900_ and _uca1400_) in queries to
INFORMATION_SCHEMA tables COLLATIONS and
COLLATION_CHARACTER_SET_APPLICABILITY,
as well as in SHOW COLLATION statements.

Deep initialization is now performed only when a collation
(either the _0900_ alias or the corresponding  _uca1400_ collation)
is used for the very first time after the server startup.

Refactoring was done to maintain the code easier:
- most of the _uca1400_ code was moved from ctype-uca.c
  to a new file ctype-uca1400.c
- most of the _0900_ code was moved from type-uca.c
  to a new file ctype-uca0900.c

Change details:

- The original function add_alias_for_collation() added by the patch for
   "MDEV-20912 Add support for utf8mb4_0900_* collations in MariaDB Server"
  was removed from mysys/charset.c, as it had two two problems:

  a. it forced deep initialization of the _uca1400_ collations
     when adding _0900_ aliases for them at the server startup
     (the main reported problem)

  b. the collation initialization code in add_alias_for_collation()
     was related more to collations rather than to memory management,
     so /strings should be a better place for it than /mysys.

  The code from add_alias_for_collation() was split into separate functions.
  Cyclic dependency was removed. `#include <my_sys.h>` was removed
  from /strings/ctype-uca.c. Collations are now added using a callback
  function MY_CHARSET_LOADED::add_collation, like it is done for
  user collations defined in Index.xml. The code in /mysys sets
  MY_CHARSET_LOADED::add_collation to add_compiled_collation().

- The function compare_collations() was removed.
  A new virtual function was added into my_collation_handler_st instead:

    my_bool (*eq_collation)(CHARSET_INFO *self, CHARSET_INFO *other);

  because it is the collation handler who knows how to detect equal
  collations by comparing only some of CHARSET_INFO members without
  their deep initialization.

  Three implementations were added:
  - my_ci_eq_collation_uca() for UCA collations, it compares
    _0900_ collations as equal to their corresponding _uca1400_ collations.
  - my_ci_eq_collation_utf8mb4_bin(), it compares
    utf8mb4_nopad_bin and utf8mb4_0900_bin as equal.
  - my_ci_eq_collation_generic() - the default implementation,
    which compares all collations as not equal.

  A C++ wrapper CHARSET_INFO::eq_collations() was added.
  The code in /sql was changes to use the wrapper instead of
  the former calls for the removed function compare_collations().

- A part of add_alias_for_collation() was moved into a new function
  my_ci_alloc(). It allocates a memory for a new charset_info_st
  instance together with the collation name and the comment using a single
  MY_CHARSET_LOADER::once_alloc call, which points to my_once_alloc()
  in the server.

- A part of add_alias_for_collation() was moved into a new function
  my_ci_make_comment_for_alias(). It makes an "Alias for xxx" string,
  e.g. "Alias for utf8mb4_uca1400_swedish_ai_ci" in case of
  utf8mb4_sv_0900_ai_ci.

- A part of the code in create_tailoring() was moved to
  a new function my_uca1400_collation_get_initialized_shared_uca(),
  to reuse the code between _uca1400_ and _0900_ collations.

- A new function my_collation_id_is_mysql_uca0900() was added
  in addition to my_collation_id_is_mysql_uca1400().

- Functions to build collation names were added:
   my_uca0900_collation_build_name()
   my_uca1400_collation_build_name()

- A shared function function was added:

  my_bool
  my_uca1400_collation_alloc_and_init(MY_CHARSET_LOADER *loader,
                                      LEX_CSTRING name,
                                      LEX_CSTRING comment,
                                      const uca_collation_def_param_t *param,
                                      uint id)

  It's reused to add _uca1400_ and _0900_ collations, with basic
  initialization (without deep initialization).

- The function add_compiled_collation() changed its return type from
  void to int, to make it compatible with MY_CHARSET_LOADER::add_collation.

- Functions mysql_uca0900_collation_definition_add(),
  mysql_uca0900_utf8mb4_collation_definitions_add(),
  mysql_utf8mb4_0900_bin_add() were added into ctype-uca0900.c.
  They get MY_CHARSET_LOADER as a parameter.

- Functions my_uca1400_collation_definition_add(),
  my_uca1400_collation_definitions_add() were moved from
  charset-def.c to strings/ctype-uca1400.c.
  The latter now accepts MY_CHARSET_LOADER as the first parameter
  instead of initializing a MY_CHARSET_LOADER inside.

- init_compiled_charsets() now initializes a MY_CHARSET_LOADER
  variable and passes it to all functions adding collations:
  - mysql_utf8mb4_0900_collation_definitions_add()
  - mysql_uca0900_utf8mb4_collation_definitions_add()
  - mysql_utf8mb4_0900_bin_add()

- A new structure was added into ctype-uca.h:

  typedef struct uca_collation_def_param
  {
    my_cs_encoding_t cs_id;
    uint tailoring_id;
    uint nopad_flags;
    uint level_flags;
  } uca_collation_def_param_t;

  It simplifies reusing the code for _uca1400_ and _0900_ collations.

- The definition of MY_UCA1400_COLLATION_DEFINITION was
  moved from ctype-uca.c to ctype-uca1400.h, to reuse
  the code for _uca1400_ and _0900_ collations.

- The definitions of "MY_UCA_INFO my_uca_v1400" and
  "MY_UCA_INFO my_uca1400_info_tailored[][]" were moved from
  ctype-uca.c to ctype-uca1400.c.

- The definitions/declarations of:
  - mysql_0900_collation_start,
  - struct mysql_0900_to_mariadb_1400_mapping
  - mysql_0900_to_mariadb_1400_mapping
  - mysql_utf8mb4_0900_collation_definitions_add()
  were moved from ctype-uca.c to ctype-uca0900.c

- Functions
  my_uca1400_make_builtin_collation_id()
  my_uca1400_collation_definition_init()
  my_uca1400_collation_id_uca400_compat()
  my_ci_get_collation_name_uca1400_context()
  were moved from ctype-uca.c to ctype-uca1400.c and ctype-uca1400.h

- A part of my_uca1400_collation_definition_init()
  was moved into my_uca0520_builtin_collation_by_id(),
  to make functions smaller.
This commit is contained in:
Alexander Barkov
2025-03-22 12:45:13 +04:00
parent 0dad1458e7
commit 10c063f9f0
38 changed files with 1498 additions and 646 deletions
Download Patch File Download Diff File Expand all files Collapse all files

422
strings/ctype-uca.c
View File

@@ -34,8 +34,8 @@
#include "strings_def.h"
#include <m_ctype.h>
#include <my_sys.h>
#include "ctype-uca.h"
#include "ctype-uca0520.h"
#include "ctype-unidata.h"
#include "my_bit.h"
@@ -30214,78 +30214,6 @@ MY_UCA_INFO my_uca_v520=
};
#include "ctype-uca1400data.h"
static MY_UCA_INFO my_uca_v1400=
{
{
{
0x10FFFF, /* maxchar */
(uchar *) uca1400_length,
(uint16 **) uca1400_weight,
{ /* Contractions: */
array_elements(uca1400_contractions), /* nitems */
uca1400_contractions, /* item */
NULL /* flags */
},
0, /* levelno */
{0}, /* contraction_hash */
NULL /* booster */
},
{
0x10FFFF, /* maxchar */
(uchar *) uca1400_length_secondary,
(uint16 **) uca1400_weight_secondary,
{ /* Contractions: */
array_elements(uca1400_contractions_secondary), /* nitems */
uca1400_contractions_secondary, /* item */
NULL /* flags */
},
1, /* levelno */
{0}, /* contraction_hash */
NULL /* booster */
},
{
0x10FFFF, /* maxchar */
(uchar *) uca1400_length_tertiary,
(uint16 **) uca1400_weight_tertiary,
{ /* Contractions: */
array_elements(uca1400_contractions_tertiary), /* nitems */
uca1400_contractions_tertiary, /* item */
NULL /* flags */
},
2, /* levelno */
{0}, /* contraction_hash */
NULL /* booster */
}
},
uca1400_non_ignorable_first,
uca1400_non_ignorable_last,
uca1400_primary_ignorable_first,
uca1400_primary_ignorable_last,
uca1400_secondary_ignorable_first,
uca1400_secondary_ignorable_last,
uca1400_tertiary_ignorable_first,
uca1400_tertiary_ignorable_last,
0x0000, /* first_trailing */
0x0000, /* last_trailing */
uca1400_variable_first,
uca1400_variable_last,
/* Misc */
uca1400_version
};
/******************************************************/
/*
@@ -31247,25 +31175,12 @@ static const char myanmar[]= "[shift-after-method expand]"
;
typedef struct my_uca1400_collation_definition_st
{
const char * tailoring;
const char * name;
uint16 id_utf8mb3;
uint16 id_utf8mb4;
uint16 id_ucs2;
uint16 id_utf16;
uint16 id_utf32;
} MY_UCA1400_COLLATION_DEFINITION;
/*
UCA1400 collation definitions in the order of their UCA400 counterparts,
with IDs of their closest UCA1400 counterparts, for character sets
utf8mb3, utf8mb4, ucs2, utf16, utf32.
*/
static MY_UCA1400_COLLATION_DEFINITION
MY_UCA1400_COLLATION_DEFINITION
my_uca1400_collation_definitions[MY_UCA1400_COLLATION_DEFINITION_COUNT]=
{
#define COLDEF(tl,name,id_utf8mb3,id_utf8mb4,id_ucs2,id_utf16,id_utf32) \
@@ -31309,9 +31224,17 @@ my_uca1400_collation_definitions[MY_UCA1400_COLLATION_DEFINITION_COUNT]=
};
static MY_UCA_INFO
my_uca1400_info_tailored[MY_CS_ENCODING_LAST+1]
[MY_UCA1400_COLLATION_DEFINITION_COUNT];
static my_bool
my_ci_eq_collation_uca(CHARSET_INFO *a, CHARSET_INFO *b)
{
return a->cset == b->cset &&
a->coll == b->coll &&
a->uca == b->uca &&
a->casefold == b->casefold &&
(a->state & MY_CS_NOPAD) == (b->state & MY_CS_NOPAD) &&
a->levels_for_order == b->levels_for_order &&
a->tailoring == b->tailoring;
}
typedef struct my_uca_scanner_param_st
@@ -34722,6 +34645,30 @@ my_uca_info_init(MY_CHARSET_LOADER *loader,
}
/*
Initialize (if needed) an element of the array my_uca1400_info_tailored[].
UCA1400 collations with equal character set and tailoring
(but with different level flags) share the same MY_UCA_INFO.
*/
static MY_UCA_INFO *
my_uca1400_collation_get_initialized_shared_uca(MY_CHARSET_LOADER *loader,
struct charset_info_st *cs,
MY_COLL_RULES *rules,
const MY_UCA_INFO *src_uca,
uint id)
{
my_cs_encoding_t enc= my_uca1400_collation_id_to_charset_id(id);
uint tailoring= my_uca1400_collation_id_to_tailoring_id(id);
MY_UCA_INFO *dst_uca= &my_uca1400_info_tailored[enc][tailoring];
DBUG_ASSERT(my_collation_id_is_uca1400(id));
if (!dst_uca->level[0].weights/*Check if already initialized*/ &&
(my_uca_info_init(loader, dst_uca, rules, cs, src_uca,
(1<<MY_UCA_WEIGHT_LEVELS)-1)))
return NULL; /* EOM or an error in rules */
return dst_uca;
}
/*
This function copies an UCS2 collation from
the default Unicode Collation Algorithm (UCA)
@@ -34789,20 +34736,23 @@ create_tailoring(struct charset_info_st *cs,
my_ci_set_strength(cs, 1);
if (my_collation_id_is_uca1400(cs->number))
if (my_collation_id_is_mysql_uca0900(cs->number))
{
/*
UCA1400 collations with equal character set and tailoring
(but with different level flags) share the same MY_UCA_INFO.
*/
my_cs_encoding_t enc= my_uca1400_collation_id_to_charset_id(cs->number);
uint tailoring= my_uca1400_collation_id_to_tailoring_id(cs->number);
MY_UCA_INFO *dst_uca= &my_uca1400_info_tailored[enc][tailoring];
if (!dst_uca->level[0].weights &&
(rc= my_uca_info_init(loader, dst_uca, &rules, cs, src_uca,
(1<<MY_UCA_WEIGHT_LEVELS)-1)))
goto ex;
cs->uca= dst_uca;
uint id1400= mysql_0900_mapping[cs->number - mysql_0900_collation_start].
collation_id;
if (!(cs->uca= my_uca1400_collation_get_initialized_shared_uca(loader, cs,
&rules,
src_uca,
id1400)))
goto ex;
}
else if (my_collation_id_is_uca1400(cs->number))
{
if (!(cs->uca= my_uca1400_collation_get_initialized_shared_uca(loader, cs,
&rules,
src_uca,
cs->number)))
goto ex;
}
else
{
@@ -39397,122 +39347,6 @@ struct charset_info_st my_charset_utf16_unicode_520_nopad_ci=
#endif /* HAVE_CHARSET_utf16 */
uint
my_uca1400_make_builtin_collation_id(my_cs_encoding_t charset_id,
uint tailoring_id,
my_bool nopad,
my_bool secondary_level,
my_bool tertiary_level)
{
if (!my_uca1400_collation_definitions[tailoring_id].tailoring)
return 0;
return MY_UCA1400_COLLATION_ID_POSSIBLE_MIN +
(charset_id << 8) +
(tailoring_id << 3) +
(nopad << 2) +
(secondary_level << 1) +
(tertiary_level << 0);
}
my_bool
my_uca1400_collation_definition_init(MY_CHARSET_LOADER *loader,
struct charset_info_st *dst,
uint id)
{
my_cs_encoding_t cs_id= my_uca1400_collation_id_to_charset_id(id);
uint tailoring_id= my_uca1400_collation_id_to_tailoring_id(id);
my_bool nopad= my_uca1400_collation_id_to_nopad_flag(id);
my_bool secondary_level= my_uca1400_collation_id_to_secondary_level_flag(id);
my_bool tertiary_level= my_uca1400_collation_id_to_tertiary_level_flag(id);
const MY_UCA1400_COLLATION_DEFINITION *def=
&my_uca1400_collation_definitions[tailoring_id];
char tmp[128], *coll_name;
size_t length;
switch (cs_id) {
case MY_CS_ENCODING_UTF8MB3:
*dst= nopad ? my_charset_utf8mb3_unicode_520_nopad_ci :
my_charset_utf8mb3_unicode_520_ci;
break;
case MY_CS_ENCODING_UTF8MB4:
*dst= nopad ? my_charset_utf8mb4_unicode_520_nopad_ci :
my_charset_utf8mb4_unicode_520_ci;
break;
#ifdef HAVE_CHARSET_ucs2
case MY_CS_ENCODING_UCS2:
*dst= nopad ? my_charset_ucs2_unicode_520_nopad_ci :
my_charset_ucs2_unicode_520_ci;
break;
#endif
#ifdef HAVE_CHARSET_utf16
case MY_CS_ENCODING_UTF16:
*dst= nopad ? my_charset_utf16_unicode_520_nopad_ci :
my_charset_utf16_unicode_520_ci;
break;
#endif
#ifdef HAVE_CHARSET_utf32
case MY_CS_ENCODING_UTF32:
*dst= nopad ? my_charset_utf32_unicode_520_nopad_ci :
my_charset_utf32_unicode_520_ci;
break;
#endif
}
dst->number= id;
dst->uca= &my_uca_v1400;
dst->tailoring= def->tailoring;
if (def->tailoring == turkish)
dst->casefold= &my_casefold_unicode1400tr;
else
dst->casefold= &my_casefold_unicode1400;
if (nopad)
dst->state|= MY_CS_NOPAD;
my_ci_set_level_flags(dst, (1 << MY_CS_LEVEL_BIT_PRIMARY) |
(secondary_level ?
1 << MY_CS_LEVEL_BIT_SECONDARY : 0) |
(tertiary_level ?
1 << MY_CS_LEVEL_BIT_TERTIARY : 0));
length= my_snprintf(tmp, sizeof(tmp), "%.*s_uca1400%s%s%s%s%s",
(int) dst->cs_name.length, dst->cs_name.str,
def->name[0] ? "_" : "",
def->name,
nopad ? "_nopad" : "",
secondary_level ? "_as" : "_ai",
tertiary_level ? "_cs" : "_ci");
if (!(coll_name= loader->once_alloc(length + 1)))
return TRUE;
strcpy(coll_name, tmp);
dst->coll_name.str= coll_name;
dst->coll_name.length= length;
return FALSE;
}
/*
Return UCA-4.0.0 compatible ID, e.g. for use in the protocol
with the old clients.
*/
static uint my_uca1400_collation_id_uca400_compat(uint id)
{
uint tlid= my_uca1400_collation_id_to_tailoring_id(id);
my_cs_encoding_t csid= my_uca1400_collation_id_to_charset_id(id);
MY_UCA1400_COLLATION_DEFINITION *def;
DBUG_ASSERT(my_collation_id_is_uca1400(id));
if (!(def= &my_uca1400_collation_definitions[tlid])->name)
return id;
switch (csid) {
case MY_CS_ENCODING_UTF8MB3: return def->id_utf8mb3;
case MY_CS_ENCODING_UTF8MB4: return def->id_utf8mb4;
case MY_CS_ENCODING_UCS2: return def->id_ucs2;
case MY_CS_ENCODING_UTF16: return def->id_utf16;
case MY_CS_ENCODING_UTF32: return def->id_utf32;
}
return id;
}
uint my_ci_get_id_uca(CHARSET_INFO *cs, my_collation_id_type_t type)
{
switch (type)
@@ -39532,23 +39366,6 @@ uint my_ci_get_id_uca(CHARSET_INFO *cs, my_collation_id_type_t type)
}
LEX_CSTRING my_ci_get_collation_name_uca1400_context(CHARSET_INFO *cs)
{
LEX_CSTRING res;
DBUG_ASSERT(my_collation_id_is_uca1400(cs->number));
if (cs->coll_name.length <= cs->cs_name.length ||
cs->coll_name.str[cs->cs_name.length] != '_')
{
DBUG_ASSERT(0);
return cs->coll_name;
}
res.str= cs->coll_name.str + cs->cs_name.length + 1;
res.length= cs->coll_name.length - cs->cs_name.length - 1;
return res;
}
LEX_CSTRING my_ci_get_collation_name_uca(CHARSET_INFO *cs,
my_collation_name_mode_t mode)
{
@@ -39565,137 +39382,4 @@ LEX_CSTRING my_ci_get_collation_name_uca(CHARSET_INFO *cs,
return cs->coll_name;
}
/*
Add support for MySQL 8.0 utf8mb4_0900_.. collations
The collation id's where collected from fprintf() in add_alias_for_collation()
*/
#define mysql_0900_collation_start 255
struct mysql_0900_to_mariadb_1400_mapping
{
const char *mysql_col_name, *mariadb_col_name, *case_sensitivity;
uint collation_id;
};
struct mysql_0900_to_mariadb_1400_mapping mysql_0900_mapping[]=
{
/* 255 Ascent insensitive, Case insensitive 'ai_ci' */
{"", "", "ai_ci", 2308},
{"de_pb", "german2", "ai_ci", 2468},
{"is", "icelandic", "ai_ci", 2316},
{"lv", "latvian", "ai_ci", 2324},
{"ro", "romanian", "ai_ci", 2332},
{"sl", "slovenian", "ai_ci", 2340},
{"pl", "polish", "ai_ci", 2348},
{"et", "estonian", "ai_ci", 2356},
{"es", "spanish", "ai_ci", 2364},
{"sv", "swedish", "ai_ci", 2372},
{"tr", "turkish", "ai_ci", 2380},
{"cs", "czech", "ai_ci", 2388},
{"da", "danish", "ai_ci", 2396},
{"lt", "lithuanian", "ai_ci", 2404},
{"sk", "slovak", "ai_ci", 2412},
{"es_trad", "spanish2", "ai_ci", 2420},
{"la", "roman", "ai_ci", 2428},
{"fa", NullS, "ai_ci", 0}, // Disabled in MySQL
{"eo", "esperanto", "ai_ci", 2444},
{"hu", "hungarian", "ai_ci", 2452},
{"hr", "croatian", "ai_ci", 2500},
{"si", NullS, "ai_ci", 0}, // Disabled in MySQL
{"vi", "vietnamese", "ai_ci", 2492},
/* 278 Ascent sensitive, Case sensitive 'as_cs' */
{"","", "as_cs", 2311},
{"de_pb", "german2", "as_cs", 2471},
{"is", "icelandic", "as_cs", 2319},
{"lv", "latvian", "as_cs", 2327},
{"ro", "romanian", "as_cs", 2335},
{"sl", "slovenian", "as_cs", 2343},
{"pl", "polish", "as_cs", 2351},
{"et", "estonian", "as_cs", 2359},
{"es", "spanish", "as_cs", 2367},
{"sv", "swedish", "as_cs", 2375},
{"tr", "turkish", "as_cs", 2383},
{"cs", "czech", "as_cs", 2391},
{"da", "danish", "as_cs", 2399},
{"lt", "lithuanian", "as_cs", 2407},
{"sk", "slovak", "as_cs", 2415},
{"es_trad", "spanish2", "as_cs", 2423},
{"la", "roman", "as_cs", 2431},
{"fa", NullS, "as_cs", 0}, // Disabled in MySQL
{"eo", "esperanto", "as_cs", 2447},
{"hu", "hungarian", "as_cs", 2455},
{"hr", "croatian", "as_cs", 2503},
{"si", NullS, "as_cs", 0}, // Disabled in MySQL
{"vi", "vietnamese", "as_cs", 2495},
{"", NullS, "as_cs", 0}, // Missing
{"", NullS, "as_cs", 0}, // Missing
{"_ja_0900_as_cs", NullS, "as_cs", 0}, // Not supported
{"_ja_0900_as_cs_ks", NullS, "as_cs", 0}, // Not supported
/* 305 Ascent-sensitive, Case insensitive 'as_ci' */
{"","", "as_ci", 2310},
{"ru", NullS, "ai_ci", 0}, // Not supported
{"ru", NullS, "as_cs", 0}, // Not supported
{"zh", NullS, "as_cs", 0}, // Not supported
{NullS, NullS, "", 0}
};
static LEX_CSTRING
mysql_utf8mb4_0900_bin= {STRING_WITH_LEN("utf8mb4_0900_bin")},
mariadb_utf8mb4_nopad_bin= {STRING_WITH_LEN("utf8mb4_nopad_bin")};
/*
Map mysql character sets to MariaDB using the same definition but with
with the MySQL collation name and id.
*/
my_bool mysql_utf8mb4_0900_collation_definitions_add()
{
uint id= mysql_0900_collation_start;
struct mysql_0900_to_mariadb_1400_mapping *map;
for (map= mysql_0900_mapping; map->mysql_col_name ; map++, id++)
{
if (map->mariadb_col_name) /* Supported collation */
{
size_t org_length, ali_length;
char original[64], alias[64];
LEX_CSTRING org_name, alias_name;
org_length= (strxnmov(original, sizeof(original)-1,
"utf8mb4_uca1400_",
map->mariadb_col_name,
(map->mariadb_col_name[0] ? "_" : ""),
"nopad_",
map->case_sensitivity,
NullS) - original);
ali_length= (strxnmov(alias, sizeof(alias)-1,
"utf8mb4_", map->mysql_col_name,
(map->mysql_col_name[0] ? "_" : ""),
"0900_",
map->case_sensitivity,
NullS) - alias);
org_name.str= original;
org_name.length= org_length;
alias_name.str= alias;
alias_name.length= ali_length;
if (add_alias_for_collation(&org_name, map->collation_id, &alias_name,
id))
return 1;
}
}
if (add_alias_for_collation(&mariadb_utf8mb4_nopad_bin, 1070,
&mysql_utf8mb4_0900_bin, 309))
return 1;
return 0;
}
#endif /* HAVE_UCA_COLLATIONS */