ffc817f040
A TEXT value longer than `group_concat_max_len` is cut on its way into `blob_storage`, in `Field_blob::handle_group_concat()`. That happens while the group is being built, not when the answer is put together, so whether the answer is any shorter for it depends on whether the row carrying the value reaches the answer at all. Every such cut was reported as `ER_CUT_VALUE_GROUP_CONCAT`, which says that the answer lost something the user asked for, and that is true of only some of them. `Blob_mem_storage` now writes one byte in front of every value it stores and returns the pointer past it, so `was_cut()` answers for any value a reader holds a pointer to. `Field_blob::store()` sends every blob of a table that has a `Blob_mem_storage` through `handle_group_concat()`, so no value in that storage is without the byte, and `Field_blob::get_ptr()` on the record of a row hands back exactly the pointer that was stored. `dump_leaf_key()` reads the mark off each row it appends and sets `value_cut_in_result`. `val_str()` then reports: 1. **A warning**, `ER_CUT_VALUE_GROUP_CONCAT`, when a row that reached the answer carried a cut value. The answer is short by what was cut. The result being cut at `gconcat_max_len()` already gives that same warning, and a group that hits both is told once, not twice. 2. **A note**, `ER_CUT_VALUES_WHILE_PROCESSING`, when a value was cut but no row carrying one reached the answer. The answer may well be what a larger limit would have given. One note per aggregate is enough for a statement however many groups had a value cut, and `cleanup()` clears the mark so a statement run again gets its own. Reporting the loss as a warning keeps a strict `sql_mode` aborting on it, which it does because `THD::raise_condition()` promotes a warning and never promotes a note. `ST_COLLECT` is not affected. It reports `ER_CUT_VALUE_GROUP_CONCAT` itself, against `group_collect_max_len`. `main.gconcat_cut_note` covers the split with one group holding a short value and a long one, where a `LIMIT` alone decides which of them the answer is built from, over both the sort tree and the duplicate filter. `main.func_gconcat` shows the granularity: of five groups at `group_concat_max_len=499999`, the one holding exactly 499999 bytes is the one that does not warn. Note that `blob_storage` only exists when the aggregate has an `ORDER BY` or a `DISTINCT` and a blob field, so this is the only shape in which a value is cut this way.
A quicker way for adding new language translations to the errmsg-utf8.txt file
Summary
To generate a new language translation of MariaDB use the following pull request (PR) as a template for your work:
You will notice as part of your translation work, you will have to add your language translations to the file sql/share/errmsg-utf8.txt which is found in the current directory. This file is long with many sections which can make the translation work tedious. In this README, we explain a procedure and provide a script insert_translations_into_errmsg.py that cuts down the amount of tedium in accomplishing the task.
Procedure
-
Start by grepping out all the english translations from errmsg-utf8.txt using the following grep command, and redirecting the output to a file:
grep -P "^\s*eng\s" errmsg-utf8.txt > all_english_text_in_errmsg-utf8.txt
-
Next use Google translate to obtain a translation of this file. Google translate provides the ability to upload whole files for translation. For example, this technique was used to obtain Swahili translations which yielded a file with output similar to the below (output is truncated for clarity):
sw "hashchk" sw "isamchk" sw "LA" sw "NDIYO" sw "Haiwezi kuunda faili '% -.200s' (kosa: %iE)" sw "Haiwezi kuunda jedwali %
s.%s (kosa: %iE)" sw "Haiwezi kuunda hifadhidata '% -.192s' (kosa: %iE)" sw "Haiwezi kuunda hifadhidata '% -.192s'; hifadhidata ipo"
Note that Google translate removes the leading whitespace in the translation file it generates. DO NOT add that leading whitespace back!
-
Give the translated file an appropriate name (e.g.
all_swahili_text_in_errmsg-utf8.txt) and store it in the same directory witherrmsg-utf8.txtandall_english_text_in_errmsg-utf8.txt. These 3 files will be used by the script insert_translations_into_errmsg.py. -
Proof check the auto-translations in the file you downloaded from Google translate. Note that Google might omit formatting information that will cause the compilation of MariaDB to fail, so pay attention to these.
-
Reintegrate these translations into the errmsg-utf8.txt by running the insert_translations_into_errmsg.py script as follows:
chmod ugo+x insert_translations_into_errmsg.py # Make the script executable if it is not.
./insert_translations_into_errmsg.py <errmsg-utf8.txt file>
For example, for the swahili translation, we ran the following:
./insert_translations_into_errmsg.py errmsg-utf8.txt all_english_text_in_errmsg-utf8.txt all_swahili_text_in_errmsg-utf8.txt
The script uses the
errmsg-utf8.txtfile and the grepped english file to keep track of each new translation. It then creates a file in the same directory aserrmsg-utf8.txtwith the nameerrmsg-utf8-with-new-language.txt. -
Check that the reintegration of the new translations into
errmsg-utf8-with-new-language.txtwent OK, and if it did, renameerrmsg-utf8-with-new-language.txttoerrmsg-utf8.txt:mv errmsg-utf8-with-new-language.txt errmsg-utf8.txt
-
In the header of errmsg-utf8.txt make sure to add your language long form to short form mapping. E.g. for Swahili, add:
swahili=sw