If connect engineis not able to allocate connect_work_space memory for
GetUser() it will call free() twice with the same value (g).
g was freed first in user_connect::user_init() which calls PlugExit() on
errors and then again in ~user_connect() which also calls PlugExit().
Fixed by setting g to 0 in user_init() after calling PlugExit()
This code was tested 'by hand' by setting connect.work_space=600G
Other things:
- Removed some very old not relevant comments in touched code
- Added comments to clarify how some memory was freed
- Fixed indentation in changed functions.
This patch also fixes:
MDEV-33050 Build-in schemas like oracle_schema are accent insensitive
MDEV-33084 LASTVAL(t1) and LASTVAL(T1) do not work well with lower-case-table-names=0
MDEV-33085 Tables T1 and t1 do not work well with ENGINE=CSV and lower-case-table-names=0
MDEV-33086 SHOW OPEN TABLES IN DB1 -- is case insensitive with lower-case-table-names=0
MDEV-33088 Cannot create triggers in the database `MYSQL`
MDEV-33103 LOCK TABLE t1 AS t2 -- alias is not case sensitive with lower-case-table-names=0
MDEV-33109 DROP DATABASE MYSQL -- does not drop SP with lower-case-table-names=0
MDEV-33110 HANDLER commands are case insensitive with lower-case-table-names=0
MDEV-33119 User is case insensitive in INFORMATION_SCHEMA.VIEWS
MDEV-33120 System log table names are case insensitive with lower-cast-table-names=0
- Removing the virtual function strnncoll() from MY_COLLATION_HANDLER
- Adding a wrapper function CHARSET_INFO::streq(), to compare
two strings for equality. For now it calls strnncoll() internally.
In the future it will turn into a virtual function.
- Adding new accent sensitive case insensitive collations:
- utf8mb4_general1400_as_ci
- utf8mb3_general1400_as_ci
They implement accent sensitive case insensitive comparison.
The weight of a character is equal to the code point of its
upper case variant. These collations use Unicode-14.0.0 casefolding data.
The result of
my_charset_utf8mb3_general1400_as_ci.strcoll()
is very close to the former
my_charset_utf8mb3_general_ci.strcasecmp()
There is only a difference in a couple dozen rare characters, because:
- the switch from "tolower" to "toupper" comparison, to make
utf8mb3_general1400_as_ci closer to utf8mb3_general_ci
- the switch from Unicode-3.0.0 to Unicode-14.0.0
This difference should be tolarable. See the list of affected
characters in the MDEV description.
Note, utf8mb4_general1400_as_ci correctly handles non-BMP characters!
Unlike utf8mb4_general_ci, it does not treat all BMP characters
as equal.
- Adding classes representing names of the file based database objects:
Lex_ident_db
Lex_ident_table
Lex_ident_trigger
Their comparison collation depends on the underlying
file system case sensitivity and on --lower-case-table-names
and can be either my_charset_bin or my_charset_utf8mb3_general1400_as_ci.
- Adding classes representing names of other database objects,
whose names have case insensitive comparison style,
using my_charset_utf8mb3_general1400_as_ci:
Lex_ident_column
Lex_ident_sys_var
Lex_ident_user_var
Lex_ident_sp_var
Lex_ident_ps
Lex_ident_i_s_table
Lex_ident_window
Lex_ident_func
Lex_ident_partition
Lex_ident_with_element
Lex_ident_rpl_filter
Lex_ident_master_info
Lex_ident_host
Lex_ident_locale
Lex_ident_plugin
Lex_ident_engine
Lex_ident_server
Lex_ident_savepoint
Lex_ident_charset
engine_option_value::Name
- All the mentioned Lex_ident_xxx classes implement a method streq():
if (ident1.streq(ident2))
do_equal();
This method works as a wrapper for CHARSET_INFO::streq().
- Changing a lot of "LEX_CSTRING name" to "Lex_ident_xxx name"
in class members and in function/method parameters.
- Replacing all calls like
system_charset_info->coll->strcasecmp(ident1, ident2)
to
ident1.streq(ident2)
- Taking advantage of the c++11 user defined literal operator
for LEX_CSTRING (see m_strings.h) and Lex_ident_xxx (see lex_ident.h)
data types. Use example:
const Lex_ident_column primary_key_name= "PRIMARY"_Lex_ident_column;
is now a shorter version of:
const Lex_ident_column primary_key_name=
Lex_ident_column({STRING_WITH_LEN("PRIMARY")});
Under terms of MDEV 27490 we'll add support for non-BMP identifiers
and upgrade casefolding information to Unicode version 14.0.0.
In Unicode-14.0.0 conversion to lower and upper cases can increase octet length
of the string, so conversion won't be possible in-place any more.
This patch removes virtual functions performing in-place casefolding:
- my_charset_handler_st::casedn_str()
- my_charset_handler_st::caseup_str()
and fixes the code to use the non-inplace functions instead:
- my_charset_handler_st::casedn()
- my_charset_handler_st::caseup()
clang15 finally errors on old prototype definations.
Its also a lot fussier about variables that aren't used
as is the case a number of time with loop counters that
aren't examined.
RocksDB was complaining that its get_range function was
declared without the array length in ha_rocksdb.h. While
a constant is used rather than trying to import the
Rdb_key_def::INDEX_NUMBER_SIZE header (was causing a lot of
errors on the defination of other orders). If the constant
does change can be assured that the same compile warnings will
tell us of the error.
The ha_rocksdb::index_read_map_impl DBUG_EXECUTE_IF was similar
to the existing endless functions used in replication tests.
Its rather moot point as the rocksdb.force_shutdown test that
uses myrocks_busy_loop_on_row_read is currently disabled.
* MDEV-29426 Fix memory leak in CONNECT JSON/BSON
If information is being gathered on JSON/BSON tables we leak the heap
allocated to the table class. With this fix we close and cleanup
instead, just as we do for XML tables.
This commit replaces sprintf(buf, ...) with
snprintf(buf, sizeof(buf), ...),
specifically in the "easy" cases where buf is allocated with a size
known at compile time.
The changes make sure we are not write outside array/string bounds which
will lead to undefined behaviour. In case the code is trying to write
outside bounds - safe version of functions simply cut the string
messages so we process this gracefully.
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the BSD-new
license. I am contributing on behalf of my employer Amazon Web Services,
Inc.
bsonudf.cpp warnings cleanup by Daniel Black
Reviewer: Daniel Black
The easiest way to compile and test the server with UBSAN is to run:
./BUILD/compile-pentium64-ubsan
and then run mysql-test-run.
After this commit, one should be able to run this without any UBSAN
warnings. There is still a few compiler warnings that should be fixed
at some point, but these do not expose any real bugs.
The 'special' cases where we disable, suppress or circumvent UBSAN are:
- ref10 source (as here we intentionally do some shifts that UBSAN
complains about.
- x86 version of optimized int#korr() methods. UBSAN do not like unaligned
memory access of integers. Fixed by using byte_order_generic.h when
compiling with UBSAN
- We use smaller thread stack with ASAN and UBSAN, which forced me to
disable a few tests that prints the thread stack size.
- Verifying class types does not work for shared libraries. I added
suppression in mysql-test-run.pl for this case.
- Added '#ifdef WITH_UBSAN' when using integer arithmetic where it is
safe to have overflows (two cases, in item_func.cc).
Things fixed:
- Don't left shift signed values
(byte_order_generic.h, mysqltest.c, item_sum.cc and many more)
- Don't assign not non existing values to enum variables.
- Ensure that bool and enum values are properly initialized in
constructors. This was needed as UBSAN checks that these types has
correct values when one copies an object.
(gcalc_tools.h, ha_partition.cc, item_sum.cc, partition_element.h ...)
- Ensure we do not called handler functions on unallocated objects or
deleted objects.
(events.cc, sql_acl.cc).
- Fixed bugs in Item_sp::Item_sp() where we did not call constructor
on Query_arena object.
- Fixed several cast of objects to an incompatible class!
(Item.cc, Item_buff.cc, item_timefunc.cc, opt_subselect.cc, sql_acl.cc,
sql_select.cc ...)
- Ensure we do not do integer arithmetic that causes over or underflows.
This includes also ++ and -- of integers.
(Item_func.cc, Item_strfunc.cc, item_timefunc.cc, sql_base.cc ...)
- Added JSON_VALUE_UNITIALIZED to json_value_types and ensure that
value_type is initialized to this instead of to -1, which is not a valid
enum value for json_value_types.
- Ensure we do not call memcpy() when second argument could be null.
- Fixed that Item_func_str::make_empty_result() creates an empty string
instead of a null string (safer as it ensures we do not do arithmetic
on null strings).
Other things:
- Changed struct st_position to an OBJECT and added an initialization
function to it to ensure that we do not copy or use uninitialized
members. The change to a class was also motived that we used "struct
st_position" and POSITION randomly trough the code which was
confusing.
- Notably big rewrite in sql_acl.cc to avoid using deleted objects.
- Changed in sql_partition to use '^' instead of '-'. This is safe as
the operator is either 0 or 0x8000000000000000ULL.
- Added check for select_nr < INT_MAX in JOIN::build_explain() to
avoid bug when get_select() could return NULL.
- Reordered elements in POSITION for better alignment.
- Changed sql_test.cc::print_plan() to use pointers instead of objects.
- Fixed bug in find_set() where could could execute '1 << -1'.
- Added variable have_sanitizer, used by mtr. (This variable was before
only in 10.5 and up). It can now have one of two values:
ASAN or UBSAN.
- Moved ~Archive_share() from ha_archive.cc to ha_archive.h and marked
it virtual. This was an effort to get UBSAN to work with loaded storage
engines. I kept the change as the new place is better.
- Added in CONNECT engine COLBLK::SetName(), to get around a wrong cast
in tabutil.cpp.
- Added HAVE_REPLICATION around usage of rgi_slave, to get embedded
server to compile with UBSAN. (Patch from Marko).
- Added #ifdef for powerpc64 to avoid a bug in old gcc versions related
to integer arithmetic.
Changes that should not be needed but had to be done to suppress warnings
from UBSAN:
- Added static_cast<<uint16_t>> around shift to get rid of a LOT of
compiler warnings when using UBSAN.
- Had to change some '/' of 2 base integers to shift to get rid of
some compile time warnings.
Reviewed by:
- Json changes: Alexey Botchkov
- Charset changes in ctype-uca.c: Alexander Barkov
- InnoDB changes & Embedded server: Marko Mäkelä
- sql_acl.cc changes: Vicențiu Ciorbaru
- build_explain() changes: Sergey Petrunia
Just keep the /MDd flag but only for Windows Debug REST compile
modified: storage/connect/CMakeLists.txt
- Add a trace to EvalColumns
modified: storage/connect/connect.cc
- Remove a file not belonging to CONNECT
deleted: rest.h
- Subfolder Option: SELECT Query Never Ends
modified: storage/connect/tabmul.cpp
modified: storage/connect/tabmul.h
Work on MDEV-12667 Crash when using JSON tables
modified: storage/connect/connect.cc
modified: storage/connect/ha_connect.cc
modified: storage/connect/ha_connect.h
modified: storage/connect/plgdbutl.cpp
Change Base offset for DIR tables on Linux
modified: storage/connect/reldef.cpp
table (ODBC, JDBC, MYSQL) with a WHERE clause on an indexed column.
Also fix a bugs in TDBEXT::MakeCommand (use of uninitialised Quote)
Add in this function the eventual Schema (database) prefixing.
modified: storage/connect/connect.cc
modified: storage/connect/tabext.cpp
Typo
modified: storage/connect/tabjdbc.h
when joining a table to a MYSQL indexed table.
modified: storage/connect/myconn.cpp
modified: storage/connect/myconn.h
modified: storage/connect/tabmysql.cpp
- Add more tests to the mysql_index.test file
modified: storage/connect/mysql-test/connect/r/mysql_index.result
modified: storage/connect/mysql-test/connect/t/mysql_index.test
- Fix and error causing remote indexing to fail when for not unique index.
Was experienced with MYSQL, ODBC and JDBC tables.
modified: storage/connect/connect.cc
- Fix MDEV-9966 (zero lines returned)
modified: storage/connect/ha_connect.cc
modified: storage/federatedx/ha_federatedx.cc
Swap the key length when WORDS_BIGENDIAN is defined
Make the IOFF structure depending on WORDS_BIGENDIAN
modified: storage/connect/connect.cc
modified: storage/connect/xindex.h
- Added JSON OBJECT specification for pretty != 2.
- Fix NULL values not recognized for nullable JSON columns
- Issue an error message when a JSON table is created without specifying LRECL if PRETTY != 2.
- Make JSONColumns use a TDBJSON class.
- Make JSON table using MAPFAM
modified:
filamap.h
filamtxt.h
ha_connect.cc
json.result
tabjson.cpp
tabjson.h
table.cpp
- Implementing Discovery for the XML table type.
modified:
domdoc.cpp
domdoc.h
ha_connect.cc
libdoc.cpp
plgxml.cpp
plgxml.h
reldef.cpp
reldef.h
tabxml.cpp
tabxml.h
- Providing an error message when creating an ODBC table via discovery returns
columns of more than one table.
modified:
ha_connect.cc
- TableOptionStruct declaration moved from ha_connect.h to mycat.h
To make it easier to use by other classes.
modified:
ha_connect.cc
ha_connect.h
mycat.cc
mycat.h
reldef.cpp
tabmysql.cpp
taboccur.cpp
tabpivot.cpp
tabtbl.cpp
tabutil.cpp
tabxcl.cpp
tables. This should fix bug MDEV-7343.
modified:
storage/connect/ha_connect.cc
storage/connect/myconn.cpp
storage/connect/myconn.h
storage/connect/reldef.cpp
storage/connect/reldef.h
storage/connect/table.cpp
storage/connect/tabmysql.cpp
storage/connect/xtable.h
- Prevent double column evaluation when CONNECT does filtering
modified:
storage/connect/connect.cc
- Export CreateFileMap and CloseMemMap (for OEM tables)
modified:
storage/connect/maputil.h
- Add the compute function to be used on VALUE types.
Preserve precision for DOUBLE values.
modified:
storage/connect/value.cpp
storage/connect/value.h
- Typo (in preparation to the future JSON table type)
modified:
storage/connect/ha_connect.cc
storage/connect/mycat.cc
storage/connect/plgdbsem.h
They are still experimental and should not be used in production.
added:
storage/connect/encas.h
storage/connect/english.msg
storage/connect/enids.h
storage/connect/frcas.h
storage/connect/french.msg
storage/connect/frids.h
storage/connect/frmsg.h
modified:
storage/connect/connect.cc
storage/connect/engmsg.h
storage/connect/filamdbf.cpp
storage/connect/global.h
storage/connect/ha_connect.cc
storage/connect/msgid.h
storage/connect/plgdbutl.cpp
storage/connect/plugutil.c
storage/connect/rcmsg.c
storage/connect/resource.h
storage/connect/tabfmt.h
- Fix global variable not being properly initialized (MDEV-6690, MDEV-7094)
modified:
storage/connect/ha_connect.cc
storage/connect/plugutil.c
storage/connect/user_connect.cc
storage/connect/xindex.cpp
- Implement Rewind for ODBC tables (MDEV-7097)
modified:
storage/connect/odbconn.cpp
storage/connect/odbconn.h
storage/connect/tabmysql.cpp
storage/connect/tabodbc.cpp
- Reset N when reopening MYSQL tables
modified:
storage/connect/tabmysql.cpp
modified:
storage/connect/connect.cc
storage/connect/ha_connect.cc
storage/connect/ha_connect.h
storage/connect/mycat.cc
storage/connect/plgdbsem.h
- Get good message when calling ColDB
modified:
storage/connect/connect.cc
- Fix buffer preparation for BIN files
modified:
storage/connect/filamfix.cpp
fix error while updating (force fseek)
modified:
storage/connect/filamfix.cpp
fix error of XCOL column when filtered (typo)
modified:
storage/connect/tabdos.cpp
storage/connect/tabxcl.cpp
fix error when indexing on special column
modified:
storage/connect/tabdos.cpp
was not reset causing rows to be lost.
modified:
storage/connect/connect.cc
storage/connect/ha_connect.cc
storage/connect/tabxcl.cpp
- Typo
modified:
storage/connect/connect.cc
storage/connect/ha_connect.cc
the beginning. Defining the STRING class and begining to use it (MYSQL)
2) Change the xtrace, use_tempfile and exact_info connect variables from
GLOBAL to SESSION. Remaining GLOBAL variables have been made readonly.
3) Take care of LEX_STRING variables. The .str should not be regarded as
allways being 0 terminated. This is handled by the Strz functions that
make sure to return 0 terminated strings.
Bug fix:
- When inserting in MYSQL table with special column(s) a query such as:
insert into t2 values(0,4,'new04'),(0,5,'new05');
failed saying: column id (the special column) not found in t2.
It is now accepted but must be counted in values (these 0 are ignored)
- ROWID was returning row numbers based 0. Now it is from base 1.
modified:
storage/connect/array.cpp
storage/connect/blkfil.cpp
storage/connect/colblk.cpp
storage/connect/connect.cc
storage/connect/filamap.cpp
storage/connect/filamdbf.cpp
storage/connect/filamfix.cpp
storage/connect/filamtxt.cpp
storage/connect/filamvct.cpp
storage/connect/filamzip.cpp
storage/connect/filamzip.h
storage/connect/filter.cpp
storage/connect/global.h
storage/connect/ha_connect.cc
storage/connect/ha_connect.h
storage/connect/libdoc.cpp
storage/connect/mycat.cc
storage/connect/myconn.cpp
storage/connect/odbconn.cpp
storage/connect/plgdbutl.cpp
storage/connect/plugutil.c
storage/connect/reldef.cpp
storage/connect/tabcol.cpp
storage/connect/tabdos.cpp
storage/connect/tabfix.cpp
storage/connect/tabfmt.cpp
storage/connect/table.cpp
storage/connect/tabmul.cpp
storage/connect/tabmysql.cpp
storage/connect/tabmysql.h
storage/connect/taboccur.cpp
storage/connect/tabodbc.cpp
storage/connect/tabpivot.cpp
storage/connect/tabsys.cpp
storage/connect/tabtbl.cpp
storage/connect/tabutil.cpp
storage/connect/tabvct.cpp
storage/connect/tabwmi.cpp
storage/connect/tabwmi.h
storage/connect/tabxcl.cpp
storage/connect/tabxml.cpp
storage/connect/user_connect.cc
storage/connect/valblk.cpp
storage/connect/value.cpp
storage/connect/value.h
storage/connect/xindex.cpp
storage/connect/xobject.cpp
storage/connect/xobject.h
storage/connect/xtable.h
it belongs to each tables and the catalog being share between several
instances of CONNECT, when a query implied several tables belonging to
different databases, some where pointing on the wrong database. This
fix bugs occuring in queries such as:
INSERT into db1.t1 select * from db2.t2;
Where the t1 data file was made in db2.
modified:
storage/connect/catalog.h
storage/connect/connect.cc
storage/connect/filamdbf.cpp
storage/connect/filamdbf.h
storage/connect/ha_connect.cc
storage/connect/ha_connect.h
storage/connect/mycat.cc
storage/connect/mycat.h
storage/connect/plgdbsem.h
storage/connect/plgdbutl.cpp
storage/connect/reldef.cpp
storage/connect/reldef.h
storage/connect/tabfix.h
storage/connect/tabfmt.cpp
storage/connect/tabfmt.h
storage/connect/tabmul.cpp
a temporary file should be used for UPDATE/DELETE of file tables.
Also use the "sorted" argument of index_init to help decide if
sorting of positions must be done.
modified:
storage/connect/checklvl.h
storage/connect/connect.cc
storage/connect/connect.h
storage/connect/filamdbf.cpp
storage/connect/filamfix.cpp
storage/connect/filamfix.h
storage/connect/filamtxt.cpp
storage/connect/ha_connect.cc
storage/connect/mysql-test/connect/r/part_table.result
storage/connect/plgdbsem.h
storage/connect/plgdbutl.cpp
storage/connect/reldef.cpp
storage/connect/tabdos.cpp
storage/connect/tabdos.h
storage/connect/tabfix.cpp
storage/connect/tabfmt.cpp
storage/connect/tabvct.cpp
storage/connect/tabvct.h
storage/connect/xindex.cpp
- Fix a bug in TDBASE::ColDB that caused some special columns not to
be found in the column list and reallocated without their Value
causing a crash of some queries.
modified:
storage/connect/table.cpp
- Fix a bug causing RestoreNrec to be called before closing a table
causing a wrong value given to Spos
modified:
storage/connect/tabdos.cpp
storage/connect/xindex.cpp
- Add a new CONNECT global variable connect_exact_info. Set to ON, it
tells CONNECT to return exact record numbers on info queries. If OFF
it just gives an estimate. In version 10.0.13 this was unconditionally
ON and caused info queries on remote tables to be extremely long and
was the subject of MDEV-6612.
modified:
storage/connect/ha_connect.cc
storage/connect/tabdos.cpp
storage/connect/tabmysql.cpp
storage/connect/tabodbc.cpp
them sorted by file position. Firstly a new value is stored in indexes
to know if they are sorted, preventing to do the sorting when it is not
needed. Secondly, almost all in now done in connect instead of being
done by the different file access method classes. This pepares the future
use of temporary files for all table types and also fix the bug that was
occuring when partially using a multi-column index because of false MRR
like call of position followed by unsorted rnd_pos no more using indexing.
modified:
storage/connect/connect.cc
storage/connect/filamap.cpp
storage/connect/filamap.h
storage/connect/filamdbf.cpp
storage/connect/filamdbf.h
storage/connect/filamfix.cpp
storage/connect/filamfix.h
storage/connect/filamtxt.cpp
storage/connect/filamtxt.h
storage/connect/filamvct.cpp
storage/connect/filamvct.h
storage/connect/tabdos.cpp
storage/connect/tabdos.h
storage/connect/tabfix.h
storage/connect/tabfmt.cpp
storage/connect/tabfmt.h
storage/connect/xindex.cpp
storage/connect/xindex.h
storage/connect/xtable.h
