CSC default ctor was private b/c it must not allow to use CSC outside thread cache.
However there are some places in the plugin code that need a standalone syscat that
is cleaned up leaving the scope. The decision is to make the restriction mentioned
organizational rather than syntactical.
These seem to have all fallen out of a recent Boost update to 1.81 which
dropped some internal includes. All of these uses within columnstore
relied on these transitive includes, so explicitly include what we need
to fix build.
Signed-off-by: Sam James <sam@gentoo.org>
Analyze needs to be completed differently than a normal query. In server, when an ANALYZE is seen, it calls init_scan() immediatly followed by end_scan(). This leaves the sqlfrontendsession (ExeMgr) in a state where it expects to return rows. This patch fixes end_scan to clean this up via reads and writes to get everything back in synch.
ANALYZE should display the number of rows to be displayed if the query were run normally. We have that information available, but no way to return it. A modification to server side to ask for that in the handler is required.
This patch also includes a beautification of sqlfrontsessionthread.cpp since it looked bad. The important change is at line 774
if (!swallowRows)
which short circuits the actual return of data
* fix C API includes
ColumnStore used to include server's mysql.h
but link all tools with libmariadb.so
There's no guarantee that this would work, even with workarounds
it had in dbcon/mysql/sm.cpp
Fix:
* tools (linked with libmariadb.so) *must* include libmariadb's mysql.h
* as a hack prevent service_thd_timezone.h from being loaded into tools,
as it conflicts with libmariadb's mysql.h
* server plugin *must* include server's mysql.h
* also don't link every tool with libmariadb.so, link the helper library
(liblibmysqlclient.so) that actually needs it, tools use this
helper library, not libmariadb.so directly
* do *not* link ha_columnstore.so with libmariadb.so
this means some libraries have to be compiled twice -
for tools with libmariadb.so and for plugin, without.
* use system boost, if possible
boost 1.71.0 is what ubuntu focal has, so let's start with that version.
boost 1.77.0 is the first that supports c++20
* add dependency for generated header files errorids.h messageids.h
see 3edd51610
* bump the version
* MCOL-5322 This patch replaces boost::mutex with std::mutex b/c IMHO std::unique_lock::lock is
less troublesome comparing with the boost alternative
* MCOL-5310 This patch replaces move-assignment with copy-assignment to avoid memory corruption (#2661)
* Bump VERSION to 22.08.7-1
* MCOL-5306 Re-read the config (Columnstore.xml) file if it was updated.
The existing implementation of Config::makeConfig() factory method
was returning a possibly stale config to the caller, without checking
if the config file was updated since the last read. This bug triggered
a scenario as described in MCOL-5306 where after a failover in an MCS
cluster, the controllernode coordinates changed in the config file
after failover and the existing mariadbd process was still using the
old controllernode coordinates. This lead to failed network connection
between mariadbd and the new controllernode.
The change in this fix, however, is more generic and not just limited
to this above scenario.
* MCOL-5264 This patch replaces boost mutex locks with std analogs
boost::uniqie_lock dtor calls a fancy unlock logic that throws twice.
First if the mutex is 0 and second lock doesn't own the mutex.
The first condition failure causes unhandled exception for one of the clients
in DEC::writeToClient(). I was unable to find out why Linux can have a 0
mutex and replaced boost::mutex with std::mutex b/c stdlibc++ should
be more stable comparing with boost.
* MCOL-5311 Add timezone to jobList in subquerytransformer
TimeZone was uninitialized in this scenario and led to undefined behavior.
* patch_out_of_band
Some changes made to 10.6-enterprise make a build using the out-of-band method of compiling columnstore not work. Out-of band means the source for the engine is not in the storage subdir of server, but rather in a stand alone directory. This is used by developers for easier develop work. In the case of out-of-band, INSTALL_LAYOUT is false in CMakeLists.txt
* MCOL-5346 This patch forces TreeNode::getIntValue to use conversion for dict-based CHAR/VARCHAR and TEXT columns (#2657)
Co-authored-by: Roman Nozdrin <rnozdrin@mariadb.com>
* MCOL-5263 Add support to ROLLBACK when PP were restarted.
DMLProc starts ROLLBACK when SELECT part of UPDATE fails b/c EM facility in PP were restarted.
Unfortunately this ROLLBACK stuck if EM/PP are not yet available.
DMLProc must have a t/o with re-try doing ROLLBACK.
* MCOL-3561 This patch updates Connector code after MDEV-29988
* This commit applies the code style format
Co-authored-by: Sergei Golubchik <serg@mariadb.com>
Co-authored-by: Roman Nozdrin <rnozdrin@mariadb.com>
Co-authored-by: David.Hall <david.hall@mariadb.com>
Co-authored-by: Gagan Goel <gagan.nith@gmail.com>
Co-authored-by: Denis Khalikov <dennis.khalikov@gmail.com>
Disable check for correlated subqueries, basically those types of queries transforms
to join (aggr(table2), table1), table2) and post join scalar filter.
ColumnStore used to include server's mysql.h
but link all tools with libmariadb.so
There's no guarantee that this would work, even with workarounds
it had in dbcon/mysql/sm.cpp
Fix:
* tools (linked with libmariadb.so) *must* include libmariadb's mysql.h
* as a hack prevent service_thd_timezone.h from being loaded into tools,
as it conflicts with libmariadb's mysql.h
* server plugin *must* include server's mysql.h
* also don't link every tool with libmariadb.so, link the helper library
(liblibmysqlclient.so) that actually needs it, tools use this
helper library, not libmariadb.so directly
Exit early from the plugin execution of ALTER TABLE statements
on the replica nodes. This is to prevent re-execution of syscat
table population from the replica nodes which should only be
executed once by the primary node in a CS cluster setup.
The main CmakeLists.txt was using MY_CHECK_AND_SET_COMPILER_FLAG before the include. This works in-band with server because it was already included in server's CmakeLists.txt.
dbcon/mysql included curl as a build dependency. We don't build curl. It's a lib dependency. Not sure why it works in-band. One wouldn't think it should.
The following functions are created:
Create function JSON_VALID and test cases
Create function JSON_DEPTH and test cases
Create function JSON_LENGTH and test cases
Create function JSON_EQUALS and test cases
Create function JSON_NORMALIZE and test cases
Create function JSON_TYPE and test cases
Create function JSON_OBJECT and test cases
Create function JSON_ARRAY and test cases
Create function JSON_KEYS and test cases
Create function JSON_EXISTS and test cases
Create function JSON_QUOTE/JSON_UNQUOTE and test cases
Create function JSON_COMPACT/DETAILED/LOOSE and test cases
Create function JSON_MERGE and test cases
Create function JSON_MERGE_PATCH and test cases
Create function JSON_VALUE and test cases
Create function JSON_QUERY and test cases
Create function JSON_CONTAINS and test cases
Create function JSON_ARRAY_APPEND and test cases
Create function JSON_ARRAY_INSERT and test cases
Create function JSON_INSERT/REPLACE/SET and test cases
Create function JSON_REMOVE and test cases
Create function JSON_CONTAINS_PATH and test cases
Create function JSON_OVERLAPS and test cases
Create function JSON_EXTRACT and test cases
Create function JSON_SEARCH and test cases
Note:
Some functions output differs from MDB because session variables that affects functions output,e.g JSON_QUOTE/JSON_UNQUOTE
This depends on MCOL-5212
in aggregation code
The patch disables padding that forces hasher to calculate over the whole 2k buffer. This patch also moves hashing code
into the common place where it belongs.
* MCOL-5092 Ensure column width is correct for datatype
Change MODA return type to STRING
Modify MODA to handle every numeric type
* MCOL-5162 MODA to support char and varchar with collation support
Fixes to the aggregate bit functions
When we fixed the storage sign issue for MCOL-5092, it uncovered a problem in the bit aggregates (bit_and, bit_or and bit_xor). These aggregates should always return UBIGINT, but they relied on the type of the argument column, which gave bad results.
* MCOL-5074 CASE with IN and aggregate asserts
gwip-scsp wasn't set and buildPredicateItem() was called which assumes it is set. Added code to set properly in this case
Introduced UDF and stored prodecure.
usage:
set columnstore_s3_key='<s3_key>';
set columnstore_s3_secret='<s3_secret>';
set columnstore_s3_region='region';
and then use UDF
select columnstore_dataload("<tablename>", "<filename>", "<bucket>", "<db_name>");
for UDF db_name can be ommited, then current connection db will be used
or stored function
call calpontsys.columnstore_load_from_s3("<tablename>", "<filename>", "<bucket>", "<db_name>");
EM scaleability project has two parts: phase1 and phase2.
This is phase1 that brings EM index to speed up(from O(n) down
to the speed of boost::unordered_map) EM lookups looking for
<dbroot, oid, partition> tuple to turn it into LBID,
e.g. most bulk insertion meta info operations.
The basis is boost::shared_managed_object where EMIndex is
stored. Whilst it is not debug-friendly it allows to put a
nested structs into shmem. EMIndex has 3 tiers. Top down description:
vector of dbroots, map of oids to partition vectors, partition
vectors that have EM indices.
Separate EM methods now queries index before they do EM run.
EMIndex has a separate shmem file with the fixed id
MCS-shm-00060001.
* Fix clang warnings
* Remove vim tab guides
* initialize variables
* 'strncpy' output truncated before terminating nul copying as many bytes from a string as its length
* Fix ISO C++17 does not allow 'register' storage class specifier for outdated bison
* chars are unsigned on ARM, having if (ival < 0) always false
* chars are unsigned by default on ARM and comparison with -1 if always true
Part 1:
As part of MCOL-3776 to address synchronization issue while accessing
the fTimeZone member of the Func class, mutex locks were added to the
accessor and mutator methods. However, this slows down processing
of TIMESTAMP columns in PrimProc significantly as all threads across
all concurrently running queries would serialize on the mutex. This
is because PrimProc only has a single global object for the functor
class (class derived from Func in utils/funcexp/functor.h) for a given
function name. To fix this problem:
(1) We remove the fTimeZone as a member of the Func derived classes
(hence removing the mutexes) and instead use the fOperationType
member of the FunctionColumn class to propagate the timezone values
down to the individual functor processing functions such as
FunctionColumn::getStrVal(), FunctionColumn::getIntVal(), etc.
(2) To achieve (1), a timezone member is added to the
execplan::CalpontSystemCatalog::ColType class.
Part 2:
Several functors in the Funcexp code call dataconvert::gmtSecToMySQLTime()
and dataconvert::mySQLTimeToGmtSec() functions for conversion between seconds
since unix epoch and broken-down representation. These functions in turn call
the C library function localtime_r() which currently has a known bug of holding
a global lock via a call to __tz_convert. This significantly reduces performance
in multi-threaded applications where multiple threads concurrently call
localtime_r(). More details on the bug:
https://sourceware.org/bugzilla/show_bug.cgi?id=16145
This bug in localtime_r() caused processing of the Functors in PrimProc to
slowdown significantly since a query execution causes Functors code to be
processed in a multi-threaded manner.
As a fix, we remove the calls to localtime_r() from gmtSecToMySQLTime()
and mySQLTimeToGmtSec() by performing the timezone-to-offset conversion
(done in dataconvert::timeZoneToOffset()) during the execution plan
creation in the plugin. Note that localtime_r() is only called when the
time_zone system variable is set to "SYSTEM".
This fix also required changing the timezone type from a std::string to
a long across the system.
