1. Extend the calpontsys.syscolumn system catalog table
with a new column, 'charsetnum'.
'charsetnum' field is set to the 'number' member of the
'charset_info_st' struct defined in the server in m_ctype.h.
For CHAR/VARCHAR/TEXT column types, 'charset_info_st' is
initialized to the charset/collation of the column, which
is set at the column-level or at the table-level in the DDL.
For BLOB/VARBINARY binary column types, 'charset_info_st' is
initialized to my_charset_bin (charsetnum=63).
For all other column types, charsetnum is set to 0.
2. Add support for the newly added 'charsetnum' column in the
automatic system catalog upgrade logic in dbbuilder.
For existing table definitions, charsetnum for the column is
defaulted to 0.
3. Add MTR test case that creates a few table definitions with
a range of charset/collation combinations and queries the
calpontsys.syscolumn system catalog table with the charsetnum
field for the columns in the table DDLs.
As part of the charset support, a call to MY_INIT() was added at the
initialization of the above processes. This call initializes the MySQL
thread environment required by the charset library. However, the
accompanying my_end() call required to terminate this thread environment
was not added at the termination of these process, hence leaking
resources. As a fix, we move the MY_INIT() calls to the Child()
functions of these services and also add the missing my_end() call.
* Fixes of bugs from ASAN warnings, part one
* MQC as static library, with nifty counter for global map and mutex
* Switch clang to 16
* link messageqcpp to execplan
This patch improves handling of NULLs in textual fields in ColumnStore.
Previously empty strings were considered NULLs and it could be a problem
if data scheme allows for empty strings. It was also one of major
reasons of behavior difference between ColumnStore and other engines in
MariaDB family.
Also, this patch fixes some other bugs and incorrect behavior, for
example, incorrect comparison for "column <= ''" which evaluates to
constant True for all purposes before this patch.
- occured -> occurred
- reponse -> response
- seperated -> separated
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.
This reverts commit f4e3022fbdecc25a22ae6eaf072e462aa6695f35.
The commit apparently caused MCOL-5318 and MCOL-5319 which involve the
internal ColumnStore batch insert mechanism passing through the SQL
layer. The code block involved in this change is a predicate checking
for the HWM extent in WriteEngineServer at the end of the batch insert.
This is done in WE_DMLCommandProc::processBatchInsertHwm(). The original
predicate check in this function for the HWM extent is restored until
further investigation.
In the implementation of MCOL-5021, an assert was added in
`WE_DMLCommandProc::processBatchInsertHwm()` that assumed the
`WriteEngine::TableMetaData` cache is uniform across the cluster.
However, this assumption is incorrect.
This bug caused undefined behaviour in ColumnStore resulting in bugs
such as MCOL-5367. In MCOL-5367, in a multi-node ColumnStore cluster,
an INSERT ... SELECT in a transaction with system variable
`columnstore_use_import_for_batchinsert=OFF/ON` did not show inserted
records when a SELECT query was issued. Assuming a 3-node cluster setup,
DMLProc only sends a given batch of records to be inserted to one of the
3 nodes, and not all nodes. As a result, the `WriteEngine::TableMetaData`
cache is only populated for that one node and is not uniform across the
cluster, causing the assert to fail.
As a fix, we simply remove this assert as it is redundant and should not
have been added in the first place.
DMLProc starts ROLLBACK when SELECT part of UPDATE fails b/c EM facility in PP were restarted.
Unfortunately this ROLLBACK stuck if EM/PP are not yet available.
DMLProc must have a t/o with re-try doing ROLLBACK.
The idea is relatively simple - encode prefixes of collated strings as
integers and use them to compute extents' ranges. Then we can eliminate
extents with strings.
The actual patch does have all the code there but miss one important
step: we do not keep collation index, we keep charset index. Because of
this, some of the tests in the bugfix suite fail and thus main
functionality is turned off.
The reason of this patch to be put into PR at all is that it contains
changes that made CHAR/VARCHAR columns unsigned. This change is needed in
vectorization work.
* Fix clang warnings
* Remove vim tab guides
* initialize variables
* 'strncpy' output truncated before terminating nul copying as many bytes from a string as its length
* Fix ISO C++17 does not allow 'register' storage class specifier for outdated bison
* chars are unsigned on ARM, having if (ival < 0) always false
* chars are unsigned by default on ARM and comparison with -1 if always true
Part 1:
As part of MCOL-3776 to address synchronization issue while accessing
the fTimeZone member of the Func class, mutex locks were added to the
accessor and mutator methods. However, this slows down processing
of TIMESTAMP columns in PrimProc significantly as all threads across
all concurrently running queries would serialize on the mutex. This
is because PrimProc only has a single global object for the functor
class (class derived from Func in utils/funcexp/functor.h) for a given
function name. To fix this problem:
(1) We remove the fTimeZone as a member of the Func derived classes
(hence removing the mutexes) and instead use the fOperationType
member of the FunctionColumn class to propagate the timezone values
down to the individual functor processing functions such as
FunctionColumn::getStrVal(), FunctionColumn::getIntVal(), etc.
(2) To achieve (1), a timezone member is added to the
execplan::CalpontSystemCatalog::ColType class.
Part 2:
Several functors in the Funcexp code call dataconvert::gmtSecToMySQLTime()
and dataconvert::mySQLTimeToGmtSec() functions for conversion between seconds
since unix epoch and broken-down representation. These functions in turn call
the C library function localtime_r() which currently has a known bug of holding
a global lock via a call to __tz_convert. This significantly reduces performance
in multi-threaded applications where multiple threads concurrently call
localtime_r(). More details on the bug:
https://sourceware.org/bugzilla/show_bug.cgi?id=16145
This bug in localtime_r() caused processing of the Functors in PrimProc to
slowdown significantly since a query execution causes Functors code to be
processed in a multi-threaded manner.
As a fix, we remove the calls to localtime_r() from gmtSecToMySQLTime()
and mySQLTimeToGmtSec() by performing the timezone-to-offset conversion
(done in dataconvert::timeZoneToOffset()) during the execution plan
creation in the plugin. Note that localtime_r() is only called when the
time_zone system variable is set to "SYSTEM".
This fix also required changing the timezone type from a std::string to
a long across the system.
data types TEXT, CHAR, VARCHAR, FLOAT and DOUBLE are not yet supported by vectorized path
This patch introduces an example for Google benchmarking suite to measure a perf diff
b/w legacy scan/filtering code and the templated version
* Adds CompressInterfaceLZ4 which uses LZ4 API for compress/uncompress.
* Adds CMake machinery to search LZ4 on running host.
* All methods which use static data and do not modify any internal data - become `static`,
so we can use them without creation of the specific object. This is possible, because
the header specification has not been modified. We still use 2 sections in header, first
one with file meta data, the second one with pointers for compressed chunks.
* Methods `compress`, `uncompress`, `maxCompressedSize`, `getUncompressedSize` - become
pure virtual, so we can override them for the other compression algos.
* Adds method `getChunkMagicNumber`, so we can verify chunk magic number
for each compression algo.
* Renames "s/IDBCompressInterface/CompressInterface/g" according to requirement.
This patch:
1. Removes the option to declare uncompressed columns (set columnstore_compression_type = 0).
2. Ignores [COMMENT '[compression=0] option at table or column level (no error messages, just disregard).
3. Removes the option to set more than 2 extents per file (ExtentsPreSegmentFile).
4. Updates rebuildEM tool to support up to 10 dictionary extent per dictionary segment file.
5. Adds check for `DBRootStorageType` for rebuildEM tool.
6. Renamed rebuildEM to mcsRebuildEM.
mcsconfig.h and my_config.h have the following
pre-processor definitions:
1. Conflicting definitions coming from the standard cmake definitions:
- PACKAGE
- PACKAGE_BUGREPORT
- PACKAGE_NAME
- PACKAGE_STRING
- PACKAGE_TARNAME
- PACKAGE_VERSION
- VERSION
2. Conflicting definitions of other kinds:
- HAVE_STRTOLL - this is a dirt in MariaDB headers.
Should be fixed in the server code. my_config.h erroneously
performs "#define HAVE_STRTOLL" instead of "#define HAVE_STRTOLL 1".
in some cases. The former is not CMake compatible style. The latter is.
3. Non-conflicting definitions:
Otherwise, mcsconfig.h and my_config.h should be mutually compatible,
because both are generated by cmake on the same host machine. So
they should have exactly equal definitions like "HAVE_XXX", "SIZEOF_XXX", etc.
Observations:
- It's OK to include both mcsconfig.h and my_config.h providing that we
suppress duplicate definition of the above conflicting types #1 and #2.
- There is no a need to suppress duplicate definitions mentioned in #3,
as they are compatible!
- my_sys.h and m_ctype.h must always follow a CMake configuation header,
either my_config.h or mcsconfig.h (or both).
They must never be included without any preceeding configuration header.
This change make sure that we resolve conflicts by:
- either disallowing inclusion of mcsconfig.h and my_config.h
at the same time
- or by hiding conflicting definitions #1 and #2
(with their later restoring).
- also, by making sure that my_sys.h and m_ctype.h always follow
a CMake configuration file.
Details:
- idb_mysql.h can now only be included only after my_config.h
An attempt to use idb_mysql.h with mcsconfig.h instead of
my_config.h is caught by the "#error" preprocessor directive.
- mariadb_my_sys.h can now be only included after mcsconfig.h.
An attempt to use mariadb_my_sys.h without mcscofig.h
(e.g. with my_config.h) is also caught by "#error".
- collation.h now can now be included in two ways.
It now has the following effective structure:
#if defined(PREFER_MY_CONFIG_H) && defined(MY_CONFIG_H)
// Remember current conflicting definitions on the preprocessor stack
// Undefine current conflicting definitions
#endif
#include "mcsconfig.h"
#include "m_ctype.h"
#if defined(PREFER_MY_CONFIG_H) && defined(MY_CONFIG_H)
# Restore conflicting definitions from the preprocessor stack
#endif
and can be included as follows:
a. using only mcsconfig.h as a configuration header:
// my_config.h must not be included so far
#include "collation.h"
b. using my_config.h as the first included configuration file:
#define PREFER_MY_CONFIG_H // Force conflict resolution
#include "my_config.h" // can be included directly or indirectly
...
#include "collation.h"
Other changes:
- Adding helper header files
utils/common/mcsconfig_conflicting_defs_remember.h
utils/common/mcsconfig_conflicting_defs_restore.h
utils/common/mcsconfig_conflicting_defs_undef.h
to perform conflict resolution easier.
- Removing `#include "collation.h"` from a number of files,
as it's automatically included from rowgroup.h.
- Removing redundant `#include "utils_utf8.h"`.
This change is not directly related to the problem being fixed,
but it's nice to remove redundant directives for both collation.h
and utils_utf8.h from all the files that do not really need them.
(this change could probably have gone as a separate commit)
- Changing my_init() to MY_INIT(argv[0]) in the MCS services sources.
After the fix of the complitation failure it appeared that ColumnStore
services compiled with the debug build crash due to recent changes in
safemalloc. The crash happened in strcmp() with `my_progname` as an argument
(where my_progname is a mysys global variable). This problem should
probably be fixed on the server side as well to avoid passing NULL.
But, the majority of MariaDB executable programs also use MY_INIT(argv[0])
rather than my_init(). So let's make MCS do like the other programs do.
Progress keep and test commit
Progress keep and test commit
Progress keep and test commit
Again, trying to pinpoint problematic part of a change
Revert "Again, trying to pinpoint problematic part of a change"
This reverts commit 71874e7c0d7e4eeed0c201b12d306b583c07b9e2.
Revert "Progress keep and test commit"
This reverts commit 63c7bc67ae55bdb81433ca58bbd239d6171a1031.
Revert "Progress keep and test commit"
This reverts commit 121c09febd78dacd37158caeab9ac70f65b493df.
Small steps - I walk minefield here
Propagating changes - now CPInfo in convertValArray
Progress keep commit
Restoring old functionality
Progress keep commit
Small steps to avoid/better locate old problem with the write engine.
Progress keeping commit
Thread the CPInfo up to convertValArray call in writeColumnRec
About to test changes - I should get no regression and no updates in
ranges either.
Testing out why I get a regression
Investigating source of regression
Debugging prints
Fix compile error
Debugging print - debug regression
I clearly see calls to writeColumnRec and prints there added to discern
between these.
Fix warning error
Possible culprit
Add forgotten default parameter for convertValArray
New logic to test
Max/min gets updated during value conversion
To test results of updates
Debug logs
Debug logs
An attempt to provide proper sequence index
Debug logs
An attempt to provide proper sequence index - now magic for resetting
Debug logs
Debug logs
Debug logs
Trying to perform correct updates
Trying to perform correct updates - seqNum woes fight
COMMIT after INSERT performs 'mark extent as invalid' operation - investigating
To test: cut setting of CPInfo upon commit from DML processor
It may be superfluous as write engine does that too
Debug logs
Debug logs
Better interface for CPMaxMin
Old interface forgot to set isBinaryColumn field
Possible fix for the problems
I forgot to reassign the value in cpinfoList
Debug logs
Computation of 'binary' column property
logs indicated that it was not set in getExtentCPMaxMin, and it was impossible to compute there so I had to move that into writeengine.
To test: code to allow cross-extent insertion
To test: removed another assertion for probable cause of errors
Debug logs
Dropped excessive logs
Better reset code
Again, trying to fix ordering
Fixing order of rowids for LBID computation
Debug logs
Remove update of second LBID in split insert
I have to validate incorrect behaviour for this test
Restoring the case where everything almost worked
Tracking changes in newly created extents
Progress keeping commit
Fixing build errors with recent server
An ability to get old values from blocks we update
Progress keeping commit
Adding analysis of old values to write engine code.
It is needed for updates and deletes.
Progress keeping commit
Moving max/min range update from convertValArray into separate function with simpler logic.
To test and debug - logic is there
Fix build errors
Update logic to debug
There is a suspicious write engine method updateColumnRecs which
receives a vector of column types but does not iterate over them
(otherwise it will be identical to updateColumnRec in logic).
Other than that, the updateColumnRec looks like the center of all
updates - deleteRow calls it, for example, dml processor also calls it.
Debug logs for insert bookkeeping regression
Set up operation type in externally-callable interface
Internal operations depend on the operation type and consistency is what matters there.
Debug logs
Fix for extent range update failure during update operation
Fix build error
Debug logs
Fix for update on deletion
I am not completely sure in it - to debug.
Debug log
writeColumnRec cannot set m_opType to UPDATE unconditionally
It is called from deleteRow
Better diagnostics
Debug logs
Fixed search condition
Debug logs
Debugging invalid LBID appearance
Debug logs - fixed condition
Fix problems with std::vector reallocation during growth
Fix growing std::vector data dangling access error
Still fixing indexing errors
Make in-range update to work
Correct sequence numbers
Debug logs
Debug logs
Remove range drop from DML part of write engine
A hack to test the culprit of range non-keeping
Tests - no results for now
MTR-style comments
Empty test results
To be filled with actual results.
Special database and result selects for all tests
Pleasing MTR with better folder name
Pleasing MTR - testing test result comparison
Pleasing MTR by disabling warnings
All test results
Cleaning up result files
Reset ranges before update
Remove comments from results - point of failure in MTR
Remove empty line from result - another MTR failure point
Probably fix for deletes
Possible fix for remaining failed delete test
Fix a bug in writeRows
It should not affect delete-with-range test case, yet it is a bug.
Debug logs
Debug logs
Tests reorganization and description
Support for unsigned integer for new tests
Fix type omission
Fix test failure due to warnings on clean installation
Support for bigint to test
Fix for failed signed bigint test
Set proper unsignedness flag
Removed that assignment during refactoring.
Tests for types with column width 1 and 2
Support for types in new tests
Remove trailing empty lines from results
Tests had failed because of extra empty lines.
Remove debug logs
Update README with info about new tests
Move tests for easier testing
Add task tag to tests
Fix invalid unsaigned range check
Fix for signed types
Fix regressions - progress keeping commit
Do not set invalid ranges into valid state
A possible fix for mcs81_self_join test
MCOL 2044 test database cleanup
Missing expected results
Delete extraneous assignment to m_opType
nullptr instead of NULL
Refactor extended CPInfo with TypeHandler
Better handling of ranges - safer types, less copy-paste
Fix logic error related to typo
Fix logic error related to typo
Trying to figure out why invalid ranges aren't displayed as NULL..NULL
Debug logs
Debug logs
Debug logs
Debug logs for worker node
Debug logs for worker node in extent map
Debugging virtual table fill operation
Debugging virtual table fill operation
Fix for invalid range computation
Remove debug logs
Change handling of invalid ranges
They are also set, but to invalid state.
Complete change
Fix typo
Remove unused code
"Fix" for tests - -1..0 instead of NULL..NULL for invalid unsigned ranges
Not a good change, yet I cannot do better for now.
MTR output requires tabs instead of spaces
Debug logs
Debug logs
Debug logs - fix build
Debug logs and logic error fix
Fix for clearly incorrect firstLBID in CPInfo being set - to test
Fix for system catalog operations suppot
Better interface to fix build errors
Delete tests we cannot satisfy due to extent rescan due to WHERE
Tests for wide decimals
Testing support for wide decimals
Fix for wide decimals tests
Fix for delete within range
Memory leak fix and, possible, double free fix
Dispatch on CalpontSystemCatalog::ColDataType is more robust
Add support for forgotten MEDINT type
Add forgottent BIGINT
empty() instead of size() > 0
Better layout
Remove confusing comment
Sensible names for special values of seqNum field
Tests for wide decimal support
Addressing concerns of drrtuy
Remove test we cannot satisfy
Final touches for PR
Remove unused result file
Logs for research purposes
Keep progress - may not build
Good interface to collect LBIDs and CPInfo's
Write Engine compiles with new interface
New interface breaks things the least way and allows for new features to be added gradually.
Still ironing design - rewriting parts of WE
Keep progress commit
Write Engine compiles, going to test
I could introduce crashes there. Let's see.
Disable logging for tests
Fixing build problems - keep progress commit
Changed related to new interface
Add back accidentally removed m_txnLBIDMap.find
Remove printf/cout; up-to-date comment for AddLBIDtoList
Add "auto" type annotation
Work on PR comments
Descriptive vector emptines check
MCOL-4409 This patch combines VDecimal and Decimal and makes
IDB_Decimal an alias for the result class
MCOL-4409 More boilerplate reduction in Func_mod
Removed couple TSInt128::toType() methods
Removed uint128 from joblist/lbidlist.*
Another toString() method for wide-decimal that is EMPTY/NULL aware
Unified decimal processing in WF functions
Fixed a potential issue in EqualCompData::operator() for
wide-decimal processing
Fixed some signedness warnings
WF::percentile runtime threw an exception b/c of wrong DT deduced from its argument
Replaced literals with constants
Tought WF_sum_avg::checkSumLimit to use refs instead of values
2. Set Decimal precision in SimpleColumn::evaluate().
3. Add support for int128_t in ConstantColumn.
4. Set IDB_Decimal::s128Value in buildDecimalColumn().
5. Use width 16 as first if predicate for branching based on decimal width.
