Boost 1.85 removed some deprecated code in filesystem module which is
still used in columnstore:
- The boost/filesystem/convenience.hpp was removed but columnstore does
not use any functionality from that file except indirect includes.
Therefore this include is removed or replaced with more general
boost/filesystem.hpp. The convenience.hpp header file was deprecated
in filesystem V3 introduced in Boost 1.46.0.
- `normalize` method was removed and users are suggested to replace it
with `lexically_normal` method, which was introduced in Boost 1.60.0.
Original `normalize` call is preserved for backward compatibility with
old Boost version, however`, `lexically_normal` method is preferably
used with Boost 1.60.0 and newer.
- The `copy_option` was removed in favor of `copy_options` (note the
trailing 's'), but enum values were renamed. Namely, `fail_if_exists`
is replaced with `none` and `overwrite_if_exists` is replaced with
`overwrite_existing`. The `copy_options` was introduced in Boost
1.74.0.
New form is used instead, but a backward compatibility layer for Boost
1.73.0 and older was introduced in boost_copy_options_compat.hpp file.
This solution seems to be less awkward than using multiple #if #else
#endif blocks in source code.
ColumnStore used to include server's mysql.h
but link all tools with libmariadb.so
There's no guarantee that this would work, even with workarounds
it had in dbcon/mysql/sm.cpp
Fix:
* tools (linked with libmariadb.so) *must* include libmariadb's mysql.h
* as a hack prevent service_thd_timezone.h from being loaded into tools,
as it conflicts with libmariadb's mysql.h
* server plugin *must* include server's mysql.h
* also don't link every tool with libmariadb.so, link the helper library
(liblibmysqlclient.so) that actually needs it, tools use this
helper library, not libmariadb.so directly
writeengine/server/we_dataloader.cpp:894:22: error: arithmetic between floating-point type 'double' and enumeration type 'WriteEngine::WEDataLoader::<unnamed enum>' is deprecated [-Werror=enum-conversion]
Part 1:
As part of MCOL-3776 to address synchronization issue while accessing
the fTimeZone member of the Func class, mutex locks were added to the
accessor and mutator methods. However, this slows down processing
of TIMESTAMP columns in PrimProc significantly as all threads across
all concurrently running queries would serialize on the mutex. This
is because PrimProc only has a single global object for the functor
class (class derived from Func in utils/funcexp/functor.h) for a given
function name. To fix this problem:
(1) We remove the fTimeZone as a member of the Func derived classes
(hence removing the mutexes) and instead use the fOperationType
member of the FunctionColumn class to propagate the timezone values
down to the individual functor processing functions such as
FunctionColumn::getStrVal(), FunctionColumn::getIntVal(), etc.
(2) To achieve (1), a timezone member is added to the
execplan::CalpontSystemCatalog::ColType class.
Part 2:
Several functors in the Funcexp code call dataconvert::gmtSecToMySQLTime()
and dataconvert::mySQLTimeToGmtSec() functions for conversion between seconds
since unix epoch and broken-down representation. These functions in turn call
the C library function localtime_r() which currently has a known bug of holding
a global lock via a call to __tz_convert. This significantly reduces performance
in multi-threaded applications where multiple threads concurrently call
localtime_r(). More details on the bug:
https://sourceware.org/bugzilla/show_bug.cgi?id=16145
This bug in localtime_r() caused processing of the Functors in PrimProc to
slowdown significantly since a query execution causes Functors code to be
processed in a multi-threaded manner.
As a fix, we remove the calls to localtime_r() from gmtSecToMySQLTime()
and mySQLTimeToGmtSec() by performing the timezone-to-offset conversion
(done in dataconvert::timeZoneToOffset()) during the execution plan
creation in the plugin. Note that localtime_r() is only called when the
time_zone system variable is set to "SYSTEM".
This fix also required changing the timezone type from a std::string to
a long across the system.
data types TEXT, CHAR, VARCHAR, FLOAT and DOUBLE are not yet supported by vectorized path
This patch introduces an example for Google benchmarking suite to measure a perf diff
b/w legacy scan/filtering code and the templated version
* Adds CompressInterfaceLZ4 which uses LZ4 API for compress/uncompress.
* Adds CMake machinery to search LZ4 on running host.
* All methods which use static data and do not modify any internal data - become `static`,
so we can use them without creation of the specific object. This is possible, because
the header specification has not been modified. We still use 2 sections in header, first
one with file meta data, the second one with pointers for compressed chunks.
* Methods `compress`, `uncompress`, `maxCompressedSize`, `getUncompressedSize` - become
pure virtual, so we can override them for the other compression algos.
* Adds method `getChunkMagicNumber`, so we can verify chunk magic number
for each compression algo.
* Renames "s/IDBCompressInterface/CompressInterface/g" according to requirement.
This patch:
1. Removes the option to declare uncompressed columns (set columnstore_compression_type = 0).
2. Ignores [COMMENT '[compression=0] option at table or column level (no error messages, just disregard).
3. Removes the option to set more than 2 extents per file (ExtentsPreSegmentFile).
4. Updates rebuildEM tool to support up to 10 dictionary extent per dictionary segment file.
5. Adds check for `DBRootStorageType` for rebuildEM tool.
6. Renamed rebuildEM to mcsRebuildEM.
mcsconfig.h and my_config.h have the following
pre-processor definitions:
1. Conflicting definitions coming from the standard cmake definitions:
- PACKAGE
- PACKAGE_BUGREPORT
- PACKAGE_NAME
- PACKAGE_STRING
- PACKAGE_TARNAME
- PACKAGE_VERSION
- VERSION
2. Conflicting definitions of other kinds:
- HAVE_STRTOLL - this is a dirt in MariaDB headers.
Should be fixed in the server code. my_config.h erroneously
performs "#define HAVE_STRTOLL" instead of "#define HAVE_STRTOLL 1".
in some cases. The former is not CMake compatible style. The latter is.
3. Non-conflicting definitions:
Otherwise, mcsconfig.h and my_config.h should be mutually compatible,
because both are generated by cmake on the same host machine. So
they should have exactly equal definitions like "HAVE_XXX", "SIZEOF_XXX", etc.
Observations:
- It's OK to include both mcsconfig.h and my_config.h providing that we
suppress duplicate definition of the above conflicting types #1 and #2.
- There is no a need to suppress duplicate definitions mentioned in #3,
as they are compatible!
- my_sys.h and m_ctype.h must always follow a CMake configuation header,
either my_config.h or mcsconfig.h (or both).
They must never be included without any preceeding configuration header.
This change make sure that we resolve conflicts by:
- either disallowing inclusion of mcsconfig.h and my_config.h
at the same time
- or by hiding conflicting definitions #1 and #2
(with their later restoring).
- also, by making sure that my_sys.h and m_ctype.h always follow
a CMake configuration file.
Details:
- idb_mysql.h can now only be included only after my_config.h
An attempt to use idb_mysql.h with mcsconfig.h instead of
my_config.h is caught by the "#error" preprocessor directive.
- mariadb_my_sys.h can now be only included after mcsconfig.h.
An attempt to use mariadb_my_sys.h without mcscofig.h
(e.g. with my_config.h) is also caught by "#error".
- collation.h now can now be included in two ways.
It now has the following effective structure:
#if defined(PREFER_MY_CONFIG_H) && defined(MY_CONFIG_H)
// Remember current conflicting definitions on the preprocessor stack
// Undefine current conflicting definitions
#endif
#include "mcsconfig.h"
#include "m_ctype.h"
#if defined(PREFER_MY_CONFIG_H) && defined(MY_CONFIG_H)
# Restore conflicting definitions from the preprocessor stack
#endif
and can be included as follows:
a. using only mcsconfig.h as a configuration header:
// my_config.h must not be included so far
#include "collation.h"
b. using my_config.h as the first included configuration file:
#define PREFER_MY_CONFIG_H // Force conflict resolution
#include "my_config.h" // can be included directly or indirectly
...
#include "collation.h"
Other changes:
- Adding helper header files
utils/common/mcsconfig_conflicting_defs_remember.h
utils/common/mcsconfig_conflicting_defs_restore.h
utils/common/mcsconfig_conflicting_defs_undef.h
to perform conflict resolution easier.
- Removing `#include "collation.h"` from a number of files,
as it's automatically included from rowgroup.h.
- Removing redundant `#include "utils_utf8.h"`.
This change is not directly related to the problem being fixed,
but it's nice to remove redundant directives for both collation.h
and utils_utf8.h from all the files that do not really need them.
(this change could probably have gone as a separate commit)
- Changing my_init() to MY_INIT(argv[0]) in the MCS services sources.
After the fix of the complitation failure it appeared that ColumnStore
services compiled with the debug build crash due to recent changes in
safemalloc. The crash happened in strcmp() with `my_progname` as an argument
(where my_progname is a mysys global variable). This problem should
probably be fixed on the server side as well to avoid passing NULL.
But, the majority of MariaDB executable programs also use MY_INIT(argv[0])
rather than my_init(). So let's make MCS do like the other programs do.
Progress keep and test commit
Progress keep and test commit
Progress keep and test commit
Again, trying to pinpoint problematic part of a change
Revert "Again, trying to pinpoint problematic part of a change"
This reverts commit 71874e7c0d7e4eeed0c201b12d306b583c07b9e2.
Revert "Progress keep and test commit"
This reverts commit 63c7bc67ae55bdb81433ca58bbd239d6171a1031.
Revert "Progress keep and test commit"
This reverts commit 121c09febd78dacd37158caeab9ac70f65b493df.
Small steps - I walk minefield here
Propagating changes - now CPInfo in convertValArray
Progress keep commit
Restoring old functionality
Progress keep commit
Small steps to avoid/better locate old problem with the write engine.
Progress keeping commit
Thread the CPInfo up to convertValArray call in writeColumnRec
About to test changes - I should get no regression and no updates in
ranges either.
Testing out why I get a regression
Investigating source of regression
Debugging prints
Fix compile error
Debugging print - debug regression
I clearly see calls to writeColumnRec and prints there added to discern
between these.
Fix warning error
Possible culprit
Add forgotten default parameter for convertValArray
New logic to test
Max/min gets updated during value conversion
To test results of updates
Debug logs
Debug logs
An attempt to provide proper sequence index
Debug logs
An attempt to provide proper sequence index - now magic for resetting
Debug logs
Debug logs
Debug logs
Trying to perform correct updates
Trying to perform correct updates - seqNum woes fight
COMMIT after INSERT performs 'mark extent as invalid' operation - investigating
To test: cut setting of CPInfo upon commit from DML processor
It may be superfluous as write engine does that too
Debug logs
Debug logs
Better interface for CPMaxMin
Old interface forgot to set isBinaryColumn field
Possible fix for the problems
I forgot to reassign the value in cpinfoList
Debug logs
Computation of 'binary' column property
logs indicated that it was not set in getExtentCPMaxMin, and it was impossible to compute there so I had to move that into writeengine.
To test: code to allow cross-extent insertion
To test: removed another assertion for probable cause of errors
Debug logs
Dropped excessive logs
Better reset code
Again, trying to fix ordering
Fixing order of rowids for LBID computation
Debug logs
Remove update of second LBID in split insert
I have to validate incorrect behaviour for this test
Restoring the case where everything almost worked
Tracking changes in newly created extents
Progress keeping commit
Fixing build errors with recent server
An ability to get old values from blocks we update
Progress keeping commit
Adding analysis of old values to write engine code.
It is needed for updates and deletes.
Progress keeping commit
Moving max/min range update from convertValArray into separate function with simpler logic.
To test and debug - logic is there
Fix build errors
Update logic to debug
There is a suspicious write engine method updateColumnRecs which
receives a vector of column types but does not iterate over them
(otherwise it will be identical to updateColumnRec in logic).
Other than that, the updateColumnRec looks like the center of all
updates - deleteRow calls it, for example, dml processor also calls it.
Debug logs for insert bookkeeping regression
Set up operation type in externally-callable interface
Internal operations depend on the operation type and consistency is what matters there.
Debug logs
Fix for extent range update failure during update operation
Fix build error
Debug logs
Fix for update on deletion
I am not completely sure in it - to debug.
Debug log
writeColumnRec cannot set m_opType to UPDATE unconditionally
It is called from deleteRow
Better diagnostics
Debug logs
Fixed search condition
Debug logs
Debugging invalid LBID appearance
Debug logs - fixed condition
Fix problems with std::vector reallocation during growth
Fix growing std::vector data dangling access error
Still fixing indexing errors
Make in-range update to work
Correct sequence numbers
Debug logs
Debug logs
Remove range drop from DML part of write engine
A hack to test the culprit of range non-keeping
Tests - no results for now
MTR-style comments
Empty test results
To be filled with actual results.
Special database and result selects for all tests
Pleasing MTR with better folder name
Pleasing MTR - testing test result comparison
Pleasing MTR by disabling warnings
All test results
Cleaning up result files
Reset ranges before update
Remove comments from results - point of failure in MTR
Remove empty line from result - another MTR failure point
Probably fix for deletes
Possible fix for remaining failed delete test
Fix a bug in writeRows
It should not affect delete-with-range test case, yet it is a bug.
Debug logs
Debug logs
Tests reorganization and description
Support for unsigned integer for new tests
Fix type omission
Fix test failure due to warnings on clean installation
Support for bigint to test
Fix for failed signed bigint test
Set proper unsignedness flag
Removed that assignment during refactoring.
Tests for types with column width 1 and 2
Support for types in new tests
Remove trailing empty lines from results
Tests had failed because of extra empty lines.
Remove debug logs
Update README with info about new tests
Move tests for easier testing
Add task tag to tests
Fix invalid unsaigned range check
Fix for signed types
Fix regressions - progress keeping commit
Do not set invalid ranges into valid state
A possible fix for mcs81_self_join test
MCOL 2044 test database cleanup
Missing expected results
Delete extraneous assignment to m_opType
nullptr instead of NULL
Refactor extended CPInfo with TypeHandler
Better handling of ranges - safer types, less copy-paste
Fix logic error related to typo
Fix logic error related to typo
Trying to figure out why invalid ranges aren't displayed as NULL..NULL
Debug logs
Debug logs
Debug logs
Debug logs for worker node
Debug logs for worker node in extent map
Debugging virtual table fill operation
Debugging virtual table fill operation
Fix for invalid range computation
Remove debug logs
Change handling of invalid ranges
They are also set, but to invalid state.
Complete change
Fix typo
Remove unused code
"Fix" for tests - -1..0 instead of NULL..NULL for invalid unsigned ranges
Not a good change, yet I cannot do better for now.
MTR output requires tabs instead of spaces
Debug logs
Debug logs
Debug logs - fix build
Debug logs and logic error fix
Fix for clearly incorrect firstLBID in CPInfo being set - to test
Fix for system catalog operations suppot
Better interface to fix build errors
Delete tests we cannot satisfy due to extent rescan due to WHERE
Tests for wide decimals
Testing support for wide decimals
Fix for wide decimals tests
Fix for delete within range
Memory leak fix and, possible, double free fix
Dispatch on CalpontSystemCatalog::ColDataType is more robust
Add support for forgotten MEDINT type
Add forgottent BIGINT
empty() instead of size() > 0
Better layout
Remove confusing comment
Sensible names for special values of seqNum field
Tests for wide decimal support
Addressing concerns of drrtuy
Remove test we cannot satisfy
Final touches for PR
Remove unused result file
Logs for research purposes
Keep progress - may not build
Good interface to collect LBIDs and CPInfo's
Write Engine compiles with new interface
New interface breaks things the least way and allows for new features to be added gradually.
Still ironing design - rewriting parts of WE
Keep progress commit
Write Engine compiles, going to test
I could introduce crashes there. Let's see.
Disable logging for tests
Fixing build problems - keep progress commit
Changed related to new interface
Add back accidentally removed m_txnLBIDMap.find
Remove printf/cout; up-to-date comment for AddLBIDtoList
Add "auto" type annotation
Work on PR comments
Descriptive vector emptines check
MCOL-4409 This patch combines VDecimal and Decimal and makes
IDB_Decimal an alias for the result class
MCOL-4409 More boilerplate reduction in Func_mod
Removed couple TSInt128::toType() methods
Removed uint128 from joblist/lbidlist.*
Another toString() method for wide-decimal that is EMPTY/NULL aware
Unified decimal processing in WF functions
Fixed a potential issue in EqualCompData::operator() for
wide-decimal processing
Fixed some signedness warnings
WF::percentile runtime threw an exception b/c of wrong DT deduced from its argument
Replaced literals with constants
Tought WF_sum_avg::checkSumLimit to use refs instead of values
2. Set Decimal precision in SimpleColumn::evaluate().
3. Add support for int128_t in ConstantColumn.
4. Set IDB_Decimal::s128Value in buildDecimalColumn().
5. Use width 16 as first if predicate for branching based on decimal width.
There are multiple overloaded version of the low level DML write methods to
push down CSC column type. WE needs the type to convert values correctly.
Replaced WE_INT128 with CSC data type that is more informative.
Removed commented and obsolete code.
Replaced switch-case blocks with oneliners.
Create tables and schemas with lower case name only if the flag is set.
During operations, convert to lowercase in plugin. Byt the time a query gets to ExeMgr, DDLProc etc., everything must be lower case if the flag is set, and undisturbed if not.
