* feat(joblist,runtime): this is the first part of the execution model that produces a workload that can be predicted for a given query.
- forces to UM join converter to use a value from a configuration
- replaces a constant used to control a number of outstanding requests with a value depends on column width
- modifies related Columnstore.xml values
Added logical transformation of the execplan::ParseTrees with the taking out the common factor in expression of the form "(A and B) or (A and C)" for the purposes of passing a TPCH 19 query.
Co-authored-by: Leonid Fedorov <leonid.fedorov@mariadb.com>
by default), which when enabled, indiscriminately invalidates all
column extents and performs the actual DELETE only on the AUX
column. The trade-off with this approach would now be that the
first SELECT for certain query patterns (those containing a WHERE
predicate) after the DELETE operation will slow down as the
invalidated column extent would need to be scanned again to set
the min/max values.
* MCOL-4560 remove unused xml entries and code that references it.
There is reader code and variables for some of these settings, but nobody uses them.
Changes to Columnstore.xml are causing issues with automated install scripts. Removing these minor changes will get us through the release. The problem shall be fixed with MCOL-4928
* Adds CompressInterfaceLZ4 which uses LZ4 API for compress/uncompress.
* Adds CMake machinery to search LZ4 on running host.
* All methods which use static data and do not modify any internal data - become `static`,
so we can use them without creation of the specific object. This is possible, because
the header specification has not been modified. We still use 2 sections in header, first
one with file meta data, the second one with pointers for compressed chunks.
* Methods `compress`, `uncompress`, `maxCompressedSize`, `getUncompressedSize` - become
pure virtual, so we can override them for the other compression algos.
* Adds method `getChunkMagicNumber`, so we can verify chunk magic number
for each compression algo.
* Renames "s/IDBCompressInterface/CompressInterface/g" according to requirement.
* Introduce multigeneration aggregation
* Do not save unused part of RGDatas to disk
* Add IO error explanation (strerror)
* Reduce memory usage while aggregating
* introduce in-memory generations to better memory utilization
* Try to limit the qty of buckets at a low limit
* Refactor disk aggregation a bit
* pass calculated hash into RowAggregation
* try to keep some RGData with free space in memory
* do not dump more than half of rowgroups to disk if generations are
allowed, instead start a new generation
* for each thread shift the first processed bucket at each iteration,
so the generations start more evenly
* Unify temp data location
* Explicitly create temp subdirectories
whether disk aggregation/join are enabled or not
This patch:
1. Removes the option to declare uncompressed columns (set columnstore_compression_type = 0).
2. Ignores [COMMENT '[compression=0] option at table or column level (no error messages, just disregard).
3. Removes the option to set more than 2 extents per file (ExtentsPreSegmentFile).
4. Updates rebuildEM tool to support up to 10 dictionary extent per dictionary segment file.
5. Adds check for `DBRootStorageType` for rebuildEM tool.
6. Renamed rebuildEM to mcsRebuildEM.
Rename packages to MariaDB-columnstore-engine, MariaDB-columnstore-libs
and MariaDB-columnstore-platform.
Also add the "columnstore-" prefix the the components so that MariaDB's
packaging system understands then and add a line to include them in
MariaDB's packaging.
In addition
* Fix S3 building for dist source build
* Fix Debian 10 dependency issue
* Fix git handling for dist builds
* Add support for MariaDB's RPM building
* Use MariaDB's PCRE and readline
* Removes a few dead files
* Fix Boost noncopyable includes
ColumnStore now uses standard bin/lib paths for pretty much everything.
Data path is now hard-coded to /var/lib/columnstore.
This patch also:
* Removes v1 decompression
* Removes a bunch of unneeded files
* Removes COLUMNSTORE_INSTALL_DIR / $INSTALLDIR
* Makes my.cnf.d work for all platforms (MCOL-3558)
* Changes configcpp to use recursive mutex (fixes possible config write deadlock)
* Fixes MCOL-3599 Fix regr functions, The library was installed in the wrong location
* Fixes a bunch of Ubuntu packaging issues
* Changes the binary names of several of the executables so as not to
clash with potential executables from other packages
This branch enforces the use of generic MariaDB server paths for their
binaries and data rather than custom paths.
/usr/local/mariadb/columnstore is now only for columnstore with this
patch.
It should be noted that this removes the auto-mounting of external
MariaDB UM data storage for AWS.
This is also a fix for MCOL-3510 after buildbot changes are made.
Also... MCOL-3552 Use columnstore.cnf to load plugins
The ColumnStore plugins now load using a columnstore.cnf instead of a
SQL sequence to be more in-line with MariaDB's methods.
This patch:
* Moves config files from /usr/local/mariadb/columnstore/etc to
ENGINE_SYSCONFDIR/columnstore (ENGINE_SYSCONFDIR is /etc by default)
* Sets a define called MCSSYSCONFDIR whic contains the
ENGINE_SYSCONFDIR compile time setting
* Modifies scripts and code to use the new paths
* Removes a whole bunch of files we don't use
- Re-added the StorageManager/Enabled key in the default config file.
Thought I had switched all that stuff to only look at DBRootStorageType
but I was wrong.
- Made the unit_tests bin for SM stay in the SM dir
- Collapsed one level of build dirs for SM. For some reason it was
putting test data in storage-manager/storage-manager/testData
- Limited a single read() call to 100MB as a paranoid safety measure.
Normal usage won't read that much at once anyway.
