There is a potential edge case when a thread remaps EM
managed shmem segment in grabEMEntryTable() whilst another
thread access the current shmem segment mapping in virtual memory
1. Input and output RowGroup's used in GROUP_CONCAT classes
are currently allocating a raw memory buffer of size equal
to the actual width of the string datatype. As an example,
for the following query:
SELECT col1, GROUP_CONCAT(col2) FROM t GROUP BY col1;
If col2 is a TEXT field with default width, the input
RowGroup containing the target rows to be concatenated will
assign 64kb of memory for every input row in the RowGroup.
This is wasteful as actual field values in real workloads
would be much smaller. We fix this by enabling the
RowGroup to use the StringStore when the RowGroup contains
long strings.
2. RowAggregation::initialize() allocates a memory buffer
for a NULL row. The size of this buffer is equal to the
row size for the output RowGroup. For the above scenario,
using the default group_concat_max_len (which is a server
variable that sets the maximum length of the GROUP_CONCAT string)
value of 1mb, the buffer size would be
(1mb + 64kb + some additional metadata). If the user sets
group_concat_max_len to a higher value, say 3gb, this buffer
size would be ~3gb. Now if the runtime initiates several
instances of RowAggregation, total memory consumption by
PrimProc could exceed the hardware memory limits causing the
OS OOM to kill the process. We fix this problem by again
enabling the StringStore for the NULL row allocation.
3. In the plugin code in buildAggregateColumn(), there is
an integer overflow when the server group_concat_max_len
variable (which is an uint32_t) is set to a value > INT32_MAX
(such as 3gb) and is assigned to
CalpontSystemCatalog::ColType::colWidth (which is an int32_t).
As a short term fix, we saturate the assigned value to colWidth
to INT32_MAX. Proper fix would be to upgrade
CalpontSystemCatalog::ColType::colWidth to an uint32_t.
Given that idx is a RH hashmap bucket number and info is intra-bucket idx
the root cause is triggered by the difference of idx/hash pair
calculation for a certain GROUP BY generation and for generation
aggregations merging that takes place in RowAggStorage::finalize.
This patch generalizes rowHashToIdx to leverage it in both cases
mentioned above.
* cmake fixes:
* set build byproducts (otherwise ninja cannot build)
* exclude thrift from "all" target, it should only be buit as a dependency
(otherwise it builds - and pulls in boost - when columnstore is disabled)
* move test-only external project into tests/ (because it requires git)
and into storage/columnstore (where thrift and boost already are)
* Update drone to use 10.6-enterprise server branch. (#2763)
---------
Co-authored-by: Sergei Golubchik <serg@mariadb.com>
Added logical transformation of the execplan::ParseTrees with the taking out the common factor in expression of the form "(A and B) or (A and C)" for the purposes of passing a TPCH 19 query.
Co-authored-by: Leonid Fedorov <leonid.fedorov@mariadb.com>
1. In TupleUnion::writeNull(), add the missing switch case for
wide decimal with 16bytes column width.
2. MCOL-5432 Disable complete/partial pushdown of UNION operation
if the query involves an ORDER BY or a LIMIT clause, until
MCOL-5222 is fixed. Also add MTR test cases for this.
When a UNION operation involving DECIMAL datatypes with scale and digits
before the decimal exceeds the currently supported maximum precision
of 38, we throw an error to the user:
"MCS-2060: Union operation exceeds maximum DECIMAL precision of 38".
This is until MCOL-5417 is implemented where ColumnStore will have
full parity with MariaDB server in terms of maximum supported DECIMAL
precision and scale of 65 and 38 digits respectively.
* Add MTR_SUITE_LIST
* Typo
* Add data download
* Install tar and lz4
* Change the way MTR_SUITE_LIST is set up
* Use bash for MTR_SUITE_LIST
* Another one
* Fix reference results for full MTR develop, disable broken JSON test and tests with 10GB database
* Fix timestamps and truncate cos
* Fix some more references
* Fix dokcerhub step for custom build
* One more fix for dockerhub step on custom build
* Fix tests for regr functions with truncate
* Full mtr set on nghtly + MTR_FULL_SET flag
* One more fix for dockerhub
* Fix MTR_FULL_SET
* Testing MTR_FULL_SET
* sorted_result in tests + fix typo
* Truncate even more
* Typo
* truncate 2 more tests
* Disable regr_* functions tests
* fix setup mtr step
* correct settings for table creation
* Put setup for tests into drone
* Fix for debian based distros
* More truncates
* Disable the rest
---------
Co-authored-by: Leonid Fedorov <leonid.fedorov@mariadb.com>
