1
0
mirror of https://github.com/mariadb-corporation/mariadb-columnstore-engine.git synced 2025-08-07 03:22:57 +03:00
Commit Graph

27 Commits

Author SHA1 Message Date
drrtuy
d8bef9dc11 fix(UT): added dummy RM ctor to be used with UTs 2025-05-02 10:11:40 +01:00
Aleksei Antipovskii
4bea7e59a0 feat(PrimProc): MCOL-5852 disk-based GROUP_CONCAT & JSON_ARRAYAGG
* move GROUP_CONCAT/JSON_ARRAYAGG storage to the RowGroup from
  the RowAggregation*
* internal data structures (de)serialization
* get rid of a specialized classes for processing JSON_ARRAYAGG
* move the memory accounting to disk-based aggregation classes
* allow aggregation generations to be used for queries with
  GROUP_CONCAT/JSON_ARRAYAGG
* Remove the thread id from the error message as it interferes with the mtr
2025-04-11 15:21:07 +02:00
drrtuy
729db829a2 feat(RM,allocators): use atomic counters math abstraction 2025-03-27 22:12:48 +00:00
drrtuy
02b8ea1331 feat(PP,ByteStream): new counting memory allocator 2025-03-27 22:12:48 +00:00
drrtuy
dc03621e9d fix(rowgroup): RGData now uses uint64_t counter for the fixed sizes columns data buf.
The buffer can utilize > 4GB RAM that is necessary for PM side join.
	RGData ctor uses uint32_t allocating data buffer.
 	This fact causes implicit heap overflow.
2024-11-08 16:28:51 +04:00
Roman Nozdrin
62dc392476 MCOL-5499 Enable ControlFlow for same node communication processing path to avoid DEC queue overloading (#2848) 2023-06-07 15:41:59 +03:00
Roman Nozdrin
4fe9cd64a3 Revert "No boost condition (#2822)" (#2828)
This reverts commit f916e64927.
2023-04-22 15:49:50 +03:00
Leonid Fedorov
f916e64927 No boost condition (#2822)
This patch replaces boost primitives with stdlib counterparts.
2023-04-22 00:42:45 +03:00
david.hall
3b6449842f Merge branch 'develop' into MCOL-4841
# Conflicts:
#	exemgr/main.cpp
#	oam/etc/Columnstore.xml.singleserver
#	primitives/primproc/primproc.cpp
2022-06-09 10:07:26 -05:00
David.Hall
bbb168a846 Mcol 4560 (#2337)
* MCOL-4560 remove unused xml entries and code that references it.
There is reader code and variables for some of these settings, but nobody uses them.
2022-04-18 18:00:17 -04:00
Roman Nozdrin
e174696351 MCOL-5001 This patch merges ExeMgr and PrimProc runtimes
EM and PP are most resource-hungry runtimes.
        The merge enables to control their cummulative
        resource consumption, thread allocation + enables
        zero-copy data exchange b/w local EM and PP facilities.
2022-04-04 11:46:33 +00:00
Leonid Fedorov
65252df4f6 C++20 fixes 2022-03-28 12:32:29 +00:00
David Hall
27dea733c5 MCOL4841 dev port run large join without OOM 2022-02-09 17:33:55 -06:00
Leonid Fedorov
04752ec546 clang format apply 2022-01-21 16:43:49 +00:00
benthompson15
2ae3da45eb MCOL-1175: add ability to encrypt CEJ password and use in Columnstore.xml (#2045) 2021-07-13 11:42:36 -05:00
Roman Nozdrin
14dbb6a8fb MCOL-4770 Installation.PMwithUM is enabled by default so a plugin communicates with a local EM only
Fix in DEC::Setup to allow to re-establish all PM connections in EM
2021-06-24 11:07:54 +00:00
Roman Nozdrin
736b9b81fc MCOL-4694 Add the JobList.DECConnectionsPerQuery setting to allow to
distribute Jobs b/w DEC connections from EM to PPs
2021-06-14 07:15:03 +00:00
Alexey Antipovsky
475104e4d3 [MCOL-4709] Disk-based aggregation
* Introduce multigeneration aggregation

* Do not save unused part of RGDatas to disk
* Add IO error explanation (strerror)

* Reduce memory usage while aggregating
* introduce in-memory generations to better memory utilization

* Try to limit the qty of buckets at a low limit

* Refactor disk aggregation a bit
* pass calculated hash into RowAggregation
* try to keep some RGData with free space in memory

* do not dump more than half of rowgroups to disk if generations are
  allowed, instead start a new generation
* for each thread shift the first processed bucket at each iteration,
  so the generations start more evenly

* Unify temp data location

* Explicitly create temp subdirectories
  whether disk aggregation/join are enabled or not
2021-06-06 16:09:15 +03:00
Roman Nozdrin
2aa5380d51 Remove global lock from OAMCache
Config now uses a single atomic variable to speed up its operations
Config has a special method to re-read a config file if it changed on disk
2021-04-23 15:41:29 +00:00
Sergei Golubchik
586391e1ca compilation failure
error: reference to 'mutex' is ambiguous
note: candidates are: 'class boost::mutex'
note:                 'class std::mutex'
2019-12-19 18:13:39 +01:00
Jose Rojas
e9ee385a11 Merge branch 'develop' into MCOL-3585 2019-12-05 10:03:40 -06:00
jmrojas2332
608d61feb5 MCOL 3585 Minor update to Default for MaxOutstandingRequests 2019-12-05 15:54:00 +00:00
Patrick LeBlanc
c2aaf6cea2 Fixed compiler warning in resource manager. 2019-12-02 15:18:42 -05:00
jmrojas2332
e712df5d0b MCOL 3585 update some defaults in config file 2019-11-21 06:08:49 +00:00
Andrew Hutchings
01446d1e22 Reformat all code to coding standard 2017-10-26 17:18:17 +01:00
Andrew Hutchings
ffcfc41563 MCOL-507 Further ExeMgr performance improvements
This does the following:

* Switch resource manager to a singleton which reduces the amount of
times the XML data is scanned and objects allocated.
* Make the I_S tables use the FE implementation of the system catalog
* Make the I_S.columnstore_columns table use the RID list cache
* Make the extentmap pre-allocate a vector instead of many small allocs
2017-01-16 12:33:27 +00:00
david hill
f6afc42dd0 the begginning 2016-01-06 14:08:59 -06:00