1
0
mirror of https://github.com/mariadb-corporation/mariadb-columnstore-engine.git synced 2025-09-13 21:21:50 +03:00
Commit Graph

29 Commits

Author SHA1 Message Date
drrtuy
d8bef9dc11 fix(UT): added dummy RM ctor to be used with UTs 2025-05-02 10:11:40 +01:00
drrtuy
252b3ddbef fix(perf,allocator): adding unit test to check CountingAllocator behavior when it is wrapped in STLPoolAllocator 2025-05-02 10:11:40 +01:00
Aleksei Antipovskii
4bea7e59a0 feat(PrimProc): MCOL-5852 disk-based GROUP_CONCAT & JSON_ARRAYAGG
* move GROUP_CONCAT/JSON_ARRAYAGG storage to the RowGroup from
  the RowAggregation*
* internal data structures (de)serialization
* get rid of a specialized classes for processing JSON_ARRAYAGG
* move the memory accounting to disk-based aggregation classes
* allow aggregation generations to be used for queries with
  GROUP_CONCAT/JSON_ARRAYAGG
* Remove the thread id from the error message as it interferes with the mtr
2025-04-11 15:21:07 +02:00
drrtuy
729db829a2 feat(RM,allocators): use atomic counters math abstraction 2025-03-27 22:12:48 +00:00
drrtuy
a4c4d33ee7 feat(): zerocopy TNS case and JOIN results RGData with CountingAllocator 2025-03-27 22:12:48 +00:00
drrtuy
397b3ff729 feat(): change ref to atomic with ptr to atomic 2025-03-27 22:12:48 +00:00
drrtuy
02b8ea1331 feat(PP,ByteStream): new counting memory allocator 2025-03-27 22:12:48 +00:00
Aleksei Antipovskii
0ab03c7258 chore(codestyle): mark virtual methods as override 2025-02-21 20:01:34 +04:00
Roman Nozdrin
62dc392476 MCOL-5499 Enable ControlFlow for same node communication processing path to avoid DEC queue overloading (#2848) 2023-06-07 15:41:59 +03:00
Leonid Fedorov
56f2346083 Remove windows ifdefs 2023-03-02 15:59:42 +00:00
david.hall
3b6449842f Merge branch 'develop' into MCOL-4841
# Conflicts:
#	exemgr/main.cpp
#	oam/etc/Columnstore.xml.singleserver
#	primitives/primproc/primproc.cpp
2022-06-09 10:07:26 -05:00
David.Hall
bbb168a846 Mcol 4560 (#2337)
* MCOL-4560 remove unused xml entries and code that references it.
There is reader code and variables for some of these settings, but nobody uses them.
2022-04-18 18:00:17 -04:00
Roman Nozdrin
e174696351 MCOL-5001 This patch merges ExeMgr and PrimProc runtimes
EM and PP are most resource-hungry runtimes.
        The merge enables to control their cummulative
        resource consumption, thread allocation + enables
        zero-copy data exchange b/w local EM and PP facilities.
2022-04-04 11:46:33 +00:00
David Hall
27dea733c5 MCOL4841 dev port run large join without OOM 2022-02-09 17:33:55 -06:00
Leonid Fedorov
04752ec546 clang format apply 2022-01-21 16:43:49 +00:00
Leonid Fedorov
01f3ceb437 replace header guards with #pragma once 2022-01-21 15:24:58 +00:00
Roman Nozdrin
736b9b81fc MCOL-4694 Add the JobList.DECConnectionsPerQuery setting to allow to
distribute Jobs b/w DEC connections from EM to PPs
2021-06-14 07:15:03 +00:00
Alexey Antipovsky
475104e4d3 [MCOL-4709] Disk-based aggregation
* Introduce multigeneration aggregation

* Do not save unused part of RGDatas to disk
* Add IO error explanation (strerror)

* Reduce memory usage while aggregating
* introduce in-memory generations to better memory utilization

* Try to limit the qty of buckets at a low limit

* Refactor disk aggregation a bit
* pass calculated hash into RowAggregation
* try to keep some RGData with free space in memory

* do not dump more than half of rowgroups to disk if generations are
  allowed, instead start a new generation
* for each thread shift the first processed bucket at each iteration,
  so the generations start more evenly

* Unify temp data location

* Explicitly create temp subdirectories
  whether disk aggregation/join are enabled or not
2021-06-06 16:09:15 +03:00
Roman Nozdrin
2aa5380d51 Remove global lock from OAMCache
Config now uses a single atomic variable to speed up its operations
Config has a special method to re-read a config file if it changed on disk
2021-04-23 15:41:29 +00:00
jmrojas2332
c0319a5438 MCOL 3585 Related. Updated Comment in config file 2019-12-04 23:20:01 +00:00
jmrojas2332
e712df5d0b MCOL 3585 update some defaults in config file 2019-11-21 06:08:49 +00:00
David Hill
4e6e5647ef MCOL-520 2018-09-21 10:40:05 -05:00
Andrew Hutchings
0d7c0f7ae4 Merge branch 'develop-1.1' into dev-merge-up-20180202 2018-02-02 14:53:36 +00:00
David Hall
9749d825b5 MCOL-1165 use the threadpool's idle down feature 2018-01-16 13:26:44 -06:00
David Hall
504fcf0778 MCOL-1128 Allow exeMgrThreadPool to use unlimited threads. The number will be limited by the number of sessions allowed in server. 2018-01-12 16:29:01 -06:00
Andrew Hutchings
01446d1e22 Reformat all code to coding standard 2017-10-26 17:18:17 +01:00
David Hall
e09b7e10c5 MCOL-513 Threadpool to unlimited threads when queuesize = 0. Idle down after 10 minutes 2017-02-13 11:56:28 -06:00
Andrew Hutchings
ffcfc41563 MCOL-507 Further ExeMgr performance improvements
This does the following:

* Switch resource manager to a singleton which reduces the amount of
times the XML data is scanned and objects allocated.
* Make the I_S tables use the FE implementation of the system catalog
* Make the I_S.columnstore_columns table use the RID list cache
* Make the extentmap pre-allocate a vector instead of many small allocs
2017-01-16 12:33:27 +00:00
david hill
f6afc42dd0 the begginning 2016-01-06 14:08:59 -06:00