1
0
mirror of https://github.com/mariadb-corporation/mariadb-columnstore-engine.git synced 2025-07-29 08:21:15 +03:00
Commit Graph

96 Commits

Author SHA1 Message Date
449029a827 Deep build refactoring phase 2 (#3564)
* configcpp refactored

* chore(build): massive removals, auto add files to debian install file

* chore(build): configure before autobake

* chore(build): use custom cmake commands for components, mariadb-plugin-columnstore.install generated

* chore(build): install deps as separate step for build-packages

* more deps

* chore(codemanagement, build): build refactoring stage2

* chore(safety): Locked Map for MessageqCpp with a simpler way

 Please enter the commit message for your changes. Lines starting

* chore(codemanagement, ci): better coredumps handling, deps fixed

* Delete build/bootstrap_mcs.py

* Update charset.cpp (add license)
2025-07-17 16:14:10 +04:00
aa7e0fb9b4 Deep build refactoring phase 1 (#3562)
* configcpp refactored
* logging and datatypes refactored

* more dataconvert
* chore(build): massive removals, auto add files to debian install file
* chore(codemanagement): nodeps headers, potentioal library
* chore(build): configure before autobake
* chore(build): use custom cmake commands for components, mariadb-plugin-columnstore.install generated
* chore(build): install deps as separate step for build-packages
* more deps
* check  debian/mariadb-plugin-columnstore.install automatically
* chore(build): add option for multibracnh compilation
* Fix warning
2025-05-30 14:05:21 +04:00
6db2dc668f stubs and cmake formatting 2025-05-20 18:22:59 +04:00
a0bee173f6 chore(build): fixes to satisfy clang19 warnings 2025-05-15 19:05:38 +04:00
21ebd1ac20 feat(bytestream): serialize long strings in the common way 2025-04-11 15:21:07 +02:00
4bea7e59a0 feat(PrimProc): MCOL-5852 disk-based GROUP_CONCAT & JSON_ARRAYAGG
* move GROUP_CONCAT/JSON_ARRAYAGG storage to the RowGroup from
  the RowAggregation*
* internal data structures (de)serialization
* get rid of a specialized classes for processing JSON_ARRAYAGG
* move the memory accounting to disk-based aggregation classes
* allow aggregation generations to be used for queries with
  GROUP_CONCAT/JSON_ARRAYAGG
* Remove the thread id from the error message as it interferes with the mtr
2025-04-11 15:21:07 +02:00
db2c82519d chore(): another cleanup round 2025-03-27 22:12:48 +00:00
3dfc8cd454 feat(): first cleanup 2025-03-27 22:12:48 +00:00
a6de8ec1ac feat(): dangling pointer/ref issue has been solved for both RGData and BS 2025-03-27 22:12:48 +00:00
71ed9cabe0 feat(): propagate long strings SP type change 2025-03-27 22:12:48 +00:00
02b8ea1331 feat(PP,ByteStream): new counting memory allocator 2025-03-27 22:12:48 +00:00
0ab03c7258 chore(codestyle): mark virtual methods as override 2025-02-21 20:01:34 +04:00
6f6e69815d feat(bytestream,serdes): Distribute BS buf size data type change to avoid implicit data type narrowing 2024-11-08 16:28:51 +04:00
a947f7341c feat(bytestream,serdes): BS buffer size type is uint64_t
This necessary to handle 64bit RGData, that comes as
	a separate patch. The pair of patches would allow to
	have PM joins when SmallSide size > 4GB.
2024-11-08 16:28:51 +04:00
9a9b5f8036 fix(join,threadpool): MCOL-5565: MCOL-5636: MCOL-5645: port from develop-23.02 to [stable-23.10] (#3127)
* fix(threadpool): MCOL-5565 queries stuck in FairThreadScheduler. (#3100)

Meta Primitive Jobs, .e.g ADD_JOINER, LAST_JOINER stuck
	in Fair scheduler without out-of-band scheduler. Add OOB
	scheduler back to remedy the issue.

* fix(messageqcpp): MCOL-5636 same node communication crashes transmiting PP errors to EM b/c error messaging leveraged socket that was a nullptr. (#3106)

* fix(threadpool): MCOL-5645 errenous threadpool Job ctor implictly sets socket shared_ptr to nullptr causing sigabrt when threadpool returns an error (#3125)

---------

Co-authored-by: drrtuy <roman.nozdrin@mariadb.com>
2024-02-14 14:56:07 +03:00
1f71847e99 fix(packaging) dh_missing: warning are treated as errors for buildbot debians
dh_missing: warning: Compatibility levels before 10 are deprecated (level 9 in use)
dh_missing: warning: usr/lib/x86_64-linux-gnu/libmessageqcpp.a exists in debian/tmp but is not installed to anywhere
dh_missing: warning: usr/lib/x86_64-linux-gnu/libpron.a exists in debian/tmp but is not installed to anywhere

so do not install static libraries as targets on CMake
2023-10-04 13:20:24 -04:00
176eefcc57 MCOL-5456 codebase preparation for colab with CS 11.1 2023-05-08 18:54:47 +00:00
4fe9cd64a3 Revert "No boost condition (#2822)" (#2828)
This reverts commit f916e64927.
2023-04-22 15:49:50 +03:00
f916e64927 No boost condition (#2822)
This patch replaces boost primitives with stdlib counterparts.
2023-04-22 00:42:45 +03:00
c2d0fa24da replace boost::shared_array<T> to std::shared_ptr<T[]> 2023-04-14 10:33:27 +00:00
a508b86091 remove boost/shared_array include 2023-04-14 09:42:50 +00:00
2e1394149b MCOL-5464: Fixes of bugs from ASAN warnings, part one (#2792)
* Fixes of bugs from ASAN warnings, part one

* MQC as static library, with nifty counter for global map and mutex

* Switch clang to 16

* link messageqcpp to execplan
2023-04-04 02:33:23 +03:00
b53c231ca6 MCOL-271 empty strings should not be NULLs (#2794)
This patch improves handling of NULLs in textual fields in ColumnStore.
Previously empty strings were considered NULLs and it could be a problem
if data scheme allows for empty strings. It was also one of major
reasons of behavior difference between ColumnStore and other engines in
MariaDB family.

Also, this patch fixes some other bugs and incorrect behavior, for
example, incorrect comparison for "column <= ''" which evaluates to
constant True for all purposes before this patch.
2023-03-30 21:18:29 +03:00
a0cb59f5bc This patch resolves UB with non-existing mutex usage in MQC on runtime shutdown 2023-03-09 20:15:28 +00:00
56f2346083 Remove windows ifdefs 2023-03-02 15:59:42 +00:00
a9d8924683 MCOL-5166 This patch adds support for in-memory communication b/w EM to PP via a shared queue in DEC class
JobList low-level code relateod to primitive jobs now uses shared pointers instead of ByteStream refs talking to DEC
b/c same-node EM-PP communication now goes over a queue in DEC instead of a network hop.
PP now has a separate thread that processes the primitive job messages from that DEC queue.
2022-08-04 18:51:31 +03:00
1624c347f6 MCOL-5152 This patch enables PP to put ByteStreams into DEC input queue directly for a local PP-EM connection 2022-07-04 09:06:40 +00:00
9a24934728 MCOL-4841 remove BOOST_BIND_GLOBAL_PLACEHOLDERS
drone has this defined on the command line
2022-06-14 16:16:38 -05:00
d4cf894edc MCOL-4841 fix some compiler issues 2022-06-14 14:32:01 -05:00
3919c541ac New warnfixes (#2254)
* Fix clang warnings

* Remove vim tab guides

* initialize variables

* 'strncpy' output truncated before terminating nul copying as many bytes from a string as its length

* Fix ISO C++17 does not allow 'register' storage class specifier for outdated bison

* chars are unsigned on ARM, having  if (ival < 0) always false

* chars are unsigned by default on ARM and comparison with -1 if always true
2022-02-17 13:08:58 +03:00
04752ec546 clang format apply 2022-01-21 16:43:49 +00:00
01f3ceb437 replace header guards with #pragma once 2022-01-21 15:24:58 +00:00
1b6d278404 Handle error during parsing of bytestream.
The error can occur in case we send the bytestream with `old` format header.
2021-12-30 12:30:21 +03:00
6393c6d019 MCOL-4810 Add support for missed operation for longStrings. 2021-10-28 10:02:02 +03:00
7bda598fbf MCOL-4810 Redundant copying and wasting memory in PrimProc
This patch eliminates a copying `long string`s into the bytestream.
2021-08-26 12:16:23 +03:00
a292585b8c MCOL-4815 ColumnCommand was replaced with a set of derived classes specified by
column width

RTSCommand was modified to use a fabric that produces CC class based on column width

NB this patch doesn't affect PseudoCC that also leverages ColumnCommand
2021-07-21 12:54:14 +00:00
866dc25729 Merge pull request #1842 from denis0x0D/MCOL-987_LZ
MCOL-987 LZ4 compression support.
2021-07-07 13:13:18 +03:00
fb5ba84212 MCOL-4802 Removed ByteStream methods for bool manipulations and add some logging into I_S.columnstore_files 2021-07-07 07:16:30 +00:00
cc1c3629c5 MCOL-987 Add LZ4 compression.
* Adds CompressInterfaceLZ4 which uses LZ4 API for compress/uncompress.
* Adds CMake machinery to search LZ4 on running host.
* All methods which use static data and do not modify any internal data - become `static`,
  so we can use them without creation of the specific object. This is possible, because
  the header specification has not been modified. We still use 2 sections in header, first
  one with file meta data, the second one with pointers for compressed chunks.
* Methods `compress`, `uncompress`, `maxCompressedSize`, `getUncompressedSize` - become
  pure virtual, so we can override them for the other compression algos.
* Adds method `getChunkMagicNumber`, so we can verify chunk magic number
  for each compression algo.
* Renames "s/IDBCompressInterface/CompressInterface/g" according to requirement.
2021-07-06 18:04:37 +03:00
bed0b7c6bc MCOL-4173 This patch adds support for wide-DECIMAL INNER, OUTER, SEMI, functional JOINs
based on top of TypelessData
2021-06-24 08:07:23 +00:00
90397dfed0 MCOL-4675 DMLProc now automatically and gracefully shutdowns when a cluster state is set to
SS_SHUTDOWN_PENDING | SS_ROLLBACK
2021-05-27 11:07:32 +00:00
c9b353e975 Simplify PMS connection entries configuration 2021-04-16 10:52:13 +00:00
3eb26c0d4a MCOL-4313 Introduced TSInt128 that is a storage class for int128
Removed uint128 from joblist/lbidlist.*

Another toString() method for wide-decimal that is EMPTY/NULL aware

Unified decimal processing in WF functions

Fixed a potential issue in EqualCompData::operator() for
    wide-decimal processing

Fixed some signedness warnings
2020-11-18 13:53:15 +00:00
d5c6645ba1 Adding mcs_basic_types.h
For now it consists of only:

using int128_t = __int128;
using uint128_t = unsigned __int128;

All new privitive data types should go into this file in the future.
2020-11-18 13:53:15 +00:00
844472d812 MCOL-4313 Very fragile but high speed approach with inline ASM
GCC compiler uses aligned versions of SIMD instructions expecting
aligned memory blocks that is hard to implement now
2020-11-18 13:52:20 +00:00
2d044fd7f1 MCOL-4171 Fix comments in bytestream.h 2020-11-18 13:52:19 +00:00
1588ebe439 MCOL-641 Clean up primitives code
Add int128_t support into ByteStream

Fixed UTs broken after collation patch
2020-11-18 13:52:19 +00:00
638202417f MCOL-4171 2020-11-18 13:52:19 +00:00
62d0c82d75 MCOL-641 1. Templatized convertValueNum() function.
2. Allocate int128_t buffers in batchprimitiveprocessor if
a query involves wide decimal columns.
2020-11-18 13:47:44 +00:00
f73de30427 MCOL-641 This commit introduces GTest Suite into CS.
Binary NULL magic now consists of a series of BINARYEMPTYROW-s + BINARYNULL
in the end.

ByteStream now has hexbyte alias.

Added ColumnCommand::getEmptyRowValue to support 16 byte EMPTY values.
2020-11-18 13:47:01 +00:00