1
0
mirror of https://github.com/mariadb-corporation/mariadb-columnstore-engine.git synced 2025-07-29 08:21:15 +03:00

6351 Commits

Author SHA1 Message Date
e290d436e9 Merge pull request #2895 from denis0x0D/MCOL-5522
MCOL-5522 Temporary disable join match limit on BPP
2023-07-14 16:56:03 +01:00
8d06822be5 atomic stop flag 2023-07-12 18:17:13 +03:00
aead012f9f Simpler OAM 2023-07-12 18:15:51 +03:00
bab29ff495 Simpler Config 2023-07-12 18:15:26 +03:00
cd53fd31a2 Debian 12 to CI (#2896) 2023-07-12 15:35:01 +03:00
2a66ae2ed1 MCOL-5514 Parallel disk join step. 2023-07-11 14:05:14 +03:00
741774eafa Update 'configureEngine.cmake' to not write to source dir (#2893)
It is currently not possible to do a full MariaDB build without touching the
source tree, i.e. such that the the source tree could exist on a read-only
filesystem. This columnstore build file currently creates temporary files in
the source tree to evaluate how the operating system handles them, so it is
one blocker to these isolated builds.

This patch adjusts these tests to create the temporary files in the output
directory, rather than in the source tree.

Signed-off-by: Trevor Gross <tmgross@umich.edu>
2023-07-10 17:45:22 +03:00
a079834288 MCOL-5522 Temporary disable join match limit on BPP
Current join pipeline is not designed to send a temporal result and
continue execution. RowGroup for large side is the same where we store
the matched rows, we cannot continue to iterate over it, requires a
proper refactoring.
2023-07-09 17:56:44 +03:00
ebfb9face2 compiler failures with gcc 12.x
a workaround for something that looks like a bug in a compiler.
Fixes errors like

In file included from /usr/include/c++/12/string:40,
                 from /mnt/server/storage/columnstore/columnstore/utils/funcexp/func_math.cpp:26:
In static member function ‘static constexpr std::char_traits<char>::char_type* std::char_traits<char>::copy(char_type*, const char_type*, std::size_t)’,
    inlined from ‘static constexpr void std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::_S_copy(_CharT*, const _CharT*, size_type) [with _CharT = char; _Traits = std::char_traits<char>; _Alloc = std::allocator<char>]’ at /usr/include/c++/12/bits/basic_string.h:423:21,
    inlined from ‘constexpr std::__cxx11::basic_string<_CharT, _Traits, _Allocator>& std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::_M_replace(size_type, size_type, const _CharT*, size_type) [with _CharT = char; _Traits = std::char_traits<char>; _Alloc = std::allocator<char>]’ at /usr/include/c++/12/bits/basic_string.tcc:532:22,
    inlined from ‘constexpr std::__cxx11::basic_string<_CharT, _Traits, _Alloc>& std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::replace(size_type, size_type, const _CharT*, size_type) [with _CharT = char; _Traits = std::char_traits<char>; _Alloc = std::allocator<char>]’ at /usr/include/c++/12/bits/basic_string.h:2171:19,
    inlined from ‘constexpr std::__cxx11::basic_string<_CharT, _Traits, _Alloc>& std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::insert(size_type, const _CharT*) [with _CharT = char; _Traits = std::char_traits<char>; _Alloc = std::allocator<char>]’ at /usr/include/c++/12/bits/basic_string.h:1928:22,
    inlined from ‘virtual std::string funcexp::Func_format::getStrVal(rowgroup::Row&, funcexp::FunctionParm&, bool&, execplan::CalpontSystemCatalog::ColType&)’ at /mnt/server/storage/columnstore/columnstore/utils/funcexp/func_math.cpp:2008:17:
/usr/include/c++/12/bits/char_traits.h:431:56: error: ‘void* __builtin_memcpy(void*, const void*, long unsigned int)’ accessing 9223372036854775810 or more bytes at offsets 3 and [2, 2147483645] may overlap up to 9223372036854775813 bytes at offset -3 [-Werror=restrict]
  431 |         return static_cast<char_type*>(__builtin_memcpy(__s1, __s2, __n );

$ gcc --version
gcc (Ubuntu 12.2.0-3ubuntu1) 12.2.0
2023-07-04 12:58:18 -04:00
30064e3d4a cpimport.bin doesn't really need libboost_program_options
linking with unused libraries creates a difference in dependencies
between --no-as-needed (gcc default) and --as-needed (default on
Fedora rpmbuild) builds.
2023-07-04 12:58:18 -04:00
0482308b45 disable pch when building boost with gcc 8.x
according to https://bugzilla.redhat.com/show_bug.cgi?id=1806545

  GCC's PCH mechanism is fundamentally incompatible with the security
  hardening requirements for RHEL (PIE in particular).  We are currently
  evaluating the best way to present a more friendly diagnostic for this
  issue.

and also

  we don't turn on hardening the compiler and its binaries in DTS 9.
2023-07-04 12:58:18 -04:00
501da394ca Replace std::set contains method with count
to support Rocky/RHEL/Alma 8 where the std::set
in the stock STL does not have contains method
2023-07-04 12:58:18 -04:00
a8be4a3787 compiler warnings
like

dbcon/joblist/batchprimitiveprocessor-jl.cpp:893:54: error: pointer used after ‘void operator delete [](void*, std::size_t)’ [-Werror=use-after-free]
  893 |           joinResults.reset(new vector<uint32_t>[8192]);
      |                                                      ^
2023-07-04 12:58:18 -04:00
cfcda69b1d Add description for Disk Join Step. (#2882) 2023-07-03 04:03:29 +03:00
b2a573cf82 MCOL-5521: Use Columnstore VERSION file for CMAPI versioning. (#2885)
[fix] columnstore_version.cmake setting extra variables for CMAPI versioning
[fix] CMAPI CmakeLists.txt using columnstore_version.cmake file
[fix] CMAPI VERSION file to template file due to using it inside CMAPI applicatio
n
2023-06-29 15:47:55 +03:00
6d44d2e850 MCOL-5500 Remove another noisy printout. (#2886)
Co-authored-by: Roman Nozdrin <rnozdrin@mariadb.com>
2023-06-29 15:46:00 +03:00
385e580e9c MCOL-5175 Bump overlooked libmariaS3 and added more ENV variables to fine tune initial installation on S3 (#2870)
Co-authored-by: Roman Nozdrin <rnozdrin@mariadb.com>
2023-06-27 17:33:39 +03:00
2aba28d855 Merge pull request #2851 from denis0x0D/MCOL-5477
MCOL-5477 Disk join step improvement.
2023-06-26 11:02:20 +03:00
535b7d5b61 Merge pull request #2877 from tntnatbry/my_init_resource_leak_fix-dev
Fix resource leak in DDLProc/DMLProc/PrimProc/WriteengineServer processes.
2023-06-25 20:55:30 +01:00
18f4131176 Merge pull request #2871 from drrtuy/MCOL-5500-dev
MCOL-5500 Remove noisy printout from CGroupConfigurator method
2023-06-25 20:51:38 +01:00
ab7dfaa25b Fix resource leak in DDLProc/DMLProc/PrimProc/WriteengineServer processes.
As part of the charset support, a call to MY_INIT() was added at the
initialization of the above processes. This call initializes the MySQL
thread environment required by the charset library. However, the
accompanying my_end() call required to terminate this thread environment
was not added at the termination of these process, hence leaking
resources. As a fix, we move the MY_INIT() calls to the Child()
functions of these services and also add the missing my_end() call.
2023-06-23 20:18:32 +00:00
669c88007e Merge pull request #2875 from phoeinx/update-documentation
Update BUILD.md.
2023-06-23 19:06:51 +01:00
1f190a6e75 MCOL-5477 Disk join step improvement.
This patch:
1. Handles corner case when the bucket exceeded the memory limit, but we cannot redistribute the data in this bucket into new buckets based on a hash algorithm, because the rows have the same values.
2. Adds force option for disk join step.
3. Add a option to contol the depth of the partition tree.
2023-06-23 18:40:15 +03:00
cc08929578 Merge pull request #2865 from denis0x0D/MCOL-5512
MCOL-5512 Fix for post join filter.
2023-06-21 10:45:13 +01:00
79b636d853 MCOL-5500 Remove noisy printout from CGroupConfigurator method 2023-06-19 11:24:41 +00:00
d2d0e08690 Fixed stdout and stderr forking porcs handling 2023-06-16 04:38:09 +03:00
5dba0ba53a Merge branch 'mariadb-corporation:develop' into update-documentation 2023-06-13 21:53:03 +02:00
780232b29f Merge pull request #2862 from mariadb-corporation/remove-bind-mount-cgroup
ci: try to remove bind mount host sys fs cgroup
2023-06-12 10:57:53 +01:00
a71b21a491 ci: remove bind mount sys/fs/cgroup 2023-06-11 17:27:59 +02:00
375d162376 MCOL-5500 This patch adds cgroup v2 support with some sanity checks for (#2849)
values reported by cgroups v1
2023-06-09 17:37:21 +03:00
024e6bd358 MCOL-5512 Fix for post join filter.
This patch fixes certain situations where post join filter is not applying.
2023-06-09 11:15:05 +03:00
ab9b95e368 Add one more unit to logs (#2861) 2023-06-07 16:32:21 +03:00
62dc392476 MCOL-5499 Enable ControlFlow for same node communication processing path to avoid DEC queue overloading (#2848) 2023-06-07 15:41:59 +03:00
cacbbee1c2 MCOL-5175 Increase the maximum effective length of S3 secret used as SHA256 key producing S3 signature (#2859)
Co-authored-by: Roman Nozdrin <rnozdrin@mariadb.com>
2023-06-07 15:24:02 +03:00
1eca477485 MCOL-5496: Fix CI adding CMAPI steps.
[fix] deb packages deps commands
[add] several additional local variables
[fix] packages url
[fix] smoke step
[fix] mtr step
[fix] regression step
[add] cmapipython, cmapibuild, cmapitest and cmapilog steps
[fix] dockerfile step
[fix] build step to include cmapi package on repodata creating
[fix] multi_node_mtr step
[fix] pkg step
[add] cmapi steps to pipelines
[fix] cmapi/CMakeLists.txt to prevent cmake byte-compile .py files for rpm packages
[add] setup-repo.sh file
[fix] now use packages from the repos in tests steps
2023-06-07 10:00:16 +03:00
a079a2c944 MCOL-5496: Merge CMAPI code to engine repo.
[add] cmapi code to engine
2023-06-07 10:00:16 +03:00
77eedd1756 MCOL-5503: Fix broken quarter (#2855)
* Fix broken quarter function
2023-06-02 18:06:58 +03:00
8f93fc3623 MCOL-5493: First portion of UBSan fixes (#2842)
Multiple UB fixes
2023-06-02 17:02:09 +03:00
5ede14c9c0 Fix indentation in BUILD.md. 2023-06-01 19:09:11 +02:00
44633f40ec Update BUILD.md and delete duplicate build/README.md.
This commit updates the BUILD.md with some commments for contributors building from a fork and regarding dependency installation.
2023-06-01 17:03:48 +00:00
0a2e9760ee Fix for JSON_VALUE function to remove OOB stack access (#2852)
MCOL-271 introduced a bug in JSON_VALUE that was discovered during
implementation of ASAN builds. The changes here restore normal
functionality.

In short, changes in MCOL-271 introduced a local variable instead of
reference to a string in ConstantColumn's fResult.strVal. The handling
of ConstantColumn is different because ConstantColumn's value is used
to initialize JSON path once. JSON path value holds pointer to data it
does not own and if there are two or more rows the data can be corrupted
and/or be out of stack bounds.

The changes here introduce reference to a NullString that is held in the
ConstantColumn's fResult.strVal and uses appropriate functions to obtain
data from the NullString. CC's fResult is held by CC and strVal is also
neither changing nor moving during operation, which allow JSON path to
hold correct pointers during multi-row operation.
2023-05-31 15:30:40 +03:00
32482cd928 Add an option in build/bootstrap_mcs.sh to NOT perform a clean install. (#2854)
A new option is added (-n/--no-clean-install) that allows a user to
re-install ColumnStore without deleting the existing db files. This is
useful for testing new code changes in the engine without the need to
re-create the database tables to test the code changes.
2023-05-30 21:04:51 +03:00
d083650f75 Merge pull request #2846 from tntnatbry/MCOL-5480
MCOL-5480 LOAD DATA INFILE incorrectly loads values for MEDIUMINT datatype.
2023-05-26 14:39:06 +01:00
f18c556311 Fix gcc-13 warning and add support for building Fedora (#2845) 2023-05-26 16:30:53 +03:00
c598a9bbed MCOL-5480 LOAD DATA INFILE incorrectly loads values for MEDIUMINT datatype.
Internal memory representation of MEDIUMINT datatype uses 24 bits. This is
true for both MariaDB server as well as ColumnStore. MCS plugin code uses
TypeHandlerSInt24 and TypeHandlerUInt24 classes to respectively convert the
binary representation of the signed and unsigned MEDIUMINT values passed by
the server to the plugin. The plugin then outputs the text representation
of these values into an open file descriptor which is piped to cpimport
for the final load into the MCS db files.

The TypeHandlerXInt24 classes were earlier incorrectly using
WriteBatchField::ColWriteBatchXInt32() functions which operate on a 4 byte
buffer. This resulted in incorrect parsing of MEDIUMINT values. As a fix,
we implement WriteBatchField::ColWriteBatchXInt24() functions which
correctly handle the 24 bit input buffer used for MEDIUMINT datatype.
2023-05-23 16:00:05 -04:00
a90535e1a7 Merge pull request #2844 from drrtuy/obsidian
An initial Obsidian vault as a knowledge base.
2023-05-20 16:01:42 +01:00
26aa9abaf0 An initial Obsidian vault as a knowledge base. 2023-05-20 15:46:19 +03:00
de7ba854bd Merge pull request #2840 from tntnatbry/MCOL-5491
MCOL-5491 Enable StringStore for long strings in JSON_ARRAYAGG processing.
2023-05-17 12:12:03 +01:00
d3778e391e Merge pull request #2827 from mariadb-corporation/rwlockmonitor_datarace
Atomic bools for RWLockMonitor
2023-05-17 12:09:11 +01:00
fed115c9ac Merge pull request #2837 from tntnatbry/MCOL-5357-dev
MCOL-5357 Fix TPC-DS query error "MCS-3009: Unknown column '.<colname>'".
2023-05-17 12:06:49 +01:00