1
0
mirror of https://github.com/mariadb-corporation/mariadb-columnstore-engine.git synced 2025-07-29 08:21:15 +03:00
Commit Graph

6318 Commits

Author SHA1 Message Date
c97c8b672e Merge pull request #2874 from phoeinx/mcol-4632-4648-int-casts
Fix MCOL 4632 and 4648 and further issues when casting to [U]BIGINT.
2023-08-11 21:23:29 +01:00
c672edfda4 Vscode nonroot debug launch2 (#2929)
* -vscode add launch config for all processes
- -fg option used
- tested with primproc only. will others test later

* -change -fg option to -f

---------

Co-authored-by: pgmabv99 <alexey.vorovich@gmail.com>
2023-08-11 23:04:29 +03:00
426557380c fix(cmapi, brm): MCOL-5535 Use an accurate glob when forcing owner of files in /dev/shm on startup (#2923)
Co-authored-by: Roman Nozdrin <rnozdrin@mariadb.com>
2023-08-11 20:04:12 +03:00
06d7345630 Merge pull request #2926 from mariadb-corporation/fix/MCOL-5553-not-running-savebrm
MCOL-5553: Not running save_brm on a single node without CMAPI.
2023-08-11 17:10:41 +01:00
66ec9f039b MCOL-5553: Not running save_brm on a single node without CMAPI.
[add] LOCALHOSTS tuple to mcs-savebrm script
[fix] detecting primary when no working CMAPI found
2023-08-11 18:22:59 +03:00
48562e41f9 feat(datatypes): MCOL-4632 and MCOL-4648, fix cast leads to NULL.
Remove redundant cast.

As C-style casts with a type name in parantheses are interpreted as static_casts this literally just changes the interpretation around (and forces an implicit cast to match the return value of the function).

Switch UBIGINTNULL and UBIGINTEMPTYROW constants for consistency.

Make consistent with relation between BIGINTNULL and BIGINTEMPTYROW & make adapted cast behaviour due to NULL markers more intuitive. (After this change we can simply block the highest possible uint64_t value and if a cast results in it, print the next lower value (2^64 - 2). Previously, (2^64 - 1) was able to be printed, but (2^64 - 2) as being blocked by the UBIGINTNULL constant was not, making finding the appropiate replacement value to give out more confusing.

Introduce MAX_MCS_UBIGINT and MIN_MCS_BIGINT and adapt casts.

Adapt casting to BIGINT to remove NULL marker error.

Add bugfix regression test for MCOL 4632

Add regression test for mcol_4648

Revert "Switch UBIGINTNULL and UBIGINTEMPTYROW constants for consistency."

This reverts commit 83eac11b18937ecb0b4c754dd48e4cb47310f620.
Due to backwards compatability issues.

Refactor casting to MCS[U]Int to datatype functions.

Update regression tests to include other affected datatypes.

Apply formatting.

Refactor according to PR review

Remove redundant new constant, switch to using already existing constant.

Adapt nullstring casting to EMPTYROW markers for backwards compatability.

Adapt tests for backward compatability behaviour allowing text datatypes to be casted to EMPTYROW constant.

Adapt mcol641-functions test according to bug fix.

Update tests according to new expected behaviour.

Adapt tests to new understanding of issue.

Update comments/documentation for MCOL_4632 test.

Adapt to new cast limit logic.

Make bracketing consistent.

Adapt previous regression test to new expected behaviour.
2023-08-11 13:00:30 +00:00
4b20ced6a7 -add vscode/gdb support for non-root user (#2924)
Co-authored-by: pgmabv99 <alexey.vorovich@gmail.com>
2023-08-10 21:10:10 +03:00
f2affca162 MCOL-5478 upgrade tests (#2815)
* Squash all

* Typos

* Only load shared on centos

* Debug

* Install rsyslog

* More exceptions

* One more exception

* rename pkgs

* Remove unnecessary copying

* deb doesn't work on arm with 1-4

* Disable CentOS

* Remove flights

* Use setup-repo

* Exclude deb12
2023-08-04 18:43:52 +03:00
896e8dd769 MCOL-5522 Properly process pm join result count. (#2909)
This patch:
1. Properly processes situation when pm join result count is exceeded.
2. Adds session variable 'columnstore_max_pm_join_result_count` to control the limit.
2023-08-04 16:55:45 +03:00
4f580d109d Fix a compiler error related to signed v/s unsigned integer comparison. (#2915) 2023-08-04 16:54:40 +03:00
a36ea6dbb4 MCOL-5005 Add charset number to system catalog - Part 1.
This patch improves/fixes the existing handling of CHARSET and
COLLATION symbols in the ColumnStore DDL parser.

Also, add fCollate and fCharsetNum member variables to the
ddlpackage::ColumnType class.
2023-07-28 18:36:53 -04:00
1a49a09af3 Merge pull request #2878 from denis0x0D/MCOL-5514_dev_1
MCOL-5514 Parallel disk join step
2023-07-25 14:32:13 +01:00
65cde8c894 feature: pron (#2908)
* feature: Special dictionary, we can pass with session veriable to modify codepaths and behaviour for testing and debugging
2023-07-21 14:02:03 +03:00
510bdd0e7b ci: add backtraces for all CS processes after 4800secs (=1.20 hour) o… (#2903)
* ci: add backtraces for all CS processes after 4800secs (=1.20 hour) of test001.sh

---------

Co-authored-by: mariadb-AndreyPiskunov <andrey.piskunov@mariadb.com>
2023-07-18 20:31:44 +03:00
d86204b9c8 Merge pull request #2899 from mariadb-corporation/simpler_singletons
MCOL-5532: Simpler singletons
2023-07-17 11:28:15 +01:00
70951db9c6 Merge pull request #2901 from mariadb-corporation/pidof-binary-develop
ci: add centos7 pidof package, format jsonnet
2023-07-16 10:49:15 +01:00
5a370cae8c ci: add centos7 for pidof, eu-stack for deb, format jsonnet 2023-07-15 21:18:51 +02:00
e290d436e9 Merge pull request #2895 from denis0x0D/MCOL-5522
MCOL-5522 Temporary disable join match limit on BPP
2023-07-14 16:56:03 +01:00
8d06822be5 atomic stop flag 2023-07-12 18:17:13 +03:00
aead012f9f Simpler OAM 2023-07-12 18:15:51 +03:00
bab29ff495 Simpler Config 2023-07-12 18:15:26 +03:00
cd53fd31a2 Debian 12 to CI (#2896) 2023-07-12 15:35:01 +03:00
2a66ae2ed1 MCOL-5514 Parallel disk join step. 2023-07-11 14:05:14 +03:00
741774eafa Update 'configureEngine.cmake' to not write to source dir (#2893)
It is currently not possible to do a full MariaDB build without touching the
source tree, i.e. such that the the source tree could exist on a read-only
filesystem. This columnstore build file currently creates temporary files in
the source tree to evaluate how the operating system handles them, so it is
one blocker to these isolated builds.

This patch adjusts these tests to create the temporary files in the output
directory, rather than in the source tree.

Signed-off-by: Trevor Gross <tmgross@umich.edu>
2023-07-10 17:45:22 +03:00
a079834288 MCOL-5522 Temporary disable join match limit on BPP
Current join pipeline is not designed to send a temporal result and
continue execution. RowGroup for large side is the same where we store
the matched rows, we cannot continue to iterate over it, requires a
proper refactoring.
2023-07-09 17:56:44 +03:00
ebfb9face2 compiler failures with gcc 12.x
a workaround for something that looks like a bug in a compiler.
Fixes errors like

In file included from /usr/include/c++/12/string:40,
                 from /mnt/server/storage/columnstore/columnstore/utils/funcexp/func_math.cpp:26:
In static member function ‘static constexpr std::char_traits<char>::char_type* std::char_traits<char>::copy(char_type*, const char_type*, std::size_t)’,
    inlined from ‘static constexpr void std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::_S_copy(_CharT*, const _CharT*, size_type) [with _CharT = char; _Traits = std::char_traits<char>; _Alloc = std::allocator<char>]’ at /usr/include/c++/12/bits/basic_string.h:423:21,
    inlined from ‘constexpr std::__cxx11::basic_string<_CharT, _Traits, _Allocator>& std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::_M_replace(size_type, size_type, const _CharT*, size_type) [with _CharT = char; _Traits = std::char_traits<char>; _Alloc = std::allocator<char>]’ at /usr/include/c++/12/bits/basic_string.tcc:532:22,
    inlined from ‘constexpr std::__cxx11::basic_string<_CharT, _Traits, _Alloc>& std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::replace(size_type, size_type, const _CharT*, size_type) [with _CharT = char; _Traits = std::char_traits<char>; _Alloc = std::allocator<char>]’ at /usr/include/c++/12/bits/basic_string.h:2171:19,
    inlined from ‘constexpr std::__cxx11::basic_string<_CharT, _Traits, _Alloc>& std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::insert(size_type, const _CharT*) [with _CharT = char; _Traits = std::char_traits<char>; _Alloc = std::allocator<char>]’ at /usr/include/c++/12/bits/basic_string.h:1928:22,
    inlined from ‘virtual std::string funcexp::Func_format::getStrVal(rowgroup::Row&, funcexp::FunctionParm&, bool&, execplan::CalpontSystemCatalog::ColType&)’ at /mnt/server/storage/columnstore/columnstore/utils/funcexp/func_math.cpp:2008:17:
/usr/include/c++/12/bits/char_traits.h:431:56: error: ‘void* __builtin_memcpy(void*, const void*, long unsigned int)’ accessing 9223372036854775810 or more bytes at offsets 3 and [2, 2147483645] may overlap up to 9223372036854775813 bytes at offset -3 [-Werror=restrict]
  431 |         return static_cast<char_type*>(__builtin_memcpy(__s1, __s2, __n );

$ gcc --version
gcc (Ubuntu 12.2.0-3ubuntu1) 12.2.0
2023-07-04 12:58:18 -04:00
30064e3d4a cpimport.bin doesn't really need libboost_program_options
linking with unused libraries creates a difference in dependencies
between --no-as-needed (gcc default) and --as-needed (default on
Fedora rpmbuild) builds.
2023-07-04 12:58:18 -04:00
0482308b45 disable pch when building boost with gcc 8.x
according to https://bugzilla.redhat.com/show_bug.cgi?id=1806545

  GCC's PCH mechanism is fundamentally incompatible with the security
  hardening requirements for RHEL (PIE in particular).  We are currently
  evaluating the best way to present a more friendly diagnostic for this
  issue.

and also

  we don't turn on hardening the compiler and its binaries in DTS 9.
2023-07-04 12:58:18 -04:00
501da394ca Replace std::set contains method with count
to support Rocky/RHEL/Alma 8 where the std::set
in the stock STL does not have contains method
2023-07-04 12:58:18 -04:00
a8be4a3787 compiler warnings
like

dbcon/joblist/batchprimitiveprocessor-jl.cpp:893:54: error: pointer used after ‘void operator delete [](void*, std::size_t)’ [-Werror=use-after-free]
  893 |           joinResults.reset(new vector<uint32_t>[8192]);
      |                                                      ^
2023-07-04 12:58:18 -04:00
cfcda69b1d Add description for Disk Join Step. (#2882) 2023-07-03 04:03:29 +03:00
b2a573cf82 MCOL-5521: Use Columnstore VERSION file for CMAPI versioning. (#2885)
[fix] columnstore_version.cmake setting extra variables for CMAPI versioning
[fix] CMAPI CmakeLists.txt using columnstore_version.cmake file
[fix] CMAPI VERSION file to template file due to using it inside CMAPI applicatio
n
2023-06-29 15:47:55 +03:00
6d44d2e850 MCOL-5500 Remove another noisy printout. (#2886)
Co-authored-by: Roman Nozdrin <rnozdrin@mariadb.com>
2023-06-29 15:46:00 +03:00
385e580e9c MCOL-5175 Bump overlooked libmariaS3 and added more ENV variables to fine tune initial installation on S3 (#2870)
Co-authored-by: Roman Nozdrin <rnozdrin@mariadb.com>
2023-06-27 17:33:39 +03:00
2aba28d855 Merge pull request #2851 from denis0x0D/MCOL-5477
MCOL-5477 Disk join step improvement.
2023-06-26 11:02:20 +03:00
535b7d5b61 Merge pull request #2877 from tntnatbry/my_init_resource_leak_fix-dev
Fix resource leak in DDLProc/DMLProc/PrimProc/WriteengineServer processes.
2023-06-25 20:55:30 +01:00
18f4131176 Merge pull request #2871 from drrtuy/MCOL-5500-dev
MCOL-5500 Remove noisy printout from CGroupConfigurator method
2023-06-25 20:51:38 +01:00
ab7dfaa25b Fix resource leak in DDLProc/DMLProc/PrimProc/WriteengineServer processes.
As part of the charset support, a call to MY_INIT() was added at the
initialization of the above processes. This call initializes the MySQL
thread environment required by the charset library. However, the
accompanying my_end() call required to terminate this thread environment
was not added at the termination of these process, hence leaking
resources. As a fix, we move the MY_INIT() calls to the Child()
functions of these services and also add the missing my_end() call.
2023-06-23 20:18:32 +00:00
669c88007e Merge pull request #2875 from phoeinx/update-documentation
Update BUILD.md.
2023-06-23 19:06:51 +01:00
1f190a6e75 MCOL-5477 Disk join step improvement.
This patch:
1. Handles corner case when the bucket exceeded the memory limit, but we cannot redistribute the data in this bucket into new buckets based on a hash algorithm, because the rows have the same values.
2. Adds force option for disk join step.
3. Add a option to contol the depth of the partition tree.
2023-06-23 18:40:15 +03:00
cc08929578 Merge pull request #2865 from denis0x0D/MCOL-5512
MCOL-5512 Fix for post join filter.
2023-06-21 10:45:13 +01:00
79b636d853 MCOL-5500 Remove noisy printout from CGroupConfigurator method 2023-06-19 11:24:41 +00:00
d2d0e08690 Fixed stdout and stderr forking porcs handling 2023-06-16 04:38:09 +03:00
5dba0ba53a Merge branch 'mariadb-corporation:develop' into update-documentation 2023-06-13 21:53:03 +02:00
780232b29f Merge pull request #2862 from mariadb-corporation/remove-bind-mount-cgroup
ci: try to remove bind mount host sys fs cgroup
2023-06-12 10:57:53 +01:00
a71b21a491 ci: remove bind mount sys/fs/cgroup 2023-06-11 17:27:59 +02:00
375d162376 MCOL-5500 This patch adds cgroup v2 support with some sanity checks for (#2849)
values reported by cgroups v1
2023-06-09 17:37:21 +03:00
024e6bd358 MCOL-5512 Fix for post join filter.
This patch fixes certain situations where post join filter is not applying.
2023-06-09 11:15:05 +03:00
ab9b95e368 Add one more unit to logs (#2861) 2023-06-07 16:32:21 +03:00
62dc392476 MCOL-5499 Enable ControlFlow for same node communication processing path to avoid DEC queue overloading (#2848) 2023-06-07 15:41:59 +03:00