1
0
mirror of https://github.com/mariadb-corporation/mariadb-columnstore-engine.git synced 2025-09-13 21:21:50 +03:00
Commit Graph

6318 Commits

Author SHA1 Message Date
drrtuy
c97c8b672e Merge pull request #2874 from phoeinx/mcol-4632-4648-int-casts
Fix MCOL 4632 and 4648 and further issues when casting to [U]BIGINT.
2023-08-11 21:23:29 +01:00
mariadb-AlexeyVorovich
c672edfda4 Vscode nonroot debug launch2 (#2929)
* -vscode add launch config for all processes
- -fg option used
- tested with primproc only. will others test later

* -change -fg option to -f

---------

Co-authored-by: pgmabv99 <alexey.vorovich@gmail.com>
2023-08-11 23:04:29 +03:00
drrtuy
426557380c fix(cmapi, brm): MCOL-5535 Use an accurate glob when forcing owner of files in /dev/shm on startup (#2923)
Co-authored-by: Roman Nozdrin <rnozdrin@mariadb.com>
2023-08-11 20:04:12 +03:00
drrtuy
06d7345630 Merge pull request #2926 from mariadb-corporation/fix/MCOL-5553-not-running-savebrm
MCOL-5553: Not running save_brm on a single node without CMAPI.
2023-08-11 17:10:41 +01:00
mariadb-AlanMologorsky
66ec9f039b MCOL-5553: Not running save_brm on a single node without CMAPI.
[add] LOCALHOSTS tuple to mcs-savebrm script
[fix] detecting primary when no working CMAPI found
2023-08-11 18:22:59 +03:00
Theresa Hradilak
48562e41f9 feat(datatypes): MCOL-4632 and MCOL-4648, fix cast leads to NULL.
Remove redundant cast.

As C-style casts with a type name in parantheses are interpreted as static_casts this literally just changes the interpretation around (and forces an implicit cast to match the return value of the function).

Switch UBIGINTNULL and UBIGINTEMPTYROW constants for consistency.

Make consistent with relation between BIGINTNULL and BIGINTEMPTYROW & make adapted cast behaviour due to NULL markers more intuitive. (After this change we can simply block the highest possible uint64_t value and if a cast results in it, print the next lower value (2^64 - 2). Previously, (2^64 - 1) was able to be printed, but (2^64 - 2) as being blocked by the UBIGINTNULL constant was not, making finding the appropiate replacement value to give out more confusing.

Introduce MAX_MCS_UBIGINT and MIN_MCS_BIGINT and adapt casts.

Adapt casting to BIGINT to remove NULL marker error.

Add bugfix regression test for MCOL 4632

Add regression test for mcol_4648

Revert "Switch UBIGINTNULL and UBIGINTEMPTYROW constants for consistency."

This reverts commit 83eac11b18937ecb0b4c754dd48e4cb47310f620.
Due to backwards compatability issues.

Refactor casting to MCS[U]Int to datatype functions.

Update regression tests to include other affected datatypes.

Apply formatting.

Refactor according to PR review

Remove redundant new constant, switch to using already existing constant.

Adapt nullstring casting to EMPTYROW markers for backwards compatability.

Adapt tests for backward compatability behaviour allowing text datatypes to be casted to EMPTYROW constant.

Adapt mcol641-functions test according to bug fix.

Update tests according to new expected behaviour.

Adapt tests to new understanding of issue.

Update comments/documentation for MCOL_4632 test.

Adapt to new cast limit logic.

Make bracketing consistent.

Adapt previous regression test to new expected behaviour.
2023-08-11 13:00:30 +00:00
mariadb-AlexeyVorovich
4b20ced6a7 -add vscode/gdb support for non-root user (#2924)
Co-authored-by: pgmabv99 <alexey.vorovich@gmail.com>
2023-08-10 21:10:10 +03:00
Andrey Piskunov
f2affca162 MCOL-5478 upgrade tests (#2815)
* Squash all

* Typos

* Only load shared on centos

* Debug

* Install rsyslog

* More exceptions

* One more exception

* rename pkgs

* Remove unnecessary copying

* deb doesn't work on arm with 1-4

* Disable CentOS

* Remove flights

* Use setup-repo

* Exclude deb12
2023-08-04 18:43:52 +03:00
Denis Khalikov
896e8dd769 MCOL-5522 Properly process pm join result count. (#2909)
This patch:
1. Properly processes situation when pm join result count is exceeded.
2. Adds session variable 'columnstore_max_pm_join_result_count` to control the limit.
2023-08-04 16:55:45 +03:00
Gagan Goel
4f580d109d Fix a compiler error related to signed v/s unsigned integer comparison. (#2915) 2023-08-04 16:54:40 +03:00
Gagan Goel
a36ea6dbb4 MCOL-5005 Add charset number to system catalog - Part 1.
This patch improves/fixes the existing handling of CHARSET and
COLLATION symbols in the ColumnStore DDL parser.

Also, add fCollate and fCharsetNum member variables to the
ddlpackage::ColumnType class.
2023-07-28 18:36:53 -04:00
drrtuy
1a49a09af3 Merge pull request #2878 from denis0x0D/MCOL-5514_dev_1
MCOL-5514 Parallel disk join step
2023-07-25 14:32:13 +01:00
Leonid Fedorov
65cde8c894 feature: pron (#2908)
* feature: Special dictionary, we can pass with session veriable to modify codepaths and behaviour for testing and debugging
2023-07-21 14:02:03 +03:00
Leonid Fedorov
510bdd0e7b ci: add backtraces for all CS processes after 4800secs (=1.20 hour) o… (#2903)
* ci: add backtraces for all CS processes after 4800secs (=1.20 hour) of test001.sh

---------

Co-authored-by: mariadb-AndreyPiskunov <andrey.piskunov@mariadb.com>
2023-07-18 20:31:44 +03:00
drrtuy
d86204b9c8 Merge pull request #2899 from mariadb-corporation/simpler_singletons
MCOL-5532: Simpler singletons
2023-07-17 11:28:15 +01:00
drrtuy
70951db9c6 Merge pull request #2901 from mariadb-corporation/pidof-binary-develop
ci: add centos7 pidof package, format jsonnet
2023-07-16 10:49:15 +01:00
mariadb-RomanNavrotskiy
5a370cae8c ci: add centos7 for pidof, eu-stack for deb, format jsonnet 2023-07-15 21:18:51 +02:00
drrtuy
e290d436e9 Merge pull request #2895 from denis0x0D/MCOL-5522
MCOL-5522 Temporary disable join match limit on BPP
2023-07-14 16:56:03 +01:00
Leonid Fedorov
8d06822be5 atomic stop flag 2023-07-12 18:17:13 +03:00
Leonid Fedorov
aead012f9f Simpler OAM 2023-07-12 18:15:51 +03:00
Leonid Fedorov
bab29ff495 Simpler Config 2023-07-12 18:15:26 +03:00
Leonid Fedorov
cd53fd31a2 Debian 12 to CI (#2896) 2023-07-12 15:35:01 +03:00
Denis Khalikov
2a66ae2ed1 MCOL-5514 Parallel disk join step. 2023-07-11 14:05:14 +03:00
Trevor Gross
741774eafa Update 'configureEngine.cmake' to not write to source dir (#2893)
It is currently not possible to do a full MariaDB build without touching the
source tree, i.e. such that the the source tree could exist on a read-only
filesystem. This columnstore build file currently creates temporary files in
the source tree to evaluate how the operating system handles them, so it is
one blocker to these isolated builds.

This patch adjusts these tests to create the temporary files in the output
directory, rather than in the source tree.

Signed-off-by: Trevor Gross <tmgross@umich.edu>
2023-07-10 17:45:22 +03:00
Denis Khalikov
a079834288 MCOL-5522 Temporary disable join match limit on BPP
Current join pipeline is not designed to send a temporal result and
continue execution. RowGroup for large side is the same where we store
the matched rows, we cannot continue to iterate over it, requires a
proper refactoring.
2023-07-09 17:56:44 +03:00
Sergei Golubchik
ebfb9face2 compiler failures with gcc 12.x
a workaround for something that looks like a bug in a compiler.
Fixes errors like

In file included from /usr/include/c++/12/string:40,
                 from /mnt/server/storage/columnstore/columnstore/utils/funcexp/func_math.cpp:26:
In static member function ‘static constexpr std::char_traits<char>::char_type* std::char_traits<char>::copy(char_type*, const char_type*, std::size_t)’,
    inlined from ‘static constexpr void std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::_S_copy(_CharT*, const _CharT*, size_type) [with _CharT = char; _Traits = std::char_traits<char>; _Alloc = std::allocator<char>]’ at /usr/include/c++/12/bits/basic_string.h:423:21,
    inlined from ‘constexpr std::__cxx11::basic_string<_CharT, _Traits, _Allocator>& std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::_M_replace(size_type, size_type, const _CharT*, size_type) [with _CharT = char; _Traits = std::char_traits<char>; _Alloc = std::allocator<char>]’ at /usr/include/c++/12/bits/basic_string.tcc:532:22,
    inlined from ‘constexpr std::__cxx11::basic_string<_CharT, _Traits, _Alloc>& std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::replace(size_type, size_type, const _CharT*, size_type) [with _CharT = char; _Traits = std::char_traits<char>; _Alloc = std::allocator<char>]’ at /usr/include/c++/12/bits/basic_string.h:2171:19,
    inlined from ‘constexpr std::__cxx11::basic_string<_CharT, _Traits, _Alloc>& std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::insert(size_type, const _CharT*) [with _CharT = char; _Traits = std::char_traits<char>; _Alloc = std::allocator<char>]’ at /usr/include/c++/12/bits/basic_string.h:1928:22,
    inlined from ‘virtual std::string funcexp::Func_format::getStrVal(rowgroup::Row&, funcexp::FunctionParm&, bool&, execplan::CalpontSystemCatalog::ColType&)’ at /mnt/server/storage/columnstore/columnstore/utils/funcexp/func_math.cpp:2008:17:
/usr/include/c++/12/bits/char_traits.h:431:56: error: ‘void* __builtin_memcpy(void*, const void*, long unsigned int)’ accessing 9223372036854775810 or more bytes at offsets 3 and [2, 2147483645] may overlap up to 9223372036854775813 bytes at offset -3 [-Werror=restrict]
  431 |         return static_cast<char_type*>(__builtin_memcpy(__s1, __s2, __n );

$ gcc --version
gcc (Ubuntu 12.2.0-3ubuntu1) 12.2.0
2023-07-04 12:58:18 -04:00
Sergei Golubchik
30064e3d4a cpimport.bin doesn't really need libboost_program_options
linking with unused libraries creates a difference in dependencies
between --no-as-needed (gcc default) and --as-needed (default on
Fedora rpmbuild) builds.
2023-07-04 12:58:18 -04:00
Sergei Golubchik
0482308b45 disable pch when building boost with gcc 8.x
according to https://bugzilla.redhat.com/show_bug.cgi?id=1806545

  GCC's PCH mechanism is fundamentally incompatible with the security
  hardening requirements for RHEL (PIE in particular).  We are currently
  evaluating the best way to present a more friendly diagnostic for this
  issue.

and also

  we don't turn on hardening the compiler and its binaries in DTS 9.
2023-07-04 12:58:18 -04:00
Leonid Fedorov
501da394ca Replace std::set contains method with count
to support Rocky/RHEL/Alma 8 where the std::set
in the stock STL does not have contains method
2023-07-04 12:58:18 -04:00
Sergei Golubchik
a8be4a3787 compiler warnings
like

dbcon/joblist/batchprimitiveprocessor-jl.cpp:893:54: error: pointer used after ‘void operator delete [](void*, std::size_t)’ [-Werror=use-after-free]
  893 |           joinResults.reset(new vector<uint32_t>[8192]);
      |                                                      ^
2023-07-04 12:58:18 -04:00
Denis Khalikov
cfcda69b1d Add description for Disk Join Step. (#2882) 2023-07-03 04:03:29 +03:00
Alan Mologorsky
b2a573cf82 MCOL-5521: Use Columnstore VERSION file for CMAPI versioning. (#2885)
[fix] columnstore_version.cmake setting extra variables for CMAPI versioning
[fix] CMAPI CmakeLists.txt using columnstore_version.cmake file
[fix] CMAPI VERSION file to template file due to using it inside CMAPI applicatio
n
2023-06-29 15:47:55 +03:00
drrtuy
6d44d2e850 MCOL-5500 Remove another noisy printout. (#2886)
Co-authored-by: Roman Nozdrin <rnozdrin@mariadb.com>
2023-06-29 15:46:00 +03:00
drrtuy
385e580e9c MCOL-5175 Bump overlooked libmariaS3 and added more ENV variables to fine tune initial installation on S3 (#2870)
Co-authored-by: Roman Nozdrin <rnozdrin@mariadb.com>
2023-06-27 17:33:39 +03:00
Denis Khalikov
2aba28d855 Merge pull request #2851 from denis0x0D/MCOL-5477
MCOL-5477 Disk join step improvement.
2023-06-26 11:02:20 +03:00
Roman Nozdrin
535b7d5b61 Merge pull request #2877 from tntnatbry/my_init_resource_leak_fix-dev
Fix resource leak in DDLProc/DMLProc/PrimProc/WriteengineServer processes.
2023-06-25 20:55:30 +01:00
Roman Nozdrin
18f4131176 Merge pull request #2871 from drrtuy/MCOL-5500-dev
MCOL-5500 Remove noisy printout from CGroupConfigurator method
2023-06-25 20:51:38 +01:00
Gagan Goel
ab7dfaa25b Fix resource leak in DDLProc/DMLProc/PrimProc/WriteengineServer processes.
As part of the charset support, a call to MY_INIT() was added at the
initialization of the above processes. This call initializes the MySQL
thread environment required by the charset library. However, the
accompanying my_end() call required to terminate this thread environment
was not added at the termination of these process, hence leaking
resources. As a fix, we move the MY_INIT() calls to the Child()
functions of these services and also add the missing my_end() call.
2023-06-23 20:18:32 +00:00
Roman Nozdrin
669c88007e Merge pull request #2875 from phoeinx/update-documentation
Update BUILD.md.
2023-06-23 19:06:51 +01:00
Denis Khalikov
1f190a6e75 MCOL-5477 Disk join step improvement.
This patch:
1. Handles corner case when the bucket exceeded the memory limit, but we cannot redistribute the data in this bucket into new buckets based on a hash algorithm, because the rows have the same values.
2. Adds force option for disk join step.
3. Add a option to contol the depth of the partition tree.
2023-06-23 18:40:15 +03:00
Roman Nozdrin
cc08929578 Merge pull request #2865 from denis0x0D/MCOL-5512
MCOL-5512 Fix for post join filter.
2023-06-21 10:45:13 +01:00
Roman Nozdrin
79b636d853 MCOL-5500 Remove noisy printout from CGroupConfigurator method 2023-06-19 11:24:41 +00:00
Leonid Fedorov
d2d0e08690 Fixed stdout and stderr forking porcs handling 2023-06-16 04:38:09 +03:00
Theresa Hradilak
5dba0ba53a Merge branch 'mariadb-corporation:develop' into update-documentation 2023-06-13 21:53:03 +02:00
Roman Nozdrin
780232b29f Merge pull request #2862 from mariadb-corporation/remove-bind-mount-cgroup
ci: try to remove bind mount host sys fs cgroup
2023-06-12 10:57:53 +01:00
mariadb-RomanNavrotskiy
a71b21a491 ci: remove bind mount sys/fs/cgroup 2023-06-11 17:27:59 +02:00
Roman Nozdrin
375d162376 MCOL-5500 This patch adds cgroup v2 support with some sanity checks for (#2849)
values reported by cgroups v1
2023-06-09 17:37:21 +03:00
Denis Khalikov
024e6bd358 MCOL-5512 Fix for post join filter.
This patch fixes certain situations where post join filter is not applying.
2023-06-09 11:15:05 +03:00
Leonid Fedorov
ab9b95e368 Add one more unit to logs (#2861) 2023-06-07 16:32:21 +03:00
Roman Nozdrin
62dc392476 MCOL-5499 Enable ControlFlow for same node communication processing path to avoid DEC queue overloading (#2848) 2023-06-07 15:41:59 +03:00