* fix(threadpool): MCOL-5565 queries stuck in FairThreadScheduler. (#3100)
Meta Primitive Jobs, .e.g ADD_JOINER, LAST_JOINER stuck
in Fair scheduler without out-of-band scheduler. Add OOB
scheduler back to remedy the issue.
* fix(messageqcpp): MCOL-5636 same node communication crashes transmiting PP errors to EM b/c error messaging leveraged socket that was a nullptr. (#3106)
* fix(threadpool): MCOL-5645 errenous threadpool Job ctor implictly sets socket shared_ptr to nullptr causing sigabrt when threadpool returns an error (#3125)
---------
Co-authored-by: drrtuy <roman.nozdrin@mariadb.com>
CI ocassionaly stuck running test001 b/c PP threadpool endlessly reschedules
meta jobs, e.g. BATCH_PRIMITIVE_CREATE, which ByteStreams were somehow damaged or read out.
Co-authored-by: Leonid Fedorov <leonid.fedorov@mariadb.com>
* more build dependencies
* fix for cmake < 3.11
It cannot do ADD_LIBRARY(... ALIAS ...) on IMPORTED targets
* another fix for cmake 3.10.2
It doesn't know about CMAKE_CXX_STANDARD=20,
let's add the correct flag manually
* gcc 8 on aarch64
utils/common/simd_arm.h:241:16: error: need ‘typename’ before ‘simd::TypeToVecWrapperType<T>::WrapperType’ because ‘simd::TypeToVecWrapperType<T>’ is a dependent scope
JobList low-level code relateod to primitive jobs now uses shared pointers instead of ByteStream refs talking to DEC
b/c same-node EM-PP communication now goes over a queue in DEC instead of a network hop.
PP now has a separate thread that processes the primitive job messages from that DEC queue.
operations + morsel size weight model to equally allocate CPU b/w parallel query morsels.
This patch delivers better parallel query timings distribution(timings graph resembles normal
distribution with a bigger left side thus more queries runs faster comparing with PrioThreadPool-based
single-node installation).
See changes in batchprimitiveprocessor-jl.h and comments in fair_threadpool.h for
important implementation details
PP now uses PriorityThreadPool that arbitrary picks another jobs pack
to run. This scheduling discipline tend to run portions of a single query
forcing other simultaneous queries to wait. In result parallel queries
timings variance is high. The FairThreadPool picks the job with the smallest
amount of work done so far(see the code for details)
respondWait could be set to false
while other threads were waiting. With respondWait false, okToRrespond
wouldn't ever get notify_one(). Get rid of respondWait and use
fProcessorPool->blockedThreadCount to determine if any threads may be
waiting.
* Fix clang warnings
* Remove vim tab guides
* initialize variables
* 'strncpy' output truncated before terminating nul copying as many bytes from a string as its length
* Fix ISO C++17 does not allow 'register' storage class specifier for outdated bison
* chars are unsigned on ARM, having if (ival < 0) always false
* chars are unsigned by default on ARM and comparison with -1 if always true
This patch changes the logic of the `receiveMultiPrimitiveMessages`
function in the following way:
1. We have only one aggregation thread which reads the data from Queue (which is populated
by messages from BPPs).
2. Processing of the received `bytestream vector` could be in parallel depends on the
type of `TupleBPS` operation (join, fe2, ...) and actual thread pool workload.
The motivation is to eliminate some amount of context switches.
Fixes:
* Irrelevant where conditions
* Irrelevant const
* A potential infinite loop in treenode
* Bad implicit case fallthroughs
* Explicit markings for required case fallthroughs
* Unused variables
* Unused function
Also disabled some warnings for now which we should fix later.
Rename packages to MariaDB-columnstore-engine, MariaDB-columnstore-libs
and MariaDB-columnstore-platform.
Also add the "columnstore-" prefix the the components so that MariaDB's
packaging system understands then and add a line to include them in
MariaDB's packaging.
In addition
* Fix S3 building for dist source build
* Fix Debian 10 dependency issue
* Fix git handling for dist builds
* Add support for MariaDB's RPM building
* Use MariaDB's PCRE and readline
* Removes a few dead files
* Fix Boost noncopyable includes
This branch enforces the use of generic MariaDB server paths for their
binaries and data rather than custom paths.
/usr/local/mariadb/columnstore is now only for columnstore with this
patch.
It should be noted that this removes the auto-mounting of external
MariaDB UM data storage for AWS.
This is also a fix for MCOL-3510 after buildbot changes are made.
Also... MCOL-3552 Use columnstore.cnf to load plugins
The ColumnStore plugins now load using a columnstore.cnf instead of a
SQL sequence to be more in-line with MariaDB's methods.
I reformulate the messages.
Changed version in preprocessor conditions to avoid compilation
warnings in Debian 9.
Disabled sign-compare check for generated files in DML/DDL.
