PP now uses PriorityThreadPool that arbitrary picks another jobs pack
to run. This scheduling discipline tend to run portions of a single query
forcing other simultaneous queries to wait. In result parallel queries
timings variance is high. The FairThreadPool picks the job with the smallest
amount of work done so far(see the code for details)
respondWait could be set to false while other threads were waiting. With respondWait false, okToRrespond wouldn't ever get notify_one().
Get rid of respondWait and use fProcessorPool->blockedThreadCount to determine if any threads may be waiting.
* MCOL-4846 dev-6 Handle large join results
Use a loop to shrink the number of results reported per message to something manageable.
* MCOL-4841 small changes requested by review
* Add EXTRA threads to prioritythreadpool
prioritythreadpool is configured at startup with a fixed number of threads available. This is to prevent thread thrashing. Since most of the time, BPP job steps are short lived, and a rescheduling mechanism exist if no threads are available, this works to keep cpu wastage to a minimum.
However, if a query or queries consume all the threads in prioritythreadpool and then block (due to the consumer not consuming fast enough) we can run out of threads and no work will be done until some threads unblock. A new mechanism allows for EXTRA threads to be generated for the duration of the blocking action. These threads can act on new queries. When all blocking is completed, these threads will be released when idle.
* MCOL-4841 dev6 Reconcile with changes in develop-6
* MCOL-4841 Some format corrections
* MCOL-4841 dev clean up some things based on review
* MCOL-4841 dev 6 ExeMgr Crashes after large join
This commit fixes up memory accounting issues in ExeMgr
* MCOL-4841 remove LDI change
Opened MCOL-4968 to address the issue
* MCOL-4841 Add fMaxBPPSendQueue to ResourceManager
This causes the setting to be loaded at run time (requires restart to accept a change) BPPSendthread gets this in it's ctor
Also rolled back changes to TupleHashJoinStep::smallRunnerFcn() that used a local variable to count locally allocated memory, then added it into the global counter at function's end. Not counting the memory globally caused conversion to UM only join way later than it should. This resulted in MCOL-4971.
* MCOL-4841 make blockedThreads and extraThreads atomic
Also restore previous scope of locks in bppsendthread. There is some small chance the new scope could be incorrect, and the performance boost is negligible. Better safe than sorry.
This patch changes the logic of the `receiveMultiPrimitiveMessages`
function in the following way:
1. We have only one aggregation thread which reads the data from Queue (which is populated
by messages from BPPs).
2. Processing of the received `bytestream vector` could be in parallel depends on the
type of `TupleBPS` operation (join, fe2, ...) and actual thread pool workload.
The motivation is to eliminate some amount of context switches.
Fixes:
* Irrelevant where conditions
* Irrelevant const
* A potential infinite loop in treenode
* Bad implicit case fallthroughs
* Explicit markings for required case fallthroughs
* Unused variables
* Unused function
Also disabled some warnings for now which we should fix later.
Rename packages to MariaDB-columnstore-engine, MariaDB-columnstore-libs
and MariaDB-columnstore-platform.
Also add the "columnstore-" prefix the the components so that MariaDB's
packaging system understands then and add a line to include them in
MariaDB's packaging.
In addition
* Fix S3 building for dist source build
* Fix Debian 10 dependency issue
* Fix git handling for dist builds
* Add support for MariaDB's RPM building
* Use MariaDB's PCRE and readline
* Removes a few dead files
* Fix Boost noncopyable includes
This branch enforces the use of generic MariaDB server paths for their
binaries and data rather than custom paths.
/usr/local/mariadb/columnstore is now only for columnstore with this
patch.
It should be noted that this removes the auto-mounting of external
MariaDB UM data storage for AWS.
This is also a fix for MCOL-3510 after buildbot changes are made.
Also... MCOL-3552 Use columnstore.cnf to load plugins
The ColumnStore plugins now load using a columnstore.cnf instead of a
SQL sequence to be more in-line with MariaDB's methods.
I reformulate the messages.
Changed version in preprocessor conditions to avoid compilation
warnings in Debian 9.
Disabled sign-compare check for generated files in DML/DDL.
Fixed pragmas that disables compilation checks.
DDLProc now returns an error if it couldn't cwd.
Use either auto_ptr or unique_ptr depending on GCC version.
Anything that links against joblist will spin up a threadpool upon
startup. This includes the tools setConfig/getConfig. It is possible on
a low core count machine or low CPU speed that the signal to the prune
thread to shutdown is sent before the thread has completed startup when
these quick-running tools are used.
This fix adds a mutex so that spin up and shutdown can't happen at the
same time as well as a stop watch in case we are shutting down when
either the thread is running or we haven't fully started.
Anything that links against joblist will spin up a threadpool upon
startup. This includes the tools setConfig/getConfig. It is possible on
a low core count machine or low CPU speed that the signal to the prune
thread to shutdown is sent before the thread has completed startup when
these quick-running tools are used.
This fix adds a mutex so that spin up and shutdown can't happen at the
same time as well as a stop watch in case we are shutting down when
either the thread is running or we haven't fully started.
When a thread has been idle for 10 minutes and we have too many threads
in the threadpool the thread will be pruned. This is done by the
thread's main function just returning. Unfortunately this does not free
up the memory, the thread either needs to be joined or detatched.
We cannot use detached threads since there are mutexes and conditional
variables between the main thread and the threadpool threads. If the
main thread finishes before the threadpool threads (as would happen in
cpimport) then crashes occur. The parent needs to wait on the child
threads which is the whole point in joining.
So this fix spawns a new thread which every minute will check the list
of threads to be joined due to timeout and join them.
We have had to use an adapted version of boost::thread_group so that we
can join a single thread based off its thread ID.
In addition with have modified PriorityThreadPool to use detached
threads since this does not need to signal the child threads at the end.
PriorityThreadPool didn't have very good error handling. If something
failed it would just ignore whatever was being processed. This could
lead to a query continuing without retreiving all of the required data.
This patch adds error handling, sending a message back to the client
and a log message. It also destroys and recreates the pool thread.
Ubuntu 18.04 uses GCC 7.3 which is a little stricter than before.
Fixes a few errors due to implicit includes that are no longer implicit
and a ton of warnings about the implied alignment of code in
utils/common/any.hpp
