Instead of a seperate param for cache size, we decided
to allow an override of the NumBlocksPct param instead.
If NumBlocksPct ends in an 'm' or a 'g', it will be interpreted
as megabytes or gigabytes instead of a %age.
This adds options which are user enabled to debug the LRU cache inside
ColumnStore. Specifically cache flushing.
It adds the following:
* PrimProc flush information when SIGUSR2 mode is enabled
* cpimport dictionary flush information when -d2 is used
* WriteEngineServer DML flush information to STDERR
The FIFO buffer could get data in next() whilst a data swap is happening
due to a rare race condition. This patch adds mutexes around the parts
that could race.
The observed effect of this race was during a complex aggregate query
the results would occasionally be incorrect.
In addition this fixes a race condition in PrimProc's regex processor.
PriorityThreadPool didn't have very good error handling. If something
failed it would just ignore whatever was being processed. This could
lead to a query continuing without retreiving all of the required data.
This patch adds error handling, sending a message back to the client
and a log message. It also destroys and recreates the pool thread.
SQL-92 basically specifies for a NOPAD collation that only space should
be ignored for matches. Tabs and other whitespace are handled
differently. We don't fully support collations yet so we assume the
defaults.
For TEXT columns (and some other scenarios) we don't do a DSS step to
scan dictionaries and do it directly in the BPS step instead. This patch
applies the previous fix to this case too.
For equality string matches other engines ignore trailing whitespace
(this does not apply to LIKE matches). So we should do the same. This
patch trims whitespace for MIN/MAX extent elimination checks, fixed
width columns and dictionary columns during equality matches against
constants (SELECT * FROM t1 WHERE b = 'ABC').
With 1.1 we have removed libdrizzle and used MariaDB's client library
instead for both CrossEngine and QueryStats. Unfortunately MariaDB 10.2
has two client libraries which have different structs with the same
name. When QueryStats was running inside the ColumnStore plugin this
symbol conflict was causing a crash.
The server's built-in client API has several different and several
missing functions so some additions to sm.cpp were made to fill the
gaps.
This patch does the following:
* Make sure that libmariadb is only linked to executables, not the
ColumnStore Plugin (to avoid symbol conflicts). Note that all
executables that link to CrossEngine and/or QueryStats need to link to
libmariadb to avoid missing symbol issues.
* Use the server's built-in client API for QueryStats when run in the
plugin
* Replace missing server built-in client API calls in sm.cpp (this is
for QueryStats and CrossEngine to keep the dynamic linker happy)
* Fixes issue where using 'localhost' as the MariaDB Server hostname
would fail in QueryStats.
Whilst very rare we can hit a case where we attempt to unlock objLock
when it is already unlocked. With the Boost version in Ubuntu 16.04 this
triggers an abort() effectively crashing PrimProc.
In this patch we switch to a pthread mutex instead which does not have
this limitation. At a later date we can look into refactoring how BPP
and this mutex works.
Whilst very rare we can hit a case where we attempt to unlock objLock
when it is already unlocked. With the Boost version in Ubuntu 16.04 this
triggers an abort() effectively crashing PrimProc.
In this patch we switch to a pthread mutex instead which does not have
this limitation. At a later date we can look into refactoring how BPP
and this mutex works.
ByteStream::advance can throw an exception if there isn't enough data in
the buffer yet. PrimProc's BPP processor would not catch this causing a
thread to be leaked every time. This was happening on BPP destroy and
abort.
ByteStream::advance can throw an exception if there isn't enough data in
the buffer yet. PrimProc's BPP processor would not catch this causing a
thread to be leaked every time. This was happening on BPP destroy and
abort.
If ExeMgr disconnects (such as a crash) whilst queries are being
executed some BPP threads get orphaned. This patch tracks the BPP usage
for each threads and cleans up appropriately.
If ExeMgr disconnects (such as a crash) whilst queries are being
executed some BPP threads get orphaned. This patch tracks the BPP usage
for each threads and cleans up appropriately.
pDictionaryScan won't work for BLOB/TEXT since it requires searching the
data file and rebuilding the token from matches. The tokens can't be
rebuild correctly due the bits in the token used for block counts. This
patch forces the use of pDictionaryStep instead for WHERE conditions.
In addition this patch adds support for TEXT/BLOB in various parts of
the job step processing. This fixes things like error 202 during an
UPDATE with a join condition on TEXT/BLOB columns.
* TEXT and BLOB now have separate identifiers internally
* TEXT columns are identified as such in system catalog
* cpimport only requires hex input for BLOB, not TEXT
* DML writes for multi-block dictionary (blob) now works
* PrimProc fixed so that the first block in multi-block is read
correctly
* Performance optimisation (removed string copy into stack) for new
dictionary entries
