There were numerous memory leaks in plugin's code and associated code.
During typical run of MTR tests it leaked around 65 megabytes of
objects. As a result they may severely affect long-lived connections.
This patch fixes (almost) all leaks found in the plugin. The exceptions
are two leaks associated with SHOW CREATE TABLE columnstore_table and
getting information of columns of columnstore-handled table. These
should be fixed on the server side and work is on the way.
- occured -> occurred
- reponse -> response
- seperated -> separated
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.
Part 1:
As part of MCOL-3776 to address synchronization issue while accessing
the fTimeZone member of the Func class, mutex locks were added to the
accessor and mutator methods. However, this slows down processing
of TIMESTAMP columns in PrimProc significantly as all threads across
all concurrently running queries would serialize on the mutex. This
is because PrimProc only has a single global object for the functor
class (class derived from Func in utils/funcexp/functor.h) for a given
function name. To fix this problem:
(1) We remove the fTimeZone as a member of the Func derived classes
(hence removing the mutexes) and instead use the fOperationType
member of the FunctionColumn class to propagate the timezone values
down to the individual functor processing functions such as
FunctionColumn::getStrVal(), FunctionColumn::getIntVal(), etc.
(2) To achieve (1), a timezone member is added to the
execplan::CalpontSystemCatalog::ColType class.
Part 2:
Several functors in the Funcexp code call dataconvert::gmtSecToMySQLTime()
and dataconvert::mySQLTimeToGmtSec() functions for conversion between seconds
since unix epoch and broken-down representation. These functions in turn call
the C library function localtime_r() which currently has a known bug of holding
a global lock via a call to __tz_convert. This significantly reduces performance
in multi-threaded applications where multiple threads concurrently call
localtime_r(). More details on the bug:
https://sourceware.org/bugzilla/show_bug.cgi?id=16145
This bug in localtime_r() caused processing of the Functors in PrimProc to
slowdown significantly since a query execution causes Functors code to be
processed in a multi-threaded manner.
As a fix, we remove the calls to localtime_r() from gmtSecToMySQLTime()
and mySQLTimeToGmtSec() by performing the timezone-to-offset conversion
(done in dataconvert::timeZoneToOffset()) during the execution plan
creation in the plugin. Note that localtime_r() is only called when the
time_zone system variable is set to "SYSTEM".
This fix also required changing the timezone type from a std::string to
a long across the system.
mcsconfig.h and my_config.h have the following
pre-processor definitions:
1. Conflicting definitions coming from the standard cmake definitions:
- PACKAGE
- PACKAGE_BUGREPORT
- PACKAGE_NAME
- PACKAGE_STRING
- PACKAGE_TARNAME
- PACKAGE_VERSION
- VERSION
2. Conflicting definitions of other kinds:
- HAVE_STRTOLL - this is a dirt in MariaDB headers.
Should be fixed in the server code. my_config.h erroneously
performs "#define HAVE_STRTOLL" instead of "#define HAVE_STRTOLL 1".
in some cases. The former is not CMake compatible style. The latter is.
3. Non-conflicting definitions:
Otherwise, mcsconfig.h and my_config.h should be mutually compatible,
because both are generated by cmake on the same host machine. So
they should have exactly equal definitions like "HAVE_XXX", "SIZEOF_XXX", etc.
Observations:
- It's OK to include both mcsconfig.h and my_config.h providing that we
suppress duplicate definition of the above conflicting types #1 and #2.
- There is no a need to suppress duplicate definitions mentioned in #3,
as they are compatible!
- my_sys.h and m_ctype.h must always follow a CMake configuation header,
either my_config.h or mcsconfig.h (or both).
They must never be included without any preceeding configuration header.
This change make sure that we resolve conflicts by:
- either disallowing inclusion of mcsconfig.h and my_config.h
at the same time
- or by hiding conflicting definitions #1 and #2
(with their later restoring).
- also, by making sure that my_sys.h and m_ctype.h always follow
a CMake configuration file.
Details:
- idb_mysql.h can now only be included only after my_config.h
An attempt to use idb_mysql.h with mcsconfig.h instead of
my_config.h is caught by the "#error" preprocessor directive.
- mariadb_my_sys.h can now be only included after mcsconfig.h.
An attempt to use mariadb_my_sys.h without mcscofig.h
(e.g. with my_config.h) is also caught by "#error".
- collation.h now can now be included in two ways.
It now has the following effective structure:
#if defined(PREFER_MY_CONFIG_H) && defined(MY_CONFIG_H)
// Remember current conflicting definitions on the preprocessor stack
// Undefine current conflicting definitions
#endif
#include "mcsconfig.h"
#include "m_ctype.h"
#if defined(PREFER_MY_CONFIG_H) && defined(MY_CONFIG_H)
# Restore conflicting definitions from the preprocessor stack
#endif
and can be included as follows:
a. using only mcsconfig.h as a configuration header:
// my_config.h must not be included so far
#include "collation.h"
b. using my_config.h as the first included configuration file:
#define PREFER_MY_CONFIG_H // Force conflict resolution
#include "my_config.h" // can be included directly or indirectly
...
#include "collation.h"
Other changes:
- Adding helper header files
utils/common/mcsconfig_conflicting_defs_remember.h
utils/common/mcsconfig_conflicting_defs_restore.h
utils/common/mcsconfig_conflicting_defs_undef.h
to perform conflict resolution easier.
- Removing `#include "collation.h"` from a number of files,
as it's automatically included from rowgroup.h.
- Removing redundant `#include "utils_utf8.h"`.
This change is not directly related to the problem being fixed,
but it's nice to remove redundant directives for both collation.h
and utils_utf8.h from all the files that do not really need them.
(this change could probably have gone as a separate commit)
- Changing my_init() to MY_INIT(argv[0]) in the MCS services sources.
After the fix of the complitation failure it appeared that ColumnStore
services compiled with the debug build crash due to recent changes in
safemalloc. The crash happened in strcmp() with `my_progname` as an argument
(where my_progname is a mysys global variable). This problem should
probably be fixed on the server side as well to avoid passing NULL.
But, the majority of MariaDB executable programs also use MY_INIT(argv[0])
rather than my_init(). So let's make MCS do like the other programs do.
For a query of the form:
SELECT COUNT(c2) FROM (SELECT * FROM t1) q;
where t1 contains 10 columns c1, c2, ... , c10.
We currently create an intermediate RowGroup in ExeMgr with
a row of the form (1, c2_value1, 1, 1, 1, 1, 1, 1, 1, 1), i.e.
for all the columns of the subquery which are not referenced in
the outer query, we substitute a constant value, which is wasteful.
With this optimization, we are trimming the RowGroup to a row
of the form (1, c2_value1). This can have non-trivial query
execution time improvements if the subquery contains large number
of columns (such as a "select *" on a very wide table) and the outer
query is only referencing a subset of these columns with lower
index values from the subquery (as an example, c1 or c2 above).
That is, the current limitation of this optimization is we are not
removing those non-referenced subquery columns (c1 in the query above)
which are to the left of a referenced column.
Create tables and schemas with lower case name only if the flag is set.
During operations, convert to lowercase in plugin. Byt the time a query gets to ExeMgr, DDLProc etc., everything must be lower case if the flag is set, and undisturbed if not.
Original SH implementation sends the result set back to the client
thus it can't be used in INSERT..SELECT, SELECT INTO OUTFILE,CREATE
TABLE AS SELECT etc.
CLX-77 feature has been backported into MDB to enable SH to run
query part of the mentioned queries.
Disabled 4th if block in buildOuterJoin to handle non-optimized MDB query
structures.
Broke getSelectPlan into pieces: processFrom, processWhere.
MCOL-3593 UNION processing depends on two flags isUnion that comes as
arg of getSelectPlan and unionSel that is a local variable in
getSelectPlan. Modularization of getSelectPlan broke the mechanizm.
This patch is supposed to partially fix it.
MCOL-3593 Removed unused if condition from buildOuterJoin that allows
unsupported construct subquery in ON expression.
Fixed an improper if condition that ignors tableMap entries w/o condition
in external_lock thus external_lock doesn't clean up when the query
finishes.
Fixed wrong logging for queries processed in tableMode. Now rnd_init
properly sends queryText down to ExeMgr to be logged.
MCOL-3593 Unused attribute FromSubQuery::fFromSub was removed.
getSelectPlan has been modularized into: setExecutionParams,
processFrom, processWhere. SELECT, HAVING, GROUP BY, ORDER BY
still lives in getSelectPlan.
Copied optimization function simplify_joins_ into our pushdown
code to provide the plugin code with some rewrites from MDB it
expects.
The columnstore_processing_handlers_fallback session variable
has been removed thus CS can't fallback from SH to partial
execution paths, e.g. DH, GBH or plugin API.
MCOL-3602 Moved MDB optimizer rewrites into a separate file.
Add SELECT_LEX::optimize_unflattened_subqueries() call to fix IN
into EXISTS rewrite for semi-JOINs with subqueries.
disable_indices_for_CEJ() add index related hints to disable
index access methods in Cross Engine Joins.
create_SH() now flattens JOIN that has both physical tables and
views. This fixes most of views related tests in the regression.
SH query execution migrated from SH::init() into create_SH().
There is a session variable columnstore_processing_handlers_fallback
that allows to fallback to DH, GBH if SH fails. DH now uses semantic
tree check for unsupported features to allow to fallback to GBH or
storage API.
Fixes GBH related bug when create_GBH() returns a handler for
queries with impossible WHERE/HAVING.
Fixed bug in FromSubquery::transform() where isUnion is set to true.
Enabled RTTI b/c server team enabled it for MDB.
Removed unused code supposed to be used with vtable.
Fix crash in ha_calpont_impl_close_connection()
Fix leak in ci.tableMap.
Removed extra returns in pushdown_init to avoid crashes.
create_select_handler now detects INSERT..SELECT.
buildConstColFromFilter now uses any kind of filter to supply
relevant columns.
Remove strings used by vtable redo phase.
Make FromSubQuery aware of Pushdown handlers.
Changed debug_walk to work around changed Item framework.
Temporary disabled derived handler and unsupported features checks.
A view calling a view as part of a subquery outer join was not getting
the view name for the derived table columns. Which caused ColumnStore to
think it was joining outside of the view and triggered a missing column
error.
This fix adds the view name from the subquery if one cannot be obtained
from the field object.
