Part 1:
As part of MCOL-3776 to address synchronization issue while accessing
the fTimeZone member of the Func class, mutex locks were added to the
accessor and mutator methods. However, this slows down processing
of TIMESTAMP columns in PrimProc significantly as all threads across
all concurrently running queries would serialize on the mutex. This
is because PrimProc only has a single global object for the functor
class (class derived from Func in utils/funcexp/functor.h) for a given
function name. To fix this problem:
(1) We remove the fTimeZone as a member of the Func derived classes
(hence removing the mutexes) and instead use the fOperationType
member of the FunctionColumn class to propagate the timezone values
down to the individual functor processing functions such as
FunctionColumn::getStrVal(), FunctionColumn::getIntVal(), etc.
(2) To achieve (1), a timezone member is added to the
execplan::CalpontSystemCatalog::ColType class.
Part 2:
Several functors in the Funcexp code call dataconvert::gmtSecToMySQLTime()
and dataconvert::mySQLTimeToGmtSec() functions for conversion between seconds
since unix epoch and broken-down representation. These functions in turn call
the C library function localtime_r() which currently has a known bug of holding
a global lock via a call to __tz_convert. This significantly reduces performance
in multi-threaded applications where multiple threads concurrently call
localtime_r(). More details on the bug:
https://sourceware.org/bugzilla/show_bug.cgi?id=16145
This bug in localtime_r() caused processing of the Functors in PrimProc to
slowdown significantly since a query execution causes Functors code to be
processed in a multi-threaded manner.
As a fix, we remove the calls to localtime_r() from gmtSecToMySQLTime()
and mySQLTimeToGmtSec() by performing the timezone-to-offset conversion
(done in dataconvert::timeZoneToOffset()) during the execution plan
creation in the plugin. Note that localtime_r() is only called when the
time_zone system variable is set to "SYSTEM".
This fix also required changing the timezone type from a std::string to
a long across the system.
on a non-ColumnStore table does not work.
As part of MCOL-4617, we moved the in-to-exists predicate creation
and injection from the server into the engine. However, when query
with an IN Subquery contains a non-ColumnStore table, the server
still performs the in-to-exists predicate transformation for the
foreign engine table. This caused ColumnStore's execution plan to
contain incorrect WHERE predicates. As a fix, we call
mutate_optimizer_flags() for the WRITE lock, in addition to the READ
table lock. And in mutate_optimizer_flags(), we change the optimizer
flag from OPTIMIZER_SWITCH_IN_TO_EXISTS to OPTIMIZER_SWITCH_MATERIALIZATION.
cross-engine join with a ColumnStore table errors out.
ColumnStore cannot directly update a foreign table. We detect whether
a multi-table UPDATE operation is performed on a foreign table, if so,
do not create the select_handler and let the server execute the UPDATE
operation instead.
This feature allows a query execution to fallback to the server,
in case query execution using the select_handler (SH) fails. In case
of fallback, a warning message containing the original reason for
query failure using SH is generated.
To accomplish this task, SH execution is moved to an earlier step when
we create the SH in create_columnstore_select_handler(), instead of the
previous call to SH execution in ha_columnstore_select_handler::init_scan().
This requires some pre-requisite steps that occur in the server in
JOIN::optimize() and JOIN::exec() to be performed before starting SH execution.
In addition, missing test cases from MCOL-424 are also added to the MTR suite,
and the corresponding fix using disable_indices_for_CEJ() is reverted back
since the original fix now appears to be redundant.
This is a subtask of MCOL-4525 Implement select_handler=AUTO.
Server performs outer join to inner join conversion using simplify_joins()
in sql/sql_select.cc, by updating the TABLE_LIST::outer_join variable.
In order to perform this conversion, permanent changes are made in some
cases to the SELECT_LEX::JOIN::conds and/or TABLE_LIST::on_expr.
This is undesirable for MCOL-4525 which will attemp to fallback and execute
the query inside the server, in case the query execution fails in ColumnStore
using the select_handler.
For a query such as:
SELECT * FROM t1 LEFT JOIN t2 ON expr1 LEFT JOIN t3 ON expr2
In some cases, server can update the original SELECT_LEX::JOIN::conds
and/or TABLE_LIST::on_expr and create new Item_cond_and objects
(e.g. with 2 Item's expr1 and expr2 in Item_cond_and::list).
Instead of making changes to the original query structs, we use
gp_walk_info::tableOnExprList and gp_walk_info::condList. 2 Item's,
expr1 and expr2, in the condList, mean Item_cond_and(expr1, expr2), and
hence avoid permanent transformations to the SELECT_LEX.
We also define a new member variable
ha_columnstore_select_handler::tableOuterJoinMap
which saves the original TABLE_LIST::outer_join values before they are
updated. This member variable will be used later on to restore to the original
state of TABLE_LIST::outer_join in case of a query fallback to server execution.
The original simplify_joins() implementation in the server also performs a
flattening of the JOIN nest, however we don't perform this operation in
convertOuterJoinToInnerJoin() since it is not required for ColumnStore.
in the query projection list.
With this patch, we enable SELECT/DERIVED handler execution for
queries involving 'get_user_var' function. Whereas for 'set_user_var'
function, the handlers are still disabled and the query execution
fallsback to the server, except for some edges case described in
the test file mcol-4410.test, in which case, an appropriate error
message is returned to the client.
We earlier leveraged the server functionality provided by
Item_in_subselect::create_in_to_exists_cond and
Item_in_subselect::inject_in_to_exists_cond
to create and inject the in-to-exists predicate into an IN
subquery's JOIN struct. With this patch, we leave the IN subquery's
JOIN unaltered and instead directly perform this predicate creation
and injection into ColumnStore's select execution plan.
Mutate the optimizer flags for prepared statements in:
1. ha_mcs::open
2. ha_mcs::discover_check_version
This is done to ensure the optimizer flags are disabled before
JOIN::prepare() is called during "PREPARE stmt FROM ...".
This patch enables select handler for executing prepared
statements. Most importantly, we are now activating a
persistent arena which will allocate any new items in a
permanent MEMROOT for prepared statements and stored procedures.
Refer to JOIN::optimize_inner() for details.
In processWhere(), we now use SELECT_LEX::prep_where in case
we are executing a prepared statement, as this is where the saved
WHERE clause is stored for prepared statement processing.
In addition, we also disable derived handler for prepared
statements.
Original SH implementation sends the result set back to the client
thus it can't be used in INSERT..SELECT, SELECT INTO OUTFILE,CREATE
TABLE AS SELECT etc.
CLX-77 feature has been backported into MDB to enable SH to run
query part of the mentioned queries.
internally(thus can remove equi-JOIN predicates) with a recursive f()
to inject equi-JOIN conditions for IN + correlated subquery Item.
This change both solves MCOL-3747 issue and remove issues I first
faced in union.sql test.
This commit also fixes a clean up for conditions pushed for table API
execution path and replaces NULL with nullptr in ha_mcs_impl.cpp.
Disabled 4th if block in buildOuterJoin to handle non-optimized MDB query
structures.
Broke getSelectPlan into pieces: processFrom, processWhere.
MCOL-3593 UNION processing depends on two flags isUnion that comes as
arg of getSelectPlan and unionSel that is a local variable in
getSelectPlan. Modularization of getSelectPlan broke the mechanizm.
This patch is supposed to partially fix it.
MCOL-3593 Removed unused if condition from buildOuterJoin that allows
unsupported construct subquery in ON expression.
Fixed an improper if condition that ignors tableMap entries w/o condition
in external_lock thus external_lock doesn't clean up when the query
finishes.
Fixed wrong logging for queries processed in tableMode. Now rnd_init
properly sends queryText down to ExeMgr to be logged.
MCOL-3593 Unused attribute FromSubQuery::fFromSub was removed.
getSelectPlan has been modularized into: setExecutionParams,
processFrom, processWhere. SELECT, HAVING, GROUP BY, ORDER BY
still lives in getSelectPlan.
Copied optimization function simplify_joins_ into our pushdown
code to provide the plugin code with some rewrites from MDB it
expects.
The columnstore_processing_handlers_fallback session variable
has been removed thus CS can't fallback from SH to partial
execution paths, e.g. DH, GBH or plugin API.
MCOL-3602 Moved MDB optimizer rewrites into a separate file.
Add SELECT_LEX::optimize_unflattened_subqueries() call to fix IN
into EXISTS rewrite for semi-JOINs with subqueries.
disable_indices_for_CEJ() add index related hints to disable
index access methods in Cross Engine Joins.
create_SH() now flattens JOIN that has both physical tables and
views. This fixes most of views related tests in the regression.
Set a proper type for string literals on ConstantColumn ctor
to fix the regression produced by MCOL-174.
Removed OPTIMIZER_SWITCH_EXISTS_TO_IN b/c MDB produces
unsupported optimization with it and CS couldn't create
ExistsFilter.
any function in projection looking for either aggregation function or
GROUP BY key column.
MDB doesn't convert Item_equal into corresponding Item_func_eq w/o
OPTIMIZER_SWITCH_COND_PUSHDOWN_FROM_HAVING optimizer flag.
Activation of the flag fixes queries from having test scenario w
equal operator in HAVING, e.g. HAVING c1 = 50.
This happens in external_lock() whilst locking the table.
Fixes LIMIT=1 optimization for EXISTS_SUBS subqueries.
external_lock() contains if condition that gives false positive
for SH + pushed conditions.
external_lock() now resets in_subquery_conversion_threshold
variable that governs IN_INTO_SUBQUERY optimization for
queries run in table mode.
external_lock() now purges dynamicall allocated condInfo for
SH and DH execution path.
Commented out UNION check b/c if condition gives false positives
and silently enables table mode execution for queries w/o
UNION.
call.
CS doesn't use SH for SELECT..INTO OUTFILE queries.
Clean up gwi::physTableList when processing Storage API request.
SH now explicitly set an execution error in THD::stmt_da.
SH now set queryState in select_next() to mark a begging of the execution.
SH query execution migrated from SH::init() into create_SH().
There is a session variable columnstore_processing_handlers_fallback
that allows to fallback to DH, GBH if SH fails. DH now uses semantic
tree check for unsupported features to allow to fallback to GBH or
storage API.
Fixes GBH related bug when create_GBH() returns a handler for
queries with impossible WHERE/HAVING.
Fixed bug in FromSubquery::transform() where isUnion is set to true.
Enabled RTTI b/c server team enabled it for MDB.
Removed unused code supposed to be used with vtable.
We preserve union related condition blocks until UNION-capable smart handler.
Removed a number of commented code blocks.
Add projection list REAL and TIME constants, e.g
SELECT TIME'10:20:30', TIMESTAMP'2001-01-01 10:20:30', DATE'2001-01-01' FROM cs1;
Marked potentially useless block in ORDER BY processing for future removal.
Removed string variables used for reconstructed query in the pre-1.4 fork.
Removed cp_get_plan().
GBH doesn't step in if SH processes the query.
All smart handlers now correctly processes impossible WHERE and HAVING.
to avoid accedental crashes.
Add check for Conversion of Big IN Predicates Into Subqueries optimization
conditions.
Enabled derivedTableOptimization() for group by and derived handlers.
Disabled Conversion of Big IN Predicates Into Subqueries optimization.
Disabled most of optimizer_flags for now.
RowGroup + operator now correctly sets useStringTable flag that
instructs code to check StringStore instead of plain data buffer.
Fix crash in ha_calpont_impl_close_connection()
Fix leak in ci.tableMap.
Removed extra returns in pushdown_init to avoid crashes.
create_select_handler now detects INSERT..SELECT.
buildConstColFromFilter now uses any kind of filter to supply
relevant columns.
Remove strings used by vtable redo phase.
Make FromSubQuery aware of Pushdown handlers.
Changed debug_walk to work around changed Item framework.
Temporary disabled derived handler and unsupported features checks.
Used Item attribute getters introduced by 10.4
Make changes to support Item::CONST_ITEM introduced by 10.4
as a replacement for INT_,REAL_,STRING_ ITEM.
Replaced QT_INFINIDB_DERIVED and similar flags
with correponded flags for Item->print().
Replaced or commented out infinidb_ variable names with columnstore_
where applicable.
SELECT_LEX had been moved in THD so changed all references.
Avoid writing CS decimal scales into MDB decimal fields
d-only dec attribute. WIP
Replaced infinidb_vtable with a singleton MIGR.
Merged with MCOL-2121.
Added new wsrep include paths needed by UDaF code.
Removed .vcxproj from Connector code.
Renamed isInfiniDB() into isMCSTable
Changed getSelectPlan() to reuse it with derived and
other handler types.
Separate pushdown handlers methods and functions.
Removed vcxproj files from the source.
Added fix for MCOL-2166.
Merged with MCOL-2121
