Found memory leaks were introduced by the commit
a896bebfa6
MDEV-18844 Implement EXCEPT ALL and INTERSECT ALL operations
and caused by using a statement arena instead a runtime arena for
allocation of objects having temporary life span by their nature.
Aforementioned memory leaks were produced by running queries
that typically use select with intersect, union or table values
constructors.
To fix these memory leaks use the runtime arena for allocation
of Item_field objects used by set operations.
Additionally, OOM handling added on allocation of aforementioned
Item_field objects.
This patch fixes the issue with passing the DEFAULT or IGNORE values to
positional parameters for some kind of SQL statements to be executed
as prepared statements.
The main idea of the patch is to associate an actual value being passed
by the USING clause with the positional parameter represented by
the Item_param class. Such association must be performed on execution of
UPDATE statement in PS/SP mode. Other corner cases that results in
server crash is on handling CREATE TABLE when positional parameter
placed after the DEFAULT clause or CALL statement and passing either
the value DEFAULT or IGNORE as an actual value for the positional parameter.
This case is fixed by checking whether an error is set in diagnostics
area at the function pack_vcols() on return from the function pack_expression()
MDEV-32441 SENT_ROWS shows random wrong values when stored function
is selected.
MDEV-32281 EXAMINED_ROWS is not populated in
information_schema.processlist upon SELECT.
Added ROWS_SENT to information_schema.processlist
This is to have the same information as Percona server (SENT_ROWS)
To ensure that information_schema.processlist has correct values for
sent_rows and examined_rows I introduced two new variables to hold the
total counts so far. This was needed as stored functions and stored
procedures will reset the normal counters to be able to count rows for
each statement individually for slow query log.
Other things:
- Selects with functions shows in processlist the total examined_rows
and sent_rows by the main statement and all functions.
- Stored procedures shows in processlist examined_rows and sent_rows
per stored procedure statement.
- Fixed some double accounting for sent_rows and examined_rows.
- HANDLER operations now also supports send_rows and examined_rows.
- Display sizes for MEMORY_USED, MAX_MEMORY_USED, EXAMINED_ROWS and
QUERY_ID in information_schema.processlist changed to 10 characters.
- EXAMINED_ROWS and SENT_ROWS changed to bigint.
- INSERT RETURNING and DELETE RETURNING now updates SENT_ROWS.
- As thd is always up to date with examined_rows, we do not need
to handle examined row counting for unions or filesort.
- I renamed SORT_INFO::examined_rows to m_examined_rows to ensure that
we don't get bugs in merges that tries to use examined_rows.
- Removed calls of type "thd->set_examined_row_count(0)" as they are
not needed anymore.
- Removed JOIN::join_examined_rows
- Removed not used functions:
THD::set_examined_row_count()
- Made inline some functions that where called for each row.
Fake_select_lex->join was prepared at the unit execution stage so
the validation of fake_select_lex before the unit pushdown
was incomplete. That caused pushing down of statements having
an incorrect ORDER BY clause.
This commit moves preparation of the fake_select_lex->join to the unit
prepare() method, before initializing of the pushdown handler,
so incorrect clauses error out before being pushed down
During st_select_lex_unit::prepare() the member select_unit*
st_select_lex_unit::union_result is being assigned to an instance
of one of the following classes:
- select_unit
- select_unit_ext
- select_unit_recursive
- select_union_direct
Select_union_direct used to pass the result of the query directly to
the receiving select_result without filling a temporary table. This class
wraps a select_result object and is currently used to process UNION ALL
queries. Other select_unit_* classes involve some additional result processing.
Pushed down units are processed on the engine side so the results must be
also passed directly to a select_result object. So in the case when
the unit pushdown is employed st_select_lex_unit::union_result must be
assigned to an instance of select_union_direct.
Allow queries of multiple SELECTs combined together with
UNIONs/EXCEPTs/INTERSECTs to be pushed down to foreign engines.
If the foreign engine provides an interface method "create_unit"
and the UNIT is a top-level unit of the SQL query then the server
tries to push the whole SELECT_LEX_UNIT down to the engine for execution.
The engine should perform necessary checks and if they succeed,
execute the query. If the engine is unable to execute the whole unit,
then another attempt is made to push down SELECTs composing the unit
separately using the "create_select" interface method. In this case
the results of separate SELECTs are combined at the server side
thus composing the final result
This error was discovered while working on
MDEV-30540 Wrong result with IN list length reaching
IN_PREDICATE_CONVERSION_THRESHOLD
If there is read error from handler::ha_rnd_next() during a recursive
query, st_select_lex_unit::exec_recursive() will crash as it will try to
get the error code from a structure that was deleted by the callee.
The code was using the construct:
sl->join->exec();
saved_error=sl->join->error;
This does not work as sl->join was freed by the exec() and sl->join would
be set to 0.
Fixed by having JOIN::exec() return the error code.
The included test case simulates the error in ha_rnd_next(), which causes
a crash without the patch.
scovered whle working on
MDEV-30540 Wrong result with IN list length reaching
IN_PREDICATE_CONVERSION_THRESHOLD
If there is read error from handler::ha_rnd_next() during a recursive
query, st_select_lex_unit::exec_recursive() will crash as it will try to
get the error code from a structure that was deleted by the callee.
The code was using the construct:
sl->join->exec();
saved_error=sl->join->error;
This does not work as sl->join was freed by the exec() and sl->join was
set to 0.
Fixed by having JOIN::exec() return the error code.
The included test case simulates the error in ha_rnd_next(), which causes
a crash without the patch.
This patch also fixes some bugs detected by valgrind after this
patch:
- Not enough copy_func elements was allocated by Create_tmp_table() which
causes an memory overwrite in Create_tmp_table::add_fields()
I added an ASSERT() to be able to detect this also without valgrind.
The bug was that TMP_TABLE_PARAM::copy_fields was not correctly set
when calling create_tmp_table().
- Aria::empty_bits is not allocated if there is no varchar/char/blob
fields in the table. Fixed code to take this into account.
This cannot cause any issues as this is just a memory access
into other Aria memory and the content of the memory would not be used.
- Aria::last_key_buff was not allocated big enough. This may have caused
issues with rtrees and ma_extra(HA_EXTRA_REMEMBER_POS) as they
would use the same memory area.
- Aria and MyISAM didn't take extended key parts into account, which
caused problems when copying rec_per_key from engine to sql level.
- Mark asan builds with 'asan' in version strihng to detect these in
not_valgrind_build.inc.
This is needed to not have main.sp-no-valgrind fail with asan.
The bug was some old code that, without any explanation, reset
PART_KEY_FLAG from fields in temporary tables. This caused
join_tab->key_dependent to not be updated properly, which caused
an assert.
The idea is that instead of marking all select_lex's with DISTINCT, we
only mark those that really need distinct result.
Benefits of this change:
- Temporary tables used with derived tables, UNION, IN are now smaller
as duplicates are removed already on the insert phase.
- The optimizer can now produce better plans with EQ_REF. This can be
seen from the tests where several queries does not anymore materialize
derived tables twice.
- Queries affected by 'in_predicate_conversion_threshold' where large IN
lists are converted to sub query produces better plans.
Other things:
- Removed on duplicate call to sel->init_select() in
LEX::add_primary_to_query_expression_body()
- I moved the testing of
tab->table->pos_in_table_list->is_materialized_derived()
in join_read_const_table() to the caller as it caused problems for
derived tables that could be proven to be const tables.
This also is likely to fix some bugs as if join_read_const_table()
was aborted, the table was left marked as JT_CONST, which cannot
be good. I added an ASSERT there for now that can be removed when
the code has been properly tested.
select_union_direct::send_data() only sends a record when
the LIMIT ... OFFSET clause of the individual select won't skip it.
Thus, select_union_direct::send_data() should not do any actions
related to a sending a record if the offset of a select isn't
reached yet
1. system_versioning_insert_history session variable allows
pseudocolumns ROW_START and ROW_END be specified in INSERT,
INSERT..SELECT and LOAD DATA.
2. Cleaned up select_insert::send_data() from setting vers_write as
this parameter is now set on TABLE initialization.
4. Replication of system_versioning_insert_history via option_bits in
OPTIONS_WRITTEN_TO_BIN_LOG.
Nowdays subquery in a UNION's ORDER BY placed correctly in fake select,
the only problem was incorrect Name_resolution_contect is fixed by this
patch in parsing, so we do not need scanning/reseting of ORDER BY of
a union.
When creating a recursive CTE, the column types are taken from the
non recursive part of the CTE (this is according to the SQL standard).
This patch adds code to abort the CTE if the calculated values in the
recursive part does not fit in the fields in the created temporary table.
The new code only affects recursive CTE, so it should not cause any notable
problems for old applications.
Other things:
- Fixed that we get correct row numbers for warnings generated with
WITH RECURSIVE
Reviewer: Alexander Barkov <bar@mariadb.com>
from view
A crash of the server happened when executing a stored procedure whose the
only query calculated window functions over a mergeable view specified
as a select from non-mergeable view. The crash could be reproduced if
the window specifications of the window functions were identical and both
contained PARTITION lists and ORDER BY lists. A crash also happened on
the second execution of the prepared statement created for such query.
If to use derived tables or CTE instead of views the problem still
manifests itself crashing the server.
When optimizing the window specifications of a window function the
server can substitute the partition lists and the order lists for
the corresponding lists from another window specification in the case
when the lists are identical. This substitution is not permanent and should
be rolled back before the second execution. It was not done and this
ultimately led to a crash when resolving the column names at the second
execution of SP/PS.
The bug affected execution of queries with With clauses containing so-called
hanging recursive CTEs in PREPARE mode. A CTE is hanging if it's not used
in the query. Preparation of a prepared statement from a query with a
hanging CTE caused a leak in the server and execution of this prepared
statement led to an assert failure of the server built in the debug mode.
This happened because the units specifying recursive CTEs erroneously were
not cleaned up if those CTEs were hanging.
The patch enforces cleanup of hanging recursive CTEs in the same way as
other hanging CTEs.
Approved by dmitry.shulga@mariadb.com
The ROWNUM() function is for SELECT mapped to JOIN->accepted_rows, which is
incremented for each accepted rows.
For Filesort, update, insert, delete and load data, we map ROWNUM() to
internal variables incremented when the table is changed.
The connection between the row counter and Item_func_rownum is done
in sql_select.cc::fix_items_after_optimize() and
sql_insert.cc::fix_rownum_pointers()
When ROWNUM() is used anywhere in query, the optimization to ignore ORDER
BY in sub queries are disabled. This was done to get the following common
Oracle query to work:
select * from (select * from t1 order by a desc) as t where rownum() <= 2;
MDEV-3926 "Wrong result with GROUP BY ... WITH ROLLUP" contains a discussion
about this topic.
LIMIT optimization is enabled when in a top level WHERE clause comparing
ROWNUM() with a numerical constant using any of the following expressions:
- ROWNUM() < #
- ROWNUM() <= #
- ROWNUM() = 1
ROWNUM() can be also be the right argument to the comparison function.
LIMIT optimization is done in two cases:
- For the current sub query when the ROWNUM comparison is done on the top
level:
SELECT * from t1 WHERE rownum() <= 2 AND t1.a > 0
- For an inner sub query, when the upper level has only a ROWNUM comparison
in the WHERE clause:
SELECT * from (select * from t1) as t WHERE rownum() <= 2
In Oracle mode, one can also use ROWNUM without parentheses.
Other things:
- Fixed bug where the optimizer tries to optimize away sub queries
with RAND_TABLE_BIT set (non-deterministic queries). Now these
sub queries will not be converted to joins. This bug fix was also
needed to get rownum() working inside subqueries.
- In remove_const() remove setting simple_order to FALSE if ROLLUP is
USED. This code was disable a long time ago because of wrong assignment
in the following code. Instead we set simple_order to false if
RAND_TABLE_BIT was used in the SELECT list. This ensures that
we don't delete ORDER BY if the result set is not deterministic, like
in 'SELECT RAND() AS 'r' FROM t1 ORDER BY r';
- Updated parameters for Sort_param::init_for_filesort() to be able
to provide filesort with information where the number of accepted
rows should be stored
- Reordered fields in class Filesort to optimize storage layout
- Added new error messsage to tell that a function can't be used in HAVING
- Added field 'with_rownum' to THD to mark that ROWNUM() is used in the
query.
Co-author: Oleksandr Byelkin <sanja@mariadb.com>
LIMIT optimization for sub query
One should instead use Item::fixed() and Item::with_subquery()
Removed Item::is_fixed() and has_subquery() and did the following replace:
replace is_fixed() fixed() -- *.*
replace 'has_subquery()' 'with_subquery()' -- *.*
The function was originally introduced by eb0804ef5e
MDEV-18553: MDEV-16327 pre-requisits part 1: isolation of LIMIT/OFFSET handling
set_unlimited had an overloaded notion of both clearing the offset value
and the limit value. The code is used for SQL_CALC_ROWS option to
disable the limit clause after the limit is reached, while at the same
time the calling code suppreses sending of rows.
Proposed solution:
Dedicated clear method for query initialization (to ensure no garbage
remains between executions).
Dedicated set_unlimited that only alters the limit value.
Replace
* select_lex::offset_limit
* select_lex::select_limit
* select_lex::explicit_limit
with select_lex::Lex_select_limit
The Lex_select_limit already existed with the same elements and was used in
by the yacc parser.
This commit is in preparation for FETCH FIRST implementation, as it
simplifies a lot of the code.
Additionally, the parser is simplified by making use of the stack to
return Lex_select_limit objects.
Cleanup of init_query() too. Removes explicit_limit= 0 as it's done a bit later
in init_select() with limit_params.empty()
