JOIN::cur_dups_producing_tables was not maintained correctly in
the cases of greedy optimization (search_depth < n_tables).
Moved it to POSITION structure where it will be maintained automatically.
Removed POSITION::prefix_dups_producing_tables since its value can now
be calculated.
This bug manifests due to wrong computation and evaluation of
keyinfo->key_length. The issues were:
* Using table->file->max_key_length() as an absolute value that must not be
reached for a key, while it represents the maximum number of bytes
possible for a table key.
* Incorrectly computing the keyinfo->key_length size during
KEY_PART_INFO creation. The metadata information regarding the key
such the field length (for strings) was added twice.
Problem:
find_order_by_list does not update the address of order_item
correctly after resolving.
Solution:
Change the ref_by address for a order_by field if its
SUM_FUNC_ITEM to the address of the field present in
all_fields.
Problem:
While getting the temp table field for a REF_ITEM
make_sortorder is using the real_item. As a result
server fails later with an assert.
Solution:
Do not use real_item to get the temp table field.
Instead use the REF_ITEM itself as temp table fields
are created for REF_ITEM not the real_item.
generate_derived_keys_for_table() did not work correctly in the case where
- it had a potential index on derived table
- however, TABLE::check_tmp_key() would disallow creation of this index
after looking at its future key parts (because of the key parts exceeding
max. index length)
- the code would leave a KEYUSE structure that refers to a non-existant index.
Depending on further optimizer calculations, this could cause a crash.
Analysis:
--------
Certain queries using intrinsic temporary tables may fail due to
name clashes in the file name for the temporary table when the
'temp-pool' enabled.
'temp-pool' tries to reduce the number of different filenames used for
temp tables by allocating them from small pool in order to avoid
problems in the Linux kernel by using a three part filename:
<tmp_file_prefix>_<pid>_<temp_pool_slot_num>.
The bit corresponding to the temp_pool_slot_num is set in the bit
map maintained for the temp-pool when it used for the file name.
It is cleared after the temp table is deleted for re-use.
The 'create_tmp_table()' function call under error condition
tries to clear the same bit twice by calling 'free_tmp_table()'
and 'bitmap_lock_clear_bit()'. 'free_tmp_table()' does a delete
of the table/file and clears the bit by calling the same function
'bitmap_lock_clear_bit()'.
The issue reported can be triggered under the timing window mentioned
below for an error condition while creating the temp table:
a) THD1: Due to an error clears the temp pool slot number used by it
by calling 'free_tmp_table'.
b) THD2: In the process of creating the temp table by using an unused
slot number in the bit map.
c) THD1: Clears the slot number used THD2 by calling
'bitmap_lock_clear_bit()' after completing the call 'free_tmp_table'.
d) THD3: Uses the slot number used the THD2 since it is freed by THD1.
When it tries to create the temp file using that slot number,
an error is reported since it is currently in use by THD2.
[The error: Error 'Can't create/write to file
'/tmp/#sql_277e_0.MYD' (Errcode: 17)']
Another issue which may occur in 5.6 and trunk is that:
When the open temporary table fails after its creation(due to ulimit
or OOM error), the file is not deleted. Thus further attempts to use
the same slot number in the 'temp-pool' results in failure.
Fix:
---
a) Under the error condition calling the 'bitmap_lock_clear_bit()'
function to clear the bit is unnecessary since 'free_tmp_table()'
deletes the table/file and clears the bit. Hence removed the
redundant call 'bitmap_lock_clear_bit()' in 'create_tmp_table()'
This prevents the timing window under which the issue reported
can be seen.
b) If open of the temporary table fails, then the file is deleted
thus allowing the temp-pool slot number to be utilized for the
subsequent temporary table creation.
c) Also if the attempt to create temp table fails since it already
exists, the temp-pool slot for it is marked as used, to avoid
the problem from re-appearing.
(Backport to 5.3)
(Attempt #2)
- Don't attempt to use BKA for materialized derived tables. The
table is neither filled nor fully opened yet, so attempt to
call handler->multi_range_read_info() causes crash.
- TABLE::create_key_part_by_field() should not set PART_KEY_FLAG in field->flags
= The reason is that it is used by hash join code which calls it to create a hash
table lookup structure. It doesn't create a real index.
= Another caller of the function is TABLE::add_tmp_key(). Made it to set the flag itself.
- The differences in join_cache.result could also be observed before this patch: one
could put "FLUSH TABLES" before the queries and get exactly the same difference.
- test_if_skip_sort_order()/create_ref_for_key() may change table
access from EQ_REF(index1) to REF(index2).
- Doing so doesn't make much sense from optimization POV, but since
they are doing it, they should update tab->read_record.unlock_row
accordingly.
Don't double-check privileges for a column in the GROUP BY that refers to
the same column in SELECT clause. Privileges were already checked for SELECT clause.
- When traversing JOIN_TABs with first_linear_tab/next_linear_tab(), don't forget
to enter the semi-join nest when it is the first table in the join order.
Failure to do so could cause e.g. I_S tables not to be filled.
- When the optimizer chose LooseScan, make_join_readinfo() should
use the index that was chosen for LooseScan, and should not try
to find a better (shortest) index.
It is triple bug with one test suite:
1. Incorrect outer table detection
2. Incorrect leaf table processing for multi-update (should be full like for usual updates and inserts)
3. ON condition fix_fields() fould be called for all tables of the query.
WHERE ONE OF SELECT* IS DISTINCT FAILS.
ISSUE:
------
There are 2 issues related to explain union.
1. If we have subquery with union of selects. And, one of
the select need temp table to materialize its results
then it will replace its query structure with a simple
select from temporary table. Trying to display new
internal temporary table scan resulted in crash. But to
display the query plan, we should save the original
query structure.
2. Multiple execution of prepared explain statement which
have union of subqueries resulted in crash. If we have
constant subqueries, fake select used in union operation
will be evaluated once before using it for explain.
During first execution we have set fake select options to
SELECT_DESCRIBE, but did not reset after the explain.
Hence during next execution of prepared statement during
first time evaluation of fake select we had our select
options as SELECT_DESCRIBE this resulted in improperly
initialized data structures and crash.
SOLUTION:
---------
1. If called by explain now we save the original query
structure. And this will be used for displaying.
2. Reset the fake select options after it is called for
explain of union.
Both bugs are caused by the same problem: the function optimize_cond() should
update the value of *cond_equal rather than the value of join->cond_equal,
because it is called not only for the WHERE condition, but for the HAVING
condition as well.
- With big_tables=ON, materialized table will use Aria (or MyISAM) SE, which
allows prefix key reads. However, the temp.table has rec_per_key=NULL which
causes the optimizer to crash when attempting to read index statistics for a
prefix index read.
- Fixed by providing a rec_per_key array with zeros (i.e. "no statistics data")
The calls of the function remove_eq_conds() may change the and/or structure
of the where conditions. So JOIN::equal_cond should be updated for non-recursive
calls of remove_eq_conds().
update_used_tables for the the where condition to update cached
indicators of constant subexpressions. It should be done before further
possible simplification of the where condition.
This change caused simplification of the executed where conditions
in many test cases.
- The problem was that JOIN::prepare() tried to set TABLE::maybe_null
for a table in join. Non-merged semi-join tables 1) are present as
join's base tables on second EXECUTE, but 2) do not yet have a TABLE
object.
Worked around the problem by putting mixed_implicit_grouping into JOIN
object, and then passing it to JTBM tables in setup_jtbm_semi_joins().
The field JOIN::select_lex->where should be updated after the call
of remove_eq_conds() in the function make_join_statistics(). This
matters for subselects.
Old code in create_tmp_table(), that created an extra one-byte field (recinfo)
before every NULL-able grouping field (Field) in the tmp table, did not actually work.
Because the matching code in end_update(), that was supposed to update this byte,
was using a wrong offset, updating the first byte of the Field, not a byte before it.
Normally this wasn't an issue, because the Field value (written later in end_update)
was overwriting this byte anyway. But in this bug the Field was Field_null, with zero
length, so end_update() was overwriting the first byte of the following field.
And the following field was not-nullable constant, which was stored only once in
create_tmp_table and never updated later.
Fixed by removing the code that didn't do any useful work anyway.
