database/mariadb
1
0
Fork 0
mirror of https://github.com/MariaDB/server.git synced 2026-10-23 18:49:29 +03:00
Code Activity
Files
25a366f10fc41d2f939431155d56b15e55ccaca5
mariadb/sql/sql_expression_cache.cc
Arcadiy Ivanov f1a72e2cca MDEV-40376 Protocol::end_statement assertion when window function fills HEAP tmp table 
`save_window_function_values()` did not take into account that one can
get `HA_ERR_RECORD_FILE_FULL` on `ha_update_row()`. With blob support in
heap, it can now happen more easily (expressions wider than 512
characters are promoted to TEXT in tmp tables, and each update of a blob
result column allocates a continuation record). However, it could also
happen with Aria tables, which is apparently not tested. The error was
returned as a plain `true` with an empty diagnostics area: debug builds
hit `Assertion '0'` in `Protocol::end_statement()`, release builds send
a bare OK packet after the result set metadata, which the client
mis-parses (appears as a hang / lost connection).

Errors of the computation are now exposed:
`save_window_function_values()` calls `handler::print_error()` for any
failure it does not recover from, the previously ignored `ha_rnd_pos()`
return values are checked and reported, and `compute_window_func()`
distinguishes a read error from EOF and stops the row scan as soon as
the per-row loop fails.

The overflow itself is recovered from by converting the tmp table to
the disk-based tmp engine in place: the rows keep their identity and
their contents, only their positions change, and those are translated
in the rowid sequence that the computation is built on.

The filesort result of a window sort is a sequence of row positions
that covers every row of the table exactly once (the sort is set up
without a limit) and is the only place where the positions are kept:
the row scan and all frame cursors read the rows through it. The
conversion (`Window_rowid_remapper`) therefore copies the rows in the
order of that sequence, which makes the new position of a row known as
soon as the row has been written, and stores it back into the very slot
the old position was read from.

For the new position to always fit into the slot, a window sort stores
the row positions in slots of a fixed 8 bytes
(`Filesort::min_ref_length`, set to the new `TMP_TABLE_MAX_REF_LENGTH`),
which holds the position of any engine an internal tmp table can use: a
pointer into memory or a data file offset. `make_sortkey()` zero-pads a
shorter position to its slot, and `SORT_INFO::ref_length` carries the
slot width to the readers, so `init_read_record()` no longer re-derives
it from `handler::ref_length`, which changes when the table is
converted. Sorts that do not set a minimum, and engines whose positions
are wider than 8 bytes, are unchanged byte for byte. A sequence held in
memory is thereby rewritten in place, keeping its layout, so the frame
cursors' places in it stay valid.

A sequence held in a temporary file is written and read through an
`IO_CACHE`, which encrypts the file when tmp file encryption is
enabled, so the file can not be rewritten in place: the old sequence is
streamed out of its cache and the translated sequence into a new cached
temporary file, which then replaces the old one under the cache. The
frame cursors' slave caches stay linked to the master through
`next_file_user` across the replacement, and are re-created from the
new master afterwards (`Frame_cursor::rowids_rewritten()`). Should
re-creating one fail, the cursor forgets its already-released cache, so
that it is not released a second time when the cursor is destroyed, and
reports the failure.

The row whose update did not fit is written from `record[0]`, which
holds its new image, instead of being read from the table, so the
conversion applies the update that failed. Blob values of `record[0]`
that were read from the HEAP table can point into the handler's shared
blob reassembly buffer (`hp_read_blobs()`, blob values that span
multiple allocation blocks), which reading the other rows overwrites,
so they are first given memory of their own.

When the window sort was set up with a deferred filter (a HAVING clause
deferred to the final ORDER BY sort), the sequence omits the rows the
filter rejected and the conversion drops them: every later reader of
the table applies the same filter, either directly or by reading the
table through a sort that does, so such rows can never reach the
result.

`create_internal_tmp_table_from_heap()` gains a `Tmp_table_row_copier`
hook for the copy from the HEAP table to the Aria or MyISAM table that
replaces it. The plain copy loop and the append of the pending
`record[0]` become `Tmp_table_default_copier`, used when the caller
passes no copier, so both cases of the conversion run through the same
code, defined together with `Window_rowid_remapper` just before the
function. `ha_end_bulk_insert()` is called also when the copy fails, so
the new handler does not keep bulk insert state when its table is
dropped; the pending row is thereby written while the bulk insert is
still active, which does not delay its duplicate detection, as bulk
insert does not cache unique keys (`maria_init_bulk_insert()` skips
`HA_NOSAME` keys). An ignored duplicate of the pending row is passed to
the caller in `Tmp_table_row_copier::duplicate_row_error`, which sets
`*is_duplicate`.

Carrying the running computation over the conversion, instead of
re-running it, has two visible consequences:

 - Compound expressions containing window functions (`items_to_copy`)
   are evaluated exactly once per row, as they are when the table does
   not overflow, so the conversion does not have to be refused when
   such an expression is non-deterministic (`RAND_TABLE_BIT`: a user
   variable assignment, a non-deterministic stored function); those
   statements produce their result.
 - The statement sorts once, so sort related aggregate warnings, e.g.
   the `max_sort_length` truncation counts, are reported for one pass.

The rows are copied in the order of the sequence, so the converted
table does not hold them in the order they were inserted in. This is
safe even when that order is what the table was built for
(`TABLE::keep_row_order`, set for `ROWNUM()` with GROUP BY or ORDER
BY): before HEAP supported blobs, such a tmp table was created in the
disk engine from the start and these statements simply ran, so refusing
here would be a regression against that behavior.

1. `ROWNUM()` values are materialized into the tmp table rows during
   the fill, before the window computation begins; the copy moves them
   verbatim, so their pairing with the rows can not change.
2. The rowid sequence is translated in place, so the running window
   computation continues over exactly the same row order, and
   tie-sensitive window function values (`ROW_NUMBER`, frames over tied
   keys) keep the values the interrupted pass had already produced.
3. Consumers that scan the converted table afterwards see the rows in
   the copy order. Among rows with equal sort keys that order differs
   from the insertion order, but such tie order is unspecified and
   already differs between the memory and disk tmp engines.

The converted table keeps `keep_row_order= true` so the copy order is
also the order the disk engine preserves from then on.

Tests, in `heap.blob_window_overflow` unless noted: a single window
over all rows and a partitioned window overflow mid-computation and
must transparently convert; the side-effecting window expression test
verifies that the user variable is assigned exactly once per row,
against the same query that does not overflow; a fourth test covers the
sequence being an array in memory instead of a merge file; a fifth test
covers the `keep_row_order` conversion, verifying that every `ROWNUM()`
value keeps its insertion-order pairing with its row across the
conversion; a sixth test covers blob values larger than a HEAP
allocation block surviving the conversion of the pending row;
`heap.blob_window_overflow_encrypt` runs the conversion with encrypted
temporary files; `heap.blob_window_overflow_debug` covers the failure
to re-create a slave cache of the sequence
(`simulate_window_seq_slave_oom`), which previously hung the server in
the cursor destructor.
2026-10-11 18:42:16 +03:00

367 lines
10 KiB
C++
Raw Blame History

/* Copyright (C) 2010-2011 Monty Program Ab & Oleksandr Byelkin
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; version 2 of the License.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1335 USA */
#include "mariadb.h"
#include "sql_base.h"
#include "sql_select.h"
#include "sql_expression_cache.h"
/**
Minimum hit ration to proceed on disk if in memory table overflowed.
hit_rate = hit / (miss + hit);
*/
#define EXPCACHE_MIN_HIT_RATE_FOR_DISK_TABLE 0.7
/**
Minimum hit ratio to keep in memory table (do not switch cache off)
hit_rate = hit / (miss + hit);
*/
#define EXPCACHE_MIN_HIT_RATE_FOR_MEM_TABLE 0.2
/**
Number of cache miss to check hit ratio (maximum cache performance
impact in the case when the cache is not applicable)
*/
#define EXPCACHE_CHECK_HIT_RATIO_AFTER 200
/*
Expression cache is used only for caching subqueries now, so its statistic
variables we call subquery_cache*.
*/
ulong subquery_cache_miss, subquery_cache_hit;
Expression_cache_tmptable::Expression_cache_tmptable(THD *thd,
List<Item> &dependants,
Item *value)
:cache_table(NULL), table_thd(thd), tracker(NULL), items(dependants), val(value),
hit(0), miss(0), inited (0)
{
DBUG_ENTER("Expression_cache_tmptable::Expression_cache_tmptable");
DBUG_VOID_RETURN;
};
/**
Disable cache
*/
void Expression_cache_tmptable::disable_cache()
{
if (cache_table->file->inited)
cache_table->file->ha_index_end();
free_tmp_table(table_thd, cache_table);
cache_table= NULL;
update_tracker();
if (tracker)
tracker->detach_from_cache();
}
/**
Field enumerator for TABLE::add_tmp_key
@param arg reference variable with current field number
@return field number
*/
static uint field_enumerator(uchar *arg)
{
return ((uint*)arg)[0]++;
}
/**
Initialize temporary table and auxiliary structures for the expression
cache
@details
The function creates a temporary table for the expression cache, defines
the search index and initializes auxiliary search structures used to check
whether a given set of values of the expression parameters is in some
cache entry.
*/
void Expression_cache_tmptable::init()
{
List_iterator<Item> li(items);
Item_iterator_list it(li);
uint field_counter;
LEX_CSTRING cache_table_name= { STRING_WITH_LEN("subquery-cache-table") };
DBUG_ENTER("Expression_cache_tmptable::init");
DBUG_ASSERT(!inited);
inited= TRUE;
cache_table= NULL;
if (items.elements == 0)
{
DBUG_PRINT("info", ("All parameters were removed by optimizer."));
DBUG_VOID_RETURN;
}
/* add result field */
items.push_front(val);
cache_table_param.init();
/* dependent items and result */
cache_table_param.field_count= cache_table_param.func_count= items.elements;
/* postpone table creation to index description */
cache_table_param.skip_create_table= 1;
if (!(cache_table= create_tmp_table(table_thd, &cache_table_param,
items, (ORDER*) NULL,
FALSE, TRUE,
((table_thd->variables.option_bits |
TMP_TABLE_ALL_COLUMNS) &
~TMP_TABLE_FORCE_MYISAM),
HA_POS_ERROR,
&cache_table_name,
TRUE)))
{
DBUG_PRINT("error", ("create_tmp_table failed, caching switched off"));
DBUG_VOID_RETURN;
}
if (cache_table->s->db_type() != heap_hton)
{
DBUG_PRINT("error", ("we need only heap table"));
goto error;
}
/*
HEAP hash indexes on blob columns use a pointer-based key format
(4-byte length + data pointer). This is incompatible with the SQL
layer's key format (2-byte length + inline data) because
Field_blob::new_key_field() returns a Field_varstring.
This check is slightly conservative: a blob only in the result
value would not affect the key. However, it matches the pre-blob
behavior where blobs forced Aria, which failed the heap_hton check
above and disabled the cache anyway.
*/
if (cache_table->s->blob_fields)
{
DBUG_PRINT("error", ("blob fields not supported in heap expression cache"));
goto error;
}
field_counter= 1;
if (cache_table->alloc_keys(1) ||
cache_table->add_tmp_key(0, items.elements - 1, &field_enumerator,
(uchar*)&field_counter, TRUE) ||
ref.tmp_table_index_lookup_init(table_thd, cache_table->key_info, it,
TRUE, 1 /* skip result field*/))
{
DBUG_PRINT("error", ("creating index failed"));
goto error;
}
cache_table->s->keys= 1;
ref.null_rejecting= 1;
ref.const_ref_part_map= 0;
ref.disable_cache= FALSE;
ref.has_record= 0;
ref.use_count= 0;
if (open_tmp_table(cache_table))
{
DBUG_PRINT("error", ("Opening (creating) temporary table failed"));
goto error;
}
if (!(cached_result= new (table_thd->mem_root)
Item_field(table_thd, cache_table->field[0])))
{
DBUG_PRINT("error", ("Creating Item_field failed"));
goto error;
}
update_tracker();
DBUG_VOID_RETURN;
error:
disable_cache();
DBUG_VOID_RETURN;
}
Expression_cache_tmptable::~Expression_cache_tmptable()
{
/* Add accumulated statistics */
statistic_add(subquery_cache_miss, miss, &LOCK_status);
statistic_add(subquery_cache_hit, hit, &LOCK_status);
if (cache_table)
disable_cache();
else
{
update_tracker();
if (tracker)
tracker->detach_from_cache();
tracker= NULL;
}
}
/**
Check if a given set of parameters of the expression is in the cache
@param [out] value the expression value found in the cache if any
@details
For a given set of the parameters of the expression the function
checks whether it can be found in some entry of the cache. If so
the function returns the result of the expression extracted from
the cache.
@retval Expression_cache::HIT if the set of parameters is in the cache
@retval Expression_cache::MISS - otherwise
*/
Expression_cache::result Expression_cache_tmptable::check_value(Item **value)
{
int res;
DBUG_ENTER("Expression_cache_tmptable::check_value");
if (cache_table)
{
DBUG_PRINT("info", ("status: %u has_record %u",
(uint)cache_table->status, (uint)ref.has_record));
if ((res= join_read_key2(table_thd, NULL, cache_table, &ref)) == 1)
DBUG_RETURN(ERROR);
if (res)
{
if (((++miss) == EXPCACHE_CHECK_HIT_RATIO_AFTER) &&
((double)hit / ((double)hit + miss)) <
EXPCACHE_MIN_HIT_RATE_FOR_MEM_TABLE)
{
DBUG_PRINT("info",
("Early check: hit rate is not so good to keep the cache"));
disable_cache();
}
DBUG_RETURN(MISS);
}
hit++;
*value= cached_result;
DBUG_RETURN(Expression_cache::HIT);
}
DBUG_RETURN(Expression_cache::MISS);
}
/**
Put a new entry into the expression cache
@param value the result of the expression to be put into the cache
@details
The function evaluates 'value' and puts the result into the cache as the
result of the expression for the current set of parameters.
@retval FALSE OK
@retval TRUE Error
*/
my_bool Expression_cache_tmptable::put_value(Item *value)
{
int error;
DBUG_ENTER("Expression_cache_tmptable::put_value");
DBUG_ASSERT(inited);
if (!cache_table)
{
DBUG_PRINT("info", ("No table so behave as we successfully put value"));
DBUG_RETURN(FALSE);
}
*(items.head_ref())= value;
fill_record(table_thd, cache_table, cache_table->field, items, true, true,
true);
if (unlikely(table_thd->is_error()))
goto err2;
if (unlikely((error=
cache_table->file->ha_write_tmp_row(cache_table->record[0]))))
{
/* create_myisam_from_heap will generate error if needed */
if (cache_table->file->is_fatal_error(error, HA_CHECK_DUP) &&
error != HA_ERR_RECORD_FILE_FULL)
goto err;
else
{
double hit_rate= ((double)hit / ((double)hit + miss));
DBUG_ASSERT(miss > 0);
if (hit_rate < EXPCACHE_MIN_HIT_RATE_FOR_MEM_TABLE)
{
DBUG_PRINT("info", ("hit rate is not so good to keep the cache"));
disable_cache();
DBUG_RETURN(FALSE);
}
else if (hit_rate < EXPCACHE_MIN_HIT_RATE_FOR_DISK_TABLE)
{
DBUG_PRINT("info", ("hit rate is not so good to go to disk"));
if (cache_table->file->ha_delete_all_rows() ||
cache_table->file->ha_write_tmp_row(cache_table->record[0]))
goto err;
}
else
{
bool is_duplicate;
if (create_internal_tmp_table_from_heap(table_thd, cache_table,
cache_table_param.start_recinfo,
&cache_table_param.recinfo,
error, 1, &is_duplicate, NULL))
goto err2;
}
}
}
cache_table->status= 0; /* cache_table->record contains an existed record */
ref.has_record= TRUE; /* the same as above */
DBUG_PRINT("info", ("has_record: TRUE status: 0"));
DBUG_RETURN(FALSE);
err:
cache_table->file->print_error(error, MYF(0));
err2:
disable_cache();
DBUG_RETURN(TRUE);
}
void Expression_cache_tmptable::print(String *str, enum_query_type query_type)
{
List_iterator<Item> li(items);
Item *item;
bool is_first= TRUE;
str->append('<');
li++; // skip result field
while ((item= li++))
{
if (!is_first)
str->append(',');
item->print(str, query_type);
is_first= FALSE;
}
str->append('>');
}
const char *Expression_cache_tracker::state_str[3]=
{"uninitialized", "disabled", "enabled"};
View Git Blame Copy Permalink