REGULAR SQL VS PREPARED STATEMENT
Analysis:
---------
When passing user variables as parameters to the
prepared statements, the IF() function evaluation
turns out to be incorrect.
Consider the example:
SET @var1='0.038687';
SELECT @var1 , IF( @var1 = 0 , 1 ,@var1 ) AS sqlif ;
+----------+----------+
| @var1 | sqlif |
+----------+----------+
| 0.038687 | 0.038687 |
+----------+----------+
Executing a prepared statement where the parameters are
supplied:
PREPARE fail_stmt FROM "SELECT ? ,
IF( ? = 0 , 1 , ? ) AS ps_if_fail" ;
EXECUTE fail_stmt USING @var1 ,@var1 , @var1 ;
+----------+------------+
| ? | ps_if_fail |
+----------+------------+
| 0.038687 | 1 |
+----------+------------+
1 row in set (0.00 sec)
In the regular statement or while executing the prepared
statements without passing parameters, the decimal
precision is set for the user variable of type string.
The comparison function used for evaluation considered
the precision while comparing the values.
But while executing the prepared statement with the
parameters supplied, the decimal precision was not
set. Thus the comparison function chosen was different
which looked at the absolute values for comparison.
Fix:
----
The fix is to set 'decimals' field of Item_param to the
default value which is nothing but the maximum number of
decimals(NOT_FIXED_DEC). This is set for cases where the
strings are converted to the numeric form within certain
functions. Thus the value is not rounded off during
comparison, ensuring correct evaluation.
Some queries with the "SELECT ... FROM DUAL" nested subqueries
failed with an assertion on debug builds.
Non-debug builds were not affected.
There were a few different issues with similar assertion
failures on different queries:
1. The first problem was related to the incomplete propagation
of the "non-constant" item status from underlying subquery
items to the outer item tree: in some cases non-constants were
interpreted as constants and evaluated at the preparation stage
(val_int() calls withing fix_fields() etc).
Thus, the default implementation of Item_ref::const_item() from
the Item parent class didn't take into account the "const_item"
status of the referenced item tree -- it used the insufficient
"used_tables() == 0" check instead. This worked in most cases
since our "non-constant" functions like RAND() and SLEEP() set
the RAND_TABLE_BIT in the used table map, so they aren't
non-constant from Item_ref's "point of view". However, the
"SELECT ... FROM DUAL" subquery may have an empty map of used
tables, but at the same time subqueries are never "constant" at
the context analysis stage (preparation, view creation etc).
So, the non-contantness of such subqueries was missed.
Fix: the Item_ref::const_item() function has been overloaded to
take into account both (*ref)->const_item() status and tricky
Item_ref::used_tables() return values, since the only
(*ref)->const_item() call is not enough there.
2. In some cases instead of the const_item() call we check a
value of the Item::with_subselect field to recognize items
with nested subqueries. However, the Item_ref class didn't
propagate this value from the referenced item tree.
Fix: Item::has_subquery() and Item_ref::has_subquery()
functions have been backported from 5.6. All direct
references to the with_subselect fields of nested items have
been with the has_subquery() function call.
3. The Item_func_regex class didn't propagate with_subselect
as well, since it overloads the Item_func::fix_fields()
function with insufficient fix_fields() implementation.
Fix: the Item_func_regex::fix_fields() function has been
modified to gather "constant" statuses from inner items.
4. The Item_func_isnull::update_used_tables() function has
a special branch for the underlying item where the maybe_null
value is false: in this case it marks the Item_func_isnull
as a "const_item" and sets the cached_value to false.
However, the Item_func_isnull::val_int() was not in sync with
update_used_tables(): it didn't take into account neither
const_item_cache nor cached_value for the case of
"args[0]->maybe_null == false optimization".
As far as such an Item_func_isnull has "const_item() == true",
it's ok to call Item_func_isnull::val_int() etc from outer
items on preparation stage. In this case the server tried to
call Item_func_isnull::args[0]->isnull(), and if the args[0]
item contained a nested not-nullable subquery, it failed
with an assertion.
Fix: take the value of Item_func_isnull::const_item_cache into
account in the val_int() function.
5. The auxiliary Item_is_not_null_test class has a similar
optimization in the update_used_tables() function as the
Item_func_isnull class has, and the same issue in the val_int()
function.
In addition to that the Item_is_not_null_test::update_used_tables()
doesn't update the const_item_cache value, so the "maybe_null"
optimization is useless there. Thus, we missed some optimizations
of cases like these (before and after the fix):
< <is_not_null_test>(a),
---
> <cache>(<is_not_null_test>(a)),
or
< having (<is_not_null_test>(a) and <is_not_null_test>(a))
---
> having 1
etc.
Fix: update Item_is_not_null_test::const_item_cache in
update_used_tables() and take in into account in val_int().
When resolving outer fields, Item_field::fix_outer_fields()
creates new Item_refs for each execution of a prepared statement, so
these must be allocated in the runtime memroot. The memroot switching
before resolving JOIN::having causes these to be allocated in the
statement root, leaking memory for each PS execution.
Problem: Some queries with subqueries and a HAVING clause that
consists only of a column not in the select or grouping lists causes
the server to crash.
During parsing, an Item_ref is constructed for the HAVING column. The
name of the column is resolved when JOIN::prepare calls fix_fields()
on its having clause. Since the column is not mentioned in the select
or grouping lists, a ref pointer is not found and a new Item_field is
created instead. The Item_ref is replaced by the Item_field in the
tree of HAVING clauses. Since the tree consists only of this item, the
pointer that is updated is JOIN::having. However,
st_select_lex::having still points to the Item_ref as the root of the
tree of HAVING clauses.
The bug is triggered when doing filesort for create_sort_index(). When
find_all_keys() calls select->cond->walk() it eventually reaches
Item_subselect::walk() where it continues to walk the having clauses
from lex->having. This means that it finds the Item_ref instead of the
new Item_field, and Item_ref::walk() tries to dereference the ref
pointer, which is still null.
The crash is reproducible only in 5.5, but the problem lies latent in
5.1 and trunk as well.
Fix: After calling fix_fields on the having clause in JOIN::prepare(),
set select_lex::having to point to the same item as JOIN::having.
This patch also fixes a bug in 5.1 and 5.5 that is triggered if the
query is executed as a prepared statement. The Item_field is created
in the runtime arena when the query is prepared, and the pointer to
the item is saved by st_select_lex::fix_prepare_information() and
brought back as a dangling pointer when the query is executed, after
the runtime arena has been reclaimed.
Fix: Backport fix from trunk that switches to the permanent arena
before calling Item_ref::fix_fields() in JOIN::prepare().
The table contains one time value: '00:00:32'
This value is converted to timestamp by a subquery.
In convert_constant_item we call (*item)->is_null()
which triggers execution of the Item_singlerow_subselect subquery,
and the string "0000-00-00 00:00:32" is cached
by Item_cache_datetime.
We continue execution and call update_null_value, which calls val_int()
on the cached item, which converts the time value to ((longlong) 32)
Then we continue to do (*item)->save_in_field()
which ends up in Item_cache_datetime::val_str() which fails,
since (32 < 101) in number_to_datetime, and val_str() returns NULL.
Item_singlerow_subselect::val_str isnt prepared for this:
if exec() succeeds, and return !null_value, then val_str()
*must* succeed.
Solution: refuse to cache strings like "0000-00-00 00:00:32"
in Item_cache_datetime::cache_value, and return NULL instead.
This is similar to the solution for
Bug#11766860 - 60085: CRASH IN ITEM::SAVE_IN_FIELD() WITH TIME DATA TYPE
This patch is for 5.5 only.
The issue is not present after WL#946, since a time value
will be converted to a proper timestamp, with the current date
rather than "0000-00-00"
Problem: Grouping results by VALUES(alias for string literal) causes
the server to crash.
Item_insert_values is not constructed to handle other types of
arguments than field and reference to field. In this case, the
argument is an Item_string, and this causes
Item_insert_values::fix_fields() to crash.
Fix: Issue an error message when the argument to Item_insert_values is
not a field or a reference to a field.
This is slightly in breach with documentation, which states that
VALUES should return NULL, but the error message is only issued in
cases where the server otherwise would crash, so there is no change in
behavior for queries that already work. Future versions will restrict
syntax so that using VALUES in this way is illegal.
Analysis:
========================
sql_mode "NO_BACKSLASH_ESCAPES": When user want to use backslash as character input,
instead of escape character in a string literal then sql_mode can be set to
"NO_BACKSLASH_ESCAPES". With this mode enabled, backslash becomes an ordinary
character like any other.
SQL_MODE set applies to the current client session. And while creating the stored
procedure, MySQL stores the current sql_mode and always executes the stored
procedure in sql_mode stored with the Procedure, regardless of the server SQL
mode in effect when the routine is invoked.
In the scenario (for which bug is reported), the routine is created with
sql_mode=NO_BACKSLASH_ESCAPES. And routine is executed with the invoker sql_mode
is "" (NOT SET) by executing statement "call testp('Axel\'s')".
Since invoker sql_mode is "" (NOT_SET), the '\' in 'Axel\'s'(argument to function)
is considered as escape character and column "a" (of table "t1") values are
updated with "Axel's". The binary log generated for above update operation is as below,
set sql_mode=XXXXXX (for no_backslash_escapes)
update test.t1 set a= NAME_CONST('var',_latin1'Axel\'s' COLLATE 'latin1_swedish_ci');
While logging stored procedure statements, the local variables (params) used in
statements are replaced with the NAME_CONST(var_name, var_value) (Internal function)
(http://dev.mysql.com/doc/refman/5.6/en/miscellaneous-functions.html#function_name-const)
On slave, these logs are applied. NAME_CONST is parsed to get the variable and its
value. Since, stored procedure is created with sql_mode="NO_BACKSLASH_ESCAPES", the sql_mode
is also logged in. So that at slave this sql_mode is set before executing the statements
of routine. So at slave, sql_mode is set to "NO_BACKSLASH_ESCAPES" and then while
parsing NAME_CONST of string variable, '\' is considered as NON ESCAPE character
and parsing reported error for "'" (as we have only one "'" no backslash).
At slave, parsing was proper with sql_mode "NO_BACKSLASH_ESCAPES".
But above error reported while writing bin log, "'" (of Axel's) is escaped with
"\" character. Actually, all special characters (n, r, ', ", \, 0...) are escaped
while writing NAME_CONST for string variable(param, local variable) in bin log
Airrespective of "NO_BACKSLASH_ESCAPES" sql_mode. So, basically, the problem is
that logging string parameter does not take into account sql_mode value.
Fix:
========================
So when sql_mode is set to "NO_BACKSLASH_ESCAPES", escaping characters as
(n, r, ', ", \, 0...) should be avoided. To do so, added a check to not to
escape such characters while writing NAME_CONST for string variables in bin
log.
And when sql_mode is set to NO_BACKSLASH_ESCAPES, quote character "'" is
represented as ''.
http://dev.mysql.com/doc/refman/5.6/en/string-literals.html (There are several
ways to include quote characters within a string: )
BUG#13519696 - 62940: SELECT RESULTS VARY WITH VERSION AND
WITH/WITHOUT INDEX RANGE SCAN
BUG#13453382 - REGRESSION SINCE 5.1.39, RANGE OPTIMIZER WRONG
RESULTS WITH DECIMAL CONVERSION
BUG#13463488 - 63437: CHAR & BETWEEN WITH INDEX RETURNS WRONG
RESULT AFTER MYSQL 5.1.
Those are all cases where the range optimizer got it wrong
with > and >=.
Fixing the 5.5 part (the 5.6 part will go in a separate commit soon).
Problem:
Item_direct_ref::get_date() incorrectly calculated its "null_value",
which made UNIX_TIMESTAMP(view_column) incorrectly return NULL
for a NOT NULL view_column.
Fix:
Make Item_direct_ref::get_date() calculate null_value
in the similar way with the other methods
(val_real,val_str,val_int,val_decimal):
copy null_value from the referenced Item.
modified:
mysql-test/r/func_time.result
mysql-test/t/func_time.test
sql/item.cc
The predicate is re-written from
((`test`.`g1`.`a` = geometryfromtext('')) or ...
to
((`test`.`g1`.`a` = <cache>(geometryfromtext(''))) or ...
The range optimizer calls save_in_field_no_warnings, in order to fetch keys.
save_in_field_no_warnings returns 0 because of the cache wrapper,
and get_mm_leaf() proceeded to call Field_blob::get_key_image()
which accesses un-initialized data.
and innodb
The 5.5 version of the patch.
The server doesn't restrict the data that can be inserted into integer columns
with explicitly specified length that's smaller than what the type can handle,
e.g. 1234 can be inserted into an INT(2) column just fine.
Thus, when calcualting the maximum width of expressions involving such
restricted integer columns we need to use the implicit maximum width of
the field instead of the explicitly speficied one.
Fixed the server to use the implicit maximum in such cases and made sure
the implicit maximum is addjusted the same way as the explicit one wrt
signedness.
Fixed several test case results (ctype_*.result, metadata.result and
type_ranges.result) to reflect the extended column widths.
Added a regression test case in distinct.test.
Note : this is the behavior preserving fix that makes 5.5 behave as 5.1 and
earlier. In the mysql trunk we'll add a insert time check for the explict
maximum size.
Attempts to assign value to a table column from trigger by using
NEW.column_name pseudo-variable might result in garbled data.
That happened when:
- the column had a BLOB-based type (e.g. TEXT)
and
- the value being assigned was retrieved from stored routine variable
of the same type.
The problem was that BLOB values were not copied correctly in this
case. Instead of doing a copy of a real value, the value's representation
in record buffer was copied. This representation is essentially a
pointer to a buffer associated with the virtual table for routine
variables where the real value is stored. Since this buffer got
freed once trigger was left or could have changed its contents when
new value was assigned to corresponding routine variable such a shallow
copying resulted in garbled data in NEW.colum_name column.
It worked in 5.1 due to a subtle bug in create_virtual_tmp_table():
- in 5.1 create_virtual_tmp_table() returned a table which
had db_low_byte_first == false.
- in 5.5 and up create_virtual_tmp_table() returns a table which
has db_low_byte_first == true.
Actually, db_low_byte_first == false only for ISAM storage engine,
which was deprecated and removed in 5.0.
Having db_low_byte_first == false led to getting false in the
complex condition for the 2nd "if" in field_conv(), which in turn
led to copy-blob-behavior as a fall-back strategy:
- to->table->s->db_low_byte_first was true (correct value)
- from->table->s->db_low_byte_first was false (incorrect value)
In 5.5 and up that condition is true, which means blob-values are
not copied.
(SUBSTRING inside a stored function works too slow).
The user-visible problem was that the server started to consume memory if a
stored-routine of some sort is executed subsequently. The memory was freed
only after the corresponding connection was closed.
Technically, the problem was that the memory needed for temporary string
conversions was allocated on the connection ("persistent") memory root,
instead of statement one.
The root cause of this problem was the incorrect patch for Bug 55744.
That patch wrongly fixed a crash in prepared-statement-mode introduced by
another patch. The patch for Bug 55744 used wrong condition to check if
prepared statement mode is active (or whether the connection-scoped or
statement-scoped memory root should be used). The thing is that for
prepared statements such conversions should be done in the connection
memory root, so that that the transformations of item-tree were correctly
remembered in the PREPARE-phase.
The fix is to use proper condition to detect prepared-statement-mode and
use proper memory root.
(SUBSTRING inside a stored function works too slow).
Background:
- THD classes derives from Query_arena, thus inherits the 'state'
attribute and related operations (is_stmt_prepare() & co).
- Although these operations are available in THD, they must not
be used. THD has its own attribute to point to the active
Query_arena -- stmt_arena.
- So, instead of using thd->is_stmt_prepare(),
thd->stmt_arena->is_stmt_prepare() must be used. This was the root
cause of Bug 60025.
This patch enforces the proper way of calling those operations.
is_stmt_prepare() & co are declared as private operations
in THD (thus, they are hidden from being called on THD instance).
The patch tries to minimize changes in 5.5.
Problem: comparison of a DATETIME sp variable and NOW()
led to Illegal mix of collations error when
character_set_connection=utf8.
Introduced by "WL#2649 Number-to-string conversions".
Error happened in Arg_comparator::set_compare_func(),
because the first argument was errouneously converted to utf8,
while the second argument was not.
Fix: separate agg_arg_charsets_for_comparison() into two functions:
- agg_arg_charsets_for_comparison() - for pure comparison,
when we don't need to return any string result and therefore
don't need to convert arguments to @@character_set_connection:
SELECT a = b;
- agg_arg_charsets_for_string_results_with_comparison() - when
we need to return a string result, but we also need to do
comparison internally: SELECT REPLACE(a,b,c)
If all arguments are numbers:
SELECT REPLACE(123,2,3) -> 133
we convert arguments to @@character_set_connection.
@ mysql-test/include/ctype_numconv.inc
@ mysql-test/r/ctype_binary.result
@ mysql-test/r/ctype_cp1251.result
@ mysql-test/r/ctype_latin1.result
@ mysql-test/r/ctype_ucs.result
@ mysql-test/r/ctype_utf8.result
Adding tests
@ sql/item.cc
@ sql/item.h
@ sql/item_func.cc
@ sql/item_func.h
@ sql/item_strfunc.cc
Introducing and using new function
agg_item_charsets_for_string_result_with_comparison() and
its Item_func wrapper agg_arg_charsets_for_string_result_with_comparison().
Select from a view with the underlying HAVING clause failed with a
message: "1356: View '...' references invalid table(s) or column(s)
or function(s) or definer/invoker of view lack rights to use them"
The bug is a regression of the fix for bug 11750328 - 40825 (similar
case, but the HAVING cause references an aliased field).
In the old fix for bug 40825 the Item_field::name_length value has
been used in place of the real length of Item_field::name. However,
in some cases Item_field::name_length is not in sync with the
actual name length (TODO: combine name and name_length into a
solid String field).
The Item_ref::print() method has been modified to calculate actual
name length every time.
In the string context the MIN() and MAX() functions don't take
into account the unsignedness of the UNSIGNED BIGINT argument
column.
I.e.:
CREATE TABLE t1 (a BIGINT UNSIGNED);
INSERT INTO t1 VALUES (18446668621106209655);
SELECT CONCAT(MAX(a)) FROM t1;
returns -75452603341961.
The problem was that server didn't check resulting size of prepared
statement argument which was set using mysql_send_long_data() API.
By calling mysql_send_long_data() several times it was possible
to create overly big string and thus force server to allocate
memory for it. There was no way to limit this allocation.
The solution is to add check for size of result string against
value of max_long_data_size start-up parameter. When intermediate
string exceeds max_long_data_size value an appropriate error message
is emitted.
We can't use existing max_allowed_packet parameter for this purpose
since its value is limited by 1GB and therefore using it as a limit
for data set through mysql_send_long_data() API would have been an
incompatible change. Newly introduced max_long_data_size parameter
gets value from max_allowed_packet parameter unless its value is
specified explicitly. This new parameter is marked as deprecated
and will be eventually replaced by max_allowed_packet parameter.
Value of max_long_data_size parameter can be set only at server
startup.
@ mysql-test/r/ctype_latin1.result
@ mysql-test/r/ctype_utf8.result
@ mysql-test/t/ctype_latin1.test
@ mysql-test/t/ctype_utf8.test
Adding tests
@ sql/mysqld.h
@ sql/item.cc
@ sql/sql_parse.cc
@ sql/sql_view.cc
Refactoring (thanks to Guilhem for the idea):
Item_string::print() was hard to understand because of the different
QT_ constants: in "query_type==QT_x", QT_x is explicitely included
but the other two QT_ are implicitely excluded. The combinations
with '||' and '&&' make this even harder.
- logic is now more "explicit" by changing QT_ constants to a bitmap of flags:
QT_ORDINARY: no change,
QT_IS -> QT_TO_SYSTEM_CHARSET | QT_WITHOUT_INTRODUCERS,
QT_EXPLAIN -> QT_TO_SYSTEM_CHARSET
(QT_EXPLAIN was introduced in the first version of the Bug#57341 patch)
- Item_string::print() is rewritten using those flags
Bugfix itself:
When QT_TO_SYSTEM_CHARSET is used alone (with no QT_WITHOUT_INTRODUCERS),
we print string literals as follows:
- display introducers if they were in the original query
- print ASCII characters as is
- print non-ASCII characters using hex-escape
Note: as "EXPLAIN" output is only for human readability purposes
and does not need to be a pasrable SQL, so using hex-escape is Ok.
ErrConvString class perfectly suites for hex escaping purposes.
This assumption in Item_cache_datetime::cache_value_int
was wrong:
- /* Assume here that the underlying item will do correct conversion.*/
- int_value= example->val_int_result();
