It was possible to construct a PCRE expression that exceeded the stack.
resulting in a crash:
With fix:
MariaDB [(none)]> SELECT 1
-> FROM dual
-> WHERE ('Alpha,Bravo,Charlie,Delta,Echo,Foxtrot,StrataCentral,Golf,Hotel,India,Juliet,Kilo,Lima,Mike,StrataL3,November,Oscar,StrataL2,Sand,P3,P4SwitchTest,Arsys,Poppa,ExtensionMgr,Arp,Quebec,Romeo,StrataApiV2,PtReyes,Sierra,SandAcl,Arrow,Artools,BridgeTest,Tango,SandT,PAlaska,Namespace,Agent,Qos,PatchPanel,ProjectReport,Ark,Gimp,Agent,SliceAgent,Arnet,Bgp,Ale,Tommy,Central,AsicPktTestLib,Hsc,SandL3,Abuild,Pca9555,Standby,ControllerDut,CalSys,SandLib,Sb820,PointV2,BfnLib,Evpn,BfnSdk,Sflow,ManagementActive,AutoTest,GatedTest,Bgp,Sand,xinetd,BfnAgentLib,bf-utils,Hello,BfnState,Eos,Artest,Qos,Scd,ThermoMgr,Uniform,EosUtils,Eb,FanController,Central,BfnL3,BfnL2,tcp_wrappers,Victor,Environment,Route,Failover,Whiskey,Xray,Gimp,BfnFixed,Strata,SoCal,XApi,Msrp,XpProfile,tcpdump,PatchPanel,ArosTest,FhTest,Arbus,XpAcl,MacConc,XpApi,telnet,QosTest,Alpha2,BfnVlan,Stp,VxlanControllerTest,MplsAgent,Bravo2,Lanz,BfnMbb,Intf,XCtrl,Unicast,SandTunnel,L3Unicast,Ipsec,MplsTest,Rsvp,EthIntf,StageMgr,Sol,MplsUtils,Nat,Ira,P4NamespaceDut,Counters,Charlie2,Aqlc,Mlag,Power,OpenFlow,Lag,RestApi,BfdTest,strongs,Sfa,CEosUtils,Adt746,MaintenanceMode,MlagDut,EosImage,IpEth,MultiProtocol,Launcher,Max3179,Snmp,Acl,IpEthTest,PhyEee,bf-syslibs,tacc,XpL2,p4-ar-switch,p4-bf-switch,LdpTest,BfnPhy,Mirroring,Phy6,Ptp'
->
-> REGEXP '^((?!\b(Strata|StrataApi|StrataApiV2)\b).)*$');
Empty set, 1 warning (0.00 sec)
MariaDB [(none)]> show warnings;
+---------+------+---------------------------------------------------------+
| Level | Code | Message |
+---------+------+---------------------------------------------------------+
| Warning | 1139 | Got error 'pcre_exec: Internal error (-21)' from regexp |
+---------+------+---------------------------------------------------------+
- Adding Type_handler::make_table_field() and moving pieces of the code
from Item::tmp_table_field_from_field_type() to virtual implementations
for various type handlers.
- Adding a new Type_all_attributes, to access to Item's extended
attributes, such as decimal_precision() and geometry_type().
- Adding a new class Record_addr, to pass record related information
to Type_handler methods (ptr, null_ptr and null_bit) as a single structure.
Note, later it will possibly be extended for BIT-alike field purposes,
by adding new members (bit_ptr_arg, bit_ofs_arg).
- Moving the code from Field_new_decimal::create_from_item()
to Type_handler_newdecimal::make_table_field().
- Removing Field_new_decimal() and Field_geom() helper constructor
variants that were used for temporary field creation.
- Adding Item_field::type_handler(), Field::type_handler() and
Field_blob::type_handler() to return correct type handlers for
blob variants, according to Field_blob::packlength.
- Adding Type_handler_blob_common, as a common parent for
Type_handler_tiny_blob, Type_handler_blob, Type_handler_medium_blob
and Type_handler_long_blob.
- Implementing Type_handler_blob_common::Item_hybrid_func_fix_attributes().
It's needed for cases when TEXT variants of different character sets are mixed
in LEAST, GREATEST, CASE and its abreviations (IF, IFNULL, COALESCE), e.g.:
CREATE TABLE t1 (
a TINYTEXT CHARACTER SET latin1,
b TINYTEXT CHARACTER SET utf8
);
CREATE TABLE t2 AS SELECT COALESCE(a,b) FROM t1;
Type handler aggregation returns TINYTEXT as a common data type
for the two columns. But as conversion from latin1 to utf8
happens for "a", the maximum possible length of "a" grows from 255 to 255*3.
Type_handler_blob_common::Item_hybrid_func_fix_attributes() makes sure
to update the blob type handler according to max_length.
- Adding Type_handler::blob_type_handler(uint max_octet_length).
- Adding a few m_type_aggregator_for_result.add() pairs, because
now Item_xxx::type_handler() can return pointers to type_handler_tiny_blob,
type_handler_blob, type_handler_medium_blob, type_handler_long_blob.
Before the patch only type_handler_blob was possible result of type_handler().
- Making type_handler_tiny_blob, type_handler_blob, type_handler_medium_blob,
type_handler_long_blob public.
- Removing the condition in Item_sum_avg::create_tmp_field()
checking Item_sum_avg::result_type() against DECIMAL_RESULT.
Now both REAL_RESULT and DECIMAL_RESULT are symmetrically handled
by tmp_table_field_from_field_type().
- Removing Item_geometry_func::create_field_for_create_select(),
as the inherited version perfectly works.
- Fixing Item_func_as_wkb::field_type() to return MYSQL_TYPE_LONG_BLOB
rather than MYSQL_TYPE_BLOB. It's needed to make sure that
tmp_table_field_from_field_type() creates a LONGBLOB field for AsWKB().
- Fixing Item_func_as_wkt::fix_length_and_dec() to set max_length to
UINT32_MAX rather than MAX_BLOB_WIDTH, to make sure that
tmp_table_field_from_field_type() creates a LONGTEXT field for AsWKT().
- Removing Item_func_set_user_var::create_field_for_create_select(),
as the inherited version works fine.
- Adding Item_func_get_user_var::create_field_for_create_select() to
make sure that "CREATE TABLE t1 AS SELECT @string_user variable"
always creates a field of LONGTEXT/LONGBLOB type.
- Item_func_ifnull::create_field_for_create_select()
behavior has changed. Before the patch it passed set_blob_packflag=false,
which meant to create LONGBLOB for all blob variants.
Now it takes into account max_length, which gives better column
data types for:
CREATE TABLE t2 AS SELECT IFNULL(blob_column1, blob_column2) FROM t1;
- Fixing Item_func_nullif::fix_length_and_dec() to use
set_handler(args[2]->type_handler()) instead of
set_handler_by_field_type(args[2]->field_type()).
This is needed to distinguish between BLOB variants.
- Implementing Item_blob::type_handler(), to make sure to create
proper BLOB field variant, according to max_length, for queries like:
CREATE TABLE t1 AS
SELECT some_blob_field FROM INFORMATION_SCHEMA.SOME_TABLE;
- Fixing Item_field::real_type_handler() to make sure that
the code aggregating fields for UNION gets a proper BLOB
variant type handler from fields.
- Adding a special code into Item_type_holder::make_field_by_type(),
to make sure that after aggregating field types it also properly
takes into account max_length when mixing TEXT variants of different
character sets and chooses a proper TEXT variant:
CREATE TABLE t1 (
a TINYTEXT CHARACTER SET latin1,
b TINYTEXT CHARACTER SET utf8
);
CREATE TABLE t2 AS SELECT a FROM t1 UNION SELECT b FROM t1;
- Adding tests, for better coverage of IFNULL, NULLIF, UNION.
- The fact that tmp_table_field_from_field_type() now takes
into account BLOB variants (instead of always creating LONGBLOB),
tests results for WEIGHT_STRING() and NULLIF() and UNION
have become more precise.
The patch actually fixes the old defect of the optimizer that
could not extract keys for range access from IN predicates
with row arguments.
This problem was resolved in the mysql-5.7 code. The patch
supersedes what was done there:
- it can build range access when not all components of
the first row argument are refer to the columns of the table
for which the range access is constructed.
- it can use equality predicates to build range access
to the table that is not referred to in this argument.
Also, implement MDEV-11027 a little differently from 5.5 and 10.0:
recv_apply_hashed_log_recs(): Change the return type back to void
(DB_SUCCESS was always returned).
Report progress also via systemd using sd_notifyf().
Item_func_le included Arg_comparator. Arg_comparator remembered
the current_thd during fix_fields and used that value during
execution to allocate Item_cache in get_datetime_value().
But for vcols fix_fields and val_int can happen in different threads.
Same bug for Item_func_in using in_datetime or cmp_item_datetime,
both also remembered current_thd at fix_fields() to use it later
for get_datetime_value().
As a fix, these objects no longer remember the current_thd,
and get_datetime_value() uses current_thd at run time. This
should not increase the number of current_thd calls much, as
Item_cache is created only once anyway.
Now after the patch for MDEV-11478 added a new method Type_handler::name(),
using the new method in debug output to make test results for MDEV-11514
more readable (func_debug.result).
When a condition containing NULLIF is pushed into a materialized
view/derived table the clone of the Item_func_nullif item must
be processed in a special way to guarantee that the first argument
points to the same item as the third argument.
This patch fixes a number of data type aggregation problems in IN and CASE:
- MDEV-11497 Wrong result for (int_expr IN (mixture of signed and unsigned expressions))
- MDEV-11514 IN with a mixture of TIME and DATETIME returns a wrong result
- MDEV-11554 Wrong result for CASE on a mixture of signed and unsigned expressions
- MDEV-11555 CASE with a mixture of TIME and DATETIME returns a wrong result
1. The problem reported in MDEV-11514 and MDEV-11555 was in the wrong assumption
that items having the same cmp_type() can reuse the same cmp_item instance.
So Item_func_case and Item_func_in used a static array of cmp_item*,
one element per one XXX_RESULT.
TIME and DATETIME cannot reuse the same cmp_item, because arguments of
these types are compared very differently. TIME and DATETIME must have
different instances in the cmp_item array. Reusing the same cmp_item
for TIME and DATETIME leads to unexpected result and unexpected warnings.
Note, after adding more data types soon (e.g. INET6), the problem would
become more serious, as INET6 will most likely have STRING_RESULT, but
it won't be able to reuse the same cmp_item with VARCHAR/TEXT.
This patch introduces a new class Predicant_to_list_comparator,
which maintains an array of cmp_items, one element per distinct
Type_handler rather than one element per XXX_RESULT.
2. The problem reported in MDEV-11497 and MDEV-11554 happened because
Item_func_in and Item_func_case did not take into account the fact
that UNSIGNED and SIGNED values must be compared as DECIMAL rather than INT,
because they used item_cmp_type() to aggregate the arguments.
The relevant code now resides in Predicant_to_list_comparator::add_value()
and uses Type_handler_hybrid_field_type::aggregate_for_comparison(),
like Item_func_between does.
This patch implements the task according to the description:
1. The old code from Item_func_in::fix_length_and_dec() was decomposed
into smaller methods:
- all_items_are_consts()
- compatible_types_scalar_bisection_possible()
- compatible_types_row_bisection_possible()
- fix_in_vector()
- fix_for_scalar_comparison_using_bisection()
- fix_for_scalar_comparison_using_cmp_items()
- fix_for_row_comparison_using_bisection()
- fix_for_row_comparison_using_cmp_items()
The data type dependend pieces where moved as methods to Type_handler.
2. Splits in_datetime into separate:
- in_datetime, for DATETIME and DATE,
- in_time, for TIME
to make the code more symmetric across data types.
Additionally:
- Adds a test func_debug.test to see which calculation strategy
(bisect or no bisect) is chosen to handle IN with various arguments.
- Adds a new helper method (to avoid duplicate code):
cmp_item_rows::prepare_comparators()
- Changes the propotype for cmp_item_row::alloc_comparators(),
to avoid duplicate code, and to use less current_thd.
- Changes "friend" sections in cmp_item_row and in_row from
an exact Item_func_in method to the entire class Item_func_in,
as their internals are now needed in multiple Item_func_in methods.
- Added more comments (e.g. on bisection, on the problem reported in MDEV-11511)
- Removes "Item_result Item_func_opt_neg::m_compare_type" and introduces
"Type_handler_hybrid_field_type Item_func_opt_neg::m_comparator" instead.
- Removes Item_func_between::compare_as_dates, because
the new member m_comparator now contains the precise information
about the data type that is used for comparison, which is important
for TIME vs DATETIME.
- Adds a new method:
Type_handler_hybrid_field_type::aggregate_for_comparison(const Type_handler*),
as a better replacement for item_cmp_type(), which additionally can handle
TIME vs DATE/DATETIME/TIMESTAMP correctly. Additionally, it correctly
handles TIMESTAMP which fixes the problem reported in MDEV-11482.
The old compare_as_dates/find_date_time_item() based code didn't handle
comparison between TIME and TIMESTAMP correctly and erroneously used TIME
comparison instead of DATETIME comparison.
- Adds a new method:
Type_handler_hybrid_field_type::aggregate_for_comparison(Item **, uint nitems),
as a better replacement for agg_cmp_type(), which can handle TIME.
- Splits Item_func_between::val_int() into pieces val_int_cmp_xxx(),
one new method per XXX_RESULT.
- Adds a new virtual method Type_handler::Item_func_between_val_int()
whose implementations use Item_func_between::val_int_cmp_xxx().
- Makes type_handler_longlong and type_handler_newdecimal public,
as they are now needed in item_cmpfunc.cc.
Note:
This patch does not change Item_func_in to use the new aggregation methods,
so it still uses collect_cmp_type()/item_cmp_type() based aggregation.
Item_func_in will be changed in a separate patch and item_cmp_type() will be
removed.
This patch:
- Adds a new virtual method Type_handler::Item_get_cache
- Splits moves Item_cache::get_cache() into the new method, every
"case XXX_RESULT" to the corresponding Type_handler_xxx::Item_get_cache.
- Adds Item::get_cache as a convenience wrapper, to make the caller code
shorter.
- Changes the last argument of Arg_comparator::cache_converted_constant()
from Item_result to "const Type_handler *".
- Removes subselect_engine::cmp_type, subselect_engine::res_type,
subselect_engine::res_field_type and derives subselect_engine
from Type_handler_hybrid_field_type instead.
- Makes Type_handler_varchar public, as it's now needed as the
default data type handler for subselect_engine.
This patch:
- Introduces a new virtuial method Type_handler::set_comparator_func
and moves pieces of the code from the switch in
Arg_comparator::set_compare_func into the corresponding
Type_handler_xxx::set_comparator_func.
- Adds Type_handler::get_handler_by_cmp_type()
- Moves Type_handler_hybrid_field_type::get_handler_by_result_type() to
a static method Type_handler::get_handler_by_result_type(),
for symmetry with similar methods:
* Type_handler::get_handler_by_field_type()
* Type_handler::get_handler_by_real_type()
* Type_handler::get_handler_by_cmp_type()
- Introduces Type_handler_row, to unify the code for the scalar
data types and the ROW data type (currently for comparison purposes only).
- Adds public type_handler_row, as it's now needed in item_row.h
- Makes type_handler_null public, as it's now needed in item_cmpfunc.h
Note, other type_handler_xxx will become public as well later.
- Removes the global variable Arg_comparator::comparator_matrix,
as it's not needed any more.
The function Item_func_isnull::update_used_tables() must
handle the case when the predicate is over not nullable
column in a special way.
This is actually a bug of MariaDB 5.3/5.5, but it's probably
hard to demonstrate that it can cause problems there.
The class Item_func_nop_all missed an implementation
of the virtual method get_copy.
As a result if the condition that can be pushed into
into a materialized view / derived table contained
an ANY subselect then the pushdown condition was built
incorrectly.
