This allows
create table t1 (v vector(64) not null);
insert into t1 select vec_fromtext(concat('[',group_concat(1),']')) from seq_1_to_64;
which was banned in the previous fix
bb0ac43701, though this also introduces
the inconsistency(?) where
create table t1 as select vec_fromtext(concat('[',group_concat(1),']')) from seq_1_to_64;
still fails ER_TRUNCATED_WRONG_VALUE
see updated tests
TODO:
- Changes in vector_utf16.result does not look right
- the following tests crashes:
CREATE TABLE t1 (v VECTOR(2));
--error ER_TOO_BIG_FIELDLENGTH
INSERT INTO t1 VALUES (VEC_FROMTEXT(CONCAT('[1.', REPEAT('0',70000), ',2]')));
DROP TABLE t1;
SELECT VEC_FROMTEXT('😀😀😀');
The changes of MDEV-39558 2b6529426a
added length check assertion in Field_varstring constructors, and
length check in type inference for SELECT set operations, to emit
errors before reaching the assertions.
That change caused an error to turn into an assertion failure in a
separate path, when the length limit violation is not detected before
tripping the assertion. So in this patch we fix it by adding an
earlier length check in that path.
The reason that we place this check inside
Item_func_vec_fromtext::fix_length_and_dec rather than say
`create_field_for_create_select is for consistency:
If
create table t1 as select
vec_fromtext(concat('[',group_concat(1),']')) as c1 from seq_1_to_64;
fails due to length limit violation, then so should
create table t1 (v vector(64) not null);
insert into t1 select vec_fromtext(concat('[',group_concat(1),']'))
from seq_1_to_64;
Also use max_char_length() instead of max_length. This is a more
accurate length of characters. And add handling of empty string edge
case. Added testcases accordingly.
The change that uses max_char_length() causes side effects where
creating a table using a VEC_FROMTEXT(CHAR(1)) would result in a
0-dimensional vector field. This is accurate but 0-dim vector table
fields should not be allowed. So we make cases like this result in
a one-dimensional field.
Also fixed the underflow in (args[0]->max_length - 1) * 2 when the
arg's max length is 0. Previously this underflow would cause
create table t1 select vec_fromtext(NULL)
to fail with ER_TOO_BIG_FIELDLENGTH. Now it will be a VECTOR(1) field
The initial hard capped limit on the depth was 32. It was implemented using
static arrays of relevant type and size 32.
Hence, to implement unlimited depth, dynamic array on mem_root was
implemented which grows by 3200 as needed. Relevant arrays were replaced
with this dynamic array.
* Let GCC `-Wformat` check formats sent to these `my_vsnprintf_ex` users
* Migrate them from the old extension specifiers
to the new `-Wformat`-compatible suffixes
switch to a more predictable, shorter, and more correct output
that is, print as many significant digits as necessary.
but not more (they'd be just zeros) and not less (it'd lose precision)
This commit introduces two utility functions meant to make working with
vectors simpler.
Vec_ToText converts a binary vector into a json array of numbers
(floats).
Vec_FromText takes in a json array of numbers and converts it into a
little-endian IEEE float sequence of bytes (4 bytes per float).
This commit includes the work done in collaboration with Hugo Wen from
Amazon:
MDEV-33408 Alter HNSW graph storage and fix memory leak
This commit changes the way HNSW graph information is stored in the
second table. Instead of storing connections as separate records, it now
stores neighbors for each node, leading to significant performance
improvements and storage savings.
Comparing with the previous approach, the insert speed is 5 times faster,
search speed improves by 23%, and storage usage is reduced by 73%, based
on ann-benchmark tests with random-xs-20-euclidean and
random-s-100-euclidean datasets.
Additionally, in previous code, vector objects were not released after
use, resulting in excessive memory consumption (over 20GB for building
the index with 90,000 records), preventing tests with large datasets.
Now ensure that vectors are released appropriately during the insert and
search functions. Note there are still some vectors that need to be
cleaned up after search query completion. Needs to be addressed in a
future commit.
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.
As well as the commit:
Introduce session variables to manage HNSW index parameters
Three variables:
hnsw_max_connection_per_layer
hnsw_ef_constructor
hnsw_ef_search
ann-benchmark tool is also updated to support these variables in commit
https://github.com/HugoWenTD/ann-benchmarks/commit/e09784e for branch
https://github.com/HugoWenTD/ann-benchmarks/tree/mariadb-configurable
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.
Co-authored-by: Hugo Wen <wenhug@amazon.com>
MDEV-33407 Parser support for vector indexes
The syntax is
create table t1 (... vector index (v) ...);
limitation:
* v is a binary string and NOT NULL
* only one vector index per table
* temporary tables are not supported
MDEV-33404 Engine-independent indexes: subtable method
added support for so-called "high level indexes", they are not visible
to the storage engine, implemented on the sql level. For every such
an index in a table, say, t1, the server implicitly creates a second
table named, like, t1#i#05 (where "05" is the index number in t1).
This table has a fixed structure, no frm, not accessible directly,
doesn't go into the table cache, needs no MDLs.
MDEV-33406 basic optimizer support for k-NN searches
for a query like SELECT ... ORDER BY func() optimizer will use
item_func->part_of_sortkey() to decide what keys can be used
to resolve ORDER BY.
