44f295ce63
fix index out of bounds for large bvector
...
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com >
2024-02-21 17:38:21 +08:00
c5dc49193e
merge 'main' into 'feat/binary-vector'
...
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com >
2024-02-20 17:02:04 +08:00
d50a75beb5
feat: support binary vector
...
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com >
2024-02-20 16:33:08 +08:00
7b9ae2eef0
refactor: define detailed error types ( #356 )
...
* refactor: define detailed error types
Signed-off-by: usamoi <usamoi@outlook.com >
* do not check version in _vectors_main
Signed-off-by: usamoi <usamoi@outlook.com >
* rename NORMAL to STARTED
Signed-off-by: usamoi <usamoi@outlook.com >
* fix MacOS and FreeBSD implementation
Signed-off-by: usamoi <usamoi@outlook.com >
* fix 512KB memory leak for each session
Signed-off-by: usamoi <usamoi@outlook.com >
* remove skip on delete_threshold for fixing bincode
Signed-off-by: usamoi <usamoi@outlook.com >
* add WorkerOperations and InstanceViewOperations for name conflicts
Signed-off-by: usamoi <usamoi@outlook.com >
---------
Signed-off-by: usamoi <usamoi@outlook.com >
2024-02-19 09:36:16 +00:00
e7e808e835
chore: drop support for pg12, pg13 ( #344 )
...
Signed-off-by: usamoi <usamoi@outlook.com >
2024-02-18 07:38:51 +00:00
d7a490c900
feat: support sparse vector ( #299 )
...
* basic support
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com >
* cargo clippy
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com >
* optimize for ivf
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com >
* patch for merge
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com >
* set vector cast to non-implicit
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com >
* add dimension in sparse vector
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com >
* support creating sparse vector from spilt array and kv string
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com >
* cargo clippy
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com >
* move vector type from Storage trait to G trait
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com >
* export svector function by schema manually
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com >
* spilt index and value for sparse vector
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com >
* update offsets from u32 to usize
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com >
* align SVecf32 to 8 && make SVecf32 support binary
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com >
* cargo clippy
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com >
* convert the computation to branchless
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com >
* refactor
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com >
* cargo fmt
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com >
* update to_svector arguments name
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com >
---------
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com >
2024-02-18 07:16:32 +00:00
c6dabc79ea
chore: consistent slice behavior with python ( #352 )
...
Signed-off-by: usamoi <usamoi@outlook.com >
2024-02-07 08:09:34 +00:00
a4f5d28c10
refactor: refine delete RPC ( #351 )
...
Signed-off-by: usamoi <usamoi@outlook.com >
2024-02-07 02:31:40 +00:00
35701dd2c6
feat: support binary representation ( #314 )
...
* feat: support binary representation
Signed-off-by: usamoi <usamoi@outlook.com >
* fix: check data corruption in send/recv
Signed-off-by: usamoi <usamoi@outlook.com >
---------
Signed-off-by: usamoi <usamoi@outlook.com >
2024-02-05 08:04:06 +00:00
7ec3c88c1e
feat: support vector subscript ( #329 )
...
* feat: support vector subscript
Signed-off-by: usamoi <usamoi@outlook.com >
* test: tests for vector subscript
Signed-off-by: usamoi <usamoi@outlook.com >
* test: pg14 tests
Signed-off-by: usamoi <usamoi@outlook.com >
---------
Signed-off-by: usamoi <usamoi@outlook.com >
2024-02-02 08:52:15 +00:00
c975c74314
chore: set storage = EXTERNAL by default for vector types ( #321 )
...
* chore: set storage = EXTERNAL by default for vector types
Signed-off-by: usamoi <usamoi@outlook.com >
* test: test setting storage by users
Signed-off-by: usamoi <usamoi@outlook.com >
---------
Signed-off-by: usamoi <usamoi@outlook.com >
2024-02-02 07:22:06 +00:00
b22cd7887e
refactor: faster algorithm for parsing vector strings ( #316 )
...
Signed-off-by: usamoi <usamoi@outlook.com >
2024-02-02 07:21:54 +00:00
5b2626691e
chore: nicer cast error message ( #322 )
...
Signed-off-by: usamoi <usamoi@outlook.com >
2024-01-30 09:34:17 +00:00
bb80544407
fix: add schema for referring vector_index_stat ( #319 )
...
* fix: add schema for referring vector_index_stat
Signed-off-by: usamoi <usamoi@outlook.com >
* test: add test for out-of-search_path usage
Signed-off-by: usamoi <usamoi@outlook.com >
---------
Signed-off-by: usamoi <usamoi@outlook.com >
2024-01-30 07:57:28 +00:00
fc93a64b75
feat: support pgvector gucs compact ( #296 )
...
* feat: support pgvector gucs compact
Signed-off-by: cutecutecat <junyuchen@tensorchord.ai >
* Update src/gucs/executing.rs
Signed-off-by: usamoi <usamoi@outlook.com >
---------
Signed-off-by: cutecutecat <junyuchen@tensorchord.ai >
Signed-off-by: usamoi <usamoi@outlook.com >
Co-authored-by: Usamoi <usamoi@outlook.com >
2024-01-22 10:06:38 +00:00
32fcf4ec3d
feat: add compat with pgvector ( #256 )
...
Signed-off-by: cutecutecat <junyuchen@tensorchord.ai >
2024-01-19 04:33:51 +00:00
b12962fc7c
chore: Add test for dropping index when indexing ( #283 )
...
Signed-off-by: xieydd <xieydd@gmail.com >
2024-01-18 13:01:00 +00:00
727d69feab
feat: transactional CREATE/DROP ( #279 )
...
* feat: transactional CREATE/DROP
Signed-off-by: usamoi <usamoi@outlook.com >
* test: transaction ddl
Signed-off-by: usamoi <usamoi@outlook.com >
---------
Signed-off-by: usamoi <usamoi@outlook.com >
2024-01-17 11:47:02 +00:00
30a0f7043b
chore: add test for partition/partial/fp16 index ( #276 )
...
Signed-off-by: cutecutecat <junyuchen@tensorchord.ai >
2024-01-16 06:00:53 +00:00
e1fde86249
feat: Add vector to vecf16 CAST ( #266 )
...
Signed-off-by: jepassanante <j.e.passanante@gmail.com >
2024-01-14 04:26:59 +00:00
4bdf192412
refactor: orthogonal support of vbase and prefilter ( #248 )
...
Signed-off-by: usamoi <usamoi@outlook.com >
2024-01-10 12:22:19 +00:00
98622ed077
chore: add crash test ( #242 )
...
* chore: add crash test
Signed-off-by: cutecutecat <junyuchen@tensorchord.ai >
* fix
Signed-off-by: cutecutecat <junyuchen@tensorchord.ai >
---------
Signed-off-by: cutecutecat <junyuchen@tensorchord.ai >
2024-01-09 08:13:13 +00:00
eb44c263b6
chore: fix typos ( #228 )
...
Signed-off-by: Keming <kemingyang@tensorchord.ai >
2024-01-04 06:24:45 +00:00
cc1b7d0f17
feat: deny unknown field in options ( #184 )
...
* fix: deny unknown options
Signed-off-by: usamoi <usamoi@outlook.com >
* test: deny unknown options
Signed-off-by: usamoi <usamoi@outlook.com >
---------
Signed-off-by: usamoi <usamoi@outlook.com >
2023-12-15 11:02:04 +00:00
2869fbd44c
fix: cast test ( #182 )
...
Signed-off-by: usamoi <usamoi@outlook.com >
2023-12-14 10:52:49 +00:00
5c0450274d
feat: fp16 vector ( #178 )
...
* feat: fp16 vector
Signed-off-by: usamoi <usamoi@outlook.com >
* feat: detect avx512fp16
Signed-off-by: usamoi <usamoi@outlook.com >
* fix: install clang-16 for ci
Signed-off-by: usamoi <usamoi@outlook.com >
* fix: clippy
Signed-off-by: usamoi <usamoi@outlook.com >
* fix: rename c to pgvectorsc
Signed-off-by: usamoi <usamoi@outlook.com >
* feat: hand-writing avx512fp16
Signed-off-by: usamoi <usamoi@outlook.com >
* fix: index on fp16
Signed-off-by: usamoi <usamoi@outlook.com >
* feat: hand-writing avx2
Signed-off-by: usamoi <usamoi@outlook.com >
* fix: clippy
Signed-off-by: usamoi <usamoi@outlook.com >
* fix: add rerun in build script
Signed-off-by: usamoi <usamoi@outlook.com >
* fix: cross compilation
Signed-off-by: usamoi <usamoi@outlook.com >
* fix: do not leave uninitialized bytes in datatype input function
Signed-off-by: usamoi <usamoi@outlook.com >
* fix: compiler built-in function calling convention workaround
Signed-off-by: usamoi <usamoi@outlook.com >
* fix: cross compile on aarch64
Signed-off-by: usamoi <usamoi@outlook.com >
* fix: fix detect avx512fp16
Signed-off-by: usamoi <usamoi@outlook.com >
* fix: avx512 codegen by multiversion
Signed-off-by: usamoi <usamoi@outlook.com >
* fix: enable more target features for c
Signed-off-by: usamoi <usamoi@outlook.com >
* fix: use tensorchord/stdarch
Signed-off-by: usamoi <usamoi@outlook.com >
* fix: ci
Signed-off-by: usamoi <usamoi@outlook.com >
* fix: remove no-run cross test
Signed-off-by: usamoi <usamoi@outlook.com >
* fix: vbase
Signed-off-by: usamoi <usamoi@outlook.com >
* fix: error and document
Signed-off-by: usamoi <usamoi@outlook.com >
* [skip ci]
Signed-off-by: usamoi <usamoi@outlook.com >
---------
Signed-off-by: usamoi <usamoi@outlook.com >
2023-12-14 09:50:52 +00:00
2ab76118fc
feat: vbase optimization ( #177 )
...
* feat: vbase optimization
Signed-off-by: silver-ymz <yinmingzhuo@gmail.com >
* cargo fmt && cargo clippy
Signed-off-by: silver-ymz <yinmingzhuo@gmail.com >
* fix
Signed-off-by: silver-ymz <yinmingzhuo@gmail.com >
* make the second progress generate by using heap
Signed-off-by: silver-ymz <yinmingzhuo@gmail.com >
* move filter out of algorrithm
Signed-off-by: silver-ymz <yinmingzhuo@gmail.com >
* cargo fmt
Signed-off-by: silver-ymz <yinmingzhuo@gmail.com >
* return all points in the first stage
Signed-off-by: silver-ymz <yinmingzhuo@gmail.com >
* change heap to a vec to be sorted
Signed-off-by: silver-ymz <yinmingzhuo@gmail.com >
* fix
Signed-off-by: silver-ymz <yinmingzhuo@gmail.com >
---------
Signed-off-by: silver-ymz <yinmingzhuo@gmail.com >
2023-12-11 03:01:27 +00:00
189fe96b1d
fix: reindex failed due to existing old index ( #169 )
...
* fix: reindex failed due to existing old index
Signed-off-by: silver-ymz <yinmingzhuo@gmail.com >
* fix: destory index according to smgrGetPendingDeletes list
Signed-off-by: silver-ymz <yinmingzhuo@gmail.com >
* update pgrx repo url
Signed-off-by: silver-ymz <yinmingzhuo@gmail.com >
* remove incorrect panic
Signed-off-by: silver-ymz <yinmingzhuo@gmail.com >
* typo
Signed-off-by: silver-ymz <yinmingzhuo@gmail.com >
* replace rpc id from indexoid to relfileid
Signed-off-by: silver-ymz <yinmingzhuo@gmail.com >
* remove optimize_handle
Signed-off-by: silver-ymz <yinmingzhuo@gmail.com >
---------
Signed-off-by: silver-ymz <yinmingzhuo@gmail.com >
2023-11-28 13:23:00 +00:00
39e8ee9797
feat: mmap transport for macos ( #137 )
...
* feat: mmap transport for macos
Signed-off-by: usamoi <usamoi@outlook.com >
* fix: building with feature pg12, pg13
Signed-off-by: usamoi <usamoi@outlook.com >
* fix: call unlink for shmem
Signed-off-by: usamoi <usamoi@outlook.com >
* fix: reduce shmem filename on macos
Signed-off-by: usamoi <usamoi@outlook.com >
* chore: enable testing on all Postgresql versions
Signed-off-by: usamoi <usamoi@outlook.com >
* fix: use file instead of shmem for macos
Signed-off-by: usamoi <usamoi@outlook.com >
* chore: select simpler matrix for pull requests in CI
Signed-off-by: usamoi <usamoi@outlook.com >
* fix: remove macos-latest-16
Signed-off-by: usamoi <usamoi@outlook.com >
* chore: reduce use of nightly features
Signed-off-by: usamoi <usamoi@outlook.com >
---------
Signed-off-by: usamoi <usamoi@outlook.com >
2023-11-17 16:26:15 +00:00
7b85fe4030
feat: mmap for ipc ( #133 )
...
* feat: use mmap for session-bgworker commutating
Signed-off-by: usamoi <usamoi@outlook.com >
* use fcntl_lock to replace flock
Signed-off-by: usamoi <usamoi@outlook.com >
* chore: improve error messages
Signed-off-by: usamoi <usamoi@outlook.com >
* fix: fix test
Signed-off-by: usamoi <usamoi@outlook.com >
* chore: add nice message for IPC error
Signed-off-by: usamoi <usamoi@outlook.com >
---------
Signed-off-by: usamoi <usamoi@outlook.com >
2023-11-13 16:42:39 +00:00
4bc6e114c8
feat: indexing on segments ( #117 )
...
* feat: indexing on segments
Signed-off-by: usamoi <usamoi@outlook.com >
* fix: add vamana back
Signed-off-by: usamoi <usamoi@outlook.com >
* test: update tests for error message
Signed-off-by: usamoi <usamoi@outlook.com >
---------
Signed-off-by: usamoi <usamoi@outlook.com >
2023-11-08 16:13:53 +00:00
889fac48cd
feat: support concurrent index building ( #101 )
...
Signed-off-by: silver-ymz <yinmingzhuo@gmail.com >
2023-10-24 10:14:20 +00:00
34bca2a408
test: add e2e test ( #91 )
...
* test: add e2e test
Signed-off-by: silver-ymz <yinmingzhuo@gmail.com >
* add github action
Signed-off-by: silver-ymz <yinmingzhuo@gmail.com >
---------
Signed-off-by: silver-ymz <yinmingzhuo@gmail.com >
2023-10-15 19:48:28 -07:00
