Mingzhuo Yin
44f295ce63
fix index out of bounds for large bvector
...
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com>
2024-02-21 17:38:21 +08:00
Mingzhuo Yin
16c8f2c127
fix invalid cmp
...
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com>
2024-02-21 10:19:51 +08:00
Mingzhuo Yin
1bd49d76b6
cargo fmt
...
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com>
2024-02-21 01:14:45 +08:00
Mingzhuo Yin
90e9a91ce8
cargo clippy
...
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com>
2024-02-21 01:11:30 +08:00
Mingzhuo Yin
3dae8dd9e6
remove unused feature
...
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com>
2024-02-21 01:00:29 +08:00
Mingzhuo Yin
50824d64e5
remove crate bitvec
...
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com>
2024-02-21 00:59:06 +08:00
Mingzhuo Yin
20f13e25fd
add simd for binary vector
...
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com>
2024-02-21 00:01:16 +08:00
Mingzhuo Yin
c5dc49193e
merge 'main' into 'feat/binary-vector'
...
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com>
2024-02-20 17:02:04 +08:00
Mingzhuo Yin
d50a75beb5
feat: support binary vector
...
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com>
2024-02-20 16:33:08 +08:00
usamoi
42fa5835d1
refactor: add crate "base" ( #367 )
...
Signed-off-by: usamoi <usamoi@outlook.com>
2024-02-20 07:35:49 +00:00
Usamoi
7b9ae2eef0
refactor: define detailed error types ( #356 )
...
* refactor: define detailed error types
Signed-off-by: usamoi <usamoi@outlook.com>
* do not check version in _vectors_main
Signed-off-by: usamoi <usamoi@outlook.com>
* rename NORMAL to STARTED
Signed-off-by: usamoi <usamoi@outlook.com>
* fix MacOS and FreeBSD implementation
Signed-off-by: usamoi <usamoi@outlook.com>
* fix 512KB memory leak for each session
Signed-off-by: usamoi <usamoi@outlook.com>
* remove skip on delete_threshold for fixing bincode
Signed-off-by: usamoi <usamoi@outlook.com>
* add WorkerOperations and InstanceViewOperations for name conflicts
Signed-off-by: usamoi <usamoi@outlook.com>
---------
Signed-off-by: usamoi <usamoi@outlook.com>
2024-02-19 09:36:16 +00:00
Mingzhuo Yin
915f956ff0
fix comparsion of svector ( #357 )
...
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com>
2024-02-18 09:42:24 +00:00
Mingzhuo Yin
d7a490c900
feat: support sparse vector ( #299 )
...
* basic support
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com>
* cargo clippy
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com>
* optimize for ivf
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com>
* patch for merge
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com>
* set vector cast to non-implicit
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com>
* add dimension in sparse vector
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com>
* support creating sparse vector from spilt array and kv string
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com>
* cargo clippy
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com>
* move vector type from Storage trait to G trait
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com>
* export svector function by schema manually
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com>
* spilt index and value for sparse vector
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com>
* update offsets from u32 to usize
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com>
* align SVecf32 to 8 && make SVecf32 support binary
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com>
* cargo clippy
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com>
* convert the computation to branchless
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com>
* refactor
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com>
* cargo fmt
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com>
* update to_svector arguments name
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com>
---------
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com>
2024-02-18 07:16:32 +00:00
Usamoi
a4f5d28c10
refactor: refine delete RPC ( #351 )
...
Signed-off-by: usamoi <usamoi@outlook.com>
2024-02-07 02:31:40 +00:00
Usamoi
4eb1f87ae6
fix: remove ctor ( #339 )
...
Signed-off-by: usamoi <usamoi@outlook.com>
2024-02-02 10:37:07 +00:00
Usamoi
f26ffba75d
refactor: replace heap of heaps with loser tree in merging results ( #315 )
...
Signed-off-by: usamoi <usamoi@outlook.com>
2024-02-02 07:21:45 +00:00
Usamoi
3b06670b9d
fix: check if prefilter enabled ( #302 )
...
Signed-off-by: Usamoi <usamoi@outlook.com>
Co-authored-by: cutecutecat <junyuchen@tensorchord.ai>
2024-01-23 13:58:43 +00:00
Usamoi
132b304811
fix: checking of user options ( #297 )
...
Signed-off-by: usamoi <usamoi@outlook.com>
2024-01-22 10:02:48 +00:00
cutecutecat
c95f99f846
fix: prevent overflow at kmeans ( #287 )
...
* fix: prevent overflow at kmeans
Signed-off-by: cutecutecat <junyuchen@tensorchord.ai>
* fix
Signed-off-by: cutecutecat <junyuchen@tensorchord.ai>
---------
Signed-off-by: cutecutecat <junyuchen@tensorchord.ai>
2024-01-22 03:24:01 +00:00
Mingzhuo Yin
3c3ec7119e
fix: correct fp16 dot operation ( #291 )
...
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com>
2024-01-20 06:28:33 +00:00
Usamoi
2b95a23325
fix: promise flushing when committing ( #280 )
...
Signed-off-by: usamoi <usamoi@outlook.com>
2024-01-17 12:37:07 +00:00
Usamoi
727d69feab
feat: transactional CREATE/DROP ( #279 )
...
* feat: transactional CREATE/DROP
Signed-off-by: usamoi <usamoi@outlook.com>
* test: transaction ddl
Signed-off-by: usamoi <usamoi@outlook.com>
---------
Signed-off-by: usamoi <usamoi@outlook.com>
2024-01-17 11:47:02 +00:00
Usamoi
4bdf192412
refactor: orthogonal support of vbase and prefilter ( #248 )
...
Signed-off-by: usamoi <usamoi@outlook.com>
2024-01-10 12:22:19 +00:00
Usamoi
cbd42caa89
refactor: rework vbase ( #238 )
...
* refactor: rework vbase abstraction and remove HNSW vbase implementation
Signed-off-by: usamoi <usamoi@outlook.com>
* refactor: rework HNSW vbase implementation
Signed-off-by: usamoi <usamoi@outlook.com>
---------
Signed-off-by: usamoi <usamoi@outlook.com>
2024-01-08 02:25:58 +00:00
Keming
4aafd8c312
fix: cosine distance ( #231 )
...
* fix: cosine distance
Signed-off-by: Keming <kemingyang@tensorchord.ai>
* fix sq and pq
Signed-off-by: Keming <kemingyang@tensorchord.ai>
* fix doc
Signed-off-by: Keming <kemingyang@tensorchord.ai>
---------
Signed-off-by: Keming <kemingyang@tensorchord.ai>
2024-01-06 14:23:21 +00:00
Ce Gao
a7a9a592c1
chore: Refine readme ( #234 )
...
Signed-off-by: Ce Gao <cegao@tensorchord.ai>
2024-01-05 12:27:39 +00:00
Usamoi
1411fefccf
docs: development docs ( #226 )
...
Signed-off-by: usamoi <usamoi@outlook.com>
2024-01-05 08:45:30 +00:00
Keming
eb44c263b6
chore: fix typos ( #228 )
...
Signed-off-by: Keming <kemingyang@tensorchord.ai>
2024-01-04 06:24:45 +00:00
Usamoi
d0349f1aa8
chore: fine-grained upgrade hint ( #220 )
...
* chore: upgrade instruction for every index
Signed-off-by: usamoi <usamoi@outlook.com>
* fix: soft_version check
Signed-off-by: usamoi <usamoi@outlook.com>
* fix: index_stat view if need upgrade
Signed-off-by: usamoi <usamoi@outlook.com>
* fix: size info of write segment
Signed-off-by: usamoi <usamoi@outlook.com>
---------
Signed-off-by: usamoi <usamoi@outlook.com>
2024-01-03 09:19:11 +00:00
Keming
5669253a98
feat: show index size in index info table ( #222 )
...
* feat: show index size in index info table
Signed-off-by: Keming <kemingyang@tensorchord.ai>
* fix clippy
Signed-off-by: Keming <kemingyang@tensorchord.ai>
* compute all the file for sealed segments
Signed-off-by: Keming <kemingyang@tensorchord.ai>
* fix dir_size
Signed-off-by: Keming <kemingyang@tensorchord.ai>
* display the sum size in the table
Signed-off-by: Keming <kemingyang@tensorchord.ai>
---------
Signed-off-by: Keming <kemingyang@tensorchord.ai>
2024-01-03 07:39:58 +00:00
cutecutecat
e530742f0f
feat: add nprobe into guc ( #212 )
...
Signed-off-by: cutecutecat <junyuchen@tensorchord.ai>
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com>
2023-12-29 07:14:04 +00:00
Usamoi
2dc6b721f3
chore: move memfd detect to detect crate ( #213 )
...
Signed-off-by: usamoi <usamoi@outlook.com>
2023-12-28 11:14:55 +00:00
Usamoi
4d34b45b23
fix: upgrade instructions ( #203 )
...
* fix: upgrade instructions
Signed-off-by: usamoi <usamoi@outlook.com>
* fix: rename scripts to install
Signed-off-by: usamoi <usamoi@outlook.com>
* fix: ipc error-handling
Signed-off-by: usamoi <usamoi@outlook.com>
* feat: soft version
Signed-off-by: usamoi <usamoi@outlook.com>
* chore: freebsd
Signed-off-by: usamoi <usamoi@outlook.com>
---------
Signed-off-by: usamoi <usamoi@outlook.com>
2023-12-28 08:16:22 +00:00
Mingzhuo Yin
4f60c4c824
feat: allow user to set vbase range ( #206 )
...
* feat: allow user to set vbase range
Signed-off-by: silver-ymz <yinmingzhuo@gmail.com>
* separate the enable_vbase and vbase_range settings
Signed-off-by: silver-ymz <yinmingzhuo@gmail.com>
---------
Signed-off-by: silver-ymz <yinmingzhuo@gmail.com>
2023-12-27 06:32:23 +00:00
Usamoi
cc1b7d0f17
feat: deny unknown field in options ( #184 )
...
* fix: deny unknown options
Signed-off-by: usamoi <usamoi@outlook.com>
* test: deny unknown options
Signed-off-by: usamoi <usamoi@outlook.com>
---------
Signed-off-by: usamoi <usamoi@outlook.com>
2023-12-15 11:02:04 +00:00
Usamoi
c50912e87d
C code tests & avx512f f16 implement ( #183 )
...
* test: add tests for c code
Signed-off-by: usamoi <usamoi@outlook.com>
* fix: relax EPSILON for tests
Signed-off-by: usamoi <usamoi@outlook.com>
---------
Signed-off-by: usamoi <usamoi@outlook.com>
2023-12-15 08:00:08 +00:00
Usamoi
5c0450274d
feat: fp16 vector ( #178 )
...
* feat: fp16 vector
Signed-off-by: usamoi <usamoi@outlook.com>
* feat: detect avx512fp16
Signed-off-by: usamoi <usamoi@outlook.com>
* fix: install clang-16 for ci
Signed-off-by: usamoi <usamoi@outlook.com>
* fix: clippy
Signed-off-by: usamoi <usamoi@outlook.com>
* fix: rename c to pgvectorsc
Signed-off-by: usamoi <usamoi@outlook.com>
* feat: hand-writing avx512fp16
Signed-off-by: usamoi <usamoi@outlook.com>
* fix: index on fp16
Signed-off-by: usamoi <usamoi@outlook.com>
* feat: hand-writing avx2
Signed-off-by: usamoi <usamoi@outlook.com>
* fix: clippy
Signed-off-by: usamoi <usamoi@outlook.com>
* fix: add rerun in build script
Signed-off-by: usamoi <usamoi@outlook.com>
* fix: cross compilation
Signed-off-by: usamoi <usamoi@outlook.com>
* fix: do not leave uninitialized bytes in datatype input function
Signed-off-by: usamoi <usamoi@outlook.com>
* fix: compiler built-in function calling convention workaround
Signed-off-by: usamoi <usamoi@outlook.com>
* fix: cross compile on aarch64
Signed-off-by: usamoi <usamoi@outlook.com>
* fix: fix detect avx512fp16
Signed-off-by: usamoi <usamoi@outlook.com>
* fix: avx512 codegen by multiversion
Signed-off-by: usamoi <usamoi@outlook.com>
* fix: enable more target features for c
Signed-off-by: usamoi <usamoi@outlook.com>
* fix: use tensorchord/stdarch
Signed-off-by: usamoi <usamoi@outlook.com>
* fix: ci
Signed-off-by: usamoi <usamoi@outlook.com>
* fix: remove no-run cross test
Signed-off-by: usamoi <usamoi@outlook.com>
* fix: vbase
Signed-off-by: usamoi <usamoi@outlook.com>
* fix: error and document
Signed-off-by: usamoi <usamoi@outlook.com>
* [skip ci]
Signed-off-by: usamoi <usamoi@outlook.com>
---------
Signed-off-by: usamoi <usamoi@outlook.com>
2023-12-14 09:50:52 +00:00
