43b4864a98
Add tests for using tokendata=1 and contentless_delete=1 together.
...
FossilOrigin-Name: a2506b8c9718054912270055638204753c4156bbc115e55194e6df9d7e76cb10
2023-12-04 15:08:21 +00:00
c44041e03b
Ensure that tokendata=1 queries avoid loading large doclists for queries like "common AND uncommon", just as tokendata=0 queries do.
...
FossilOrigin-Name: 7bda09ab404a110d57449e149a3281fca8dc4cacf7bd9832ea2a1356ad20fe8e
2023-12-02 17:32:16 +00:00
f4c2962558
Remove old code for tokendata=1 queries.
...
FossilOrigin-Name: b0a489e8e1bf0290c2117ab32d78b1cc7d67bcb226b55ec044c8367ebde3815b
2023-12-01 20:37:11 +00:00
b5effc0605
Different approach to querying a tokendata=1 table. Saves cpu and memory.
...
FossilOrigin-Name: c523f40895866e6fc979a26483dbea8206126b4bbdf4b73b77263c09e13c855e
2023-12-01 20:09:59 +00:00
0d4a888485
Merge latest trunk updates into this branch.
...
FossilOrigin-Name: 554fc13f2ca5f2ebd9ad0206034c25b556ff40db3106051c5e539f2e142e88ea
2023-11-28 19:43:08 +00:00
226aebae07
Fix an fts5 problem that could occur when mixing regular and secure delete operations on a single table.
...
FossilOrigin-Name: 8eb3f40021994f0d25ef9d246873796c84886e5951bb8ab991e1e2df31998484
2023-11-23 11:18:06 +00:00
5c268bbf67
Fix tokendata=1 and xInstToken() APIs for detail=none and detail=column tables.
...
FossilOrigin-Name: 37b271c19d772bd06524db816ded03377b426efed7a7783c8a96f6fb156ecd86
2023-11-22 19:02:54 +00:00
56bb07bad6
Fix an assert() in fts5 that could be true following an OOM or IO error in contentless-delete mode.
...
FossilOrigin-Name: 3fe89238c31ca163d021ef1fecde594328815aaba7053fef9e9c789562a62f45
2023-11-20 11:40:18 +00:00
a0764f63a8
When querying a tokendata=1 fts5 table, do not use a prefix cursor for the case where the term has only one variant.
...
FossilOrigin-Name: d711c96ba855686d6881a50498418de3492144f005684b5ae55bca24413dce47
2023-11-16 21:11:56 +00:00
50b0e25a55
Add implementation of xInstToken() API.
...
FossilOrigin-Name: a34b26fe7f60b74e7ae5cf64900920a3d352a20da2496401bcbc27041689cd07
2023-11-15 11:45:19 +00:00
e108029332
Add new fts5 API xQueryToken().
...
FossilOrigin-Name: 828566392b3ea8db603cb1ae5eccbc8ac035efaa284bc7c15ba89874f634aec9
2023-11-13 14:29:12 +00:00
b4fc827c0f
Merge latest trunk into this branch.
...
FossilOrigin-Name: 3a869cf1f84b0e9bdcc4de53685430ab41eafacbba1ca7b87e727aa98811c6c5
2023-11-06 19:16:38 +00:00
a3e8822c4b
Fix an fts5 problem where a transaction consisting of (a) a DELETE on rowid X, (b) a prefix query, and (c) an INSERT on rowid X, could corrupt the index.
...
FossilOrigin-Name: c2058a045b57571b2b5d342adb212fe606717c633a0422755691ae6bf5725d25
2023-11-06 18:40:25 +00:00
4cad31912e
Merge trunk changes into this branch.
...
FossilOrigin-Name: ac5570614ebb77feff4943c7a78d877508e31087b550f337c11b45f1016e3c55
2023-11-06 16:32:50 +00:00
ab8a4b2bb0
Fix a case where adjacent tokens are handled incorrectly by the fts5 snippet() function.
...
FossilOrigin-Name: 8f5e9c192ff2820d8cfb076ab28f30697d10c22710583d6c7fd7019c4a0ea795
2023-11-03 17:20:20 +00:00
e186fe20f5
Add the "remove_diacritics" option to the fts5 trigram tokenizer.
...
FossilOrigin-Name: 83da80135b6105f47d1de560232449562ae8ac176c8011a6f75589f62bc9b1db
2023-11-02 17:31:06 +00:00
a238c57eb8
Fix problems with previous commit on Makefile.msc. Also some win32-specific test failures in fts5misc.test.
...
FossilOrigin-Name: 527f2237a6fb5e2ac97afddd7c65c0afa971efc616cb3b738e7245ee0b8d9570
2023-10-30 18:07:18 +00:00
d548f74024
Fix a problem with the fts5 highlight() and snippet() functions when used with tokenizers like "trigram" that output overlapping tokens. Forum post [forum:/forumpost/63735293ec|63735293ec].
...
FossilOrigin-Name: d570aa02f79b1d7d3889e33f9eebab1b7edcf5231b1357451eed9a538607de54
2023-10-24 15:53:02 +00:00
8c86f56e90
New test cases for PRAGMA integrity_check against FTS4 and FTS5 tables.
...
FossilOrigin-Name: 3950d20c3a4433543546ce4495ae07aab3d6cefb163b1f30813a9776c5b962e8
2023-10-23 15:23:10 +00:00
ef6bf1bbe5
Fix a problem allowing a COMMIT following an OOM to cause fts3/4 corruption.
...
FossilOrigin-Name: 7f41d7006db4225cf9b3d197d3a76842778669ac079e76361214a8023c9976e6
2023-10-21 18:12:07 +00:00
71bae9f19f
Fix a problem allowing a COMMIT following an OOM to cause fts5 corruption.
...
FossilOrigin-Name: fba3129d1362dd3ee04d61cb3d3c48518cd98dde3febc85298f411776d60073e
2023-10-20 19:59:12 +00:00
d7cdfe74f6
Fix an fts5 problem caused by a 'rebuild' followed by a DELETE in secure-delete mode.
...
FossilOrigin-Name: 18be505c628d9b13431ca6cfe822d4aeae119c53ae08aef26a67f310a8bd7bd3
2023-10-13 15:59:11 +00:00
3046039309
Fix a problem with an fts5 secure-delete on a rowid/term pair that follows a legacy delete of the same pair.
...
FossilOrigin-Name: 579aea0c28e01a79620ac758edc02db3a01baaa073e7773b8f0b6f610479520b
2023-10-12 19:46:58 +00:00
03204e9106
Add the tokendata=1 option to ignore trailing token-data when querying an fts5 table.
...
FossilOrigin-Name: 122935182ad5869ce3a4c6d796c38a0509f6f3384dd1b3e60a3f2f0f366cc5f5
2023-10-11 21:08:12 +00:00
1846de49a4
Fixes for fts5 expression parser module to allow embedded 0x00 bytes in tokens.
...
FossilOrigin-Name: 342c8d0783f449817d3f565ff6b9f010a6c690beeea32f1861640810490a8b5f
2023-10-03 19:06:52 +00:00
a35ae44150
Changes so that fts5 can handle tokens with embedded '\0' bytes.
...
FossilOrigin-Name: c027c092c4af53bd6ae3cc6e2b4439167d9eeb0f9de549b6a2c2a72a67ee886c
2023-09-30 18:13:35 +00:00
2d3c5385bf
Add extra tests for the fts5 xPhraseQuery API.
...
FossilOrigin-Name: b5edea16317ccc896394011f45442af44d5ea1041b6abfe152c90412c78bfc5b
2023-09-23 15:34:02 +00:00
2599705105
Enable "OR IGNORE" support via SQLITE_VTAB_CONSTRAINT_SUPPORT for internal-content (not contentless or external-content) fts5 tables.
...
FossilOrigin-Name: c362bde4f4b8489947f080154d7fddcfd6e8e21d742a483c496fb7fbe59969d2
2023-09-16 17:11:44 +00:00
12439f9c16
Fix a use-after-free error in fts5 that could occur when querying the "rank" column immediately after another connection changes its definition. [forum:a2dd636330|forum post a2dd636330].
...
FossilOrigin-Name: cb54c2da52d3175844127ead41cfed18793e91ea52e02af2d57c1d17b4312437
2023-09-12 18:36:46 +00:00
1b2d170a8f
Add extra tests for [4b60a1c3].
...
FossilOrigin-Name: 6357491bd1b97f7abddf1a83860f3763cd870033bddc74bd9994a6440fe8c762
2023-09-08 19:18:02 +00:00
410acf6944
Fix a problem with fts5 secure-delete mode causing fts5 to corrupt its records.
...
FossilOrigin-Name: 4b60a1c358c966f161601e749ccb0a7c5f4126e20350f4d6c83b6f35841fbe0a
2023-09-08 18:34:42 +00:00
961c2a9f36
Add the xIntegrity method to the sqlite3_module object. Implement this
...
method in RTREE, FTS3/4, and FTS5 so that "PRAGMA integrity_check" also
verifies the correctness of shadow tables associated with those virtual
tables.
FossilOrigin-Name: 17bede8cdefd968210dd8a5a2617acbe12ba2c99fdd5e88c5def8665e7bec2d7
2023-09-06 12:52:00 +00:00
dd24b1c8a1
Add tests for, and source code comments to, fts5. No changes to code.
...
FossilOrigin-Name: b12afff4efe4de84388d102060e8c312abd2f9eca8270b3c0f01ac6d1f2e329a
2023-09-04 16:48:31 +00:00
1f1169ad09
Updates to testrunner.tcl so that it runs "make fuzztest" using multiple jobs.
...
FossilOrigin-Name: 7596ea7074e0ac73312586ed3d28cdacf97f54b8af73f804cbc8066c94d4b4ef
2023-08-26 21:04:54 +00:00
da7662a4b0
Fix an undefined integer overflow in fts5.
...
FossilOrigin-Name: f5c50349b1e4da233b554f371be49297b8b40ca5fe0188899ed906f682bf70d6
2023-08-21 16:14:58 +00:00
fab6882fd6
Add extra test to fts5synonym2.test.
...
FossilOrigin-Name: 39ea11be3b35c5ecde9bc0fe66af673b8f440dc523422887d05a29feef036b01
2023-08-17 16:49:06 +00:00
8c5611a169
Fix problems in fts5 test code.
...
FossilOrigin-Name: c16ed3b1cc7b4c21bfa96062bf9028ae071c89e63df3b444b59f1753594e4e94
2023-08-15 14:53:49 +00:00
be3ab76bfb
Changes so that sqlite3_vtab_nochange() works with "UPDATE ... FROM...". Use this to allow UPDATE on a contentless fts5 table if new values are supplied for all indexed columns.
...
FossilOrigin-Name: 16cd2161e312cf97129011fc829079db8f762b822b2f4fabf7ff6742c071302f
2023-08-07 16:15:56 +00:00
5326953e57
Add extra test for 'deletemerge'.
...
FossilOrigin-Name: bc33cff4203cef12518e0f43d380a06d53d67c725fb96cfe6e934b7dc97a7efd
2023-07-25 16:48:54 +00:00
3f874b58fb
Change the name of the fts5 'delete-automerge' option to 'deletemerge'. And add tests for it.
...
FossilOrigin-Name: 1079300db2a7d1fbc86a01c215c234a3af64889c5396e6da63ff4f3c7efae4c5
2023-07-25 15:48:58 +00:00
039d494d5f
Further tests for 'delete-automerge'.
...
FossilOrigin-Name: ca26c7a37a7e680be633f43be28f8877bdf9917448ea51c3bedc9b2352a00601
2023-07-25 13:53:42 +00:00
24730de8d1
Add the fts5 'delete-automerge' integer option. A level is eligible for auto-merging if it has a greater than or equal percentage of its entries deleted by tombstones than the 'delete-automerge' option. Default value is 10.
...
FossilOrigin-Name: b314be66b9ac0190b5373b3b6baec012382bc588c2d86c2edab796669a4303c3
2023-07-24 19:13:06 +00:00
2159292ce0
Integrate contentless delete with auto-merge.
...
FossilOrigin-Name: 85c1589ab1fc69d1eef4bbc1bdefa2b10af5f6b9c08e813130b93829b592f416
2023-07-22 19:47:46 +00:00
330e36c2c6
Fix fts5 incremental optimization so that it too can handle an index that consists of a single segment with one or more tombstone hash pages.
...
FossilOrigin-Name: e61c9b083f5e0b6b6ee18f9394581ad816f445dbfb72ed1fe954f4182755a576
2023-07-21 21:10:33 +00:00
4e0c157d02
Ensure the fts5 'optimize' command correctly rewrites any index that consists of a single segment and one or more tombstone hash pages.
...
FossilOrigin-Name: f4926006b371d9a1439a25384bd50a50c2f1c03f75a7c2c3134ae72abb971c91
2023-07-21 19:33:35 +00:00
3a51f8c307
Fix some divide-by-zero errors that could occur when handling corrupt tombstone hash records.
...
FossilOrigin-Name: 7567ca0676f0d45026f5cd4f3fbcd09119c2eaab8ec1711499609c16c452b5e4
2023-07-20 20:29:56 +00:00
d6f5aa824e
Avoid an infinite loop that could be entered when dealing with corrupt fts5 tombstone hash pages.
...
FossilOrigin-Name: 69ce2ce035279f2a00c2238187cf4d2a9092c3410f5900e4613fe4e46311169e
2023-07-20 20:09:26 +00:00
0b3791b537
Add tests and fixes for the new code on this branch.
...
FossilOrigin-Name: 5aac50e92e956b15367c75c20c17bc1c75e84e2752bfffe4ad0a266cb9bd3b8a
2023-07-20 16:07:04 +00:00
d1fbaa071b
Fix various issues with code added to this branch.
...
FossilOrigin-Name: 8d09011fa2c6ae9cc88e1766f9aad4578efbf9e0e311b8c6efdffe7a3f88f923
2023-07-19 18:47:02 +00:00
d05bf0fe61
Fix various problems with fts5 contentless_delete=1 tables.
...
FossilOrigin-Name: 0d005112b8aca9e9eca9d86d5fed9168f6a0218fd290b5489b9e7b05714610f4
2023-07-18 19:52:32 +00:00