mirror of
https://github.com/facebook/rocksdb.git
synced 2026-10-23 08:09:02 +03:00
3b38fde293
Summary: Add `use_udi_as_primary_index` option to `BlockBasedTableOptions`. When enabled, the UDI becomes the primary index — all reads (including internal operations like compaction and VerifyChecksum) automatically route through the UDI without needing `ReadOptions::table_index_factory`. Both the standard binary search index and the UDI are always fully built. The standard index serves as a safety fallback (e.g., for backup/restore or rollback to a non-UDI configuration). A future refactor will extract the index abstraction to allow skipping the standard index build when the UDI is primary (see discussion below). ## Write path - `UserDefinedIndexBuilderWrapper` always forwards `AddIndexEntry` and `OnKeyAdded` to both the internal standard builder and the UDI builder - New `udi_is_primary_index` table property marks primary-mode SSTs - Validates incompatible options at `DB::Open` and builder creation: partitioned index, partitioned filters, missing `user_defined_index_factory` ## Read path - `UserDefinedIndexReaderWrapper` defaults to UDI when `udi_is_primary_`, even when `ReadOptions::table_index_factory` is null — this handles the 15+ internal call sites that don't set `table_index_factory` - `use_udi_as_primary_index` automatically enforces `fail_if_no_udi_on_open` to prevent silent data loss if SSTs are opened without UDI support ## Rollback Since the standard index is always fully populated, rollback from primary mode is straightforward: set `use_udi_as_primary_index=false`. No compaction required — SSTs written in primary mode are immediately readable through the standard index. ## Public API - `BlockBasedTableOptions::use_udi_as_primary_index` (default: false) - `UserDefinedIndexBuilder::EstimatedSize()` — pure virtual, O(1) via running counter in the trie implementation ## Bug fixes (issues https://github.com/facebook/rocksdb/issues/14560, https://github.com/facebook/rocksdb/issues/14561, https://github.com/facebook/rocksdb/issues/14562) Fixed trie index correctness bugs that caused crash test failures: - **Always-on seqno encoding**: `must_use_separator_with_seq_` is now unconditionally true. Non-boundary separators store tag=0 (sentinel), same-user-key boundaries store the real tag, and the last block stores its real last-key tag. This fixes the `NonBoundaryTag` bug where non-boundary separators with `kMaxSequenceNumber` caused the post-seek correction to incorrectly advance past the correct block. - **Standard index always built in primary mode**: An empty (stub) standard index block caused behavioral divergence in `BlockBasedTableIterator` under concurrent flush/compaction, leading to `test_batches_snapshots` prefix scan inconsistencies. ## Stress test - `use_udi_as_primary_index` flag randomized by `db_crashtest.py` - Both primary and secondary UDI modes exercised in crash tests - Trie index probability halved (~6%) per reviewer request - Re-enables trie crash tests (disabled by https://github.com/facebook/rocksdb/issues/14559) ## Tests - Parameterized `TrieIndexDBTest` on UDI mode (secondary vs primary) - New factory-level tests: non-boundary separator seek, Prev within overflow, SeekToLast with overflow, empty trie, overflow exhaustion, all-scans-exhausted - New DB-level tests: multi-CF coalescing iterator, GetEntity with explicit snapshot, reverse iteration across same-user-key blocks, non-boundary separator seek correctness, rollback from primary Pull Request resolved: https://github.com/facebook/rocksdb/pull/14547 Reviewed By: anand1976 Differential Revision: D99494181 Pulled By: xingbowang fbshipit-source-id: ca52a0d5c0e523770c80e1fe2b9b5d50406b67bc