499 Commits

Author	SHA1	Message	Date
Yoshinori Matsunobu	d6a7847d37	Add a DB option to log long-held global mutexes and release sites (#15436) ... Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15436 ## Why Long-held global mutexes can stall foreground and background work. Existing mutex statistics measure acquisition waits, not continuous ownership. This adds opt-in logging to identify the mutex, owning thread, operation and release site. ## Enable for an investigation Provide an INFO logger and shared `Statistics` object when opening the DB, then: - Set the shared statistics level to `StatsLevel::kAll`. - Set `mutex_hold_log_threshold_us` through `DB::SetDBOptions`; `100000` means 100 ms. - Restore threshold 0 and the previous statistics level afterward. The threshold defaults to **0 (disabled)**. `kAll` alone does not enable these hold probes; threshold 0 leaves existing statistics timers unchanged. ## Output Illustrative records, not production observations: ``` SLOW_GLOBAL_MUTEX mutex=DBImpl::mutex_ mutex_addr=0x12340000 owner_thread_id=12345 thread_operation=Compaction hold_us=200000 threshold_us=100000 observed_end_us=1792205400000000 buffer_dropped_records=0 unlock=db_impl_compaction_flush.cc:4660 BackgroundCallCompaction() SLOW_GLOBAL_MUTEX mutex=DBImpl::wal_write_mutex_ mutex_addr=0x12340100 owner_thread_id=12346 thread_operation= hold_us=35000 threshold_us=25000 observed_end_us=1792205401000000 buffer_dropped_records=0 scope_exit=db_impl_write.cc:2250 PreprocessWrite() ``` The first proves a completed 200 ms DB-mutex ownership interval on a thread tagged `Compaction`. The second uses a 25 ms threshold; an empty operation tag means none was available. Neither record reports acquisition wait or proves the cause of a stall. Release attribution uses compiler-embedded strings available in optimized, stripped builds, with a GCC/Clang fallback where standard source locations are unavailable. Direct unlock and condition wait identify their call sites; scope exit and temporary unlock identify guard declarations, not closing braces. No stack unwinding or symbol lookup. ## Scope and safety - Track six selected mutexes: DB, WAL, point-lock wait graph, two prepare-log bookkeeping mutexes and transaction name map. This is not comprehensive synchronization tracing. - Acquisition waits and condition-variable release gaps are excluded from hold duration. Configuration is pre-first-use; repeated setup retains the first valid context. - Sample at most once per mutex per second, with eight deferred records per thread. `buffer_dropped_records` counts capacity overflow only; rate limiting, pause and emission contention can omit samples. - Emit after covered outer lock scopes release. Emission is nonblocking best-effort; shutdown drains/pauses emission and stops diagnostics before the owned logger closes. Closing from that logger's own callback is unsupported. - No data-file format change. The mutable option is persisted in OPTIONS and wired into bench/stress tools. Typed C binding is deferred; string DB-option APIs are available. Disabled bookkeeping still has a cost. Release A/B measurements found about 3 ns added per selected mutex pair at level 1, and about 70 ns for hold timing over `kAll`/0. Get and reused-iterator Seek showed no resolved small probe effect; fresh iterator+Seek at 16 threads showed +8.7% CPU/query and -9.4% QPS. No logs were emitted in those benchmarks; logging cost and broader workload qualification remain open. Reviewed By: xingbowang Differential Revision: D125790955 fbshipit-source-id: 1096163f3bd25ffee585dc6ec8ebfd7f1759851a	2026-10-22 09:50:20 -07:00
zaidoon	553b52b4a7	Wire IndexFactory through block-based tables (#14963) ... Summary: Build block-based table indexes through `IndexFactory`, with modes for writing both indexes or only the custom index and a per-read index selector. Built-in reads retain their existing cache and prefetch paths. The review fixes preserve the existing UDI behavior at the migration and rollback boundaries: - Reopening an SST with index caching enabled reuses the serialized custom block. Each table reader constructs its own factory reader over those bytes and passes `NewReader()` a `Slice` view stored in the table reader. That view stays valid for the reader's lifetime, so a legacy reader can advance it or keep a pointer to it without changing the cached block. The view and the raw block are destroyed after the custom reader. - `IndexFactoryReader::ApproximateMemoryUsage()` keeps its documented meaning: the reader's own allocations, excluding the serialized block. `MemoryUsageIncludesIndexBlock()` defaults to false. A reader whose estimate includes the block opts in, and the wrapper subtracts the block size from that estimate. - Copying factory options and clearing a legacy UDI flag now rolls back that flag's policy. Legacy-derived options keep the requested default and resolve their effective mode without changing the inputs. Explicit factory modes retain precedence through cloning, failed updates, reconstruction, and the C API, including an explicitly selected `kStandardDefault`. - OK empty custom-index output still produces a zero-size meta block and reads through the full standard index. This covers point data and files containing only range tombstones. Custom-only mode rejects empty output because it has no standard fallback. - Default primary reads retain the table factory's precedence over a legacy per-read factory pointer. Explicit `kPreferCustom` still checks a supplied factory name; `kBuiltin` ignores that pointer. - OPTIONS files write equivalent legacy booleans for `kStandardDefault`, `kStandardRequired`, and `kCustomDefault`. Older releases retain those policies when loading newer files with `ignore_unknown_options=true`; the explicit-mode marker remains authoritative for current readers. Custom-only SSTs carry a footer feature bit so older readers reject them. Reject unsupported filter/index combinations before constructing filters. Trie size estimates account for shared prefixes, and block offsets or sizes beyond the 32-bit format limit return `NotSupported` instead of truncating. ## Safety and compatibility - Custom-only files require format version 6 or later and the factory. Rewrite every custom-only SST with a full standard index before removing the factory or downgrading. The other modes have no equivalent legacy configuration beyond the three listed above. - A raw C++ options struct cannot distinguish an explicitly assigned `kStandardDefault` from its default value. Configure the factory to make that choice explicit. `GetEffectiveIndexMode()` reports the resolved policy while preserving legacy inputs for copy-and-clear rollback. - A custom factory with user-defined timestamps is rejected in every mode except `kStandardOnly`, which ignores the factory. - Trie construction rejects a data-block offset or size greater than `UINT32_MAX`, including in `kStandardDefault`. That can fail flush, compaction, or `SstFileWriter::Finish()`; total SST size above 4 GiB alone is not the rejection condition. File-size targets do not split L0 or `SstFileWriter` outputs. - Custom-index cache hits, misses, and adds are reported under `BLOCK_CACHE_INDEX_*` instead of `BLOCK_CACHE_DATA_*`. The existing cache setting still controls ownership and cache use. - A table reader now charges a custom index block it owns (`cache_index_and_filter_blocks=false`); before, that block was not charged anywhere. Cached blocks are still charged only to the block cache. A reader whose `ApproximateMemoryUsage()` includes the serialized block, such as IntegerTrie or TimeSeries, must override `MemoryUsageIncludesIndexBlock()` to return true, or that block is charged twice. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14963 Test Plan: For this commit, on macOS: `table_test`, `trie_index_test`, `trie_index_db_test`, and `block_based_table_reader_test` pass on the full stack (11,416 cases), and the full `table_test` passes at this PR's head alone (6,969 cases). The four custom-reader compatibility tests pass 100 times with `ASSERT_STATUS_CHECKED=1` and `COERCE_CONTEXT_SWITCH=1`, and 84 UDI table cases pass under ASAN with stack-use-after-return detection. Release-mode syntax checks with `-DNDEBUG -fno-rtti` on the changed sources, source checks, formatting checks, and `git diff --check` pass. Two new regressions cover the latest review, and both were checked failing before the fix: - `RetainedReaderSliceOutlivesTableOpen`: a reader keeps a pointer to the `Slice` it received, the stack is overwritten after open, and the reader then serves a seek and a full scan. It covers all three custom-index modes with owned and cached blocks, including a cached reopen. Without the fix, the retained `Slice` no longer matched the block. - `AuxiliaryReaderMemoryIsFullyCharged`: a reader that implements only the older reader methods reports a fixed allocation of half and twice the raw block size. Table usage must grow by that amount with owned and cached blocks, and a strict cache sized just below the required reservation must reject the open with `MemoryLimit`. Without the fix, usage fell short by `min(allocation, raw block size)` and the strict-cache open succeeded. The consuming-reader and strict-cache accounting tests are kept, now on shared helpers. From the previous revision: 498 focused C++ cases passed (82 table/factory, 156 trie, and 260 trie DB), along with the C API test and generated C API verification. Coverage includes both comparators, copied-options rollback, C factory reconstruction, explicit modes, OPTIONS serialization, SST Get/iteration, point and range-only flushes, compaction, reopen, cache disabled/enabled/evicted, and legacy factory precedence. The rollback, empty-output, primary-selector, and OPTIONS regressions were checked failing before their fixes. The cache test measures file reads at the custom block's actual offset: one on a cold open, zero on cached reopens, and one after eviction or with caching disabled. Disabling custom-block cache reuse makes that assertion fail. The full suite and Linux/Windows/TSAN coverage are left to CI. Earlier full-suite runs had failures/timeouts (173/2095 jobs normally and 152/2088 with Status checking); that broad suite was not rerun for this commit. ## Series Previous: https://github.com/facebook/rocksdb/issues/14962 (landed through import). Next: https://github.com/facebook/rocksdb/issues/14965. Reviewed By: jaykorean Differential Revision: D122796629 Pulled By: xingbowang fbshipit-source-id: 4c9d6f3f548c588a37b08fd94de25e8c0b5ef743	2026-10-21 20:20:38 -07:00
Josh Kang	014e1c43c1	Apply readability fixes (isolate decls, braces) (#15394) ... Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15394 ## Summary RocksDB's `.clang-tidy` enables several readability checks that existing code still violates: `readability-isolate-declaration` (476 findings), `readability-braces-around-statements` (49), `readability-redundant-string-init` (25), `modernize-use-using` (10), `misc-definitions-in-headers` (3) and `readability-duplicate-include` (1). They keep getting flagged whenever nearby lines change. This diff fixes 553 of the 564; upstream has since fixed the other 11. What changed: - Multi-variable declarations such as `std::string a, b;` are split into one declaration per variable. Pointer and reference declarators are kept per variable. - Braces are added around single-statement `if`/`else`/`for`/`while` bodies, and empty spin loops get an empty `{}` body. - `std::string s = ""` / `member("")` initializers are removed. - Remaining `typedef`s in the touched C++ files become `using`. C headers and `extern "C"` code are untouched. - `inline` is added to non-template function definitions in test headers. - A duplicate `#include "util/udt_util.h"` in `db_impl.cc` is removed. These are clang-tidy's own fix-its, applied mechanically and then clang-formatted on the changed lines. Three kinds of broken fix-it output were repaired by hand. Removing a `member("")` initializer left a dangling comma in constructor initializer lists (`ExternalSstFileInfo`, `ColumnFamilyMetaData` and others). The function-type typedefs in `block_test.cc` came out malformed. Empty `while (cond);` spin loops got a `{;` body. Splitting declarations also exposed two vectors in `db_etc2_test.cc` to `performance-inefficient-vector-operation`, so a `reserve()` was added for each. The lint bot reports every finding in a file a diff touches. This diff splits declarations in `options/configurable_test.cc`, so it also fixes that file's 14 older `cert-err58-cpp` and `cppcoreguidelines-avoid-non-const-global-variables` findings, which come from seven mutable file-level maps. Six are now `const` and built where they are used (function-local statics, or locals in their only user). The unused `imm_struct_option_info` is deleted. Compatibility: no behavior change. In public headers, the only edits are removed redundant `""` initializers, split declarations and added braces. Reviewed By: rban1 Differential Revision: D123186310 fbshipit-source-id: f9c8e88b00e997db0196f8725c7c8d880cf798da	2026-10-13 14:06:55 -07:00
Josh Kang	4b983da055	Spell out special member functions (rule of five) (#15387) ... Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15387 ## Summary RocksDB's `.clang-tidy` enables `cppcoreguidelines-special-member-functions`, `cppcoreguidelines-virtual-class-destructor` and the noexcept-move checks, and clang warns `-Wdeprecated-copy-with-user-provided-dtor`. A full scan reports 476 findings. In most of them, a class declares a destructor (or copy operations) but leaves the rest of copy/move implicit. That has two costs. Readers can't tell whether a class may be copied or moved. And resource-owning classes keep an implicit copy that would double-free if anyone ever used it. This diff resolves 453 of the 476. Approach, picked per class from least to most invasive: 1. An empty user-provided destructor becomes `= default`. This covers most interface classes, such as `virtual ~X() {}` -> `= default`. If the class is still copied, that would only rename the deprecated-copy warning, so the destructor is removed instead when nothing needs it (`Customizable`, `TableFactory` and `InternalKeyComparator` inherit a virtual one; `IteratorWrapperBase` isn't polymorphic), and `MemTableRep::KeyComparator` gets defaulted copy and move. 2. Classes whose copy is already deleted get explicitly deleted move operations too. Move was already unavailable, so this only documents it. 3. Resource-owning classes that are never copied get deleted copy and move operations. Examples: the C API wrapper structs, which run a `destructor_` callback (edited in `tools/c_api_gen/c_base.cc` and regenerated into `db/c.cc`); `ShardedCache`, which frees its shards; iterator wrappers; and RAII handles. 4. Value types that are copied or moved get the missing operations as `= default`, which keeps their current behavior. 5. Exception specifications of existing functions are unchanged, except for `WriteBatch`'s move assignment (below): no other `noexcept` is added or removed. Only the move operations this diff newly declares (`Comparator`, `FileSampledStats` and the thread-local `Entry`) are `noexcept`; they copy only scalars or atomics. `WriteBatch`'s copy and move assignment destroyed `*this` and placement-new'ed a new `WriteBatch` over it. That is undefined behavior when `*this` is a subclass, and if the copy constructor threw, `*this` was left destroyed and was later destroyed again. Move assignment now assigns member-wise and ends in the same state as the move constructor, which leaves `has_key_with_ts_`, the timestamp-size tracking and the other fields it doesn't copy at their defaults. `rep_` is swapped through a temporary, because a plain string move-assignment can keep the old buffer (e.g. a large batch reset with `batch = WriteBatch()` would keep its memory). Nothing in it can throw, so it is now `noexcept`. Copy assignment copies into a temporary and moves it in, so a throwing copy leaves `*this` unchanged. Bugs this exposed and fixed: - `RestoreAfterCopyOrCreateWorkItem` (backup engine): the hand-written move constructor and assignment moved only some fields, so `from_file` and `to_file` were empty after `push_back(std::move(...))`. A restore that hit a checksum mismatch then reported "While restoring -> : ..." without file names. It now uses the compiler-generated moves, which are still `noexcept`. - `ExternalSstFileIngestionJob` lived in a `std::vector`, and its destructor frees raw-owned pointers, so the implicit copy used on vector reallocation would double-free. Its file batches also point into its own inline file storage, so neither a copy nor a move is safe after `Prepare()`. Copy and move are now deleted, and `DBImpl` holds jobs through `std::unique_ptr`. Compatibility: no public class loses an operation users can call. Public interfaces mostly only get `virtual ~X() = default;`. `WriteBatch`'s move assignment becomes `noexcept`, which is not part of the mangled name, so the ABI doesn't change. `SubcompactionJobInfo` and `CompactionJobInfo` intentionally stay C++20 aggregates, so users' aggregate and designated initializers keep compiling. `Configurable`, `CompareInterface`, `WriteBatchBase`, `ObjectLibrary::Entry` and `MemTableRep::KeyComparator` declare defaulted copy and move, and `Customizable` no longer declares a destructor. As a result, `Configurable` and `Customizable` subclasses that declare no destructor or copy operations of their own now move their registered-options list instead of copying it. In-tree, these are the mutable-options adapters, the table factories, the memory allocators and test classes; all are held through smart pointers or used in place, and `Clone()` copies, so none is moved by value. Not changed (23): `IODebugContext` (custom deep copy plus a `std::shared_mutex`), `DBWithColumnFamilies` in db_bench (stored in a `std::vector` with non-movable members), `Snapshot`'s protected virtual destructor (intended), `ManagedSnapshot`, the `SubcompactionJobInfo` and `CompactionJobInfo` aggregates, and the 16 noexcept-move-constructor and noexcept-swap findings on existing functions, including `autovector`'s move assignment (adding `noexcept` would change their exception specifications). Copying `ManagedSnapshot` already double-releases the snapshot. Fixing that needs a real move implementation and is a behavior change, so it is left for a separate diff. Reviewed By: rban1 Differential Revision: D123186309 fbshipit-source-id: 6f8e57e233e12199a007407130bb9a1f6d52e8d1	2026-10-12 16:00:09 -07:00
Xingbo Wang	baa4d373a6	Commit authoritative OPTIONS file numbers in MANIFEST (#15305) ... Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15305 Why the change is required: Column-family topology is durable in MANIFEST, while its operational configuration snapshot is stored in a separate OPTIONS file. CreateColumnFamily and DropColumnFamily previously committed MANIFEST first and wrote OPTIONS afterward. An OPTIONS I/O failure could therefore make the API report failure after the topology change was already durable. Persisted-options readers also selected the highest file number and could consume an orphan left by a failed publication. Previous behavior: RocksDB had no durable correspondence between a MANIFEST state and the OPTIONS snapshot describing it. LoadLatestOptions and GetLatestOptionsFileName treated the highest-numbered OPTIONS file as authoritative. Column-family creation and deletion returned the later OPTIONS persistence error even when their MANIFEST edit had already succeeded. Multi-column-family operations wrote a complete OPTIONS snapshot after every successful member, producing quadratic OPTIONS bytes. Ordinary DB::Open consumes caller-supplied DB and column-family options rather than loading OPTIONS. This change preserves that contract. New behavior: Add the default-disabled experimental DB option track_options_file_number_in_manifest. Its value is serialized into each OPTIONS file so the selection rule is self-describing and participates in the normal DB-options compatibility checks. When enabled, RocksDB first durably creates and renames OPTIONS-N and then commits N with one safely ignorable MANIFEST field. A normal OPTIONS rotation uses one MANIFEST commit. Column-family creation and deletion prewrite the proposed snapshot and commit its number together with a topology edit in the same VersionEdit. A failure before the OPTIONS file is durable changes no topology. A failure after the file is durable but before MANIFEST commit leaves an orphan that binding-aware readers ignore, while the previous committed snapshot remains authoritative. Each multi-CF create or drop writes one proposed snapshot, independent of the batch size. Create commits the new number with the first successful add; drop commits it with the last successful drop. Existing partial-success semantics can therefore leave a committed snapshot with extra CFs, but never one missing a live MANIFEST CF. Persisted-options APIs filter this safe superset against recovered MANIFEST membership. A snapshot missing a live CF is corruption. DB::Open with create_missing_column_families uses the same batched path. If the newest OPTIONS file has tracking disabled, it retains the legacy highest-numbered-file behavior, including after an old or disabled writer has run. If it has tracking enabled, readers use the recovered MANIFEST commit and reject newer tracked orphans. A missing usable commit falls back to the highest tracked snapshot for RepairDB and legacy-rewrite recovery. DB open logs that fallback through its configured logger; logger-less utility APIs remain silent. Writable open removes rejected tracked orphans when local obsolete-file deletion is enabled. Read-only open never mutates the DB directory. A DB with no recovered commit and tracking disabled takes the legacy fast path without parsing the newest OPTIONS file. When inspection is required, RocksDB reads only the tracking marker. Marker-read failures used for selection do not fail DB open; a recovered commit remains authoritative and ambiguous newer candidates are conservatively retained. The first topology change after enabling tracking first binds the current topology, so failure of the proposed change cannot make its orphan authoritative. MANIFEST failure during an ordinary tracked OPTIONS rotation also enters the existing manifest-write background-error path. Implementation: - Add the safely ignorable CommittedOptionsFileNumber VersionEdit field and carry it through recovery, MANIFEST snapshots, rollover, and subset checkpoints. - Persist the tracking marker through C++, C, and Java options, generated round-trip coverage, compatibility checking, and db_stress configuration. - Centralize binding-aware selection for DB open and public persisted-options APIs. Reuse RecoveryContext's directory listing, use a narrow marker scan, and recover live CF names while replaying MANIFEST. - Filter manifest-selected safe-superset snapshots to live MANIFEST CFs in LoadLatestOptions and compatibility verification without discarding their option maps. - Prevalidate and sanitize CF batches outside the DB mutex, write one complete snapshot while options_mutex_ serializes topology changes, and hold the DB mutex only for the existing individual MANIFEST commits. Batch missing-CF creation during DB open follows the same locking boundary. - Keep the committed snapshot live in cleanup and live-file APIs, and make subset checkpoints write an OPTIONS snapshot and filtered MANIFEST with identical membership in tracked and legacy modes. - Preserve the first partial multi-CF drop error while consuming legacy OPTIONS persistence Status, and sync the checkpoint MANIFEST directory after replacement. - Pass checkpoint helper callbacks by reference and group VersionEdit storage by alignment so the new paths add neither callback copies nor avoidable object padding. Safety boundaries and unsupported cases: - Tracking defaults to false. Disabled writers and never-tracked databases retain legacy write ordering and highest-numbered selection. - Older binaries safely ignore the new MANIFEST tag and continue choosing the highest OPTIONS number. Downgrade is readable but not crash-atomic: an old binary can consume a tracked orphan. That is the same class of OPTIONS/MANIFEST inconsistency as the previous behavior. - A rollback binary must know the serialized option name before the feature is enabled. Exact compatibility checks intentionally report a mismatch when the caller's tracking setting differs from the persisted snapshot. - A newer untracked OPTIONS file deliberately supersedes an older committed number, allowing an old or disabled writer to become authoritative without rewriting MANIFEST. - Reactive secondary and follower instances do not enforce or repair the binding. Ordinary DB::Open still uses caller-supplied options. - Writable open does not delete rejected tracked or ambiguous files when a compaction service owns OPTIONS-file retention. Read-only open performs no repair writes. - Direct consumers that scan OPTIONS filenames themselves are unchanged; the binding is enforced by RocksDB's persisted-options APIs. - If the committed snapshot is missing and only tracked candidates remain, selection intentionally falls back to the highest candidate so RepairDB output remains usable. Validation coverage and known limitations: Focused coverage exercises encoding compatibility, tracked and legacy selection, the never-tracked open fast path, marker-read failures, exact tracking-marker compatibility, activation and disable/re-enable transitions, batched create/drop and create-missing behavior, partial-batch safe supersets, OPTIONS/MANIFEST failure injection, background-error propagation, orphan cleanup and retention, read-only non-mutation, subset-checkpoint behavior, stale OPTIONS recovery, RepairDB option sanitization, and transient read-corruption reconstruction. The assertion-enabled full executable suite passes. The repository check-format and check-sources wrappers cannot run because they invoke Git inside this Sapling checkout; fbsource lint is used instead. Cross-platform CI is not part of local validation. Reviewed By: pdillinger Differential Revision: D123104845 fbshipit-source-id: 77d94c989243c3f3f9fedfa81fa364c166250303	2026-10-11 10:39:30 -07:00
Josh Kang	c0512c4c34	Remove deprecated write_global_seqno ingestion support (#15287) ... Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15287 `IngestExternalFileOptions::write_global_seqno` has been deprecated since RocksDB 8.0 and will soon be removed from the API. It requires ingestion to mutate SST files, sync them, regenerate checksums, and expose file-offset plumbing solely for that mutation. Remove the option and its C, Java, and `ldb` APIs. Assigned sequence numbers now live only in the MANIFEST, leaving newly ingested SST bytes and whole-file checksums unchanged. ## Backward Compatibility New readers continue to support existing SSTs containing a nonzero global sequence number written by legacy ingestion. Because that old write did not update the properties-block checksum, checksum verification retries against a temporary copy with only the global-seqno value restored to its original zero placeholder. The parsed sequence number and file contents remain unchanged. This fallback only accepts mismatches explained by the legacy rewrite. Other checksum failures continue through normal corruption handling, and `verify_checksums=false` continues to bypass verification. Reviewed By: xingbowang Differential Revision: D122405856 fbshipit-source-id: 934e9bc755d58ea8d84e1c085e9769015bd8dfd7	2026-10-05 09:53:52 -07:00
Josh Kang	4d180247ae	Add PreparedFileInfo serialization APIs (#15274) ... Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15274 Expose `SerializePreparedFileInfo()` and `DeserializePreparedFileInfo()` so opaque metadata produced by SST writers or DBs can be persisted and transferred across process boundaries. The representation uses RocksDB `OptionTypeInfo` adapters for `InternalKey` and `TableProperties` and includes a CRC32C integrity check. Deserialization now rejects malformed nested string-map entries, including missing key/value separators and invalid hex, with `InvalidArgument` instead of asserting in debug builds or accepting malformed input in release builds. Valid serialized data and the wire format are unchanged. The `PreparedFileInfo` type metadata is also initialized on first use to avoid throwable namespace-scope initialization. Reviewed By: xingbowang Differential Revision: D121897852 fbshipit-source-id: 59a3d221bfd20e6c257bd60477823a12d05a35c5	2026-10-01 13:45:31 -07:00
Josh Kang	dbc2db5228	Make external table APIs more consistent with native tables (#15241) ... Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15241 Native `TableFactory` implementations receive the RocksDB `WritableFileWriter` in `NewTableBuilder()` and `RandomAccessFileReader` in `NewTableReader()`. These wrappers provide buffering, checksums, rate limiting, statistics, listeners, tracing, and direct-I/O handling. External table implementations previously received a limited writable-file adapter and reopened readers from a path, bypassing parts of this wrapper logic. This change makes `WritableFileWriter` and `RandomAccessFileReader` implement `FSWritableFile` and `FSRandomAccessFile`, respectively, so the same RocksDB wrappers can be passed to external tables. `ExternalTableFactory::NewTableReader()` now receives the already-opened reader and file size. The filesystem interfaces expose standard append, flush, sync, fsync, file metadata, synchronous reads, asynchronous reads, and batched reads. `RandomAccessFileReader::SubmitReadAsync()` schedules its wrapped coroutine read on the configured filesystem read executor, and the DB table-cache and `SstFileReader` construction paths pass that filesystem through. `SstFileReader` also supplies its configured statistics, listeners, rate limiter, and file temperature, removing the need for a separate external-table metrics filesystem wrapper. For direct I/O, `MultiRead()` checks whether requests are already sorted. When alignment and merging are needed, it constructs a sorted index only for unsorted input, preserving caller request order and result mapping. Fully aligned requests using aligned caller buffers bypass this path and may be supplied in any order. Reviewed By: xingbowang Differential Revision: D121035539 fbshipit-source-id: 27cb897726c31cb1a4dc811c65c8d79e690d82b1	2026-09-24 11:46:13 -07:00
Peter Dillinger	b03a058d61	Persist more SST table properties for compression analysis (#15231) ... Summary: These changes improve the block-based SST table properties available for compression benchmarking and optimization. - Fix: `uncompressed_data_size` was computed while building a file but was never written to the properties block, so it always read back as 0. It is now persisted (written in AddTableProperty and read back in ParsePropertiesBlock). - Add: a new packed `lsm_info_at_creation` property records the LSM level and bottommost status at the file's creation. A new `LsmInfoAtCreation` struct owns the (extensible) uint64 encoding via symmetric Encode()/DecodeFrom(), keeping TableProperties a bare data struct. It distinguishes three states: "unknown" (nothing recorded, e.g. older files), "not applicable" (no LSM position, e.g. SstFileWriter/misc output), and "applicable" (level + bottommost, with an unknown-level sentinel for the DB repairer). The level is stored as its natural value. - Change: the `_compression_manager` detail in the `compression_options` property now records the manager's full serialized configuration (via ToString) instead of only its GetId(). New properties are additive and backward compatible: older files without them read back as "unknown"/0. Pull Request resolved: https://github.com/facebook/rocksdb/pull/15231 Test Plan: Added coverage in db/db_table_properties_test.cc: LsmInfoAtCreation encode/decode round-trips (natural levels, unknown level, not-applicable vs unknown); end-to-end persistence of uncompressed_data_size and lsm_info_at_creation after flush (L0, not bottommost) and a forced bottommost compaction; not-applicable recorded for an SstFileWriter file; and full compression manager configuration present in compression_options. Updated options/options_settable_test.cc for the new field. Also made table_test's BlockBasedTableTest.CompressionRatioThreshold resilient to per-file overhead changes (such as these added table properties): it now verifies the compression decision via the persisted data_size vs uncompressed_data_size rather than modeling total file size against a fixed overhead constant, which had become too tight. Full table_test passes. Reviewed By: rban1 Differential Revision: D120553038 Pulled By: pdillinger fbshipit-source-id: 07ede62dbcbe31c040fd9b037c7636a49558c4d9	2026-09-17 13:18:07 -07:00
Jay Huh	897162a6bc	Add session-scoped temporary directory for remote compaction (#15169) ... Summary: Remote compaction writes output SSTs into caller-selected job directories. If the primary crashes before installation or falls back to local compaction, those directories can outlive the DB session and accumulate under the DB directory. This change defines `kSessionTmpDirName` as `session_tmp`, the canonical name for a DB-local temporary directory whose contents have the lifetime of a DB session. The immutable boolean option `use_session_tmp_dir_for_remote_compaction` opts remote compaction into this directory. When enabled, an `OpenAndCompact()` directory name is resolved as `<dbname>/session_tmp/<job>`. During DB open, RocksDB creates the session temporary directory, completes recovery, and then synchronously deletes contents left by the previous DB session before scheduling compactions. `DestroyDB()` removes it as part of database deletion, and normal completion, failure, and cancellation paths clean the resolved per-job directory. The option defaults to false. Disabled callers retain the existing `<dbname>/<job>` layout and no session-directory cleanup. For rollout compatibility, legacy callers that already pass a DB-rooted output path are accepted unchanged only while the option is false. Directory creation and invalid-path errors are delegated to the configured filesystem rather than prevalidated by RocksDB. After D118301531 landed, per-job directory creation and resumption cleanup are kept together in `InitializeCompactionWorkspace()`. `OpenAndCompact()` resolves the client-provided name and expects DB open to have initialized the session temporary directory when the feature is enabled. The boolean is wired through C++ and generated C APIs, option serialization and reconstruction, `OpenAndCompact()`, ldb, db_bench, and db_stress. Tests cover path resolution, cross-session deletion, ownership boundaries, disabled behavior, `DestroyDB()`, configured compaction output, resumption fallback, and option round trips. Pull Request resolved: https://github.com/facebook/rocksdb/pull/15169 Reviewed By: pdillinger Differential Revision: D118227112 fbshipit-source-id: 01aa834b0b3f75638102130a86d95d537a2797bd	2026-09-02 21:46:24 -07:00
Peter Dillinger	e4a7136d3e	Preferred-phase scheduling for periodic (time-based) compaction ... Summary: `periodic_compaction_seconds` softly guarantees every SST is rewritten at least once per interval. Across a fleet of similarly-configured DBs, many events line those intervals up in lockstep -- a shared enable, a coordinated restart (e.g. DC power-outage recovery), or just similar write history -- so the periodic compactions all come due together, a recurring "thundering herd" that is especially painful with remote compaction (SEV S698635). The obvious fixes trade one problem for another. Spreading the herd out means some files get compacted off their natural schedule, which is wasted work; and a clumsy spread just relocates the herd or spawns a new one. So the real goal is narrow: **de-herd while keeping "badness" -- extra and/or late compaction -- small, and without inducing fresh herding at any step.** This change adds an EXPERIMENTAL, opt-in mechanism that gives each `(DB, column family)` a stable, well-spread *preferred phase* within the interval and steers periodic compactions toward it. It is randomized and *uncoordinated* (DBs never talk to each other), which both keeps it simple and, as it turns out, de-herds far more cheaply than adding jitter to the interval. It is controlled by a single mutable DBOption and is **off by default**: - `periodic_compaction_phase_recovery_percent` (default `0` = disabled / kill switch; recommended `33`; clamped to `[0, 100]`) -- how aggressively each trigger converges toward the phase, as a percent of the remaining gap, only ever pulling the trigger *earlier*. `0` reproduces the classic scheduler exactly; `100` jumps to the phase in one cycle; lower values approach gradually with much less extra work. The preferred phase itself needs no tuning: it is derived from the DB ID (stable across re-open and physical cloning/migration, but distinct across DBs) and each column family is offset from the DB's base phase, so the CFs of one DB don't fire together. (An earlier draft exposed a `compaction_schedule_seed` string option for choosing the phase source; it was over-engineered for the one behavior anyone needed, and as a mutable string it also broke OPTIONS-file round-trip verification in crash testing. It has been removed in favor of always using the DB ID.) ## The two kinds of badness "Badness" here is the fraction of compaction work that is ideally unnecessary: over-compaction (rewriting a file earlier than the interval required) plus a penalty for lateness (going overdue past the interval). Steady state is 0; both "never compact" and "compact twice as often as needed" approach 1. De-herding inherently spends a little badness to spread load out; every design choice below exists to bound one of those two costs *without* re-clustering the fleet. The scenario names cited in the design bullets below (`upgrade_dynamic_r33`, `naive_upgrade_r25`, `burst_r*`, `turndown_r33`, `turnup_r33`, ...) all come from the out-of-tree simulator; see its results page (the "Simulator and visualization" section below, with an interactive plot per scenario) for the evidence behind each claim. ## Design - **A stable per-(DB, CF) preferred phase = one fixed attractor per DB.** Each DB gets a base phase from its DB ID; each CF is offset from it. Because the target is fixed (not re-rolled each cycle) and unique per DB, the fleet converges to a uniform spread and then compacts *on time* -- badness decays toward ~0. This is the crucial contrast with naive per-cycle jitter, which de-herds but never stops jittering and so pays a permanent over-compaction cost (see `naive_upgrade_r25`: flat ~12-14%, only diffusive spreading) vs. `upgrade_dynamic_r33`, whose badness decays to ~0 for the same anti-herding. - **Converge gradually, and only ever earlier.** Each trigger closes a `recovery_percent` *fraction* of the gap to the phase, reaching it over a few cycles rather than in one burst, and never later than `periodic_compaction_seconds` in steady state. A lower rate does less extra work, which matters when the phase keeps moving (write-driven / bursty workloads), where a high rate wastefully "chases" it -- the burst sweep shows `burst_r25 < burst_r33 < burst_r50 < burst_r100` in sustained badness. `33` is the default balance: quick spreading, low extra work. - **Fractional recovery (not "snap") is latency-tolerant and doesn't re-herd.** Closing a fraction of the gap gives a single stable convergence point, so real-world scheduling/queueing latency -- the delay between a file becoming due and its compaction actually running -- merely shifts the whole fleet uniformly instead of re-clustering it. A tried "snap onto the phase when in range" variant converged a bit faster with no latency but re-herded around the snap threshold under latency; it was rejected. Graceful degradation beat peak speed. - **Spread a suddenly-due cohort instead of bursting it (bounded lateness).** Enabling the feature, turning `periodic_compaction_seconds` down, or catching up after downtime can leave a whole cohort past due at the same instant. Firing it all at once would just be the herd again, so such a cohort is spread over the first quarter-interval (~N/4) after the phasing anchor on an *absolute*, epoch-aligned per-(DB, CF) grid. This trades a bounded lateness (<= ~N/4 past the deadline, so the interval is a *soft* upper bound during these transients) for anti-herding, and the absolute grid keeps a rapidly restarting DB from being perpetually re-deferred and starved. `turndown_r33` shows the past-due cohort smeared over N/4 rather than bursting; `turnup_r33` confirms a turn-up creates no past-due cohort (deadlines only grow) and so no burst. For immediate enforcement at such a transition, trigger a manual compaction. - **Off by default, with a kill switch.** Inert unless `periodic_compaction_phase_recovery_percent > 0`; at `0` behavior is identical to the classic scheduler. ## Implementation notes - Phase math is integer-only (`Hash64` + `FastRange64`); floating point is avoided for its pitfalls. The per-CF offset is a golden-ratio (Weyl/Kronecker) additive sequence -- low-discrepancy for any number of CFs. - Logic lives in `db/periodic_compaction_phaser.{h,cc}` (`PeriodicCompactionPhaser`), which owns the recovery percent and the phasing anchor and exposes the stateless trigger-time math; `VersionSet` holds one and caches the per-CF result on each `Version`. The anchor is (re)set to now on DB open, on a recovery-percent change, and on any `periodic_compaction_seconds` change -- the three events that can create a past-due cohort. - Plumbed through the C API, `db_bench`, `db_stress`, and `db_crashtest`. ## Simulator and visualization (out-of-tree, not part of this change) A standalone simulator drives 40 real RocksDB instances through a real periodic-task scheduler, records every compaction (periodic vs write-driven), and scores the badness metric above across ~17 scenarios: dynamic/restart enable, interval turn-down/turn-up via `SetOptions`, on-cadence writes, random write bursts with a recovery-rate sweep, and the naive-jitter baseline. It is the basis for the numbers cited above. Interactive HTML (**NOTE** fb-internal URL sanitization might require you to copy-paste the URL yourself): 5759c6a62d/pc_phase_sim.html Reviewed By: jaykorean Differential Revision: D117487278 fbshipit-source-id: a62439548b7dd6fe7dfdbbe21b69531d8b57a1e8	2026-08-26 20:14:26 -07:00
Peter Dillinger	8f492b2f47	Store common user-key prefix once per block (format_version 8) (#15117) ... Summary: Building on D116105558, this change finishes and publishes `format_version = 8` as production-ready. The headline feature is a tweaked block encoding that stores the common user-key prefix shared by all keys in a block **once**, in the block's leading bytes `[0, restarts[0])`, instead of repeating it at every restart point. This shrinks prefix-heavy blocks (and their block-cache footprint) and, for bytewise-ordered keys, speeds up `Seek` by comparing prefix-stripped suffixes. It is applied to **data blocks** and **every index tier** (leaf/partition index, the partitioned-index top-level, and the partitioned-filter top-level). Motivation: MyRocks/ZippyDB/timeseries-style keys carry a long shared prefix (table/index id, partition, etc.). Repeating it at each restart wastes space and cache, and every Seek re-compares those bytes. Profiles show index-block seek CPU is comparable to data-block seek CPU (partitioned index does two index seeks per data seek), so index blocks are a first-class target, not just data blocks. ### Format (self-describing, no new footer bit) - The common prefix occupies the block's leading bytes; a non-zero `restarts[0]` (the offset of the first entry, i.e. the prefix length) self-signals its presence. `restarts[0]` is 0 in every block that does not use the feature (all versions, all block types), so a single fv8 reader transparently handles any mix of stripped and unstripped blocks -- no footer bit is consumed. - Previous versions would misinterpret this encoding by quietly ignoring the common prefix, which (along with the lack of extensibility in data-block metadata) is why it is gated behind format_version 8. - Only restart-point entries change on disk; non-restart entries are byte identical to fv7, so the writer produces the exact same bytes regardless of implementation. ### Public option New `BlockBasedTableOptions::optimize_key_common_prefix` (`enum class OptimizeKeyCommonPrefix`), default `kIfFastSeek`: - `kDisabled` -- never use it. - `kIfFastSeek` (default) -- use it only where it also enables the Seek speedup: format_version >= 8 with a built-in (reverse-)bytewise comparator. - `kEnabled` -- use it for all comparators at fv8; (reverse-)bytewise get space + fast seek, other comparators get the space savings only (the reader reconstructs full keys and compares with the real comparator). Expressed as a criteria so it need not be toggled per CF based on the comparator (which can vary by CF). It additionally requires `use_delta_encoding` (the feature builds on delta encoding) and `ts_sz == 0`; when `use_delta_encoding` is false -- the setting that also preserves zero-copy key pinning -- it is a no-op. Default format_version is unchanged (still 7); the feature is opt-in via fv8. ### Scope: every index tier The reader handles all index-like blocks uniformly (a non-zero `restarts[0]` self-signals a stripped block regardless of block type), so the writer strips the common prefix from every tier: - **Data blocks** and **leaf / partition index blocks**: enabled. - **Partitioned-index top-level block**: enabled. Under the default `index_shortening = kShortenSeparatorsAndSuccessor` the file's last separator would otherwise be a short successor sharing no leading bytes, collapsing the single-block top-level's common prefix to 0. When common-prefix is on the builder suppresses that one successor (keeps the full last key -- i.e. `kShortenSeparators` behavior for that one entry), so the top-level strips uniformly under every shortening mode; the full last key is also a tighter file upper bound, so it is not worse. - **Partitioned-filter top-level index**: enabled (same uniform treatment). Its space/CPU effect is small at default partitioning (the top-level filter index is tiny) but grows with many small partitions -- a filter miss short-circuits at, and is dominated by, the top-level filter-index seek (see Performance). Included for a uniform on-disk surface across tiers. Bloom filter *contents* still hash full user keys, so prefix/whole-key filtering is unaffected. ### Implementation - Writer (`block_builder.{h,cc}`): keys are encoded incrementally in the fv7 layout while the running common prefix is tracked cheaply (reusing the delta `shared`); at `Finish()` only the restart-point entries are rewritten in place with the prefix stripped, and the prefix is written once at the block start. The rewrite is generalized to the value-delta (fv4) index layout used by index blocks (no `value_size` header; bulk-copy the inline BlockHandle and non-restart entries verbatim). - Reader (`block.{h,cc}`, `dbformat.h`): `Block` decodes the prefix from `restarts[0]`; `BlockIter::ParseNextKey` prepends it on a restart point (`shared == 0`). The suffix-seek helpers (strip target once, compare shortened suffixes) live in the templated `BlockIter` base and are shared by data and index iterators (binary search, templated on `DecodeKey`/`DecodeKeyV4`); for non-bytewise comparators (data) and the hash `PrefixSeek` refine step, the reader reconstructs the probed restart key and compares full keys. - Gating (`block_based_table_builder.cc`, `index_builder.{h,cc}`, `partitioned_filter_block.{h,cc}`): the criteria is threaded to the data-block builder and to every index tier -- leaf/partition index, the partitioned-index top-level (`use_common_prefix_top`), and the partitioned-filter top-level -- together with the last-separator successor suppression (`ShortenedIndexBuilder::GetSeparatorWithSeq`) that lets the top-level strip under the default shortening mode. The index base gate also requires `super_block_alignment_size == 0` (block alignment can trigger `skip_delta_encoding`, producing non-restart `shared == 0` entries the reader could not distinguish from stripped restart keys). - Memory: the per-block/per-iterator prefix is stored as a `uint32_t` length (the prefix is always at offset 0) rather than a full `Slice`, and `Block` / `BlockBuilder` members were repacked so the feature adds little or no struct size. - Java: a new `OptimizeKeyCommonPrefix` enum and a `BlockBasedTableConfig` getter/setter, plumbed through JNI in both directions (the `newTableFactoryHandle` writer and the options-reconstruction constructor). ### Whole-file (table-level) prefix: investigated and rejected A natural extension is to store the whole-file common prefix once per *file* (when an `SstPartitioner` guarantees every key in the file shares a fixed prefix) instead of once per *block* -- e.g. the MyRocks index/table id, or a future 128-bit global id. It was prototyped end-to-end (partitioner-provided prefix captured at file open, stored in a `user_key_common_prefix` table property, and stripped from every data/index block; the reader re-prepends it) and then dropped. The `user_key_common_prefix` table property reserved in the previous diff is dropped in this one. The rationale, in case it is revisited: - Marginal benefit over per-block. Per-block already strips the full shared prefix from every key; the only thing whole-file removes on top is the single per-block copy of that prefix, i.e. `~= prefix_len / block_size` of the uncompressed (block-cache) footprint, with no meaningful read-CPU change (per-block already delivers strip-once + suffix compares). Measured at prefix_len=12, 4 KB blocks this was ~-0.3% incremental space; at the 8-16 KB data blocks typical of the space-constrained target (MyRocks/UDB) it is ~0.1-0.2% even with a 16-byte global id -- a fraction of a percent, seen only by partitioner users. Small block-size workloads are becoming more niche, and even where they exist, they are also CPU-sensitive (see next paragraph). - Real, if small, CPU tax on the *per-block* path. Reconstructing a key from two non-contiguous parts (file prefix + block residual) cost ~0.5-1.0% additional retired instructions on the per-block read path (deterministic across gcc and clang), wall-clock-neutral in cache-resident tests but present even when no whole-file prefix is configured. Attempts to remove it did not pan out: a compile-time specialization of the reconstruction *increased* instructions (+2.9%) by defeating inlining; a per-block combined-prefix buffer trades the CPU cost for a memory cost on every (default) block; and a per-iterator buffer was the original larger regression. Only invasive iterator-level monomorphization would truly zero it, which is not justified on this hot path. - Decision. A fraction-of-a-percent, partitioner-only space win does not justify a permanent per-block-read-path CPU tax plus additional on-disk format surface (a table property that changes whole-file interpretation). Dropped. The generic `SstPartitioner::ShouldPartitionByPrefix` hook is kept (cheap, and useful independently), and the reserved `user_key_common_prefix` table property is removed since no writer will emit it. ### Performance (db_bench, release/native, cache-warm, best-of-3, pinned) Three arms: (A) fv7 on the base/main branch, (B) fv7 with this change (should be neutral / byte-identical writing), (C) fv8 with this change and `optimize_key_common_prefix=enabled` (the optimization is active). Partitioned index + filters; `compression_type=none`, so SST bytes are the uncompressed block-cache footprint. `int` = db_bench default keys (variation-early, near worst case); `pfxN` = an N-byte shared prefix + varying suffix (MyRocks-like). Setup: `num=3,000,000`, `reads=1,500,000`, single thread, `metadata_block_size=4096`, cache holds the working set, pinned to one core. | shape (key/val) | arm | SST bytes | readrandom us/op | seekrandom us/op | |---|---|---:|---:|---:| | int (32/8) | A fv7 main | 138,983,658 | 3.484 | 4.995 | | | B fv7 ours | 138,983,146 | 3.518 | 5.024 | | | C fv8 ours | 137,681,420 | 3.490 | 4.986 | | pfx24 (32/8) | A fv7 main | 71,697,531 | 3.417 | 5.561 | | | B fv7 ours | 71,698,043 | 3.448 | 5.579 | | | C fv8 ours | 65,578,236 | 3.284 | 5.386 | | pfx56 (64/16) | A fv7 main | 102,942,266 | 3.624 | 5.830 | | | B fv7 ours | 102,942,266 | 3.678 | 5.835 | | | C fv8 ours | 90,554,642 | 3.444 | 5.626 | fv8 vs fv7-main (C vs A) -- space / readrandom / seekrandom: - int (32/8): **-0.9%** / ~neutral / ~neutral - pfx24 (32/8): **-8.5%** / **-3.9%** / **-3.1%** - pfx56 (64/16): **-12.0%** / **-5.0%** / **-3.5%** fv7-ours vs fv7-main (B vs A): SST bytes byte-identical (within 512 B; the fv8 paths are gated off), read/seek us/op within run-to-run noise (~1%, best-of-3 on a shared host). Net: fv8 is a space **and** read/seek-CPU win on prefix-heavy data and neutral on variation-early (`int`) data, so `kIfFastSeek` is safe on by default at fv8. ### Space by tier (compression=none, num=3M, mdb=4096; fv7 -> fv8) Where the win comes from -- the index tiers strip the most: | shape | data | leaf/partition index | top-level index | filter | |---|---:|---:|---:|---:| | int | -0.8% | -29% | -45% | -0.2% | | pfx24 | -8.3% | -69% | -89% | -0.8% | | pfx56 | -11.1% | -83% | -96% | -1.6% | Data bytes dominate the absolute total, but every index tier is stripped: leaf/partition index -29% to -83% and the partitioned-index top-level -45% to -96%. Filter *blocks* barely move (-0.2% to -1.6%) because filter bits hash full user keys; only the small top-level filter index is stripped. ### Point-miss lookups (readmissing) A partitioned-filter miss short-circuits at the filter (no data-block read), so it is dominated by the top-level filter-index seek plus the partition bloom. At a typical `metadata_block_size=4096` (few partitions, tiny top-level filter index) fv8 is neutral on misses (int +0.9%, pfx24 -1.0%, pfx56 -0.7% us/op -- all noise). The top-level-filter stripping pays only with many small partitions: at `metadata_block_size=256` it is pfx24 1.166->1.138 (**-2.4%**) and pfx56 1.273->1.188 (**-6.7%**) us/op. ### Backward / forward compatibility - Default `format_version` remains 7; the feature is inert unless a CF opts into fv8. Older RocksDB rejects fv8 files at footer decode (fail-safe). - Within fv8, the stripped-block set a writer can emit is fixed and every fv8 reader can read it (the `restarts[0]` signal plus uniform reader handling), so the on-disk surface is self-consistent across builds/downgrade. ### Follow-ups (not in this change) - Suffix-aware interpolation / hash index seek (currently correct via the reconstruction fallback, not perf-tuned). Pull Request resolved: https://github.com/facebook/rocksdb/pull/15117 Test Plan: - New focused unit tests (`table_test`), run at the full `format_version` matrix including the now-supported fv8: `GeneralTableTest.OptimizeKeyCommonPrefix` (data-block space+read+Seek correctness: existing and missing keys straddling the prefix, before/after-all, SeekForPrev), `OptimizeKeyCommonPrefixNonBytewise` (reconstruction path), and `IndexBlockCommonPrefixEquivalence` across `{kBinarySearch, kBinarySearchWithFirstKey, kHashSearch, kTwoLevelIndexSearch}` x `{partitioned filters}` x `{bytewise, reverse-bytewise}` (so both top-level tiers -- partitioned-index and partitioned-filter -- are exercised), asserting identical reads and a shrunken fv8 index. - The randomized `ParameterizedHarnessTest` now runs at fv8 (reverse-bytewise, restart intervals `{1,16,1024}`, special/empty keys fuzzed against the feature). - `db_crashtest.py` randomizes `format_version` including 8 and `optimize_key_common_prefix` in {0,1,2}. - Removed the reserved-property carve-out: dropped the `ReservedUserKeyCommonPrefix` reader test and the C API getter for the property; C API regenerated via `tools/c_api_gen/` (`make regen-c-api`), and the intentional public-C-API removal is recorded in `tools/c_api_gen/api_compatibility_allowlist.json`. - Java: `BlockBasedTableConfigTest.optimizeKeyCommonPrefix` (default `kIfFastSeek`, plus set/get of every value). - `table_test` `FooterTests` updated for the now-published fv8 footer, whose metaindex-gap encoding stores the metaindex block size in 16 bits (the test caps that size for fv >= 8). - Manual verification: the db_bench runs in Performance above (three-arm space + read/seek CPU, per-tier byte breakdown via `sst_dump`, and `readmissing`); and `readrandom`/`seekrandom` with `--block_protection_bytes_per_key=8` find 100% of keys at both fv7 and fv8 (reconstructed keys pass per-key checksum). Reviewed By: xingbowang Differential Revision: D116365016 Pulled By: pdillinger fbshipit-source-id: 66f4b5c60ea4fe0ff48d34c98d191ba8ebdf0665	2026-08-20 17:32:41 -07:00
Peter Dillinger	e8554f1b11	Faster and safer MANIFEST recovery for remote compaction (OpenAndCompact) (#15133) ... Summary: Faster and safer MANIFEST recovery for remote compaction (OpenAndCompact) `DB::OpenAndCompact` (remote/distributed compaction) opens the target DB as a secondary and reconstructs the LSM state from the MANIFEST before running the requested compaction. Previously, point-in-time recovery would stat/open the SST (and blob) files referenced by the MANIFEST to classify them found/missing and materialize candidate versions, and eagerly load their table handlers. That is (a) a lot of unnecessary file operations and open latency, especially on remote/distributed storage, and (b) fragile: when a file referenced by the MANIFEST but not an input to the compaction was momentarily unavailable to the worker -- e.g. an obsolete file the primary had already deleted -- recovery could fail the job (surfacing as an incomplete "no valid result") or roll back to an earlier version, even though the compaction only needs its own input files, which the primary keeps referenced for the job's duration. This was seen as a flaky failure of CompactionServiceTest.BasicCompactions on aarch64, and reproduces on x86 under COERCE_CONTEXT_SWITCH. Efficiency + reliability: recovery for the OpenAndCompact path now trusts the MANIFEST. It reconstructs the current version from FileMetaData without stat-ing/opening files to classify them and without eagerly opening candidate-version table handlers, so it does far fewer file operations and never fails or rolls back over a file the compaction does not need. The compaction's input files are opened (and unique-id verified) on demand as it reads them. Correctness hardening: trusting the MANIFEST means the worker's own (possibly eventually-consistent) view of CURRENT/MANIFEST drives the reconstructed LSM shape, which feeds correctness-critical decisions (Compaction::IsBottommostLevel / KeyNotExistsBeyondOutputLevel). A worker that read a stale CURRENT (older MANIFEST), a truncated MANIFEST, or a MANIFEST with a corrupt/torn tail could reconstruct against an older LSM shape and silently drop keys that should be kept. To prevent this, the primary records the MANIFEST position (file number and size) it scheduled the compaction from and passes it in CompactionServiceInput as a floor. After recovery the worker compares the size of the maximal *valid* prefix it reconstructed -- which stops at any corrupt or torn record and at a partial atomic group at EOF, so tolerated tail garbage cannot inflate it -- against that floor, and refuses to compact against an older view (an older file number, or the same file number whose valid prefix is shorter), returning Status::Incomplete so the CompactionService can fall back to a local compaction. Accept-equal-or-later is safe by monotonicity: while the inputs stay locked (being_compacted), a later view can only have moved data further down, never resurrecting a shadowed value, so only an older view is unsafe. Single kill switch: both behaviors are bundled behind one signal and one option. Presence of the floor on the request (min_manifest_file_number != 0) couples on both trust-manifest recovery and the floor check in the worker; absence reverts the worker to the original point-in-time recovery with no floor check. A single mutable DBOption, remote_compaction_manifest_floor (default on), gates whether the primary populates the floor, so it is the one escape hatch that disables the entire new recovery path on a running DB via SetDBOptions if it ever misbehaves. The new CompactionServiceInput fields are wire-compatible in both directions (0 = not provided), so an old worker or an old primary simply behaves as if the switch were off. Follow-up: recovery is not yet strictly "open only the compaction's inputs". A limited number of non-input files are still opened by the bounded file-property sampling in Version::PrepareAppend -> UpdateAccumulatedStats (tolerating any that are missing), which today populates input FileMetaData::num_entries for compaction input-record-count verification. A natural follow-up is to record the needed per-file info (e.g. num_entries) in the MANIFEST so the worker can skip these auxiliary opens entirely; alternatively, source input num_entries from the worker's own read of its input table properties. Relatedly, the floor comparand is the recovered "maximal valid prefix" size, which today equals the MANIFEST file size but is defined to stay correct if a future MANIFEST format writes trailing padding/footer past the last record. Pull Request resolved: https://github.com/facebook/rocksdb/pull/15133 Test Plan: Added tests in compaction_service_test.cc: - RemoteCompactionMissingLowerFileTrustsManifest: a non-input file referenced by the MANIFEST is unavailable during a remote compaction; the job completes with correct results (fails without trust-manifest recovery). The underlying race also reproduces under COERCE_CONTEXT_SWITCH. - RemoteCompactionManifestFloorRejectsStaleView: inflate the floor so the worker's recovered MANIFEST is below it; assert the worker rejects, the job falls back to local, and a below-output deleted key stays deleted (no silent resurrection). - RemoteCompactionManifestFloorSendsPosition: a non-zero floor is attached and the worker accepts its equal-or-later view. - RemoteCompactionManifestFloorKillSwitchOff: with the option off, no floor is sent, the worker uses the original recovery, and a transiently-missing non-input file makes the job fail/roll back (the escape hatch reverts behavior). - RemoteCompactionManifestFloorMutable: SetDBOptions flips the option; the floor stops being attached and the worker skips the check. options_settable_test covers the new mutable DBOption. Crash test: remote compaction is already exercised (db_crashtest randomly enables remote_compaction_worker_threads, driving DB::OpenAndCompact), so with remote_compaction_manifest_floor defaulting on, the new recovery path is covered by the crash test by default -- no new crash-test plumbing is needed. The kill switch is intentionally left un-randomized (always on): turning it off re-enables the original, occasionally-broken recovery, which would only add noisy failures for no benefit. UPDATE: crash test is not well covered because of `"remote_compaction_failure_fall_back_to_local": 1` which is masking other existing failures. Addressing this is a high-priority follow-up item. (Not practical to fix in this diff.) Reviewed By: jaykorean Differential Revision: D116738823 Pulled By: pdillinger fbshipit-source-id: c02e0a437d62fc526dbbfcf70edf43434dfc7c1f	2026-08-20 16:07:48 -07:00
zaidoon	136d0d1fa1	Add UDI index mode API vocabulary (#14959) ... Summary: Part 2 of 13 in the UDI split. Stack order: 1. https://github.com/facebook/rocksdb/issues/14954 Add IndexFactory compatibility names 2. https://github.com/facebook/rocksdb/issues/14959 Add UDI index mode API vocabulary 3. https://github.com/facebook/rocksdb/issues/14960 Add optional UDI builder protocols 4. https://github.com/facebook/rocksdb/issues/14961 Add built-in index factory wrappers 5. https://github.com/facebook/rocksdb/issues/14969 Add built-in index factory tests 6. https://github.com/facebook/rocksdb/issues/14962 Promote IndexFactory as UDI SPI 7. https://github.com/facebook/rocksdb/issues/14963 Wire IndexFactory through block-based tables 8. https://github.com/facebook/rocksdb/issues/14964 Add IndexFactory table routing tests 9. https://github.com/facebook/rocksdb/issues/14970 Add IndexFactory parallel routing tests 10. https://github.com/facebook/rocksdb/issues/14965 Account UDI blocks as index blocks 11. https://github.com/facebook/rocksdb/issues/14966 Support parallel AddIndexEntry in trie index 12. https://github.com/facebook/rocksdb/issues/14967 Cover trie index modes 13. https://github.com/facebook/rocksdb/issues/14968 Add IndexFactory stress and benchmark flags Previous: https://github.com/facebook/rocksdb/issues/14954. Next: https://github.com/facebook/rocksdb/issues/14960. Depends on https://github.com/facebook/rocksdb/issues/14954. Until https://github.com/facebook/rocksdb/issues/14954 lands, GitHub shows the cumulative diff against main. After that lands, the intended review diff is: `zaidoon/udi-01-index-factory-compat-shim..zaidoon/udi-02-index-mode-api` That final review delta is 4 files with 52 insertions. What changed: - Add inert IndexMode and ReadIndex vocabulary to the public options headers. - Block those experimental enum fields from automatic C API generation until the explicit C API mapping lands later in the stack. - Keep behavior unchanged. Runtime wiring comes in later PRs. Validation: - `git diff --check` - `make check-sources` - `python3 tools/c_api_gen/verify_generated_up_to_date.py` - `AUTO_CLEAN=1 make check-c-api-gen` - `./c_test` Pull Request resolved: https://github.com/facebook/rocksdb/pull/14959 Reviewed By: jaykorean Differential Revision: D116211178 Pulled By: xingbowang fbshipit-source-id: 42edd3f0991d90b0b5c90232e5023de6e3740dac	2026-08-19 10:31:27 -07:00
Xingbo Wang	db1adf47bb	Add bootstrap configuration for RocksDB external tables ... Summary: Add an opaque external_table_config option to the external table adapter. RocksDB persists the string in OPTIONS files and passes it to ExternalTableFactory::Configure during bootstrap, allowing implementations such as Nimble to parse implementation-specific settings without adding RocksDB-specific fields. The option is immutable after DB open, and attempted runtime updates fail safely for non-clonable table factories. A reopen test verifies that callers can supply a different valid configuration between DB opens. Initialize the adapter option metadata lazily as const to avoid namespace-scope dynamic initialization and mutable-global lint issues. PR https://github.com/facebook/rocksdb/pull/15119 Reviewed By: anand1976 Differential Revision: D116362444 fbshipit-source-id: 225a4aedc455f320b5c90ac2f0862917da477ccb	2026-08-18 10:12:47 -07:00
Peter Dillinger	021374c62c	Start format_version=8 development toward new features (#15104) ... Summary: format_version background: SST files are shared across RocksDB versions. format_version is a critical tool in ensuring prior (and later) versioned readers do not misinterpret a file created by some version. Opt-in features can sometimes rely on previously-rejected corners of the schema space, but not always. This change: lands much of the format_version=8 (fv8) format surface as an internal-only draft -- readable and testable but not user-writable -- so upcoming features can build on a stable base. fv8 is NOT published: kLatestBbtFormatVersion stays 7, and reading or writing fv8 is gated behind the existing TEST-only TEST_AllowUnsupportedFormatVersion() hook. Publishing fv8 (raising the ceiling) is deferred to a later change. The pieces of fv8 added here, and what each unlocks: - Index value-delta escape. Existing behavior: index blocks (format_version >= 4) store each block handle as a delta against the previous entry's handle, and signal "this value is a full handle instead" by reusing the key sentinel `shared == 0`. Because the two are coupled, an index entry whose handle is not contiguous with the previous one must be forced to a restart, costing a full key as well as a full value. fv8 decouples them with an in-value escape codepoint: varint 0 means "a full BlockHandle follows," and genuine size deltas are remapped to zigzag(delta)+1 so that 0 stays free. A non-contiguous entry can then remain an ordinary key-delta-encoded, on-cadence entry. The escape applies to every index-shaped block, which includes the top-level index of index partitions and the index of filter partitions, not just the index over data blocks. Unlocks: the upcoming optimize_key_common_prefix index feature (removes its need to special-case super_block_alignment) and value-delta encoding for embedded-blob SSTs (previously disabled outright because interleaved blob records break handle contiguity). Guarded strictly on fv8; format_version <= 7 encoding and decoding are byte-for-byte unchanged. Note that restart points must not delta-encode, so they still have full values without a sentinel. - Data block footer bit 30 is reserved as an "extended metadata present" escape. Decode rejects a set bit as unsupported/corrupt at all versions. Unlocks: adding block-level metadata in the future through the one shared footer decoder, without another format_version bump or plumbing format_version into every block parser. (Older reader versions relied on this bit being zero for some integrity/consistency checking; format_version bump now makes it safe to use.) - Footer metaindex gap. For fv8, the top 16 bits of the footer's 4-byte metaindex-size field record the gap (in bytes) between the end of the metaindex block and the start of the footer; the low 16 bits hold the metaindex block size. Unlocks: a space very near the end of the file to save full file checksum state, for a "files carry their own checksums" feature that doesn't require SST parsing to extract the checksum states (to run them to completion on the trivial file tail). - Reserved "user_key_common_prefix" table property. The reader returns NotSupported when it is present and non-empty. Unlocks: a future whole-file optimization that stores the common prefix of all user keys once instead of in every restart key; until implemented, fv8 readers refuse such a file rather than mis-read it. - Because of the problem of inferring such a prefix at the start of constructing an SST file, we expect it will rely on SstPartitioner. Here we've added SstPartitioner::ShouldPartitionByPrefix() (not yet used) that will enable this optimization, and avoid the overhead of a virtual callback to SstPartitioner on every key written to an SST file. Supporting cleanups: the Footer's "unset" format_version sentinel is named (kInvalidFormatVersion) and the FormatVersionUses*() predicates assert against it, as each is a `version >= N` test that an unpopulated Footer would silently satisfy. Index block reader entry points (Block::NewIndexIterator() and friends) drop their defaulted trailing arguments so each caller states which codec it expects, and BlockBuilder::Add() takes delta_value by const reference rather than a pointer whose null case no caller used. Pull Request resolved: https://github.com/facebook/rocksdb/pull/15104 Test Plan: New unit tests, all exercising fv8 through TEST_AllowUnsupportedFormatVersion(): - block_test: value-delta escape codec round-trip (delta 0, negative, large, and the escape; asserts format_version <= 7 bytes match the legacy encoding and a legacy 0x00 still decodes as delta 0), and rejection of footer bit 30 at all versions. - db_flush_test: fv7-vs-fv8 read equivalence for super_block_alignment, including the separate_key_value_in_data_block intersection, confirming padding events (hence escape entries) occur and fv8 index values are delta encoded. - sst_file_reader_test: fv7-vs-fv8 read equivalence for embedded-blob SSTs, confirming fv8 (unlike fv7) delta-encodes their index values and actually emits escape entries (counted through a sync point). - block_based_table_reader_test: a builder+reader test that injects a footer metaindex gap via a sync point (verifies the gap bytes are written and the reader locates the metaindex via the recorded gap, and that too large a gap surfaces as a builder error), and that a non-empty reserved user_key_common_prefix property makes the reader return NotSupported. - table_test: two-level index with partitioned filters and index_block_restart_interval > 1, the configuration in which both top-level indexes delta-encode their values; verifies scan and point lookup round trips at each tested format_version including fv8. - (Not strictly fv8) Basic SstPartitionerFixedPrefix tests in slice_test (ShouldPartition, ShouldPartitionByPrefix, their mutual consistency, and CanDoTrivialMove), which had no non-DB coverage before. fv8 is also added to the table_test harness (GenerateArgList) and the shared footer version set (test::kFooterFormatVersionsToTest), so the existing table-level and DB-level property/checksum suites now also run at fv8. And added to crash test (minor revisions to db_stress). ## Performance details Release db_bench (PORTABLE=0, -march=native) A/B/C on the read/write fast path, comparing (a) parent, (b) this commit @ fv7, (c) this commit @ fv8. All DBs on tmpfs, 20M keys, block_size=2048, compression=none. Note index_block_restart_ interval=16 (not the default 1) is required to actually exercise the fv7<->fv8 index value-delta codec -- with restart interval 1 every index entry is a full- handle restart and the changed path never runs (confirmed delta-value=1 via sst_dump). Reads single-thread + core-pinned, warm cache, interleaved/rotated rounds, paired deltas. Reads (readrandom + seekrandom, non-partitioned single ~1.9MB index block and partitioned index+filters): fv7->fv8 in the *same* binary is within noise on all four workloads (largest -1.0%, CIs cross zero). before->after (fv7 path) is also within noise; perf shows a deterministic +0.4-0.6% instructions/op from the added branches (the format_version>=8 value_delta_escape guard + refactored data-block footer decode), which is below the throughput noise floor. Writes (fillseq + compact total CPU): fv7->fv8 null. compact (which rewrites all index blocks) is null before->after. fillseq showed +3.5% CPU before->after, but perf attributes it entirely to cycles/IPC with *identical* instructions/op (+0.01-0.04%) -- a code-layout effect on the untouched memtable-insert loop, not this change. Sanity: any single throughput/CPU delta <~1% is within the build-to-build noise floor, measured directly by comparing two byte-identical binaries (swings up to ~0.9% reads; the pre-native/PORTABLE=1 build showed a spurious -1.85% that vanished under -march=native). Net: no measurable regression from fv8 draft; fv7 encoding/decoding byte-for-byte unchanged, only the noted sub-1% instruction bump on the shared read path. Reviewed By: xingbowang Differential Revision: D116105558 Pulled By: pdillinger fbshipit-source-id: ead776cdd396d497d47872c0956ea8c651c0d13f	2026-08-17 12:08:59 -07:00
Michael C Huang	276495843d	Add blob file writer buffer size option (#15061) ... Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15061 Adds mutable ColumnFamilyOptions field `blob_file_writable_file_max_buffer_size` to cap blob-file WritableFileWriter buffers independently from DBOptions::writable_file_max_buffer_size. The default value is 0, which preserves existing behavior by inheriting the DB-wide writer buffer cap. Non-zero values apply to both flush/compaction blob writers and direct-write blob writers. Direct-write updates are propagated through DBImpl::SetOptions to newly opened blob files without requiring a DB restart. Also wires option parsing, dumping, db_bench, db_stress, crashtest, and release-note coverage. Reviewed By: xingbowang Differential Revision: D114979265 fbshipit-source-id: 3f764b797628eee0ce379110c1a1f8a16beb3fa2	2026-08-11 17:38:34 -07:00
Peter Dillinger	fcf382c95c	Integrated and ratio-aware compression auto-skip (#15079) ... Summary: Auto-skip compression (stop attempting compression on data that isn't compressing well, to save CPU) was previously an experimental CompressionManager wrapper (CreateAutoSkipCompressionManager) whose predictor lived in each per-thread working area. Besides being fragmented across parallel workers and reset on a tumbling window, it decided based on how OFTEN blocks failed to compress, ignoring how MUCH they compress -- so on mixed data it could give up substantial compression to save a little CPU (see performance results). This makes auto-skip a first-class, optional, ratio-aware feature of the block-based table builder, controlled by two new CompressionOptions fields: - auto_skip (default false): the on/off switch. There is no separate threshold knob; the decision reuses max_compressed_bytes_per_kb (the existing per-block "minimum compression worth keeping" bar) as the bar for whether attempting compression on the stream is worthwhile. - auto_skip_min_sample_every: sampling interval that bounds the local impact around a transition back to compressible data. 0 selects an internal default (currently 64 skipped data blocks between forced samples). The emit thread makes the skip/attempt decision once per data block (serial and parallel paths alike; data blocks only), so a skipped block is never dispatched to a compression worker. It is driven by a continuously updated, shared estimate q_hat of the aggregate stored/uncompressed ratio the builder would achieve by attempting -- each block counted at its achieved ratio, or 1.0 when stored raw: attempt while q_hat is at or below the bar, otherwise skip. Because the estimate is payoff-weighted, a stream whose blocks each save little -- or where only a few blocks compress well -- does not clear the bar and is skipped, while a stream that compresses substantially is attempted. The compression given up per KB is bounded by (1024 - max_compressed_bytes_per_kb). q_hat uses an asymmetric exponentially weighted moving average (EWMA), slightly faster toward better compression than toward worse; the fast side is deliberately gentle so a single marginally-compressible sample does not flip a whole file back to attempting. It is kept on its own cache line to avoid false sharing. Under parallel compression q_hat is updated with relaxed atomics, and the emit thread can decide from a value that lags the learning it drives by the in-flight depth -- tolerable concessions of accuracy and responsiveness for a coarse heuristic. The first data block of each file is always attempted (the sample countdown starts at 0), and while skipping the builder keeps sampling (one block after a randomized gap) so a return to compressible data is detected. q_hat is carried across the files a flush/compaction thread emits via thread_local state keyed by preferred compression type, so a sustained region does not re-learn on every file. The old CompressionManager-based AutoSkip is removed (CostAware is retained); db_stress, db_bench, and db_crashtest are migrated to the new option. db_bench is extended so it can generate the relevant regimes: --compressible_data_fraction and --compressible_data_run_length produce a mix of compressible/incompressible blocks -- i.i.d. per value (run length 1) or in alternating regions (large run length). The default (fraction 1.0) reproduces the historical single-pool behavior. Pull Request resolved: https://github.com/facebook/rocksdb/pull/15079 Test Plan: New/updated unit tests in `util/compression_test.cc` (`DBAutoSkipTest`): - skips sustained incompressible data; - never skips compressible data; - disabled by default never bypasses a block; - 4-thread parallel compression round-trips all data across alternating compressible/incompressible regions. **Performance** (`db_bench fillseq`, release, zstd level 3; 100k values of 16 KiB each, roughly one value per data block; WAL and auto-compaction off; median of 5 runs on a quiet host). Auto-skip trades one cost for another, so each cell reports two costs, both lower-is-better: CPU cost as process CPU time (user+sys) per GB of uncompressed input (s/GB; ~1.64 GB uncompressed here), and storage cost as the resulting DB size. Auto-skip does not appear to change parallelizability, so wall-clock s/GB tracks CPU s/GB (the two agree here); CPU s/GB is reported. Percentages are relative to the no-auto-skip baseline: negative CPU = cheaper, positive size = more storage (shown only where storage grows). | Workload | baseline (off) | (a) old wrapper | (b) draft | (c) this version | |---|---|---|---|---| | Incompressible | 2.33 s/GB, 1568 MB | 0.96 (-59%), 1568 MB | 0.82 (-65%), 1568 MB | 0.82 (-65%), 1568 MB | | Compressible | 3.55 s/GB, 710 MB | 3.54 (~0%), 710 MB | 3.56 (~0%), 710 MB | 3.55 (~0%), 710 MB | | Marginal-payoff mix | 3.80 s/GB, 1376 MB | 1.54 (-59%), 1523 MB (+11%) | 3.81 (~0%), 1376 MB | 1.43 (-62%), 1531 MB (+11%) | | High-payoff mix | 2.20 s/GB, 929 MB | 1.14 (-48%), 1417 MB (+53%) | 2.16 (~0%), 929 MB | 2.17 (~0%), 929 MB | A strategy wins by cutting CPU cost without paying much storage cost. On incompressible data all three cut CPU ~60% at no storage cost; on compressible data none skip (within run-to-run noise). The two mixes (50% incompressible, 50% compressible per block) separate the strategies: - Marginal-payoff mix: the compressible half barely clears the per-block bar, so the stream is not worth compressing -- skipping is right. (c) and (a) cut CPU ~60% for ~11% more storage. That extra storage is compression (c) intentionally declines because its aggregate benefit is below the configured worthwhile threshold (max_compressed_bytes_per_kb, ~12.5% per KB); it stays within the (1024 - max_compressed_bytes_per_kb) bound. The ratio-unaware draft (b) cannot see the low payoff, keeps compressing, and saves no CPU. - High-payoff mix: the compressible half compresses hard, so compressing is right. (c) and (b) keep full compression at baseline storage; the old wrapper (a) cuts CPU 48% but inflates storage +53% -- far beyond that bound -- giving up worthwhile compression. Only (c) makes the right call on both mixes: it never gives up worthwhile compression (unlike (a)) and does not waste CPU on low-payoff data (unlike (b)). Reviewed By: xingbowang Differential Revision: D115444567 Pulled By: pdillinger fbshipit-source-id: 05e122d25de95ddcd0edeb0dd07c9d25bead7293	2026-08-11 14:29:51 -07:00
Peter Dillinger	afc922bf5f	Make ReadOptions::table_filter a caller-owned pointer for cheaper copies (#15005) ... Summary: ReadOptions is copied frequently on read paths, and its only non-trivially-copyable member, table_filter, is a std::function that is almost always empty yet still gets deep-copied on every ReadOptions copy (and can heap-allocate). This changes table_filter to a pointer to a caller-owned std::function instead of storing one by value: when non-nullptr, the caller is responsible for keeping the pointed-to function alive for the duration of the read (like the other pointer fields such as iterate_upper_bound). With this change every ReadOptions member is trivially copyable, so copying ReadOptions no longer touches a std::function at all. That trivial-copyability is locked in with a static_assert in options_test.cc (kept out of the public header). This is inspired by D113109635, which targeted the same ReadOptions-copy hotspot by relocating table_filter to the end of the struct so the trivial members memcpy in one contiguous block. This is intended to be a superior solution: rather than only defragmenting the copy, it removes the std::function copy entirely. This is a breaking API change, tolerated because table_filter is a relatively obscure option. A nullptr, or a pointer to an empty std::function, means no filtering (preserving the previous empty-function behavior). A shared internal helper HasTableFilter(const ReadOptions&) in options_helper.h centralizes the nullptr + empty-function check used on the read paths. The C API and Java bindings now own the std::function and hand ReadOptions a pointer to it; their public signatures are unchanged, and the C API rocksdb_readoptions_t clears the owned function when the callback is null. Making ReadOptions trivially copyable also let -Werror=unused-but-set-variable fire on a pre-existing dead local in db_bench's MultiScan: a ReadOptions that set async_io/readahead_size but was never passed to NewMultiScan (the warning had been suppressed while ReadOptions had a non-trivial member). That local was in fact inert for multiscan -- BlockBasedTableIterator::Prepare overrides ReadOptions::async_io with MultiScanArgs::use_async_io for the coalesced prefetch, so multiscan async I/O is driven by --multiscan_use_async_io, not ReadOptions::async_io. Removed the dead local; MultiScan now uses read_options_ directly, consistent with MultiScanRandom. Pull Request resolved: https://github.com/facebook/rocksdb/pull/15005 Test Plan: Existing table_filter coverage was adapted to the new pointer API and still exercises the behavior end to end: db_iterator_test.TableFilter (callback invoked once per table; returning false hides a table) and its range-conversion rejection case; db_bloom_filter_test SstQueryFilter/FixedWidthSegments range queries; and trie_index_db_test's range-query iterator (which returns iterators outliving the filter, exercising the caller-owned lifetime). db/c_test covers the C API set/clear/has_table_filter path. options_test asserts std::is_trivially_copyable_v<ReadOptions>. Manually verified the db_bench MultiScan async behavior with a ~200MB DB and --use_direct_reads: with ReadOptions::async_io held true, async reads tracked only --multiscan_use_async_io (rocksdb.read.async.micros count 329511 with use_async_io=true vs 0 with false), confirming ReadOptions::async_io does not drive multiscan async and the removed local had no effect. Reviewed By: anand1976 Differential Revision: D113613525 Pulled By: pdillinger fbshipit-source-id: 38d73b607500a5c7ca2b89dd8f4187ef43940aa4	2026-07-31 10:20:25 -07:00
Josh Kang	c64f682946	Fix IoUringBackend crash in ReadIOExecutor worker threads (#15020) ... Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15020 Each read executor worker creates its own `IoUringBackend`, and Folly's I/O thread pool creates its worker threads and EventBases lazily. A successful calling-thread probe or `DB::Open()` therefore does not validate every worker. A later read can start a new worker; previously, an `IoUringBackend` initialization exception on that worker could escape startup and terminate the process. The previous 512-submit setup also allowed Folly's capacity retry to reduce the completion queue to 512 while retaining a 1024-entry submission queue, which the kernel rejects with `EINVAL`. Use a 256-submit limit so the 512-entry retry remains valid. Configure the shared read executor during `DB::Open()` through `DBOptions::read_io_executor_threads`, defaulting to one thread. The backend factory catches per-worker io_uring initialization failures and uses Folly's default backend. Each worker's thread-local availability marker starts false and is set true only after `IoUringBackend` construction succeeds. When construction fails, the marker remains false and `SubmitReadAsync()` performs the read synchronously. The executor's maximum thread capacity can be raised or lowered; `db_stress` requests eight threads for async DB APIs. To expose this degradation, `FSRandomAccessFile::SubmitReadAsync()` now reports whether the asynchronous path was used. `RandomAccessFileReader` increments the `FILE_SUBMIT_ASYNC_READ_FALLBACK` ticker for every submission that falls back to synchronous I/O, making the fallback visible through the DB's `Statistics`. Reviewed By: xingbowang Differential Revision: D114031018 fbshipit-source-id: 1b7f6c1cea693e5dd743ea1ada0fca1cf757b3f9	2026-07-30 11:18:08 -07:00
zaidoon	7affaee1c4	Add use_direct_io_for_compaction_reads option (#14743) ... Summary: Adds a new `DBOption use_direct_io_for_compaction_reads` (default false). When on, compaction-input SST files are opened with `O_DIRECT` so the sequential read-once data from compaction doesn't pollute the OS page cache and evict the hot user-read working set. User reads keep going through the buffered fast path. This protects user-read tail latency on write-heavy workloads without forcing user reads onto the existing global `use_direct_reads` knob (which pays in throughput and P50 — see the bench below). The interesting bit is that just flipping the FileOptions returned by `FileSystem::OptimizeForCompactionTableRead` doesn't actually trigger `O_DIRECT` at the kernel level. The TableCache (and `FileMetaData::pinned_reader`) is already holding buffered handles opened at flush time or at `DB::Open` via `LoadTableHandlers`. When compaction asks for an iterator, it gets back the cached buffered handle and the kernel never sees the `O_DIRECT` flag. So this PR also adds a small bypass path: - `TableCache::FindTable` / `NewIterator` learn a `open_ephemeral_table_reader` mode. When set, the pinned-reader fast path and the shared cache are skipped, `GetTableReader` is called directly with the caller's FileOptions, and ownership of the freshly opened TableReader is handed back via a `unique_ptr`. The iterator takes ownership via `RegisterCleanup` and frees the reader on destruction. - `VersionSet::MakeInputIterator` and `LevelIterator` plumb the flag through both L0 and L1+ compaction-input paths. - `CompactionJob::ProcessKeyValueCompaction` turns the bypass on when `use_direct_io_for_compaction_reads` is set, the global `use_direct_reads` is off, and `OptimizeForCompactionTableRead` produced `use_direct_reads=true` in the compaction-read FileOptions. The option is opt-in: when off, nothing changes for existing users. When on, only the compaction-input opens take the bypass path; user reads keep hitting the TableCache and the buffered fast path normally. There's also a small db_bench helper in the same PR: a new `--bgwriter_num` flag that lets the writer thread in `readwhilewriting` (and the other "while writing" variants) spread its puts across `[0, bgwriter_num)` instead of `[0, num)`. Without this the readers and writer share a key range and you can't have both a hot read subset and meaningful compaction work — this lets you have both. ### Benchmark Setup: Ubuntu 24.04 (kernel 7.0.5, OrbStack Linux VM on Apple Silicon), 14 vCPUs, virtio-blk disk, btrfs. MGLRU disabled (`echo 0 > /sys/kernel/mm/lru_gen/enabled`) so the kernel uses the classic active/inactive LRU. 14 GB DB (3.5M keys × 4 KB values), no compression. Each measurement run is pinned to a 1 GB cgroup via `systemd-run --scope -p MemoryMax=1G -p MemorySwapMax=0`. Page cache is dropped between configs. db_bench is Release build. Workload: `readwhilewriting` for 120s. 4 reader threads doing random reads over a hot key subset, plus 1 writer thread spreading overwrites across the full 3.5M-key keyspace (via `--bgwriter_num=3500000`) throttled at 200 MB/s, so there's continuous compaction running while the readers go. The size of the hot reader subset relative to available page cache controls how visible the optimization is. The Cassandra blog ([Lightfoot 2026](https://lightfoot.dev/direct-i-o-for-cassandra-compaction-cutting-p99-read-latency-by-5x/)) documented the same thing: biggest wins when the hot set is big enough to actually compete for cache, smaller wins when the hot set trivially fits, neutral when the hot set is way bigger than cache. So I ran two hot-set sizes. #### Small hot set: ~30 MB (~3% of the 1 GB cgroup) — N=5 iterations, mean (CV) `--num=7500`. The hot set is small enough that the page cache holds it without much trouble even under compaction, so the wins here are real but on the modest side. | Config | Throughput (ops/s) | Read P50 (µs) | Read P99 (µs) | Read P99.9 (µs) | Read P99.99 (µs) | |---|---|---|---|---|---| | buffered (default) | 233,477 (8.2%) | 16.09 | 82.24 | 721.0 | 2,102.5 | | direct_compaction_writes_only (existing knob alone) | 287,405 (2.8%) — **+23.1%** | 13.00 (−19.2%) | **66.77 (−18.8%)** | 553.9 (−23.2%) | 1,787.6 (−15.0%) | | direct_compaction_read_only (new knob alone) | 250,669 (2.4%) — +7.4% | 14.16 (−12.0%) | 102.99 (+25.2%) | 689.8 (−4.3%) | 1,801.3 (−14.3%) | | direct_compaction_read_write (new + existing, recommended) | 277,920 (3.3%) — **+19.0%** | **12.99 (−19.3%)** | 84.23 (+2.4%) | 613.4 (−14.9%) | **1,738.2 (−17.3%)** | | use_direct_reads=true (existing global) + write-side | 249,014 (2.5%) — +6.7% | 15.95 (−0.9%) | 68.78 (−16.4%) | **450.8 (−37.5%)** | 1,814.5 (−13.7%) | CV is 2.4–3.3% on the optimized configs (8.2% on buffered), so the deltas are real. With a hot set this small, the existing `use_direct_io_for_flush_and_compaction` knob is already doing most of the work — the new flag's main extra contribution here is P99.99 (combined wins it by ~2 points vs writes-only-alone). Worth noting: the new flag *alone* (without the existing write-side flag) improves P99.99 but regresses P99 by 25% on this small-hot-set workload, because direct compaction reads lose kernel readahead and compaction-output writes are still hitting the page cache. That regression goes away once you combine with the existing write-side flag, or once the hot set is bigger (see next table). So if you're using just one knob, use the existing one. If you're using this PR's flag, pair it with `use_direct_io_for_flush_and_compaction=true`. #### Larger hot set: ~400 MB (~40% of cache) — N=5 iterations, mean (CV) `--num=100000`. This is the case the Cassandra blog calls out — hot set big enough to actually fight compaction for cache. Their analogous setup (1M hot partitions, ~33% hot/cache) reported 1.93× p99 improvement. Numbers here are the headline: | Config | Throughput (ops/s) | Read P50 (µs) | Read P99 (µs) | Read P99.9 (µs) | Read P99.99 (µs) | |---|---|---|---|---|---| | buffered (default) | 68,959 (7.7%) | 44.81 | 541.22 | 2,225.2 | 11,334.5 | | direct_compaction_writes_only (existing knob alone) | 73,973 (10.3%) — +7.3% | 42.22 (−5.8%) | 456.27 (−15.7%) | 2,016.9 (−9.4%) | 9,190.0 (−18.9%) | | direct_compaction_read_only (new knob alone) | 84,337 (2.3%) — +22.3% | 38.66 (−13.7%) | 386.97 (−28.5%) | 1,644.8 (−26.1%) | 4,837.9 (−57.3%, 2.34×) | | direct_compaction_read_write (new + existing, recommended) | **104,923 (8.4%) — +52.2%** | **34.26 (−23.5%)** | **290.97 (−46.2%)** | **1,143.4 (−48.6%)** | **3,080.3 (−72.8%, 3.68×)** | | use_direct_reads=true (existing global) + write-side | 71,598 (9.1%) — +3.8% | 51.33 (+14.5%) | 297.91 (−45.0%) | 1,663.6 (−25.2%) | 6,530.0 (−42.4%) | Combined config gets a 3.68× p99.99 win, 1.86× p99, p50 down 23%, throughput up 52%. Same shape as the Cassandra blog's 1.93× p99 result — the improvement just lands at deeper percentiles for us because RocksDB's baseline data path is roughly 40× faster than Cassandra's (their buffered p99 was 35 ms, ours is 0.54 ms), so the cache-miss tail is further out. A few things worth calling out from this table: - The new flag is doing real work on top of the existing write-side flag here, not just shifting things around. Combined throughput is +42% over `direct_compaction_writes_only` alone, and combined p99.99 is 3× better. The existing knob alone gives a fairly modest +7% throughput / -19% p99.99 in this case — there's a clear gap that the new flag fills. - The new flag *alone* (no existing write-side flag) is also a real improvement here: +22% throughput, p99.99 down 57%. The P99 regression we saw in the small-hot-set case is gone, because the cache-protection effect now dominates the lost-readahead cost. - `use_direct_reads=true` (the existing global flag) actually regresses P50 by 14.5% in this workload — taking user reads off the page cache hurts you when the hot data could have been cached. It also gets the worst throughput of any direct config. It's not an equivalent way to get these gains. ### `compaction_readahead_size` matters when this flag is on Direct I/O bypasses kernel readahead, so RocksDB's own `DBOptions::compaction_readahead_size` becomes the only prefetch the iterator has. The default of 2 MB is enough and real users will get it automatically. **But `db_bench`'s `--compaction_readahead_size` CLI default is 0**, which defeats prefetch and makes direct compaction look slower than it actually is. If you're reproducing the numbers above, pass `--compaction_readahead_size=2097152` (or larger). - Recommended production config is `use_direct_io_for_compaction_reads=true` + `use_direct_io_for_flush_and_compaction=true`. Strongest configuration at every percentile and throughput in both benches. - The new flag is the read-side counterpart to `use_direct_io_for_flush_and_compaction`, which handles compaction-write cache pollution. They address different sources of pollution and compose. The gap between "combined" and "writes-only-alone" is 17 percentage points on p99.99 in the small-hot-set bench and 54 points in the larger one, so the new flag is contributing real value, especially as the hot set grows. - The new flag alone is also a real improvement when the hot set is big enough to compete with cache (+22% throughput, 2.34× p99.99 in the larger-hot-set bench). On a very small hot set it improves p99.99 but regresses p99, so pairing with the existing write-side flag is safer. - The benefit is workload-dependent. Small hot sets get modest tail-latency wins. Hot sets sized to actually compete for cache get the big multi-percentile wins shown above. Hot sets bigger than cache (not benched here but covered in the Cassandra blog) see no change either way — every read misses regardless. ### Reproducing Any Linux host (or a Linux VM on macOS via OrbStack / Multipass / lima): ```bash sudo apt-get install -y build-essential clang cmake git pkg-config \ libgflags-dev libsnappy-dev zlib1g-dev libbz2-dev liblz4-dev libzstd-dev cmake -DCMAKE_BUILD_TYPE=Release -DPORTABLE=1 -DWITH_GFLAGS=1 -DWITH_TESTS=0 .. make -j db_bench echo 0 | sudo tee /sys/kernel/mm/lru_gen/enabled ``` Build the source DB once, unrestricted memory: ```bash ./db_bench --benchmarks=fillrandom,compact,waitforcompaction,stats \ --db=/path/to/source_db --num=3500000 --key_size=16 --value_size=4096 \ --write_buffer_size=16777216 --target_file_size_base=16777216 \ --max_background_jobs=4 --compression_type=none --cache_size=4194304 \ --max_bytes_for_level_base=67108864 --disable_wal=1 --sync=0 ``` For each config, copy `source_db -> scratch_db`, run `sync && echo 3 > /proc/sys/vm/drop_caches`, then: ```bash sudo systemd-run --scope -p MemoryMax=1G -p MemorySwapMax=0 \ ./db_bench --use_existing_db=1 \ --benchmarks=readwhilewriting,stats --db=/path/to/scratch_db \ --threads=5 --duration=120 --statistics=true --histogram=1 \ --num=7500 --bgwriter_num=3500000 \ --key_size=16 --value_size=4096 \ --write_buffer_size=16777216 --target_file_size_base=16777216 \ --max_background_jobs=4 --compression_type=none \ --cache_size=4194304 --open_files=200 \ --skip_stats_update_on_db_open=true \ --max_bytes_for_level_base=67108864 \ --benchmark_write_rate_limit=209715200 \ --compaction_readahead_size=2097152 \ --rate_limiter_bytes_per_sec=0 \ --use_direct_reads={true|false} \ --use_direct_io_for_compaction_reads={true|false} \ --use_direct_io_for_flush_and_compaction={true|false} ``` For the larger hot-set table, change `--num=7500` to `--num=100000`. The five configs in the tables: - `buffered`: all three flags false. - `direct_compaction_writes_only`: `use_direct_io_for_flush_and_compaction=true`, the other two false. This is what users have today without this PR. - `direct_compaction_read_only`: `use_direct_io_for_compaction_reads=true`, the other two false. - `direct_compaction_read_write`: `use_direct_io_for_compaction_reads=true`, `use_direct_io_for_flush_and_compaction=true`, `use_direct_reads=false`. **Recommended.** - `direct_all`: `use_direct_reads=true`, `use_direct_io_for_flush_and_compaction=true`, `use_direct_io_for_compaction_reads=false`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14743 Reviewed By: pdillinger Differential Revision: D108017601 Pulled By: xingbowang fbshipit-source-id: 4039d490d7e77b476db7a477a2f3d24738db6336	2026-06-09 17:02:53 -07:00
Xingbo Wang	76b8cce2f7	Fix spelling of memtable_verify_per_key_checksum_on_seek (#14811) ... Summary: - Rename the misspelled `memtable_veirfy_per_key_checksum_on_seek` option and related flags/config keys to `memtable_verify_per_key_checksum_on_seek`. - Update memtable option plumbing, options serialization/logging, db_bench/db_stress/crash-test flags, tests, and the option-addition guide to use the corrected name. - Keep the checksum-on-seek behavior unchanged. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14811 Test Plan: CI Reviewed By: pdillinger Differential Revision: D107382276 Pulled By: xingbowang fbshipit-source-id: 7621dc718d61503b982a7e3f65cc9293a1ad085b	2026-06-05 10:18:47 -07:00
Peter Dillinger	8053b9414f	Change default compression from Snappy to LZ4 (#14818) ... Summary: The historical default block compression `kSnappyCompression` dates to when Snappy was the obvious fast/cheap choice. On modern server CPUs LZ4 matches or beats Snappy on compression ratio while decompressing substantially cheaper, so it is a better default. This changes the default `ColumnFamilyOptions::compression` to `kLZ4Compression`, with a runtime fallback of LZ4 -> Snappy -> none depending on what is compiled into the binary (new `GetDefaultCompressionType()` in util/compression.h). Only column families that do not explicitly set `compression` are affected (including compaction output when `CompactionOptions::compression == kDisableCompressionOption`), and only newly written SST files; existing data is read as before. Doc comments, the Java bindings, and the sorted_run_builder example are updated to describe the new default. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14818 Test Plan: Adjusted two unit tests that implicitly depended on the old Snappy default to pin `compression = kSnappyCompression`: db_iterator_test's ReadAhead (readahead byte thresholds assume Snappy-sized files) and compaction_service_test's CustomFileChecksum (LSM shape after auto-compaction determined whether a manual CompactRange had work to do). This change is primarily validated by performance testing. Added a `compressreject` db_bench benchmark (output buffer sized just below the predicted compressed size) to measure the cost of attempting then declining compression, alongside `compress`/`uncompress`. Both db_bench and sst_dump compress/decompress a modest block at a time, as in a real workload. sst_dump on SST files from production workloads (4 files, 16KB blocks, single thread) on recent server-class AMD, Intel, and ARM CPUs, LZ4 vs Snappy: - Compression ratio: comparable; LZ4 is slightly smaller on the more compressible files (up to ~8%) and within ~2% on the rest. - Compression (write) CPU: a wash, within ~2% either direction. - Decompression (read) CPU: the clear win -- Snappy costs ~1.2x-1.5x as much as LZ4, i.e. LZ4 saves ~25-30% read CPU, consistently across AMD, Intel, and ARM. db_bench synthetic workload (100-byte values), at 1 / 12 / 160 threads, LZ4 vs Snappy: - Compression throughput: LZ4 ~10-30% higher. - Decompression throughput: LZ4 much higher, and the advantage grows with core count -- from ~+12% at 12 threads to ~+45-50% at 160 threads, i.e. better multi-core scaling. - Rejection (insufficient ratio) path: comparable; LZ4 ~15-18% faster on compressible-but-rejected blocks and Snappy within ~10% on barely-compressible blocks. No meaningful regression, confirming incompressible data is still efficiently detected and stored raw. Reviewed By: xingbowang Differential Revision: D107536490 Pulled By: pdillinger fbshipit-source-id: f8abaee630d782674778338148e36ed0c84e3661	2026-06-04 10:48:27 -07:00
Peter Dillinger	9ef369c606	Make StringToMap entries self-contained for direct round-trip (#14805) ... Summary: StringToMap previously stripped the outer braces from nested values, making single-entry round-trip via `key=value;` (e.g. SetOptions) silently corrupt values that contain ';'. For example, a filter_policy with sub-options serialized as filter_policy={id=ribbonfilter:10:-1;bloom_before_level=-1;} parsed to map[filter_policy] = "id=ribbonfilter:10:-1;bloom_before_level=-1;", and embedding that map entry directly into another `key=value;` string re-exposed the inner ';' as a top-level delimiter. This change is a step toward the "essential view": the outer `{...}` of a nested value is part of the value's identity. StringToMap preserves it, so each map entry is in self-contained form (a simple value or a single balanced `{...}` block) and can be embedded directly without further escaping by the caller. However some permissiveness is kept for compatibility: list-style typed parsers (ParseVector, ParseArray, kStringMap, the listener parser) and scalar dispatch in OptionTypeInfo::Parse accept either the bare or the wrapped form via a new OptionTypeInfo::StripOuterBraces helper that peels at most one outer-pair-matched layer. So braced scalar input like `key={42};` and `key={true};` still parses, and `key={};` denotes an empty list, distinct from `key={{}};` which denotes a one-element list with an empty element. A new public MapToString is added to convenience.h as the symmetric inverse of StringToMap: a trivial `key=value;` joiner that relies on each value already being self-contained (which StringToMap guarantees). Also fixed and refactored our trim() function because it would do the wrong thing for a single space. This problem was detected by expanding the StringToMapRandomTest based on code review feedback, and should only improve existing callers to trim(). The OPTIONS-file serializer code is untouched, so the on-disk format is byte-for-byte identical and format-compatibility is preserved. Bonus: fixes / clarifications to CLAUDE.md's build-system note. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14805 Test Plan: db_bloom_filter_test extends MutableFilterPolicy with the OPTIONS-file format for filter_policy to confirm that form round-trips through SetOptions. options_test updates StringToMapTest expectations for the new brace-preserving behavior, and adds: - MapToStringTest covering the new joiner. - StringToMapMapToStringRoundTripTest including a single-entry pull-and-embed scenario that previously corrupted on round-trip. - FullOptionsStringToMapRoundTripTest that drives a populated DBOptions and CFOptions through GetStringFromXxx -> StringToMap -> per-entry single round-trip -> MapToString -> GetXxxFromString -> VerifyXxxOptions, catching any custom serializer that emits a value with ';' without enclosing it in '{}'. - EmptyBracedVectorAndBracedScalarTest covering `key={};` (empty list) and braced scalar input like `key={42};`, `key={true};`. - Expanded StringToMapRandomTest with round-tripping. Some explicit trim() tests added to string_util_test.cc Format compatibility verified with SHORT_TEST=1 tools/check_format_compatible.sh Other existing tests in options_test, customizable_test, options_settable_test, listener_test, options_file_test, options_util_test, db_options_test, and compaction_service_test have extensive coverage of serialization / deserialization logic. Reviewed By: hx235, xingbowang Differential Revision: D106855466 Pulled By: pdillinger fbshipit-source-id: 872aac8d819c7b90c807b92bab85569b48fa2aa0	2026-06-01 17:07:11 -07:00
Xingbo Wang	21723bbbef	Add async WAL precreation ... Summary: - Add experimental immutable `DBOptions::async_wal_precreate` to reserve and open one future WAL on a background HIGH-priority task, with sanitization that disables the optimization when WAL recycling is configured. - Split WAL creation into open/preallocate and start phases so `SwitchMemtable()` can consume a prepared WAL after writing normal WAL metadata, wait for in-flight precreation, fall back to synchronous creation, and delete an unstarted prepared WAL on start failure. - Keep WAL numbering, close, recovery, and read-only open safe for empty future WAL files left by async precreation; `error_if_wal_file_exists=true` now rejects non-empty WALs while tolerating empty WALs. - Add public option plumbing for the C API, options parsing/stringification, random option testing, `db_bench`, `db_stress`, and crash-test configuration. - Add WAL precreate statistics counters plus Java `TickerType`/JNI mappings, and update C++, C, and Java read-only-open documentation for the empty-WAL behavior. - Add focused WAL/option/C/Java tests for async precreate ready/wait/failure/recovery paths, read-only WAL detection, option sanitization, and API plumbing, plus write-flow docs and unreleased history entries for the new feature and behavior change. PR https://github.com/facebook/rocksdb/pull/14738 Reviewed By: pdillinger Differential Revision: D105020559 fbshipit-source-id: 5059b424702e021abb8de65ceeb6d3b975280ffc	2026-05-15 10:59:47 -07:00
Anand Ananthabhotla	c734b7cc60	Add reuse_manifest_on_open DBOption (#14704) ... Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/14704 Add an immutable DBOption `reuse_manifest_on_open` (default false). When enabled, `DB::Open` can keep using the recovered MANIFEST for the first post-open metadata update instead of rebuilding a fresh MANIFEST, which can reduce warm-open latency for DBs whose MANIFEST is expensive to regenerate. Reuse is still best-effort. If RocksDB cannot safely resume appending to the recovered MANIFEST, it falls back to the existing fresh-MANIFEST path. The option is also disabled under `best_efforts_recovery`. This diff also teaches the reopened MANIFEST writer to adopt the existing file size before appending, documents the small-`max_manifest_file_size` caveat for the reused path, and keeps the full warm-reopen composition working with `optimize_manifest_for_recovery`. Reviewed By: hx235, pdillinger Differential Revision: D103568447 fbshipit-source-id: f4f5c35ea3ef0b80a0d52d94be40c6bd11505999	2026-05-07 13:10:59 -07:00
Anand Ananthabhotla	02a2b3501d	Add optimize_manifest_for_recovery DBOption (#14702) ... Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/14702 Add a mutable DBOption `optimize_manifest_for_recovery` (default false) as a temporary rollout / kill switch for warm-reopen MANIFEST optimizations. In this diff, enabling the option lets recovery skip MANIFEST updates during `DB::Open` when the recovered state is already reflected on disk, which reduces metadata appends after a clean shutdown and can lower warm-reopen latency on storage where MANIFEST appends are expensive. If the option is disabled, RocksDB follows the existing recovery path unchanged. The optimization is disabled under `best_efforts_recovery`, where recovery intentionally rewrites metadata as part of salvage, and the option is mutable so later diffs in this stack can share the same rollout knob. Reviewed By: pdillinger, hx235 Differential Revision: D103568448 fbshipit-source-id: 9ec930343e434f1bee6130bcdbd7738dddd92b6d	2026-05-07 11:44:37 -07:00
Anand Ananthabhotla	1cc216a45b	Fast file open: retrieve, persist, and pass file open metadata (#14596) ... Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/14596 When DBOptions::fast_sst_open is enabled, RocksDB retrieves opaque file system metadata for SST files after flush, compaction, and external file ingestion via FSRandomAccessFile::GetFileOpenMetadata(). This metadata is persisted in the MANIFEST using a new forward-compatible NewFileCustomTag (kFileOpenMetadata = 17), and passed back to the file system via FileOptions::file_metadata on subsequent file opens. This accelerates DB open time on remote storage systems by allowing the file system to skip expensive metadata RPCs. The feature is gated by DBOptions::fast_sst_open (default false). Everything works seamlessly regardless of the option setting, file metadata support, or presence/absence of the metadata in the MANIFEST. The MANIFEST change is backward compatible - older RocksDB versions safely ignore the new tag. Reviewed By: xingbowang Differential Revision: D100220973 fbshipit-source-id: f52de9dd853a50653b3297ab4a37a868fe41cc04	2026-04-14 19:07:29 -07:00
zaidoon	3b38fde293	Support UDI as primary index (#14547) ... Summary: Add `use_udi_as_primary_index` option to `BlockBasedTableOptions`. When enabled, the UDI becomes the primary index — all reads (including internal operations like compaction and VerifyChecksum) automatically route through the UDI without needing `ReadOptions::table_index_factory`. Both the standard binary search index and the UDI are always fully built. The standard index serves as a safety fallback (e.g., for backup/restore or rollback to a non-UDI configuration). A future refactor will extract the index abstraction to allow skipping the standard index build when the UDI is primary (see discussion below). ## Write path - `UserDefinedIndexBuilderWrapper` always forwards `AddIndexEntry` and `OnKeyAdded` to both the internal standard builder and the UDI builder - New `udi_is_primary_index` table property marks primary-mode SSTs - Validates incompatible options at `DB::Open` and builder creation: partitioned index, partitioned filters, missing `user_defined_index_factory` ## Read path - `UserDefinedIndexReaderWrapper` defaults to UDI when `udi_is_primary_`, even when `ReadOptions::table_index_factory` is null — this handles the 15+ internal call sites that don't set `table_index_factory` - `use_udi_as_primary_index` automatically enforces `fail_if_no_udi_on_open` to prevent silent data loss if SSTs are opened without UDI support ## Rollback Since the standard index is always fully populated, rollback from primary mode is straightforward: set `use_udi_as_primary_index=false`. No compaction required — SSTs written in primary mode are immediately readable through the standard index. ## Public API - `BlockBasedTableOptions::use_udi_as_primary_index` (default: false) - `UserDefinedIndexBuilder::EstimatedSize()` — pure virtual, O(1) via running counter in the trie implementation ## Bug fixes (issues https://github.com/facebook/rocksdb/issues/14560, https://github.com/facebook/rocksdb/issues/14561, https://github.com/facebook/rocksdb/issues/14562) Fixed trie index correctness bugs that caused crash test failures: - **Always-on seqno encoding**: `must_use_separator_with_seq_` is now unconditionally true. Non-boundary separators store tag=0 (sentinel), same-user-key boundaries store the real tag, and the last block stores its real last-key tag. This fixes the `NonBoundaryTag` bug where non-boundary separators with `kMaxSequenceNumber` caused the post-seek correction to incorrectly advance past the correct block. - **Standard index always built in primary mode**: An empty (stub) standard index block caused behavioral divergence in `BlockBasedTableIterator` under concurrent flush/compaction, leading to `test_batches_snapshots` prefix scan inconsistencies. ## Stress test - `use_udi_as_primary_index` flag randomized by `db_crashtest.py` - Both primary and secondary UDI modes exercised in crash tests - Trie index probability halved (~6%) per reviewer request - Re-enables trie crash tests (disabled by https://github.com/facebook/rocksdb/issues/14559) ## Tests - Parameterized `TrieIndexDBTest` on UDI mode (secondary vs primary) - New factory-level tests: non-boundary separator seek, Prev within overflow, SeekToLast with overflow, empty trie, overflow exhaustion, all-scans-exhausted - New DB-level tests: multi-CF coalescing iterator, GetEntity with explicit snapshot, reverse iteration across same-user-key blocks, non-boundary separator seek correctness, rollback from primary Pull Request resolved: https://github.com/facebook/rocksdb/pull/14547 Reviewed By: anand1976 Differential Revision: D99494181 Pulled By: xingbowang fbshipit-source-id: ca52a0d5c0e523770c80e1fe2b9b5d50406b67bc	2026-04-08 21:05:03 -07:00
Josh Kang	3f51c0a185	Convert sequential single deletes into range tombstones (#14448) ... Summary: Add a read-path optimization that converts contiguous point tombstones into range tombstones during forward/reverse iteration. When a configurable threshold of consecutive point deletions (kTypeDeletion, kTypeDeletionWithTimestamp, kTypeSingleDeletion — with no live keys between them) is detected, a range tombstone covering `[first_tombstone_key, next_live_key)` is inserted into the active mutable memtable. This benefits future iterators by enabling efficient skipping via range tombstone fragmentation. If there is a memtable switch during the read iteration, then the range deletion entry is discarded. The inserted range tombstones are logically redundant (they don't delete anything that isn't already deleted by point tombstones), skip WAL (they're a derived optimization regenerated by future reads on crash), and use the max tombstone sequence number so they don't interfere with newer writes. ## Key changes - **New option `min_tombstones_for_range_conversion`** (`AdvancedColumnFamilyOptions`): Threshold of contiguous point tombstones before converting to a range tombstone. Default 0 (disabled). Dynamically changeable via `SetOptions()`. - **`DBIter` tracking logic** (`db/db_iter.cc`): Tracks contiguous tombstones during `FindNextUserEntryInternal()` (forward) and `PrevInternal()` (reverse). When a live key terminates a run that meets the threshold, `MaybeInsertRangeTombstone()` inserts `[first_tombstone, live_key)` into the active memtable. - **`FindValueForCurrentKey` `found_visible` output** (`db/db_iter.cc`): Distinguishes "key deleted at this snapshot" from "no visible entries" so reverse tracking doesn't treat post-snapshot keys as tombstones. - **`IterKey::Swap()`** (`db/dbformat.h`): Efficiently tracks reverse tombstone run end keys without extra allocations. - **`MemTable::AddLogicallyRedundantRangeTombstone()`** (`db/memtable.cc`): Concurrent-safe range tombstone insertion into the active memtable. Range tombstone skiplist always uses concurrent inserts. - **`ConstructFragmentedRangeTombstones` race fix** (`db/db_impl/db_impl_write.cc`): Moved after `MarkImmutable()` to prevent lost entries. - **`MarkImmutable` ordering fix** (`db/memtable_list.cc`): Called before `current_->Add()` to close a race window. - **Prefix filter awareness** (`db/db_iter.cc`): Tombstone tracking scoped to the seek prefix when prefix filtering is active. See dedicated section below. - **Transaction awareness** (`db/db_iter.cc`): Tombstones with `seq > snapshot` excluded from tracking. `min_uncommitted` guard uses `insert_seq` (which may be bumped to `earliest_seq`) instead of `range_tomb_max_seq_`. See dedicated section below. - **Duplicate range check**: Skips insertion if the memtable already covers `[start, end)`. - **New statistics**: `READ_PATH_RANGE_TOMBSTONES_INSERTED` and `READ_PATH_RANGE_TOMBSTONES_DISCARDED`. - **Memtable MultiGet batch lookup** (`memtable/inlineskiplist.h`, `db/memtable.cc`): `InlineSkipList::MultiGet()` with cached search path ("finger") for sorted key lookups. - **New option `memtable_batch_lookup_optimization`** (`AdvancedColumnFamilyOptions`): Enables batch lookup for memtable MultiGet. Default false. Immutable. ## Deciding Range Tombstone Seqno - The range tombstone is inserted with `insert_seq = max(range_tomb_max_seq_, earliest_seq)` where `range_tomb_max_seq_` is the maximum sequence number across all point tombstones in the contiguous run, and `earliest_seq` is the memtable's earliest sequence number. This preserves the memtable's `earliest_seqno_` invariant. - If the iterator's snapshot sequence (`sequence_`) predates the memtable's `earliest_seq`, insertion is skipped entirely to avoid unintentionally covering entries between `sequence_` and `earliest_seq`. ## ConstructFragmentedRangeTombstones Race Fix - `MarkImmutable()` and `ConstructFragmentedRangeTombstones()` are now called before `mutex_.Lock()` in `SwitchMemtable`, keeping this work outside the DB mutex. `MarkImmutable()` blocks concurrent `AddLogicallyRedundantRangeTombstone()` calls via `immutable_mutex_`, ensuring no range tombstones are inserted after the fragmented list is built. `MarkImmutable()` is idempotent, so `MemTableList::Add()` calling it again inside the mutex is harmless. ## Prefix Filter Safety - When prefix filtering is active, the BBTI bloom filter may reject SST files outside the seek prefix, but the memtable (no bloom filter) returns keys across prefix boundaries. Tombstone tracking is scoped to the seek prefix so that converted range tombstones cannot cover live keys hidden in filtered files. - `total_order_seek=true` disables prefix filtering — all files are visible, so tombstones safely span prefix boundaries. - **Behavior change**: Seeking to an out-of-domain key with `total_order_seek=false` now treats it as total-order (prefix_ not set). When `prefix_same_as_start=true`, iterating past an out-of-domain key cleanly invalidates the iterator instead of calling `Transform()` on it (which was UB in release builds with `FixedPrefixTransform`). This is a requirement because an incorrect iterator scan could lead to a range tombstone covering a live key. ## Transaction Support - Tombstones written by the transaction's own uncommitted writes (sequence > snapshot) are now excluded from contiguous tombstone tracking entirely at the tracking site in `FindNextUserEntryInternal()` and `PrevInternal()`. Previously, tracking relied on a `min_uncommitted` check at insertion time, but this was insufficient — a transaction's own Delete with `seq > snapshot` could extend a run of committed tombstones, and the resulting range tombstone would cover data visible to other snapshots. - The fix skips any tombstone with `ikey_.sequence > sequence_` during tracking. If a transaction-owned tombstone appears mid-run, it flushes the accumulated committed run first, then resets tracking. This ensures only tombstones visible to the current snapshot are ever converted. - Both WritePrepared and WriteUnprepared transactions are supported with dedicated test coverage: - **WritePrepared**: When tombstones are committed before `Prepare()`, their seqnos are below `min_uncommitted` and insertion proceeds safely. When `Prepare()` happens first, tombstone seqnos exceed `min_uncommitted` and insertion is blocked. - **WriteUnprepared**: Multiple unprepared batches with different seqno ranges are handled correctly. Own transaction Deletes that extend a committed tombstone run block insertion of the entire run. After rollback, data correctness is verified. ## UDT support - When user-defined timestamps (UDT) are enabled, keys include an 8-byte timestamp suffix. The comparator, Put/Delete APIs, and ReadOptions all require timestamps. - Forward exhaustion with UDT: `iterate_upper_bound_` is a plain user key without a timestamp suffix. It is padded with min timestamp via `AppendKeyWithMinTimestamp()` so it sorts after all entries with this user key, preserving the exclusive bound semantics. - Reverse exhaustion with UDT: The end key comes from either the previous live key (which already has a proper timestamp suffix) or the seek target set by `SetSavedKeyToSeekForPrevTarget()` (which appends a timestamp via `SetInternalKey(..., timestamp_ub_)` and `UpdateInternalKey(..., ts)`). In both cases, the end key is already properly timestamped, so no additional padding is needed unlike forward exhaustion. - Contiguous tombstone detection works correctly with UDT because the underlying `kTypeDeletionWithTimestamp` entries are tracked the same way as `kTypeDeletion`. ## Concurrent Iterators - Should concurrent iterators happen to read the range at the same time, both will produce the same range and seqno entry. Only one will be accepted by the skip list and the others will be rejected. Future iterators will read the range and not even attempt to insert the range. - There is nothing preventing similar ranges from being inserted however. Two iterators can produce overlapping ranges, but this protection would be complicated to implement and there is no evidence that it is a likely scenario yet. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14448 Test Plan: - **Unit tests** (`db/db_iterator_test.cc`): `ReadPathRangeTombstoneTest` parameterized by forward/reverse with cases for basic insertion, non-contiguous (below threshold), memtable switch, exhausted iterator with/without bounds, direction change, mixed Delete/SingleDelete, single-delete-only runs, snapshot predating memtable, block cache tier incomplete, skip when covered by existing range, UDT basic scan, UDT exhaustion, prefix filter cross-prefix scan (`PrefixFilterCrossPrefixScanCoversLiveKey` with default/total_order_seek/prefix_same_as_start variants), stale ikey from forward-then-reverse scan (`StaleIkeyFromForwardThenReverse`), and reseek stale ikey (`ReseekStaleIkey`). - **Concurrency test** (`db/db_test2.cc`): `DBTestConcurrentRangeTombstoneConversions` parameterized by `(allow_concurrent_memtable_write, min_tombstones_for_range_conversion)` with mixed writers, deleters, range deleters, and concurrent forward/reverse readers. - **Transaction tests** (`utilities/transactions/write_prepared_transaction_test.cc`, `write_unprepared_transaction_test.cc`): Tests for WritePrepared (insertion allowed when tombstones committed before prepare, blocked when after; seqno bump shadowing prepared writes `RangeTombstoneSeqnoBumpShadowsPreparedWrite`) and WriteUnprepared (multiple batches, extended visibility with CalcMaxVisibleSeq, own deletions with rollback). - **IterKey::Swap tests** (`db/dbformat_test.cc`): `IterKeySwapTest` parameterized over `(key_len, copy, use_secondary)` × 2 covering all inline/heap/pinned/secondary combinations. - **InlineSkipList MultiGet tests** (`memtable/inlineskiplist_test.cc`): Basic, exact matches, empty, single key, randomized validation against `std::set::lower_bound`, duplicate keys with callback walk, and concurrent MultiGet with read-after-write consistency. - **Memtable MultiGet tests** (`db/db_basic_test.cc`): Batch lookup, overwrite, flush, merge, disabled by default, paranoid checks, and snapshot tests. - **Stress test coverage**: `min_tombstones_for_range_conversion` and `memtable_batch_lookup_optimization` options added to `db_crashtest.py` and `db_stress` flags. - `make check` passes all tests. ### Benchmark results Tombstones scattered randomly in clusters via `seek_nexts_to_delete` for realistic workloads. **DB Setup A (scattered deletes, 100 per seek)**: ``` # Step 1: Create and compact ./db_bench --benchmarks=fillseq,compact --seed=1 --compression_type=none --num=1000000 --db=<DB> # Step 2: Scatter tombstones (5000 seeks × 100 deletes ≈ 500k tombstones) ./db_bench --benchmarks=seekrandom,flush --seed=1 --compression_type=none --num=2000 \ --seek_nexts=0 --seek_nexts_to_delete=100 --use_existing_db=1 --threads=1 --db=<DB> ``` **DB Setup A2 (scattered deletes, 8 per seek)**: ``` # Step 1: Create and compact ./db_bench --benchmarks=fillseq,compact --seed=1 --compression_type=none --num=1000000 --db=<DB> # Step 2: Scatter tombstones (5000 seeks × 8 deletes ≈ 40k tombstones) ./db_bench --benchmarks=seekrandom,flush --seed=1 --compression_type=none --num=2000 \ --seek_nexts=0 --seek_nexts_to_delete=8 --use_existing_db=1 --threads=1 --db=<DB> ``` **DB Setup B (no deletes)**: ``` ./db_bench --benchmarks=fillseq,compact --seed=1 --compression_type=none --num=1000000 \ [--key_size=100] --db=<DB> ``` **Read workload** (same for all): ``` ./db_bench --benchmarks=seekrandom --seek_nexts=100 --threads=8 \ --reverse_iterator={true,false} --seed=1 --use_existing_db=1 \ --compression_type=none --num=1000000 --duration=10 \ --disable_auto_compactions \ [--key_size=100] [--min_tombstones_for_range_conversion=X] --db=<DB_COPY> ``` Each workload averaged over 3 runs. **Table 1: seekrandom forward, scattered deletes (2000 seeks × 100 deletes/seek)** | Variant | avg ops/s | % vs main | |---------|-----------|-----------| | main | 2,895 | - | | threshold=0 | 2,869 | -0.9% | | threshold=8 | 287,334 | +9,824% | **Table 2: seekrandom reverse, scattered deletes (2000 seeks × 100 deletes/seek)** | Variant | avg ops/s | % vs main | |---------|-----------|-----------| | main | 544 | - | | threshold=0 | 548 | +0.7% | | threshold=8 | 206,491 | +37,860% | **Table 3: seekrandom forward, scattered deletes (2000 seeks × 8 deletes/seek)** | Variant | avg ops/s | % vs main | |---------|-----------|-----------| | main | 194,049 | - | | threshold=0 | 195,703 | +0.9% | | threshold=8 | 310,740 | +60.1% | **Table 4: seekrandom reverse, scattered deletes (2000 seeks × 8 deletes/seek)** | Variant | avg ops/s | % vs main | |---------|-----------|-----------| | main | 63,854 | - | | threshold=0 | 69,266 | +8.5% | | threshold=8 | 218,101 | +241.6% | **Table 5: seekrandom forward, no deletes (regression check)** | Variant | key=16B avg ops/s | % vs main | key=100B avg ops/s | % vs main | |---------|-------------------|-----------|---------------------|-----------| | main | 330,901 | - | 236,048 | - | | threshold=0 | 328,398 | -0.8% | 238,055 | +0.9% | | threshold=8 | 332,539 | +0.5% | 233,776 | -1.0% | **Table 6: seekrandom reverse, no deletes (regression check)** | Variant | key=16B avg ops/s | % vs main | key=100B avg ops/s | % vs main | |---------|-------------------|-----------|---------------------|-----------| | main | 261,445 | - | 192,177 | - | | threshold=0 | 265,020 | +1.4% | 191,616 | -0.3% | | threshold=8 | 250,881 | -4.0% | 189,239 | -1.5% | Reviewed By: xingbowang Differential Revision: D96203950 Pulled By: joshkang97 fbshipit-source-id: 06ba66ebde3c355f04671d1e681f1b1586e8751d	2026-04-03 22:47:51 -07:00
Jialiang Tan	a0ff5b143f	Add num_data_blocks_compression_rejected/bypassed to TableProperties (#14551) ... Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/14551 Add two new per-SST TableProperties fields that track how many data blocks were stored uncompressed: - `num_data_blocks_compression_rejected`: blocks where compression was attempted but the compressed output exceeded the ratio limit set by `CompressionOptions::max_compressed_bytes_per_kb`. - `num_data_blocks_compression_bypassed`: blocks where compression was never attempted (e.g., kNoCompression type, no compressor available). Together with `num_data_blocks`, these give a full decomposition: num_data_blocks = compressed + rejected + bypassed Previously this information was only available via global Statistics tickers (NUMBER_BLOCK_COMPRESSION_REJECTED / BYPASSED) which aggregate across all SSTs. Now it is persisted per-SST in the properties block and visible via sst_dump --show_properties. Reviewed By: anand1976 Differential Revision: D99229339 fbshipit-source-id: 8333f51acff30a759d725e5e94bd80f4fcb18b57	2026-04-03 16:48:09 -07:00
Rajat Jain	64f8d5765b	Add blob_compression_opts to ColumnFamilyOptions (#14568) ... Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/14568 Port of D98326754 to internal_repo_rocksdb (GitHub-synced repo). BlobDB blob file compression currently uses hardcoded default CompressionOptions{} in BlobFileBuilder, ignoring any user-specified compression options like level, window_bits, or strategy. This means ZSTD always uses level 3 (doubleFast strategy with two hash tables), even when a lower level would significantly reduce CPU usage. This diff adds a new `blob_compression_opts` field to AdvancedColumnFamilyOptions (and MutableCFOptions) and plumbs it into BlobFileBuilder::GetCompressor(). The option is registered as mutable and dynamically changeable via SetOptions(). For example, setting level=1 with ZSTD switches from "doubleFast" (two hash tables per thread) to "fast" (single hash table), reducing L3 cache pressure on flush-heavy workloads. There was an existing TODO in blob_file_builder.cc requesting exactly this feature — this diff fulfills it. This change is safe as the default value is always dFast. So this will not impact the current code. Reviewed By: xingbowang Differential Revision: D99478562 fbshipit-source-id: e79b847a854f60dea552442b18fd3f1384efac70	2026-04-03 16:42:16 -07:00
Xingbo Wang	4cf28450a5	Add customizable blob partition strategy for blob direct write (#14565) ... Summary: This PR adds a `BlobFilePartitionStrategy` interface that allows users to plug in custom partition selection logic for blob direct writes. The default behavior (round-robin across partitions) is preserved as the built-in `RoundRobinBlobFilePartitionStrategy`. ### Motivation The existing blob direct write implementation distributes writes across partitions using a fixed round-robin strategy (atomic counter mod num_partitions). While this works well for uniform workloads, some use cases benefit from custom routing: - **Key-based routing**: co-locate related keys in the same blob partition for locality-aware reads. - **CF-based routing**: direct writes for different column families to dedicated partitions. - **Value-size routing**: send large vs. small values to different partitions. - **Application-defined affinity**: any domain-specific grouping that the default round-robin cannot express. ### Design A new pure-virtual interface `BlobFilePartitionStrategy` is added to `include/rocksdb/advanced_options.h`: ```cpp class BlobFilePartitionStrategy { public: virtual ~BlobFilePartitionStrategy() = default; // Select a partition for the given blob direct write. // The return value can be any uint32_t; the caller applies // modulo num_partitions internally. // Implementations must be thread-safe. virtual uint32_t SelectPartition(uint32_t num_partitions, uint32_t column_family_id, const Slice& key, const Slice& value) = 0; }; ``` The strategy receives: - `num_partitions`: the configured partition count (for implementations that want to be partition-count-aware). - `column_family_id`: useful for CF-based routing. - `key` / `value`: the key and value being written. The return value is taken modulo `num_partitions` internally, so implementations can return any `uint32_t` without worrying about bounds. A new column family option `blob_direct_write_partition_strategy` (type `std::shared_ptr<BlobFilePartitionStrategy>`, default `nullptr`) wires the strategy into `BlobFilePartitionManager`. When `nullptr`, the built-in `RoundRobinBlobFilePartitionStrategy` is used, preserving existing behavior exactly. ### Changes | File | Change | |---|---| | `include/rocksdb/advanced_options.h` | New `BlobFilePartitionStrategy` interface; new `blob_direct_write_partition_strategy` option | | `db/blob/blob_file_partition_manager.h/.cc` | Accept strategy in constructor; replace atomic counter with strategy call; move `RoundRobinBlobFilePartitionStrategy` here as default | | `options/cf_options.h/.cc` | Thread strategy through `ImmutableCFOptions` | | `options/options_helper.cc` | Propagate strategy in `UpdateColumnFamilyOptions` | | `options/options_settable_test.cc` | Register new option field in options settability test | | `db/db_impl/db_impl.cc` | Pass strategy when constructing `BlobFilePartitionManager` | | `db/blob/db_blob_direct_write_test.cc` | New test with `FixedBlobDirectWritePartitionStrategy` | Pull Request resolved: https://github.com/facebook/rocksdb/pull/14565 Test Plan: New test `DirectWriteCustomPartitionStrategyRoutesWritesToOneBlobFile` verifies that a `FixedBlobDirectWritePartitionStrategy` that always returns partition 3 correctly routes all writes to a single blob file across 4 configured partitions. ``` make -j128 db_blob_direct_write_test && ./db_blob_direct_write_test --gtest_filter='*CustomPartition*' make -j128 db_blob_direct_write_test && ./db_blob_direct_write_test ``` Reviewed By: joshkang97 Differential Revision: D99458813 Pulled By: xingbowang fbshipit-source-id: 54cfbb0f75e24b58a8db61ad8755204d0db6ece7	2026-04-03 15:27:57 -07:00
Xingbo Wang	af4e32945b	Blob direct write v1: write-path blob separation with partitioned files (reduced scope) (#14535) ... Summary: This PR introduces **blob direct write v1**, a reduced-scope write-path optimization where large values (>= `min_blob_size`) are written directly to blob files during `Put()` and replaced in the memtable with compact `BlobIndex` references. This avoids holding full values in memory until flush time. ### Motivation In the existing BlobDB architecture, values are written to the WAL and memtable in their full form and separated into blob files only at flush time. This means: - Large values are held in memory twice (raw in memtable + blob file at flush) - Blob I/O is serialized through a single flush thread per column family Blob direct write addresses both: values leave the write path as small `BlobIndex` references, and multiple **partitions** (configurable via `blob_direct_write_partitions`) allow concurrent blob writes with independent locks. ### Design (v1 — single-writer, WAL-disabled, reduced scope) The v1 design intentionally keeps scope narrow for correctness and reviewability: - **Single writer thread assumption**: no concurrent writes to the same partition file. One logical writer serializes the batch. - **WAL-disabled**: direct-write blob files are only registered in MANIFEST at flush time. WAL replay cannot recover unregistered blob references, so WAL is disabled for this v1. - **Flush-on-write**: each `AddRecord` call flushes to the OS immediately. - **FIFO generation batching**: each memtable switch creates one generation batch. Direct-write files for that memtable are sealed and registered atomically when the batch is flushed to MANIFEST. - **Round-robin partitions**: blob writes are distributed across `blob_direct_write_partitions` files using an atomic counter. ### New components | Component | Description | |---|---| | `BlobFilePartitionManager` | Owns N partition files per CF. Manages open/seal/register lifecycle tied to memtable generations. | | `BlobWriteBatchTransformer` | A `WriteBatch::Handler` that rewrites qualifying `Put` values as `BlobIndex` entries before the batch enters the write group. | ### Write path integration 1. `DBImpl::WriteImpl` calls `BlobWriteBatchTransformer::TransformBatch` before entering the writer group (for default write path), or before joining the batch group (for pipelined/unordered write). 2. Values >= `min_blob_size` are written to a partition file; the key is stored with a `BlobIndex` in the transformed batch. A rollback guard marks blob bytes as initial garbage if the write fails. 3. On `SwitchMemtable`, `RotateCurrentGeneration` moves active partitions into the next immutable batch. 4. `FlushMemTableToOutputFile` / `AtomicFlushMemTablesToOutputFiles` call `PrepareFlushAdditions` to seal partition files and collect `BlobFileAddition` + `BlobFileGarbage` entries registered to MANIFEST alongside the flush. 5. Shutdown paths (`CancelAllBackgroundWork`, `WaitForCompact` with `close_db=true`) force-flush all CFs with active direct-write managers to ensure blob files are registered before close. ### Read path - **Get/MultiGet**: `MaybeResolveBlobForWritePath` resolves `BlobIndex` references found in memtable or immutable memtable via `BlobFilePartitionManager::ResolveBlobDirectWriteIndex`, which first checks manifest-visible state and falls back to direct blob-file reads via `BlobFileCache`. - **Iterator**: `DBIter::BlobReader` is extended with a `BlobFilePartitionManager*` to resolve direct-write blob indexes during iteration. The unified `ResolveBlobDirectWriteIndex` path handles both manifest-visible and not-yet-flushed files. ### New options | Option | Default | Description | |---|---|---| | `enable_blob_direct_write` | `false` | Enable write-path blob separation for this CF. Requires `enable_blob_files = true`. Not dynamically changeable. | | `blob_direct_write_partitions` | `1` | Number of parallel partition files per CF. Not dynamically changeable. | ### Feature incompatibilities (reduced v1 scope) The following features are *not supported* when `enable_blob_direct_write = true`, and are enforced both in `db_stress_tool` validation and `db_crashtest.py` sanitization: **Write model constraints:** - `threads` must be 1 (single writer assumption) - `allow_concurrent_memtable_write` = 0 - `enable_pipelined_write` = 0 (transformation done before batch group, but pipelined path supported with pre-transform) - `two_write_queues` = 0 - `unordered_write` = 0 (transformation done before batch group, but unordered path supported with pre-transform) **WAL and recovery:** - `disable_wal` = 1 (required — WAL replay of unregistered blob files is out of v1 scope) - `best_efforts_recovery` = 0 - `reopen` = 0 (no crash-restart with WAL replay) - All WAL-related stress features disabled: `manual_wal_flush_one_in`, `sync_wal_one_in`, `lock_wal_one_in`, `get_sorted_wal_files_one_in`, `get_current_wal_file_one_in`, `track_and_verify_wals`, `rate_limit_auto_wal_flush`, `recycle_log_file_num` **Blob GC and dynamic options:** - `use_blob_db` = 0 (stacked BlobDB not supported) - `allow_setting_blob_options_dynamically` = 0 - `enable_blob_garbage_collection` = 0 - `blob_compaction_readahead_size` = 0 - `blob_file_starting_level` = 0 **Unsupported value types and APIs:** - Merge (`use_merge`, `use_full_merge_v1`) — merge values pass through untransformed - Entity APIs (`use_put_entity_one_in`, `use_get_entity`, `use_multi_get_entity`, `use_attribute_group`) - `use_timed_put_one_in` - User-defined timestamps (`user_timestamp_size`, `persist_user_defined_timestamps`, `create_timestamped_snapshot_one_in`) - Transactions (`use_txn`, `use_optimistic_txn`, `test_multi_ops_txns`, `commit_bypass_memtable_one_in`) — though `WriteCommittedTxn::CommitInternal` falls back from bypass-memtable to normal path when BDW is active - `IngestWriteBatchWithIndex` returns `NotSupported` - `inplace_update_support` = 0 **Fault injection:** - All write/read/metadata fault injection disabled (`sync_fault_injection`, `write_fault_one_in`, `metadata_write_fault_one_in`, `read_fault_one_in`, `metadata_read_fault_one_in`, `open_*_fault_one_in`) **Infrastructure/snapshot APIs:** - `remote_compaction_worker_threads` = 0 - `test_secondary` = 0 - `backup_one_in` = 0 - `checkpoint_one_in` = 0 - `get_live_files_apis_one_in` = 0 - `ingest_external_file_one_in` = 0 - `ingest_wbwi_one_in` = 0 ### Tests - `db/blob/db_blob_basic_test.cc`: ~660 lines of new direct-write unit tests covering basic put/get, multi-partition, flush/compaction, recovery, and error injection. - `db/blob/blob_file_cache_test.cc`: ~96 lines of new tests for direct-write blob file cache behavior. - `db/write_batch_test.cc`: ~96 lines of tests for WriteBatch with blob index entries. - `utilities/transactions/transaction_test.cc`: verifies transaction commit path falls back correctly with direct write enabled. - `db_stress_tool/`: full stress test support with `--enable_blob_direct_write` and `--blob_direct_write_partitions` flags, integrated into `db_crashtest.py` with 10% random selection alongside regular blob params. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14535 Test Plan: ``` make -j128 db_blob_basic_test && ./db_blob_basic_test make -j128 blob_file_cache_test && ./blob_file_cache_test make -j128 write_batch_test && ./write_batch_test make -j128 transaction_test && ./transaction_test make -j128 check ``` Stress test: ``` python3 tools/db_crashtest.py blackbox --enable_blob_direct_write=1 \ --enable_blob_files=1 --blob_direct_write_partitions=4 \ --disable_wal=1 --threads=1 ``` Reviewed By: pdillinger Differential Revision: D98766843 Pulled By: xingbowang fbshipit-source-id: 1577653826913a59d05680a87bce5534ac5a5e69	2026-04-02 07:31:56 -07:00
Josh Kang	f897c1b2ef	Add uniform block tracking stats (#14513) ... Summary: Add tracking of uniform index blocks as a table property (`num_uniform_blocks`). When `uniform_cv_threshold` is set, the block builder detects uniformly distributed keys via coefficient of variation of key gaps. This information is now surfaced end-to-end: from block building through index builders to SST table properties. ## Key changes - **BlockBuilder**: Persist the uniformity result from `ScanForUniformity()` as a member (`is_uniform_`) exposed via `IsUniform()`, rather than a local variable discarded after `Finish()`. - **Index builders**: All three index builder types (`ShortenedIndexBuilder`, `HashIndexBuilder`, `PartitionedIndexBuilder`) implement `NumUniformIndexBlocks()`. For partitioned indexes, the count accumulates across partition `Finish()` calls. - **Table property serialization**: Added `TablePropertiesNames::kNumUniformBlocks` (`"rocksdb.num.uniform.blocks"`) with full serialization/deserialization support. Without this, the property was computed in memory but never persisted to SST files. - **Table property aggregation**: `num_uniform_blocks` included in `Add()` and `GetAggregatablePropertiesAsMap()` for correct cross-SST aggregation. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14513 Test Plan: - **Unit tests**: `block_test` (7504 passed), `table_test` (6917 passed), `table_properties_collector_test` (8 passed). - **End-to-end verification via db_bench + sst_dump**: - Positive: `./db_bench --benchmarks=fillseq,compact --num=10000 --uniform_cv_threshold=0.2 --index_shortening_mode=0 --compression_type=none` produces SST with `# uniform blocks: 1`. - Negative: Same with `--uniform_cv_threshold=-1` (disabled) produces `# uniform blocks: 0`. Reviewed By: xingbowang Differential Revision: D98343170 Pulled By: joshkang97 fbshipit-source-id: ed08e83b2dcfb70f074bfb0f7bb5c31d75dc6da9	2026-03-30 11:58:45 -07:00
Josh Kang	5db0603613	Read-triggered compactions (#14426) ... Summary: Add read-triggered compaction, a new feature that reduces read amplification by compacting SST files that receive high read traffic. When an SST file's read frequency (`num_reads_sampled / file_size`) exceeds a configurable threshold, it is marked for compaction to a lower level. The feature introduces two new options: a CF option `read_triggered_compaction_threshold` (default 0, disabled) and a DB option `max_periodic_compaction_trigger_seconds` (default 43200s) that controls how often the background thread re-evaluates compaction scores on quiet databases. Both options are dynamically changeable. Lowering `max_periodic_compaction_trigger_seconds` does add some overhead, but generally is minimal, so running this every couple of minutes in a production environment seems fairly reasonable. ## Key changes - **New CF option `read_triggered_compaction_threshold`** (`advanced_options.h`): When positive, files with `reads_per_byte > threshold` are marked for compaction. Files at the last non-empty level are skipped (bottommost compaction handles those separately). Marked files are sorted by hotness (reads_per_byte descending). - **New DB option `max_periodic_compaction_trigger_seconds`** (`options.h`): Replaces the hardcoded 12-hour ceiling in `ComputeTriggerCompactionPeriod()`. Essential for read-triggered compaction on quiet DBs since there are no writes to trigger score re-evaluation. - **Leveled compaction picker** (`compaction_picker_level.cc`): Adds read-triggered as the lowest-priority compaction reason in `SetupInitialFiles()`, using the existing `PickFileToCompact` helper. - **Universal compaction picker** (`compaction_picker_universal.cc`): Adds `PickReadTriggeredCompaction` as lowest priority. Refactors shared "find output level + compute overlapping inputs + create Compaction" logic from both `PickDeleteTriggeredCompaction` and `PickReadTriggeredCompaction` into `BuildCompactionToNextLevel`, handling both single-level and multi-level universal cases. - **Periodic trigger integration** (`db_impl.cc`): `TriggerPeriodicCompaction` now also fires for CFs with `read_triggered_compaction_threshold > 0`, even without time-based compaction configured. - **Stress test & db_bench support**: Both `db_stress` and `db_bench` support the new options. `db_crashtest.py` randomly enables read-triggered compaction and sets a short periodic trigger interval when enabled. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14426 Test Plan: **Unit tests**: - `compaction_picker_test` — 7 new tests: `ReadTriggeredCompactionDisabled`, `ReadTriggeredCompactionBelowThreshold`, `ReadTriggeredCompactionAboveThreshold`, `NeedsCompactionReadTriggered`, `ReadTriggeredPicksFile`, `UniversalReadTriggeredCompaction`, `ReadTriggeredSkipsLastLevel`, `UniversalReadTriggeredNoPickWhenNotMarked` - `db_compaction_test` — `ReadTriggeredCompaction` integration test verifying end-to-end behavior with sync points - Stress test coverage **Stress test**: ``` make V=1 -j "CRASH_TEST_EXT_ARGS=--duration=600 --max_key=2500000 --max_compaction_trigger_wakeup_seconds=10 --read_triggered_compaction_threshold=0.0001 --interval=600" blackbox_crash_test ``` - confirmed read triggered compactions from LOGS **Benchmark** (`db_bench`): Setup: 5M keys (100B values, 16B keys), leveled compaction, 5 levels, 4MB target file size. DB fully compacted, then 2M overlapping keys written without compaction to create L0/L1 overlap (82 files, ~294MB). LSM shape change during readrandom with read-triggered compaction: ``` BEFORE: L0=9 files (15MB), L1=4 (16MB), L2=20 (69MB), L3=49 (194MB) — 82 files, 294MB AFTER: L3=66 files (223MB) ``` | Benchmark | Config | avg ops/s | % change | |-----------|--------|-----------|----------| | readrandom (8 threads, 5M reads) | baseline (threshold=0) | 1,086,965 | — | | readrandom (8 threads, 5M reads) | threshold=0.000001, trigger=5s | 1,453,697 | **+33.7%** | Reviewed By: xingbowang Differential Revision: D97838716 Pulled By: joshkang97 fbshipit-source-id: a21fcb270c7fadd4f78d98b9c821982f220dd3f0	2026-03-27 14:43:52 -07:00
Danny Chen	b6f498b2c9	Add verify_manifest_content_on_close option (#14451) ... Summary: Add a new mutable DB option `verify_manifest_content_on_close` (default: false). When enabled, on DB close the MANIFEST file is read back and all records are validated (CRC checksums via log::Reader and logical content via VersionEdit::DecodeFrom). If corruption is detected, a fresh MANIFEST is written from in-memory state using the existing LogAndApply recovery path. This complements the existing size validation in VersionSet::Close() with content validation, reusing the same manifest reading pattern as VersionSet::Recover(). Implementation plan: ## Part 1: New DB Option — verify_manifest_content_on_close - A new mutable bool DB option (default: false) that can be dynamically toggled via SetDBOptions() at runtime, following the pattern of other mutable manifest options like max_manifest_file_size. - Propagation: SetDBOptions() -> DBImpl::mutable_db_options_ -> versions_->UpdatedMutableDbOptions() -> VersionSet::verify_manifest_content_on_close_ ## Part 2: Core Implementation — Content Validation in VersionSet::Close() - Inserted after existing size check, before closed_ = true - Opens manifest as SequentialFileReader, creates log::Reader with checksum=true - Loops ReadRecord with WALRecoveryMode::kAbsoluteConsistency, decodes each record as VersionEdit - On corruption: fires OnIOError listeners, logs error, calls LogAndApply with empty edit to trigger manifest rewrite from in-memory state - If manifest can't be opened for reading: logs warning, doesn't fail close ## Part 3: Unit Tests (in version_set_test.cc) - ManifestContentValidationOnClose_Clean: enable option, normal close, verify no manifest rotation - ManifestContentValidationOnClose_CorruptRecord: enable option, corrupt manifest via SyncPoint, verify rotation occurs and DB reopens cleanly - ManifestContentValidationOnClose_Disabled: default off, verify content validation does not run - ManifestContentValidationOnClose_SizeCheckFails: truncate manifest so size check fails first, verify recovery via size-check path ## What Happens If a Corruption is Detected If corruption was detected, four things happen: 1. **Notify listeners** — Fires `OnIOError` on all registered event listeners (from db_options_->listeners) so monitoring/alerting systems can observe the corruption event. Uses `FileOperationType::kVerify` to categorize it. 2. **Permit unchecked errors** — `PermitUncheckedError()` silences RocksDB's debug-mode assertion that every `IOStatus` must be inspected. These statuses are informational-only here; the real recovery is via `LogAndApply`. 3. **Log the error** — Writes a `ROCKS_LOG_ERROR` message with the filename for operational visibility (grep-able in production logs). 4. **Rewrite the manifest via `LogAndApply`** — This is the actual recovery. `LogAndApply` is called with an empty `VersionEdit` (no changes). Internally, `LogAndApply` detects that the current `descriptor_log_` is null (it was reset at line 5551, or by the previous `LogAndApply` in the size-check path) and creates a brand-new MANIFEST file. It serializes the entire current in-memory LSM state — all column families, all levels, all file metadata, sequence numbers, etc. — into this new file. It then atomically updates the `CURRENT` file pointer to reference the new MANIFEST. This works because the in-memory state was built from the original manifest during `DB::Open()` and has been kept fully up to date through all subsequent operations (flushes, compactions, etc.) during the DB's lifetime. The on-disk manifest is essentially a journal of changes; `LogAndApply` with an empty edit produces a fresh, compacted snapshot of that state. ## Flow Diagram of Manifest Content Validation VersionSet::Close() │ ├─ Close descriptor_log_ and check size │ └─ Size mismatch? → LogAndApply (rewrite manifest) │ ├─ Content validation (if s.ok() && option enabled) │ ├─ Open manifest for sequential reading │ │ └─ Can't open? → WARN log, continue │ │ │ ├─ For each record: │ │ ├─ ReadRecord (CRC32 check, kAbsoluteConsistency) │ │ └─ DecodeFrom (VersionEdit logical check) │ │ │ └─ Corruption detected? │ ├─ Notify OnIOError listeners │ ├─ LOG_ERROR │ └─ LogAndApply (rewrite manifest from in-memory state) │ └─ closed_ = true; return s; ## How This Relates to the Existing Size Check The existing size check (lines 5556-5582) and the new content validation are complementary: | Check | What it catches | How it checks | |----------------|-----------------------------------------|----------------------------| | Size check | Truncation, partial writes, extra bytes | Compare expected vs actual file size | | Content check | Bit-rot, silent corruption, bad records | CRC32 + VersionEdit decode | The size check catches gross corruption (file too short or too long). The content check catches subtle corruption where the file is the right size but individual bytes have been flipped (e.g., storage media bit-rot, buggy filesystem, incomplete block write). Both recovery paths use the same mechanism: `LogAndApply` with an empty `VersionEdit` to rewrite the manifest from in-memory state. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14451 Reviewed By: xingbowang Differential Revision: D96004906 Pulled By: dannyhchen fbshipit-source-id: 0b0ecdada3a74e97d2cadbba2091b8b577f1d684	2026-03-19 12:01:23 -07:00
anand76	c66c14258f	Add memtable MultiGet finger search optimization (#14428) ... Summary: Add memtable batch lookup optimization with finger search Optimize memtable MultiGet by using a finger search on the skip list. After finding key[i], the search path (Splice) is retained as a "finger" for key[i+1]. The next search walks up the finger until the forward pointer overshoots, then descends -- costing O(log d) where d is the distance between consecutive sorted keys, rather than O(log N) from the head each time. Controlled by the new `memtable_batch_lookup_optimization` column family option (default: false). ## Changes - Add `FindGreaterOrEqualWithFinger()` and `MultiGet()` to `InlineSkipList` with optional paranoid validation support (key ordering checks and per-key checksum verification via default parameters) - Add virtual `MultiGet()` to `MemTableRep`, override in `SkipListRep` - Add `memtable_batch_lookup_optimization` CF option - Integrate finger search into `MemTable::MultiGet` -- sorts keys, performs batched finger search, then unscrambles results - Add `db_bench`, `db_stress`, and crash test support - Add unit tests for `InlineSkipList::MultiGet` (5 tests) and integration tests at the DB level (25 `BatchLookup` tests), including paranoid validation tests ## Benchmark Results Setup: 2M keys in memtable, release build (`DEBUG_LEVEL=0`). ### Single-threaded `multireadrandom` | Batch Size | Baseline | BatchOpt | vs Base | |:---:|---:|---:|---:| | 2 | 363,593 | 376,562 | **+3.6%** | | 8 | 385,204 | 386,082 | **+0.2%** | | 32 | 360,339 | 375,105 | **+4.1%** | | 64 | 352,696 | 378,497 | **+7.3%** | ### Multithreaded `multireadrandom` (batch_size=64) | Threads | Baseline | BatchOpt | vs Base | |:---:|---:|---:|---:| | 4 | 171,356 | 185,633 | **+8.3%** | | 8 | 161,478 | 171,392 | **+6.1%** | ### `multireadwhilewriting` (batch_size=64) | Threads | Baseline | BatchOpt | vs Base | |:---:|---:|---:|---:| | 1 | 163,938 | 175,068 | **+6.8%** | | 4 | 109,698 | 120,790 | **+10.1%** | | 8 | 116,623 | 123,658 | **+6.0%** | No regression at any batch size or thread count. The finger is stack-allocated per `MultiGet` call, so there is no shared state or cache-line contention between threads. Concurrent writes don't invalidate the finger's bracket since it only reads via acquire-load `Next()` pointers. Small Batch Size Regression Check: memtable_batch_lookup_optimization ===================================================================== All values are ops/sec. Measured with db_bench (release build). 2M keys in memtable, value_size=100, duration=30s per run. Multithreaded multireadrandom — small batch sizes -------------------------------------------------- Threads | Batch Size | Baseline | BatchOpt | Change 4 | 2 | 532,302 | 540,682 | +1.6% 8 | 2 | 1,044,046 | 1,046,920 | +0.3% 4 | 8 | 633,733 | 630,039 | -0.6% 8 | 8 | 1,260,818 | 1,241,792 | -1.5% multireadwhilewriting — small batch sizes ----------------------------------------- Threads | Batch Size | Baseline | BatchOpt | Change 1 | 2 | 107,284 | 105,217 | -1.9% 4 | 2 | 428,508 | 423,747 | -1.1% 8 | 2 | 854,654 | 864,923 | +1.2% 1 | 8 | 114,788 | 118,496 | +3.2% 4 | 8 | 467,995 | 473,437 | +1.2% 8 | 8 | 935,636 | 960,791 | +2.7% No regression at small batch sizes. Variations at batch_size=2 are within noise (~1-2%). At batch_size=8, modest positive trend (+1-3% in read-while-writing). Pull Request resolved: https://github.com/facebook/rocksdb/pull/14428 Test Plan: - `make check` -- all tests pass - `ASSERT_STATUS_CHECKED=1 make check` -- all tests pass - 2-hour `db_crashtest.py blackbox` with `--memtable_batch_lookup_optimization=1` -- passed - 2-hour `db_crashtest.py blackbox` with `--memtable_batch_lookup_optimization=1 --paranoid_memory_checks=1` -- passed Reviewed By: pdillinger Differential Revision: D95813786 Pulled By: anand1976 fbshipit-source-id: b182a023e1026021f1b9682d1cc024c7729326c7	2026-03-16 10:45:49 -07:00
Jay Huh	5494bc1d76	Add option to verify file checksum of output files (#14433) ... Summary: One of the follow ups from https://github.com/facebook/rocksdb/pull/14103. Users will have the option to verify file checksums for all compaction output files before they are installed. This feature helps prevent corrupted SST files from being added to the LSM tree. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14433 Test Plan: Unit test added Reviewed By: archang19 Differential Revision: D95648000 Pulled By: jaykorean fbshipit-source-id: 512f3c1a7449b96a660865531f3537624c89a9cc	2026-03-11 21:14:28 -07:00
Josh Kang	3ad23b2d94	Support automated interpolation search (#14383) ... Summary: Add automatic per-block interpolation search selection (`kAuto` mode) for index blocks. During SST construction, each index block's key distribution is analyzed using the coefficient of variation (CV) of gaps between restart-point keys. Blocks with uniformly distributed keys are flagged via a new bit in the data block footer, and at read time, `kAuto` resolves to interpolation search for uniform blocks and binary search otherwise. ## Key changes - **New `BlockSearchType::kAuto` enum value**: Resolves per-block at read time to either `kInterpolation` or `kBinary` based on the block's uniformity flag. Falls back to `kBinary` on older versions that don't recognize it. - **Write-path uniformity analysis**: `BlockBuilder::ScanForUniformity()` uses Welford's online algorithm to incrementally compute the CV of key gaps at restart points. The result is stored in a new bit (bit 30) of the data block footer's packed restart count. - **New table option `uniform_cv_threshold`** (default: -1 `disabled`): Controls how strict the uniformity check is. Set to negative to disable. Exposed in C++, Java (JNI), and `db_bench`. - **Code reorganization**: Block entry decode helpers (`DecodeEntry`, `DecodeKey`, `DecodeKeyV4`, `ReadBe64FromKey`) moved from `block.cc` to a new shared header `block_util.h` so they can be reused by `BlockBuilder` on the write path. - **New histogram `BLOCK_KEY_DISTRIBUTION_CV`**: Records the CV (scaled by 10000) of each index block's key distribution for observability. - **Java bindings**: `IndexSearchType.kAuto`, `uniformCvThreshold` getter/setter, JNI portal constructor signature updated, and `HistogramType.BLOCK_KEY_DISTRIBUTION_CV` added. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14383 Test Plan: - `IndexBlockTest.IndexValueEncodingTest` parameterized to include `kAuto` search type alongside `kBinary` and `kInterpolation`, verifying correct seek/iteration behavior across all combinations of key distributions, restart intervals, and key lengths. - Uniformity detection validated: blocks with uniform key distribution correctly set `is_uniform = true`, blocks with clustered/non-uniform keys set `is_uniform = false`. - Stress test coverage - Updated check_format_compatible to also include a "uniform" dataset. By default using uniform_cv_threshold=-1 does not result in an incompatibility issues. When manually changing the threshold (e.g. `uniform_cv_threshold=1000`), I see `bad block contents`, which is expected ## Benchmark readrandom with `fillrandom,compact -seed=1 --statistics`: | Benchmark | Branch | Params | avg ops/s | % change vs main | CV P50 | |-----------|--------|--------|-----------|------------------|--------| | readrandom | main | `binary_search, shortening=1` | 335,791 | baseline | N/A | | readrandom | feature | `binary_search, shortening=1` (default) | 335,749 | -0.0% | 1,500 | | readrandom | feature | `auto_search, shortening=1` (kAuto) | 366,832 | **+9.2%** | 1,500 | | readrandom | feature | `interpolation_search, shortening=1` | 366,598 | **+9.2%** | 1,500 | | readrandom | feature | `auto_search, shortening=2` (kAuto) | 344,631 | **+2.6%** | 1,030,000 | | readrandom | feature | `interpolation_search, shortening=2` | 201,178 | **-40.1%** | 1,030,000 | As seen with shortening=2, a non-uniform distribution produces a high CV, which does not use interpolation search. ## Write benchmark There is a write overhead which scans each restart entry for a block upon Finish. In practice this is very low because currently it is only applied to index blocks. See cpu profile (https://fburl.com/strobelight/io5hwj9h) here of `-benchmarks=fillseq,compact -compression_type=none -disable_wal=1`. Only 0.08% attributed to `ScanForUniformity`. Reviewed By: pdillinger Differential Revision: D94738890 Pulled By: joshkang97 fbshipit-source-id: 9661ac593c5fef89d49f3a8a027f1338a0c96766	2026-03-06 10:13:51 -08:00
Jay Huh	3aa706c2bf	Enforce WriteBufferManager during WAL recovery (#14305) ... Summary: Add a new immutable DB option `enforce_write_buffer_manager_during_recovery` to control whether WriteBufferManager buffer_size is enforced during WAL recovery. When multiple RocksDB instances share a WriteBufferManager, a recovering instance could exceed the global memory limit by replaying large amounts of WAL data into memtables. This can cause OOM, especially when other instances are actively using the shared memory budget. When this option is enabled (and a WriteBufferManager is configured), RocksDB will check `WriteBufferManager::ShouldFlush()` after each batch insertion during WAL recovery and schedule flushes when needed to keep memory bounded. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14305 Test Plan: Two unit tests added - `WriteBufferManagerLimitDuringWALRecoverySingleDB` and `WriteBufferManagerLimitDuringWALRecoveryMultipleDBs` ``` ./db_write_buffer_manager_test --gtest_filter="*WriteBufferManagerLimitDuringWALRecovery*" ``` Basic stress test added with options toggled ## To follow up Multiple DB scenario in stress test Reviewed By: hx235 Differential Revision: D92533792 Pulled By: jaykorean fbshipit-source-id: 35ca70e2300a8bfd6b81a6646a65ef284fb90a9a	2026-03-05 15:56:46 -08:00
Josh Kang	f25fb41da6	Add option to validate sst files in the background on DB open (#14322) ... Summary: Add `open_files_async` option for faster DB startup. When enabled, SST file opening and validation is deferred to a background thread after `DB::Open` returns, reducing startup latency for databases with many SST files. WAL recovery remains synchronous. To support this, `FindTable` is extended with a pinning mechanism that stores the cache handle directly on `FileMetaData` via a new `PinnedTableReader` class, and sets the table reader atomically so subsequent reads skip cache lookups. `FileDescriptor::table_reader` is replaced with `PinnedTableReader pinned_reader` which wraps a `std::atomic<TableReader*>` with acquire/release ordering to safely handle concurrent access between the background opener and read threads. Should validations fail, the background opener sets a `kAsyncFileOpen` background error. Future read requests will look up the table reader again via the cache, and if any validations fail there it will get propagated to the user (existing behavior when `max_open_files > 0`). This feature is most useful when `max_open_files=-1`, because otherwise file opening is already capped at 16 files and DB open should be fast. ## Restrictions - This feature also is incompatible with fifo compaction because fifo compaction requires reading table properties under DB mutex. When table reader is unpinned, this may cause a DB hang. - This feature is also incompatible with `skip_stats_update_on_db_open=false` because it will result in even longer DB open ## Key changes - New `open_files_async` DB option with C, Java, and `db_bench` bindings - `BGWorkAsyncFileOpen` background worker that opens all SST files post-`DB::Open`, with shutdown awareness via `shutting_down_` flag - New `PinnedTableReader` class in `version_edit.h` — thread-safe wrapper holding `std::atomic<TableReader*>` and `Cache::Handle*` with proper acquire/release ordering. Replaces the old `FileDescriptor::table_reader` raw pointer and `FileMetaData::table_reader_handle` - Extract `LoadTableHandlersHelper` into `db/version_util.cc` — shared between `VersionBuilder::LoadTableHandlers` (for version edits during recovery) and `BGWorkAsyncFileOpen` (for base storage post-open) - `FindTable` extended with `pin_table_handle` and `out_table_reader` params — when pinning is enabled, the table reader is stored on `FileMetaData` so Get/MultiGet/Iterator skip redundant cache lookups. `FindTable` now performs the pinned-reader fast-path check internally instead of requiring callers to check `fd.table_reader` beforehand - Note: pinning is explicit (not default) because some callers create temporary `FileMetaData`s that would need to properly clean up table handles - `CompactedDBImpl` updated to use `FindTable` + pinning instead of raw `fd.table_reader` access for Get/MultiGet - New `kAsyncFileOpen` background error reason in `listener.h` and `error_handler.cc` - Add a check in ~DBImpl to ensure async file open task has not been forgotten to be scheduled in (future) subclasses of DBImpl. Certain subclasses that never use it will need to explicitly mark it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14322 Test Plan: - `OpenFilesAsyncTest` parameterized over `num_flushes` (1, 20), `ReadType` (Get, MultiGet, Iterator), `max_open_files` (-1, 10), and `read_only` (true, false) - **ConcurrentFileAccess**: concurrent reads and compactions race with async opener - **AfterRead**: reads happen before async opener, verifying lazy open and that the opener sees already-pinned readers - **BeforeRead**: async opener completes first, verifying reads use pre-loaded table readers - **Shutdown**: DB closes before async opener starts, verifying clean cancellation with 0 file opens - **Error**: corrupted SST files, verifying `kAsyncFileOpen` background error is set and reads return corruption - **DropColumnFamily**: CF dropped before async opener runs, verifying the opener gracefully skips dropped CFs - Added to crash test ### Benchmark To simulate a high-latency remote filesystem, I set up a virtual filesystem with dm-delay using 10ms reads, 0 ms writes. ``` # Generate a DB with many L0 files TEST_TMPDIR=/data/users/jkangs/dm-delay-test/mnt ./db_bench -benchmarks=fillseq -disable_auto_compactions=true -write_buffer_size=1000 -num=1000000 ``` ``` ./db_bench -use_existing_db=true -db=/data/users/jkangs/dm-delay-test/mnt/dbbench -benchmarks=readrandom -reads=1 -report_open_timing=true -open_files_async=true -use_direct_reads -file_opening_threads=1 -skip_stats_update_on_db_open OpenDb: 25.1419 milliseconds ``` ``` ./db_bench -use_existing_db=true -db=/data/users/jkangs/dm-delay-test/mnt/dbbench -benchmarks=readrandom -reads=1 -report_open_timing=true -open_files_async=false -use_direct_reads -file_opening_threads=1 -skip_stats_update_on_db_open OpenDb: 23109.4 milliseconds ``` ### No read regressions On main branch ``` ./db_bench -use_existing_db=true -db=/dev/shm/dbbench -benchmarks=readrandom -seed=1 -threads=8 -duration=30 readrandom : 4.827 micros/op 1657100 ops/sec 30.005 seconds 49720992 operations; 183.3 MB/s (6198999 of 6198999 found) ``` On this branch ``` ./db_bench -use_existing_db=true -db=/dev/shm/dbbench -benchmarks=readrandom -seed=1 -threads=8 -duration=30 readrandom : 4.863 micros/op 1644808 ops/sec 30.007 seconds 49354992 operations; 182.0 MB/s (6099999 of 6099999 found) ./db_bench -use_existing_db=true -db=/dev/shm/dbbench -benchmarks=readrandom -seed=1 -threads=8 -duration=30 -open_files_async=true readrandom : 4.803 micros/op 1665392 ops/sec 30.004 seconds 49968992 operations; 184.2 MB/s (6222999 of 6222999 found) ``` Reviewed By: pdillinger, xingbowang Differential Revision: D93538033 Pulled By: joshkang97 fbshipit-source-id: 32ac70c112cd733b7c1e1c1e2e7ce6422318a5ae	2026-03-02 16:18:14 -08:00
Josh Kang	901c88e37b	Separate keys and values in data blocks (#14287) ... Summary: Introduce new table option with separated key-value storage in data blocks. This PR implements a new SST block format where keys and values are stored in separate sections within data blocks, rather than interleaved. Keys are stored first, followed by all values in a contiguous section. The motivation is better cpu cache hit rate during seeks and potentially better compression. The additional storage cost is a varint per restart point, and 4 bytes additional block footer. For a data block with a restart interval of 16, it is approximately 1 bit of overhead per entry. But compression actually performs better, resulting in ~3% storage savings from benchmark. For now I've opted to not separate kvs in non-data blocks since restart interval for those blocks is typically 1, and values are typically small and probably better inlined. ### New block layout ``` +------------------+ | Keys Section | <- Key entries with delta encoding +------------------+ | Values Section | <- (new) Values stored contiguously +------------------+ | Restart Array | <- Fixed32 offsets to restart points +------------------+ | Values Offset | <- (new) 4 bytes: offset to values section | Footer | <- 4 bytes: packed index_type + num_restarts +------------------+ ``` ### Entry Format **At restart points** ``` +--------------+------------------+----------------+-----------------+-----------+ | shared (v32) | non_shared (v32) | value_sz (v32) | value_off (v32) | key_delta | +--------------+------------------+----------------+-----------------+-----------+ ``` **At non-restart points** ``` +--------------+------------------+----------------+-----------+ | shared (v32) | non_shared (v32) | value_sz (v32) | key_delta | +--------------+------------------+----------------+-----------+ ``` - `value_offset` is only stored at restart points to save space - For non-restart entries, value offset is computed as: `prev_value_offset + prev_value_size` ### Forward Compatibility - We make use of reserved block footer bits to mark if a block has separated kv format. Should an older version read this, it will assume a very large block restart interval and result in a corruption error. ### Key Changes - **BlockBuilder**: Accumulates values in a separate buffer; value offsets are stored only at restart points (other entries derive offset from previous value's position). There is an additional memcpy cost to place the value data after the key data. - **Block iteration**: Iteration now needs to know if we are at a restart point. This will rely on `cur_entry_idx_`, which was previously only used for per-kv checksum purposes. In this new format, we also need to know the block_restart_interval, which was previously also only calculated for per-kv checksums. - **Table properties**: Store `data_block_restart_interval`, `index_block_restart_interval`, and `separate_key_value_in_data_block` in table properties Pull Request resolved: https://github.com/facebook/rocksdb/pull/14287 Test Plan: - Extended block_test, table_test, compaction_test to contain new separated_kv param - Added new parameter to crash test --- ## Benchmark ### Varying Value Size **Write:** `db_bench --num=1000000 --separate_key_value_in_data_block=<bool> --value_size=<X> --benchmarks=fillrandom,compact` **Read:** `db_bench --db=$DB --use_existing_db --benchmarks=readrandom,readseq` | Value Size | FillRandom (ops/s) | | Compact (s) | | ReadRandom (ops/s) | | ReadSeq (ops/s) | | SST Size (MB) | | |----------:|-------------------:|-----:|----------:|-----:|-------------------:|-----:|----------------:|-----:|-------------:|-----:| | | baseline | sep_kv | baseline | sep_kv | baseline | sep_kv | baseline | sep_kv | baseline | sep_kv | | 1 | 253,280 | 264,977 | 0.516 | 0.497 (**-3.7%**) | 596,328 | 586,625 (**-1.6%**) | 5,904,681 | 6,069,581 (**+2.8%**) | 5.1 | 4.2 (**-18.6%**) | | 16 | 235,757 | 242,367 | 0.572 | 0.533 (**-6.8%**) | 570,751 | 572,815 (**+0.4%**) | 5,371,138 | 5,604,908 (**+4.4%**) | 14.7 | 13.4 (**-8.5%**) | | 100 | 264,299 | 265,427 | 0.461 | 0.454 (**-1.5%**) | 323,696 | 332,790 (**+2.8%**) | 4,239,725 | 4,232,416 (**-0.2%**) | 38.8 | 37.6 (**-3.2%**) | | 1,000 | 238,992 | 242,764 | 2.349 | 2.329 (**-0.9%**) | 244,608 | 261,403 (**+6.9%**) | 1,285,394 | 1,265,868 (**-1.5%**) | 342.1 | 342.0 (**-0.0%**) | ### Varying Block Restart Interval **Write:** `db_bench --num=1000000 --separate_key_value_in_data_block=<bool> --block_restart_interval=<X> --benchmarks=fillrandom,compact` **Read:** `db_bench --db=$DB --use_existing_db --benchmarks=readrandom,readseq` | BRI | FillRandom (ops/s) | | Compact (s) | | ReadRandom (ops/s) | | ReadSeq (ops/s) | | SST Size (MB) | | |----:|-------------------:|-----:|----------:|-----:|-------------------:|-----:|----------------:|-----:|-------------:|-----:| | | baseline | sep_kv | baseline | sep_kv | baseline | sep_kv | baseline | sep_kv | baseline | sep_kv | | 1 | 251,654 | 263,707 | 0.453 | 0.485 (**+7.1%**) | 334,653 | 328,708 (**-1.8%**) | 4,194,342 | 3,954,291 (**-5.7%**) | 40.6 | 42.4 (**+4.5%**) | | 4 | 253,797 | 252,394 | 0.476 | 0.476 (**+0.0%**) | 332,719 | 341,676 (**+2.7%**) | 4,135,691 | 4,051,151 (**-2.0%**) | 39.2 | 39.3 (**+0.3%**) | | 8 | 260,143 | 262,273 | 0.496 | 0.460 (**-7.3%**) | 330,859 | 337,567 (**+2.0%**) | 4,144,081 | 4,187,389 (**+1.0%**) | 38.9 | 38.1 (**-2.1%**) | | 16 | 252,875 | 263,176 | 0.464 | 0.455 (**-1.9%**) | 323,783 | 335,418 (**+3.6%**) | 4,127,310 | 4,217,028 (**+2.2%**) | 38.8 | 37.6 (**-3.2%**) | | 32 | 260,224 | 269,422 | 0.464 | 0.451 (**-2.8%**) | 304,001 | 314,989 (**+3.6%**) | 4,310,162 | 4,247,248 (**-1.5%**) | 38.8 | 37.3 (**-3.8%**) | ### Varying Compression **Write:** `db_bench --num=1000000 --separate_key_value_in_data_block=<bool> --compression_type=<X> [--compression_level=<N>] --benchmarks=fillrandom,compact` **Read:** `db_bench --db=$DB --use_existing_db --benchmarks=readrandom,readseq` | Compression | FillRandom (ops/s) | | Compact (s) | | ReadRandom (ops/s) | | ReadSeq (ops/s) | | SST Size (MB) | | |------------|-------------------:|-----:|----------:|-----:|-------------------:|-----:|----------------:|-----:|-------------:|-----:| | | baseline | sep_kv | baseline | sep_kv | baseline | sep_kv | baseline | sep_kv | baseline | sep_kv | | None | 252,494 | 260,552 | 0.413 | 0.419 (**+1.5%**) | 356,290 | 371,535 (**+4.3%**) | 4,479,261 | 4,507,133 (**+0.6%**) | 73.5 | 73.6 (**+0.2%**) | | LZ4 | 246,010 | 256,360 | 0.477 | 0.455 (**-4.6%**) | 342,497 | 345,882 (**+1.0%**) | 4,400,570 | 4,268,102 (**-3.0%**) | 38.3 | 37.6 (**-2.0%**) | | ZSTD (L3) | 254,748 | 258,556 | 1.067 | 1.055 (**-1.1%**) | 176,724 | 177,566 (**+0.5%**) | 2,736,841 | 2,717,739 (**-0.7%**) | 32.9 | 31.3 (**-4.7%**) | | ZSTD (L6) | 256,459 | 259,388 | 1.556 | 1.462 (**-6.0%**) | 177,390 | 176,691 (**-0.4%**) | 2,754,336 | 2,688,682 (**-2.4%**) | 32.8 | 31.1 (**-5.1%**) | ### Varying Block Size **Write:** `db_bench --num=1000000 --separate_key_value_in_data_block=<bool> --block_size=<X> --benchmarks=fillrandom,compact` **Read:** `db_bench --db=$DB --use_existing_db --benchmarks=readrandom,readseq` | Block Size | FillRandom (ops/s) | | Compact (s) | | ReadRandom (ops/s) | | ReadSeq (ops/s) | | SST Size (MB) | | |-----------:|-------------------:|-----:|----------:|-----:|-------------------:|-----:|----------------:|-----:|-------------:|-----:| | | baseline | sep_kv | baseline | sep_kv | baseline | sep_kv | baseline | sep_kv | baseline | sep_kv | | 4 KB | 263,203 | 260,362 | 0.469 | 0.461 (**-1.7%**) | 324,178 | 332,249 (**+2.5%**) | 4,231,537 | 4,217,763 (**-0.3%**) | 38.8 | 37.6 (**-3.1%**) | | 16 KB | 252,742 | 263,161 | 0.426 | 0.428 (**+0.5%**) | 227,805 | 222,873 (**-2.2%**) | 5,146,997 | 5,081,080 (**-1.3%**) | 38.1 | 36.7 (**-3.6%**) | | 64 KB | 257,490 | 260,225 | 0.423 | 0.414 (**-2.1%**) | 86,807 | 91,586 (**+5.5%**) | 5,380,403 | 5,372,372 (**-0.1%**) | 36.3 | 35.0 (**-3.5%**) | ### Varying Key Size **Write:** `db_bench --num=1000000 --separate_key_value_in_data_block=<bool> --min_key_size=10 --max_key_size=100 --benchmarks=fillrandom,compact` **Read:** `db_bench --db=$DB --use_existing_db --benchmarks=readrandom,readseq` | Key Size | FillRandom (ops/s) | | Compact (s) | | ReadRandom (ops/s) | | ReadSeq (ops/s) | | SST Size (MB) | | |---------:|-------------------:|-----:|----------:|-----:|-------------------:|-----:|----------------:|-----:|-------------:|-----:| | | baseline | sep_kv | baseline | sep_kv | baseline | sep_kv | baseline | sep_kv | baseline | sep_kv | | 10–100 | 243,740 | 255,183 | 0.618 | 0.622 (**+0.6%**) | 284,304 | 307,569 (**+8.2%**) | 3,738,921 | 3,686,676 (**-1.4%**) | 41.5 | 41.2 (**-0.8%**) | ### CPU Profile Notes - No compression: DataBlock::SeekForGet uses less cpu (13.2% vs 13.9%) - https://fburl.com/strobelight/6mwwebft with separated KV - https://fburl.com/strobelight/m9m798ka without - ZSTD compression: rocksdb::DecompressSerializedBlock uses more CPU (45.8% vs 44.9%), while DataBlock::SeekForGet uses less cpu (5.09% vs 6.52%) - https://fburl.com/strobelight/3x5nw1k4 with separated KV - https://fburl.com/strobelight/e7809046 without --- Reviewed By: xingbowang, pdillinger Differential Revision: D92103024 Pulled By: joshkang97 fbshipit-source-id: 47cfeb656ff3c20d34975f0b6c4c0462935a83dc	2026-02-23 12:42:05 -08:00
Hui Xiao	29819f37e1	Remove deprecated ReadOptions::managed, `ColumnFamilyOptions::snap_refresh_nanos (#14350) ... Summary: **Context/Summary:** Remove deprecated, unused APIs and options: - ReadOptions::managed: This option was not used anymore. The functionality it controlled has been removed long ago. - ColumnFamilyOptions::snap_refresh_nanos: Deprecated and unused option. Corresponding C API (rocksdb_readoptions_set_managed) and Java API (ReadOptions.managed/setManaged) are also removed. All related checks an db_impl and db_impl_secondary iterators are cleaned up. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14350 Test Plan: make check Reviewed By: pdillinger Differential Revision: D93812438 Pulled By: hx235 fbshipit-source-id: e4a9d21c65f83294b6d0878286ba14024f049bac	2026-02-20 14:00:41 -08:00
Hui Xiao	407f02da19	Remove deprecated SliceTransform::InRange() virtual method (#14353) ... Summary: **Summary/Context:** Remove the `InRange()` virtual method from `SliceTransform` and all its overrides. This method was marked DEPRECATED, never called by RocksDB, and existed only for backward compatibility. Also removes the `in_range` callback parameter from `rocksdb_slicetransform_create()` in the C API, which is a breaking change appropriate for a major version release. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14353 Test Plan: Make check Reviewed By: xingbowang Differential Revision: D93795070 Pulled By: hx235 fbshipit-source-id: 5eba23f1d038b19c494997a55e5d8ca379fbedcb	2026-02-19 16:45:51 -08:00
xingbowang	3556c22059	Remove deprecated option skip_checking_sst_file_sizes_on_db_open (#14346) ... Summary: Remove deprecated option skip_checking_sst_file_sizes_on_db_open Pull Request resolved: https://github.com/facebook/rocksdb/pull/14346 Test Plan: Unit test Reviewed By: hx235 Differential Revision: D93602683 Pulled By: xingbowang fbshipit-source-id: f576825cb107bb0aeb14f4ff29fef0df269b8728	2026-02-19 14:12:38 -08:00
Xingbo Wang	b040ab83e1	Add a new picking algorithm in fifo compaction (#14326) ... Summary: Add a new kv ratio based compaction picking algorithm in fifo compaction Pull Request resolved: https://github.com/facebook/rocksdb/pull/14326 Test Plan: Unit test Reviewed By: pdillinger Differential Revision: D93257941 Pulled By: xingbowang fbshipit-source-id: fd2d0e1356c7b54682a1197475a1bd26cb45c9d4	2026-02-15 10:04:58 -08:00
Josh Kang	9f47518676	Add interpolation search as an alternative to binary search (#14247) ... Summary: Interpolation search is an alternative algorithm to binary search, which performs better on uniformly distributed keys. Instead of binary search always computing the mid point of the left and right boundaries, interpolation search "interpolates" the mid point based on the distance to the target. Fortunately, we can re-use existing block format to support interpolation search. For a given block, we compute the shared_prefix length of the first and last key. Interpolation search is usually done with numerical target values, so for a variable binary length key, we calculate the "value" as the first 8 non-shared bytes. This also means interpolation search would only really be effective for bytewise comparator (guarded via options validations). #### Fallback to binary search - if the the val(left_key) == val(right_key) then we fallback to classic binary search (to avoid divide by 0) - interpolation search is significantly more computationally expensive than binary search, so when the search distance is small, we also fallback to binary search. - if interpolation search does not make significant progress (i.e. reduces search space by more than half each iteration), we can assume data is non-uniform and fallback. Interpolation search also performs best when there is minimal shortening, especially shortening of the last block, as it can heavily skew the distribution of the actual keys. Note that each search algorithm is guaranteed to make progress because at each iteration the search space is guaranteed to be reduce by at least 1. For now this change only applies to index block seeks, as data block seeks and other blocks do not have as many entries and would not require significant number of search rounds, but it could be easily extended to include that support. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14247 Test Plan: Updated unit tests and crash test with new search option ### Benchmark The default benchmark sets up keys in generally uniform distribution, so it was a good way to test performance improvements. Setup: `./db_bench -benchmarks=fillseq,compact -index_shortening_mode=1` #### Before this change ``` ./db_bench -use_existing_db=true -benchmarks=readrandom -seed=1 readrandom : 2.899 micros/op 344973 ops/sec 2.899 seconds 1000000 operations; 38.2 MB/s (1000000 of 1000000 found) ``` #### After this change Notice how key comparison counts are the same between the two. ``` ./db_bench -use_existing_db=true -benchmarks=readrandom -seed=1 -index_search_type=binary_search readrandom : 2.881 micros/op 347128 ops/sec 2.881 seconds 1000000 operations; 38.4 MB/s (1000000 of 1000000 found) ``` ``` ./db_bench -use_existing_db=true -benchmarks=readrandom -seed=1 -index_search_type=interpolation_search readrandom : 2.609 micros/op 383209 ops/sec 2.610 seconds 1000000 operations; 42.4 MB/s (1000000 of 1000000 found) ``` With a non-uniform distribution, `i.e. index_shortening_mode=2` ``` ./db_bench -use_existing_db=true -benchmarks=readrandom -seed=1 -index_search_type=binary_search readrandom : 2.958 micros/op 338075 ops/sec 2.958 seconds 1000000 operations; 37.4 MB/s (1000000 of 1000000 found) ``` ``` ./db_bench -use_existing_db=true -benchmarks=readrandom -seed=1 -index_search_type=interpolation_search readrandom : 5.502 micros/op 181750 ops/sec 5.502 seconds 1000000 operations; 20.1 MB/s (1000000 of 1000000 found) ``` Reviewed By: pdillinger Differential Revision: D91063163 Pulled By: joshkang97 fbshipit-source-id: 151d6aa76f8713740b714de6e406aff40d28ccbc	2026-02-13 17:15:10 -08:00
Peter Dillinger	871f79d6ef	Reformat source files (#14331) ... Summary: probably something changed, maybe https://github.com/facebook/rocksdb/issues/14311 Full command: ``` git ls-files '*.cc' '*.h' | grep -v '^third-party/' | grep -v 'range_tree' | xargs clang-format -i ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/14331 Test Plan: CI Reviewed By: mszeszko-meta Differential Revision: D93246992 Pulled By: pdillinger fbshipit-source-id: 6bc5b97978fef8aee52823dadb6daa4bea57343d	2026-02-13 11:56:22 -08:00
Peter Dillinger	672389fd8c	Remove obsolete compression code and some .h->.cc movement (#14325) ... Summary: In follow-up to https://github.com/facebook/rocksdb/issues/14315 Remove obsolete code replaced by new Compressor/Decompressor interface: * OLD_CompressData and OLD_UncompressData * Individual compression/decompression functions (Snappy_*, Zlib_*, BZip2_*, LZ4_*, LZ4HC_*, XPRESS_*, ZSTD_Compress, ZSTD_Uncompress) * CompressionInfo and UncompressionInfo classes * UncompressionDict class * compression::PutDecompressedSizeInfo and GetDecompressedSizeInfo The only small refactoring in this change that is not pure code removal or movement is in blob_file_builder_test.cc. Move some function implementations etc. from compression.h to compression.cc: * CompressionTypeToString, CompressionTypeFromString, CompressionOptionsToString * ZSTD_TrainDictionary (both overloads), ZSTD_FinalizeDictionary * DecompressorDict::Populate * Most compression library includes Also cleaned up other includes of compression.h, which caused some other files to need new includes. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14325 Test Plan: existing tests Reviewed By: hx235 Differential Revision: D93120580 Pulled By: pdillinger fbshipit-source-id: ab5c50db7379c0387a8c0e379642c9ea2799eae5	2026-02-13 11:18:05 -08:00