510 Commits

Author	SHA1	Message	Date
Yoshinori Matsunobu	d6a7847d37	Add a DB option to log long-held global mutexes and release sites (#15436) ... Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15436 ## Why Long-held global mutexes can stall foreground and background work. Existing mutex statistics measure acquisition waits, not continuous ownership. This adds opt-in logging to identify the mutex, owning thread, operation and release site. ## Enable for an investigation Provide an INFO logger and shared `Statistics` object when opening the DB, then: - Set the shared statistics level to `StatsLevel::kAll`. - Set `mutex_hold_log_threshold_us` through `DB::SetDBOptions`; `100000` means 100 ms. - Restore threshold 0 and the previous statistics level afterward. The threshold defaults to **0 (disabled)**. `kAll` alone does not enable these hold probes; threshold 0 leaves existing statistics timers unchanged. ## Output Illustrative records, not production observations: ``` SLOW_GLOBAL_MUTEX mutex=DBImpl::mutex_ mutex_addr=0x12340000 owner_thread_id=12345 thread_operation=Compaction hold_us=200000 threshold_us=100000 observed_end_us=1792205400000000 buffer_dropped_records=0 unlock=db_impl_compaction_flush.cc:4660 BackgroundCallCompaction() SLOW_GLOBAL_MUTEX mutex=DBImpl::wal_write_mutex_ mutex_addr=0x12340100 owner_thread_id=12346 thread_operation= hold_us=35000 threshold_us=25000 observed_end_us=1792205401000000 buffer_dropped_records=0 scope_exit=db_impl_write.cc:2250 PreprocessWrite() ``` The first proves a completed 200 ms DB-mutex ownership interval on a thread tagged `Compaction`. The second uses a 25 ms threshold; an empty operation tag means none was available. Neither record reports acquisition wait or proves the cause of a stall. Release attribution uses compiler-embedded strings available in optimized, stripped builds, with a GCC/Clang fallback where standard source locations are unavailable. Direct unlock and condition wait identify their call sites; scope exit and temporary unlock identify guard declarations, not closing braces. No stack unwinding or symbol lookup. ## Scope and safety - Track six selected mutexes: DB, WAL, point-lock wait graph, two prepare-log bookkeeping mutexes and transaction name map. This is not comprehensive synchronization tracing. - Acquisition waits and condition-variable release gaps are excluded from hold duration. Configuration is pre-first-use; repeated setup retains the first valid context. - Sample at most once per mutex per second, with eight deferred records per thread. `buffer_dropped_records` counts capacity overflow only; rate limiting, pause and emission contention can omit samples. - Emit after covered outer lock scopes release. Emission is nonblocking best-effort; shutdown drains/pauses emission and stops diagnostics before the owned logger closes. Closing from that logger's own callback is unsupported. - No data-file format change. The mutable option is persisted in OPTIONS and wired into bench/stress tools. Typed C binding is deferred; string DB-option APIs are available. Disabled bookkeeping still has a cost. Release A/B measurements found about 3 ns added per selected mutex pair at level 1, and about 70 ns for hold timing over `kAll`/0. Get and reused-iterator Seek showed no resolved small probe effect; fresh iterator+Seek at 16 threads showed +8.7% CPU/query and -9.4% QPS. No logs were emitted in those benchmarks; logging cost and broader workload qualification remain open. Reviewed By: xingbowang Differential Revision: D125790955 fbshipit-source-id: 1096163f3bd25ffee585dc6ec8ebfd7f1759851a	2026-10-22 09:50:20 -07:00
Josh Kang	014e1c43c1	Apply readability fixes (isolate decls, braces) (#15394) ... Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15394 ## Summary RocksDB's `.clang-tidy` enables several readability checks that existing code still violates: `readability-isolate-declaration` (476 findings), `readability-braces-around-statements` (49), `readability-redundant-string-init` (25), `modernize-use-using` (10), `misc-definitions-in-headers` (3) and `readability-duplicate-include` (1). They keep getting flagged whenever nearby lines change. This diff fixes 553 of the 564; upstream has since fixed the other 11. What changed: - Multi-variable declarations such as `std::string a, b;` are split into one declaration per variable. Pointer and reference declarators are kept per variable. - Braces are added around single-statement `if`/`else`/`for`/`while` bodies, and empty spin loops get an empty `{}` body. - `std::string s = ""` / `member("")` initializers are removed. - Remaining `typedef`s in the touched C++ files become `using`. C headers and `extern "C"` code are untouched. - `inline` is added to non-template function definitions in test headers. - A duplicate `#include "util/udt_util.h"` in `db_impl.cc` is removed. These are clang-tidy's own fix-its, applied mechanically and then clang-formatted on the changed lines. Three kinds of broken fix-it output were repaired by hand. Removing a `member("")` initializer left a dangling comma in constructor initializer lists (`ExternalSstFileInfo`, `ColumnFamilyMetaData` and others). The function-type typedefs in `block_test.cc` came out malformed. Empty `while (cond);` spin loops got a `{;` body. Splitting declarations also exposed two vectors in `db_etc2_test.cc` to `performance-inefficient-vector-operation`, so a `reserve()` was added for each. The lint bot reports every finding in a file a diff touches. This diff splits declarations in `options/configurable_test.cc`, so it also fixes that file's 14 older `cert-err58-cpp` and `cppcoreguidelines-avoid-non-const-global-variables` findings, which come from seven mutable file-level maps. Six are now `const` and built where they are used (function-local statics, or locals in their only user). The unused `imm_struct_option_info` is deleted. Compatibility: no behavior change. In public headers, the only edits are removed redundant `""` initializers, split declarations and added braces. Reviewed By: rban1 Differential Revision: D123186310 fbshipit-source-id: f9c8e88b00e997db0196f8725c7c8d880cf798da	2026-10-13 14:06:55 -07:00
Josh Kang	4b983da055	Spell out special member functions (rule of five) (#15387) ... Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15387 ## Summary RocksDB's `.clang-tidy` enables `cppcoreguidelines-special-member-functions`, `cppcoreguidelines-virtual-class-destructor` and the noexcept-move checks, and clang warns `-Wdeprecated-copy-with-user-provided-dtor`. A full scan reports 476 findings. In most of them, a class declares a destructor (or copy operations) but leaves the rest of copy/move implicit. That has two costs. Readers can't tell whether a class may be copied or moved. And resource-owning classes keep an implicit copy that would double-free if anyone ever used it. This diff resolves 453 of the 476. Approach, picked per class from least to most invasive: 1. An empty user-provided destructor becomes `= default`. This covers most interface classes, such as `virtual ~X() {}` -> `= default`. If the class is still copied, that would only rename the deprecated-copy warning, so the destructor is removed instead when nothing needs it (`Customizable`, `TableFactory` and `InternalKeyComparator` inherit a virtual one; `IteratorWrapperBase` isn't polymorphic), and `MemTableRep::KeyComparator` gets defaulted copy and move. 2. Classes whose copy is already deleted get explicitly deleted move operations too. Move was already unavailable, so this only documents it. 3. Resource-owning classes that are never copied get deleted copy and move operations. Examples: the C API wrapper structs, which run a `destructor_` callback (edited in `tools/c_api_gen/c_base.cc` and regenerated into `db/c.cc`); `ShardedCache`, which frees its shards; iterator wrappers; and RAII handles. 4. Value types that are copied or moved get the missing operations as `= default`, which keeps their current behavior. 5. Exception specifications of existing functions are unchanged, except for `WriteBatch`'s move assignment (below): no other `noexcept` is added or removed. Only the move operations this diff newly declares (`Comparator`, `FileSampledStats` and the thread-local `Entry`) are `noexcept`; they copy only scalars or atomics. `WriteBatch`'s copy and move assignment destroyed `*this` and placement-new'ed a new `WriteBatch` over it. That is undefined behavior when `*this` is a subclass, and if the copy constructor threw, `*this` was left destroyed and was later destroyed again. Move assignment now assigns member-wise and ends in the same state as the move constructor, which leaves `has_key_with_ts_`, the timestamp-size tracking and the other fields it doesn't copy at their defaults. `rep_` is swapped through a temporary, because a plain string move-assignment can keep the old buffer (e.g. a large batch reset with `batch = WriteBatch()` would keep its memory). Nothing in it can throw, so it is now `noexcept`. Copy assignment copies into a temporary and moves it in, so a throwing copy leaves `*this` unchanged. Bugs this exposed and fixed: - `RestoreAfterCopyOrCreateWorkItem` (backup engine): the hand-written move constructor and assignment moved only some fields, so `from_file` and `to_file` were empty after `push_back(std::move(...))`. A restore that hit a checksum mismatch then reported "While restoring -> : ..." without file names. It now uses the compiler-generated moves, which are still `noexcept`. - `ExternalSstFileIngestionJob` lived in a `std::vector`, and its destructor frees raw-owned pointers, so the implicit copy used on vector reallocation would double-free. Its file batches also point into its own inline file storage, so neither a copy nor a move is safe after `Prepare()`. Copy and move are now deleted, and `DBImpl` holds jobs through `std::unique_ptr`. Compatibility: no public class loses an operation users can call. Public interfaces mostly only get `virtual ~X() = default;`. `WriteBatch`'s move assignment becomes `noexcept`, which is not part of the mangled name, so the ABI doesn't change. `SubcompactionJobInfo` and `CompactionJobInfo` intentionally stay C++20 aggregates, so users' aggregate and designated initializers keep compiling. `Configurable`, `CompareInterface`, `WriteBatchBase`, `ObjectLibrary::Entry` and `MemTableRep::KeyComparator` declare defaulted copy and move, and `Customizable` no longer declares a destructor. As a result, `Configurable` and `Customizable` subclasses that declare no destructor or copy operations of their own now move their registered-options list instead of copying it. In-tree, these are the mutable-options adapters, the table factories, the memory allocators and test classes; all are held through smart pointers or used in place, and `Clone()` copies, so none is moved by value. Not changed (23): `IODebugContext` (custom deep copy plus a `std::shared_mutex`), `DBWithColumnFamilies` in db_bench (stored in a `std::vector` with non-movable members), `Snapshot`'s protected virtual destructor (intended), `ManagedSnapshot`, the `SubcompactionJobInfo` and `CompactionJobInfo` aggregates, and the 16 noexcept-move-constructor and noexcept-swap findings on existing functions, including `autovector`'s move assignment (adding `noexcept` would change their exception specifications). Copying `ManagedSnapshot` already double-releases the snapshot. Fixing that needs a real move implementation and is a behavior change, so it is left for a separate diff. Reviewed By: rban1 Differential Revision: D123186309 fbshipit-source-id: 6f8e57e233e12199a007407130bb9a1f6d52e8d1	2026-10-12 16:00:09 -07:00
Peter Dillinger	34d5074bfa	Cross-key coalescing for lazy wide-column blob reads (#15178) ... Summary: Follow-up (Phase 3) to the experimental lazy wide-column read API (DB::GetEntityLazy / DB::MultiGetEntityLazy). A MultiGetEntityLazy batch now resolves blob references across keys with coalesced I/O instead of key-by-key. - MultiGetEntityLazy acquires one SuperVersion and one consistent (implicit) sequence number for the whole batch via MultiCFSnapshot (as multi-CF iterators do) and transfers a single shared per-column-family pin into the LazyWideColumnsBatch, replacing the former per-key SuperVersion + explicit-snapshot loop. Single column family for now; a cross-CF ColumnFamily** overload remains a follow-up (the batch already models a per-CF pin map). - LazyWideColumnsBatch::MultiResolve groups the batch's reads by (Version, blob file) for separate-file references and by SST for embedded references, and issues one coalesced read per group -- for whole-column and byte-range reads alike -- caching whole values in each entity's resolver and pinning partial reads directly. - Adds the coalesced blob-read primitives this needs (none existed before): range-aware BlobFileReader::MultiGetBlobRange / BlobSource::MultiGetBlobRange / Version::MultiGetBlobLazy for separate-file references, and an embedded (SimpleGen2) batch read path (blob_gen2_format batch readers + BlobSource::MultiGetSimpleGen2Blob[Range] + BlockBasedTable::MultiGetSameFileBlob). Whole-value separate-file coalescing reuses the existing MultiGetBlob. The batch readers sort their requests by file offset before issuing the underlying MultiRead (which requires ascending offsets), and gen2 checksum verification is factored into one helper shared by the scalar and batch readers. - The hot non-lazy Get / MultiGet / iterator / compaction paths are untouched: the new multis are lazy-only, and the GetImpl change is gated on a new opt-in GetImplOptions field (an injected shared SuperVersion + sequence number) used only by the batched lazy read. - Blob read failures on these paths (short reads, invalid offsets, out-of-range requests, compression mismatches) now carry file/offset/size and expected-vs-actual context, mirroring the block-checksum diagnostics. - db_bench gains a multireadrandomentitylazy benchmark for measuring the coalesced batch path. No public API change, so this is a performance follow-up to the still-recent lazy API rather than a new feature. EXCEPT that `LazyColumnReadRequest::status == nullptr is now explicitly disallowed`. Left for later follow-up: - When a whole fetch is planned for an (entity, column), bind ordinary range requests for that column to the same fetch and slice them after adoption instead of issuing separate range I/O. Pull Request resolved: https://github.com/facebook/rocksdb/pull/15178 Test Plan: New unit tests in db/wide/db_lazy_entity_test.cc assert cross-key coalescing via a blob-file / SST MultiRead counter added to the test FileSystem wrapper: - N whole-column separate-file reads across keys in one blob file collapse to a single coalesced MultiRead (and the fetched values are cached, so a repeat read does no further blob I/O); - N byte-range separate-file reads collapse to one MultiRead, save the un-read bytes (rocksdb.blobdb.lazy.partial.bytes.saved), and never fill the blob cache; - embedded (same-file) reads across keys in one SST collapse to one MultiRead over that SST; - an out-of-order embedded batch (reads issued in descending record offset) still resolves correctly in one coalesced read, covering the batch readers' internal offset sort; - a mixed batch (inline + whole + byte-range across keys) returns the correct bytes for each read; - a batched MultiGetEntityLazy + one MultiResolve matches resolving each key via a separate GetEntityLazy. The existing db_stress lazy-vs-eager differential (MaybeTestMultiGetEntityLazy) exercises the new batched read and cross-key MultiResolve unchanged (same public API). Reviewed By: xingbowang Differential Revision: D118660682 Pulled By: pdillinger fbshipit-source-id: 6a832175e730917eddeedab8e0b9cb2b8d6ae7b4	2026-09-16 18:10:21 -07:00
Xingbo Wang	4c2a977aa5	Track newer snapshot versions in RocksDB metadata reads (#15086) ... Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15086 Why: An application built on top of RocksDB could use cache population reads through explicit snapshots and needs to know when the returned value has already been superseded, without a second point read. Previous behavior: Get and MultiGet returned the snapshot-visible value and optional timestamp but could not report a newer point update or covering range deletion. New behavior: experimental C++, C, and Java metadata-read APIs report newer committed puts, deletes, merges, and covering range tombstones; ordinary primary-DB writes completed before the read are guaranteed to be reported, racing writes remain best-effort, and true has no false positives. Implementation: metadata reads widen lookup behind read-callback hooks while keeping metadata-only state off passive common object layouts; per-key state stays in KeyContext/GetContext, ordinary MultiGet row-cache replay semantics are preserved, BlobDB/TTL use shared output helpers, dead single-CF plumbing and an invalid cross-key stress assertion are removed, and the range-tombstone ordering invariant is documented without changing behavior. Safety and unsupported cases: tracking is opt-in and requires an explicit snapshot; persisted-tier reads, custom transactional visibility callbacks, secondary DBs, and unsupported subclasses return NotSupported, while read-only/compacted DBs return false because their in-process views are static. Tests and limitations: 1,265 Buck tests plus focused metadata, 100x coerced-context-switch row-cache, write-committed/write-prepared crash tests, builds, and lint passed. Current-parent release db_bench found no material passive regression: readrandom +0.65%, MultiGet batch 4 -1.40%, and batch 32 +0.48%. Reviewed By: pdillinger Differential Revision: D104182397 fbshipit-source-id: 36f7be1cd50fe2383689adaae31a6bb7d7e462dc	2026-09-11 04:44:59 -07:00
Xingbo Wang	7d3150ffce	Report RocksDB benchmark interval latency and perf metrics (#15101) ... Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15101 This diff contains all changes under fbcode/internal_repo_rocksdb/repo for the RocksDB regression benchmark observability work. It adds opt-in per-interval latency percentile reporting to db_bench. When --report_interval_percentiles is enabled with histogram and a positive stats interval, each worker records into pre-created per-operation interval histograms and counters. A background reporter periodically merges and resets those thread-owned stats, skips excluded background-writer stats, performs a final partial-interval flush, and emits IntervalPercentiles lines with sample count, operation count, and P50/P75/P99/P99.9/P99.99 latency. Worker updates and reporter snapshots use a double-buffer handoff, so the reporter never reads or clears a histogram still being updated. The legacy histogram timing path remains unchanged when interval reporting is disabled. The interval implementation uses RocksDB UnorderedMap aliases for the hot lookup maps and UnownedPtr for non-owning reporter references. It also aggregates each worker thread PerfContext and IOStatsContext into the final merged benchmark Stats and emits final PERF_CONTEXT and IOSTATS_CONTEXT blocks, so multi-thread benchmark metrics represent all workers rather than one thread. For options-file runs, explicit benchmark-critical CLI overrides are applied after loading the file. This includes --disable_auto_compactions, fixing an existing precedence bug where disable_auto_compactions=false in the options file silently overrode an explicit CLI true value. The regression harness relies on that CLI setting, so the bug allowed automatic compactions during origin construction and write-heavy benchmarks. The regression test persists false in an options file, passes --disable_auto_compactions=true, reloads the opened DB options, and verifies the effective value is true. One shared WriteBufferManager is created after option loading and reused by all DBs, enforcing the intended aggregate memory cap for multi-DB benchmarks. The compactall benchmark now accepts --compact_all_parallelism. It compacts independent DBs through a bounded worker set, applies --subcompactions to each DB, emits per-DB progress and elapsed time, and propagates the first CompactRange failure after joining all workers. Its default parallelism remains 1 for backward compatibility. The dependent harness diff enables these features, parses their output, and publishes the resulting interval and final metrics. Reviewed By: pdillinger Differential Revision: D116025321 fbshipit-source-id: d44185ba2110aacbe9fadc65a59799bf8c1c54f0	2026-09-10 08:08:38 -07:00
Jay Huh	91b96f41cf	Retry the OpenAndCompact secondary open when the primary replaces CURRENT (#15179) ... Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15179 A remote compaction worker (`DB::OpenAndCompact()`) opens the *live* primary's DB directory read-only, and that open reads `CURRENT`. The primary publishes each MANIFEST rotation by writing a new `CURRENT` under a temp name and renaming it over the old `CURRENT`. POSIX keeps an already-opened `CURRENT` readable after that replacement, but file systems without read-after-unlink semantics (remote or object stores) do not: the replaced object is reported as gone, surfacing in RocksDB as `IO error: No such file or directory` (`Status::PathNotFound`). The secondary open was not retried, so this transient race failed the whole compaction. There is an equivalent, already-recognized race one step later for the MANIFEST file (`ReactiveVersionSet::MaybeSwitchManifest()` converts it into `Status::TryAgain`), which was fatal here for the same reason. `DB::OpenAndCompact()` now retries the secondary open (up to 3 attempts) when it fails with `TryAgain` / `PathNotFound` / `NotFound` -- exactly the statuses a concurrent `CURRENT`/MANIFEST replacement produces. The retry limit is configurable via `OpenAndCompactOptions::max_secondary_open_retries`. Genuine failures are still returned after the bounded retries, and no safety check is weakened or skipped. Found by a `db_stress` crash test failure. Two non-default options in that run make this reachable and fatal: * `remote_compaction_worker_threads=8` with `remote_compaction_failure_fall_back_to_local=0` -- the latter was only recently randomized to 0; before that a remote compaction failure was always silently retried locally. Now `DbStressCompactionService::Wait()` maps the failure to `kFailure`, which becomes a sticky background error on the primary, so every subsequent `Flush()` returns it and `StressTest::ProcessStatus()` aborts with `assert(false)`. * `max_manifest_file_size=8192` -- the primary rotates MANIFEST (and therefore replaces `CURRENT`) constantly, which is what makes the race window hit in practice. Full chain: primary renames a new `CURRENT` over the old one -> the file system reports the replaced object as gone for the reader that already resolved it -> `GetCurrentManifestPath()` inside the worker's secondary open fails with `IO error: No such file or directory` -> `DB::OpenAndCompact()` returns that error with no retry -> `kFailure` (no local fallback) -> primary background error -> `Flush failed: IO error: No such file or directory: ... /CURRENT ...` -> assertion failure in `db_stress`. Reviewed By: joshkang97 Differential Revision: D118480192 fbshipit-source-id: a2618ee24e582e2e713af04464ee1e4038925ccb	2026-09-09 19:32:00 -07:00
Josh Kang	ac0f4ae69f	Isolate coroutine statistics across async reads (#15142) ... Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15142 Coroutine statistics use TLS, but coroutine reads can interleave on the same executor thread. Previously, when a stats-enabled request suspended, it saved its counters without disabling the executor's TLS configuration. A stats-disabled request running next could inherit those enabled settings and collect statistics unexpectedly. Stats setup also lived in individual DB implementations, so wrapper early returns and future stackable DB implementations could bypass it. Move statistics ownership to the public `CoroDB` and callback-based async read boundaries, before virtual dispatch. Each operation captures its caller's configuration, enabled requests preserve their counters across suspensions, and every suspension or completion leaves executor TLS disabled. Because the scope wraps stackable DB dispatch, current and future `CoroStackableDB` implementations automatically inherit the correct behavior without adding their own stats reset or scope. For consistency between coroutine and callback-based reads, the experimental callback API now uses TLS for both configuring and consuming statistics. `AsyncCallback::OnComplete()` no longer receives context arguments; RocksDB publishes the completed per-operation counters to TLS before invoking it. ``` Before time --------------------------------------------------------------> stats-on A: onSet(enable A) -- work -- onUnset(save A) ...... onSet(A) executor TLS: [A enabled] ------------> [still enabled] ------> [A enabled] stats-off B: run ^ inherits A's enabled TLS After time --------------------------------------------------------------> stats-on A: onSet(enable A) -- work -- onUnset(save A, disable) ... onSet(A) executor TLS: [A enabled] ------------> [disabled] ----------------> [A enabled] stats-off B: run ^ remains stats-disabled ``` Reviewed By: xingbowang Differential Revision: D117017089 fbshipit-source-id: 8a85f7ba43e3b73225995f5d0b79da3892baafb9	2026-09-04 11:19:16 -07:00
Jay Huh	897162a6bc	Add session-scoped temporary directory for remote compaction (#15169) ... Summary: Remote compaction writes output SSTs into caller-selected job directories. If the primary crashes before installation or falls back to local compaction, those directories can outlive the DB session and accumulate under the DB directory. This change defines `kSessionTmpDirName` as `session_tmp`, the canonical name for a DB-local temporary directory whose contents have the lifetime of a DB session. The immutable boolean option `use_session_tmp_dir_for_remote_compaction` opts remote compaction into this directory. When enabled, an `OpenAndCompact()` directory name is resolved as `<dbname>/session_tmp/<job>`. During DB open, RocksDB creates the session temporary directory, completes recovery, and then synchronously deletes contents left by the previous DB session before scheduling compactions. `DestroyDB()` removes it as part of database deletion, and normal completion, failure, and cancellation paths clean the resolved per-job directory. The option defaults to false. Disabled callers retain the existing `<dbname>/<job>` layout and no session-directory cleanup. For rollout compatibility, legacy callers that already pass a DB-rooted output path are accepted unchanged only while the option is false. Directory creation and invalid-path errors are delegated to the configured filesystem rather than prevalidated by RocksDB. After D118301531 landed, per-job directory creation and resumption cleanup are kept together in `InitializeCompactionWorkspace()`. `OpenAndCompact()` resolves the client-provided name and expects DB open to have initialized the session temporary directory when the feature is enabled. The boolean is wired through C++ and generated C APIs, option serialization and reconstruction, `OpenAndCompact()`, ldb, db_bench, and db_stress. Tests cover path resolution, cross-session deletion, ownership boundaries, disabled behavior, `DestroyDB()`, configured compaction output, resumption fallback, and option round trips. Pull Request resolved: https://github.com/facebook/rocksdb/pull/15169 Reviewed By: pdillinger Differential Revision: D118227112 fbshipit-source-id: 01aa834b0b3f75638102130a86d95d537a2797bd	2026-09-02 21:46:24 -07:00
Josh Kang	96ed085402	Blog post for coroutines (#15143) ... Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15143 ## Summary Adds a RocksDB blog post describing the native async/coroutine read APIs introduced in RocksDB 11.10.0. It covers the callback and `folly::coro::Task` interfaces, differences from `ReadOptions::async_io`, executor and I/O-completion handoffs, filesystem integration, request-local statistics across suspension, and API lifetime constraints. The post includes three explanatory diagrams and benchmark results comparing synchronous and coroutine point reads across task counts and `MultiGet` batch sizes. https://www.internalfb.com/intern/px/p/cDGqv/ Reviewed By: xingbowang Differential Revision: D117159178 fbshipit-source-id: 5452e6b07e29abb5594505021995e9b5bac9d198	2026-09-01 15:55:37 -07:00
Peter Dillinger	e4a7136d3e	Preferred-phase scheduling for periodic (time-based) compaction ... Summary: `periodic_compaction_seconds` softly guarantees every SST is rewritten at least once per interval. Across a fleet of similarly-configured DBs, many events line those intervals up in lockstep -- a shared enable, a coordinated restart (e.g. DC power-outage recovery), or just similar write history -- so the periodic compactions all come due together, a recurring "thundering herd" that is especially painful with remote compaction (SEV S698635). The obvious fixes trade one problem for another. Spreading the herd out means some files get compacted off their natural schedule, which is wasted work; and a clumsy spread just relocates the herd or spawns a new one. So the real goal is narrow: **de-herd while keeping "badness" -- extra and/or late compaction -- small, and without inducing fresh herding at any step.** This change adds an EXPERIMENTAL, opt-in mechanism that gives each `(DB, column family)` a stable, well-spread *preferred phase* within the interval and steers periodic compactions toward it. It is randomized and *uncoordinated* (DBs never talk to each other), which both keeps it simple and, as it turns out, de-herds far more cheaply than adding jitter to the interval. It is controlled by a single mutable DBOption and is **off by default**: - `periodic_compaction_phase_recovery_percent` (default `0` = disabled / kill switch; recommended `33`; clamped to `[0, 100]`) -- how aggressively each trigger converges toward the phase, as a percent of the remaining gap, only ever pulling the trigger *earlier*. `0` reproduces the classic scheduler exactly; `100` jumps to the phase in one cycle; lower values approach gradually with much less extra work. The preferred phase itself needs no tuning: it is derived from the DB ID (stable across re-open and physical cloning/migration, but distinct across DBs) and each column family is offset from the DB's base phase, so the CFs of one DB don't fire together. (An earlier draft exposed a `compaction_schedule_seed` string option for choosing the phase source; it was over-engineered for the one behavior anyone needed, and as a mutable string it also broke OPTIONS-file round-trip verification in crash testing. It has been removed in favor of always using the DB ID.) ## The two kinds of badness "Badness" here is the fraction of compaction work that is ideally unnecessary: over-compaction (rewriting a file earlier than the interval required) plus a penalty for lateness (going overdue past the interval). Steady state is 0; both "never compact" and "compact twice as often as needed" approach 1. De-herding inherently spends a little badness to spread load out; every design choice below exists to bound one of those two costs *without* re-clustering the fleet. The scenario names cited in the design bullets below (`upgrade_dynamic_r33`, `naive_upgrade_r25`, `burst_r*`, `turndown_r33`, `turnup_r33`, ...) all come from the out-of-tree simulator; see its results page (the "Simulator and visualization" section below, with an interactive plot per scenario) for the evidence behind each claim. ## Design - **A stable per-(DB, CF) preferred phase = one fixed attractor per DB.** Each DB gets a base phase from its DB ID; each CF is offset from it. Because the target is fixed (not re-rolled each cycle) and unique per DB, the fleet converges to a uniform spread and then compacts *on time* -- badness decays toward ~0. This is the crucial contrast with naive per-cycle jitter, which de-herds but never stops jittering and so pays a permanent over-compaction cost (see `naive_upgrade_r25`: flat ~12-14%, only diffusive spreading) vs. `upgrade_dynamic_r33`, whose badness decays to ~0 for the same anti-herding. - **Converge gradually, and only ever earlier.** Each trigger closes a `recovery_percent` *fraction* of the gap to the phase, reaching it over a few cycles rather than in one burst, and never later than `periodic_compaction_seconds` in steady state. A lower rate does less extra work, which matters when the phase keeps moving (write-driven / bursty workloads), where a high rate wastefully "chases" it -- the burst sweep shows `burst_r25 < burst_r33 < burst_r50 < burst_r100` in sustained badness. `33` is the default balance: quick spreading, low extra work. - **Fractional recovery (not "snap") is latency-tolerant and doesn't re-herd.** Closing a fraction of the gap gives a single stable convergence point, so real-world scheduling/queueing latency -- the delay between a file becoming due and its compaction actually running -- merely shifts the whole fleet uniformly instead of re-clustering it. A tried "snap onto the phase when in range" variant converged a bit faster with no latency but re-herded around the snap threshold under latency; it was rejected. Graceful degradation beat peak speed. - **Spread a suddenly-due cohort instead of bursting it (bounded lateness).** Enabling the feature, turning `periodic_compaction_seconds` down, or catching up after downtime can leave a whole cohort past due at the same instant. Firing it all at once would just be the herd again, so such a cohort is spread over the first quarter-interval (~N/4) after the phasing anchor on an *absolute*, epoch-aligned per-(DB, CF) grid. This trades a bounded lateness (<= ~N/4 past the deadline, so the interval is a *soft* upper bound during these transients) for anti-herding, and the absolute grid keeps a rapidly restarting DB from being perpetually re-deferred and starved. `turndown_r33` shows the past-due cohort smeared over N/4 rather than bursting; `turnup_r33` confirms a turn-up creates no past-due cohort (deadlines only grow) and so no burst. For immediate enforcement at such a transition, trigger a manual compaction. - **Off by default, with a kill switch.** Inert unless `periodic_compaction_phase_recovery_percent > 0`; at `0` behavior is identical to the classic scheduler. ## Implementation notes - Phase math is integer-only (`Hash64` + `FastRange64`); floating point is avoided for its pitfalls. The per-CF offset is a golden-ratio (Weyl/Kronecker) additive sequence -- low-discrepancy for any number of CFs. - Logic lives in `db/periodic_compaction_phaser.{h,cc}` (`PeriodicCompactionPhaser`), which owns the recovery percent and the phasing anchor and exposes the stateless trigger-time math; `VersionSet` holds one and caches the per-CF result on each `Version`. The anchor is (re)set to now on DB open, on a recovery-percent change, and on any `periodic_compaction_seconds` change -- the three events that can create a past-due cohort. - Plumbed through the C API, `db_bench`, `db_stress`, and `db_crashtest`. ## Simulator and visualization (out-of-tree, not part of this change) A standalone simulator drives 40 real RocksDB instances through a real periodic-task scheduler, records every compaction (periodic vs write-driven), and scores the badness metric above across ~17 scenarios: dynamic/restart enable, interval turn-down/turn-up via `SetOptions`, on-cadence writes, random write bursts with a recovery-rate sweep, and the naive-jitter baseline. It is the basis for the numbers cited above. Interactive HTML (**NOTE** fb-internal URL sanitization might require you to copy-paste the URL yourself): 5759c6a62d/pc_phase_sim.html Reviewed By: jaykorean Differential Revision: D117487278 fbshipit-source-id: a62439548b7dd6fe7dfdbbe21b69531d8b57a1e8	2026-08-26 20:14:26 -07:00
Peter Dillinger	8f492b2f47	Store common user-key prefix once per block (format_version 8) (#15117) ... Summary: Building on D116105558, this change finishes and publishes `format_version = 8` as production-ready. The headline feature is a tweaked block encoding that stores the common user-key prefix shared by all keys in a block **once**, in the block's leading bytes `[0, restarts[0])`, instead of repeating it at every restart point. This shrinks prefix-heavy blocks (and their block-cache footprint) and, for bytewise-ordered keys, speeds up `Seek` by comparing prefix-stripped suffixes. It is applied to **data blocks** and **every index tier** (leaf/partition index, the partitioned-index top-level, and the partitioned-filter top-level). Motivation: MyRocks/ZippyDB/timeseries-style keys carry a long shared prefix (table/index id, partition, etc.). Repeating it at each restart wastes space and cache, and every Seek re-compares those bytes. Profiles show index-block seek CPU is comparable to data-block seek CPU (partitioned index does two index seeks per data seek), so index blocks are a first-class target, not just data blocks. ### Format (self-describing, no new footer bit) - The common prefix occupies the block's leading bytes; a non-zero `restarts[0]` (the offset of the first entry, i.e. the prefix length) self-signals its presence. `restarts[0]` is 0 in every block that does not use the feature (all versions, all block types), so a single fv8 reader transparently handles any mix of stripped and unstripped blocks -- no footer bit is consumed. - Previous versions would misinterpret this encoding by quietly ignoring the common prefix, which (along with the lack of extensibility in data-block metadata) is why it is gated behind format_version 8. - Only restart-point entries change on disk; non-restart entries are byte identical to fv7, so the writer produces the exact same bytes regardless of implementation. ### Public option New `BlockBasedTableOptions::optimize_key_common_prefix` (`enum class OptimizeKeyCommonPrefix`), default `kIfFastSeek`: - `kDisabled` -- never use it. - `kIfFastSeek` (default) -- use it only where it also enables the Seek speedup: format_version >= 8 with a built-in (reverse-)bytewise comparator. - `kEnabled` -- use it for all comparators at fv8; (reverse-)bytewise get space + fast seek, other comparators get the space savings only (the reader reconstructs full keys and compares with the real comparator). Expressed as a criteria so it need not be toggled per CF based on the comparator (which can vary by CF). It additionally requires `use_delta_encoding` (the feature builds on delta encoding) and `ts_sz == 0`; when `use_delta_encoding` is false -- the setting that also preserves zero-copy key pinning -- it is a no-op. Default format_version is unchanged (still 7); the feature is opt-in via fv8. ### Scope: every index tier The reader handles all index-like blocks uniformly (a non-zero `restarts[0]` self-signals a stripped block regardless of block type), so the writer strips the common prefix from every tier: - **Data blocks** and **leaf / partition index blocks**: enabled. - **Partitioned-index top-level block**: enabled. Under the default `index_shortening = kShortenSeparatorsAndSuccessor` the file's last separator would otherwise be a short successor sharing no leading bytes, collapsing the single-block top-level's common prefix to 0. When common-prefix is on the builder suppresses that one successor (keeps the full last key -- i.e. `kShortenSeparators` behavior for that one entry), so the top-level strips uniformly under every shortening mode; the full last key is also a tighter file upper bound, so it is not worse. - **Partitioned-filter top-level index**: enabled (same uniform treatment). Its space/CPU effect is small at default partitioning (the top-level filter index is tiny) but grows with many small partitions -- a filter miss short-circuits at, and is dominated by, the top-level filter-index seek (see Performance). Included for a uniform on-disk surface across tiers. Bloom filter *contents* still hash full user keys, so prefix/whole-key filtering is unaffected. ### Implementation - Writer (`block_builder.{h,cc}`): keys are encoded incrementally in the fv7 layout while the running common prefix is tracked cheaply (reusing the delta `shared`); at `Finish()` only the restart-point entries are rewritten in place with the prefix stripped, and the prefix is written once at the block start. The rewrite is generalized to the value-delta (fv4) index layout used by index blocks (no `value_size` header; bulk-copy the inline BlockHandle and non-restart entries verbatim). - Reader (`block.{h,cc}`, `dbformat.h`): `Block` decodes the prefix from `restarts[0]`; `BlockIter::ParseNextKey` prepends it on a restart point (`shared == 0`). The suffix-seek helpers (strip target once, compare shortened suffixes) live in the templated `BlockIter` base and are shared by data and index iterators (binary search, templated on `DecodeKey`/`DecodeKeyV4`); for non-bytewise comparators (data) and the hash `PrefixSeek` refine step, the reader reconstructs the probed restart key and compares full keys. - Gating (`block_based_table_builder.cc`, `index_builder.{h,cc}`, `partitioned_filter_block.{h,cc}`): the criteria is threaded to the data-block builder and to every index tier -- leaf/partition index, the partitioned-index top-level (`use_common_prefix_top`), and the partitioned-filter top-level -- together with the last-separator successor suppression (`ShortenedIndexBuilder::GetSeparatorWithSeq`) that lets the top-level strip under the default shortening mode. The index base gate also requires `super_block_alignment_size == 0` (block alignment can trigger `skip_delta_encoding`, producing non-restart `shared == 0` entries the reader could not distinguish from stripped restart keys). - Memory: the per-block/per-iterator prefix is stored as a `uint32_t` length (the prefix is always at offset 0) rather than a full `Slice`, and `Block` / `BlockBuilder` members were repacked so the feature adds little or no struct size. - Java: a new `OptimizeKeyCommonPrefix` enum and a `BlockBasedTableConfig` getter/setter, plumbed through JNI in both directions (the `newTableFactoryHandle` writer and the options-reconstruction constructor). ### Whole-file (table-level) prefix: investigated and rejected A natural extension is to store the whole-file common prefix once per *file* (when an `SstPartitioner` guarantees every key in the file shares a fixed prefix) instead of once per *block* -- e.g. the MyRocks index/table id, or a future 128-bit global id. It was prototyped end-to-end (partitioner-provided prefix captured at file open, stored in a `user_key_common_prefix` table property, and stripped from every data/index block; the reader re-prepends it) and then dropped. The `user_key_common_prefix` table property reserved in the previous diff is dropped in this one. The rationale, in case it is revisited: - Marginal benefit over per-block. Per-block already strips the full shared prefix from every key; the only thing whole-file removes on top is the single per-block copy of that prefix, i.e. `~= prefix_len / block_size` of the uncompressed (block-cache) footprint, with no meaningful read-CPU change (per-block already delivers strip-once + suffix compares). Measured at prefix_len=12, 4 KB blocks this was ~-0.3% incremental space; at the 8-16 KB data blocks typical of the space-constrained target (MyRocks/UDB) it is ~0.1-0.2% even with a 16-byte global id -- a fraction of a percent, seen only by partitioner users. Small block-size workloads are becoming more niche, and even where they exist, they are also CPU-sensitive (see next paragraph). - Real, if small, CPU tax on the *per-block* path. Reconstructing a key from two non-contiguous parts (file prefix + block residual) cost ~0.5-1.0% additional retired instructions on the per-block read path (deterministic across gcc and clang), wall-clock-neutral in cache-resident tests but present even when no whole-file prefix is configured. Attempts to remove it did not pan out: a compile-time specialization of the reconstruction *increased* instructions (+2.9%) by defeating inlining; a per-block combined-prefix buffer trades the CPU cost for a memory cost on every (default) block; and a per-iterator buffer was the original larger regression. Only invasive iterator-level monomorphization would truly zero it, which is not justified on this hot path. - Decision. A fraction-of-a-percent, partitioner-only space win does not justify a permanent per-block-read-path CPU tax plus additional on-disk format surface (a table property that changes whole-file interpretation). Dropped. The generic `SstPartitioner::ShouldPartitionByPrefix` hook is kept (cheap, and useful independently), and the reserved `user_key_common_prefix` table property is removed since no writer will emit it. ### Performance (db_bench, release/native, cache-warm, best-of-3, pinned) Three arms: (A) fv7 on the base/main branch, (B) fv7 with this change (should be neutral / byte-identical writing), (C) fv8 with this change and `optimize_key_common_prefix=enabled` (the optimization is active). Partitioned index + filters; `compression_type=none`, so SST bytes are the uncompressed block-cache footprint. `int` = db_bench default keys (variation-early, near worst case); `pfxN` = an N-byte shared prefix + varying suffix (MyRocks-like). Setup: `num=3,000,000`, `reads=1,500,000`, single thread, `metadata_block_size=4096`, cache holds the working set, pinned to one core. | shape (key/val) | arm | SST bytes | readrandom us/op | seekrandom us/op | |---|---|---:|---:|---:| | int (32/8) | A fv7 main | 138,983,658 | 3.484 | 4.995 | | | B fv7 ours | 138,983,146 | 3.518 | 5.024 | | | C fv8 ours | 137,681,420 | 3.490 | 4.986 | | pfx24 (32/8) | A fv7 main | 71,697,531 | 3.417 | 5.561 | | | B fv7 ours | 71,698,043 | 3.448 | 5.579 | | | C fv8 ours | 65,578,236 | 3.284 | 5.386 | | pfx56 (64/16) | A fv7 main | 102,942,266 | 3.624 | 5.830 | | | B fv7 ours | 102,942,266 | 3.678 | 5.835 | | | C fv8 ours | 90,554,642 | 3.444 | 5.626 | fv8 vs fv7-main (C vs A) -- space / readrandom / seekrandom: - int (32/8): **-0.9%** / ~neutral / ~neutral - pfx24 (32/8): **-8.5%** / **-3.9%** / **-3.1%** - pfx56 (64/16): **-12.0%** / **-5.0%** / **-3.5%** fv7-ours vs fv7-main (B vs A): SST bytes byte-identical (within 512 B; the fv8 paths are gated off), read/seek us/op within run-to-run noise (~1%, best-of-3 on a shared host). Net: fv8 is a space **and** read/seek-CPU win on prefix-heavy data and neutral on variation-early (`int`) data, so `kIfFastSeek` is safe on by default at fv8. ### Space by tier (compression=none, num=3M, mdb=4096; fv7 -> fv8) Where the win comes from -- the index tiers strip the most: | shape | data | leaf/partition index | top-level index | filter | |---|---:|---:|---:|---:| | int | -0.8% | -29% | -45% | -0.2% | | pfx24 | -8.3% | -69% | -89% | -0.8% | | pfx56 | -11.1% | -83% | -96% | -1.6% | Data bytes dominate the absolute total, but every index tier is stripped: leaf/partition index -29% to -83% and the partitioned-index top-level -45% to -96%. Filter *blocks* barely move (-0.2% to -1.6%) because filter bits hash full user keys; only the small top-level filter index is stripped. ### Point-miss lookups (readmissing) A partitioned-filter miss short-circuits at the filter (no data-block read), so it is dominated by the top-level filter-index seek plus the partition bloom. At a typical `metadata_block_size=4096` (few partitions, tiny top-level filter index) fv8 is neutral on misses (int +0.9%, pfx24 -1.0%, pfx56 -0.7% us/op -- all noise). The top-level-filter stripping pays only with many small partitions: at `metadata_block_size=256` it is pfx24 1.166->1.138 (**-2.4%**) and pfx56 1.273->1.188 (**-6.7%**) us/op. ### Backward / forward compatibility - Default `format_version` remains 7; the feature is inert unless a CF opts into fv8. Older RocksDB rejects fv8 files at footer decode (fail-safe). - Within fv8, the stripped-block set a writer can emit is fixed and every fv8 reader can read it (the `restarts[0]` signal plus uniform reader handling), so the on-disk surface is self-consistent across builds/downgrade. ### Follow-ups (not in this change) - Suffix-aware interpolation / hash index seek (currently correct via the reconstruction fallback, not perf-tuned). Pull Request resolved: https://github.com/facebook/rocksdb/pull/15117 Test Plan: - New focused unit tests (`table_test`), run at the full `format_version` matrix including the now-supported fv8: `GeneralTableTest.OptimizeKeyCommonPrefix` (data-block space+read+Seek correctness: existing and missing keys straddling the prefix, before/after-all, SeekForPrev), `OptimizeKeyCommonPrefixNonBytewise` (reconstruction path), and `IndexBlockCommonPrefixEquivalence` across `{kBinarySearch, kBinarySearchWithFirstKey, kHashSearch, kTwoLevelIndexSearch}` x `{partitioned filters}` x `{bytewise, reverse-bytewise}` (so both top-level tiers -- partitioned-index and partitioned-filter -- are exercised), asserting identical reads and a shrunken fv8 index. - The randomized `ParameterizedHarnessTest` now runs at fv8 (reverse-bytewise, restart intervals `{1,16,1024}`, special/empty keys fuzzed against the feature). - `db_crashtest.py` randomizes `format_version` including 8 and `optimize_key_common_prefix` in {0,1,2}. - Removed the reserved-property carve-out: dropped the `ReservedUserKeyCommonPrefix` reader test and the C API getter for the property; C API regenerated via `tools/c_api_gen/` (`make regen-c-api`), and the intentional public-C-API removal is recorded in `tools/c_api_gen/api_compatibility_allowlist.json`. - Java: `BlockBasedTableConfigTest.optimizeKeyCommonPrefix` (default `kIfFastSeek`, plus set/get of every value). - `table_test` `FooterTests` updated for the now-published fv8 footer, whose metaindex-gap encoding stores the metaindex block size in 16 bits (the test caps that size for fv >= 8). - Manual verification: the db_bench runs in Performance above (three-arm space + read/seek CPU, per-tier byte breakdown via `sst_dump`, and `readmissing`); and `readrandom`/`seekrandom` with `--block_protection_bytes_per_key=8` find 100% of keys at both fv7 and fv8 (reconstructed keys pass per-key checksum). Reviewed By: xingbowang Differential Revision: D116365016 Pulled By: pdillinger fbshipit-source-id: 66f4b5c60ea4fe0ff48d34c98d191ba8ebdf0665	2026-08-20 17:32:41 -07:00
Peter Dillinger	d3a82f28ce	Optimize SstPartitioner overhead in compaction via ShouldPartitionByPrefix() (#15118) ... Summary: When a column family configures an SstPartitioner, compaction paid a per-key cost for every key-value written to an output SST: a full user-key copy (into `last_key_for_partitioner_`) plus a virtual `SstPartitioner::ShouldPartition()` call (which also builds a PartitionerRequest and truncates/compares two Slices). This is motivated by an internal user whose custom SstPartitioner parses a variable-length prefix (a prefix code) out of each key to decide partition boundaries; today that prefix parsing runs on every key written during compaction, on top of the copy and virtual call above. This wires up the `SstPartitioner::ShouldPartitionByPrefix()` API (added, but unused, alongside the format_version=8 groundwork) so compaction can skip that per-key work. When a partitioner can express its next boundary as a key prefix, `CompactionOutputs` caches that prefix once per output file (re-derived from the file's first key) and replaces the per-key copy + virtual call with an inlined `Slice::starts_with()` check. That also lets a partitioner like the one above do its prefix parsing once per output file instead of once per key. When the partitioner returns no prefix (ambiguous, e.g. `SstPartitionerFixedPrefix` for keys shorter than its length), compaction falls back to the previous per-key `ShouldPartition()` path. Output file boundaries are byte-for-byte identical to before. `SstPartitionerFixedPrefix` implements `ShouldPartitionByPrefix()`; the base-class default returns "no prefix", so existing custom partitioners are unaffected until they opt in. Also: - db_bench: new `--sst_partitioner_fixed_prefix_len` to configure a fixed-prefix partitioner (enables benchmarking; no default behavior change). - `ShouldPartitionByPrefix()` API doc updated now that it is used and recommended for compaction CPU efficiency (any prefix code works, not just fixed-size). No new option and no behavior or format change. Pull Request resolved: https://github.com/facebook/rocksdb/pull/15118 Test Plan: Added db_compaction_test cases covering the new fast path and its fallbacks: many keys sharing a prefix (one output file per prefix, no spurious cuts), keys shorter than the fixed prefix length (fallback to ShouldPartition), the zero-length-prefix "never partition" case, and short<->long key transitions across output-file boundaries. Existing SstPartitioner tests and the range tombstone + partitioner tests (db_range_del_test) provide regression coverage that file boundaries are unchanged, including the fast path with range tombstones. The crash test already exercises the partitioner via `enable_sst_partitioner_factory`. Benchmark (db_bench `compact` of a pre-built 50M-key DB; 16B keys / 8B values, 8-byte shared key prefixes with FixedPrefix length 8, no compression, single subcompaction). A/B compares this change against the same binary with only the compaction hot-path change reverted. Means of 6 trials, optimized faster on every trial and every metric; both variants produced identical output (50 files, i.e. identical partition boundaries): | Metric | Baseline | Optimized | Delta | |---------------------------------|----------|-----------|--------| | Instructions (whole process) | 92.46 B | 90.09 B | -2.6% | | CPU cycles | 46.20 B | 45.28 B | -2.0% | | CPU time (task-clock) | 18.74 s | 18.39 s | -1.9% | | `compact` wall time | 20.58 s | 20.08 s | -2.4% | DB-internal compaction stats corroborate: CompMergeCPU 15.84s -> 15.42s (-2.6%), compaction write throughput 38.9 -> 39.9 MB/s. The relative gain shrinks with larger values or compression enabled, where the partitioner is a smaller fraction of total compaction work; it is a pure win whenever a prefix partitioner is configured. Reviewed By: rban1 Differential Revision: D116372363 Pulled By: pdillinger fbshipit-source-id: 63de190329112aa2ce5b48da9fc77ef386494558	2026-08-19 11:43:10 -07:00
Peter Dillinger	495c21dc5c	Unify lazy wide-column blob read verify policy; tool coverage (#15115) ... Summary: Follow-ups to the lazy wide-column read API (DB::GetEntityLazy / MultiGetEntityLazy). Three related, low-risk changes under one theme; no public API change. 1. Internal read-API cleanup (retires TODO(lazy-blob-resolution-cleanup)). The lazy byte-range work had left a whole/range/force-verify method fan-out on the blob read path: VersionBlobFetcherBase::FetchBlobRange plus a separate FetchBlobForceVerify, and SameFileBlobReader::GetSameFileBlob paired with GetSameFileBlobRange, with the (verify_checksums, force_verify) decision duplicated across branches. This folds them into a single 3-valued BlobVerifyPolicy (kVerifyIfPresent / kVerifyIfNoAmplification / kSkip, added to db/blob/blob_constants.h) derived once at the resolver boundary and threaded down. FetchBlobRange and GetSameFileBlob now take a range descriptor (range_length == kWholeBlobLength means the whole value) plus the policy; FetchBlobForceVerify and the *Range twins are removed. The public LazyColumnReadRequest::force_verify bool is unchanged. The widely-called lower-layer whole-value primitives (BlobFileReader / BlobSource / Version GetBlob) are intentionally left as delegated primitives to bound blast radius; folding those is a later cleanup. 2. Embedded block-cache-tier quirk fix (retires TODO(lazy-blob-resolution-quirks)). ValidateEmbeddedBlobIndex rejected read_tier == kBlockCacheTier before the blob-cache probe, so a same-file (embedded) column already in the blob cache returned Incomplete under kBlockCacheTier -- unlike the separate-file path, which probes the cache first. The up-front rejection moves to the two no-cache paths (ResolveEmbeddedBlob / ResolveEmbeddedBlobPinned); the cache-probing paths now serve a cached record under kBlockCacheTier, matching separate-file behavior. This is a shared lower-layer change: the non-lazy Get / GetEntity and iterator embedded read paths (get_context / MaybeResolveEmbeddedValue -> ResolveEmbeddedBlobCached) get the same corrected behavior, not just the lazy resolver. 3. Coverage. db_bench gains a readrandomentitylazy benchmark (GetEntityLazy + MultiResolve of --lazy_entity_read_length bytes per column) for measuring byte-range (partial) blob read savings; it requires open_files == -1. The crash test now also fuzzes LazyColumnReadRequest::force_verify in the lazy-vs-eager differential (force_verify affects only verification, not the returned bytes, so the comparison is unchanged). This also sets up the next planned step -- cross-key coalescing for MultiGetEntityLazy / LazyWideColumnsBatch::MultiResolve. The internal BlobReadRequest already carries a {offset, len} range descriptor, and with the verify decision now a single BlobVerifyPolicy rather than a bool plus a separate force-verify method, a batched resolve can group a vector of range+policy requests per (CF, Version, blob file) and coalesce them onto the existing MultiGetBlob machinery -- without re-introducing whole/range/force-verify method twins in the multi-read path. Pull Request resolved: https://github.com/facebook/rocksdb/pull/15115 Test Plan: - New unit test DBLazyEntityTest.EmbeddedBlockCacheTierServedFromCachedValue: a cached embedded column serves whole- and sub-range reads under kBlockCacheTier instead of returning Incomplete. Existing force_verify / BlockCacheTier tests are unchanged (public behavior preserved). - New unit test DBBlobIndexTest.EmbeddedBlobWideColumnGetEntityBlockCacheTierServesCached: covers the non-lazy GetEntity path -- a cached embedded column is served under kBlockCacheTier; the existing ...GetEntityBlockCacheTier test still asserts a genuine miss yields Incomplete. - Crash test: extended the existing lazy differential to fuzz force_verify (reuses --lazy_entity_read_one_in). - db_bench: readrandomentitylazy over 2000 entities x 2 uncompressed embedded blob columns (4000 B each), reading 100 B per column: | metric | value | |---|---| | rocksdb.blobdb.lazy.partial.read.count | 4000 | | rocksdb.blobdb.lazy.read.bytes | 400000 | | rocksdb.blobdb.lazy.partial.bytes.saved | 15600000 | i.e. 100 B read per column instead of the 4000 B whole-column value. Reviewed By: xingbowang Differential Revision: D116334969 Pulled By: pdillinger fbshipit-source-id: cebe90456b23be0f5be60cdc4bff85c8a37b5442	2026-08-17 17:06:32 -07:00
Michael C Huang	276495843d	Add blob file writer buffer size option (#15061) ... Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15061 Adds mutable ColumnFamilyOptions field `blob_file_writable_file_max_buffer_size` to cap blob-file WritableFileWriter buffers independently from DBOptions::writable_file_max_buffer_size. The default value is 0, which preserves existing behavior by inheriting the DB-wide writer buffer cap. Non-zero values apply to both flush/compaction blob writers and direct-write blob writers. Direct-write updates are propagated through DBImpl::SetOptions to newly opened blob files without requiring a DB restart. Also wires option parsing, dumping, db_bench, db_stress, crashtest, and release-note coverage. Reviewed By: xingbowang Differential Revision: D114979265 fbshipit-source-id: 3f764b797628eee0ce379110c1a1f8a16beb3fa2	2026-08-11 17:38:34 -07:00
Peter Dillinger	fcf382c95c	Integrated and ratio-aware compression auto-skip (#15079) ... Summary: Auto-skip compression (stop attempting compression on data that isn't compressing well, to save CPU) was previously an experimental CompressionManager wrapper (CreateAutoSkipCompressionManager) whose predictor lived in each per-thread working area. Besides being fragmented across parallel workers and reset on a tumbling window, it decided based on how OFTEN blocks failed to compress, ignoring how MUCH they compress -- so on mixed data it could give up substantial compression to save a little CPU (see performance results). This makes auto-skip a first-class, optional, ratio-aware feature of the block-based table builder, controlled by two new CompressionOptions fields: - auto_skip (default false): the on/off switch. There is no separate threshold knob; the decision reuses max_compressed_bytes_per_kb (the existing per-block "minimum compression worth keeping" bar) as the bar for whether attempting compression on the stream is worthwhile. - auto_skip_min_sample_every: sampling interval that bounds the local impact around a transition back to compressible data. 0 selects an internal default (currently 64 skipped data blocks between forced samples). The emit thread makes the skip/attempt decision once per data block (serial and parallel paths alike; data blocks only), so a skipped block is never dispatched to a compression worker. It is driven by a continuously updated, shared estimate q_hat of the aggregate stored/uncompressed ratio the builder would achieve by attempting -- each block counted at its achieved ratio, or 1.0 when stored raw: attempt while q_hat is at or below the bar, otherwise skip. Because the estimate is payoff-weighted, a stream whose blocks each save little -- or where only a few blocks compress well -- does not clear the bar and is skipped, while a stream that compresses substantially is attempted. The compression given up per KB is bounded by (1024 - max_compressed_bytes_per_kb). q_hat uses an asymmetric exponentially weighted moving average (EWMA), slightly faster toward better compression than toward worse; the fast side is deliberately gentle so a single marginally-compressible sample does not flip a whole file back to attempting. It is kept on its own cache line to avoid false sharing. Under parallel compression q_hat is updated with relaxed atomics, and the emit thread can decide from a value that lags the learning it drives by the in-flight depth -- tolerable concessions of accuracy and responsiveness for a coarse heuristic. The first data block of each file is always attempted (the sample countdown starts at 0), and while skipping the builder keeps sampling (one block after a randomized gap) so a return to compressible data is detected. q_hat is carried across the files a flush/compaction thread emits via thread_local state keyed by preferred compression type, so a sustained region does not re-learn on every file. The old CompressionManager-based AutoSkip is removed (CostAware is retained); db_stress, db_bench, and db_crashtest are migrated to the new option. db_bench is extended so it can generate the relevant regimes: --compressible_data_fraction and --compressible_data_run_length produce a mix of compressible/incompressible blocks -- i.i.d. per value (run length 1) or in alternating regions (large run length). The default (fraction 1.0) reproduces the historical single-pool behavior. Pull Request resolved: https://github.com/facebook/rocksdb/pull/15079 Test Plan: New/updated unit tests in `util/compression_test.cc` (`DBAutoSkipTest`): - skips sustained incompressible data; - never skips compressible data; - disabled by default never bypasses a block; - 4-thread parallel compression round-trips all data across alternating compressible/incompressible regions. **Performance** (`db_bench fillseq`, release, zstd level 3; 100k values of 16 KiB each, roughly one value per data block; WAL and auto-compaction off; median of 5 runs on a quiet host). Auto-skip trades one cost for another, so each cell reports two costs, both lower-is-better: CPU cost as process CPU time (user+sys) per GB of uncompressed input (s/GB; ~1.64 GB uncompressed here), and storage cost as the resulting DB size. Auto-skip does not appear to change parallelizability, so wall-clock s/GB tracks CPU s/GB (the two agree here); CPU s/GB is reported. Percentages are relative to the no-auto-skip baseline: negative CPU = cheaper, positive size = more storage (shown only where storage grows). | Workload | baseline (off) | (a) old wrapper | (b) draft | (c) this version | |---|---|---|---|---| | Incompressible | 2.33 s/GB, 1568 MB | 0.96 (-59%), 1568 MB | 0.82 (-65%), 1568 MB | 0.82 (-65%), 1568 MB | | Compressible | 3.55 s/GB, 710 MB | 3.54 (~0%), 710 MB | 3.56 (~0%), 710 MB | 3.55 (~0%), 710 MB | | Marginal-payoff mix | 3.80 s/GB, 1376 MB | 1.54 (-59%), 1523 MB (+11%) | 3.81 (~0%), 1376 MB | 1.43 (-62%), 1531 MB (+11%) | | High-payoff mix | 2.20 s/GB, 929 MB | 1.14 (-48%), 1417 MB (+53%) | 2.16 (~0%), 929 MB | 2.17 (~0%), 929 MB | A strategy wins by cutting CPU cost without paying much storage cost. On incompressible data all three cut CPU ~60% at no storage cost; on compressible data none skip (within run-to-run noise). The two mixes (50% incompressible, 50% compressible per block) separate the strategies: - Marginal-payoff mix: the compressible half barely clears the per-block bar, so the stream is not worth compressing -- skipping is right. (c) and (a) cut CPU ~60% for ~11% more storage. That extra storage is compression (c) intentionally declines because its aggregate benefit is below the configured worthwhile threshold (max_compressed_bytes_per_kb, ~12.5% per KB); it stays within the (1024 - max_compressed_bytes_per_kb) bound. The ratio-unaware draft (b) cannot see the low payoff, keeps compressing, and saves no CPU. - High-payoff mix: the compressible half compresses hard, so compressing is right. (c) and (b) keep full compression at baseline storage; the old wrapper (a) cuts CPU 48% but inflates storage +53% -- far beyond that bound -- giving up worthwhile compression. Only (c) makes the right call on both mixes: it never gives up worthwhile compression (unlike (a)) and does not waste CPU on low-payoff data (unlike (b)). Reviewed By: xingbowang Differential Revision: D115444567 Pulled By: pdillinger fbshipit-source-id: 05e122d25de95ddcd0edeb0dd07c9d25bead7293	2026-08-11 14:29:51 -07:00
Josh Kang	5dfde1c262	Add coroutine reads to stackable RocksDB utilities (#15080) ... Summary: ## Summary Pull Request resolved: https://github.com/facebook/rocksdb/pull/15080 Introduce reusable `CoroStackableDBBase` forwarding and public `CoroDB::CoGet()` / `CoMultiGet()` entry points matching the `DB::Get()` / `MultiGet()` overloads. These helpers schedule native coroutine reads on the read executor and use the synchronous DB API when coroutine support or a read executor is unavailable. Use the stackable coroutine hooks for TTL and WritePrepared transaction reads. TTL preserves expiration decoding and timestamp behavior, while WritePrepared preserves transaction visibility. TTL keeps a single `DB` inheritance path while adding `CoroDB`: ``` DBWithTTLImpl `-- CoroStackableDBBase<DBWithTTL> |-- DBWithTTL -> StackableDB -> DB `-- CoroDB ``` `CoroDB::CoGet()` dispatches through the stackable coroutine hook and schedules the operation on the read EventBase. `CoMultiGet()` follows the same path: ``` CoroDB::CoGet(db, ...) |-- db->GetCoroDB() -> DBWithTTLImpl |-- no CoroDB or read executor -> db->Get() `-- schedule on read EventBase `-- virtual CoroDB::GetCoroutine() `-- DBWithTTLImpl::GetCoroutine() |-- CoroStackableDBBase<DBWithTTL>::GetCoroutine() | `-- wrapped CoroDB::GetCoroutine() `-- strip TTL timestamp ``` Reviewed By: xingbowang Differential Revision: D115232510 fbshipit-source-id: 59e5090db28d15bcff1a8b9b8c0b30ca7eeb742d	2026-08-10 20:28:02 -07:00
Josh Kang	88752298ff	Add native coroutine read dispatch to RocksDB (#15073) ... Summary: ## Summary Pull Request resolved: https://github.com/facebook/rocksdb/pull/15073 The callback-based async read APIs make coroutine-native implementations and StackableDB composition awkward because wrappers must bridge callbacks back into tasks. This change introduces `CoroDB` as a side capability exposing lazy `GetCoroutine` and `MultiGetCoroutine` tasks. `DBImpl` and `CompactedDBImpl` implement this capability directly, while `DB::GetAsync` and `DB::MultiGetAsync` remain available as compatibility entry points with synchronous fallback. The native interface should also be more efficient because callers directly await the DB coroutine. It avoids the callback/Baton adapter, callback context handoff, and explicit rescheduling required for each request. Native tasks must run on the DB filesystem read executor's EventBase; debug builds verify this affinity. Public API documentation, a release note, and native-coroutine support in `db_bench` and `db_stress` are included. ## Dispatch DB::GetAsync -> GetCoroDB() on the concrete DB -> virtual CoroDB::GetCoroutine(...) -> DBImpl::GetCoroutine or CompactedDBImpl::GetCoroutine -> AsyncCallback::OnComplete(...) Virtual dispatch selects the coroutine implementation belonging to the concrete DB while preserving the existing callback API. ## Statistics Coroutine reads preserve RocksDB's TLS statistics behavior while keeping concurrent requests isolated. Each top-level task copies only the caller's statistics configuration—`PerfLevel`, per-level perf context enablement, and IO statistics enablement—into fresh request-local `PerfContext` and `IOStatsContext` objects. Existing counters are not inherited. If both perf and IO statistics are disabled, no request statistics context is installed. Folly `RequestContext` preserves these contexts across suspension and resumption on the read executor's EventBase. At completion, RocksDB publishes the request's collected values to TLS, allowing the awaiting caller to use `get_perf_context()` and `get_iostats_context()` as after a synchronous read. Callers must consume or copy them before leaving that EventBase. Statistics scopes are installed only by top-level coroutine reads and cannot be nested. ## Benchmarks Compared the previous `GetAsync` callback bridge with native `GetCoroutine`. Both variants used optimized builds, the same compacted DB, a fixed seed, eight executor threads, ten jobs per thread, and 2.56M reads per invocation. Results are the mean of three independent invocations using the default RocksDB block cache. A sequential read warmed the OS page cache before the non-direct-read measurements. | Workload | `GetAsync` bridge | Native `GetCoroutine` | Change | | Direct I/O | 237,710 ops/s | 243,588 ops/s | +2.47% | | Warm OS page cache | 783,859 ops/s | 891,531 ops/s | +13.74% | The direct-I/O difference is within observed run-to-run variation. When reads are served from the OS page cache and API overhead is a larger portion of latency, the native interface improves throughput by 13.74%. Every invocation completed all requested reads. ```bash db_bench \ --benchmarks=readrandomcoroutine \ --db=$CORO_BENCH_DB \ --use_existing_db=true \ --num=2000000 \ --value_size=1024 \ --compression_type=none \ --threads=8 \ --coro_jobs_per_thread=10 \ --reads=320000 \ --use_direct_reads=<true|false> \ --perf_level=1 \ --statistics=true \ --seed=20260807 ``` Reviewed By: xingbowang Differential Revision: D115232511 fbshipit-source-id: 30b48da2a798094d9080178f5430aae26e6c5954	2026-08-10 09:43:05 -07:00
Peter Dillinger	afc922bf5f	Make ReadOptions::table_filter a caller-owned pointer for cheaper copies (#15005) ... Summary: ReadOptions is copied frequently on read paths, and its only non-trivially-copyable member, table_filter, is a std::function that is almost always empty yet still gets deep-copied on every ReadOptions copy (and can heap-allocate). This changes table_filter to a pointer to a caller-owned std::function instead of storing one by value: when non-nullptr, the caller is responsible for keeping the pointed-to function alive for the duration of the read (like the other pointer fields such as iterate_upper_bound). With this change every ReadOptions member is trivially copyable, so copying ReadOptions no longer touches a std::function at all. That trivial-copyability is locked in with a static_assert in options_test.cc (kept out of the public header). This is inspired by D113109635, which targeted the same ReadOptions-copy hotspot by relocating table_filter to the end of the struct so the trivial members memcpy in one contiguous block. This is intended to be a superior solution: rather than only defragmenting the copy, it removes the std::function copy entirely. This is a breaking API change, tolerated because table_filter is a relatively obscure option. A nullptr, or a pointer to an empty std::function, means no filtering (preserving the previous empty-function behavior). A shared internal helper HasTableFilter(const ReadOptions&) in options_helper.h centralizes the nullptr + empty-function check used on the read paths. The C API and Java bindings now own the std::function and hand ReadOptions a pointer to it; their public signatures are unchanged, and the C API rocksdb_readoptions_t clears the owned function when the callback is null. Making ReadOptions trivially copyable also let -Werror=unused-but-set-variable fire on a pre-existing dead local in db_bench's MultiScan: a ReadOptions that set async_io/readahead_size but was never passed to NewMultiScan (the warning had been suppressed while ReadOptions had a non-trivial member). That local was in fact inert for multiscan -- BlockBasedTableIterator::Prepare overrides ReadOptions::async_io with MultiScanArgs::use_async_io for the coalesced prefetch, so multiscan async I/O is driven by --multiscan_use_async_io, not ReadOptions::async_io. Removed the dead local; MultiScan now uses read_options_ directly, consistent with MultiScanRandom. Pull Request resolved: https://github.com/facebook/rocksdb/pull/15005 Test Plan: Existing table_filter coverage was adapted to the new pointer API and still exercises the behavior end to end: db_iterator_test.TableFilter (callback invoked once per table; returning false hides a table) and its range-conversion rejection case; db_bloom_filter_test SstQueryFilter/FixedWidthSegments range queries; and trie_index_db_test's range-query iterator (which returns iterators outliving the filter, exercising the caller-owned lifetime). db/c_test covers the C API set/clear/has_table_filter path. options_test asserts std::is_trivially_copyable_v<ReadOptions>. Manually verified the db_bench MultiScan async behavior with a ~200MB DB and --use_direct_reads: with ReadOptions::async_io held true, async reads tracked only --multiscan_use_async_io (rocksdb.read.async.micros count 329511 with use_async_io=true vs 0 with false), confirming ReadOptions::async_io does not drive multiscan async and the removed local had no effect. Reviewed By: anand1976 Differential Revision: D113613525 Pulled By: pdillinger fbshipit-source-id: 38d73b607500a5c7ca2b89dd8f4187ef43940aa4	2026-07-31 10:20:25 -07:00
Josh Kang	7247edbf0a	Add opt-in stats for async RocksDB reads (#15017) ... Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15017 ## Summary RocksDB async `GetAsync` and `MultiGetAsync` callbacks now default to a completion-only path. Callbacks can override `EnableStats()` when they need request-specific `PerfContext` or `IOStatsContext`; otherwise RocksDB avoids creating a coroutine stats context and returns null stats pointers. Callbacks must not initiate another async read from `OnComplete()`. RocksDB stats normally use thread-local storage, which does not compose with interleaved coroutine reads. A stats-enabled request captures only the caller's configuration: perf level, per-level perf enablement, and whether io stats are disabled. It then starts with empty request contexts. `onSet()` loads that request's contexts into TLS, while `onUnset()` saves the accumulated contexts back into request data. On resume, the same contexts are loaded again. This isolates request counters without preserving or restoring pre-existing executor TLS state. The synchronous fallback similarly resets TLS before executing a stats-enabled read. `PerfContext` and `IOStatsContext` gain `Merge()` support for caller-side aggregation. `PerfContext` move construction and assignment now transfer ownership of per-level state rather than deep-copying it, avoiding allocations during coroutine context handoffs. A fbcode-wide audit found no caller that reads a moved-from `PerfContext`. `db_bench` opts in only for coroutine jobs that print per-job perf context and merges every completed request's callback stats into the job aggregate. ## Example: Two Async Gets on One Thread Both callbacks enable stats. Each request starts with empty counters and carries its contexts across suspensions. ```text Shared executor thread | | onSet(A): load A's empty contexts into TLS | A records 3 reads | onUnset(A): save TLS contexts back into A | | onSet(B): load B's empty contexts into TLS | B records 4 reads | onUnset(B): save TLS contexts back into B | | onSet(A): load A's contexts containing 3 reads | A records 2 more reads | callback A receives block_read_count = 5 | | onSet(B): load B's contexts containing 4 reads | callback B receives block_read_count = 4 ``` Isolation comes from replacing TLS with the active request's saved contexts on every `onSet()`. ### Timers Existing wall-clock timers remain in coroutine frames, so timers spanning `co_await` include suspended time and measure end-to-end latency. Request-level `get_cpu_nanos` is suspension-aware: it starts in `onSet()` and stops in `onUnset()` or `Finalize()`, charging only time while the request is active. Nested CPU timers remain disabled because normal RAII CPU timers would also charge executor work performed while their coroutine is suspended. `MultiGetAsync` fanout additionally requires per-child timer state to avoid double counting. ## Benchmarks Benchmarked to confirm no performance regressions when perf is disabled, and also to see the overhead of enabling perf. ```bash db_bench --benchmarks=<readrandomcoroutine|multireadrandomcoroutine> --db=<db_path> --coro_jobs_per_thread=10 --use_existing_db=true --num=2000000 --value_size=1024 --compression_type=none --threads=8 --batch_size=16 --multiread_batched=true --use_direct_reads=true --cache_size=1428076626 --reads=320000 --perf_level=<1|3> --seed=<seed> ``` | Workload | Before change | Perf level 1 | Before -> 1 | Perf level 3 | 1 -> 3 | | -------- | ------------- | ------------ | ----------- | ------------ | ------ | | Async Get | 475.6K ops/s | 476.7K ops/s | +0.2% | 470.5K ops/s | -1.3% | | Async MultiGet | 567.3K ops/s | 567.4K ops/s | +0.0% | 552.2K ops/s | -2.7% | Reviewed By: xingbowang Differential Revision: D112634874 fbshipit-source-id: 94e240aeb1496fa12dac995b5aa5a9c3543cc5fc	2026-07-28 09:32:26 -07:00
Josh Kang	a4645e864d	Implement Get/MultiGet coroutine (#15014) ... Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15014 Implements coroutine-backed execution behind the callback-based `DB::GetAsync` and `DB::MultiGetAsync` APIs that issue SST block reads asynchronously, so a single application thread can keep many point-lookup reads in flight at once instead of paying one blocking read per thread. The motivation is point-lookup throughput on disk-bound workloads (working set larger than the block cache), where the win comes from queue depth rather than per-read latency. The async capability rests on a small, generic FileSystem API that a `FileSystem` must implement to participate. Every method is virtual with a working synchronous default, so existing and custom `FileSystem` implementations compile and run unchanged (they simply stay synchronous): - `FSRandomAccessFile::SubmitReadAsync(req, opts, cb, dbg)` -- a fire-and-forget read: it starts the read and returns without blocking, then invokes `cb` once `req.result`/`req.status` are filled in (the callback may run on a different thread). The default implementation is synchronous -- a blocking `Read` followed immediately by `cb`. - `FileSystem::GetReadExecutor()` -- returns the `folly::Executor*` that async queries and read completions run on. The default returns `nullptr`, which the query layer treats as "async reads unavailable" and falls back to the synchronous path. It is exposed to users, so they may add additional application logic to run on the same thread pool, thus avoiding expensive context switches. - `FileSystem::SetReadIOExecutorThreads(int)` -- sizes that executor. The default is a no-op. `RandomAccessFileReader::Read`/`MultiRead` are `DEFINE_SYNC_AND_ASYNC` coroutines (new `file/random_access_file_reader_sync_and_async.h`), and `DBImpl::GetAsync`/`MultiGetAsync` bind the query to `GetReadExecutor()` with `co_withExecutor`. For a MultiRead, `SubmitMultiReadAsync` issues all of the block reads through `SubmitReadAsync` back-to-back and then `co_await`s a `folly::coro::Baton`: awaiting the baton suspends the query and releases the worker thread while the reads are in flight, and the callback for the last outstanding read posts the baton to resume it. A MultiGet's per-file `MultiGetFromSST` reads additionally fan out across the current eventbase so overlapping-level lookups run in parallel. This layer works against any executor and any `SubmitReadAsync` implementation. The Posix `FileSystem` is the concrete async implementation, and is where io_uring enters. `GetReadExecutor()` is a `folly::IOThreadPoolExecutor` whose workers are EventBases that each drive an io_uring ring (sized by `SetReadIOExecutorThreads`). `PosixRandomAccessFile::SubmitReadAsync` -- which must be invoked on one of those EventBase workers -- places the read on that worker's submission queue via `IoUringBackend::queueRead` and returns immediately: no blocking `pread`, no parked thread. When the kernel posts the completion the backend runs the callback on that same worker, and a short read re-queues its remainder from inside the completion callback, so even multi-syscall reads stay on the non-blocking path. The net effect is that one worker thread keeps many reads outstanding at once, bounded by io_uring queue depth rather than thread count. ## Thread-local state handling A coroutine suspends at every `co_await` and can resume on a different read-executor worker, and another coroutine can run on that worker in between. Any thread-local state held across a suspension point is therefore unsafe -- it can be observed or mutated by an interleaved coroutine, restored onto the wrong thread, or trip an in-use assertion. The read path handles each thread-local it touches explicitly, and introduces no new thread-local storage of its own: - SuperVersion cache: the thread-local SuperVersion slot is marked `kSVInUse` for the duration of a read, so a second coroutine resuming on the same worker would trip that assertion. The coroutine path instead takes an independent heap reference that releases the slot before any suspension -- `cfd->GetReferencedSuperVersion()` for single Get, and `kExtraSvRef = true` threaded through `MultiCFSnapshot` for MultiGet -- and releases it with `CleanupSuperVersion` (via a `Defer` for single Get). The synchronous path is unchanged and keeps using the thread-local slot (`GetAndRefSuperVersion` / `ReturnAndCleanupSuperVersion`). - perf_context / iostats_context / perf level: the timing guards that save-and-restore or accumulate into these thread-locals across an IO -- `GetPerfLevel`/`SetPerfLevel`, `IOSTATS_TIMER_GUARD(read_nanos)`, `IOSTATS_CPU_TIMER_GUARD(cpu_read_nanos)` -- are compiled out on the coroutine path (`#if defined(WITHOUT_COROUTINES)`), so none of that thread-local timing state spans a `co_await`. The tradeoff is that these perf/IO counters are not populated for coroutine reads (a known limitation of this path). - TLS-free helpers are left as-is: `StopWatch`, for instance, holds only clock/statistics pointers and a start time, so it is safe across a suspension (it can over-count wall time under oversubscription, which is a metrics-accuracy issue, not corruption). ## Benchmarks Benchmarked using 8 threads for both multiget (with/without async_io enabled), and single get on both direct-io enabled and disabled on a ~11 GB DB. Non direct-io benchmarks are not a good representation because my dev machine is capable of paging the entire DB into memory. But, it is still useful to compare against async_io=true, since that feature only benefits non direct-io. Jobs-per-thread (jpt) is the multiplier for the number of tasks being submitted to the thread pool. E.g. a threadpool of 8 threads with jpt = 4 has 32 running tasks on the thread pool. It is important to oversubscribe so that when a task suspends, another one can resume on the thread pool. MultiGet (batch 16): | IO | cache hit % | sync | async_io | coro jpt=1 | coro jpt=2 | coro jpt=5 | coro jpt=10 | | -------- | --- | ---- | -------- | ---------- | ---------- | ---------- | ----------- | | direct | 10% | 65K | 64K | 145K | 151K | 161K | 166K | | direct | 50% | 88K | 80K | 239K | 287K | 327K | 340K | | direct | 90% | 435K | 399K | 732K | 991K | 1,093K | 1,119K | | buffered | 10% | 449K | 561K | 555K | 593K | 572K | 566K | | buffered | 50% | 495K | 682K | 656K | 724K | 700K | 701K | | buffered | 90% | 600K | 878K | 828K | 1,021K | 1,048K | 1,060K | single Get: | IO | cache hit % | sync | coro jpt=1 | coro jpt=2 | coro jpt=5 | coro jpt=10 | | -------- | --- | ---- | ---------- | ---------- | ---------- | ----------- | | direct | 10% | 41K | 37K | 69K | 125K | 144K | | direct | 50% | 74K | 66K | 127K | 238K | 281K | | direct | 90% | 390K | 330K | 540K | 767K | 849K | | buffered | 10% | 403K | 291K | 369K | 447K | 447K | | buffered | 50% | 441K | 337K | 443K | 554K | 527K | | buffered | 90% | 544K | 433K | 606K | 753K | 810K | Example command (MultiGet coroutine, ~50% hit): ``` db_bench -benchmarks=multireadrandomcoroutine -coro_jobs_per_thread=10 -use_existing_db=true -num=2000000 -value_size=1024 -compression_type=none -threads=8 -batch_size=16 -use_direct_reads=true -cache_size=402653184 -reads=320000 -statistics ``` ### Coroutine vs context switch overhead Coroutines are not zero overhead, so we also compare it against the context switching of using many sync threads. Both variants were pinned to the same 8 CPUs and ran 80 concurrent MultiGet operations. | variant | throughput | | ---------------------------------------- | ------------ | | sync, 80 threads | 330.6K ops/s | | coroutine, 8 executor threads / 80 jobs | 446.0K ops/s | Reviewed By: xingbowang Differential Revision: D111116635 fbshipit-source-id: bd0ac052efe79261a552e9e66137e5e70f7cd8a1	2026-07-27 15:09:34 -07:00
Peter Dillinger	0347b4df61	Add db_bench benchmarks for embedded-blob and wide-column entity reads (#14995) ... Summary: db_bench had no way to exercise SST embedded (same-file) blob reads or wide-column entity reads (GetEntity/MultiGetEntity), which are needed to measure the zero-copy embedded-blob wide-column entity read path. This adds: - Two setup benchmarks, fillembeddedentity and fillembeddedblob, that build an SST via SstFileWriter::OpenWithEmbeddedBlobs and ingest it (embedded blobs can only be produced this way, not by fillrandom/flush/compaction). Entity mode writes wide-column entities whose large columns are embedded blobs; blob mode writes whole-value embedded blobs. Both require format_version >= 7. - readrandomentity (GetEntity) and multireadentity (MultiGetEntity) read benchmarks; plain readrandom is reused for the whole-value embedded blob baseline and for entity default-column Get. - Flags embedded_blob_min_size, num_wide_columns, and num_short_wide_columns to control the embedded threshold and the inline/embedded column mix per entity. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14995 Test Plan: Manual db_bench runs of the new benchmarks, used to validate D113347622. Reviewed By: xingbowang Differential Revision: D113455538 Pulled By: pdillinger fbshipit-source-id: c6b62f5a2d6a61ae0b7cc32ac1b445b566ea8842	2026-07-23 16:01:07 -07:00
Jay Huh	990827e916	db_bench: tolerate remote IO errors by skipping ops and continuing (#14971) ... Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/14971 Follow-up to https://github.com/facebook/rocksdb/pull/14942. That PR made db_bench read ops end the run with an INVALID result (`SetFatal` -> exit 1) on a tolerable remote IO error and left write ops on unconditional `SetFatal`. In practice that still failed the remote benchmark TW jobs (exit 1) on transient WS infra errors, and did not cover write ops or `DB::Open`. Change the toleration to skip-and-continue so a transient remote IO error no longer fails the run: - `HandleBenchmarkIOError` now just logs and returns on a tolerable error (was: `SetFatal`); the caller skips the failed op and keeps going. Non-tolerable, data-loss, and local-DB errors still `abort()` (core dump for debugging). - Applied uniformly to all worker op error paths: `Get` / `MultiGet`, put / merge, delete / delete-range, `DoDelete`, `RandomWithVerify` put/delete-many, and the `BGScan` iterator (stops the scan and continues). - `DB::Open`: on a tolerable remote IO error, exit cleanly (0) -- the benchmark cannot run, but it is not a db_bench / RocksDB failure. Gated by `--tolerate_io_errors_for_remote_dbs` plus a remote `--env_uri` / `--fs_uri` backend, so local runs and non-IO / data-loss errors are unaffected. The result is best-effort (skipped ops are not counted); the goal is to keep remote benchmark jobs green through transient WS infrastructure errors instead of failing or crashing. db_stress is intentionally NOT changed: it is a correctness (crash-recovery) test and must keep failing on real errors. Differential Revision: D112749712 fbshipit-source-id: a8ccd46ef66e6c3c97a1baffd35d27830879d428	2026-07-21 08:15:35 -07:00
Jay Huh	2809ce9d0f	Tolerate remote IO errors in db_stress crash test and db_bench (#14942) ... Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/14942 Remote storage reached over `--env_uri` / `--fs_uri` (e.g. Warm Storage) can surface transient infrastructure IO errors that are unrelated to any RocksDB bug. Previously such an error would take down the test/benchmark process, making remote runs unusable. This diff lets both `db_stress` and `db_bench` tolerate non-data-loss IO errors, behind default-off flags that only take effect against a remote backend, so local behavior is unchanged and local IO errors are never masked. **db_stress:** Add a default-off flag `--tolerate_non_injected_io_errors_for_remote_dbs` that treats non-injected, non-data-loss IO errors as retryable, so remote infrastructure errors do not fail the stress harness through `SharedState::SafeTerminate()`. `IsErrorInjectedAndRetryable` enforces at the binary level that this only applies when a remote backend (`--env_uri` / `--fs_uri`) is in use; `db_crashtest.py` additionally force-disables it for local DBs. Remote crash-test jobs opt in by passing the flag on the command line (see the companion Tupperware config change). **db_bench:** Add a default-off flag `--tolerate_io_errors_for_remote_dbs`. When set and a remote backend (`--env_uri` / `--fs_uri`) is in use, a non-data-loss IO error surfaced by a benchmark operation ends the run gracefully with an INVALID result through the existing cooperative shutdown path (`SharedState::SetFatal`) instead of calling `abort()` (which previously produced a `Received signal 6 (Aborted)` process crash). All read `Get` / `MultiGet` abort sites and the `BGScan` iterator-error site are routed through a shared `HandleBenchmarkIOError` helper; IO errors on local DBs and data-loss errors are never tolerated. With the flag off (default), the original `abort()` behavior is preserved so local runs still produce a core dump for debugging. Reviewed By: anand1976 Differential Revision: D112136438 fbshipit-source-id: 510dbc644da78c4315a476ec766e26b2711dc060	2026-07-17 18:57:46 -07:00
Jay Huh	e1ea04d9ef	Add reverse order support to MultiScan (#14876) ... Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/14876 Add reverse scan support to `MultiScanArgs`/`MultiScan` so callers can iterate bounded scan ranges in descending order. Plumb reverse scan state through `DBIter`, `BlockBasedTableIterator`, and `MultiScanIndexIterator`, including reverse prefetch windowing and async I/O support. Update `db_bench` `multiscanrandom` to honor `--reverse_iterator` and report rows scanned so reverse MultiScan and iterator baselines can be compared on row throughput. Add `db_stress --multiscan_reverse` coverage so the prepared MultiScan path can be stressed with `SeekForPrev()`/`Prev()` against a control iterator. Also fix the table child iterator path to accept reverse `SeekForPrev()` targets above a file's prepared ranges. `MergingIterator` and `LevelIterator` can legally seek every child to the current range limit; a child file may only have lower prepared ranges, so treating that as invalid caused intermittent `SeekForPrev target is outside prepared ranges` exceptions in the benchmark. Reviewed By: xingbowang Differential Revision: D109357730 fbshipit-source-id: eccece5ca7d8f5d6760d51ebe76801a332a49061	2026-07-08 13:58:00 -07:00
Josh Kang	e292aa95b3	Allow skipping lmax index and filter block prefetches for external file ingestion ... Summary: Bulk external file ingestion can spend significant commit time prefetching index/filter blocks for large Lmax files. This adds `IngestExternalFileOptions::prefetch_lmax_index_and_filter_blocks` (default true) so bulk-load callers can defer that metadata prefetch when the cache doesn't need immediate warming. The option threads through `FileMetaData::skip_index_and_filter_blocks_prefetch` to drive the existing `LoadTableHandlers()` prefetch logic. Benchmarks show ~40% commit latency reduction (3459 → 2112 μs) and ~55% lower IO time (2560 → 1145 μs) when disabled. Benchmarks: ``` db_bench --benchmarks=ingestexternalfile --num=2200000 --ingest_external_file_num_batches=1 --ingest_external_file_batch_size=1 --ingest_external_file_use_file_info=true --ingest_external_file_fill_cache=true --cache_index_and_filter_blocks=true --bloom_bits=10 --statistics=true --stats_level=3 --ingest_external_file_prefetch_lmax_index_and_filter_blocks=<true|false> ``` | `ingest_external_file_prefetch_lmax_index_and_filter_blocks` | `rocksdb.ingest.external.file.run.micros` | `rocksdb.table.open.io.micros` | index/filter cache adds | last-level read bytes | | `true` | `3459 us` | `2560 us` | `1 / 1` | `3551559` | | `false` | `2112 us` | `1145 us` | `0 / 0` | `1333` | Reviewed By: xingbowang Differential Revision: D108678511 fbshipit-source-id: e8951e3aad36cc0accffb33ecc3cc1b4aeb89459	2026-06-16 11:20:07 -07:00
Peter Dillinger	123030ffee	Graceful shutdown on fatal benchmark errors (#14855) ... Summary: db_bench worker operations historically responded to a fatal error (failed Get/Put/Merge/Delete, checksum failure, etc.) by calling db_bench_exit() -> exit() in place. When that happened on a worker thread while sibling workers and background (compaction/flush) threads were still running, exit() began static/global destruction underneath those live threads. A thread mid-call into a user callback (e.g. a custom Env/ FileSystem) could then touch a global/singleton that had already been destroyed -> cross-thread use-after-free. This is the same class of bug fixed for db_stress in https://github.com/facebook/rocksdb/issues/14850, but it also matters for db_bench's secondary role as an *integration validation* tool: we want production-like processes to shut down cleanly after an error (so a custom FileSystem can log the adverse condition and run its teardown) rather than crash or _exit at the first sign of trouble. This change makes fatal errors on benchmark worker threads unwind instead of exiting in place, and performs an ordered shutdown on the main thread: * SharedState gains a cooperative fatal flag + first-error Status/message and SetFatal(). A worker that hits a fatal error records it and returns out of its operation (releasing its DbUseGuard) instead of exiting. * Op loops stop promptly via Duration: the loop-termination helper now consults the run's fatal flag at its existing throttled check boundary (~every FLAGS_ops_between_duration_checks ops), so peers wind down with no added per-key cost. * After RunBenchmark joins all workers, the (guard-free) main thread runs the ordered shutdown -- DeleteDBs() closes all DBs, joining background threads and running user Env/FileSystem teardown -- and only then exits nonzero. No static destructors run while RocksDB threads or user callbacks are still live. * The worker fatal sites that previously called db_bench_exit() (BGWriter, DoDelete, RandomWithVerify, UpdateRandom, XORUpdateRandom, MergeRandom, ReadRandomMergeRandom, VerifyChecksum, VerifyFileChecksums, RandomReplaceKeys, Replay, DoDeterministicCompact) now SetFatal + return. Flush (main-thread) routes to ErrorExit() which closes DBs first. Supporting changes: * Duration is now a nested member type of SharedState, constructible only via SharedState::MakeDuration() (movable, non-copyable). This binds every Duration to its run's fatal flag by construction and removes the previous mutable global (g_cooperative_abort_flag) and its set/clear lifecycle. A `using Duration = SharedState::Duration;` alias keeps call sites tidy. * New test-only flag --fault_injection_read_after_reads: wraps the FileSystem so random-access reads start returning a non-retryable IOError after N reads, to exercise the fatal/shutdown path on demand. * New flag --fatal_shutdown_watchdog_seconds (default 180, 0 disables): arms a one-shot detached watchdog on the first fatal error (and before ErrorExit's cleanup) that forces port::ImmediateExit() if the careful shutdown itself hangs (e.g. a background thread or misbehaving Env that never returns). This is a backstop, not the path under test. * Added an "Error handling background" comment documenting the rationale and the machinery, and an invariant comment at db_bench_exit covering pre-Open vs worker vs main-thread exit rules. Existing _Exit()/abort() sites are intentionally left unchanged. Future work: * Worker ops that still abort() on error (ReadRandom, ReadRandomFast, ReadToRowCache, MultiReadRandom, MixGraph, AppendRandom, UpdateRandom's read path, BGScan, RandomTransaction, CreateNewCf) are not the use-after-free class (abort() skips static destruction) but are not graceful either; converting them would extend clean-shutdown coverage. RandomTransaction needs a rollback decision; MultiReadRandom needs a structured break-out. * Post-Open main-thread db_bench_exit() sites in Open()/Run()/ VerifyDBFromDB() are the harder, https://github.com/facebook/rocksdb/issues/14850-class case: Open() is reachable both under and outside a DbStateMutationGuard, so it cannot unconditionally call ErrorExit() without deadlock. Needs guard-aware handling or restructuring Open() to return Status. * Pre-Open setup exits (option/flag parsing, cache setup) correctly stay plain exit() -- no live threads -- and are out of scope. * Fault injection currently covers read faults only; verifying graceful shutdown for write/open/transaction error paths would need write-fault, open-fault, and conflict injection. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14855 Test Plan: Built debug db_bench (make db_bench) and validated manually: * Correctness of loop termination (unchanged by the Duration refactor): - Count-bounded is exact: `db_bench --benchmarks=fillrandom --num=300` reports "300 operations". - Time-bounded honors duration and is not capped by max_ops: `db_bench --benchmarks=readrandom --use_existing_db --duration=5 --reads=3000` ran 5.002 s / ~2.0-2.7M operations. * Graceful shutdown on a fatal error (multi-threaded, peers live): `db_bench --benchmarks=xorupdaterandom --use_existing_db --threads=8 --duration=30 --fault_injection_read_after_reads=5000` -> exit code 1, ZERO "Received signal", prints 'Fatal error during "xorupdaterandom"' after the ordered shutdown. * Confirmed this is a real fix, not just masking: rebuilt pristine main + only the fault-injection flag (graceful-shutdown changes reverted) and ran the same command -> SIGSEGV (rc=139), crash stack in __run_exit_handlers -> jemalloc free, triggered from XORUpdateRandom -> db_bench_exit() -> exit() with sibling threads still live. With the fix in place the same workload is graceful (above). * Watchdog: - Default (180 s): graceful runs above complete with no watchdog message (no spurious firing). - --fatal_shutdown_watchdog_seconds=0 disables it (still graceful). - Positive: temporarily inserted a 60 s sleep before DeleteDBs in the fatal path, ran with --fatal_shutdown_watchdog_seconds=3 -> process exited at ~3 s (not 60 s) with the watchdog message and rc=1; reverted the temporary sleep and rebuilt clean. * No benchmark performance impact (normal, error-free runs): - Structural: the cooperative-abort check adds no per-key work. It is evaluated only at Duration's pre-existing throttled boundary (~every FLAGS_ops_between_duration_checks ops, same site as the existing time check); there is no atomic load on the per-op hot path. Fatal recording (SetFatal) lives only in cold error branches. A Duration is constructed once per benchmark via the factory (returned by value, elided), not per op. - Observed: error-free fillrandom / fillseq / readrandom throughput (e.g. fillrandom ~300K ops/s, fillseq ~800K-870K ops/s) was in line with pre-change runs; no regression observed. * make format-auto applied; debug build is clean. Reviewed By: anand1976 Differential Revision: D108690137 Pulled By: pdillinger fbshipit-source-id: 7ffdf4abc665b4c2172b718670c5c2fa1da58500	2026-06-15 19:53:56 -07:00
Josh Kang	7785264e5a	Make file opens multi-threaded for commit external sst file ingestion (#14853) ... Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/14853 `IngestExternalFileOptions` now exposes `file_opening_threads` so external file ingestion can open table readers for newly committed SSTs in parallel. The requested thread count is carried from each ingestion job into commit and then into manifest `LogAndApply`, where `VersionBuilder::LoadTableHandlers()` uses it while installing the ingested files. This is intended for workloads that ingest many external SSTs in one call, where serial table-reader opens can dominate commit latency. The ingestexternalfile benchmark now has a flag for this option, and `db_stress` now creates multiple data SSTs per ingest operation so stress runs cover multi-file ingestion, file-info ingestion, and the parallel file-opening path together. Benchmarks Ran `db_bench --benchmarks=ingestexternalfile --use_existing_db=0 --num=2200000 --compression_type=none --statistics --use_direct_reads=true --ingest_external_file_batch_size=200 --ingest_external_file_num_batches=1 --ingest_external_file_use_file_info=true --ingest_external_file_fill_cache=false` while varying `--ingest_external_file_file_opening_threads`. The generated SST files were about 256 MiB each. | `file_opening_threads` | `rocksdb.ingest.external.file.prepare.micros` | `rocksdb.ingest.external.file.run.micros` | File opens | | 1 | 16,150 us | 327,136 us | 200 | | 32 | 16,665 us | 38,349 us | 200 | At the same ~256 MiB SST size, 2,000 files is roughly 500 GiB of SST data. A straight 10x extrapolation of the 200-file ingest histograms puts prepare+run at about 3.43s with `file_opening_threads=1` and 0.55s with `file_opening_threads=32`; run alone is about 3.27s vs. 0.38s. Reviewed By: pdillinger Differential Revision: D108347952 fbshipit-source-id: 49efa48524df64cace010b881030f63f771cbb95	2026-06-15 14:45:29 -07:00
Josh Kang	ad43a5fbe8	Pass file metadata to IngestExternalFile to improve ingestion latency (#14837) ... Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/14837 External file ingestion (`DB::IngestExternalFile[s]`) re-opens every SST file and scans it -- footer, properties, index, filter, and the first/last data blocks -- to recompute the boundary keys, sequence-number bounds, and table properties before committing. For cold, I/O-bound files this scan dominates ingest latency, even when the file is moved/linked rather than copied. This change lets a caller skip that work when it already has the file's metadata. `SstFileWriter::Finish()` now returns a `PreparedFileInfo` with the file size, table properties, and prepared internal `smallest`/`largest` bounds, and a new `IngestExternalFileArg::file_infos` field carries one `PreparedFileInfo` per file into `IngestExternalFiles()`. When set, ingestion reuses that metadata instead of re-opening and scanning each file. The file is still copied/linked, and the checksum is still verified when `verify_checksums_before_ingest` is set (the fast path opens the file only for that). Point-key and range-deletion bounds are folded into the same prepared bound pair, and user-defined-timestamp files (including the "UDT in Memtables only" format, whose boundary keys carry no timestamp) are handled. Internally, ingestion-job metadata acquisition was split into `GetIngestedFileInfoFromFileInfo` (reuse caller metadata) and `GetIngestedFileInfoFromFile` (open + scan). Prepared boundary updates use RocksDB comparators, and the timestamp-stripping path pads prepared bounds back to the internal timestamp shape before installing the file. The `ingestexternalfile` `db_bench` benchmark now also exposes `--ingest_external_file_fill_cache` so ingestion-read block-cache effects can be controlled independently while comparing the file-info fast path against the normal open-and-scan path. A future PR will support creating `PreparedFileInfo` from a existing DB-generated file, so this optimization is not just limited to SstFileWriter. ## Benchmark Results Existing `db_bench ingestexternalfile` benchmark, release build, using an XFS flash-backed filesystem. Files are linked (`move_files`) so the measurement isolates the ingest path rather than file-copy throughput. These numbers used direct reads and `fill_cache=false` to avoid warming the block cache while comparing the file-info fast path against the normal open-and-scan path. Each run used 1M keys/SST, one ingest call, and `db_bench` reported a 62.9 MB estimated file size. The comparison varied the number of files in that ingest call across 10, 30, and 50 files. | Files/batch | Config | Prepare P50 | Run P50 | Run P50/file | Total ingestion P50 | Total P50 drop | | --- | --- | --- | --- | --- | --- | --- | | 10 | Baseline | 10.909 ms | 7.593 ms | 0.759 ms/file | 18.502 ms | -- | | 10 | `file_info=true` | 1.127 ms | 7.541 ms | 0.754 ms/file | 8.668 ms | 53.2% | | 30 | Baseline | 29.049 ms | 23.161 ms | 0.772 ms/file | 52.210 ms | -- | | 30 | `file_info=true` | 2.734 ms | 23.384 ms | 0.779 ms/file | 26.118 ms | 50.0% | | 50 | Baseline | 49.036 ms | 39.224 ms | 0.784 ms/file | 88.260 ms | -- | | 50 | `file_info=true` | 4.133 ms | 39.417 ms | 0.788 ms/file | 43.550 ms | 50.7% | With `fill_cache=false` and direct reads, the metadata fast path cuts `Prepare()` by roughly 90-92%. `Run()` is roughly 0.75-0.79 ms per file across these runs. End-to-end `db_bench` micros/op is almost unchanged because this benchmark includes SST generation and compaction work; use `rocksdb.ingest.external.file.prepare.micros` and `rocksdb.ingest.external.file.run.micros` to isolate ingestion. Benchmark args: `--benchmarks=ingestexternalfile --use_existing_db=0 --num=1000000 --compression_type=none --statistics --use_direct_reads=true --ingest_external_file_batch_size=<10|30|50> --ingest_external_file_num_batches=1 --ingest_external_file_use_file_info=<false|true> --ingest_external_file_fill_cache=false`. Reviewed By: xingbowang Differential Revision: D107721261 fbshipit-source-id: b06ecb7b35f260ec02acf837e7986057bc23cdbf	2026-06-15 11:14:42 -07:00
Peter Dillinger	6d4a8144e0	Unify LZ4 and LZ4HC compression levels (#14819) ... Summary: kLZ4Compression and kLZ4HCCompression share the same on-disk format and decompressor, but historically kLZ4Compression only honored negative (acceleration) levels while kLZ4HCCompression only honored positive levels. This unifies them so `compression_opts.level` alone selects the variant: level <= 0 uses LZ4 fast (acceleration = -level) and level >= 1 uses LZ4HC (1..12), regardless of which of the two types is configured. The configured type now only determines the default compression level (LZ4: acceleration 1, equivalent to level -1; LZ4HC: level 9). For code simplicity, the recorded per-block type comes from the compression type derived from the level, which could differ from the configured type. To preserve the originally configured choice for debugging/tracking, it is recorded as a `_type=<decimal>` pseudo-option in the rocksdb.compression_options SST table property. Out-of-range non-default levels are clamped to the nearest effective value (LZ4 acceleration capped at 65537, which also avoids signed overflow negating INT_MIN; LZ4HC level capped at 12). The cost-aware (auto-tune) compressor's LZ4 level grid is changed to negative accelerations so it actually exercises fast LZ4 (positive levels now route to LZ4HC). Related inclusion: The ZSTD library has a discontinuity at level=0, which maps to level 3, which is more aggressive than levels 1 and 2, which are more aggressive than levels -1, -2, etc. For better friendliness to auto-tuning (etc.), we now map level 0 to be the same as level -1, so that increasing compression level numbers have non-decreasing aggressiveness. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14819 Test Plan: New unit tests in compression_test.cc: - UnifiedLZ4LZ4HCLevels: for a representative set of non-default levels, both configured types produce identical output and the same recorded type (selected by the level), levels that clamp to the same effective parameter compress identically, and each round-trips; plus per-type default-level behavior. - ConfiguredCompressionTypeRecordedInProperties: the `_type=` pseudo- option appears in the SST table property for each configured type. - ZSTDLevelZeroMapsToMinusOne: level 0 behaves like -1, not like 3. Reviewed By: joshkang97 Differential Revision: D107536580 Pulled By: pdillinger fbshipit-source-id: a1281956f70a75d0620cb73d0bfb9ad76c52cca3	2026-06-11 14:10:21 -07:00
Peter Dillinger	01084e7e8e	Disable parallel compression for fast built-in compressors (#14841) ... Summary: Use Compressor::GetRecommendedParallelThreads() overrides to disable parallel compression for Snappy, LZ4 (accelerated, not LZ4HC), and for accelerated levels of ZSTD (level < 0). These do not generally benefit from parallel compression. Also add --verify_compression option to sst_dump for some basic "recompress" benchmarking with that option. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14841 Test Plan: unit test updated. In planning the scope of this change, I manually tested some production SST files with release build sst_dump --command=recompress and various settings for --compression_parallel_threads, --compression_types, and --compression_level, while I had the CPUs mostly busy using cache_bench in the background. Here's an example, before the change to override parallel_threads: ``` $ for PT in 1 8; do /usr/bin/time ./sst_dump --command=recompress --compression_parallel_threads=$PT --block_size=16384 --compression_types=kLZ4Compression --verify_compression=1 test.sst; done ... Compression: kLZ4Compression Block Size: 16384 Threads: 1 Cx level: 32767 Cx size: 168501634 Uncx size: 791721664 Ratio: 4.698599 Write usec: 2345894 Read usec: 429022 Cx count: 48661 (100.0%) Not cx for ratio: 0 ( 0.0%) Not cx otherwise: 0 ( 0.0%) 2.54user 0.29system 0:02.84elapsed 99%CPU (0avgtext+0avgdata 460816maxresident)k ... Compression: kLZ4Compression Block Size: 16384 Threads: 8 Cx level: 32767 Cx size: 168501634 Uncx size: 791721664 Ratio: 4.698599 Write usec: 2476459 Read usec: 439464 Cx count: 48661 (100.0%) Not cx for ratio: 0 ( 0.0%) Not cx otherwise: 0 ( 0.0%) 3.95user 0.33system 0:02.98elapsed 143%CPU (0avgtext+0avgdata 455504maxresident)k ``` Here as in many of the cases I'm changing, it actually takes longer to compress with parallel, despite the added parallel opportunity of verify_compression. And overall CPU is much higher from 2.54 `CPU*s` to 3.95 `CPU*s`. The difference disappears with the change, because both use single-threaded SST construction. Reviewed By: joshkang97 Differential Revision: D108075938 Pulled By: pdillinger fbshipit-source-id: 5d1fc77ddbccf9f3b24a4f0b20b2b3c43074e89d	2026-06-10 12:58:47 -07:00
zaidoon	7affaee1c4	Add use_direct_io_for_compaction_reads option (#14743) ... Summary: Adds a new `DBOption use_direct_io_for_compaction_reads` (default false). When on, compaction-input SST files are opened with `O_DIRECT` so the sequential read-once data from compaction doesn't pollute the OS page cache and evict the hot user-read working set. User reads keep going through the buffered fast path. This protects user-read tail latency on write-heavy workloads without forcing user reads onto the existing global `use_direct_reads` knob (which pays in throughput and P50 — see the bench below). The interesting bit is that just flipping the FileOptions returned by `FileSystem::OptimizeForCompactionTableRead` doesn't actually trigger `O_DIRECT` at the kernel level. The TableCache (and `FileMetaData::pinned_reader`) is already holding buffered handles opened at flush time or at `DB::Open` via `LoadTableHandlers`. When compaction asks for an iterator, it gets back the cached buffered handle and the kernel never sees the `O_DIRECT` flag. So this PR also adds a small bypass path: - `TableCache::FindTable` / `NewIterator` learn a `open_ephemeral_table_reader` mode. When set, the pinned-reader fast path and the shared cache are skipped, `GetTableReader` is called directly with the caller's FileOptions, and ownership of the freshly opened TableReader is handed back via a `unique_ptr`. The iterator takes ownership via `RegisterCleanup` and frees the reader on destruction. - `VersionSet::MakeInputIterator` and `LevelIterator` plumb the flag through both L0 and L1+ compaction-input paths. - `CompactionJob::ProcessKeyValueCompaction` turns the bypass on when `use_direct_io_for_compaction_reads` is set, the global `use_direct_reads` is off, and `OptimizeForCompactionTableRead` produced `use_direct_reads=true` in the compaction-read FileOptions. The option is opt-in: when off, nothing changes for existing users. When on, only the compaction-input opens take the bypass path; user reads keep hitting the TableCache and the buffered fast path normally. There's also a small db_bench helper in the same PR: a new `--bgwriter_num` flag that lets the writer thread in `readwhilewriting` (and the other "while writing" variants) spread its puts across `[0, bgwriter_num)` instead of `[0, num)`. Without this the readers and writer share a key range and you can't have both a hot read subset and meaningful compaction work — this lets you have both. ### Benchmark Setup: Ubuntu 24.04 (kernel 7.0.5, OrbStack Linux VM on Apple Silicon), 14 vCPUs, virtio-blk disk, btrfs. MGLRU disabled (`echo 0 > /sys/kernel/mm/lru_gen/enabled`) so the kernel uses the classic active/inactive LRU. 14 GB DB (3.5M keys × 4 KB values), no compression. Each measurement run is pinned to a 1 GB cgroup via `systemd-run --scope -p MemoryMax=1G -p MemorySwapMax=0`. Page cache is dropped between configs. db_bench is Release build. Workload: `readwhilewriting` for 120s. 4 reader threads doing random reads over a hot key subset, plus 1 writer thread spreading overwrites across the full 3.5M-key keyspace (via `--bgwriter_num=3500000`) throttled at 200 MB/s, so there's continuous compaction running while the readers go. The size of the hot reader subset relative to available page cache controls how visible the optimization is. The Cassandra blog ([Lightfoot 2026](https://lightfoot.dev/direct-i-o-for-cassandra-compaction-cutting-p99-read-latency-by-5x/)) documented the same thing: biggest wins when the hot set is big enough to actually compete for cache, smaller wins when the hot set trivially fits, neutral when the hot set is way bigger than cache. So I ran two hot-set sizes. #### Small hot set: ~30 MB (~3% of the 1 GB cgroup) — N=5 iterations, mean (CV) `--num=7500`. The hot set is small enough that the page cache holds it without much trouble even under compaction, so the wins here are real but on the modest side. | Config | Throughput (ops/s) | Read P50 (µs) | Read P99 (µs) | Read P99.9 (µs) | Read P99.99 (µs) | |---|---|---|---|---|---| | buffered (default) | 233,477 (8.2%) | 16.09 | 82.24 | 721.0 | 2,102.5 | | direct_compaction_writes_only (existing knob alone) | 287,405 (2.8%) — **+23.1%** | 13.00 (−19.2%) | **66.77 (−18.8%)** | 553.9 (−23.2%) | 1,787.6 (−15.0%) | | direct_compaction_read_only (new knob alone) | 250,669 (2.4%) — +7.4% | 14.16 (−12.0%) | 102.99 (+25.2%) | 689.8 (−4.3%) | 1,801.3 (−14.3%) | | direct_compaction_read_write (new + existing, recommended) | 277,920 (3.3%) — **+19.0%** | **12.99 (−19.3%)** | 84.23 (+2.4%) | 613.4 (−14.9%) | **1,738.2 (−17.3%)** | | use_direct_reads=true (existing global) + write-side | 249,014 (2.5%) — +6.7% | 15.95 (−0.9%) | 68.78 (−16.4%) | **450.8 (−37.5%)** | 1,814.5 (−13.7%) | CV is 2.4–3.3% on the optimized configs (8.2% on buffered), so the deltas are real. With a hot set this small, the existing `use_direct_io_for_flush_and_compaction` knob is already doing most of the work — the new flag's main extra contribution here is P99.99 (combined wins it by ~2 points vs writes-only-alone). Worth noting: the new flag *alone* (without the existing write-side flag) improves P99.99 but regresses P99 by 25% on this small-hot-set workload, because direct compaction reads lose kernel readahead and compaction-output writes are still hitting the page cache. That regression goes away once you combine with the existing write-side flag, or once the hot set is bigger (see next table). So if you're using just one knob, use the existing one. If you're using this PR's flag, pair it with `use_direct_io_for_flush_and_compaction=true`. #### Larger hot set: ~400 MB (~40% of cache) — N=5 iterations, mean (CV) `--num=100000`. This is the case the Cassandra blog calls out — hot set big enough to actually fight compaction for cache. Their analogous setup (1M hot partitions, ~33% hot/cache) reported 1.93× p99 improvement. Numbers here are the headline: | Config | Throughput (ops/s) | Read P50 (µs) | Read P99 (µs) | Read P99.9 (µs) | Read P99.99 (µs) | |---|---|---|---|---|---| | buffered (default) | 68,959 (7.7%) | 44.81 | 541.22 | 2,225.2 | 11,334.5 | | direct_compaction_writes_only (existing knob alone) | 73,973 (10.3%) — +7.3% | 42.22 (−5.8%) | 456.27 (−15.7%) | 2,016.9 (−9.4%) | 9,190.0 (−18.9%) | | direct_compaction_read_only (new knob alone) | 84,337 (2.3%) — +22.3% | 38.66 (−13.7%) | 386.97 (−28.5%) | 1,644.8 (−26.1%) | 4,837.9 (−57.3%, 2.34×) | | direct_compaction_read_write (new + existing, recommended) | **104,923 (8.4%) — +52.2%** | **34.26 (−23.5%)** | **290.97 (−46.2%)** | **1,143.4 (−48.6%)** | **3,080.3 (−72.8%, 3.68×)** | | use_direct_reads=true (existing global) + write-side | 71,598 (9.1%) — +3.8% | 51.33 (+14.5%) | 297.91 (−45.0%) | 1,663.6 (−25.2%) | 6,530.0 (−42.4%) | Combined config gets a 3.68× p99.99 win, 1.86× p99, p50 down 23%, throughput up 52%. Same shape as the Cassandra blog's 1.93× p99 result — the improvement just lands at deeper percentiles for us because RocksDB's baseline data path is roughly 40× faster than Cassandra's (their buffered p99 was 35 ms, ours is 0.54 ms), so the cache-miss tail is further out. A few things worth calling out from this table: - The new flag is doing real work on top of the existing write-side flag here, not just shifting things around. Combined throughput is +42% over `direct_compaction_writes_only` alone, and combined p99.99 is 3× better. The existing knob alone gives a fairly modest +7% throughput / -19% p99.99 in this case — there's a clear gap that the new flag fills. - The new flag *alone* (no existing write-side flag) is also a real improvement here: +22% throughput, p99.99 down 57%. The P99 regression we saw in the small-hot-set case is gone, because the cache-protection effect now dominates the lost-readahead cost. - `use_direct_reads=true` (the existing global flag) actually regresses P50 by 14.5% in this workload — taking user reads off the page cache hurts you when the hot data could have been cached. It also gets the worst throughput of any direct config. It's not an equivalent way to get these gains. ### `compaction_readahead_size` matters when this flag is on Direct I/O bypasses kernel readahead, so RocksDB's own `DBOptions::compaction_readahead_size` becomes the only prefetch the iterator has. The default of 2 MB is enough and real users will get it automatically. **But `db_bench`'s `--compaction_readahead_size` CLI default is 0**, which defeats prefetch and makes direct compaction look slower than it actually is. If you're reproducing the numbers above, pass `--compaction_readahead_size=2097152` (or larger). - Recommended production config is `use_direct_io_for_compaction_reads=true` + `use_direct_io_for_flush_and_compaction=true`. Strongest configuration at every percentile and throughput in both benches. - The new flag is the read-side counterpart to `use_direct_io_for_flush_and_compaction`, which handles compaction-write cache pollution. They address different sources of pollution and compose. The gap between "combined" and "writes-only-alone" is 17 percentage points on p99.99 in the small-hot-set bench and 54 points in the larger one, so the new flag is contributing real value, especially as the hot set grows. - The new flag alone is also a real improvement when the hot set is big enough to compete with cache (+22% throughput, 2.34× p99.99 in the larger-hot-set bench). On a very small hot set it improves p99.99 but regresses p99, so pairing with the existing write-side flag is safer. - The benefit is workload-dependent. Small hot sets get modest tail-latency wins. Hot sets sized to actually compete for cache get the big multi-percentile wins shown above. Hot sets bigger than cache (not benched here but covered in the Cassandra blog) see no change either way — every read misses regardless. ### Reproducing Any Linux host (or a Linux VM on macOS via OrbStack / Multipass / lima): ```bash sudo apt-get install -y build-essential clang cmake git pkg-config \ libgflags-dev libsnappy-dev zlib1g-dev libbz2-dev liblz4-dev libzstd-dev cmake -DCMAKE_BUILD_TYPE=Release -DPORTABLE=1 -DWITH_GFLAGS=1 -DWITH_TESTS=0 .. make -j db_bench echo 0 | sudo tee /sys/kernel/mm/lru_gen/enabled ``` Build the source DB once, unrestricted memory: ```bash ./db_bench --benchmarks=fillrandom,compact,waitforcompaction,stats \ --db=/path/to/source_db --num=3500000 --key_size=16 --value_size=4096 \ --write_buffer_size=16777216 --target_file_size_base=16777216 \ --max_background_jobs=4 --compression_type=none --cache_size=4194304 \ --max_bytes_for_level_base=67108864 --disable_wal=1 --sync=0 ``` For each config, copy `source_db -> scratch_db`, run `sync && echo 3 > /proc/sys/vm/drop_caches`, then: ```bash sudo systemd-run --scope -p MemoryMax=1G -p MemorySwapMax=0 \ ./db_bench --use_existing_db=1 \ --benchmarks=readwhilewriting,stats --db=/path/to/scratch_db \ --threads=5 --duration=120 --statistics=true --histogram=1 \ --num=7500 --bgwriter_num=3500000 \ --key_size=16 --value_size=4096 \ --write_buffer_size=16777216 --target_file_size_base=16777216 \ --max_background_jobs=4 --compression_type=none \ --cache_size=4194304 --open_files=200 \ --skip_stats_update_on_db_open=true \ --max_bytes_for_level_base=67108864 \ --benchmark_write_rate_limit=209715200 \ --compaction_readahead_size=2097152 \ --rate_limiter_bytes_per_sec=0 \ --use_direct_reads={true|false} \ --use_direct_io_for_compaction_reads={true|false} \ --use_direct_io_for_flush_and_compaction={true|false} ``` For the larger hot-set table, change `--num=7500` to `--num=100000`. The five configs in the tables: - `buffered`: all three flags false. - `direct_compaction_writes_only`: `use_direct_io_for_flush_and_compaction=true`, the other two false. This is what users have today without this PR. - `direct_compaction_read_only`: `use_direct_io_for_compaction_reads=true`, the other two false. - `direct_compaction_read_write`: `use_direct_io_for_compaction_reads=true`, `use_direct_io_for_flush_and_compaction=true`, `use_direct_reads=false`. **Recommended.** - `direct_all`: `use_direct_reads=true`, `use_direct_io_for_flush_and_compaction=true`, `use_direct_io_for_compaction_reads=false`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14743 Reviewed By: pdillinger Differential Revision: D108017601 Pulled By: xingbowang fbshipit-source-id: 4039d490d7e77b476db7a477a2f3d24738db6336	2026-06-09 17:02:53 -07:00
Xingbo Wang	76b8cce2f7	Fix spelling of memtable_verify_per_key_checksum_on_seek (#14811) ... Summary: - Rename the misspelled `memtable_veirfy_per_key_checksum_on_seek` option and related flags/config keys to `memtable_verify_per_key_checksum_on_seek`. - Update memtable option plumbing, options serialization/logging, db_bench/db_stress/crash-test flags, tests, and the option-addition guide to use the corrected name. - Keep the checksum-on-seek behavior unchanged. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14811 Test Plan: CI Reviewed By: pdillinger Differential Revision: D107382276 Pulled By: xingbowang fbshipit-source-id: 7621dc718d61503b982a7e3f65cc9293a1ad085b	2026-06-05 10:18:47 -07:00
Peter Dillinger	8053b9414f	Change default compression from Snappy to LZ4 (#14818) ... Summary: The historical default block compression `kSnappyCompression` dates to when Snappy was the obvious fast/cheap choice. On modern server CPUs LZ4 matches or beats Snappy on compression ratio while decompressing substantially cheaper, so it is a better default. This changes the default `ColumnFamilyOptions::compression` to `kLZ4Compression`, with a runtime fallback of LZ4 -> Snappy -> none depending on what is compiled into the binary (new `GetDefaultCompressionType()` in util/compression.h). Only column families that do not explicitly set `compression` are affected (including compaction output when `CompactionOptions::compression == kDisableCompressionOption`), and only newly written SST files; existing data is read as before. Doc comments, the Java bindings, and the sorted_run_builder example are updated to describe the new default. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14818 Test Plan: Adjusted two unit tests that implicitly depended on the old Snappy default to pin `compression = kSnappyCompression`: db_iterator_test's ReadAhead (readahead byte thresholds assume Snappy-sized files) and compaction_service_test's CustomFileChecksum (LSM shape after auto-compaction determined whether a manual CompactRange had work to do). This change is primarily validated by performance testing. Added a `compressreject` db_bench benchmark (output buffer sized just below the predicted compressed size) to measure the cost of attempting then declining compression, alongside `compress`/`uncompress`. Both db_bench and sst_dump compress/decompress a modest block at a time, as in a real workload. sst_dump on SST files from production workloads (4 files, 16KB blocks, single thread) on recent server-class AMD, Intel, and ARM CPUs, LZ4 vs Snappy: - Compression ratio: comparable; LZ4 is slightly smaller on the more compressible files (up to ~8%) and within ~2% on the rest. - Compression (write) CPU: a wash, within ~2% either direction. - Decompression (read) CPU: the clear win -- Snappy costs ~1.2x-1.5x as much as LZ4, i.e. LZ4 saves ~25-30% read CPU, consistently across AMD, Intel, and ARM. db_bench synthetic workload (100-byte values), at 1 / 12 / 160 threads, LZ4 vs Snappy: - Compression throughput: LZ4 ~10-30% higher. - Decompression throughput: LZ4 much higher, and the advantage grows with core count -- from ~+12% at 12 threads to ~+45-50% at 160 threads, i.e. better multi-core scaling. - Rejection (insufficient ratio) path: comparable; LZ4 ~15-18% faster on compressible-but-rejected blocks and Snappy within ~10% on barely-compressible blocks. No meaningful regression, confirming incompressible data is still efficiently detected and stored raw. Reviewed By: xingbowang Differential Revision: D107536490 Pulled By: pdillinger fbshipit-source-id: f8abaee630d782674778338148e36ed0c84e3661	2026-06-04 10:48:27 -07:00
Maciej Szeszko	d1ab4bf12c	Guard db_bench DB lifetime with RWMutex against shutdown race (#14754) ... Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/14754 **Problem** When db_bench shuts down (fatal IO error, unknown benchmark name, etc.), ErrorExit() or ~Benchmark() destroys db_/multi_dbs_ while worker threads may still be inside benchmark methods holding raw DBWithColumnFamilies pointers returned by SelectDBWithCfh(). In single-DB mode, db_.db becomes nullptr and multi_dbs_ is empty, so workers evaluate rand_int % 0 and SIGFPE. The cascade across all workers masks whatever originally triggered the shutdown. Observed in the reliable_volumes crash-fault test at ~5 SIGFPE events per hour. **Approach** Guard DB lifetime at the worker-thread boundary with a reader-writer lock (port::RWMutex): workers acquire a read lock for the duration of their benchmark method; mutators acquire the write lock (which waits for active readers to drain) before destroying DBs. This ensures raw pointers returned by SelectDBWithCfh() remain valid for their entire usage scope. **Fix** - Workers hold a DbUseGuard (RAII wrapper over a read lock on db_lifecycle_rwlock_) for the entire benchmark method in ThreadBody. Cost: one rdlock/unlock pair per worker lifetime, zero hot-path overhead. - Every DB mutation path (ErrorExit, ~Benchmark, fresh-DB reopen) takes DbStateMutationGuard, which acquires the write lock and then stops the secondary update thread before mutation may proceed. - ErrorExit() routes to std::_Exit(1) when called from any thread that cannot safely run the mutation cleanup path: a DbUseGuard holder (would self-wait on the write lock) or the secondary update thread (would self-join via StopSecondaryUpdateThread). Tracked via two thread-locals: holds_db_use_guard_ and is_secondary_update_thread_. Main thread takes DbStateMutationGuard, cleans up, dispatches through db_bench_exit() / ToolHooks::Exit. - StopSecondaryUpdateThread() resets secondary_update_stopped_ to 0 after joining, so a replacement thread created by a subsequent Open() is not immediately killed. - Protocol is mechanically enforced in debug builds: * SelectDBWithCfh asserts holds_db_use_guard_ (read-side ownership) * DeleteDBs asserts holds_db_state_mutation_guard_ (write-side ownership) * DbUseGuard and DbStateMutationGuard ctor/dtor assert correct imbalance (no nested or stray release). * DbStateMutationGuard ctor additionally asserts !holds_db_use_guard_ and !is_secondary_update_thread_, catching the new deadlock modes the single-lock design makes reachable (WriteLock while holding ReadLock; secondary thread self-joining via StopSecondaryUpdateThread). These convert "trust me" invariants into "the assert fires if you break it." Direct field accesses (e.g. db_.db->NewIterator inside a benchmark method, multi_dbs_.clear() in the reopen branch) are protected by the enclosing guard scope but are not per-site asserted. **Caveat** port::RWMutex is pthread_rwlock_t with default attrs -> reader-preferred on Linux/glibc. The current call graph has no concurrent worker spawn during mutator wait (mutator paths run only from the main thread, not concurrently with RunBenchmark spawning workers), so writer starvation is not reachable. Documented as a constraint; revisit if that invariant changes. The port layer doesn't expose pthread_rwlockattr_setkind_np cross-platform. **Alternatives considered** - std::_Exit(1) on every shutdown path, skip cleanup entirely. Loses ToolHooks::Exit dispatch, flushed traces, and end-of-run stats -- those matter for the RV crash-fault test image which consumes db_bench output. Rejected. - shared_ptr<DBWithColumnFamilies> from SelectDBWithCfh, let DB lifetime extend naturally to the last reader. Adds a per-call atomic refcount bump in the hot path; the RWMutex approach is per-method-call instead of per-op, making the hot-path cost zero. Rejected for hot-path neutrality. Reviewed By: xingbowang Differential Revision: D104974784 fbshipit-source-id: 3a04d9e1c0b5042436d690f573cf369de6b4c9df	2026-05-21 11:49:59 -07:00
Xingbo Wang	21723bbbef	Add async WAL precreation ... Summary: - Add experimental immutable `DBOptions::async_wal_precreate` to reserve and open one future WAL on a background HIGH-priority task, with sanitization that disables the optimization when WAL recycling is configured. - Split WAL creation into open/preallocate and start phases so `SwitchMemtable()` can consume a prepared WAL after writing normal WAL metadata, wait for in-flight precreation, fall back to synchronous creation, and delete an unstarted prepared WAL on start failure. - Keep WAL numbering, close, recovery, and read-only open safe for empty future WAL files left by async precreation; `error_if_wal_file_exists=true` now rejects non-empty WALs while tolerating empty WALs. - Add public option plumbing for the C API, options parsing/stringification, random option testing, `db_bench`, `db_stress`, and crash-test configuration. - Add WAL precreate statistics counters plus Java `TickerType`/JNI mappings, and update C++, C, and Java read-only-open documentation for the empty-WAL behavior. - Add focused WAL/option/C/Java tests for async precreate ready/wait/failure/recovery paths, read-only WAL detection, option sanitization, and API plumbing, plus write-flow docs and unreleased history entries for the new feature and behavior change. PR https://github.com/facebook/rocksdb/pull/14738 Reviewed By: pdillinger Differential Revision: D105020559 fbshipit-source-id: 5059b424702e021abb8de65ceeb6d3b975280ffc	2026-05-15 10:59:47 -07:00
Anand Ananthabhotla	06699cd0fb	Add IODispatcher memory limiting and multiscanrandom benchmark to db_bench (#14570) ... Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/14570 Add support for benchmarking MultiScan with IODispatcher memory limiting in db_bench. This includes: 1. **New flags for existing `multiscan` benchmark:** - `--io_dispatcher_max_prefetch_memory_bytes`: Sets the global memory budget for IODispatcher prefetching across all ReadSets. When this limit is reached, `SubmitJob()` blocks until memory is released. 0 means unlimited (no IODispatcher created). - `--multiscan_max_prefetch_size`: Sets the per-file prefetch size limit (`MultiScanArgs::max_prefetch_size`). 2. **New `multiscanrandom` benchmark with randomized workload:** - Random batch sizes (1-64 ranges per MultiScan call) - Random range sizes (up to ~1MB worth of keys per range) - Generates sorted, non-overlapping ranges each iteration - Supports all IODispatcher flags above - `--use_multiscan` flag: when true (default), uses MultiScan API; when false, uses normal iterators (Seek + Next) for A/B comparison ## Benchmark Results Setup: 5M keys, key=16B, value=256B, 5-level LSM (267 SSTs), direct reads, 8MB block cache, 30s duration. ### Normal Iterators vs MultiScan (`multiscanrandom`) ``` | Mode | ops/sec | |-----------------------------------------|---------| | Normal iterators (use_multiscan=false) | 6 | | MultiScan sync (async=off) | 15 | | MultiScan async (async=on) | 25 | ``` MultiScan sync is **2.5x** faster than normal iterators. MultiScan async is **4.2x** faster. ### MultiScan async=off with Memory Limits (P=4MB peak concurrent memory) ``` | Config | Limit | ops/sec | blocked | |-----------------|--------|---------|---------| | Baseline (10GB) | unlim | 15 | 0 | | Fits budget | 4MB | 15 | 246 | | Exceeds 10% | 3.6MB | 14 | 698 | | Exceeds 50% | 2.67MB | 14 | 1,314 | | Exceeds 100% | 2MB | 15 | 10,724 | ``` Sync IO is disk-bound so memory limiting has negligible throughput impact, even at 2x oversubscription. ### MultiScan async=on with Memory Limits (P=35MB peak concurrent memory) ``` | Config | Limit | ops/sec | blocked | |-----------------|--------|---------|---------| | Baseline (10GB) | unlim | 25 | 0 | | Fits budget | 35MB | 24 | 204,525 | | Exceeds 10% | 31.8MB | 19 | 361,444 | | Exceeds 50% | 23.3MB | 14 | 289,495 | | Exceeds 100% | 17.5MB | 19 | 464,878 | ``` Async IO uses ~9x more peak concurrent memory (35MB vs 4MB) due to multiple in-flight reads. At ~50% oversubscription, async throughput degrades to sync-level, negating the async benefit. Reviewed By: xingbowang Differential Revision: D99489650 fbshipit-source-id: e88d3a980f592e4e86628604e6389a1f2f71cfef	2026-04-06 11:59:33 -07:00
Josh Kang	3f51c0a185	Convert sequential single deletes into range tombstones (#14448) ... Summary: Add a read-path optimization that converts contiguous point tombstones into range tombstones during forward/reverse iteration. When a configurable threshold of consecutive point deletions (kTypeDeletion, kTypeDeletionWithTimestamp, kTypeSingleDeletion — with no live keys between them) is detected, a range tombstone covering `[first_tombstone_key, next_live_key)` is inserted into the active mutable memtable. This benefits future iterators by enabling efficient skipping via range tombstone fragmentation. If there is a memtable switch during the read iteration, then the range deletion entry is discarded. The inserted range tombstones are logically redundant (they don't delete anything that isn't already deleted by point tombstones), skip WAL (they're a derived optimization regenerated by future reads on crash), and use the max tombstone sequence number so they don't interfere with newer writes. ## Key changes - **New option `min_tombstones_for_range_conversion`** (`AdvancedColumnFamilyOptions`): Threshold of contiguous point tombstones before converting to a range tombstone. Default 0 (disabled). Dynamically changeable via `SetOptions()`. - **`DBIter` tracking logic** (`db/db_iter.cc`): Tracks contiguous tombstones during `FindNextUserEntryInternal()` (forward) and `PrevInternal()` (reverse). When a live key terminates a run that meets the threshold, `MaybeInsertRangeTombstone()` inserts `[first_tombstone, live_key)` into the active memtable. - **`FindValueForCurrentKey` `found_visible` output** (`db/db_iter.cc`): Distinguishes "key deleted at this snapshot" from "no visible entries" so reverse tracking doesn't treat post-snapshot keys as tombstones. - **`IterKey::Swap()`** (`db/dbformat.h`): Efficiently tracks reverse tombstone run end keys without extra allocations. - **`MemTable::AddLogicallyRedundantRangeTombstone()`** (`db/memtable.cc`): Concurrent-safe range tombstone insertion into the active memtable. Range tombstone skiplist always uses concurrent inserts. - **`ConstructFragmentedRangeTombstones` race fix** (`db/db_impl/db_impl_write.cc`): Moved after `MarkImmutable()` to prevent lost entries. - **`MarkImmutable` ordering fix** (`db/memtable_list.cc`): Called before `current_->Add()` to close a race window. - **Prefix filter awareness** (`db/db_iter.cc`): Tombstone tracking scoped to the seek prefix when prefix filtering is active. See dedicated section below. - **Transaction awareness** (`db/db_iter.cc`): Tombstones with `seq > snapshot` excluded from tracking. `min_uncommitted` guard uses `insert_seq` (which may be bumped to `earliest_seq`) instead of `range_tomb_max_seq_`. See dedicated section below. - **Duplicate range check**: Skips insertion if the memtable already covers `[start, end)`. - **New statistics**: `READ_PATH_RANGE_TOMBSTONES_INSERTED` and `READ_PATH_RANGE_TOMBSTONES_DISCARDED`. - **Memtable MultiGet batch lookup** (`memtable/inlineskiplist.h`, `db/memtable.cc`): `InlineSkipList::MultiGet()` with cached search path ("finger") for sorted key lookups. - **New option `memtable_batch_lookup_optimization`** (`AdvancedColumnFamilyOptions`): Enables batch lookup for memtable MultiGet. Default false. Immutable. ## Deciding Range Tombstone Seqno - The range tombstone is inserted with `insert_seq = max(range_tomb_max_seq_, earliest_seq)` where `range_tomb_max_seq_` is the maximum sequence number across all point tombstones in the contiguous run, and `earliest_seq` is the memtable's earliest sequence number. This preserves the memtable's `earliest_seqno_` invariant. - If the iterator's snapshot sequence (`sequence_`) predates the memtable's `earliest_seq`, insertion is skipped entirely to avoid unintentionally covering entries between `sequence_` and `earliest_seq`. ## ConstructFragmentedRangeTombstones Race Fix - `MarkImmutable()` and `ConstructFragmentedRangeTombstones()` are now called before `mutex_.Lock()` in `SwitchMemtable`, keeping this work outside the DB mutex. `MarkImmutable()` blocks concurrent `AddLogicallyRedundantRangeTombstone()` calls via `immutable_mutex_`, ensuring no range tombstones are inserted after the fragmented list is built. `MarkImmutable()` is idempotent, so `MemTableList::Add()` calling it again inside the mutex is harmless. ## Prefix Filter Safety - When prefix filtering is active, the BBTI bloom filter may reject SST files outside the seek prefix, but the memtable (no bloom filter) returns keys across prefix boundaries. Tombstone tracking is scoped to the seek prefix so that converted range tombstones cannot cover live keys hidden in filtered files. - `total_order_seek=true` disables prefix filtering — all files are visible, so tombstones safely span prefix boundaries. - **Behavior change**: Seeking to an out-of-domain key with `total_order_seek=false` now treats it as total-order (prefix_ not set). When `prefix_same_as_start=true`, iterating past an out-of-domain key cleanly invalidates the iterator instead of calling `Transform()` on it (which was UB in release builds with `FixedPrefixTransform`). This is a requirement because an incorrect iterator scan could lead to a range tombstone covering a live key. ## Transaction Support - Tombstones written by the transaction's own uncommitted writes (sequence > snapshot) are now excluded from contiguous tombstone tracking entirely at the tracking site in `FindNextUserEntryInternal()` and `PrevInternal()`. Previously, tracking relied on a `min_uncommitted` check at insertion time, but this was insufficient — a transaction's own Delete with `seq > snapshot` could extend a run of committed tombstones, and the resulting range tombstone would cover data visible to other snapshots. - The fix skips any tombstone with `ikey_.sequence > sequence_` during tracking. If a transaction-owned tombstone appears mid-run, it flushes the accumulated committed run first, then resets tracking. This ensures only tombstones visible to the current snapshot are ever converted. - Both WritePrepared and WriteUnprepared transactions are supported with dedicated test coverage: - **WritePrepared**: When tombstones are committed before `Prepare()`, their seqnos are below `min_uncommitted` and insertion proceeds safely. When `Prepare()` happens first, tombstone seqnos exceed `min_uncommitted` and insertion is blocked. - **WriteUnprepared**: Multiple unprepared batches with different seqno ranges are handled correctly. Own transaction Deletes that extend a committed tombstone run block insertion of the entire run. After rollback, data correctness is verified. ## UDT support - When user-defined timestamps (UDT) are enabled, keys include an 8-byte timestamp suffix. The comparator, Put/Delete APIs, and ReadOptions all require timestamps. - Forward exhaustion with UDT: `iterate_upper_bound_` is a plain user key without a timestamp suffix. It is padded with min timestamp via `AppendKeyWithMinTimestamp()` so it sorts after all entries with this user key, preserving the exclusive bound semantics. - Reverse exhaustion with UDT: The end key comes from either the previous live key (which already has a proper timestamp suffix) or the seek target set by `SetSavedKeyToSeekForPrevTarget()` (which appends a timestamp via `SetInternalKey(..., timestamp_ub_)` and `UpdateInternalKey(..., ts)`). In both cases, the end key is already properly timestamped, so no additional padding is needed unlike forward exhaustion. - Contiguous tombstone detection works correctly with UDT because the underlying `kTypeDeletionWithTimestamp` entries are tracked the same way as `kTypeDeletion`. ## Concurrent Iterators - Should concurrent iterators happen to read the range at the same time, both will produce the same range and seqno entry. Only one will be accepted by the skip list and the others will be rejected. Future iterators will read the range and not even attempt to insert the range. - There is nothing preventing similar ranges from being inserted however. Two iterators can produce overlapping ranges, but this protection would be complicated to implement and there is no evidence that it is a likely scenario yet. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14448 Test Plan: - **Unit tests** (`db/db_iterator_test.cc`): `ReadPathRangeTombstoneTest` parameterized by forward/reverse with cases for basic insertion, non-contiguous (below threshold), memtable switch, exhausted iterator with/without bounds, direction change, mixed Delete/SingleDelete, single-delete-only runs, snapshot predating memtable, block cache tier incomplete, skip when covered by existing range, UDT basic scan, UDT exhaustion, prefix filter cross-prefix scan (`PrefixFilterCrossPrefixScanCoversLiveKey` with default/total_order_seek/prefix_same_as_start variants), stale ikey from forward-then-reverse scan (`StaleIkeyFromForwardThenReverse`), and reseek stale ikey (`ReseekStaleIkey`). - **Concurrency test** (`db/db_test2.cc`): `DBTestConcurrentRangeTombstoneConversions` parameterized by `(allow_concurrent_memtable_write, min_tombstones_for_range_conversion)` with mixed writers, deleters, range deleters, and concurrent forward/reverse readers. - **Transaction tests** (`utilities/transactions/write_prepared_transaction_test.cc`, `write_unprepared_transaction_test.cc`): Tests for WritePrepared (insertion allowed when tombstones committed before prepare, blocked when after; seqno bump shadowing prepared writes `RangeTombstoneSeqnoBumpShadowsPreparedWrite`) and WriteUnprepared (multiple batches, extended visibility with CalcMaxVisibleSeq, own deletions with rollback). - **IterKey::Swap tests** (`db/dbformat_test.cc`): `IterKeySwapTest` parameterized over `(key_len, copy, use_secondary)` × 2 covering all inline/heap/pinned/secondary combinations. - **InlineSkipList MultiGet tests** (`memtable/inlineskiplist_test.cc`): Basic, exact matches, empty, single key, randomized validation against `std::set::lower_bound`, duplicate keys with callback walk, and concurrent MultiGet with read-after-write consistency. - **Memtable MultiGet tests** (`db/db_basic_test.cc`): Batch lookup, overwrite, flush, merge, disabled by default, paranoid checks, and snapshot tests. - **Stress test coverage**: `min_tombstones_for_range_conversion` and `memtable_batch_lookup_optimization` options added to `db_crashtest.py` and `db_stress` flags. - `make check` passes all tests. ### Benchmark results Tombstones scattered randomly in clusters via `seek_nexts_to_delete` for realistic workloads. **DB Setup A (scattered deletes, 100 per seek)**: ``` # Step 1: Create and compact ./db_bench --benchmarks=fillseq,compact --seed=1 --compression_type=none --num=1000000 --db=<DB> # Step 2: Scatter tombstones (5000 seeks × 100 deletes ≈ 500k tombstones) ./db_bench --benchmarks=seekrandom,flush --seed=1 --compression_type=none --num=2000 \ --seek_nexts=0 --seek_nexts_to_delete=100 --use_existing_db=1 --threads=1 --db=<DB> ``` **DB Setup A2 (scattered deletes, 8 per seek)**: ``` # Step 1: Create and compact ./db_bench --benchmarks=fillseq,compact --seed=1 --compression_type=none --num=1000000 --db=<DB> # Step 2: Scatter tombstones (5000 seeks × 8 deletes ≈ 40k tombstones) ./db_bench --benchmarks=seekrandom,flush --seed=1 --compression_type=none --num=2000 \ --seek_nexts=0 --seek_nexts_to_delete=8 --use_existing_db=1 --threads=1 --db=<DB> ``` **DB Setup B (no deletes)**: ``` ./db_bench --benchmarks=fillseq,compact --seed=1 --compression_type=none --num=1000000 \ [--key_size=100] --db=<DB> ``` **Read workload** (same for all): ``` ./db_bench --benchmarks=seekrandom --seek_nexts=100 --threads=8 \ --reverse_iterator={true,false} --seed=1 --use_existing_db=1 \ --compression_type=none --num=1000000 --duration=10 \ --disable_auto_compactions \ [--key_size=100] [--min_tombstones_for_range_conversion=X] --db=<DB_COPY> ``` Each workload averaged over 3 runs. **Table 1: seekrandom forward, scattered deletes (2000 seeks × 100 deletes/seek)** | Variant | avg ops/s | % vs main | |---------|-----------|-----------| | main | 2,895 | - | | threshold=0 | 2,869 | -0.9% | | threshold=8 | 287,334 | +9,824% | **Table 2: seekrandom reverse, scattered deletes (2000 seeks × 100 deletes/seek)** | Variant | avg ops/s | % vs main | |---------|-----------|-----------| | main | 544 | - | | threshold=0 | 548 | +0.7% | | threshold=8 | 206,491 | +37,860% | **Table 3: seekrandom forward, scattered deletes (2000 seeks × 8 deletes/seek)** | Variant | avg ops/s | % vs main | |---------|-----------|-----------| | main | 194,049 | - | | threshold=0 | 195,703 | +0.9% | | threshold=8 | 310,740 | +60.1% | **Table 4: seekrandom reverse, scattered deletes (2000 seeks × 8 deletes/seek)** | Variant | avg ops/s | % vs main | |---------|-----------|-----------| | main | 63,854 | - | | threshold=0 | 69,266 | +8.5% | | threshold=8 | 218,101 | +241.6% | **Table 5: seekrandom forward, no deletes (regression check)** | Variant | key=16B avg ops/s | % vs main | key=100B avg ops/s | % vs main | |---------|-------------------|-----------|---------------------|-----------| | main | 330,901 | - | 236,048 | - | | threshold=0 | 328,398 | -0.8% | 238,055 | +0.9% | | threshold=8 | 332,539 | +0.5% | 233,776 | -1.0% | **Table 6: seekrandom reverse, no deletes (regression check)** | Variant | key=16B avg ops/s | % vs main | key=100B avg ops/s | % vs main | |---------|-------------------|-----------|---------------------|-----------| | main | 261,445 | - | 192,177 | - | | threshold=0 | 265,020 | +1.4% | 191,616 | -0.3% | | threshold=8 | 250,881 | -4.0% | 189,239 | -1.5% | Reviewed By: xingbowang Differential Revision: D96203950 Pulled By: joshkang97 fbshipit-source-id: 06ba66ebde3c355f04671d1e681f1b1586e8751d	2026-04-03 22:47:51 -07:00
Xingbo Wang	af4e32945b	Blob direct write v1: write-path blob separation with partitioned files (reduced scope) (#14535) ... Summary: This PR introduces **blob direct write v1**, a reduced-scope write-path optimization where large values (>= `min_blob_size`) are written directly to blob files during `Put()` and replaced in the memtable with compact `BlobIndex` references. This avoids holding full values in memory until flush time. ### Motivation In the existing BlobDB architecture, values are written to the WAL and memtable in their full form and separated into blob files only at flush time. This means: - Large values are held in memory twice (raw in memtable + blob file at flush) - Blob I/O is serialized through a single flush thread per column family Blob direct write addresses both: values leave the write path as small `BlobIndex` references, and multiple **partitions** (configurable via `blob_direct_write_partitions`) allow concurrent blob writes with independent locks. ### Design (v1 — single-writer, WAL-disabled, reduced scope) The v1 design intentionally keeps scope narrow for correctness and reviewability: - **Single writer thread assumption**: no concurrent writes to the same partition file. One logical writer serializes the batch. - **WAL-disabled**: direct-write blob files are only registered in MANIFEST at flush time. WAL replay cannot recover unregistered blob references, so WAL is disabled for this v1. - **Flush-on-write**: each `AddRecord` call flushes to the OS immediately. - **FIFO generation batching**: each memtable switch creates one generation batch. Direct-write files for that memtable are sealed and registered atomically when the batch is flushed to MANIFEST. - **Round-robin partitions**: blob writes are distributed across `blob_direct_write_partitions` files using an atomic counter. ### New components | Component | Description | |---|---| | `BlobFilePartitionManager` | Owns N partition files per CF. Manages open/seal/register lifecycle tied to memtable generations. | | `BlobWriteBatchTransformer` | A `WriteBatch::Handler` that rewrites qualifying `Put` values as `BlobIndex` entries before the batch enters the write group. | ### Write path integration 1. `DBImpl::WriteImpl` calls `BlobWriteBatchTransformer::TransformBatch` before entering the writer group (for default write path), or before joining the batch group (for pipelined/unordered write). 2. Values >= `min_blob_size` are written to a partition file; the key is stored with a `BlobIndex` in the transformed batch. A rollback guard marks blob bytes as initial garbage if the write fails. 3. On `SwitchMemtable`, `RotateCurrentGeneration` moves active partitions into the next immutable batch. 4. `FlushMemTableToOutputFile` / `AtomicFlushMemTablesToOutputFiles` call `PrepareFlushAdditions` to seal partition files and collect `BlobFileAddition` + `BlobFileGarbage` entries registered to MANIFEST alongside the flush. 5. Shutdown paths (`CancelAllBackgroundWork`, `WaitForCompact` with `close_db=true`) force-flush all CFs with active direct-write managers to ensure blob files are registered before close. ### Read path - **Get/MultiGet**: `MaybeResolveBlobForWritePath` resolves `BlobIndex` references found in memtable or immutable memtable via `BlobFilePartitionManager::ResolveBlobDirectWriteIndex`, which first checks manifest-visible state and falls back to direct blob-file reads via `BlobFileCache`. - **Iterator**: `DBIter::BlobReader` is extended with a `BlobFilePartitionManager*` to resolve direct-write blob indexes during iteration. The unified `ResolveBlobDirectWriteIndex` path handles both manifest-visible and not-yet-flushed files. ### New options | Option | Default | Description | |---|---|---| | `enable_blob_direct_write` | `false` | Enable write-path blob separation for this CF. Requires `enable_blob_files = true`. Not dynamically changeable. | | `blob_direct_write_partitions` | `1` | Number of parallel partition files per CF. Not dynamically changeable. | ### Feature incompatibilities (reduced v1 scope) The following features are *not supported* when `enable_blob_direct_write = true`, and are enforced both in `db_stress_tool` validation and `db_crashtest.py` sanitization: **Write model constraints:** - `threads` must be 1 (single writer assumption) - `allow_concurrent_memtable_write` = 0 - `enable_pipelined_write` = 0 (transformation done before batch group, but pipelined path supported with pre-transform) - `two_write_queues` = 0 - `unordered_write` = 0 (transformation done before batch group, but unordered path supported with pre-transform) **WAL and recovery:** - `disable_wal` = 1 (required — WAL replay of unregistered blob files is out of v1 scope) - `best_efforts_recovery` = 0 - `reopen` = 0 (no crash-restart with WAL replay) - All WAL-related stress features disabled: `manual_wal_flush_one_in`, `sync_wal_one_in`, `lock_wal_one_in`, `get_sorted_wal_files_one_in`, `get_current_wal_file_one_in`, `track_and_verify_wals`, `rate_limit_auto_wal_flush`, `recycle_log_file_num` **Blob GC and dynamic options:** - `use_blob_db` = 0 (stacked BlobDB not supported) - `allow_setting_blob_options_dynamically` = 0 - `enable_blob_garbage_collection` = 0 - `blob_compaction_readahead_size` = 0 - `blob_file_starting_level` = 0 **Unsupported value types and APIs:** - Merge (`use_merge`, `use_full_merge_v1`) — merge values pass through untransformed - Entity APIs (`use_put_entity_one_in`, `use_get_entity`, `use_multi_get_entity`, `use_attribute_group`) - `use_timed_put_one_in` - User-defined timestamps (`user_timestamp_size`, `persist_user_defined_timestamps`, `create_timestamped_snapshot_one_in`) - Transactions (`use_txn`, `use_optimistic_txn`, `test_multi_ops_txns`, `commit_bypass_memtable_one_in`) — though `WriteCommittedTxn::CommitInternal` falls back from bypass-memtable to normal path when BDW is active - `IngestWriteBatchWithIndex` returns `NotSupported` - `inplace_update_support` = 0 **Fault injection:** - All write/read/metadata fault injection disabled (`sync_fault_injection`, `write_fault_one_in`, `metadata_write_fault_one_in`, `read_fault_one_in`, `metadata_read_fault_one_in`, `open_*_fault_one_in`) **Infrastructure/snapshot APIs:** - `remote_compaction_worker_threads` = 0 - `test_secondary` = 0 - `backup_one_in` = 0 - `checkpoint_one_in` = 0 - `get_live_files_apis_one_in` = 0 - `ingest_external_file_one_in` = 0 - `ingest_wbwi_one_in` = 0 ### Tests - `db/blob/db_blob_basic_test.cc`: ~660 lines of new direct-write unit tests covering basic put/get, multi-partition, flush/compaction, recovery, and error injection. - `db/blob/blob_file_cache_test.cc`: ~96 lines of new tests for direct-write blob file cache behavior. - `db/write_batch_test.cc`: ~96 lines of tests for WriteBatch with blob index entries. - `utilities/transactions/transaction_test.cc`: verifies transaction commit path falls back correctly with direct write enabled. - `db_stress_tool/`: full stress test support with `--enable_blob_direct_write` and `--blob_direct_write_partitions` flags, integrated into `db_crashtest.py` with 10% random selection alongside regular blob params. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14535 Test Plan: ``` make -j128 db_blob_basic_test && ./db_blob_basic_test make -j128 blob_file_cache_test && ./blob_file_cache_test make -j128 write_batch_test && ./write_batch_test make -j128 transaction_test && ./transaction_test make -j128 check ``` Stress test: ``` python3 tools/db_crashtest.py blackbox --enable_blob_direct_write=1 \ --enable_blob_files=1 --blob_direct_write_partitions=4 \ --disable_wal=1 --threads=1 ``` Reviewed By: pdillinger Differential Revision: D98766843 Pulled By: xingbowang fbshipit-source-id: 1577653826913a59d05680a87bce5534ac5a5e69	2026-04-02 07:31:56 -07:00
Josh Kang	5db0603613	Read-triggered compactions (#14426) ... Summary: Add read-triggered compaction, a new feature that reduces read amplification by compacting SST files that receive high read traffic. When an SST file's read frequency (`num_reads_sampled / file_size`) exceeds a configurable threshold, it is marked for compaction to a lower level. The feature introduces two new options: a CF option `read_triggered_compaction_threshold` (default 0, disabled) and a DB option `max_periodic_compaction_trigger_seconds` (default 43200s) that controls how often the background thread re-evaluates compaction scores on quiet databases. Both options are dynamically changeable. Lowering `max_periodic_compaction_trigger_seconds` does add some overhead, but generally is minimal, so running this every couple of minutes in a production environment seems fairly reasonable. ## Key changes - **New CF option `read_triggered_compaction_threshold`** (`advanced_options.h`): When positive, files with `reads_per_byte > threshold` are marked for compaction. Files at the last non-empty level are skipped (bottommost compaction handles those separately). Marked files are sorted by hotness (reads_per_byte descending). - **New DB option `max_periodic_compaction_trigger_seconds`** (`options.h`): Replaces the hardcoded 12-hour ceiling in `ComputeTriggerCompactionPeriod()`. Essential for read-triggered compaction on quiet DBs since there are no writes to trigger score re-evaluation. - **Leveled compaction picker** (`compaction_picker_level.cc`): Adds read-triggered as the lowest-priority compaction reason in `SetupInitialFiles()`, using the existing `PickFileToCompact` helper. - **Universal compaction picker** (`compaction_picker_universal.cc`): Adds `PickReadTriggeredCompaction` as lowest priority. Refactors shared "find output level + compute overlapping inputs + create Compaction" logic from both `PickDeleteTriggeredCompaction` and `PickReadTriggeredCompaction` into `BuildCompactionToNextLevel`, handling both single-level and multi-level universal cases. - **Periodic trigger integration** (`db_impl.cc`): `TriggerPeriodicCompaction` now also fires for CFs with `read_triggered_compaction_threshold > 0`, even without time-based compaction configured. - **Stress test & db_bench support**: Both `db_stress` and `db_bench` support the new options. `db_crashtest.py` randomly enables read-triggered compaction and sets a short periodic trigger interval when enabled. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14426 Test Plan: **Unit tests**: - `compaction_picker_test` — 7 new tests: `ReadTriggeredCompactionDisabled`, `ReadTriggeredCompactionBelowThreshold`, `ReadTriggeredCompactionAboveThreshold`, `NeedsCompactionReadTriggered`, `ReadTriggeredPicksFile`, `UniversalReadTriggeredCompaction`, `ReadTriggeredSkipsLastLevel`, `UniversalReadTriggeredNoPickWhenNotMarked` - `db_compaction_test` — `ReadTriggeredCompaction` integration test verifying end-to-end behavior with sync points - Stress test coverage **Stress test**: ``` make V=1 -j "CRASH_TEST_EXT_ARGS=--duration=600 --max_key=2500000 --max_compaction_trigger_wakeup_seconds=10 --read_triggered_compaction_threshold=0.0001 --interval=600" blackbox_crash_test ``` - confirmed read triggered compactions from LOGS **Benchmark** (`db_bench`): Setup: 5M keys (100B values, 16B keys), leveled compaction, 5 levels, 4MB target file size. DB fully compacted, then 2M overlapping keys written without compaction to create L0/L1 overlap (82 files, ~294MB). LSM shape change during readrandom with read-triggered compaction: ``` BEFORE: L0=9 files (15MB), L1=4 (16MB), L2=20 (69MB), L3=49 (194MB) — 82 files, 294MB AFTER: L3=66 files (223MB) ``` | Benchmark | Config | avg ops/s | % change | |-----------|--------|-----------|----------| | readrandom (8 threads, 5M reads) | baseline (threshold=0) | 1,086,965 | — | | readrandom (8 threads, 5M reads) | threshold=0.000001, trigger=5s | 1,453,697 | **+33.7%** | Reviewed By: xingbowang Differential Revision: D97838716 Pulled By: joshkang97 fbshipit-source-id: a21fcb270c7fadd4f78d98b9c821982f220dd3f0	2026-03-27 14:43:52 -07:00
Danny Chen	b6f498b2c9	Add verify_manifest_content_on_close option (#14451) ... Summary: Add a new mutable DB option `verify_manifest_content_on_close` (default: false). When enabled, on DB close the MANIFEST file is read back and all records are validated (CRC checksums via log::Reader and logical content via VersionEdit::DecodeFrom). If corruption is detected, a fresh MANIFEST is written from in-memory state using the existing LogAndApply recovery path. This complements the existing size validation in VersionSet::Close() with content validation, reusing the same manifest reading pattern as VersionSet::Recover(). Implementation plan: ## Part 1: New DB Option — verify_manifest_content_on_close - A new mutable bool DB option (default: false) that can be dynamically toggled via SetDBOptions() at runtime, following the pattern of other mutable manifest options like max_manifest_file_size. - Propagation: SetDBOptions() -> DBImpl::mutable_db_options_ -> versions_->UpdatedMutableDbOptions() -> VersionSet::verify_manifest_content_on_close_ ## Part 2: Core Implementation — Content Validation in VersionSet::Close() - Inserted after existing size check, before closed_ = true - Opens manifest as SequentialFileReader, creates log::Reader with checksum=true - Loops ReadRecord with WALRecoveryMode::kAbsoluteConsistency, decodes each record as VersionEdit - On corruption: fires OnIOError listeners, logs error, calls LogAndApply with empty edit to trigger manifest rewrite from in-memory state - If manifest can't be opened for reading: logs warning, doesn't fail close ## Part 3: Unit Tests (in version_set_test.cc) - ManifestContentValidationOnClose_Clean: enable option, normal close, verify no manifest rotation - ManifestContentValidationOnClose_CorruptRecord: enable option, corrupt manifest via SyncPoint, verify rotation occurs and DB reopens cleanly - ManifestContentValidationOnClose_Disabled: default off, verify content validation does not run - ManifestContentValidationOnClose_SizeCheckFails: truncate manifest so size check fails first, verify recovery via size-check path ## What Happens If a Corruption is Detected If corruption was detected, four things happen: 1. **Notify listeners** — Fires `OnIOError` on all registered event listeners (from db_options_->listeners) so monitoring/alerting systems can observe the corruption event. Uses `FileOperationType::kVerify` to categorize it. 2. **Permit unchecked errors** — `PermitUncheckedError()` silences RocksDB's debug-mode assertion that every `IOStatus` must be inspected. These statuses are informational-only here; the real recovery is via `LogAndApply`. 3. **Log the error** — Writes a `ROCKS_LOG_ERROR` message with the filename for operational visibility (grep-able in production logs). 4. **Rewrite the manifest via `LogAndApply`** — This is the actual recovery. `LogAndApply` is called with an empty `VersionEdit` (no changes). Internally, `LogAndApply` detects that the current `descriptor_log_` is null (it was reset at line 5551, or by the previous `LogAndApply` in the size-check path) and creates a brand-new MANIFEST file. It serializes the entire current in-memory LSM state — all column families, all levels, all file metadata, sequence numbers, etc. — into this new file. It then atomically updates the `CURRENT` file pointer to reference the new MANIFEST. This works because the in-memory state was built from the original manifest during `DB::Open()` and has been kept fully up to date through all subsequent operations (flushes, compactions, etc.) during the DB's lifetime. The on-disk manifest is essentially a journal of changes; `LogAndApply` with an empty edit produces a fresh, compacted snapshot of that state. ## Flow Diagram of Manifest Content Validation VersionSet::Close() │ ├─ Close descriptor_log_ and check size │ └─ Size mismatch? → LogAndApply (rewrite manifest) │ ├─ Content validation (if s.ok() && option enabled) │ ├─ Open manifest for sequential reading │ │ └─ Can't open? → WARN log, continue │ │ │ ├─ For each record: │ │ ├─ ReadRecord (CRC32 check, kAbsoluteConsistency) │ │ └─ DecodeFrom (VersionEdit logical check) │ │ │ └─ Corruption detected? │ ├─ Notify OnIOError listeners │ ├─ LOG_ERROR │ └─ LogAndApply (rewrite manifest from in-memory state) │ └─ closed_ = true; return s; ## How This Relates to the Existing Size Check The existing size check (lines 5556-5582) and the new content validation are complementary: | Check | What it catches | How it checks | |----------------|-----------------------------------------|----------------------------| | Size check | Truncation, partial writes, extra bytes | Compare expected vs actual file size | | Content check | Bit-rot, silent corruption, bad records | CRC32 + VersionEdit decode | The size check catches gross corruption (file too short or too long). The content check catches subtle corruption where the file is the right size but individual bytes have been flipped (e.g., storage media bit-rot, buggy filesystem, incomplete block write). Both recovery paths use the same mechanism: `LogAndApply` with an empty `VersionEdit` to rewrite the manifest from in-memory state. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14451 Reviewed By: xingbowang Differential Revision: D96004906 Pulled By: dannyhchen fbshipit-source-id: 0b0ecdada3a74e97d2cadbba2091b8b577f1d684	2026-03-19 12:01:23 -07:00
anand76	c66c14258f	Add memtable MultiGet finger search optimization (#14428) ... Summary: Add memtable batch lookup optimization with finger search Optimize memtable MultiGet by using a finger search on the skip list. After finding key[i], the search path (Splice) is retained as a "finger" for key[i+1]. The next search walks up the finger until the forward pointer overshoots, then descends -- costing O(log d) where d is the distance between consecutive sorted keys, rather than O(log N) from the head each time. Controlled by the new `memtable_batch_lookup_optimization` column family option (default: false). ## Changes - Add `FindGreaterOrEqualWithFinger()` and `MultiGet()` to `InlineSkipList` with optional paranoid validation support (key ordering checks and per-key checksum verification via default parameters) - Add virtual `MultiGet()` to `MemTableRep`, override in `SkipListRep` - Add `memtable_batch_lookup_optimization` CF option - Integrate finger search into `MemTable::MultiGet` -- sorts keys, performs batched finger search, then unscrambles results - Add `db_bench`, `db_stress`, and crash test support - Add unit tests for `InlineSkipList::MultiGet` (5 tests) and integration tests at the DB level (25 `BatchLookup` tests), including paranoid validation tests ## Benchmark Results Setup: 2M keys in memtable, release build (`DEBUG_LEVEL=0`). ### Single-threaded `multireadrandom` | Batch Size | Baseline | BatchOpt | vs Base | |:---:|---:|---:|---:| | 2 | 363,593 | 376,562 | **+3.6%** | | 8 | 385,204 | 386,082 | **+0.2%** | | 32 | 360,339 | 375,105 | **+4.1%** | | 64 | 352,696 | 378,497 | **+7.3%** | ### Multithreaded `multireadrandom` (batch_size=64) | Threads | Baseline | BatchOpt | vs Base | |:---:|---:|---:|---:| | 4 | 171,356 | 185,633 | **+8.3%** | | 8 | 161,478 | 171,392 | **+6.1%** | ### `multireadwhilewriting` (batch_size=64) | Threads | Baseline | BatchOpt | vs Base | |:---:|---:|---:|---:| | 1 | 163,938 | 175,068 | **+6.8%** | | 4 | 109,698 | 120,790 | **+10.1%** | | 8 | 116,623 | 123,658 | **+6.0%** | No regression at any batch size or thread count. The finger is stack-allocated per `MultiGet` call, so there is no shared state or cache-line contention between threads. Concurrent writes don't invalidate the finger's bracket since it only reads via acquire-load `Next()` pointers. Small Batch Size Regression Check: memtable_batch_lookup_optimization ===================================================================== All values are ops/sec. Measured with db_bench (release build). 2M keys in memtable, value_size=100, duration=30s per run. Multithreaded multireadrandom — small batch sizes -------------------------------------------------- Threads | Batch Size | Baseline | BatchOpt | Change 4 | 2 | 532,302 | 540,682 | +1.6% 8 | 2 | 1,044,046 | 1,046,920 | +0.3% 4 | 8 | 633,733 | 630,039 | -0.6% 8 | 8 | 1,260,818 | 1,241,792 | -1.5% multireadwhilewriting — small batch sizes ----------------------------------------- Threads | Batch Size | Baseline | BatchOpt | Change 1 | 2 | 107,284 | 105,217 | -1.9% 4 | 2 | 428,508 | 423,747 | -1.1% 8 | 2 | 854,654 | 864,923 | +1.2% 1 | 8 | 114,788 | 118,496 | +3.2% 4 | 8 | 467,995 | 473,437 | +1.2% 8 | 8 | 935,636 | 960,791 | +2.7% No regression at small batch sizes. Variations at batch_size=2 are within noise (~1-2%). At batch_size=8, modest positive trend (+1-3% in read-while-writing). Pull Request resolved: https://github.com/facebook/rocksdb/pull/14428 Test Plan: - `make check` -- all tests pass - `ASSERT_STATUS_CHECKED=1 make check` -- all tests pass - 2-hour `db_crashtest.py blackbox` with `--memtable_batch_lookup_optimization=1` -- passed - 2-hour `db_crashtest.py blackbox` with `--memtable_batch_lookup_optimization=1 --paranoid_memory_checks=1` -- passed Reviewed By: pdillinger Differential Revision: D95813786 Pulled By: anand1976 fbshipit-source-id: b182a023e1026021f1b9682d1cc024c7729326c7	2026-03-16 10:45:49 -07:00
Maciej Szeszko	1ed5052486	Prepopulate block cache during compaction (#14445) ... Summary: When RocksDB operates with tiered or remote storage (e.g., Warm Storage, HDFS, S3), reading recently compacted data incurs high-latency remote reads because compaction output files are not present in the block cache. The existing `prepopulate_block_cache = kFlushOnly` avoids this for flush output but leaves compaction output cold until first access. Add a new `PrepopulateBlockCache::kFlushAndCompaction` enum value that warms all block types (data, index, filter, compression dict) into the block cache during both flush and compaction. Flush-warmed blocks use `LOW` priority (unchanged from kFlushOnly behavior), while compaction-warmed blocks use `BOTTOM` priority — compaction data is less temporally local than freshly flushed data, so it should be the first to be evicted when the cache is full. This gives the remote-read avoidance benefit without risking cache thrashing. The enum uses `kFlushAndCompaction` rather than separate `kCompactionOnly` + `kFlushAndCompaction` values because there is no practical use case for warming compaction output without also warming flush output. Flush output is by definition the hottest data (just written by the user), so if a workload benefits from warming the colder compaction output, it would always benefit from warming flush output too. The implementation reuses the existing `InsertBlockInCacheHelper` / `WarmInCache` infrastructure in `BlockBasedTableBuilder`. The only internal change is adding a `warm_cache_priority` field to `Rep` alongside the existing `warm_cache` bool, and plumbing it through to the `WarmInCache` call instead of the previously hardcoded `Cache::Priority::LOW`. ### Key changes - New `PrepopulateBlockCache::kFlushAndCompaction` enum value in table.h - `Rep::warm_cache_priority` field in BlockBasedTableBuilder for per-reason priority control - Serialization support ("kFlushAndCompaction" in string map) - db_bench support (--prepopulate_block_cache=2) - Crash test coverage (random choice includes new value) **NOTE:** Unlike flush output (which is inherently hot — just written by the user), it is hard to distinguish hot from cold blocks in compaction output. Warming all compaction output therefore risks polluting the block cache and evicting genuinely hot entries. The kFlushAndCompaction mode is recommended only for use cases where most or all of the database is expected to reside in cache (e.g., the working set fits in cache). For workloads where only a fraction of the data is hot, kFlushOnly remains the safer choice. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14445 Test Plan: - New `WarmCacheWithDataBlocksDuringCompaction` test: verifies data blocks from compaction output are present in the block cache and served without misses - Extended `DynamicOptions` test: verifies dynamic switching through kDisable -> kFlushAndCompaction -> kFlushOnly -> kDisable via SetOptions - Existing `WarmCacheWithDataBlocksDuringFlush` and parameterized `WarmCacheWithBlocksDuringFlush` tests continue to pass (kFlushOnly behavior unchanged) - db_block_cache_test: 81/81 passed - options_test: 74/74 passed - table_test: 6910/6910 passed Reviewed By: xingbowang Differential Revision: D95997952 Pulled By: mszeszko-meta fbshipit-source-id: 4ad568264992532053947df298e63e343821ddb5	2026-03-10 16:19:33 -07:00
Josh Kang	3ad23b2d94	Support automated interpolation search (#14383) ... Summary: Add automatic per-block interpolation search selection (`kAuto` mode) for index blocks. During SST construction, each index block's key distribution is analyzed using the coefficient of variation (CV) of gaps between restart-point keys. Blocks with uniformly distributed keys are flagged via a new bit in the data block footer, and at read time, `kAuto` resolves to interpolation search for uniform blocks and binary search otherwise. ## Key changes - **New `BlockSearchType::kAuto` enum value**: Resolves per-block at read time to either `kInterpolation` or `kBinary` based on the block's uniformity flag. Falls back to `kBinary` on older versions that don't recognize it. - **Write-path uniformity analysis**: `BlockBuilder::ScanForUniformity()` uses Welford's online algorithm to incrementally compute the CV of key gaps at restart points. The result is stored in a new bit (bit 30) of the data block footer's packed restart count. - **New table option `uniform_cv_threshold`** (default: -1 `disabled`): Controls how strict the uniformity check is. Set to negative to disable. Exposed in C++, Java (JNI), and `db_bench`. - **Code reorganization**: Block entry decode helpers (`DecodeEntry`, `DecodeKey`, `DecodeKeyV4`, `ReadBe64FromKey`) moved from `block.cc` to a new shared header `block_util.h` so they can be reused by `BlockBuilder` on the write path. - **New histogram `BLOCK_KEY_DISTRIBUTION_CV`**: Records the CV (scaled by 10000) of each index block's key distribution for observability. - **Java bindings**: `IndexSearchType.kAuto`, `uniformCvThreshold` getter/setter, JNI portal constructor signature updated, and `HistogramType.BLOCK_KEY_DISTRIBUTION_CV` added. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14383 Test Plan: - `IndexBlockTest.IndexValueEncodingTest` parameterized to include `kAuto` search type alongside `kBinary` and `kInterpolation`, verifying correct seek/iteration behavior across all combinations of key distributions, restart intervals, and key lengths. - Uniformity detection validated: blocks with uniform key distribution correctly set `is_uniform = true`, blocks with clustered/non-uniform keys set `is_uniform = false`. - Stress test coverage - Updated check_format_compatible to also include a "uniform" dataset. By default using uniform_cv_threshold=-1 does not result in an incompatibility issues. When manually changing the threshold (e.g. `uniform_cv_threshold=1000`), I see `bad block contents`, which is expected ## Benchmark readrandom with `fillrandom,compact -seed=1 --statistics`: | Benchmark | Branch | Params | avg ops/s | % change vs main | CV P50 | |-----------|--------|--------|-----------|------------------|--------| | readrandom | main | `binary_search, shortening=1` | 335,791 | baseline | N/A | | readrandom | feature | `binary_search, shortening=1` (default) | 335,749 | -0.0% | 1,500 | | readrandom | feature | `auto_search, shortening=1` (kAuto) | 366,832 | **+9.2%** | 1,500 | | readrandom | feature | `interpolation_search, shortening=1` | 366,598 | **+9.2%** | 1,500 | | readrandom | feature | `auto_search, shortening=2` (kAuto) | 344,631 | **+2.6%** | 1,030,000 | | readrandom | feature | `interpolation_search, shortening=2` | 201,178 | **-40.1%** | 1,030,000 | As seen with shortening=2, a non-uniform distribution produces a high CV, which does not use interpolation search. ## Write benchmark There is a write overhead which scans each restart entry for a block upon Finish. In practice this is very low because currently it is only applied to index blocks. See cpu profile (https://fburl.com/strobelight/io5hwj9h) here of `-benchmarks=fillseq,compact -compression_type=none -disable_wal=1`. Only 0.08% attributed to `ScanForUniformity`. Reviewed By: pdillinger Differential Revision: D94738890 Pulled By: joshkang97 fbshipit-source-id: 9661ac593c5fef89d49f3a8a027f1338a0c96766	2026-03-06 10:13:51 -08:00
Josh Kang	f25fb41da6	Add option to validate sst files in the background on DB open (#14322) ... Summary: Add `open_files_async` option for faster DB startup. When enabled, SST file opening and validation is deferred to a background thread after `DB::Open` returns, reducing startup latency for databases with many SST files. WAL recovery remains synchronous. To support this, `FindTable` is extended with a pinning mechanism that stores the cache handle directly on `FileMetaData` via a new `PinnedTableReader` class, and sets the table reader atomically so subsequent reads skip cache lookups. `FileDescriptor::table_reader` is replaced with `PinnedTableReader pinned_reader` which wraps a `std::atomic<TableReader*>` with acquire/release ordering to safely handle concurrent access between the background opener and read threads. Should validations fail, the background opener sets a `kAsyncFileOpen` background error. Future read requests will look up the table reader again via the cache, and if any validations fail there it will get propagated to the user (existing behavior when `max_open_files > 0`). This feature is most useful when `max_open_files=-1`, because otherwise file opening is already capped at 16 files and DB open should be fast. ## Restrictions - This feature also is incompatible with fifo compaction because fifo compaction requires reading table properties under DB mutex. When table reader is unpinned, this may cause a DB hang. - This feature is also incompatible with `skip_stats_update_on_db_open=false` because it will result in even longer DB open ## Key changes - New `open_files_async` DB option with C, Java, and `db_bench` bindings - `BGWorkAsyncFileOpen` background worker that opens all SST files post-`DB::Open`, with shutdown awareness via `shutting_down_` flag - New `PinnedTableReader` class in `version_edit.h` — thread-safe wrapper holding `std::atomic<TableReader*>` and `Cache::Handle*` with proper acquire/release ordering. Replaces the old `FileDescriptor::table_reader` raw pointer and `FileMetaData::table_reader_handle` - Extract `LoadTableHandlersHelper` into `db/version_util.cc` — shared between `VersionBuilder::LoadTableHandlers` (for version edits during recovery) and `BGWorkAsyncFileOpen` (for base storage post-open) - `FindTable` extended with `pin_table_handle` and `out_table_reader` params — when pinning is enabled, the table reader is stored on `FileMetaData` so Get/MultiGet/Iterator skip redundant cache lookups. `FindTable` now performs the pinned-reader fast-path check internally instead of requiring callers to check `fd.table_reader` beforehand - Note: pinning is explicit (not default) because some callers create temporary `FileMetaData`s that would need to properly clean up table handles - `CompactedDBImpl` updated to use `FindTable` + pinning instead of raw `fd.table_reader` access for Get/MultiGet - New `kAsyncFileOpen` background error reason in `listener.h` and `error_handler.cc` - Add a check in ~DBImpl to ensure async file open task has not been forgotten to be scheduled in (future) subclasses of DBImpl. Certain subclasses that never use it will need to explicitly mark it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14322 Test Plan: - `OpenFilesAsyncTest` parameterized over `num_flushes` (1, 20), `ReadType` (Get, MultiGet, Iterator), `max_open_files` (-1, 10), and `read_only` (true, false) - **ConcurrentFileAccess**: concurrent reads and compactions race with async opener - **AfterRead**: reads happen before async opener, verifying lazy open and that the opener sees already-pinned readers - **BeforeRead**: async opener completes first, verifying reads use pre-loaded table readers - **Shutdown**: DB closes before async opener starts, verifying clean cancellation with 0 file opens - **Error**: corrupted SST files, verifying `kAsyncFileOpen` background error is set and reads return corruption - **DropColumnFamily**: CF dropped before async opener runs, verifying the opener gracefully skips dropped CFs - Added to crash test ### Benchmark To simulate a high-latency remote filesystem, I set up a virtual filesystem with dm-delay using 10ms reads, 0 ms writes. ``` # Generate a DB with many L0 files TEST_TMPDIR=/data/users/jkangs/dm-delay-test/mnt ./db_bench -benchmarks=fillseq -disable_auto_compactions=true -write_buffer_size=1000 -num=1000000 ``` ``` ./db_bench -use_existing_db=true -db=/data/users/jkangs/dm-delay-test/mnt/dbbench -benchmarks=readrandom -reads=1 -report_open_timing=true -open_files_async=true -use_direct_reads -file_opening_threads=1 -skip_stats_update_on_db_open OpenDb: 25.1419 milliseconds ``` ``` ./db_bench -use_existing_db=true -db=/data/users/jkangs/dm-delay-test/mnt/dbbench -benchmarks=readrandom -reads=1 -report_open_timing=true -open_files_async=false -use_direct_reads -file_opening_threads=1 -skip_stats_update_on_db_open OpenDb: 23109.4 milliseconds ``` ### No read regressions On main branch ``` ./db_bench -use_existing_db=true -db=/dev/shm/dbbench -benchmarks=readrandom -seed=1 -threads=8 -duration=30 readrandom : 4.827 micros/op 1657100 ops/sec 30.005 seconds 49720992 operations; 183.3 MB/s (6198999 of 6198999 found) ``` On this branch ``` ./db_bench -use_existing_db=true -db=/dev/shm/dbbench -benchmarks=readrandom -seed=1 -threads=8 -duration=30 readrandom : 4.863 micros/op 1644808 ops/sec 30.007 seconds 49354992 operations; 182.0 MB/s (6099999 of 6099999 found) ./db_bench -use_existing_db=true -db=/dev/shm/dbbench -benchmarks=readrandom -seed=1 -threads=8 -duration=30 -open_files_async=true readrandom : 4.803 micros/op 1665392 ops/sec 30.004 seconds 49968992 operations; 184.2 MB/s (6222999 of 6222999 found) ``` Reviewed By: pdillinger, xingbowang Differential Revision: D93538033 Pulled By: joshkang97 fbshipit-source-id: 32ac70c112cd733b7c1e1c1e2e7ce6422318a5ae	2026-03-02 16:18:14 -08:00
zaidoon	ce0fff9f41	Add trie-based User Defined Index (UDI) plugin (#14310) ... Summary: Implement a Fast Succinct Trie (FST) index based on LOUDS encoding as a User Defined Index plugin for RocksDB's block-based tables. First step toward trie-based indexing per https://github.com/facebook/rocksdb/issues/12396. The trie uses hybrid LOUDS-Dense (upper levels, 256-bit bitmaps) + LOUDS-Sparse (lower levels, label arrays) encoding inspired by the [SuRF paper](https://www.pdl.cmu.edu/PDL-FTP/Storage/surf_sigmod18.pdf) (Zhang et al., SIGMOD 2018). The boundary between dense and sparse levels is automatically chosen to minimize total space. Key Components - **Bitvector** with O(1) rank and O(log n) select using a rank LUT sampled every 256 bits with popcount intrinsics. Uses uint32_t rank LUT entries (halving memory vs uint64_t). Includes word-level `AppendWord()` for efficient dense bitmap construction and `AppendMultiple()` optimized at word granularity for bulk bit fills. - **Streaming trie builder** using flat per-level arrays with deferred internal marking, handle migration for prefix keys, and lazy node creation. Infers trie structure directly from sorted keys via LCP analysis in a single pass (no intermediate tree). - **LoudsTrie** for immutable querying with BFS-ordered handle reordering built into single-pass level-by-level serialization. Move-only semantics with correct pointer re-seating after `std::string` move. - **LoudsTrieIterator** with rank-based traversal, key reconstruction from trie path, and stack-based backtracking for `Next()`. Uses packed 8-byte `LevelPos` (is_dense flag in bit 63) and `autovector<LevelPos, 24>` to avoid heap allocation. Key reconstruction uses a raw char buffer allocated once to `MaxDepth()+1` bytes. - **TrieIndexFactory/Builder/Reader/Iterator** implementing the `UserDefinedIndexFactory` interface. - **Zero-copy block handle loading** using two fixed-width uint64_t arrays (offsets + sizes) with 8-byte alignment, enabling O(1) initialization via direct pointer assignment. Seek Hot Path Optimizations - **Fanout-1 sparse fast path**: Most sparse nodes in tries built from zero-padded numeric keys have exactly one child. Detected via `start_pos + 1 == end_pos` and inlined as a single byte comparison, avoiding the full `SparseSeekLabel` call. - **Linear scan for small sparse nodes**: `SparseSeekLabel` uses sequential scan for nodes with ≤16 labels instead of binary search. Faster for common 10-child digit nodes where branch misprediction cost outweighs linear scan cost. - **Rank reuse**: `DenseLeafIndexFromRankAndHasChildRank` and `SparseLeafIndexFromHasChildRank` overloads accept pre-computed `has_child_rank` from the Seek descent, avoiding redundant `Rank1` calls. General Performance Optimizations - **Select-free sparse traversal**: Precomputed child position lookup tables (`s_child_start_pos_`/`s_child_end_pos_`) eliminate `Select1` calls during Seek. Sparse traversal tracks `(start_pos, end_pos)` directly, using only `Rank1` (O(1)) + array lookup (O(1)) for child descent. - **Cached label_rank pattern**: Eliminates redundant `Rank1` calls in hot paths (Seek, Next, Advance all cache and reuse the label_rank computed for has_child checking). - **Leaf index fast path**: When no prefix keys exist (common case), `SparseLeafIndex` and `DenseLeafIndexFromRank` skip the prefix-key `Rank1` calls entirely, reducing from 3 to 1 `Rank1` call. - **Popcount-based Select64** via 6-step binary search within 64-bit words. - MSVC portability using RocksDB's `BitsSetToOne`/`CountTrailingZeroBits`. Benchmark Results Trie Seek at 32K keys (16-byte keys, 5M lookups, median of 5 runs): | Configuration | ns/op | |---|---:| | Trie (optimized) | **118** | | Binary search (native) | 134 | Trie Seek is **~12% faster** than native binary search index at 32K keys per block. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14310 Reviewed By: anand1976 Differential Revision: D93921511 Pulled By: xingbowang fbshipit-source-id: ba18604bc6bd4f575311a1ae3047a541274869b6	2026-02-26 21:20:48 -08:00
Josh Kang	901c88e37b	Separate keys and values in data blocks (#14287) ... Summary: Introduce new table option with separated key-value storage in data blocks. This PR implements a new SST block format where keys and values are stored in separate sections within data blocks, rather than interleaved. Keys are stored first, followed by all values in a contiguous section. The motivation is better cpu cache hit rate during seeks and potentially better compression. The additional storage cost is a varint per restart point, and 4 bytes additional block footer. For a data block with a restart interval of 16, it is approximately 1 bit of overhead per entry. But compression actually performs better, resulting in ~3% storage savings from benchmark. For now I've opted to not separate kvs in non-data blocks since restart interval for those blocks is typically 1, and values are typically small and probably better inlined. ### New block layout ``` +------------------+ | Keys Section | <- Key entries with delta encoding +------------------+ | Values Section | <- (new) Values stored contiguously +------------------+ | Restart Array | <- Fixed32 offsets to restart points +------------------+ | Values Offset | <- (new) 4 bytes: offset to values section | Footer | <- 4 bytes: packed index_type + num_restarts +------------------+ ``` ### Entry Format **At restart points** ``` +--------------+------------------+----------------+-----------------+-----------+ | shared (v32) | non_shared (v32) | value_sz (v32) | value_off (v32) | key_delta | +--------------+------------------+----------------+-----------------+-----------+ ``` **At non-restart points** ``` +--------------+------------------+----------------+-----------+ | shared (v32) | non_shared (v32) | value_sz (v32) | key_delta | +--------------+------------------+----------------+-----------+ ``` - `value_offset` is only stored at restart points to save space - For non-restart entries, value offset is computed as: `prev_value_offset + prev_value_size` ### Forward Compatibility - We make use of reserved block footer bits to mark if a block has separated kv format. Should an older version read this, it will assume a very large block restart interval and result in a corruption error. ### Key Changes - **BlockBuilder**: Accumulates values in a separate buffer; value offsets are stored only at restart points (other entries derive offset from previous value's position). There is an additional memcpy cost to place the value data after the key data. - **Block iteration**: Iteration now needs to know if we are at a restart point. This will rely on `cur_entry_idx_`, which was previously only used for per-kv checksum purposes. In this new format, we also need to know the block_restart_interval, which was previously also only calculated for per-kv checksums. - **Table properties**: Store `data_block_restart_interval`, `index_block_restart_interval`, and `separate_key_value_in_data_block` in table properties Pull Request resolved: https://github.com/facebook/rocksdb/pull/14287 Test Plan: - Extended block_test, table_test, compaction_test to contain new separated_kv param - Added new parameter to crash test --- ## Benchmark ### Varying Value Size **Write:** `db_bench --num=1000000 --separate_key_value_in_data_block=<bool> --value_size=<X> --benchmarks=fillrandom,compact` **Read:** `db_bench --db=$DB --use_existing_db --benchmarks=readrandom,readseq` | Value Size | FillRandom (ops/s) | | Compact (s) | | ReadRandom (ops/s) | | ReadSeq (ops/s) | | SST Size (MB) | | |----------:|-------------------:|-----:|----------:|-----:|-------------------:|-----:|----------------:|-----:|-------------:|-----:| | | baseline | sep_kv | baseline | sep_kv | baseline | sep_kv | baseline | sep_kv | baseline | sep_kv | | 1 | 253,280 | 264,977 | 0.516 | 0.497 (**-3.7%**) | 596,328 | 586,625 (**-1.6%**) | 5,904,681 | 6,069,581 (**+2.8%**) | 5.1 | 4.2 (**-18.6%**) | | 16 | 235,757 | 242,367 | 0.572 | 0.533 (**-6.8%**) | 570,751 | 572,815 (**+0.4%**) | 5,371,138 | 5,604,908 (**+4.4%**) | 14.7 | 13.4 (**-8.5%**) | | 100 | 264,299 | 265,427 | 0.461 | 0.454 (**-1.5%**) | 323,696 | 332,790 (**+2.8%**) | 4,239,725 | 4,232,416 (**-0.2%**) | 38.8 | 37.6 (**-3.2%**) | | 1,000 | 238,992 | 242,764 | 2.349 | 2.329 (**-0.9%**) | 244,608 | 261,403 (**+6.9%**) | 1,285,394 | 1,265,868 (**-1.5%**) | 342.1 | 342.0 (**-0.0%**) | ### Varying Block Restart Interval **Write:** `db_bench --num=1000000 --separate_key_value_in_data_block=<bool> --block_restart_interval=<X> --benchmarks=fillrandom,compact` **Read:** `db_bench --db=$DB --use_existing_db --benchmarks=readrandom,readseq` | BRI | FillRandom (ops/s) | | Compact (s) | | ReadRandom (ops/s) | | ReadSeq (ops/s) | | SST Size (MB) | | |----:|-------------------:|-----:|----------:|-----:|-------------------:|-----:|----------------:|-----:|-------------:|-----:| | | baseline | sep_kv | baseline | sep_kv | baseline | sep_kv | baseline | sep_kv | baseline | sep_kv | | 1 | 251,654 | 263,707 | 0.453 | 0.485 (**+7.1%**) | 334,653 | 328,708 (**-1.8%**) | 4,194,342 | 3,954,291 (**-5.7%**) | 40.6 | 42.4 (**+4.5%**) | | 4 | 253,797 | 252,394 | 0.476 | 0.476 (**+0.0%**) | 332,719 | 341,676 (**+2.7%**) | 4,135,691 | 4,051,151 (**-2.0%**) | 39.2 | 39.3 (**+0.3%**) | | 8 | 260,143 | 262,273 | 0.496 | 0.460 (**-7.3%**) | 330,859 | 337,567 (**+2.0%**) | 4,144,081 | 4,187,389 (**+1.0%**) | 38.9 | 38.1 (**-2.1%**) | | 16 | 252,875 | 263,176 | 0.464 | 0.455 (**-1.9%**) | 323,783 | 335,418 (**+3.6%**) | 4,127,310 | 4,217,028 (**+2.2%**) | 38.8 | 37.6 (**-3.2%**) | | 32 | 260,224 | 269,422 | 0.464 | 0.451 (**-2.8%**) | 304,001 | 314,989 (**+3.6%**) | 4,310,162 | 4,247,248 (**-1.5%**) | 38.8 | 37.3 (**-3.8%**) | ### Varying Compression **Write:** `db_bench --num=1000000 --separate_key_value_in_data_block=<bool> --compression_type=<X> [--compression_level=<N>] --benchmarks=fillrandom,compact` **Read:** `db_bench --db=$DB --use_existing_db --benchmarks=readrandom,readseq` | Compression | FillRandom (ops/s) | | Compact (s) | | ReadRandom (ops/s) | | ReadSeq (ops/s) | | SST Size (MB) | | |------------|-------------------:|-----:|----------:|-----:|-------------------:|-----:|----------------:|-----:|-------------:|-----:| | | baseline | sep_kv | baseline | sep_kv | baseline | sep_kv | baseline | sep_kv | baseline | sep_kv | | None | 252,494 | 260,552 | 0.413 | 0.419 (**+1.5%**) | 356,290 | 371,535 (**+4.3%**) | 4,479,261 | 4,507,133 (**+0.6%**) | 73.5 | 73.6 (**+0.2%**) | | LZ4 | 246,010 | 256,360 | 0.477 | 0.455 (**-4.6%**) | 342,497 | 345,882 (**+1.0%**) | 4,400,570 | 4,268,102 (**-3.0%**) | 38.3 | 37.6 (**-2.0%**) | | ZSTD (L3) | 254,748 | 258,556 | 1.067 | 1.055 (**-1.1%**) | 176,724 | 177,566 (**+0.5%**) | 2,736,841 | 2,717,739 (**-0.7%**) | 32.9 | 31.3 (**-4.7%**) | | ZSTD (L6) | 256,459 | 259,388 | 1.556 | 1.462 (**-6.0%**) | 177,390 | 176,691 (**-0.4%**) | 2,754,336 | 2,688,682 (**-2.4%**) | 32.8 | 31.1 (**-5.1%**) | ### Varying Block Size **Write:** `db_bench --num=1000000 --separate_key_value_in_data_block=<bool> --block_size=<X> --benchmarks=fillrandom,compact` **Read:** `db_bench --db=$DB --use_existing_db --benchmarks=readrandom,readseq` | Block Size | FillRandom (ops/s) | | Compact (s) | | ReadRandom (ops/s) | | ReadSeq (ops/s) | | SST Size (MB) | | |-----------:|-------------------:|-----:|----------:|-----:|-------------------:|-----:|----------------:|-----:|-------------:|-----:| | | baseline | sep_kv | baseline | sep_kv | baseline | sep_kv | baseline | sep_kv | baseline | sep_kv | | 4 KB | 263,203 | 260,362 | 0.469 | 0.461 (**-1.7%**) | 324,178 | 332,249 (**+2.5%**) | 4,231,537 | 4,217,763 (**-0.3%**) | 38.8 | 37.6 (**-3.1%**) | | 16 KB | 252,742 | 263,161 | 0.426 | 0.428 (**+0.5%**) | 227,805 | 222,873 (**-2.2%**) | 5,146,997 | 5,081,080 (**-1.3%**) | 38.1 | 36.7 (**-3.6%**) | | 64 KB | 257,490 | 260,225 | 0.423 | 0.414 (**-2.1%**) | 86,807 | 91,586 (**+5.5%**) | 5,380,403 | 5,372,372 (**-0.1%**) | 36.3 | 35.0 (**-3.5%**) | ### Varying Key Size **Write:** `db_bench --num=1000000 --separate_key_value_in_data_block=<bool> --min_key_size=10 --max_key_size=100 --benchmarks=fillrandom,compact` **Read:** `db_bench --db=$DB --use_existing_db --benchmarks=readrandom,readseq` | Key Size | FillRandom (ops/s) | | Compact (s) | | ReadRandom (ops/s) | | ReadSeq (ops/s) | | SST Size (MB) | | |---------:|-------------------:|-----:|----------:|-----:|-------------------:|-----:|----------------:|-----:|-------------:|-----:| | | baseline | sep_kv | baseline | sep_kv | baseline | sep_kv | baseline | sep_kv | baseline | sep_kv | | 10–100 | 243,740 | 255,183 | 0.618 | 0.622 (**+0.6%**) | 284,304 | 307,569 (**+8.2%**) | 3,738,921 | 3,686,676 (**-1.4%**) | 41.5 | 41.2 (**-0.8%**) | ### CPU Profile Notes - No compression: DataBlock::SeekForGet uses less cpu (13.2% vs 13.9%) - https://fburl.com/strobelight/6mwwebft with separated KV - https://fburl.com/strobelight/m9m798ka without - ZSTD compression: rocksdb::DecompressSerializedBlock uses more CPU (45.8% vs 44.9%), while DataBlock::SeekForGet uses less cpu (5.09% vs 6.52%) - https://fburl.com/strobelight/3x5nw1k4 with separated KV - https://fburl.com/strobelight/e7809046 without --- Reviewed By: xingbowang, pdillinger Differential Revision: D92103024 Pulled By: joshkang97 fbshipit-source-id: 47cfeb656ff3c20d34975f0b6c4c0462935a83dc	2026-02-23 12:42:05 -08:00
Peter Dillinger	d3817f058d	Remove deprecated DB::Open raw pointer variants (and more) (#14335) ... Summary: and remove deprecated DB::MaxMemCompactionLevel(). In the process of pushing through a relatively clean refactoring of uses of the old functions, some other minor public APIs are also migrated from raw DB pointers to unique_ptr. Claude did pretty much all the work, but requiring dozens of prompts to actually push through relatively clean phase out of raw DB pointers from what needed to be touched, and leaving that code in better shape. (Hundreds of `DB*` still remain all over the place even outside C and Java bindings.) Pull Request resolved: https://github.com/facebook/rocksdb/pull/14335 Test Plan: existing tests; no functional changes intended Reviewed By: xingbowang, mszeszko-meta Differential Revision: D93523820 Pulled By: pdillinger fbshipit-source-id: e4ca22ad81cd2cfe91122d7507d7ca34fe03d043	2026-02-17 23:33:39 -08:00
Xingbo Wang	b040ab83e1	Add a new picking algorithm in fifo compaction (#14326) ... Summary: Add a new kv ratio based compaction picking algorithm in fifo compaction Pull Request resolved: https://github.com/facebook/rocksdb/pull/14326 Test Plan: Unit test Reviewed By: pdillinger Differential Revision: D93257941 Pulled By: xingbowang fbshipit-source-id: fd2d0e1356c7b54682a1197475a1bd26cb45c9d4	2026-02-15 10:04:58 -08:00
Josh Kang	9f47518676	Add interpolation search as an alternative to binary search (#14247) ... Summary: Interpolation search is an alternative algorithm to binary search, which performs better on uniformly distributed keys. Instead of binary search always computing the mid point of the left and right boundaries, interpolation search "interpolates" the mid point based on the distance to the target. Fortunately, we can re-use existing block format to support interpolation search. For a given block, we compute the shared_prefix length of the first and last key. Interpolation search is usually done with numerical target values, so for a variable binary length key, we calculate the "value" as the first 8 non-shared bytes. This also means interpolation search would only really be effective for bytewise comparator (guarded via options validations). #### Fallback to binary search - if the the val(left_key) == val(right_key) then we fallback to classic binary search (to avoid divide by 0) - interpolation search is significantly more computationally expensive than binary search, so when the search distance is small, we also fallback to binary search. - if interpolation search does not make significant progress (i.e. reduces search space by more than half each iteration), we can assume data is non-uniform and fallback. Interpolation search also performs best when there is minimal shortening, especially shortening of the last block, as it can heavily skew the distribution of the actual keys. Note that each search algorithm is guaranteed to make progress because at each iteration the search space is guaranteed to be reduce by at least 1. For now this change only applies to index block seeks, as data block seeks and other blocks do not have as many entries and would not require significant number of search rounds, but it could be easily extended to include that support. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14247 Test Plan: Updated unit tests and crash test with new search option ### Benchmark The default benchmark sets up keys in generally uniform distribution, so it was a good way to test performance improvements. Setup: `./db_bench -benchmarks=fillseq,compact -index_shortening_mode=1` #### Before this change ``` ./db_bench -use_existing_db=true -benchmarks=readrandom -seed=1 readrandom : 2.899 micros/op 344973 ops/sec 2.899 seconds 1000000 operations; 38.2 MB/s (1000000 of 1000000 found) ``` #### After this change Notice how key comparison counts are the same between the two. ``` ./db_bench -use_existing_db=true -benchmarks=readrandom -seed=1 -index_search_type=binary_search readrandom : 2.881 micros/op 347128 ops/sec 2.881 seconds 1000000 operations; 38.4 MB/s (1000000 of 1000000 found) ``` ``` ./db_bench -use_existing_db=true -benchmarks=readrandom -seed=1 -index_search_type=interpolation_search readrandom : 2.609 micros/op 383209 ops/sec 2.610 seconds 1000000 operations; 42.4 MB/s (1000000 of 1000000 found) ``` With a non-uniform distribution, `i.e. index_shortening_mode=2` ``` ./db_bench -use_existing_db=true -benchmarks=readrandom -seed=1 -index_search_type=binary_search readrandom : 2.958 micros/op 338075 ops/sec 2.958 seconds 1000000 operations; 37.4 MB/s (1000000 of 1000000 found) ``` ``` ./db_bench -use_existing_db=true -benchmarks=readrandom -seed=1 -index_search_type=interpolation_search readrandom : 5.502 micros/op 181750 ops/sec 5.502 seconds 1000000 operations; 20.1 MB/s (1000000 of 1000000 found) ``` Reviewed By: pdillinger Differential Revision: D91063163 Pulled By: joshkang97 fbshipit-source-id: 151d6aa76f8713740b714de6e406aff40d28ccbc	2026-02-13 17:15:10 -08:00