mirror of
https://github.com/facebook/rocksdb.git
synced 2026-10-23 08:09:02 +03:00
main
241 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Yoshinori Matsunobu
|
d6a7847d37 |
Add a DB option to log long-held global mutexes and release sites (#15436)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15436 ## Why Long-held global mutexes can stall foreground and background work. Existing mutex statistics measure acquisition waits, not continuous ownership. This adds opt-in logging to identify the mutex, owning thread, operation and release site. ## Enable for an investigation Provide an INFO logger and shared `Statistics` object when opening the DB, then: - Set the shared statistics level to `StatsLevel::kAll`. - Set `mutex_hold_log_threshold_us` through `DB::SetDBOptions`; `100000` means 100 ms. - Restore threshold 0 and the previous statistics level afterward. The threshold defaults to **0 (disabled)**. `kAll` alone does not enable these hold probes; threshold 0 leaves existing statistics timers unchanged. ## Output Illustrative records, not production observations: ``` SLOW_GLOBAL_MUTEX mutex=DBImpl::mutex_ mutex_addr=0x12340000 owner_thread_id=12345 thread_operation=Compaction hold_us=200000 threshold_us=100000 observed_end_us=1792205400000000 buffer_dropped_records=0 unlock=db_impl_compaction_flush.cc:4660 BackgroundCallCompaction() SLOW_GLOBAL_MUTEX mutex=DBImpl::wal_write_mutex_ mutex_addr=0x12340100 owner_thread_id=12346 thread_operation= hold_us=35000 threshold_us=25000 observed_end_us=1792205401000000 buffer_dropped_records=0 scope_exit=db_impl_write.cc:2250 PreprocessWrite() ``` The first proves a completed 200 ms DB-mutex ownership interval on a thread tagged `Compaction`. The second uses a 25 ms threshold; an empty operation tag means none was available. Neither record reports acquisition wait or proves the cause of a stall. Release attribution uses compiler-embedded strings available in optimized, stripped builds, with a GCC/Clang fallback where standard source locations are unavailable. Direct unlock and condition wait identify their call sites; scope exit and temporary unlock identify guard declarations, not closing braces. No stack unwinding or symbol lookup. ## Scope and safety - Track six selected mutexes: DB, WAL, point-lock wait graph, two prepare-log bookkeeping mutexes and transaction name map. This is not comprehensive synchronization tracing. - Acquisition waits and condition-variable release gaps are excluded from hold duration. Configuration is pre-first-use; repeated setup retains the first valid context. - Sample at most once per mutex per second, with eight deferred records per thread. `buffer_dropped_records` counts capacity overflow only; rate limiting, pause and emission contention can omit samples. - Emit after covered outer lock scopes release. Emission is nonblocking best-effort; shutdown drains/pauses emission and stops diagnostics before the owned logger closes. Closing from that logger's own callback is unsupported. - No data-file format change. The mutable option is persisted in OPTIONS and wired into bench/stress tools. Typed C binding is deferred; string DB-option APIs are available. Disabled bookkeeping still has a cost. Release A/B measurements found about 3 ns added per selected mutex pair at level 1, and about 70 ns for hold timing over `kAll`/0. Get and reused-iterator Seek showed no resolved small probe effect; fresh iterator+Seek at 16 threads showed +8.7% CPU/query and -9.4% QPS. No logs were emitted in those benchmarks; logging cost and broader workload qualification remain open. Reviewed By: xingbowang Differential Revision: D125790955 fbshipit-source-id: 1096163f3bd25ffee585dc6ec8ebfd7f1759851a |
||
Peter Dillinger
|
539532377d |
Remove Solaris support (#15449)
Summary: Solaris support was contributed by outside users starting with |
||
Andrew Chang
|
ee305ac1c3 |
Avoid a TLS init call on every perf counter update (#15428)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15428 RocksDB counts per-operation statistics (blocks read, bytes read, mutex waits, timings) in `perf_context`. It is a `thread_local` variable: each thread has its own copy, so threads never contend on it. This per-thread copy is "thread-local storage" (TLS). **Problem**: `PerfContext` has a non-trivial destructor. For such a variable, C++ must make sure the calling thread's copy is constructed before each use from another source file, so the compiler emits a call to a small "TLS init" function. That call goes through the PLT (procedure linkage table, the jump table used to call functions that may live in another shared library). Every `PERF_COUNTER_ADD` therefore paid a real function call before incrementing a counter, and these macros run in hot loops such as key comparisons during block iteration. `perf_level`, the on/off switch, has a smaller cost: it is `extern thread_local` without `constinit`, so each read also tests whether it has been initialized. **Fix**: - The perf macros read a `constinit thread_local PerfContext*` that each thread sets on first use. After that, a counter update is a pointer load, a null check and the increment. - `perf_level` is declared and defined `constinit`, so reading it is a plain thread-local load. - No public API change. `perf_context`, `get_perf_context()` and macro behavior are unchanged, and so is thread-exit behavior. **Results**: - Microbenchmark (`PERF_COUNTER_ADD` in a loop): 2.10 -> 0.84 ns per update with counting enabled; unchanged at 0.42 ns with counting disabled. - Calls to the TLS init functions drop from 40 and 96 to 0 in block.cc.o and memtable.cc.o. - In CPU profiles of large production RocksDB deployments, these accessors cost roughly 0.3-0.6% of total CPU. This change removes most of the `perf_context` part. Reviewed By: xingbowang Differential Revision: D125982209 fbshipit-source-id: f38b4fffda09611d28322e7534b135fb35631389 |
||
Josh Kang
|
4b983da055 |
Spell out special member functions (rule of five) (#15387)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15387 ## Summary RocksDB's `.clang-tidy` enables `cppcoreguidelines-special-member-functions`, `cppcoreguidelines-virtual-class-destructor` and the noexcept-move checks, and clang warns `-Wdeprecated-copy-with-user-provided-dtor`. A full scan reports 476 findings. In most of them, a class declares a destructor (or copy operations) but leaves the rest of copy/move implicit. That has two costs. Readers can't tell whether a class may be copied or moved. And resource-owning classes keep an implicit copy that would double-free if anyone ever used it. This diff resolves 453 of the 476. Approach, picked per class from least to most invasive: 1. An empty user-provided destructor becomes `= default`. This covers most interface classes, such as `virtual ~X() {}` -> `= default`. If the class is still copied, that would only rename the deprecated-copy warning, so the destructor is removed instead when nothing needs it (`Customizable`, `TableFactory` and `InternalKeyComparator` inherit a virtual one; `IteratorWrapperBase` isn't polymorphic), and `MemTableRep::KeyComparator` gets defaulted copy and move. 2. Classes whose copy is already deleted get explicitly deleted move operations too. Move was already unavailable, so this only documents it. 3. Resource-owning classes that are never copied get deleted copy and move operations. Examples: the C API wrapper structs, which run a `destructor_` callback (edited in `tools/c_api_gen/c_base.cc` and regenerated into `db/c.cc`); `ShardedCache`, which frees its shards; iterator wrappers; and RAII handles. 4. Value types that are copied or moved get the missing operations as `= default`, which keeps their current behavior. 5. Exception specifications of existing functions are unchanged, except for `WriteBatch`'s move assignment (below): no other `noexcept` is added or removed. Only the move operations this diff newly declares (`Comparator`, `FileSampledStats` and the thread-local `Entry`) are `noexcept`; they copy only scalars or atomics. `WriteBatch`'s copy and move assignment destroyed `*this` and placement-new'ed a new `WriteBatch` over it. That is undefined behavior when `*this` is a subclass, and if the copy constructor threw, `*this` was left destroyed and was later destroyed again. Move assignment now assigns member-wise and ends in the same state as the move constructor, which leaves `has_key_with_ts_`, the timestamp-size tracking and the other fields it doesn't copy at their defaults. `rep_` is swapped through a temporary, because a plain string move-assignment can keep the old buffer (e.g. a large batch reset with `batch = WriteBatch()` would keep its memory). Nothing in it can throw, so it is now `noexcept`. Copy assignment copies into a temporary and moves it in, so a throwing copy leaves `*this` unchanged. Bugs this exposed and fixed: - `RestoreAfterCopyOrCreateWorkItem` (backup engine): the hand-written move constructor and assignment moved only some fields, so `from_file` and `to_file` were empty after `push_back(std::move(...))`. A restore that hit a checksum mismatch then reported "While restoring -> : ..." without file names. It now uses the compiler-generated moves, which are still `noexcept`. - `ExternalSstFileIngestionJob` lived in a `std::vector`, and its destructor frees raw-owned pointers, so the implicit copy used on vector reallocation would double-free. Its file batches also point into its own inline file storage, so neither a copy nor a move is safe after `Prepare()`. Copy and move are now deleted, and `DBImpl` holds jobs through `std::unique_ptr`. Compatibility: no public class loses an operation users can call. Public interfaces mostly only get `virtual ~X() = default;`. `WriteBatch`'s move assignment becomes `noexcept`, which is not part of the mangled name, so the ABI doesn't change. `SubcompactionJobInfo` and `CompactionJobInfo` intentionally stay C++20 aggregates, so users' aggregate and designated initializers keep compiling. `Configurable`, `CompareInterface`, `WriteBatchBase`, `ObjectLibrary::Entry` and `MemTableRep::KeyComparator` declare defaulted copy and move, and `Customizable` no longer declares a destructor. As a result, `Configurable` and `Customizable` subclasses that declare no destructor or copy operations of their own now move their registered-options list instead of copying it. In-tree, these are the mutable-options adapters, the table factories, the memory allocators and test classes; all are held through smart pointers or used in place, and `Clone()` copies, so none is moved by value. Not changed (23): `IODebugContext` (custom deep copy plus a `std::shared_mutex`), `DBWithColumnFamilies` in db_bench (stored in a `std::vector` with non-movable members), `Snapshot`'s protected virtual destructor (intended), `ManagedSnapshot`, the `SubcompactionJobInfo` and `CompactionJobInfo` aggregates, and the 16 noexcept-move-constructor and noexcept-swap findings on existing functions, including `autovector`'s move assignment (adding `noexcept` would change their exception specifications). Copying `ManagedSnapshot` already double-releases the snapshot. Fixing that needs a real move implementation and is a behavior change, so it is left for a separate diff. Reviewed By: rban1 Differential Revision: D123186309 fbshipit-source-id: 6f8e57e233e12199a007407130bb9a1f6d52e8d1 |
||
|
Josh Kang
|
da82001d81 |
Mark overriding methods and destructors override (#15366)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15366 ## Summary RocksDB's `.clang-tidy` enables `modernize-use-override`, and clang reports `-Winconsistent-missing-destructor-override` when a class marks its other overrides `override` but not its destructor. A full scan finds 190 `modernize-use-override`, 100 `inconsistent-missing-destructor-override` and 2 `suggest-override` findings in existing code. Each one comes back whenever a nearby line is edited. They also leave classes where it isn't obvious which methods are overrides and which introduce new virtuals. This diff marks every overriding member function and destructor `override` and drops the redundant `virtual` on them. It also changes `final override` to plain `final`, following the check's default. Methods that introduce a new virtual keep `virtual`. Most edits are clang-tidy's own fix-its, applied mechanically and then clang-formatted. Compatibility: no signatures change. In public headers the only effect is `override` on existing overrides (mostly destructors, plus `Cache::Name()` and `Cache::GetPrintableOptions()`, whose base declarations are unconditional), so user code that subclasses these types is unaffected. Every touched class now marks all of its overrides consistently. This matters because a partially-marked class would make clang emit `-Winconsistent-missing-override` in user builds. Reviewed By: rban1 Differential Revision: D123186308 fbshipit-source-id: 1f489ea32d938e3ec0faf9abc4f7ff49e042d37a |
||
|
Josh Kang
|
928527b869 |
Fix bugs and false positives from clang-analyzer/bugprone (#15325)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15325 ## Summary RocksDB's `.clang-tidy` enables the `clang-analyzer-*` and `bugprone-*` checks. A full scan of the repo still reports 78 findings from these checks in existing code. Each time a nearby line is edited, they come back as noise, and a few of them are real defects. This diff resolves 73 of the 78; upstream has since fixed one more, and the four left are listed under "Intentionally not changed". Some fixes repair real bugs. The rest make the code's intent explicit to the analyzer without changing behavior. Real bugs fixed: - `port/mmap.cc`: `MemMapping::operator=(MemMapping&&)` ran the destructor on `*this`, `memcpy`'d over a non-trivially-copyable object, then placement-new'd the source. That is undefined behavior, and `*this` can be the base subobject of a `TypedMemMapping`. It now moves the members one by one, and the previous mapping is released through a local `MemMapping`. - `port/mmap.h`: `TypedMemMapping::operator=(MemMapping&&)` had no `return *this;`, so calling it was undefined behavior. - `cache/typed_cache.h`: `if constexpr (sizeof(TCreateContext) > 0)` is always true, so the branch meant for empty create contexts (for example `BlobContentsCreator`) could never run. The check is now `!std::is_empty_v<TCreateContext>`. Existing instantiations still call the same `Create` function. - `cassandra_row_merge_test.cc`: the test reused a vector after `std::move`-ing it into `RowValue::Merge(std::vector<RowValue>&&)`. That only worked because `Merge` happens not to consume its input. The test now rebuilds the input. Behavior-preserving changes: - Virtual calls in constructors and destructors are now explicitly qualified (for example `DBImpl::CloseImpl()` in `~DBImpl`). During construction and destruction, dispatch already resolved to these functions. - Stale `/*name=*/` argument comments now match the parameter names. - `unique_ptr::release()` after a successful cache `Insert` (where the cache takes ownership) uses `[[maybe_unused]] T* released = x.release();`, the pattern already used in `table_cache.cc`. - Removed `class Mutex;` and `class StaticMeta;` forward declarations that sat in the wrong namespace. - Added default member initializers for the `TablePropertiesCollectorFactory::Context` fields that its default constructor left uninitialized. - Smaller cleanups: a redundant `cond ? 3 : 3` in a test, `sizeof(std::string)` in a memory estimate, and an explicit `Cleanable` base move via `up_cast`. Tests that intentionally use moved-from objects now carry `NOLINT(bugprone-use-after-move)`, matching the existing convention. Intentionally not changed: enum casts in tests that check invalid-value handling; three `clang-analyzer-core.NullDereference` reports the analyzer can't prove safe (the C API timestamp out-parameters in `db/c.cc`, which are only written when the caller asked for a timestamp, and the waiter-list walk in `util/async_file_reader.cc`); bit-flag enums in public headers (`FileOpenContract`, `OptionTypeFlags`), where OR-combined values are valid; ownership transfers the analyzer can't see across translation units (`partitioned_index_reader.cc`, `fs_posix.cc`); and the intentional two-separator check in `file/filename.cc`. Reviewed By: rban1 Differential Revision: D123186307 fbshipit-source-id: 623e871286b1c73263b1309cb194692194199e0f |
||
Ranjan Banerjee
|
0561153b73 |
Add configurable largest-memtable flush policies to WriteBufferManager (#15047)
Summary: Motivation A WriteBufferManager can be shared by many RocksDB instances and column families. Under the existing policy, the DB whose writer first observes memory pressure flushes its own oldest mutable memtable. That DB might own only a small fraction of the reclaimable memory, so workloads with many DBs or CFs can generate many small flushes while the largest memory consumer remains untouched. Scanning every registered DB while holding process-wide or DB mutexes would improve selection accuracy but would serialize unrelated write paths and is not viable for deployments with thousands of DBs. Previous behavior kFlushOldest was the only behavior: the write-group leader that noticed WBM pressure synchronously selected the oldest eligible mutable memtable in its own DB. The WBM had no per-DB view of reclaimable memory and could not direct reclamation to another DB sharing the same budget. New behavior This change adds configurable WriteBufferFlushPolicy modes: - kFlushOldest preserves the existing behavior and remains the default. - kFlushLargest selects the largest eligible flushable column family in the writing DB. - kFlushLargestAcrossDBs selects the DB advertising the most reclaimable memory across all DBs sharing the WBM, then flushes its largest eligible CF. Atomic-flush DBs advertise and flush the aggregate eligible memory across their CFs. High-level implementation Each writable DB registers a lightweight FlushInitiator with the shared WBM. Per-DB accounting publishes two approximate values using atomics: total mutable memory and the largest flushable CF contribution, including eligible immutable memtables waiting for flush. Allocator reporting is batched, so the allocation hot path avoids registry scans and DB mutexes. A dedicated WBM coordinator thread maintains an approximate ranked snapshot of registered DBs. Refreshes are driven by accounting changes, threshold crossings, and periodic pressure checks. Ranking and O(number of DBs) work happen on that coordinator thread; writers only update counters and signal the coordinator. Registration lifetime is protected independently so DB close can safely drain an in-flight handoff. Threading and flush handoff The coordinator reserves and pins one ranked candidate at a time, without holding a registry lock while invoking the DB. The target DB uses TryLock on its DB mutex and rejects the handoff if it is closing, paused, stopped, stalled, already handling a WBM job, or no longer has flushable memory. Rejected candidates do not block consideration of the next DB. An accepted handoff queues one LOW-priority WBM job. The LOW worker joins the target DB write thread and revalidates eligibility before switching the memtable. The resulting ordinary flush is then scheduled through RocksDB's existing flush machinery on the HIGH-priority flush pool. Keeping the write-thread join off the requesting writer avoids making one DB's write path wait on another DB's mutex or write queue. Only one manager-wide remote handoff is active at a time. Completion or cancellation clears the handoff and requests a rerank. If memory reaches the hard limit while remote coordination is delayed, a rate-limited local fallback lets a writer seal a local memtable to preserve forward progress. Write stalls, shutdown, WaitForCompact, policy changes, and DB deregistration explicitly drain or reject outstanding handoffs. Compatibility The legacy constructor and kFlushOldest behavior remain unchanged unless a new policy is explicitly selected. There are no on-disk format changes. Pull Request resolved: https://github.com/facebook/rocksdb/pull/15047 Reviewed By: xingbowang Differential Revision: D112396106 fbshipit-source-id: 3049adc7034dd4e00d74d7d86e1d19004292d299 |
||
|
Peter Dillinger
|
8e1569b351 |
Lazy wide-column byte-range (partial) blob reads (#15067)
Summary: Builds on the lazy wide-column read API (DB::GetEntityLazy / DB::MultiGetEntityLazy) to serve true byte-range reads: resolving a sub-range of an uncompressed blob column now reads only the requested bytes from storage instead of fetching and slicing the whole column. This covers blobs in separate blob files and embedded (same-file) blobs in the SST, so a caller that knows a large value's layout can pull just the parts it needs. A partial read skips whole-record checksum verification (a strict sub-range cannot cover the checksum) and does not populate the blob cache (the cache entry holds the whole record). A compressed column, a whole-column read, a cache hit, or a read with LazyColumnReadRequest::force_verify still resolves (and verifies) the whole column; force_verify forces verification even when ReadOptions::verify_checksums is off. Mechanics: new range primitives mirror the whole-value read paths -- BlobFileReader::GetBlobRange, BlobSource::GetBlobRange, Version::GetBlobRange, and VersionBlobFetcherBase::FetchBlobRange for separate-file blobs; ReadSimpleGen2BlobRange, BlobSource::GetSimpleGen2BlobRange, and SameFileBlobReader::GetSameFileBlobRange (implemented by BlockBasedTable) for embedded blobs. ReadPathBlobResolver::ResolveColumnRange owns the partial-vs-whole decision and routes same-file references through the current SST's reader; the whole-column path also gained VersionBlobFetcherBase::FetchBlobForceVerify to honor force_verify. Observability: deferred resolve reads are attributed to a new Env::IOActivity::kLazyResolve (distinct from the kGetEntity / kMultiGetEntity of the initial entity read), and lazy resolution's storage reads are counted by new statistics rocksdb.blobdb.lazy.read.count / .read.bytes (all lazy reads) and rocksdb.blobdb.lazy.partial.read.count / .partial.bytes.saved (the byte-range subset and the bytes it avoided). The whole-column read sites gate recording on kLazyResolve so non-lazy reads are excluded. Also fixes a latent bug in LazyWideColumnsBatch::MultiResolve where an empty/default-constructed batch (null rep_) would accept a foreign standalone column (null owning_batch_rep_) through a nullptr == nullptr comparison instead of returning InvalidArgument. Follow-ups (in a local plan): unifying the blob-read API surface -- folding the whole/range method twins and the verify-policy flags (verify_checksums x force_verify) into one internal BlobReadRequest primitive (a range descriptor plus a 3-valued verify policy), which retires FetchBlobForceVerify and the whole/range method pairs; this is deferred for feature velocity and is not public-API affecting (the public force_verify bool stays). Marked in code with TODO(lazy-blob-resolution-cleanup) in db/blob/blob_fetcher.h and db/wide/read_path_blob_resolver.cc. Also cross-key coalescing (Phase 3) and async execution (Phase 4). Pull Request resolved: https://github.com/facebook/rocksdb/pull/15067 Test Plan: Extended db/wide/db_lazy_entity_test.cc: separate-file and embedded partial reads return the exact sub-span and read only the requested bytes; a partial read does not fill the blob cache and a cache hit slices with no I/O; a compressed column resolves whole and slices; force_verify does a full verified read and is detected via a tampered record even with verify_checksums off; the lazy statistics distinguish whole vs partial reads and are not touched by non-lazy reads; deferred reads carry kLazyResolve (verified with a FileSystem wrapper); and a regression test for the empty-batch ownership check. Added an IOActivityToString case in env/env_test.cc and registered the new tickers in the statistics name map. Crash test: extended the db_stress lazy-vs-eager differential (db_stress_tool/db_stress_test_base.cc) to fuzz per-column byte ranges. Each resolved column now randomly picks a whole-column read or a partial (offset, length) sub-range -- occasionally with an offset past the end or a length beyond the remainder to exercise clamping -- for both GetEntityLazy and MultiGetEntityLazy; when an eager reference is available the resolved bytes are compared against the same clamped sub-range of it. No new flags (reuses --lazy_entity_read_one_in); force_verify fuzzing remains a follow-up. Reviewed By: xingbowang Differential Revision: D115130388 Pulled By: pdillinger fbshipit-source-id: 454368dea8fae069d9de649c2c51611469d3c809 |
||
|
Josh Kang
|
c64f682946 |
Fix IoUringBackend crash in ReadIOExecutor worker threads (#15020)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15020 Each read executor worker creates its own `IoUringBackend`, and Folly's I/O thread pool creates its worker threads and EventBases lazily. A successful calling-thread probe or `DB::Open()` therefore does not validate every worker. A later read can start a new worker; previously, an `IoUringBackend` initialization exception on that worker could escape startup and terminate the process. The previous 512-submit setup also allowed Folly's capacity retry to reduce the completion queue to 512 while retaining a 1024-entry submission queue, which the kernel rejects with `EINVAL`. Use a 256-submit limit so the 512-entry retry remains valid. Configure the shared read executor during `DB::Open()` through `DBOptions::read_io_executor_threads`, defaulting to one thread. The backend factory catches per-worker io_uring initialization failures and uses Folly's default backend. Each worker's thread-local availability marker starts false and is set true only after `IoUringBackend` construction succeeds. When construction fails, the marker remains false and `SubmitReadAsync()` performs the read synchronously. The executor's maximum thread capacity can be raised or lowered; `db_stress` requests eight threads for async DB APIs. To expose this degradation, `FSRandomAccessFile::SubmitReadAsync()` now reports whether the asynchronous path was used. `RandomAccessFileReader` increments the `FILE_SUBMIT_ASYNC_READ_FALLBACK` ticker for every submission that falls back to synchronous I/O, making the fallback visible through the DB's `Statistics`. Reviewed By: xingbowang Differential Revision: D114031018 fbshipit-source-id: 1b7f6c1cea693e5dd743ea1ada0fca1cf757b3f9 |
||
|
Josh Kang
|
7247edbf0a |
Add opt-in stats for async RocksDB reads (#15017)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15017 ## Summary RocksDB async `GetAsync` and `MultiGetAsync` callbacks now default to a completion-only path. Callbacks can override `EnableStats()` when they need request-specific `PerfContext` or `IOStatsContext`; otherwise RocksDB avoids creating a coroutine stats context and returns null stats pointers. Callbacks must not initiate another async read from `OnComplete()`. RocksDB stats normally use thread-local storage, which does not compose with interleaved coroutine reads. A stats-enabled request captures only the caller's configuration: perf level, per-level perf enablement, and whether io stats are disabled. It then starts with empty request contexts. `onSet()` loads that request's contexts into TLS, while `onUnset()` saves the accumulated contexts back into request data. On resume, the same contexts are loaded again. This isolates request counters without preserving or restoring pre-existing executor TLS state. The synchronous fallback similarly resets TLS before executing a stats-enabled read. `PerfContext` and `IOStatsContext` gain `Merge()` support for caller-side aggregation. `PerfContext` move construction and assignment now transfer ownership of per-level state rather than deep-copying it, avoiding allocations during coroutine context handoffs. A fbcode-wide audit found no caller that reads a moved-from `PerfContext`. `db_bench` opts in only for coroutine jobs that print per-job perf context and merges every completed request's callback stats into the job aggregate. ## Example: Two Async Gets on One Thread Both callbacks enable stats. Each request starts with empty counters and carries its contexts across suspensions. ```text Shared executor thread | | onSet(A): load A's empty contexts into TLS | A records 3 reads | onUnset(A): save TLS contexts back into A | | onSet(B): load B's empty contexts into TLS | B records 4 reads | onUnset(B): save TLS contexts back into B | | onSet(A): load A's contexts containing 3 reads | A records 2 more reads | callback A receives block_read_count = 5 | | onSet(B): load B's contexts containing 4 reads | callback B receives block_read_count = 4 ``` Isolation comes from replacing TLS with the active request's saved contexts on every `onSet()`. ### Timers Existing wall-clock timers remain in coroutine frames, so timers spanning `co_await` include suspended time and measure end-to-end latency. Request-level `get_cpu_nanos` is suspension-aware: it starts in `onSet()` and stops in `onUnset()` or `Finalize()`, charging only time while the request is active. Nested CPU timers remain disabled because normal RAII CPU timers would also charge executor work performed while their coroutine is suspended. `MultiGetAsync` fanout additionally requires per-child timer state to avoid double counting. ## Benchmarks Benchmarked to confirm no performance regressions when perf is disabled, and also to see the overhead of enabling perf. ```bash db_bench --benchmarks=<readrandomcoroutine|multireadrandomcoroutine> --db=<db_path> --coro_jobs_per_thread=10 --use_existing_db=true --num=2000000 --value_size=1024 --compression_type=none --threads=8 --batch_size=16 --multiread_batched=true --use_direct_reads=true --cache_size=1428076626 --reads=320000 --perf_level=<1|3> --seed=<seed> ``` | Workload | Before change | Perf level 1 | Before -> 1 | Perf level 3 | 1 -> 3 | | -------- | ------------- | ------------ | ----------- | ------------ | ------ | | Async Get | 475.6K ops/s | 476.7K ops/s | +0.2% | 470.5K ops/s | -1.3% | | Async MultiGet | 567.3K ops/s | 567.4K ops/s | +0.0% | 552.2K ops/s | -2.7% | Reviewed By: xingbowang Differential Revision: D112634874 fbshipit-source-id: 94e240aeb1496fa12dac995b5aa5a9c3543cc5fc |
||
|
Ranjan Banerjee
|
f5c0c3e274 |
Record CheckpointEngine copy bytes under checkpoint tickers (#14997)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/14997 `CheckpointEngine`'s shared `CopyEngine` recorded copied bytes under the `BACKUP_READ_BYTES` / `BACKUP_WRITE_BYTES` tickers (the `CopyEngineOptions` defaults), conflating checkpoint I/O with backup I/O in statistics. This adds dedicated `CHECKPOINT_READ_BYTES` / `CHECKPOINT_WRITE_BYTES` tickers and points `CheckpointEngineImpl`'s `CopyEngineOptions` at them, so checkpoint copies report under their own metric names. `BackupEngine` is unchanged (still uses the backup-ticker defaults). Addresses review feedback that the shared `CopyEngine` should not hardcode backup metric names for non-backup callers. The new tickers are intentionally not mirrored in the Java bindings (`TickerType.java` / `portal.h`): the signed-`jbyte` id space there is exhausted, so they remain C++-only, documented inline in `statistics.h`. Reviewed By: xingbowang Differential Revision: D113469742 fbshipit-source-id: 2a6a7ef04c939a1240f1c48c7bcaba8b2e7a1c10 |
||
Xingbo Wang
|
03c3c090fb |
Measure IODispatcher async read latency (#15001)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15001 Add IODispatcher-specific histograms for async read observed completion latency, `FileSystem::Poll` wait time, and submit-to-consumer-poll prefetch lead time. Record submit timestamps on async read state, report completion latency from the callback, and report poll wait/prefetch lead when the block consumer polls the handle. Keep Java histogram byte mappings stable by assigning the new histogram IDs after the existing range. Add a controlled async filesystem test that verifies one deferred async read records all three histograms. Address Infer feedback by iterating `async_state->block_indices` directly when removing async IO map entries. Keep `PollAndProcessAsyncIO()` holding an owning `shared_ptr<AsyncIOState>` while erasing those entries so the coalesced request state cannot be destroyed mid-loop. Format `util/io_dispatcher_imp.cc` and `util/io_dispatcher_test.cc` with clang-format 21.1.2 to fix the format signal. Reviewed By: pdillinger, anand1976 Differential Revision: D113549401 fbshipit-source-id: dafc272c9252e1baccf404f681d33f76890ce0b2 |
||
Jay Huh
|
dcb5c2bf53 |
Restore MultiScan prefetch stats dropped in the IODispatcher refactor (#14991)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/14991 D91705195 ("Replace Prefetch Logic in BlockBasedTableIterator with IODispatcher") moved MultiScan block prefetching out of BlockBasedTableIterator and into the IODispatcher. The IO-level RecordTick/RecordInHistogram calls lived in the deleted prefetch code, and the IODispatcher was not given equivalent instrumentation, so these tickers were still defined but never populated (they published as absent): - MULTISCAN_BLOCKS_PREFETCHED - MULTISCAN_BLOCKS_FROM_CACHE - MULTISCAN_PREFETCH_BYTES - MULTISCAN_IO_REQUESTS - MULTISCAN_IO_COALESCED_NONADJACENT - MULTISCAN_BLOCKS_PER_PREPARE (histogram) The PREPARE_*, SEEK_ERRORS, and PREFETCH_BLOCKS_WASTED tickers were unaffected (recorded on the prepare/teardown path, which was not replaced). Fix: - ReadSet now exposes generic, caller-agnostic prefetch counters (GetNumBlocksPrefetched, GetNumIORequests, GetPrefetchBytes, GetNumCoalescedNonadjacent), incremented once per block in DispatchPrefetch -- the single choke point for both immediate and memory-deferred dispatch. Counting at dispatch time (not completion) keeps async and deferred prefetches accounted for without double counting the async->sync fallback. - PrepareIORequests now outputs prefetch_bytes (sum of request len) and num_coalesced_nonadjacent (within-threshold gap count) while building coalesced requests, per review feedback to avoid the extra pass over read_reqs/coalesced_block_indices at former line 836. DispatchPrefetch now gets stats directly from PrepareIORequests. - The MULTISCAN_* ticker names stay at the caller: MultiScanIndexIterator's destructor records the aggregate counters at end-of-scan (alongside the existing PREFETCH_BLOCKS_WASTED), so async/deferred reads that complete during iteration are included. MULTISCAN_BLOCKS_PER_PREPARE is recorded in BlockBasedTableIterator::Prepare. This keeps the IODispatcher a generic mechanism rather than leaking a caller's metric names into it. - Also plumb the same metrics into PerfContext for per-operation visibility. Reviewed By: xingbowang Differential Revision: D113356436 fbshipit-source-id: 660e9755d6b1b9aecccd112df94eb6838061b298 |
||
|
Andrew Chang
|
0745290ca6 |
Add OpenAndCompact DB open latency histogram (#14986)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/14986 Why: We want to measure the DB open time during offloaded compactions. What: Add a histogram for the DB open time in `DB::OpenAndCompact()`. Reviewed By: anand1976 Differential Revision: D113136652 fbshipit-source-id: f8b2d74bcda3506f7b9ac80fa3e50a172deafc5e |
||
Xingbo Wang
|
de7b8e8f7d |
Add RocksDB flush reason and atomic flush request metrics (#14898)
Summary:
This PR adds flush observability for debugging write stalls and flush bottlenecks.
Changes include:
- Add flush reason tickers for physical flush jobs:
- rocksdb.flush.reason.write_buffer_full
- rocksdb.flush.reason.write_buffer_manager
- rocksdb.flush.reason.memtable_max_range_deletions
- Add flush memtable size histograms:
- rocksdb.flush.memtable.memory.bytes
- rocksdb.flush.memtable.total.data.size.bytes
- rocksdb.flush.write_buffer_full.memtable.memory.bytes
- rocksdb.flush.write_buffer_manager.memtable.memory.bytes
- Add FlushReason::kMemtableMaxRangeDeletions and propagate it from memtables when memtable_max_range_deletions triggers a flush.
- Preserve atomic flush as a single request-level reason. For atomic flush, the reason is derived only from CFs that scheduled the flush, not every CF selected into the atomic flush cut.
- Add atomic flush request-level counters:
- rocksdb.atomic_flush.request.reason.write_buffer_full
- rocksdb.atomic_flush.request.reason.write_buffer_manager
- rocksdb.atomic_flush.request.reason.memtable_max_range_deletions
- rocksdb.atomic_flush.request.reason.other
- Keep existing per-CF flush job reason counters unchanged. Under atomic flush, rocksdb.flush.reason.* still counts physical CF flush jobs, while rocksdb.atomic_flush.request.reason.* counts one logical atomic flush request.
- Wire the new flush reason, ticker, and histogram types through Java and JNI bindings.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/14898
Test Plan: CI
Reviewed By: anand1976
Differential Revision: D110214337
Pulled By: xingbowang
fbshipit-source-id: 4a445c4175d2e09ff5edbe66fde81fa00bb9e8da
|
||
|
Xingbo Wang
|
75c1ffd4e3 |
Add RocksDB rate limiter I/O usage metrics (#14884)
Summary: Add rate limiter statistics that make it easier to diagnose whether RocksDB background IO is being throttled by the GenericRateLimiter. New ticker counters: - rocksdb.rate.limiter.bytes.read/write: bytes granted by the limiter, split by operation type. - rocksdb.rate.limiter.requests.read/write: requests granted by the limiter, split by operation type. - rocksdb.rate.limiter.delayed.requests.read/write: requests that actually had to wait for a future refill because available tokens were exhausted. - rocksdb.rate.limiter.total.wait.micros.read/write: cumulative wait time for delayed requests. New histograms: - rocksdb.rate.limiter.wait.micros.read/write: per-request wait latency for delayed requests, enabling percentile debugging. GenericRateLimiter now overrides the OpType-aware Request() path so these stats preserve the configured limiter mode. The legacy three-argument Request() path is treated as write IO for compatibility. Existing NUMBER_RATE_LIMITER_DRAINS is left intact, but the new delayed request counters are the clearer signal for out-of-token throttling because one delayed request can observe multiple drain intervals. Also update the Java ticker and histogram mappings so Java consumers can access the new statistics. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14884 Test Plan: - make format-auto - make -j16 rate_limiter_test statistics_test - ./rate_limiter_test - ./statistics_test - make check-sources Reviewed By: anand1976 Differential Revision: D109694030 Pulled By: xingbowang fbshipit-source-id: e92756788ba629f25655958c7beb91099184cda4 |
||
|
Josh Kang
|
1ede57e5b0 |
Add file ingestion histograms (#14836)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/14836 Adds a public `Statistics` histogram, `INGEST_EXTERNAL_FILE_TIME` (`"rocksdb.ingest.external.file.micros"`), recording the end-to-end latency in microseconds of each `IngestExternalFile(s)` call. Ingestion timing was previously only available through per-thread `perf_context` counters, which require setting a `PerfLevel` and are not aggregated, so there was no process-wide latency distribution (p50/p99/max) for dashboards. It is recorded with an RAII `StopWatch` at the top of `DBImpl::IngestExternalFiles` -- one sample per call (not per column family), covering all return paths. It is null-safe and self-gating on the stats level, so there is no cost when statistics are off, and ingestion is not a hot path. Java bindings are kept in sync per the `statistics.h` requirement; the C API needs no change. Reviewed By: xingbowang Differential Revision: D107721260 fbshipit-source-id: 0705f9e0f7392329a7bcbdbd9f3afd34594d20bb |
||
Michael Huang
|
9270dc8149 |
Add blob_cache_read_byte perf context counter (#14792)
Summary: Added blob_cache_read_byte in the rocksdb::PerContextBase to expose the blob cache read bytes when blob cache is enabled. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14792 Reviewed By: xingbowang Differential Revision: D106528572 Pulled By: mikechuangmeta fbshipit-source-id: 555b2f01785bb819e62ed834ee45f0436dfb2875 |
||
|
Xingbo Wang
|
21723bbbef |
Add async WAL precreation
Summary: - Add experimental immutable `DBOptions::async_wal_precreate` to reserve and open one future WAL on a background HIGH-priority task, with sanitization that disables the optimization when WAL recycling is configured. - Split WAL creation into open/preallocate and start phases so `SwitchMemtable()` can consume a prepared WAL after writing normal WAL metadata, wait for in-flight precreation, fall back to synchronous creation, and delete an unstarted prepared WAL on start failure. - Keep WAL numbering, close, recovery, and read-only open safe for empty future WAL files left by async precreation; `error_if_wal_file_exists=true` now rejects non-empty WALs while tolerating empty WALs. - Add public option plumbing for the C API, options parsing/stringification, random option testing, `db_bench`, `db_stress`, and crash-test configuration. - Add WAL precreate statistics counters plus Java `TickerType`/JNI mappings, and update C++, C, and Java read-only-open documentation for the empty-WAL behavior. - Add focused WAL/option/C/Java tests for async precreate ready/wait/failure/recovery paths, read-only WAL detection, option sanitization, and API plumbing, plus write-flow docs and unreleased history entries for the new feature and behavior change. PR https://github.com/facebook/rocksdb/pull/14738 Reviewed By: pdillinger Differential Revision: D105020559 fbshipit-source-id: 5059b424702e021abb8de65ceeb6d3b975280ffc |
||
Danny Chen
|
2e7cf42cda |
Add MANIFEST_VALIDATION_FAILURE_COUNT statistic (#14657)
Summary: CONTEXT: The manifest validation on close feature (verify_manifest_content_on_close) detects corruption but does not increment any statistics counter, making it harder to monitor in production. WHAT: Add a new ticker MANIFEST_VALIDATION_FAILURE_COUNT that is incremented each time content validation detects manifest corruption during DB::Close(). The counter fires per corruption detection, so it can increment up to 2 times per close (once on initial check, once after rewrite attempt). Updated all existing manifest validation tests to verify the counter value. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14657 Test Plan: - All 7 manifest validation tests pass with new stat assertions - 5x repeat with COERCE_CONTEXT_SWITCH=1 shows no flakiness - Full version_set_test suite (212 tests) passes Reviewed By: anand1976 Differential Revision: D102404260 Pulled By: dannyhchen fbshipit-source-id: 21a0aa1ad8de12a935caf5642e41ccf2a47b46d9 |
||
Anand Ananthabhotla
|
1cc216a45b |
Fast file open: retrieve, persist, and pass file open metadata (#14596)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/14596 When DBOptions::fast_sst_open is enabled, RocksDB retrieves opaque file system metadata for SST files after flush, compaction, and external file ingestion via FSRandomAccessFile::GetFileOpenMetadata(). This metadata is persisted in the MANIFEST using a new forward-compatible NewFileCustomTag (kFileOpenMetadata = 17), and passed back to the file system via FileOptions::file_metadata on subsequent file opens. This accelerates DB open time on remote storage systems by allowing the file system to skip expensive metadata RPCs. The feature is gated by DBOptions::fast_sst_open (default false). Everything works seamlessly regardless of the option setting, file metadata support, or presence/absence of the metadata in the MANIFEST. The MANIFEST change is backward compatible - older RocksDB versions safely ignore the new tag. Reviewed By: xingbowang Differential Revision: D100220973 fbshipit-source-id: f52de9dd853a50653b3297ab4a37a868fe41cc04 |
||
|
Josh Kang
|
3f51c0a185 |
Convert sequential single deletes into range tombstones (#14448)
Summary:
Add a read-path optimization that converts contiguous point tombstones into range tombstones during forward/reverse iteration. When a configurable threshold of consecutive point deletions (kTypeDeletion, kTypeDeletionWithTimestamp, kTypeSingleDeletion — with no live keys between them) is detected, a range tombstone covering `[first_tombstone_key, next_live_key)` is inserted into the active mutable memtable. This benefits future iterators by enabling efficient skipping via range tombstone fragmentation.
If there is a memtable switch during the read iteration, then the range deletion entry is discarded.
The inserted range tombstones are logically redundant (they don't delete anything that isn't already deleted by point tombstones), skip WAL (they're a derived optimization regenerated by future reads on crash), and use the max tombstone sequence number so they don't interfere with newer writes.
## Key changes
- **New option `min_tombstones_for_range_conversion`** (`AdvancedColumnFamilyOptions`): Threshold of contiguous point tombstones before converting to a range tombstone. Default 0 (disabled). Dynamically changeable via `SetOptions()`.
- **`DBIter` tracking logic** (`db/db_iter.cc`): Tracks contiguous tombstones during `FindNextUserEntryInternal()` (forward) and `PrevInternal()` (reverse). When a live key terminates a run that meets the threshold, `MaybeInsertRangeTombstone()` inserts `[first_tombstone, live_key)` into the active memtable.
- **`FindValueForCurrentKey` `found_visible` output** (`db/db_iter.cc`): Distinguishes "key deleted at this snapshot" from "no visible entries" so reverse tracking doesn't treat post-snapshot keys as tombstones.
- **`IterKey::Swap()`** (`db/dbformat.h`): Efficiently tracks reverse tombstone run end keys without extra allocations.
- **`MemTable::AddLogicallyRedundantRangeTombstone()`** (`db/memtable.cc`): Concurrent-safe range tombstone insertion into the active memtable. Range tombstone skiplist always uses concurrent inserts.
- **`ConstructFragmentedRangeTombstones` race fix** (`db/db_impl/db_impl_write.cc`): Moved after `MarkImmutable()` to prevent lost entries.
- **`MarkImmutable` ordering fix** (`db/memtable_list.cc`): Called before `current_->Add()` to close a race window.
- **Prefix filter awareness** (`db/db_iter.cc`): Tombstone tracking scoped to the seek prefix when prefix filtering is active. See dedicated section below.
- **Transaction awareness** (`db/db_iter.cc`): Tombstones with `seq > snapshot` excluded from tracking. `min_uncommitted` guard uses `insert_seq` (which may be bumped to `earliest_seq`) instead of `range_tomb_max_seq_`. See dedicated section below.
- **Duplicate range check**: Skips insertion if the memtable already covers `[start, end)`.
- **New statistics**: `READ_PATH_RANGE_TOMBSTONES_INSERTED` and `READ_PATH_RANGE_TOMBSTONES_DISCARDED`.
- **Memtable MultiGet batch lookup** (`memtable/inlineskiplist.h`, `db/memtable.cc`): `InlineSkipList::MultiGet()` with cached search path ("finger") for sorted key lookups.
- **New option `memtable_batch_lookup_optimization`** (`AdvancedColumnFamilyOptions`): Enables batch lookup for memtable MultiGet. Default false. Immutable.
## Deciding Range Tombstone Seqno
- The range tombstone is inserted with `insert_seq = max(range_tomb_max_seq_, earliest_seq)` where `range_tomb_max_seq_` is the maximum sequence number across all point tombstones in the contiguous run, and `earliest_seq` is the memtable's earliest sequence number. This preserves the memtable's `earliest_seqno_` invariant.
- If the iterator's snapshot sequence (`sequence_`) predates the memtable's `earliest_seq`, insertion is skipped entirely to avoid unintentionally covering entries between `sequence_` and `earliest_seq`.
## ConstructFragmentedRangeTombstones Race Fix
- `MarkImmutable()` and `ConstructFragmentedRangeTombstones()` are now called before `mutex_.Lock()` in `SwitchMemtable`, keeping this work outside the DB mutex. `MarkImmutable()` blocks concurrent `AddLogicallyRedundantRangeTombstone()` calls via `immutable_mutex_`, ensuring no range tombstones are inserted after the fragmented list is built. `MarkImmutable()` is idempotent, so `MemTableList::Add()` calling it again inside the mutex is harmless.
## Prefix Filter Safety
- When prefix filtering is active, the BBTI bloom filter may reject SST files outside the seek prefix, but the memtable (no bloom filter) returns keys across prefix boundaries. Tombstone tracking is scoped to the seek prefix so that converted range tombstones cannot cover live keys hidden in filtered files.
- `total_order_seek=true` disables prefix filtering — all files are visible, so tombstones safely span prefix boundaries.
- **Behavior change**: Seeking to an out-of-domain key with `total_order_seek=false` now treats it as total-order (prefix_ not set). When `prefix_same_as_start=true`, iterating past an out-of-domain key cleanly invalidates the iterator instead of calling `Transform()` on it (which was UB in release builds with `FixedPrefixTransform`). This is a requirement because an incorrect iterator scan could lead to a range tombstone covering a live key.
## Transaction Support
- Tombstones written by the transaction's own uncommitted writes (sequence > snapshot) are now excluded from contiguous tombstone tracking entirely at the tracking site in `FindNextUserEntryInternal()` and `PrevInternal()`. Previously, tracking relied on a `min_uncommitted` check at insertion time, but this was insufficient — a transaction's own Delete with `seq > snapshot` could extend a run of committed tombstones, and the resulting range tombstone would cover data visible to other snapshots.
- The fix skips any tombstone with `ikey_.sequence > sequence_` during tracking. If a transaction-owned tombstone appears mid-run, it flushes the accumulated committed run first, then resets tracking. This ensures only tombstones visible to the current snapshot are ever converted.
- Both WritePrepared and WriteUnprepared transactions are supported with dedicated test coverage:
- **WritePrepared**: When tombstones are committed before `Prepare()`, their seqnos are below `min_uncommitted` and insertion proceeds safely. When `Prepare()` happens first, tombstone seqnos exceed `min_uncommitted` and insertion is blocked.
- **WriteUnprepared**: Multiple unprepared batches with different seqno ranges are handled correctly. Own transaction Deletes that extend a committed tombstone run block insertion of the entire run. After rollback, data correctness is verified.
## UDT support
- When user-defined timestamps (UDT) are enabled, keys include an 8-byte timestamp suffix. The comparator, Put/Delete APIs, and ReadOptions all require timestamps.
- Forward exhaustion with UDT: `iterate_upper_bound_` is a plain user key without a timestamp suffix. It is padded with min timestamp via `AppendKeyWithMinTimestamp()` so it sorts after all entries with this user key, preserving the exclusive bound semantics.
- Reverse exhaustion with UDT: The end key comes from either the previous live key (which already has a proper timestamp suffix) or the seek target set by `SetSavedKeyToSeekForPrevTarget()` (which appends a timestamp via `SetInternalKey(..., timestamp_ub_)` and `UpdateInternalKey(..., ts)`). In both cases, the end key is already properly timestamped, so no additional padding is needed unlike forward exhaustion.
- Contiguous tombstone detection works correctly with UDT because the underlying `kTypeDeletionWithTimestamp` entries are tracked the same way as `kTypeDeletion`.
## Concurrent Iterators
- Should concurrent iterators happen to read the range at the same time, both will produce the same range and seqno entry. Only one will be accepted by the skip list and the others will be rejected. Future iterators will read the range and not even attempt to insert the range.
- There is nothing preventing similar ranges from being inserted however. Two iterators can produce overlapping ranges, but this protection would be complicated to implement and there is no evidence that it is a likely scenario yet.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/14448
Test Plan:
- **Unit tests** (`db/db_iterator_test.cc`): `ReadPathRangeTombstoneTest` parameterized by forward/reverse with cases for basic insertion, non-contiguous (below threshold), memtable switch, exhausted iterator with/without bounds, direction change, mixed Delete/SingleDelete, single-delete-only runs, snapshot predating memtable, block cache tier incomplete, skip when covered by existing range, UDT basic scan, UDT exhaustion, prefix filter cross-prefix scan (`PrefixFilterCrossPrefixScanCoversLiveKey` with default/total_order_seek/prefix_same_as_start variants), stale ikey from forward-then-reverse scan (`StaleIkeyFromForwardThenReverse`), and reseek stale ikey (`ReseekStaleIkey`).
- **Concurrency test** (`db/db_test2.cc`): `DBTestConcurrentRangeTombstoneConversions` parameterized by `(allow_concurrent_memtable_write, min_tombstones_for_range_conversion)` with mixed writers, deleters, range deleters, and concurrent forward/reverse readers.
- **Transaction tests** (`utilities/transactions/write_prepared_transaction_test.cc`, `write_unprepared_transaction_test.cc`): Tests for WritePrepared (insertion allowed when tombstones committed before prepare, blocked when after; seqno bump shadowing prepared writes `RangeTombstoneSeqnoBumpShadowsPreparedWrite`) and WriteUnprepared (multiple batches, extended visibility with CalcMaxVisibleSeq, own deletions with rollback).
- **IterKey::Swap tests** (`db/dbformat_test.cc`): `IterKeySwapTest` parameterized over `(key_len, copy, use_secondary)` × 2 covering all inline/heap/pinned/secondary combinations.
- **InlineSkipList MultiGet tests** (`memtable/inlineskiplist_test.cc`): Basic, exact matches, empty, single key, randomized validation against `std::set::lower_bound`, duplicate keys with callback walk, and concurrent MultiGet with read-after-write consistency.
- **Memtable MultiGet tests** (`db/db_basic_test.cc`): Batch lookup, overwrite, flush, merge, disabled by default, paranoid checks, and snapshot tests.
- **Stress test coverage**: `min_tombstones_for_range_conversion` and `memtable_batch_lookup_optimization` options added to `db_crashtest.py` and `db_stress` flags.
- `make check` passes all tests.
### Benchmark results
Tombstones scattered randomly in clusters via `seek_nexts_to_delete` for realistic workloads.
**DB Setup A (scattered deletes, 100 per seek)**:
```
# Step 1: Create and compact
./db_bench --benchmarks=fillseq,compact --seed=1 --compression_type=none --num=1000000 --db=<DB>
# Step 2: Scatter tombstones (5000 seeks × 100 deletes ≈ 500k tombstones)
./db_bench --benchmarks=seekrandom,flush --seed=1 --compression_type=none --num=2000 \
--seek_nexts=0 --seek_nexts_to_delete=100 --use_existing_db=1 --threads=1 --db=<DB>
```
**DB Setup A2 (scattered deletes, 8 per seek)**:
```
# Step 1: Create and compact
./db_bench --benchmarks=fillseq,compact --seed=1 --compression_type=none --num=1000000 --db=<DB>
# Step 2: Scatter tombstones (5000 seeks × 8 deletes ≈ 40k tombstones)
./db_bench --benchmarks=seekrandom,flush --seed=1 --compression_type=none --num=2000 \
--seek_nexts=0 --seek_nexts_to_delete=8 --use_existing_db=1 --threads=1 --db=<DB>
```
**DB Setup B (no deletes)**:
```
./db_bench --benchmarks=fillseq,compact --seed=1 --compression_type=none --num=1000000 \
[--key_size=100] --db=<DB>
```
**Read workload** (same for all):
```
./db_bench --benchmarks=seekrandom --seek_nexts=100 --threads=8 \
--reverse_iterator={true,false} --seed=1 --use_existing_db=1 \
--compression_type=none --num=1000000 --duration=10 \
--disable_auto_compactions \
[--key_size=100] [--min_tombstones_for_range_conversion=X] --db=<DB_COPY>
```
Each workload averaged over 3 runs.
**Table 1: seekrandom forward, scattered deletes (2000 seeks × 100 deletes/seek)**
| Variant | avg ops/s | % vs main |
|---------|-----------|-----------|
| main | 2,895 | - |
| threshold=0 | 2,869 | -0.9% |
| threshold=8 | 287,334 | +9,824% |
**Table 2: seekrandom reverse, scattered deletes (2000 seeks × 100 deletes/seek)**
| Variant | avg ops/s | % vs main |
|---------|-----------|-----------|
| main | 544 | - |
| threshold=0 | 548 | +0.7% |
| threshold=8 | 206,491 | +37,860% |
**Table 3: seekrandom forward, scattered deletes (2000 seeks × 8 deletes/seek)**
| Variant | avg ops/s | % vs main |
|---------|-----------|-----------|
| main | 194,049 | - |
| threshold=0 | 195,703 | +0.9% |
| threshold=8 | 310,740 | +60.1% |
**Table 4: seekrandom reverse, scattered deletes (2000 seeks × 8 deletes/seek)**
| Variant | avg ops/s | % vs main |
|---------|-----------|-----------|
| main | 63,854 | - |
| threshold=0 | 69,266 | +8.5% |
| threshold=8 | 218,101 | +241.6% |
**Table 5: seekrandom forward, no deletes (regression check)**
| Variant | key=16B avg ops/s | % vs main | key=100B avg ops/s | % vs main |
|---------|-------------------|-----------|---------------------|-----------|
| main | 330,901 | - | 236,048 | - |
| threshold=0 | 328,398 | -0.8% | 238,055 | +0.9% |
| threshold=8 | 332,539 | +0.5% | 233,776 | -1.0% |
**Table 6: seekrandom reverse, no deletes (regression check)**
| Variant | key=16B avg ops/s | % vs main | key=100B avg ops/s | % vs main |
|---------|-------------------|-----------|---------------------|-----------|
| main | 261,445 | - | 192,177 | - |
| threshold=0 | 265,020 | +1.4% | 191,616 | -0.3% |
| threshold=8 | 250,881 | -4.0% | 189,239 | -1.5% |
Reviewed By: xingbowang
Differential Revision: D96203950
Pulled By: joshkang97
fbshipit-source-id: 06ba66ebde3c355f04671d1e681f1b1586e8751d
|
||
Jialiang Tan
|
65e77282ce |
Add block_decompress_count to PerfContext (#14557)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/14557 RocksDB's PerfContext tracks block_decompress_time but has no counter for the number of block decompressions. The global Statistics ticker NUMBER_BLOCK_DECOMPRESSED exists but is not accessible through PerfContext, which is the thread-local, per-operation metric system used by benchmarks. Add block_decompress_count as a new PerfContext counter (gated at kEnableCount level) incremented in DecompressBlockData alongside the existing NUMBER_BLOCK_DECOMPRESSED ticker. This enables benchmarks and applications to observe how many blocks required decompression per operation, complementing the existing block_read_count. Reviewed By: anand1976 Differential Revision: D99233514 fbshipit-source-id: 40a68c1d9321f560cebdb7c30a544a0c62ae64f0 |
||
|
Yoshinori Matsunobu
|
b23fc77aca |
Add per-block-type block read byte perf counters (#14473)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/14473 Add separate PerfContext byte counters for data, index, filter, compression-dictionary, and metadata block reads while preserving block_read_byte as the aggregate. Wire the new counters through the block fetch and multi-read data block paths, expose them via the C perfcontext API, and extend table tests to verify byte attribution and that the classified counters sum back to block_read_byte. Reviewed By: xingbowang Differential Revision: D97333746 fbshipit-source-id: 3411844d7fa9c76c9ff28af477b3a72a5d6e5d9b |
||
|
Josh Kang
|
42eff8b632 |
Add new heauristic 'num_collapsible_entry_reads_sampled' (#14434)
Summary: Add per-file sampling of "collapsible" entry reads (single deletions, merges, and kNotFound results) that may later be used to help inform read-triggered compactions. This is a better metric than `num_reads_sampled` as it is more targeted towards reads that could be avoided via compaction. The existing behavior of `num_reads_sampled` is that reads only gets sampled on iterator creation for a file. It is problematic because next/prev() calls are not sampled, nor are additional seeks(). This PR moves sampling to per-seek/next granularity within `LevelIterator` and adds a new `num_collapsible_entry_reads_sampled` counter that tracks how often a file serves entries that could be eliminated by compaction. Note only L1+ files have iterator seeks/nexts/prevs sampled. Introducing this at L0 would require wrapping table reader iterators, introducing a performance cost. ## Key changes - **New counter `num_collapsible_entry_reads_sampled`** in `FileSampledStats` tracks sampled reads that encounter deletions, single deletions, merges, or kNotFound results in both Get and Iterator paths. - **Moved sampling from file-open to per-operation** in `LevelIterator`: sampling now happens in `SampleRead()` called from `Seek()`, `SeekForPrev()`, `SeekToFirst()`, `SeekToLast()`, `Next()`, `NextAndGetResult()`, and `Prev()`. The `should_sample` parameter was removed from `LevelIterator`'s constructor. - **Differentiated sampling rate for Next() vs Seek()**: `should_sample_file_read_next()` uses a 64x lower sampling rate (`kFileReadSampleRate * 64`) since Next() is cheaper than Seek() and called more frequently. - **Collapsible tracking in Get path**: `Version::Get()` now increments the collapsible counter when `GetContext::State()` is `kNotFound`, `kMerge`, or `kDeleted`. - **Collapsible tracking in MultiGet path**: `MultiGetFromSST` also increments the collapsible counter for the same states. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14434 Test Plan: - Added new DB tests for both num_reads_sampled and num_collapsible_entry_reads_sampled ### Benchmark results (readrandom, readseq) Setup: 1M keys, 16-byte keys, 100-byte values, no compression, fillrandom+compact | Benchmark | Params | ops/s (main) | ops/s (feature) | % change | |------------|--------------------|-------------|--------------------------|----------| | readrandom | seed=1, threads=1 | 387,194 | 389,449 | +0.6% | | readseq | seed=1, threads=1 | 5,598,371 | 5,572,975 | -0.5% | No meaningful performance regression observed — differences are within run-to-run noise. Reviewed By: xingbowang Differential Revision: D95613793 Pulled By: joshkang97 fbshipit-source-id: 9dd09c9b7527b148424bde5686f4157c7a9e1214 |
||
|
Josh Kang
|
3ad23b2d94 |
Support automated interpolation search (#14383)
Summary: Add automatic per-block interpolation search selection (`kAuto` mode) for index blocks. During SST construction, each index block's key distribution is analyzed using the coefficient of variation (CV) of gaps between restart-point keys. Blocks with uniformly distributed keys are flagged via a new bit in the data block footer, and at read time, `kAuto` resolves to interpolation search for uniform blocks and binary search otherwise. ## Key changes - **New `BlockSearchType::kAuto` enum value**: Resolves per-block at read time to either `kInterpolation` or `kBinary` based on the block's uniformity flag. Falls back to `kBinary` on older versions that don't recognize it. - **Write-path uniformity analysis**: `BlockBuilder::ScanForUniformity()` uses Welford's online algorithm to incrementally compute the CV of key gaps at restart points. The result is stored in a new bit (bit 30) of the data block footer's packed restart count. - **New table option `uniform_cv_threshold`** (default: -1 `disabled`): Controls how strict the uniformity check is. Set to negative to disable. Exposed in C++, Java (JNI), and `db_bench`. - **Code reorganization**: Block entry decode helpers (`DecodeEntry`, `DecodeKey`, `DecodeKeyV4`, `ReadBe64FromKey`) moved from `block.cc` to a new shared header `block_util.h` so they can be reused by `BlockBuilder` on the write path. - **New histogram `BLOCK_KEY_DISTRIBUTION_CV`**: Records the CV (scaled by 10000) of each index block's key distribution for observability. - **Java bindings**: `IndexSearchType.kAuto`, `uniformCvThreshold` getter/setter, JNI portal constructor signature updated, and `HistogramType.BLOCK_KEY_DISTRIBUTION_CV` added. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14383 Test Plan: - `IndexBlockTest.IndexValueEncodingTest` parameterized to include `kAuto` search type alongside `kBinary` and `kInterpolation`, verifying correct seek/iteration behavior across all combinations of key distributions, restart intervals, and key lengths. - Uniformity detection validated: blocks with uniform key distribution correctly set `is_uniform = true`, blocks with clustered/non-uniform keys set `is_uniform = false`. - Stress test coverage - Updated check_format_compatible to also include a "uniform" dataset. By default using uniform_cv_threshold=-1 does not result in an incompatibility issues. When manually changing the threshold (e.g. `uniform_cv_threshold=1000`), I see `bad block contents`, which is expected ## Benchmark readrandom with `fillrandom,compact -seed=1 --statistics`: | Benchmark | Branch | Params | avg ops/s | % change vs main | CV P50 | |-----------|--------|--------|-----------|------------------|--------| | readrandom | main | `binary_search, shortening=1` | 335,791 | baseline | N/A | | readrandom | feature | `binary_search, shortening=1` (default) | 335,749 | -0.0% | 1,500 | | readrandom | feature | `auto_search, shortening=1` (kAuto) | 366,832 | **+9.2%** | 1,500 | | readrandom | feature | `interpolation_search, shortening=1` | 366,598 | **+9.2%** | 1,500 | | readrandom | feature | `auto_search, shortening=2` (kAuto) | 344,631 | **+2.6%** | 1,030,000 | | readrandom | feature | `interpolation_search, shortening=2` | 201,178 | **-40.1%** | 1,030,000 | As seen with shortening=2, a non-uniform distribution produces a high CV, which does not use interpolation search. ## Write benchmark There is a write overhead which scans each restart entry for a block upon Finish. In practice this is very low because currently it is only applied to index blocks. See cpu profile (https://fburl.com/strobelight/io5hwj9h) here of `-benchmarks=fillseq,compact -compression_type=none -disable_wal=1`. Only 0.08% attributed to `ScanForUniformity`. Reviewed By: pdillinger Differential Revision: D94738890 Pulled By: joshkang97 fbshipit-source-id: 9661ac593c5fef89d49f3a8a027f1338a0c96766 |
||
|
Peter Dillinger
|
871f79d6ef |
Reformat source files (#14331)
Summary: probably something changed, maybe https://github.com/facebook/rocksdb/issues/14311 Full command: ``` git ls-files '*.cc' '*.h' | grep -v '^third-party/' | grep -v 'range_tree' | xargs clang-format -i ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/14331 Test Plan: CI Reviewed By: mszeszko-meta Differential Revision: D93246992 Pulled By: pdillinger fbshipit-source-id: 6bc5b97978fef8aee52823dadb6daa4bea57343d |
||
Ryan Hancock
|
8f9cb1a708 |
Introduce Memory restrictions for IO Dispatcher. (#14300)
Summary: Introduction of memory limiter for IO Dispatch. Currently, the user has no way of enacting policy with IO dispatcher. One important policy is the ability to restrict the amount of memory a multiscan or set of multiscans is allowed to pin. This PR introduces the max_prefetch_memory_bytes in the IODispatcherOptions, allowing for users to specify bounds on block cache memory usage. There seems to be a minor performance increase however, I have found the scans to be a bit noisy. Each benchmark is run with a stride size of 30000 keys. This was done to ensure we maintain parity with trunk. ``` Configuration: 10 concurrent scans, 1024B values, 5242880 byte SST files Scan sizes: 1024 keys = 1MiB, 2048 keys = 2MiB, 4096 keys = 4MiB per scan | Keys/Scan | Mode | Main (ops/sec) | Main (us/op) | limiter (ops/sec) | limiter (us/op) | Delta ops/sec | |-----------|-------|------------------|------------------|----------------------|--------------------|---------------| | 1024 | sync | 151.6 +/- 8.0 | 6591.14 +/- 343.30 | 170.6 +/- 4.0 | 5855.32 +/- 136.19 | +12.00% | | 1024 | async | 156.4 +/- 24.7 | 6589.64 +/- 1345.73 | 173.8 +/- 2.7 | 5744.51 +/- 91.35 | +11.00% | | 2048 | sync | 77.8 +/- 1.6 | 12785.64 +/- 286.49 | 87.6 +/- 3.4 | 11354.01 +/- 441.71 | +12.00% | | 2048 | async | 85.6 +/- 4.7 | 11658.11 +/- 618.49 | 91.4 +/- 1.2 | 10873.63 +/- 143.49 | +6.00% | | 4096 | sync | 43.2 +/- 1.5 | 22932.27 +/- 730.66 | 43.8 +/- 0.7 | 22563.90 +/- 320.93 | +1.00% | | 4096 | async | 45.4 +/- 0.8 | 21875.64 +/- 357.04 | 46.2 +/- 0.7 | 21416.95 +/- 311.89 | +1.00% | ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/14300 Reviewed By: anand1976 Differential Revision: D92316556 Pulled By: krhancoc fbshipit-source-id: dc0b7958a33b8ef5fa5af82b1c6d960041837fc1 |
||
|
Xingbo Wang
|
656b734a5f |
Support abort background compaction jobs. (#14227)
Summary: This adds a new public API to allow applications to abort all running compactions and prevent new ones from starting. Unlike DisableManualCompaction() which only pauses manual compactions and waits for them to finish naturally, AbortAllCompactions() actively signals running compactions (both automatic and manual) to terminate early and waits for them to complete before returning. The abort signal is checked periodically during compaction (every 100 keys), so ongoing compactions abort quickly. Any output files from aborted compactions are automatically cleaned up to prevent partial results from being installed. This is useful for scenarios where applications need to quickly stop all compaction activity, such as during graceful shutdown or when performing maintenance operations. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14227 Test Plan: - Unit tests in db_compaction_abort_test.cc cover various abort scenarios including: abort before/during compaction, abort with multiple subcompactions, nested abort/resume calls, abort with CompactFiles API, abort across multiple column families, and timing guarantees - Updated compaction_job_test.cc to include the new parameter Reviewed By: anand1976 Differential Revision: D91480994 Pulled By: xingbowang fbshipit-source-id: 36837971d8a540cd34d3ec28a78bc94b582625b0 |
||
Maciej Szeszko
|
6b5ccbbec6 |
Remove inline values support (#14270)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/14270 Legacy BlobDB's inline values feature (storing small values directly in the LSM tree via `min_blob_size` threshold) is unused in production - all deployments use `min_blob_size = 0`. This removes the functionality entirely. Changes: - Remove `min_blob_size` from `BlobDBOptions` - Remove `IsInlined()` check from compaction filter (dead code path) - Remove inline-related statistics (`BLOB_DB_WRITE_INLINED*`) - Remove `InlineSmallValues` test - Update stale comments referencing inlined data Reviewed By: xingbowang Differential Revision: D91088985 fbshipit-source-id: ec67848ece1a7dc071ca8e8a17faebb435394733 |
||
|
Anand Ananthabhotla
|
b89d290c20 |
Add MultiScan statistics (#14248)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/14248 ### Overview This diff introduces the addition of multi-scan statistics to RocksDB, enhancing the database's ability to monitor and analyze performance during multi-scan operations. ### Key Changes #### Implemented Multi-Scan Statistics The following statistics were implemented to provide deeper insights into multi-scan operations: - **MULTISCAN_PREPARE_MICROS**: Measures the time (in microseconds) spent preparing for multi-scan operations. - **MULTISCAN_BLOCKS_PER_PREPARE**: Tracks the number of blocks processed per multi-scan prepare operation. - **Wasted Prefetch Blocks Count**: Counts the number of prefetched blocks that were not used (i.e., wasted) if the iterator is abandoned before accessing them. - **MULTISCAN_TOTAL_BLOCKS_SCANNED**: Tracks the total number of blocks scanned during all multi-scan operations. - **MULTISCAN_TOTAL_KEYS_SCANNED**: Measures the total number of keys scanned across all multi-scan operations. - **MULTISCAN_TOTAL_MICROS**: Captures the total time (in microseconds) spent in multi-scan operations. - **MULTISCAN_PREFETCHED_BLOCKS**: Counts the number of blocks that were prefetched during multi-scan operations. - **MULTISCAN_USED_PREFETCH_BLOCKS**: Tracks the number of prefetched blocks that were actually used during multi-scan operations. ### Impact This diff provides more fine-grained statistics for multi-scan operations, allowing developers and users to better understand and optimize the performance of their RocksDB instances. Reviewed By: krhancoc Differential Revision: D91053297 fbshipit-source-id: 7158741b9f026c0b5ce8ba1264dbd137e7fe985d |
||
|
Ryan Hancock
|
b9951ded37 |
Introducing Prepare all iterators for LevelIterator (#14100)
Summary:
This diff introduces the async prepare of all iterators within a MultiScan. The current state has each iterator be prepared as its needed, and with this diff, we prepare all iterators during the prepare phase of the Level Iterator, this will allow more time for each IO to be dispatched and serviced, increasing the odds that a block is ready as the scan seeks to it.
Benchmark is prefilled using
```
KEYSIZE=64
VALUESIZE=512
NUMKEYS=5000000
SCAN_SIZE=100
DISTANCE=25000
NUM_SCANS=15
THREADS=1
./db_bench --db=$DB \
--benchmarks="fillseq" \
--write_buffer_size=5242880 \
--max_write_buffer_number=4 \
--target_file_size_base=5242880 \
--disable_wal=1 --key_size=$KEYSIZE \
--value_size=$VALUESIZE --num=$NUMKEYS --threads=32
}
```
And benchmark ran is
```
run() {
echo 1 | sudo tee /proc/sys/vm/drop_caches
./db_bench --db=$DB --use_existing_db=1 \
--benchmarks=multiscan \
--disable_auto_compactions=1 --seek_nexts=$SCAN_SIZE \
--multiscan-use-async-io=1 \
--multiscan-size=$NUM_SCANS --multiscan-stride=$DISTANCE \
--key_size=$KEYSIZE --value_size=$VALUESIZE \
--num=$NUMKEYS --threads=$THREADS --duration=60 --statistics
}
```
The benchmark uses large stride sides to ensure that two scans would touch separate files. We reduce the size of the block cache to increase likelyhood of reads (and simulate larger data sets)
**Branch:**
```
Integrated BlobDB: blob cache disabled
RocksDB: version 10.8.0
Date: Tue Nov 11 13:26:29 2025
CPU: 166 * AMD EPYC-Milan Processor
CPUCache: 512 KB
Keys: 64 bytes each (+ 0 bytes user-defined timestamp)
Values: 512 bytes each (256 bytes after compression)
Entries: 5000000
Prefix: 0 bytes
Keys per prefix: 0
RawSize: 2746.6 MB (estimated)
FileSize: 1525.9 MB (estimated)
Write rate: 0 bytes/second
Read rate: 0 ops/second
Compression: Snappy
Compression sampling rate: 0
Memtablerep: SkipListFactory
Perf Level: 1
------------------------------------------------
multiscan_stride = 25000
multiscan_size = 15
seek_nexts = 100
DB path: [/data/rocksdb/mydb]
multiscan : 837.941 micros/op 1193 ops/sec 60.001 seconds 71605 operations; (multscans:71605)
```
**Baseline:**
```
Set seed to 1762898809121995 because --seed was 0
Initializing RocksDB Options from the specified file
Initializing RocksDB Options from command-line flags
Integrated BlobDB: blob cache disabled
RocksDB: version 10.9.0
Date: Tue Nov 11 14:06:49 2025
CPU: 166 * AMD EPYC-Milan Processor
CPUCache: 512 KB
Keys: 64 bytes each (+ 0 bytes user-defined timestamp)
Values: 512 bytes each (256 bytes after compression)
Entries: 5000000
Prefix: 0 bytes
Keys per prefix: 0
RawSize: 2746.6 MB (estimated)
FileSize: 1525.9 MB (estimated)
Write rate: 0 bytes/second
Read rate: 0 ops/second
Compression: Snappy
Compression sampling rate: 0
Memtablerep: SkipListFactory
Perf Level: 1
------------------------------------------------
multiscan_stride = 25000
multiscan_size = 15
seek_nexts = 100
DB path: [/data/rocksdb/mydb]
multiscan : 1129.916 micros/op 885 ops/sec 60.001 seconds 53102 operations; (multscans:53102)
```
Repeated for confirmation.
This introduces a ~20% improvement in latency and op/s.
Note: Benchmarks are single threaded as, when increasing thread count, we start seeing large amounts of overhead being induced by block cache contention, finally resulting in both baseline and branch becoming equal.
Further on network attached storage with high latency, the level iterator, preparing all iterators so a 20% improvement even at high thread counts.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/14100
Reviewed By: anand1976
Differential Revision: D86913584
Pulled By: krhancoc
fbshipit-source-id: da9d0c890e25e392a33389ce6b80f9bfb84d3f85
|
||
|
Peter Dillinger
|
befa6b8050 |
Fix and check for potential ODR violations (#14096)
Summary: ... caused by public headers depending on build parameters (macro definitions). This change also adds a check under 'make check-headers' (already in CI) looking for potential future violations. I've audited the uses of '#if' in public headers and either * Eliminated them * Systematically excluded them because they are intentional or similar (details in comments in check-public-header.sh * Manually excluded them as being ODR-SAFE In the case of ROCKSDB_USING_THREAD_STATUS, there was no good reason for this to appear in public headers so I've replaced it with a static bool ThreadStatus::kEnabled. I considered getting rid of the ability to disable this code but some relatively recent PRs have been submitted for fixing that case. I've added a release note and updated one of the CI jobs to use this build configuration. (I didn't want to combine with some jobs like no_compression and status_checked because the interaction might limit what is checked. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14096 Test Plan: manual 'make check-headers' + manual cmake as in new CI config + CI Reviewed By: jaykorean Differential Revision: D86241864 Pulled By: pdillinger fbshipit-source-id: d16addc9e3480706b174a006720a4def0740bf2e |
||
Hui Xiao
|
8edb99f904 |
Statistics for successfully resumed compaction output bytes (#14054)
Summary: Context/Summary: as titled Pull Request resolved: https://github.com/facebook/rocksdb/pull/14054 Test Plan: new UT, manually checking Reviewed By: jaykorean Differential Revision: D84828431 Pulled By: hx235 fbshipit-source-id: 56e1a9159f7597a10d6c549657d8b22788aa0599 |
||
|
Peter Dillinger
|
1c8a012727 |
Add kCool Temperature (#14000)
Summary: also requested by internal user, like kIce in https://github.com/facebook/rocksdb/issues/13927 Pull Request resolved: https://github.com/facebook/rocksdb/pull/14000 Test Plan: unit tests updated Reviewed By: archang19 Differential Revision: D83200479 Pulled By: pdillinger fbshipit-source-id: 31f2842d87bcad40227aeee9687ff5772393689c |
||
|
Peter Dillinger
|
67af5bdc38 |
Add Temperature::kIce (#13927)
Summary: ... and associated statistics, etc. Someone needs it, so here it is. Pull Request resolved: https://github.com/facebook/rocksdb/pull/13927 Test Plan: Updated / extended / added some unit tests Reviewed By: cbi42 Differential Revision: D81981469 Pulled By: pdillinger fbshipit-source-id: 52558c08741890b781310906acbc18d9eb479363 |
||
anand76
|
0044a76d36 |
Make failure to load UDI when opening an SST a soft failure (#13921)
Summary: If user_defined_index_factory in BlockBasedTableOptions is configured and we try to open an SST file without the corresponding UDI (either during DB open or file ingestion), ignore a failure to load the UDI by default. If fail_if_no_udi_on_open in BlockBasedTableOptions is true, then treat it as a fatal error. Pull Request resolved: https://github.com/facebook/rocksdb/pull/13921 Test Plan: Update unit tests Reviewed By: xingbowang Differential Revision: D81826054 Pulled By: anand1976 fbshipit-source-id: f4fe0b13ccb02b9448622af487680131e349c52b |
||
Changyu Bi
|
b42bf48310 |
Add stats for WBWI ingestion and transaction size (#13611)
Summary: Add stats to monitor the large transaction optimization. A stat is added for how many times wbwi ingestion is used. A histogram is added to track transaction size. We could also just track write batch size for all writes but I don't want to add the overhead to all writes yet. Pull Request resolved: https://github.com/facebook/rocksdb/pull/13611 Test Plan: ran `python3 ./tools/db_crashtest.py --txn blackbox --txn_write_policy=0 --commit_bypass_memtable_one_in=50 --test_batches_snapshots=0 --stats_dump_period_sec=2 --dump_malloc_stats=0 --statistics=1` and manually check LOG files ``` rocksdb.number.wbwi.ingest COUNT : 57 ... rocksdb.num.op.per.transaction P50 : 1.000000 P95 : 1.000000 P99 : 1.000000 P100 : 1.000000 COUNT : 2265 SUM : 2265 ``` Reviewed By: jowlyzhang Differential Revision: D74829087 Pulled By: cbi42 fbshipit-source-id: 5a9c3ab2d4cb6071cedfc47201ce2cf65a77d3c6 |
||
|
Michael C Huang
|
13d865f6f1 |
Add trivial copy support when FIFO compaction reason is kChangeTemperature (#13562)
Summary: Prior to this PR, for FIFO kChangeTemperature compaction was done by iterating and reading thru the input sst and generate the output sst. This was wasteful since for FIFO we could apply the "trivial" move by copying the input sst to the out sst without need decompress/compress and reading thru the input sst content at all. This PR added "allow_trivial_copy_when_change_temperature" to the CompactionOptionsFIFO. Pull Request resolved: https://github.com/facebook/rocksdb/pull/13562 Reviewed By: cbi42 Differential Revision: D73295404 Pulled By: mikechuangmeta fbshipit-source-id: 02241c7389797730ecd4a3b636837cb5f912b424 |
||
|
Hui Xiao
|
29c6610617 |
Add compaction explicit prefetch stats (#13520)
Summary: **Context/Summary:** This PR adds new stats to measure compaction readahead size for rocksdb managed prefetching (not FS prefetching). It can be used to verify compaction read-ahead is doing what's configured. This PR also excludes compaction readahead stats from user scan readahead stats measured in existing stats so there is a cleaner separating between these two. Bonus: this PR also included some typo fixing about "io activities" Pull Request resolved: https://github.com/facebook/rocksdb/pull/13520 Test Plan: Modified existing test to verify stats Reviewed By: archang19 Differential Revision: D72892850 Pulled By: hx235 fbshipit-source-id: 1a73182061baa044c9c9193a2b0fd967ffe75c4a |
||
|
Peter Dillinger
|
2a0ee4ddd8 |
Refactor wal related naming and more (#13490)
Summary: * Clarify in API comments which `log_` options in DBOptions relate to WALs, info log, and/or manifest files. * Rename a bunch of "log" things to "wal" for clarity, especially in DBImpl. (More to go, especially some more challenging cases like `DBImpl::logs_`, but a step in the right direction IMHO) * Simplify DBImpl ctor by moving constant initializers to field definitions. * Use RelaxedAtomic for (renamed) `wals_total_size_` Pull Request resolved: https://github.com/facebook/rocksdb/pull/13490 Test Plan: existing tests Reviewed By: cbi42 Differential Revision: D71939382 Pulled By: pdillinger fbshipit-source-id: 852f4737eca83e6ad653010cc197ad1b6e6bae13 |
||
|
Hui Xiao
|
2e175124d8 |
Rename Env::IOActivity::kReadManifest (#13471)
Summary: Context/Summary: as mentioned in the [comment](https://github.com/facebook/rocksdb/pull/13178?fbclid=IwZXh0bgNhZW0CMTAAAR1nvz-1Ifh6Pm8PwFZbGHAxhLtwfi4W_XaSe-BqnBx3ICJOq-9DTdqFvs0_aem_ITO_0B6cca0kTViRmsAA8g#issuecomment-2702510373) , we want to rename this public name to align with the naming convention. Pull Request resolved: https://github.com/facebook/rocksdb/pull/13471 Test Plan: - Compilation - Manually check for no internal usage of this name. Hopefully it's good for OSS as well as this field is relatively new and the whole IOActivity is marked "EXPERIMENTAL" Reviewed By: mszeszko-meta Differential Revision: D71485300 Pulled By: hx235 fbshipit-source-id: 318c8b6c2a4d874f2f831e3ca690aa2fb8974c0f |
||
Yu Zhang
|
c8bc2b63fc |
Add time measuring metrics for file ingestion in PerfContext (#13219)
Summary: As titled. And also added some documentation for an approach to name perf context metrics that can help identify the starting `PerfLevel` that enables collecting it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/13219 Test Plan: Unit test Reviewed By: hx235 Differential Revision: D67362022 Pulled By: jowlyzhang fbshipit-source-id: 7ed1bb475b5497961612d4e331600609da42074b |
||
|
Peter Dillinger
|
c72e79a262 |
Standardize on clang-format version 18 (#13233)
Summary: ... which is the default for CentOS 9 and Ubuntu 24, the latter of which is now available in GitHub Actions. Relevant CI job updated. Re-formatted all cc|c|h files except in third-party/, using ``` clang-format -i `git ls-files | grep -E '[.](cc|c|h)$' | grep -v third-party/` ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/13233 Test Plan: CI Reviewed By: jaykorean, archang19 Differential Revision: D67461638 Pulled By: pdillinger fbshipit-source-id: 0c9ac21a3f5eea6f5ade68bb6af7b6ba16c8b301 |
||
|
Maciej Szeszko
|
be7703b27d |
Offline file checksum manifest retriever (#13178)
Summary: This change introduces a new, lightweight _experimental_ API that reconstructs the [file # -> file checksum -> file checksum function] 1-1-1 mapping directly from the `MANIFEST` file considered `CURRENT` in scope of specific DB instance at the time. The goal is to provide a cheap alternative to `DB::GetLiveFilesMetaData` that doesn't require opening the database, reconstructing version sets and/or accessing files that are _potentially_ in disaggregated storage. ### Housekeeping: 1. Moved the `GetCurrentManifestPath` out of `version_set` to a new `manifest_ops` file(s) dedicated to manifest related operations. 2. Introduced new `Env::IOActivity::kReadManifest` to better reflect the IO intent in offline file checksum retrieving function. Pull Request resolved: https://github.com/facebook/rocksdb/pull/13178 Test Plan: Added a unit test comparing the outcome of newly introduced API against the established `GetLiveFilesMetaData`: ```hcl ./db_test2 --gtest_filter="*GetFileChecksumsFromCurrentManifest_CRC32*" ``` Reviewed By: pdillinger Differential Revision: D66711910 Pulled By: mszeszko-meta fbshipit-source-id: 57091c550a14ac2e832bf7eea136dab5450e71bc |
||
|
anand76
|
c21fe1a47f |
Add ticker stats for read corruption retries (#12923)
Summary: Add a couple of ticker stats for corruption retry count and successful retries. This PR also eliminates an extra read attempt when there's a checksum mismatch in a block read from the prefetch buffer. Pull Request resolved: https://github.com/facebook/rocksdb/pull/12923 Test Plan: Update existing tests Reviewed By: jowlyzhang Differential Revision: D61024687 Pulled By: anand1976 fbshipit-source-id: 3a08403580ab244000e0d480b7ee0f5a03d76b06 |
||
|
anand76
|
98d8a85624 |
New PerfContext counters for block cache bytes read (#12459)
Summary: Add PerfContext counters for measuring the cumulative size of blocks found in the block cache. Pull Request resolved: https://github.com/facebook/rocksdb/pull/12459 Reviewed By: ajkr Differential Revision: D55170694 Pulled By: anand1976 fbshipit-source-id: 8cbba76eece116cefce7f00e2fc9d74757661d25 |
||
|
anand76
|
4868c10b44 |
Retry block reads on checksum mismatch (#12427)
Summary: On file systems that support storage level data checksum and reconstruction, retry SST block reads for point lookups, scans, and flush and compaction if there's a checksum mismatch on the initial read. A file system can indicate its support by setting the `FSSupportedOps::kVerifyAndReconstructRead` bit in `SupportedOps`. Tests: Add new unit tests Pull Request resolved: https://github.com/facebook/rocksdb/pull/12427 Reviewed By: ajkr Differential Revision: D55025941 Pulled By: anand1976 fbshipit-source-id: dbd990cb75e03f756c8a66d42956f645c0b6d55e |
||
jsteemann
|
3fff57fa6a |
fix linking without thread status support (#12400)
Summary: When compiling with `-DNROCKSDB_THREAD_STATUS`, some functions in ThreadStatusUtil are declared but their definition is missing. Their definitions are only compiled when not defining `NROCKSDB_THREAD_STATUS`. This causes problems on linking, when the linker cannot find the definitions of - ThreadStatusUtil::GetThreadOperation - ThreadStatusUtil::SetEnableTracking This PR fixes it by adding stubs for these functions in case `NROCKSDB_THREAD_STATUS` is defined. Pull Request resolved: https://github.com/facebook/rocksdb/pull/12400 Reviewed By: ajkr Differential Revision: D54510769 Pulled By: cbi42 fbshipit-source-id: e79e9257492d3dba59615e9e306df7e79838d73b |
||
|
jsteemann
|
965364972d |
fix compile warning (#12399)
Summary:
Fix compile warning
```
monitoring/thread_status_util.cc: In static member function ‘static void rocksdb::ThreadStatusUtil::NewColumnFamilyInfo(const rocksdb::DB*, const rocksdb::ColumnFamilyData*, const std::string&, const rocksdb::Env*)’: monitoring/thread_status_util.cc:193:55: warning: unused parameter ‘env’ [-Wunused-parameter]
193 | const Env* env) {}
| ~~~~~~~~~~~^~~
```
Pull Request resolved: https://github.com/facebook/rocksdb/pull/12399
Reviewed By: jaykorean
Differential Revision: D54424333
Pulled By: cbi42
fbshipit-source-id: 3dcb89f85d3a63b1b0d0d6a8b277f49ce03b6d1a
|