mirror of
https://github.com/facebook/rocksdb.git
synced 2026-10-23 08:09:02 +03:00
main
1800 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Yoshinori Matsunobu
|
d6a7847d37 |
Add a DB option to log long-held global mutexes and release sites (#15436)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15436 ## Why Long-held global mutexes can stall foreground and background work. Existing mutex statistics measure acquisition waits, not continuous ownership. This adds opt-in logging to identify the mutex, owning thread, operation and release site. ## Enable for an investigation Provide an INFO logger and shared `Statistics` object when opening the DB, then: - Set the shared statistics level to `StatsLevel::kAll`. - Set `mutex_hold_log_threshold_us` through `DB::SetDBOptions`; `100000` means 100 ms. - Restore threshold 0 and the previous statistics level afterward. The threshold defaults to **0 (disabled)**. `kAll` alone does not enable these hold probes; threshold 0 leaves existing statistics timers unchanged. ## Output Illustrative records, not production observations: ``` SLOW_GLOBAL_MUTEX mutex=DBImpl::mutex_ mutex_addr=0x12340000 owner_thread_id=12345 thread_operation=Compaction hold_us=200000 threshold_us=100000 observed_end_us=1792205400000000 buffer_dropped_records=0 unlock=db_impl_compaction_flush.cc:4660 BackgroundCallCompaction() SLOW_GLOBAL_MUTEX mutex=DBImpl::wal_write_mutex_ mutex_addr=0x12340100 owner_thread_id=12346 thread_operation= hold_us=35000 threshold_us=25000 observed_end_us=1792205401000000 buffer_dropped_records=0 scope_exit=db_impl_write.cc:2250 PreprocessWrite() ``` The first proves a completed 200 ms DB-mutex ownership interval on a thread tagged `Compaction`. The second uses a 25 ms threshold; an empty operation tag means none was available. Neither record reports acquisition wait or proves the cause of a stall. Release attribution uses compiler-embedded strings available in optimized, stripped builds, with a GCC/Clang fallback where standard source locations are unavailable. Direct unlock and condition wait identify their call sites; scope exit and temporary unlock identify guard declarations, not closing braces. No stack unwinding or symbol lookup. ## Scope and safety - Track six selected mutexes: DB, WAL, point-lock wait graph, two prepare-log bookkeeping mutexes and transaction name map. This is not comprehensive synchronization tracing. - Acquisition waits and condition-variable release gaps are excluded from hold duration. Configuration is pre-first-use; repeated setup retains the first valid context. - Sample at most once per mutex per second, with eight deferred records per thread. `buffer_dropped_records` counts capacity overflow only; rate limiting, pause and emission contention can omit samples. - Emit after covered outer lock scopes release. Emission is nonblocking best-effort; shutdown drains/pauses emission and stops diagnostics before the owned logger closes. Closing from that logger's own callback is unsupported. - No data-file format change. The mutable option is persisted in OPTIONS and wired into bench/stress tools. Typed C binding is deferred; string DB-option APIs are available. Disabled bookkeeping still has a cost. Release A/B measurements found about 3 ns added per selected mutex pair at level 1, and about 70 ns for hold timing over `kAll`/0. Get and reused-iterator Seek showed no resolved small probe effect; fresh iterator+Seek at 16 threads showed +8.7% CPU/query and -9.4% QPS. No logs were emitted in those benchmarks; logging cost and broader workload qualification remain open. Reviewed By: xingbowang Differential Revision: D125790955 fbshipit-source-id: 1096163f3bd25ffee585dc6ec8ebfd7f1759851a |
||
zaidoon
|
553b52b4a7 |
Wire IndexFactory through block-based tables (#14963)
Summary: Build block-based table indexes through `IndexFactory`, with modes for writing both indexes or only the custom index and a per-read index selector. Built-in reads retain their existing cache and prefetch paths. The review fixes preserve the existing UDI behavior at the migration and rollback boundaries: - Reopening an SST with index caching enabled reuses the serialized custom block. Each table reader constructs its own factory reader over those bytes and passes `NewReader()` a `Slice` view stored in the table reader. That view stays valid for the reader's lifetime, so a legacy reader can advance it or keep a pointer to it without changing the cached block. The view and the raw block are destroyed after the custom reader. - `IndexFactoryReader::ApproximateMemoryUsage()` keeps its documented meaning: the reader's own allocations, excluding the serialized block. `MemoryUsageIncludesIndexBlock()` defaults to false. A reader whose estimate includes the block opts in, and the wrapper subtracts the block size from that estimate. - Copying factory options and clearing a legacy UDI flag now rolls back that flag's policy. Legacy-derived options keep the requested default and resolve their effective mode without changing the inputs. Explicit factory modes retain precedence through cloning, failed updates, reconstruction, and the C API, including an explicitly selected `kStandardDefault`. - OK empty custom-index output still produces a zero-size meta block and reads through the full standard index. This covers point data and files containing only range tombstones. Custom-only mode rejects empty output because it has no standard fallback. - Default primary reads retain the table factory's precedence over a legacy per-read factory pointer. Explicit `kPreferCustom` still checks a supplied factory name; `kBuiltin` ignores that pointer. - OPTIONS files write equivalent legacy booleans for `kStandardDefault`, `kStandardRequired`, and `kCustomDefault`. Older releases retain those policies when loading newer files with `ignore_unknown_options=true`; the explicit-mode marker remains authoritative for current readers. Custom-only SSTs carry a footer feature bit so older readers reject them. Reject unsupported filter/index combinations before constructing filters. Trie size estimates account for shared prefixes, and block offsets or sizes beyond the 32-bit format limit return `NotSupported` instead of truncating. ## Safety and compatibility - Custom-only files require format version 6 or later and the factory. Rewrite every custom-only SST with a full standard index before removing the factory or downgrading. The other modes have no equivalent legacy configuration beyond the three listed above. - A raw C++ options struct cannot distinguish an explicitly assigned `kStandardDefault` from its default value. Configure the factory to make that choice explicit. `GetEffectiveIndexMode()` reports the resolved policy while preserving legacy inputs for copy-and-clear rollback. - A custom factory with user-defined timestamps is rejected in every mode except `kStandardOnly`, which ignores the factory. - Trie construction rejects a data-block offset or size greater than `UINT32_MAX`, including in `kStandardDefault`. That can fail flush, compaction, or `SstFileWriter::Finish()`; total SST size above 4 GiB alone is not the rejection condition. File-size targets do not split L0 or `SstFileWriter` outputs. - Custom-index cache hits, misses, and adds are reported under `BLOCK_CACHE_INDEX_*` instead of `BLOCK_CACHE_DATA_*`. The existing cache setting still controls ownership and cache use. - A table reader now charges a custom index block it owns (`cache_index_and_filter_blocks=false`); before, that block was not charged anywhere. Cached blocks are still charged only to the block cache. A reader whose `ApproximateMemoryUsage()` includes the serialized block, such as IntegerTrie or TimeSeries, must override `MemoryUsageIncludesIndexBlock()` to return true, or that block is charged twice. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14963 Test Plan: For this commit, on macOS: `table_test`, `trie_index_test`, `trie_index_db_test`, and `block_based_table_reader_test` pass on the full stack (11,416 cases), and the full `table_test` passes at this PR's head alone (6,969 cases). The four custom-reader compatibility tests pass 100 times with `ASSERT_STATUS_CHECKED=1` and `COERCE_CONTEXT_SWITCH=1`, and 84 UDI table cases pass under ASAN with stack-use-after-return detection. Release-mode syntax checks with `-DNDEBUG -fno-rtti` on the changed sources, source checks, formatting checks, and `git diff --check` pass. Two new regressions cover the latest review, and both were checked failing before the fix: - `RetainedReaderSliceOutlivesTableOpen`: a reader keeps a pointer to the `Slice` it received, the stack is overwritten after open, and the reader then serves a seek and a full scan. It covers all three custom-index modes with owned and cached blocks, including a cached reopen. Without the fix, the retained `Slice` no longer matched the block. - `AuxiliaryReaderMemoryIsFullyCharged`: a reader that implements only the older reader methods reports a fixed allocation of half and twice the raw block size. Table usage must grow by that amount with owned and cached blocks, and a strict cache sized just below the required reservation must reject the open with `MemoryLimit`. Without the fix, usage fell short by `min(allocation, raw block size)` and the strict-cache open succeeded. The consuming-reader and strict-cache accounting tests are kept, now on shared helpers. From the previous revision: 498 focused C++ cases passed (82 table/factory, 156 trie, and 260 trie DB), along with the C API test and generated C API verification. Coverage includes both comparators, copied-options rollback, C factory reconstruction, explicit modes, OPTIONS serialization, SST Get/iteration, point and range-only flushes, compaction, reopen, cache disabled/enabled/evicted, and legacy factory precedence. The rollback, empty-output, primary-selector, and OPTIONS regressions were checked failing before their fixes. The cache test measures file reads at the custom block's actual offset: one on a cold open, zero on cached reopens, and one after eviction or with caching disabled. Disabling custom-block cache reuse makes that assertion fail. The full suite and Linux/Windows/TSAN coverage are left to CI. Earlier full-suite runs had failures/timeouts (173/2095 jobs normally and 152/2088 with Status checking); that broad suite was not rerun for this commit. ## Series Previous: https://github.com/facebook/rocksdb/issues/14962 (landed through import). Next: https://github.com/facebook/rocksdb/issues/14965. Reviewed By: jaykorean Differential Revision: D122796629 Pulled By: xingbowang fbshipit-source-id: 4c9d6f3f548c588a37b08fd94de25e8c0b5ef743 |
||
Josh Kang
|
babce11705 |
Add public internal key encoding utilities (#15396)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15396 ## Summary `ParseEntry()` exposes the components of an internal key, but constructing one with a chosen sequence number and entry type currently requires private RocksDB utilities or duplicating the encoding. Add public `EncodeEntry()` to construct a stored internal key from `ParsedEntryInfo`, including its user key, timestamp, sequence number, and entry type. Also expose `PackSequenceAndType()` and `UnPackSequenceAndType()` to work with the sequence/type footer as a `uint64_t`. The helpers map all nine representable public entry types to their internal types and use the existing footer encoding; `EncodeEntry()` reuses the public packer. Packing `kEntryOther` or a sequence number above the 56-bit limit returns `InvalidArgument` without changing the output. Unpacking reports internal value types without a public equivalent as `kEntryOther`. Timestamp size remains the caller's responsibility, and input slices must not alias the output string. These are additive C++ utilities using the existing internal-key format. Reviewed By: anand1976, xingbowang Differential Revision: D125082298 fbshipit-source-id: 6c09a31695f1d377b9cc208fa5b6edd3ad6638e2 |
||
Xingbo Wang
|
7454655969 |
Small fix and stress coverage for MANIFEST-bound OPTIONS recovery (#15431)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15431 A filesystem failure during column family creation can leave the MANIFEST and OPTIONS snapshots at different points in the column family lifecycle. Recovery must select the snapshot consistent with the MANIFEST without deleting a newer valid legacy snapshot. Fix: When the MANIFEST committed an older tracked OPTIONS file, recovery previously rejected the highest-numbered candidate without reading its tracking marker. Writable open then deleted a newer legacy file, or an unreadable highest file, as a rejected tracked orphan; read-only open also reported the older file number. Recovery now inspects every higher candidate. A newer legacy snapshot supersedes the older committed snapshot, while an unreadable higher candidate is retained rather than deleted. Add: Crash-test invocations randomize `track_options_file_number_in_manifest`. Before writable open can replace evidence, db_stress compares the public latest-OPTIONS choice with read-only recovery, using only currently live column family descriptors and running before requested missing families are appended. After open, it verifies the persisted tracking marker and live column family set. TransactionDB still skips the unsupported read-only probe, but it now reads the MANIFEST first so an existing commit enables the post-open verification. The regression test also checks that writable open preserves the newer legacy file before creating another column family. Change: Reopen MANIFEST-avoidance checks use the loaded `options_` value directly. This is a behavior-preserving simplification, not a separate correctness fix. Safety and limitations: There is no API, file-format, or production hot-path change. Ordinary writable open still receives the caller options and complete requested column family set. The extra read-only open is stress-only and runs when tracking verification is active. The pre-open comparison remains skipped for best-efforts recovery, TransactionDB, and stacked BlobDB; post-open verification remains skipped for best-efforts recovery, TTL, and stacked BlobDB. Random crash coverage is probabilistic, and this parent diff does not add randomized column family operations; the dependent lifecycle diff adds that profile. Reviewed By: pdillinger Differential Revision: D125029493 fbshipit-source-id: 5e291deff735e3cbe6d7df0513f940e84b20f387 |
||
|
Josh Kang
|
014e1c43c1 |
Apply readability fixes (isolate decls, braces) (#15394)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15394 ## Summary RocksDB's `.clang-tidy` enables several readability checks that existing code still violates: `readability-isolate-declaration` (476 findings), `readability-braces-around-statements` (49), `readability-redundant-string-init` (25), `modernize-use-using` (10), `misc-definitions-in-headers` (3) and `readability-duplicate-include` (1). They keep getting flagged whenever nearby lines change. This diff fixes 553 of the 564; upstream has since fixed the other 11. What changed: - Multi-variable declarations such as `std::string a, b;` are split into one declaration per variable. Pointer and reference declarators are kept per variable. - Braces are added around single-statement `if`/`else`/`for`/`while` bodies, and empty spin loops get an empty `{}` body. - `std::string s = ""` / `member("")` initializers are removed. - Remaining `typedef`s in the touched C++ files become `using`. C headers and `extern "C"` code are untouched. - `inline` is added to non-template function definitions in test headers. - A duplicate `#include "util/udt_util.h"` in `db_impl.cc` is removed. These are clang-tidy's own fix-its, applied mechanically and then clang-formatted on the changed lines. Three kinds of broken fix-it output were repaired by hand. Removing a `member("")` initializer left a dangling comma in constructor initializer lists (`ExternalSstFileInfo`, `ColumnFamilyMetaData` and others). The function-type typedefs in `block_test.cc` came out malformed. Empty `while (cond);` spin loops got a `{;` body. Splitting declarations also exposed two vectors in `db_etc2_test.cc` to `performance-inefficient-vector-operation`, so a `reserve()` was added for each. The lint bot reports every finding in a file a diff touches. This diff splits declarations in `options/configurable_test.cc`, so it also fixes that file's 14 older `cert-err58-cpp` and `cppcoreguidelines-avoid-non-const-global-variables` findings, which come from seven mutable file-level maps. Six are now `const` and built where they are used (function-local statics, or locals in their only user). The unused `imm_struct_option_info` is deleted. Compatibility: no behavior change. In public headers, the only edits are removed redundant `""` initializers, split declarations and added braces. Reviewed By: rban1 Differential Revision: D123186310 fbshipit-source-id: f9c8e88b00e997db0196f8725c7c8d880cf798da |
||
Anand Ananthabhotla
|
9198b11b46 |
Make InjectedErrorLog::Entry 256 bytes on 32-bit targets (#15388)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15388 `InjectedErrorLog::Entry` in `utilities/fault_injection_fs.h` is written to the injected error log in its native layout, and a `static_assert` requires `sizeof(Entry) == 256` so the record size stays stable. That assert fails on i386 builds. Root cause: the fields add up to only 252 bytes. On targets where `alignof(uint64_t)` is 8 (x86_64, aarch64, armv7), the compiler adds 4 bytes of implicit tail padding, so the size is 256. On i386, `uint64_t` members are 4-byte aligned inside a struct, so there is no tail padding and the size is 252. Fix: add an explicit trailing `uint8_t padding[4]` field so the size no longer depends on `uint64_t` alignment. Compatibility: on 64-bit targets the new field takes the place of the old implicit tail padding, so every field offset and the 256-byte record layout are unchanged. `Record()` value-initializes the entry (`Entry entry{}`), so the padding bytes are always written as zero. No decoder in the repo depends on the padding bytes. ___ landed-with-radar-review Differential Revision: D124873369 fbshipit-source-id: f5c6f43ca85013b0e769853ad13b8d4f3f89b958 |
||
|
Josh Kang
|
4b983da055 |
Spell out special member functions (rule of five) (#15387)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15387 ## Summary RocksDB's `.clang-tidy` enables `cppcoreguidelines-special-member-functions`, `cppcoreguidelines-virtual-class-destructor` and the noexcept-move checks, and clang warns `-Wdeprecated-copy-with-user-provided-dtor`. A full scan reports 476 findings. In most of them, a class declares a destructor (or copy operations) but leaves the rest of copy/move implicit. That has two costs. Readers can't tell whether a class may be copied or moved. And resource-owning classes keep an implicit copy that would double-free if anyone ever used it. This diff resolves 453 of the 476. Approach, picked per class from least to most invasive: 1. An empty user-provided destructor becomes `= default`. This covers most interface classes, such as `virtual ~X() {}` -> `= default`. If the class is still copied, that would only rename the deprecated-copy warning, so the destructor is removed instead when nothing needs it (`Customizable`, `TableFactory` and `InternalKeyComparator` inherit a virtual one; `IteratorWrapperBase` isn't polymorphic), and `MemTableRep::KeyComparator` gets defaulted copy and move. 2. Classes whose copy is already deleted get explicitly deleted move operations too. Move was already unavailable, so this only documents it. 3. Resource-owning classes that are never copied get deleted copy and move operations. Examples: the C API wrapper structs, which run a `destructor_` callback (edited in `tools/c_api_gen/c_base.cc` and regenerated into `db/c.cc`); `ShardedCache`, which frees its shards; iterator wrappers; and RAII handles. 4. Value types that are copied or moved get the missing operations as `= default`, which keeps their current behavior. 5. Exception specifications of existing functions are unchanged, except for `WriteBatch`'s move assignment (below): no other `noexcept` is added or removed. Only the move operations this diff newly declares (`Comparator`, `FileSampledStats` and the thread-local `Entry`) are `noexcept`; they copy only scalars or atomics. `WriteBatch`'s copy and move assignment destroyed `*this` and placement-new'ed a new `WriteBatch` over it. That is undefined behavior when `*this` is a subclass, and if the copy constructor threw, `*this` was left destroyed and was later destroyed again. Move assignment now assigns member-wise and ends in the same state as the move constructor, which leaves `has_key_with_ts_`, the timestamp-size tracking and the other fields it doesn't copy at their defaults. `rep_` is swapped through a temporary, because a plain string move-assignment can keep the old buffer (e.g. a large batch reset with `batch = WriteBatch()` would keep its memory). Nothing in it can throw, so it is now `noexcept`. Copy assignment copies into a temporary and moves it in, so a throwing copy leaves `*this` unchanged. Bugs this exposed and fixed: - `RestoreAfterCopyOrCreateWorkItem` (backup engine): the hand-written move constructor and assignment moved only some fields, so `from_file` and `to_file` were empty after `push_back(std::move(...))`. A restore that hit a checksum mismatch then reported "While restoring -> : ..." without file names. It now uses the compiler-generated moves, which are still `noexcept`. - `ExternalSstFileIngestionJob` lived in a `std::vector`, and its destructor frees raw-owned pointers, so the implicit copy used on vector reallocation would double-free. Its file batches also point into its own inline file storage, so neither a copy nor a move is safe after `Prepare()`. Copy and move are now deleted, and `DBImpl` holds jobs through `std::unique_ptr`. Compatibility: no public class loses an operation users can call. Public interfaces mostly only get `virtual ~X() = default;`. `WriteBatch`'s move assignment becomes `noexcept`, which is not part of the mangled name, so the ABI doesn't change. `SubcompactionJobInfo` and `CompactionJobInfo` intentionally stay C++20 aggregates, so users' aggregate and designated initializers keep compiling. `Configurable`, `CompareInterface`, `WriteBatchBase`, `ObjectLibrary::Entry` and `MemTableRep::KeyComparator` declare defaulted copy and move, and `Customizable` no longer declares a destructor. As a result, `Configurable` and `Customizable` subclasses that declare no destructor or copy operations of their own now move their registered-options list instead of copying it. In-tree, these are the mutable-options adapters, the table factories, the memory allocators and test classes; all are held through smart pointers or used in place, and `Clone()` copies, so none is moved by value. Not changed (23): `IODebugContext` (custom deep copy plus a `std::shared_mutex`), `DBWithColumnFamilies` in db_bench (stored in a `std::vector` with non-movable members), `Snapshot`'s protected virtual destructor (intended), `ManagedSnapshot`, the `SubcompactionJobInfo` and `CompactionJobInfo` aggregates, and the 16 noexcept-move-constructor and noexcept-swap findings on existing functions, including `autovector`'s move assignment (adding `noexcept` would change their exception specifications). Copying `ManagedSnapshot` already double-releases the snapshot. Fixing that needs a real move implementation and is a behavior change, so it is left for a separate diff. Reviewed By: rban1 Differential Revision: D123186309 fbshipit-source-id: 6f8e57e233e12199a007407130bb9a1f6d52e8d1 |
||
|
Josh Kang
|
ddac37a780 |
Serialize subset-checkpoint OPTIONS persistence with SetOptions (#15386)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15386 Fixes crash-test failure T292562874: `Checkpoint failed: Corruption: [RocksDBOptionsParser]:failed the verification on BlockBasedTable::: filter_policy.id` Root cause: - D123104845 ("Commit authoritative OPTIONS file numbers in MANIFEST", 77d94c989243) made subset checkpoints (`CreateCheckpoint` with `include_cf_ids` that exclude some CFs) write a new OPTIONS file. `DBImpl::GetLiveFilesStorageInfoImpl` captures the snapshot under the DB mutex with `cfd->GetLatestCFOptions()`. That is a shallow copy: `table_factory`, and through it `filter_policy`, are `shared_ptr`s to the live objects. `DBImpl::CreateOptionsFileForSubsetCheckpoint` then calls `PersistRocksDBOptions` without holding `options_mutex_`. `PersistRocksDBOptions` serializes the file, then re-parses it and checks it against the same in-memory options. - `RibbonFilterPolicy::bloom_before_level_` is a `kMutable` atomic that `SetOptions(table_factory.filter_policy.bloom_before_level=...)` changes in place, and `RibbonFilterPolicy::GetId()` includes it (`ribbonfilter:<bits>:<bloom_before_level>`). A concurrent `SetOptions` between the write and the verify makes the in-memory `filter_policy.id` differ from the file, so verification returns Corruption and the checkpoint fails. - Every other OPTIONS writer (`WriteOptionsFile`, CF create/drop) holds `options_mutex_`. `SetOptions` holds it for its whole mutate-and-persist sequence. The new subset-checkpoint writer was the only one that skipped it. Trigger in the crash test: db_stress randomly sets `table_factory.filter_policy.bloom_before_level` through `SetOptions` (when `keepRibbonFilterPolicyOnly`, i.e. `bloom_before_level` != INT_MAX with ribbon filters; see the OPTIONS dump with `ribbonfilter:18:1`, `ribbonfilter:8.9:2147483647`, `ribbonfilter:5.67:-1` across CFs). It also takes subset checkpoints. The two run concurrently on different threads. Fix: hold `options_mutex_` in `CreateOptionsFileForSubsetCheckpoint` while persisting and verifying, the same as `WriteOptionsFile`. The only caller is `CheckpointImpl::CreateCheckpoint`, which holds no DB locks at that point, so lock ordering (options_mutex_ before mutex_) is kept. Note: the snapshot's scalar fields are still captured at live-file time. Live pointer sub-options such as the filter policy reflect their state at persist time. This was already true of the shallow snapshot. The change only makes the persisted file self-consistent with what is verified. Reviewed By: xingbowang Differential Revision: D124728996 fbshipit-source-id: adb5511c2ddd73ab138fd7fd21109199ff4f5b1e |
||
|
Xingbo Wang
|
baa4d373a6 |
Commit authoritative OPTIONS file numbers in MANIFEST (#15305)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15305 Why the change is required: Column-family topology is durable in MANIFEST, while its operational configuration snapshot is stored in a separate OPTIONS file. CreateColumnFamily and DropColumnFamily previously committed MANIFEST first and wrote OPTIONS afterward. An OPTIONS I/O failure could therefore make the API report failure after the topology change was already durable. Persisted-options readers also selected the highest file number and could consume an orphan left by a failed publication. Previous behavior: RocksDB had no durable correspondence between a MANIFEST state and the OPTIONS snapshot describing it. LoadLatestOptions and GetLatestOptionsFileName treated the highest-numbered OPTIONS file as authoritative. Column-family creation and deletion returned the later OPTIONS persistence error even when their MANIFEST edit had already succeeded. Multi-column-family operations wrote a complete OPTIONS snapshot after every successful member, producing quadratic OPTIONS bytes. Ordinary DB::Open consumes caller-supplied DB and column-family options rather than loading OPTIONS. This change preserves that contract. New behavior: Add the default-disabled experimental DB option track_options_file_number_in_manifest. Its value is serialized into each OPTIONS file so the selection rule is self-describing and participates in the normal DB-options compatibility checks. When enabled, RocksDB first durably creates and renames OPTIONS-N and then commits N with one safely ignorable MANIFEST field. A normal OPTIONS rotation uses one MANIFEST commit. Column-family creation and deletion prewrite the proposed snapshot and commit its number together with a topology edit in the same VersionEdit. A failure before the OPTIONS file is durable changes no topology. A failure after the file is durable but before MANIFEST commit leaves an orphan that binding-aware readers ignore, while the previous committed snapshot remains authoritative. Each multi-CF create or drop writes one proposed snapshot, independent of the batch size. Create commits the new number with the first successful add; drop commits it with the last successful drop. Existing partial-success semantics can therefore leave a committed snapshot with extra CFs, but never one missing a live MANIFEST CF. Persisted-options APIs filter this safe superset against recovered MANIFEST membership. A snapshot missing a live CF is corruption. DB::Open with create_missing_column_families uses the same batched path. If the newest OPTIONS file has tracking disabled, it retains the legacy highest-numbered-file behavior, including after an old or disabled writer has run. If it has tracking enabled, readers use the recovered MANIFEST commit and reject newer tracked orphans. A missing usable commit falls back to the highest tracked snapshot for RepairDB and legacy-rewrite recovery. DB open logs that fallback through its configured logger; logger-less utility APIs remain silent. Writable open removes rejected tracked orphans when local obsolete-file deletion is enabled. Read-only open never mutates the DB directory. A DB with no recovered commit and tracking disabled takes the legacy fast path without parsing the newest OPTIONS file. When inspection is required, RocksDB reads only the tracking marker. Marker-read failures used for selection do not fail DB open; a recovered commit remains authoritative and ambiguous newer candidates are conservatively retained. The first topology change after enabling tracking first binds the current topology, so failure of the proposed change cannot make its orphan authoritative. MANIFEST failure during an ordinary tracked OPTIONS rotation also enters the existing manifest-write background-error path. Implementation: - Add the safely ignorable CommittedOptionsFileNumber VersionEdit field and carry it through recovery, MANIFEST snapshots, rollover, and subset checkpoints. - Persist the tracking marker through C++, C, and Java options, generated round-trip coverage, compatibility checking, and db_stress configuration. - Centralize binding-aware selection for DB open and public persisted-options APIs. Reuse RecoveryContext's directory listing, use a narrow marker scan, and recover live CF names while replaying MANIFEST. - Filter manifest-selected safe-superset snapshots to live MANIFEST CFs in LoadLatestOptions and compatibility verification without discarding their option maps. - Prevalidate and sanitize CF batches outside the DB mutex, write one complete snapshot while options_mutex_ serializes topology changes, and hold the DB mutex only for the existing individual MANIFEST commits. Batch missing-CF creation during DB open follows the same locking boundary. - Keep the committed snapshot live in cleanup and live-file APIs, and make subset checkpoints write an OPTIONS snapshot and filtered MANIFEST with identical membership in tracked and legacy modes. - Preserve the first partial multi-CF drop error while consuming legacy OPTIONS persistence Status, and sync the checkpoint MANIFEST directory after replacement. - Pass checkpoint helper callbacks by reference and group VersionEdit storage by alignment so the new paths add neither callback copies nor avoidable object padding. Safety boundaries and unsupported cases: - Tracking defaults to false. Disabled writers and never-tracked databases retain legacy write ordering and highest-numbered selection. - Older binaries safely ignore the new MANIFEST tag and continue choosing the highest OPTIONS number. Downgrade is readable but not crash-atomic: an old binary can consume a tracked orphan. That is the same class of OPTIONS/MANIFEST inconsistency as the previous behavior. - A rollback binary must know the serialized option name before the feature is enabled. Exact compatibility checks intentionally report a mismatch when the caller's tracking setting differs from the persisted snapshot. - A newer untracked OPTIONS file deliberately supersedes an older committed number, allowing an old or disabled writer to become authoritative without rewriting MANIFEST. - Reactive secondary and follower instances do not enforce or repair the binding. Ordinary DB::Open still uses caller-supplied options. - Writable open does not delete rejected tracked or ambiguous files when a compaction service owns OPTIONS-file retention. Read-only open performs no repair writes. - Direct consumers that scan OPTIONS filenames themselves are unchanged; the binding is enforced by RocksDB's persisted-options APIs. - If the committed snapshot is missing and only tracked candidates remain, selection intentionally falls back to the highest candidate so RepairDB output remains usable. Validation coverage and known limitations: Focused coverage exercises encoding compatibility, tracked and legacy selection, the never-tracked open fast path, marker-read failures, exact tracking-marker compatibility, activation and disable/re-enable transitions, batched create/drop and create-missing behavior, partial-batch safe supersets, OPTIONS/MANIFEST failure injection, background-error propagation, orphan cleanup and retention, read-only non-mutation, subset-checkpoint behavior, stale OPTIONS recovery, RepairDB option sanitization, and transient read-corruption reconstruction. The assertion-enabled full executable suite passes. The repository check-format and check-sources wrappers cannot run because they invoke Git inside this Sapling checkout; fbsource lint is used instead. Cross-platform CI is not part of local validation. Reviewed By: pdillinger Differential Revision: D123104845 fbshipit-source-id: 77d94c989243c3f3f9fedfa81fa364c166250303 |
||
|
Josh Kang
|
da82001d81 |
Mark overriding methods and destructors override (#15366)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15366 ## Summary RocksDB's `.clang-tidy` enables `modernize-use-override`, and clang reports `-Winconsistent-missing-destructor-override` when a class marks its other overrides `override` but not its destructor. A full scan finds 190 `modernize-use-override`, 100 `inconsistent-missing-destructor-override` and 2 `suggest-override` findings in existing code. Each one comes back whenever a nearby line is edited. They also leave classes where it isn't obvious which methods are overrides and which introduce new virtuals. This diff marks every overriding member function and destructor `override` and drops the redundant `virtual` on them. It also changes `final override` to plain `final`, following the check's default. Methods that introduce a new virtual keep `virtual`. Most edits are clang-tidy's own fix-its, applied mechanically and then clang-formatted. Compatibility: no signatures change. In public headers the only effect is `override` on existing overrides (mostly destructors, plus `Cache::Name()` and `Cache::GetPrintableOptions()`, whose base declarations are unconditional), so user code that subclasses these types is unaffected. Every touched class now marks all of its overrides consistently. This matters because a partially-marked class would make clang emit `-Winconsistent-missing-override` in user builds. Reviewed By: rban1 Differential Revision: D123186308 fbshipit-source-id: 1f489ea32d938e3ec0faf9abc4f7ff49e042d37a |
||
|
Josh Kang
|
928527b869 |
Fix bugs and false positives from clang-analyzer/bugprone (#15325)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15325 ## Summary RocksDB's `.clang-tidy` enables the `clang-analyzer-*` and `bugprone-*` checks. A full scan of the repo still reports 78 findings from these checks in existing code. Each time a nearby line is edited, they come back as noise, and a few of them are real defects. This diff resolves 73 of the 78; upstream has since fixed one more, and the four left are listed under "Intentionally not changed". Some fixes repair real bugs. The rest make the code's intent explicit to the analyzer without changing behavior. Real bugs fixed: - `port/mmap.cc`: `MemMapping::operator=(MemMapping&&)` ran the destructor on `*this`, `memcpy`'d over a non-trivially-copyable object, then placement-new'd the source. That is undefined behavior, and `*this` can be the base subobject of a `TypedMemMapping`. It now moves the members one by one, and the previous mapping is released through a local `MemMapping`. - `port/mmap.h`: `TypedMemMapping::operator=(MemMapping&&)` had no `return *this;`, so calling it was undefined behavior. - `cache/typed_cache.h`: `if constexpr (sizeof(TCreateContext) > 0)` is always true, so the branch meant for empty create contexts (for example `BlobContentsCreator`) could never run. The check is now `!std::is_empty_v<TCreateContext>`. Existing instantiations still call the same `Create` function. - `cassandra_row_merge_test.cc`: the test reused a vector after `std::move`-ing it into `RowValue::Merge(std::vector<RowValue>&&)`. That only worked because `Merge` happens not to consume its input. The test now rebuilds the input. Behavior-preserving changes: - Virtual calls in constructors and destructors are now explicitly qualified (for example `DBImpl::CloseImpl()` in `~DBImpl`). During construction and destruction, dispatch already resolved to these functions. - Stale `/*name=*/` argument comments now match the parameter names. - `unique_ptr::release()` after a successful cache `Insert` (where the cache takes ownership) uses `[[maybe_unused]] T* released = x.release();`, the pattern already used in `table_cache.cc`. - Removed `class Mutex;` and `class StaticMeta;` forward declarations that sat in the wrong namespace. - Added default member initializers for the `TablePropertiesCollectorFactory::Context` fields that its default constructor left uninitialized. - Smaller cleanups: a redundant `cond ? 3 : 3` in a test, `sizeof(std::string)` in a memory estimate, and an explicit `Cleanable` base move via `up_cast`. Tests that intentionally use moved-from objects now carry `NOLINT(bugprone-use-after-move)`, matching the existing convention. Intentionally not changed: enum casts in tests that check invalid-value handling; three `clang-analyzer-core.NullDereference` reports the analyzer can't prove safe (the C API timestamp out-parameters in `db/c.cc`, which are only written when the caller asked for a timestamp, and the waiter-list walk in `util/async_file_reader.cc`); bit-flag enums in public headers (`FileOpenContract`, `OptionTypeFlags`), where OR-combined values are valid; ownership transfers the analyzer can't see across translation units (`partitioned_index_reader.cc`, `fs_posix.cc`); and the intentional two-separator check in `file/filename.cc`. Reviewed By: rban1 Differential Revision: D123186307 fbshipit-source-id: 623e871286b1c73263b1309cb194692194199e0f |
||
Peter Dillinger
|
1c490a4a4d |
Fix "ancester" spelling; deprecate TablePropertiesNames (#15335)
Summary: Follow-up to https://github.com/facebook/rocksdb/issues/15304, which renamed `TableProperties::creation_time` to `oldest_ancestor_time` but left the long-standing "ancester" misspelling everywhere else, so the same concept was spelled two ways (e.g. `TableProperties::oldest_ancestor_time` vs. `FileMetaData::oldest_ancester_time` and `SstFileMetaData::oldest_ancester_time`). This change makes the spelling consistent and addresses remaining review feedback on https://github.com/facebook/rocksdb/issues/15304. Change: * All internal identifiers and comments now use "ancestor": `FileMetaData::oldest_ancestor_time`, `CompactionServiceOutputFile::oldest_ancestor_time`, `kUnknownOldestAncestorTime`, `TryGetOldestAncestorTime()`, `Compaction::MinInputFileOldestAncestorTime()`, the `kOldestAncestorTime` manifest tag enum, and related locals and test sync point names. * Public API: `SstFileMetaData::oldest_ancestor_time` is the new name; `oldest_ancester_time` remains as a deprecated alias sharing storage via an anonymous union (same approach as `TableProperties::creation_time`, so size, layout, and copy semantics are unchanged). * Public API: `TablePropertiesNames::kOldestAncestorTime` is the new name; `kCreationTime` remains as a deprecated `static const std::string&` alias. A static reference member has no per-object cost and binds at constant initialization, so it adds no static-init-order exposure. * `TablePropertiesNames` as a whole is marked DEPRECATED for public use. An fbsource-wide BigGrep found no references outside RocksDB and its vendored copies, and it is only meaningful to table format internals; users should read the corresponding `TableProperties` fields. Fix: * The `table_file_creation` event log entry had the `oldest_key_time` and `newest_key_time` values swapped since https://github.com/facebook/rocksdb/issues/13083 (v10.0). Now each label is paired with its own value. Also: tools/c_api_gen/auto_simple_bindings.py resets the pending anonymous union field list after use and skips unnamed members inside the union (review suggestion); generated C API output is unchanged. Compatibility: names that are persisted or externally parsed intentionally keep the old spelling: the on-disk property name `rocksdb.creation.time`, the `"oldest_ancester_time"` key in remote compaction serialization, and the `oldest_ancester_time` / `OldestAncesterTime` labels in `VersionEdit` DebugString/DebugJSON (ldb manifest_dump output). The manifest tag value is unchanged. HISTORY.md is left as historical record. Pull Request resolved: https://github.com/facebook/rocksdb/pull/15335 Test Plan: DBTablePropertiesTest.CreationTimeAliasesOldestAncestorTime now also verifies `TablePropertiesNames::kCreationTime` and `kOldestAncestorTime` refer to the same object, and new DBTablePropertiesTest.SstFileMetaDataOldestAncesterTimeAlias verifies the two `SstFileMetaData` names share an address and that writes through the old name are visible through the new one. Remote compaction serialization through the unchanged `"oldest_ancester_time"` key is covered by compaction_service_test; manifest encode/decode through the renamed field by version_edit_test. `make check-c-api-gen` confirms the generated C API is up to date and backward compatible. Reviewed By: jaykorean Differential Revision: D123559299 Pulled By: pdillinger fbshipit-source-id: 480106e8d539e654f31db210ef6e1de02325d604 |
||
|
Xingbo Wang
|
866e8e9f06 |
Make full error-recovery flush atomic (#15293)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15293 Why the change is required: Full background-error recovery flushes all column families because WAL durability can be uncertain. A later recovery-flush failure or process crash must not persist different WAL replay boundaries for different column families. Otherwise reopen can recover only part of a write batch spanning those column families. The risk includes non-file-scoped WAL write or sync errors. A failed write can have reached the WAL without reaching the memtables. An earlier unsynced write can instead remain in the memtables while its WAL bytes are not durable; retrying the fsync after resetting the writer error does not by itself prove those bytes survived. Sequential recovery can make either case visible in only some column families after reopen. Previous behavior: File-scoped WAL-error recovery already set FlushOptions::force_atomic_flush. Other full kErrorRecovery paths flushed column families sequentially when DBOptions::atomic_flush was false. Each successful per-column-family flush independently advanced its MANIFEST log number, so a later flush failure or process crash could leave mismatched replay boundaries. New behavior: Every full kErrorRecovery atomically commits the flush results and WAL replay boundaries for all participating column families. A successful recovery commits every column family; a failed recovery commits none. If an atomic recovery attempt fails, all of its flush output is discarded and recreated by a later attempt instead of retaining output from column families that finished first. Implementation: Set FlushOptions::force_atomic_flush for every full error-recovery FlushAllColumnFamilies call. Remove the now-unreachable recovery-only behavior from FlushMemTable and assert that recovery does not enter the non-atomic path. Keep the two-write-queue sequence fence at the atomic memtable-switch point. Add an end-to-end fault-injection test for a retryable, non-file-scoped WAL sync error after a multi-column-family record is appended. It covers successful and failed recovery flushes, including a column family with no other memtable data, and verifies all-or-none behavior after reopen. Update the manual_wal_flush comment and add a release note. Safety, compatibility, and limitations: Only full kErrorRecovery changes. Normal, manual, shutdown, and kErrorRecoveryRetryFlush behavior is unchanged. File-scoped WAL-error recovery was already atomic. There are no API, ABI, option-default, or file-format changes. The additional work is limited to the cold recovery path. A failed atomic recovery can redo more flush work than the previous sequential path, but this avoids partial multi-column-family recovery. Automatic recovery with manual_wal_flush remains disabled as fatal pending separate analysis and testing; this change does not remove that workaround. Validation covers local Linux builds and focused recovery tests; platform CI for this version is pending. Reviewed By: pdillinger, rban1 Differential Revision: D122469648 fbshipit-source-id: f72845903d9b6bff70f5d881e02a4cb2e920ed72 |
||
xingbowang
|
4cd0450f31 |
Fix big-endian builds for the fault-injection filesystem (#15297)
Summary: The persistent injected-error log writes `InjectedErrorLog::Entry` objects using their native in-memory layout, while the log format's consumer expects little-endian multi-byte fields. The existing `static_assert(port::kLittleEndian)` protected that format by rejecting every big-endian build that included the fault-injection filesystem, even though the log is an optional diagnostic. This change keeps RocksDB and the fault-injection filesystem available on supported big-endian platforms while disabling only the incompatible persistent log. Log-file creation now returns an inactive descriptor when `port::kLittleEndian` is false, which makes `Record()` and `Flush()` remain safe no-ops. The fixed 256-byte entry-size assertion remains in place, and format-dependent tests are bypassed on big-endian targets. The change also adds a release note and documents big-endian support plus the preferred capability-check pattern for optional architecture-specific features. ## Compatibility and limitations - Little-endian behavior and the on-disk log layout are unchanged. - On big-endian platforms, configuring an injected-error log path does not create or populate a log file; the rest of fault injection remains available. - Windows already leaves this POSIX-backed diagnostic disabled. - This does not introduce a portable log encoding. Supporting persistent injected-error logs on big-endian platforms would require a format change. Pull Request resolved: https://github.com/facebook/rocksdb/pull/15297 Test Plan: - `build_tools/rockstest.sh fault_injection_fs_test '--gtest_filter=InjectedErrorLogTest.*'` (little-endian macOS): all 5 tests passed. - `make check-sources` - `git diff --check upstream/main...2026_10_01_endian_fix` Native big-endian build and test execution were not available locally; that path relies on the compile-time `port::kLittleEndian` capability and should be covered by big-endian CI. Reviewed By: jaykorean Differential Revision: D122880760 Pulled By: xingbowang fbshipit-source-id: 59dae9661a47608aa12f41078ad1b3ef0fa7231e |
||
|
Xingbo Wang
|
b58cbcf13c |
Apply transaction visibility to entity reads (#15269)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15269 Why the change is required: Transaction entity reads must enforce the same commit visibility and read-your-own-write rules as value-only reads. Without the transaction-specific read callback, GetEntity-family APIs can return a value that is not visible to the caller. Previous behavior: TransactionBaseImpl handled GetEntity and MultiGetEntity through non-virtual helpers that called WriteBatchWithIndex without a ReadCallback. WritePrepared transactions could expose a prepared but uncommitted update. WriteUnprepared transactions could miss their own write after it had been flushed as an unprepared batch, although that write remained hidden from other transactions. New behavior: GetEntity, MultiGetEntity, and GetEntityForUpdate now follow the same snapshot visibility and read-your-own-write semantics as Get and MultiGet for WritePrepared and WriteUnprepared transactions. Implementation: Make the internal entity-read helper hooks virtual. Override them in WritePreparedTxn and WriteUnpreparedTxn to construct the existing policy-specific ReadCallback and pass it through WriteBatchWithIndex. The WritePrepared batched path validates the callback refreshed read sequence so reads without an explicit snapshot do not bypass eviction validation. Invalid callback or snapshot state returns TryAgain. Add shared transaction entity-read assertions and parameterized regression coverage for prepared visibility and unprepared read-your-own-write. Safety boundaries and unsupported cases: There is no public API or persisted-format change. Other transaction policies keep the existing base implementation. The change adds only the callback and validation already used by value-only reads. Validation covers synchronous snapshot and non-snapshot entity reads, prepared visibility, and unprepared read-your-own-write in normal and checked-Status builds. Coroutine-specific and named wide-column payload cases were not separately tested, and the full repository suite was not run. Reviewed By: rban1 Differential Revision: D121793013 fbshipit-source-id: e300642ed360b640814056e4d95cced5f5c180d8 |
||
Jay Huh
|
ded3cb72a5 |
Fix trace replay on the default column family and error reporting in multi-threaded replay (#15296)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15296 **Why:** trace replay (`DB::NewDefaultReplayer()`, db_bench `replay`) cannot replay a trace of a DB with only the default column family, and multi-threaded replay reports success when records fail. Together these make db_bench replay look successful while replaying no reads. **Previous behavior:** - `TraceExecutionHandler` resolves a record's column family ID only against the `handles` passed to `NewDefaultReplayer()`. Records on a column family missing from `handles`, including the default one, fail with `Corruption: Invalid Column Family ID.`. An empty `handles` also trips `assert(!handles.empty())` in debug builds. - db_bench fills its handle list (`db_.cfh`) only when `--num_column_families` > 1, so with the default of 1 it passes an empty list. Every Get, MultiGet and iterator record fails. Write records are not affected, because `DB::Write()` resolves the column family from the batch itself. - In multi-threaded `Replay()` (`num_threads` > 1), `BackgroundWork()` sends only decode errors to the error callback. Execution errors are dropped, so `Replay()` returns OK. Single-threaded replay stops and returns the error, including a NotSupported from execution (for example `DB::Write()` of a DeleteRange with `row_cache` set). It skips only records whose trace type it cannot decode. - Result in db_bench on a single column family DB: `--trace_replay_threads=1` prints `Replay failed. Error: Corruption: Invalid Column Family ID.`. `--trace_replay_threads=4` prints `Replay completed` with `rocksdb.number.keys.read` = 0. **New behavior:** - The replay handler always includes the DB's default column family, so `handles` may be empty or leave it out. A handle the caller passes for a column family is still the one used. - Multi-threaded `Replay()` reports execution errors through the same callback as decode errors. It stops scheduling new records and returns the error of the failed record with the earliest trace timestamp, as the existing code already does for decode errors. Get records that find no key still count as success, since the handler treats NotFound as OK. - As in single-threaded replay, NotSupported is skipped only when decoding returns it (a trace type the replayer does not support). NotSupported from executing a record stops the replay. - The scheduling loop no longer reads the shared status `bg_s` while worker threads write it. It already did so for decode errors; with execution errors reported, TSAN flags it in the new test (see Test Plan). **Implementation:** - `TraceExecutionHandler` constructor: drop the non-empty assert; after the caller's handles, insert `db->DefaultColumnFamily()`. `unordered_map::insert` does not overwrite, so a caller's handle wins. - `TraceExecutionHandler` constructor: skip the default column family if `DefaultColumnFamily()` returns nullptr (some test `DB` mocks do), so such DBs behave as before. - `ReplayerImpl::BackgroundWork()`: call `error_cb` when `Accept()` returns an error. On a decode error, skip `error_cb` for NotSupported, matching single-threaded replay. - `ReplayerImpl::Replay()`: `error_cb` no longer filters out NotSupported (the decode path now does that); it still keeps the error of the earliest record. - `ReplayerImpl::Replay()`: `error_cb` also sets an `std::atomic<bool> bg_failed` while holding `mtx`, and the scheduling loop checks that flag instead of `bg_s.ok()`. `bg_s` is read only after `WaitForJobsAndJoinAllThreads()`. - Document both contracts in `db.h` (`NewDefaultReplayer`), `trace_record.h` (`NewExecutionHandler`) and `replayer.h` (`Replay`). - Release notes in `unreleased_history/bug_fixes/`. **Safety, compatibility, limitations:** - No API or ABI change. - Callers that already pass the default column family handle see no change from the first fix. - Behavior change: multi-threaded replay callers that got OK while records failed now get the error, including NotSupported from execution. That matches single-threaded replay and the new `Replay()` comment. - A record read by the scheduling loop while a failing worker is setting `bg_failed` may still be scheduled. This is allowed by the `Replay()` comment (records already scheduled still run) and does not change the returned status, which is that of the earliest failed record. - Cost: one extra map entry per replayer, and one branch per record in threaded replay. The loop's `bg_s.ok()` read becomes a relaxed atomic load. - Read records on column families that are neither passed nor the default still fail with Corruption, as before. Write records never used `handles` and are unchanged. Reviewed By: xingbowang Differential Revision: D122626982 fbshipit-source-id: 9a0973448710bac25233feedbaf264696ac85035 |
||
|
Josh Kang
|
aeebcbded4 |
Add coroutine external table reads (#15291)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15291 Add experimental coroutine interfaces for implementing asynchronous basic and full `ExternalTableReader`s. Implementations opt in through `CoroExternalTableReaderBase<Mode>` by providing `GetCoroutine()` and may override `MultiGetCoroutine()`; the default implementation runs the individual Get operations concurrently. The external-table adapter uses these APIs when supported and otherwise retains the existing synchronous behavior. As a smaller supporting fix, Single Get coroutine reads using non-mmap, non-direct I/O on a kFSBuffer-capable `FileSystem` now use `FSRandomAccessFile::SubmitReadAsync()` instead of falling back to synchronous `Read()`. Reviewed By: xingbowang Differential Revision: D122187491 fbshipit-source-id: 217e2953756938bbd3f30358a997a7ca3ed5543e |
||
|
Xingbo Wang
|
31675ca6d0 |
Recover from poisoned WAL handles on remote file systems (#15193)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15193 Keep `DB::Write`'s existing immediate `IOError` contract while allowing RocksDB to recover safely when a remote file system reports that a WAL handle is permanently unusable. The public API documentation makes the persisted-state boundary, reopen behavior, and safe post-resumption retry point explicit. Normal operation remains unchanged; the additional protocol is entered only after a supported file-scoped WAL error. ### Motivation A WAL write crosses an acknowledgement boundary: storage can accept the complete record and then fail before the caller receives confirmation. The resulting `IOError` therefore cannot tell RocksDB or the application whether the record is absent, partial, or complete. At the same time, the remote file handle can become permanently poisoned, so retrying that same handle can never succeed. The previous recovery path retried the poisoned handle. Simply replacing it is not sufficient: RocksDB must also prevent reuse of an ambiguous sequence range, keep every column family on one durable boundary, and prevent WAL tailers from publishing a physically complete record that successful live resumption deliberately abandoned. The new protocol quarantines the affected WALs, reserves the ambiguous sequence range, creates a replacement WAL, atomically persists acknowledged state, retires the quarantined handles, and only then resumes writes. ### Behavior changes #### 1. Recovering from a poisoned remote WAL handle - **Why required:** A remote file system can permanently poison a WAL handle after an I/O failure. Retrying that handle cannot succeed and leaves the database write-stopped. - **Previous behavior:** RocksDB retried the same WAL handle during recovery. - **New behavior:** The remote file system marks the failure as file-scoped. RocksDB creates a replacement WAL, atomically persists all state that could depend on the failed WAL, and then removes the quarantined WAL handles from active use. Physical file deletion remains part of normal cleanup. #### 2. Preserving an indeterminate WAL write outcome - **Why required:** A WAL record can reach storage before the append returns `IOError`; neither RocksDB nor the application can know the outcome from that error alone. - **Previous behavior:** Recovery could reuse a file-scoped failed write group's sequence numbers, allowing a later acknowledged write to overlap a record that might already be in the WAL. The public write API did not explain the acknowledgement ambiguity or distinguish live resumption from close/reopen recovery. - **New behavior:** `DB::Write` still returns the original `IOError` immediately and does not retry the logical batch. Normal DB reads can continue, but the failed write is not visible in that DB session. The persisted DB remains indeterminate with respect to the failed write until supported file-scoped live resumption succeeds without closing or reopening the DB. On reopen, or for any other WAL error without the durable-abandonment guarantee, applications must resolve consistency within RocksDB's documented guarantees, including `WALRecoveryMode`, through idempotency, deduplication, reconciliation, or an equivalent mechanism. After successful supported file-scoped live resumption, RocksDB has durably abandoned the failed batch and reserved its sequence range; retrying the same logical write is then recommended to preserve one linear logical history. #### 3. Keeping TransactionLogIterator consistent with live resumption - **Why required:** A physically complete error-returned record can remain readable from its WAL even though supported file-scoped recovery excludes that batch from the database. Returning it from `GetUpdatesSince()` would make replication or change-data-capture consumers diverge from the recovered DB. - **Previous behavior:** An iterator opened before recovery could return the abandoned batch. A new iterator could also return it while the physical WAL remained visible. - **New behavior:** For file-scoped failures, RocksDB records the exact sequence range and WAL number of each indeterminate write. For the lifetime of that open DB, existing and new WAL iterators stop with `Status::NotFound` before the range, whether the physical record is complete or absent. A caller can begin a new iterator after the reserved range. Iterator creation snapshots the recovery-gap generation together with its WAL-file view so a failure on a later, unseen WAL returns `TryAgain` for the old iterator instead of skipping its valid prefix. #### 4. Keeping all column families on one recovery boundary - **Why required:** A write batch can update multiple column families. Retiring its WAL after independently persisting only some column families can expose a partially applied batch after reopen. - **Previous behavior:** With `atomic_flush=false`, recovery could persist column families independently before abandoning the WAL. - **New behavior:** File-scoped WAL recovery force-flushes every live column family atomically and verifies that each one advanced beyond the failed WAL before writers are retired or writes resume. #### 5. Merging concurrent WAL failures - **Why required:** Another WAL write can fail while an earlier recovery attempt temporarily releases the DB mutex. - **Previous behavior:** A later failed WAL cutoff or ambiguous sequence could be overwritten or cleared by the earlier recovery attempt. - **New behavior:** Each error episode monotonically merges the largest failed WAL cutoff and the sequence range of the latest ambiguous write group. Recovery clears the quarantine cutoff only if no concurrent failure extended it; otherwise the newer failure remains write-stopping for the next attempt. Iterator fencing tracks the actual ambiguous write range separately, so an older failed `SyncWAL()` and a newer ambiguous write are not incorrectly paired. #### 6. Retaining file scope through `NoSpace` recovery - **Why required:** `NoSpace` can still identify one permanently unusable WAL handle. - **Previous behavior:** The `SstFileManager` recovery path entered generic recovery without carrying the failed WAL context. - **New behavior:** File-scoped `NoSpace` carries the same cutoff and sequence evidence through the `SstFileManager` path as other file-scoped WAL errors. #### 7. Scoping recovery context to one error episode - **Why required:** A completed recovery must not affect a later unrelated background error. - **Previous behavior:** WAL recovery context could remain latched after `ClearBGError` succeeded. - **New behavior:** Successful background-error clearing resets the context. A failed recovery retains it so a later retry has the same safety information. The separate iterator gap record remains for the lifetime of the open DB because later WAL readers must continue to reject the abandoned physical record. #### 8. Preserving successful `SyncWAL` metadata - **Why required:** A healthy WAL can finish syncing while a separate recovery error already exists. - **Previous behavior:** `SyncWAL` could return the existing background error before applying the successful WAL's `VersionEdit`, dropping its MANIFEST update. - **New behavior:** `SyncWAL` skips quarantined writers, applies metadata for successfully synced healthy WALs, and only then returns any existing recovery error. #### 9. Classifying only the intended remote-file-system failure - **Why required:** File-scoped recovery is valid only for an actual WAL `IOError` from a handle whose ownership is still valid. - **Previous behavior:** Poison state could be queried before ownership fencing was resolved, and file scope could be stamped onto a non-I/O status. - **New behavior:** Ownership fencing is resolved first, poison state is queried afterward, and only an actual `IOError` can be classified as file-scoped. `IOFenced` remains fatal. #### 10. Preserving `paranoid_checks=false` retry behavior - **Why required:** Non-paranoid mode historically cleared a writer's sticky error so a later operation could retry after storage recovered. - **Previous behavior:** Avoiding reclassification of a cached synthetic writer error also bypassed `WALIOStatusCheck`, so later `SyncWAL` calls kept returning the stale error. - **New behavior:** A cached error still fails the current `SyncWAL` call, but the non-paranoid path resets the writer for a later retry without setting a background error or entering file-scoped recovery. #### 11. Preserving async WAL precreation across recovery - **Why required:** A precreated WAL is an unpublished replacement and is not the active WAL. Recovery can also consume the prepared slot while background work is stopped. - **Previous behavior:** A file-scoped failure while creating or starting the unpublished replacement was attributed to the active WAL, unnecessarily quarantining it. After successful recovery consumed a prepared WAL, the next slot was not replenished, so the following rotation created its WAL synchronously. - **New behavior:** Only an active WAL `WriteBuffer()` failure attaches the active-WAL cutoff. A failed unpublished candidate is discarded while the current WAL remains active, and successful recovery schedules the next async precreation after `ClearBGError()` re-enables background work. ### Implementation - Carry `DBRecoverContext` through generic, `NoSpace`, and `SstFileManager` recovery, merging WAL cutoffs and failed write-group sequence ranges independently of error severity. Generic errors retain the range only so a later retry that establishes file scope can activate the stronger protocol; they do not persist a reservation or install an iterator gap on their own. - Document the WAL acknowledgement ambiguity, persisted-state boundary, reopen behavior, application consistency requirements, and post-resumption retry guidance on the public `DB::Write` API. - Publish a lock-free quarantine cutoff used by write and sync paths; replace even an empty failed current WAL; persist file-scoped ambiguous sequence reservations; force and verify an atomic all-column-family flush; and detach quarantined writers only after that durable boundary. - Record each file-scoped ambiguous write group's exact WAL number and sequence range in a DB-lifetime tracker shared by `WalManager` and all `WalIteratorImpl` instances. Stop an iterator before a batch that contains or crosses the range, and use a generation snapshot to preserve the existing `TryAgain` behavior when an iterator's fixed WAL-file view predates the failure. - Keep successful healthy-WAL tracking edits in `SyncWAL`, reset completed recovery context, and preserve legacy non-paranoid writer reset behavior. - Distinguish failures on an unpublished replacement WAL from failures flushing the active WAL. Log the failed replacement identity, keep the active WAL out of quarantine, and replenish the async-precreation slot after successful recovery. - Classify poisoned remote-file-system handles after ownership fencing and only for `IOError`. - Add lifecycle logging and a state-machine comment covering quarantine, sequence reservation, iterator fencing, replacement, atomic persistence, retirement, and concurrent-error handoff. - Add an opt-in `db_stress` mode that injects file-scoped faults only into WAL writes, disables WAL tracking in that mode, validates unsupported direct invocations, and sanitizes crash-test parameters to the supported recovery envelope. A small fraction of ordinary non-specialized crash-test runs select this mode automatically; existing stress behavior is unchanged when the flag is off. - Make the concurrent-error regression test hold the initial background flush until `SyncWAL` records the intended older-WAL error, preventing scheduler-dependent retirement of the test WAL before fault injection. ### Safety boundaries and unsupported cases - Automatic file-scoped WAL recovery fails closed with `NotSupported` for manual WAL flush, 2PC/transactions, two write queues, pipelined or unordered writes, WAL recycling, WAL TTL or size retention, either WAL-tracking mode, or blob direct writes. - Unsupported recovery leaves the DB write-stopped. `DB::Resume()` returns `NotSupported`; after correcting the I/O condition, operational recovery requires close and reopen. - The failed logical batch is never retried. Only supported file-scoped recovery durably reserves its sequence range and fences `GetUpdatesSince()`. Generic WAL errors retain the existing indeterminate outcome and can replay after close/reopen; applications must use idempotency or reconciliation unless successful file-scoped live resumption has provided the stronger guarantee. - Failed recovery attempts retain their context. Only a successful `ClearBGError` resets it. - Physical WAL deletion is not part of the recovery correctness boundary; normal obsolete-file cleanup removes files later. - The TransactionLogIterator ambiguity fence is kept for the lifetime of the DB instance that observed the file-scoped error and is not persisted as new MANIFEST metadata. After a later close/reopen, normal WAL recovery and obsolete-WAL retention/deletion semantics apply. - The in-memory iterator-gap tracker can grow with the number of recovered file-scoped failures during one DB lifetime. Safe pruning must account for retained WALs and existing iterator snapshots; it is left for follow-up. - Async WAL precreation remains disabled with WAL recycling. A prepared WAL is unpublished until its start records succeed; failure before installation does not make the active WAL file-scoped. - The new stress mode is test-only and defaults off, so normal `db_stress` and database operation without an error are unchanged. - Automatic stress-profile selection is limited to fully parsed, supported configurations. Explicit targeted runs reject options files and other unsupported combinations. ### Tests and known limitations - The affected recovery, WAL-manager, WAL-iterator, fault-injection, async-WAL-precreation, and secondary-close binaries build successfully. - Checked-status suites pass for all 70 `error_handler_fs_test` cases, all 7 `wal_manager_test` cases, all 11 `db_log_iter_test` cases, all 8 `fault_injection_fs_test` cases, and all 5 `DBWALTest.AsyncWalPrecreate*` cases. `DBSecondaryTest.SecondaryCloseFiles` passes 20 consecutive repetitions. - Focused regressions cover file-scoped gap fencing after an iterator snapshot, generic errors not installing a file-scoped sequence gap, and later recovery discovering that a generic error is file-scoped. - Earlier validation on this diff also passed crash-test unit coverage, remote-file-system adapter tests, repeated concurrency tests, two bounded multi-column-family stress runs, and bounded black-box crash/reopen dry runs. - The exact internal RocksDB format check and `arc f --take-formatters --enforce-lint-clean` report no formatting or lint issues. - The full crash/stress matrix and TSAN were not run. Reviewed By: pdillinger Differential Revision: D119324616 fbshipit-source-id: 05779d2bff7018f0ae69c4cbbe5b6455a45675d6 |
||
|
Josh Kang
|
dbc2db5228 |
Make external table APIs more consistent with native tables (#15241)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15241 Native `TableFactory` implementations receive the RocksDB `WritableFileWriter` in `NewTableBuilder()` and `RandomAccessFileReader` in `NewTableReader()`. These wrappers provide buffering, checksums, rate limiting, statistics, listeners, tracing, and direct-I/O handling. External table implementations previously received a limited writable-file adapter and reopened readers from a path, bypassing parts of this wrapper logic. This change makes `WritableFileWriter` and `RandomAccessFileReader` implement `FSWritableFile` and `FSRandomAccessFile`, respectively, so the same RocksDB wrappers can be passed to external tables. `ExternalTableFactory::NewTableReader()` now receives the already-opened reader and file size. The filesystem interfaces expose standard append, flush, sync, fsync, file metadata, synchronous reads, asynchronous reads, and batched reads. `RandomAccessFileReader::SubmitReadAsync()` schedules its wrapped coroutine read on the configured filesystem read executor, and the DB table-cache and `SstFileReader` construction paths pass that filesystem through. `SstFileReader` also supplies its configured statistics, listeners, rate limiter, and file temperature, removing the need for a separate external-table metrics filesystem wrapper. For direct I/O, `MultiRead()` checks whether requests are already sorted. When alignment and merging are needed, it constructs a sorted index only for unsorted input, preserving caller request order and result mapping. Fully aligned requests using aligned caller buffers bypass this path and may be supplied in any order. Reviewed By: xingbowang Differential Revision: D121035539 fbshipit-source-id: 27cb897726c31cb1a4dc811c65c8d79e690d82b1 |
||
|
zaidoon
|
3c30d43baf |
Promote IndexFactory as UDI SPI (#14962)
Summary: Make `IndexFactory` the public interface for custom indexes. Keep the old `UserDefinedIndex` names, header, and `NewBuilder()` / `NewReader(Slice&)` overloads so existing implementations can still compile. UDI and options tests pass on the full stack, both normally and with `ASSERT_STATUS_CHECKED=1` and `COERCE_CONTEXT_SWITCH=1`. The C API test and generation/compatibility checks pass. Full `make check` runs had test timeouts and hit the 30-minute local limit. These test results are from before the rebase. Part 5 of 9. Previous: https://github.com/facebook/rocksdb/issues/14961 (landed). Next: https://github.com/facebook/rocksdb/issues/14963. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14962 Reviewed By: jaykorean Differential Revision: D119960591 Pulled By: xingbowang fbshipit-source-id: 9703950bd48723c2c524da89cbea63bf00254cc1 |
||
|
Jay Huh
|
bcf8711165 |
Add per-column-family ReadOptions to NewIterators (#15230)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15230 Add a raw C++ `DB::NewIterators()` overload accepting parallel vectors of `ReadOptions` and column family handles. Each iterator receives its own options while all iterators share one consistent sequence snapshot. Require identical snapshot pointers and tailing modes across the options. Implement the overload for primary, secondary, read-only, stackable, and write-prepared DBs. Building on D120048299, both secondary overloads share one helper that captures all SuperVersions and the sequence number under the DB mutex. Preserve read-only `table_filter` support when range-tombstone conversion is configured, and reject that unsafe combination in write-prepared read-write iterators before allocating any iterator. Add validation, per-CF prefix-seek coverage, a secondary catch-up regression test, table-filter contract tests, and a public API release note. Reviewed By: xingbowang Differential Revision: D120296578 fbshipit-source-id: 3775884537ea29a0d32bb3fe10330cf62ac5a406 |
||
|
Xingbo Wang
|
4c2a977aa5 |
Track newer snapshot versions in RocksDB metadata reads (#15086)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15086 Why: An application built on top of RocksDB could use cache population reads through explicit snapshots and needs to know when the returned value has already been superseded, without a second point read. Previous behavior: Get and MultiGet returned the snapshot-visible value and optional timestamp but could not report a newer point update or covering range deletion. New behavior: experimental C++, C, and Java metadata-read APIs report newer committed puts, deletes, merges, and covering range tombstones; ordinary primary-DB writes completed before the read are guaranteed to be reported, racing writes remain best-effort, and true has no false positives. Implementation: metadata reads widen lookup behind read-callback hooks while keeping metadata-only state off passive common object layouts; per-key state stays in KeyContext/GetContext, ordinary MultiGet row-cache replay semantics are preserved, BlobDB/TTL use shared output helpers, dead single-CF plumbing and an invalid cross-key stress assertion are removed, and the range-tombstone ordering invariant is documented without changing behavior. Safety and unsupported cases: tracking is opt-in and requires an explicit snapshot; persisted-tier reads, custom transactional visibility callbacks, secondary DBs, and unsupported subclasses return NotSupported, while read-only/compacted DBs return false because their in-process views are static. Tests and limitations: 1,265 Buck tests plus focused metadata, 100x coerced-context-switch row-cache, write-committed/write-prepared crash tests, builds, and lint passed. Current-parent release db_bench found no material passive regression: readrandom +0.65%, MultiGet batch 4 -1.40%, and batch 32 +0.48%. Reviewed By: pdillinger Differential Revision: D104182397 fbshipit-source-id: 36f7be1cd50fe2383689adaae31a6bb7d7e462dc |
||
Omkar Gawde
|
88a8fdf849 |
Compare excluded-file lookup against the engine it searched (#15197)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15197 **What**: Compare the excluded-file lookup iterator against the engine it was obtained from, in `BackupEngineImpl::RestoreDBFromBackup()`. **How**: The loop walks `locked_restore_from_dirs` calling `find()` on each engine's `backuped_file_infos_`, but tested the result against `this` engine's `end()`. Test against the same engine's `end()`. **Why**: Comparing iterators obtained from different containers is undefined. It is inert here rather than a live bug: on libstdc++ and libc++ an `unordered_map` end iterator is a null sentinel that is not container-specific, so a miss compares equal and a hit does not. On an implementation with a container-specific end iterator - MSVC, a required CI axis - a miss would read as found and dereference the sentinel node. No test can pin this; see the test plan. Reviewed By: xingbowang Differential Revision: D119585045 fbshipit-source-id: 9ff0e770b426f6a2c4a0fdad261910a927d1bfa7 |
||
|
Josh Kang
|
ac0f4ae69f |
Isolate coroutine statistics across async reads (#15142)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15142 Coroutine statistics use TLS, but coroutine reads can interleave on the same executor thread. Previously, when a stats-enabled request suspended, it saved its counters without disabling the executor's TLS configuration. A stats-disabled request running next could inherit those enabled settings and collect statistics unexpectedly. Stats setup also lived in individual DB implementations, so wrapper early returns and future stackable DB implementations could bypass it. Move statistics ownership to the public `CoroDB` and callback-based async read boundaries, before virtual dispatch. Each operation captures its caller's configuration, enabled requests preserve their counters across suspensions, and every suspension or completion leaves executor TLS disabled. Because the scope wraps stackable DB dispatch, current and future `CoroStackableDB` implementations automatically inherit the correct behavior without adding their own stats reset or scope. For consistency between coroutine and callback-based reads, the experimental callback API now uses TLS for both configuring and consuming statistics. `AsyncCallback::OnComplete()` no longer receives context arguments; RocksDB publishes the completed per-operation counters to TLS before invoking it. ``` Before time --------------------------------------------------------------> stats-on A: onSet(enable A) -- work -- onUnset(save A) ...... onSet(A) executor TLS: [A enabled] ------------> [still enabled] ------> [A enabled] stats-off B: run ^ inherits A's enabled TLS After time --------------------------------------------------------------> stats-on A: onSet(enable A) -- work -- onUnset(save A, disable) ... onSet(A) executor TLS: [A enabled] ------------> [disabled] ----------------> [A enabled] stats-off B: run ^ remains stats-disabled ``` Reviewed By: xingbowang Differential Revision: D117017089 fbshipit-source-id: 8a85f7ba43e3b73225995f5d0b79da3892baafb9 |
||
Ranjan Banerjee
|
fe5123681e |
Fix GetEntity snapshot isolation for WritePrepared transactions (#15136)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15136 `WritePreparedTxnDB` overrides `Get` and `MultiGet` to install a `WritePreparedTxnReadCallback` that performs visibility checking for prepared-but-uncommitted data. However, `GetEntity` (both the single-CF `PinnableWideColumns` variant and the multi-CF `PinnableAttributeGroups` variant) was never overridden, causing it to fall through to `StackableDB::GetEntity` which delegates to the inner `DBImpl` without any WritePrepared visibility callback. This means `GetEntity` could read prepared-but-uncommitted data, violating snapshot isolation. The bug surfaces in the crash test with the combination: `--use_txn=1 --txn_write_policy=1 --use_attribute_group=1 --use_get_entity=1 --unordered_write=1 --two_write_queues=1 --test_batches_snapshots=1` The fix adds `GetEntity` overrides to `WritePreparedTxnDB` that create a `WritePreparedTxnReadCallback` and pass it via `GetImplOptions::callback`, matching the pattern used by the existing `Get` override. Reviewed By: xingbowang Differential Revision: D116324596 fbshipit-source-id: faf636cfbf093f47a645c1a5d5037baa02ff594a |
||
|
Xingbo Wang
|
e6a2ee0bd2 |
Fix subset checkpoints with incomplete MANIFEST tails (#15123)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15123 Subset checkpoint post-processing appends column-family drop records to a copied MANIFEST. Recovery can tolerate an incomplete atomic group at the tail while retaining the reader high-water size, so appending at that size can produce a checkpoint that fails recovery. Track the valid logical append boundary together with its MANIFEST file number, refresh it after recovery and successful writes, reset it across reactive MANIFEST switches, and trim only subset-checkpoint copies before appending drop records. Whole-DB checkpoints and the source MANIFEST remain unchanged. Reviewed By: anand1976 Differential Revision: D116449394 fbshipit-source-id: a162dd6a2558f0f9ec40d878ee83956a89df698b |
||
|
Peter Dillinger
|
8f492b2f47 |
Store common user-key prefix once per block (format_version 8) (#15117)
Summary:
Building on D116105558, this change finishes and publishes `format_version = 8` as
production-ready.
The headline feature is a tweaked block encoding that stores the common user-key
prefix shared by all keys in a block **once**, in the block's leading bytes
`[0, restarts[0])`, instead of repeating it at every restart point. This shrinks
prefix-heavy blocks (and their block-cache footprint) and, for bytewise-ordered
keys, speeds up `Seek` by comparing prefix-stripped suffixes. It is applied to
**data blocks** and **every index tier** (leaf/partition index, the
partitioned-index top-level, and the partitioned-filter top-level).
Motivation: MyRocks/ZippyDB/timeseries-style keys carry a long shared prefix
(table/index id, partition, etc.). Repeating it at each restart wastes space and
cache, and every Seek re-compares those bytes. Profiles show index-block seek CPU
is comparable to data-block seek CPU (partitioned index does two index seeks per
data seek), so index blocks are a first-class target, not just data blocks.
### Format (self-describing, no new footer bit)
- The common prefix occupies the block's leading bytes; a non-zero `restarts[0]`
(the offset of the first entry, i.e. the prefix length) self-signals its
presence. `restarts[0]` is 0 in every block that does not use the feature (all
versions, all block types), so a single fv8 reader transparently handles any
mix of stripped and unstripped blocks -- no footer bit is consumed.
- Previous versions would misinterpret this encoding by quietly ignoring the
common prefix, which (along with the lack of extensibility in data-block
metadata) is why it is gated behind format_version 8.
- Only restart-point entries change on disk; non-restart entries are byte
identical to fv7, so the writer produces the exact same bytes regardless of
implementation.
### Public option
New `BlockBasedTableOptions::optimize_key_common_prefix`
(`enum class OptimizeKeyCommonPrefix`), default `kIfFastSeek`:
- `kDisabled` -- never use it.
- `kIfFastSeek` (default) -- use it only where it also enables the Seek speedup:
format_version >= 8 with a built-in (reverse-)bytewise comparator.
- `kEnabled` -- use it for all comparators at fv8; (reverse-)bytewise get space +
fast seek, other comparators get the space savings only (the reader
reconstructs full keys and compares with the real comparator).
Expressed as a criteria so it need not be toggled per CF based on the comparator
(which can vary by CF). It additionally requires `use_delta_encoding` (the
feature builds on delta encoding) and `ts_sz == 0`; when `use_delta_encoding` is
false -- the setting that also preserves zero-copy key pinning -- it is a no-op.
Default format_version is unchanged (still 7); the feature is opt-in via fv8.
### Scope: every index tier
The reader handles all index-like blocks uniformly (a non-zero `restarts[0]`
self-signals a stripped block regardless of block type), so the writer strips the
common prefix from every tier:
- **Data blocks** and **leaf / partition index blocks**: enabled.
- **Partitioned-index top-level block**: enabled. Under the default
`index_shortening = kShortenSeparatorsAndSuccessor` the file's last separator
would otherwise be a short successor sharing no leading bytes, collapsing the
single-block top-level's common prefix to 0. When common-prefix is on the
builder suppresses that one successor (keeps the full last key -- i.e.
`kShortenSeparators` behavior for that one entry), so the top-level strips
uniformly under every shortening mode; the full last key is also a tighter file
upper bound, so it is not worse.
- **Partitioned-filter top-level index**: enabled (same uniform treatment). Its
space/CPU effect is small at default partitioning (the top-level filter index
is tiny) but grows with many small partitions -- a filter miss short-circuits
at, and is dominated by, the top-level filter-index seek (see Performance).
Included for a uniform on-disk surface across tiers.
Bloom filter *contents* still hash full user keys, so prefix/whole-key filtering
is unaffected.
### Implementation
- Writer (`block_builder.{h,cc}`): keys are encoded incrementally in the fv7
layout while the running common prefix is tracked cheaply (reusing the delta
`shared`); at `Finish()` only the restart-point entries are rewritten in place
with the prefix stripped, and the prefix is written once at the block start.
The rewrite is generalized to the value-delta (fv4) index layout used by index
blocks (no `value_size` header; bulk-copy the inline BlockHandle and
non-restart entries verbatim).
- Reader (`block.{h,cc}`, `dbformat.h`): `Block` decodes the prefix from
`restarts[0]`; `BlockIter::ParseNextKey` prepends it on a restart point
(`shared == 0`). The suffix-seek helpers (strip target once, compare shortened
suffixes) live in the templated `BlockIter` base and are shared by data and
index iterators (binary search, templated on `DecodeKey`/`DecodeKeyV4`); for
non-bytewise comparators (data) and the hash `PrefixSeek` refine step, the
reader reconstructs the probed restart key and compares full keys.
- Gating (`block_based_table_builder.cc`, `index_builder.{h,cc}`,
`partitioned_filter_block.{h,cc}`): the criteria is threaded to the data-block
builder and to every index tier -- leaf/partition index, the partitioned-index
top-level (`use_common_prefix_top`), and the partitioned-filter top-level --
together with the last-separator successor suppression
(`ShortenedIndexBuilder::GetSeparatorWithSeq`) that lets the top-level strip
under the default shortening mode. The index base gate also requires
`super_block_alignment_size == 0` (block alignment can trigger
`skip_delta_encoding`, producing non-restart `shared == 0` entries the reader
could not distinguish from stripped restart keys).
- Memory: the per-block/per-iterator prefix is stored as a `uint32_t` length
(the prefix is always at offset 0) rather than a full `Slice`, and `Block` /
`BlockBuilder` members were repacked so the feature adds little or no struct
size.
- Java: a new `OptimizeKeyCommonPrefix` enum and a `BlockBasedTableConfig`
getter/setter, plumbed through JNI in both directions (the
`newTableFactoryHandle` writer and the options-reconstruction constructor).
### Whole-file (table-level) prefix: investigated and rejected
A natural extension is to store the whole-file common prefix once per *file*
(when an `SstPartitioner` guarantees every key in the file shares a fixed prefix)
instead of once per *block* -- e.g. the MyRocks index/table id, or a future
128-bit global id. It was prototyped end-to-end (partitioner-provided prefix
captured at file open, stored in a `user_key_common_prefix` table property, and
stripped from every data/index block; the reader re-prepends it) and then
dropped. The `user_key_common_prefix` table property reserved in the
previous diff is dropped in this one. The rationale, in case it is revisited:
- Marginal benefit over per-block. Per-block already strips the full shared
prefix from every key; the only thing whole-file removes on top is the single
per-block copy of that prefix, i.e. `~= prefix_len / block_size` of the
uncompressed (block-cache) footprint, with no meaningful read-CPU change
(per-block already delivers strip-once + suffix compares). Measured at
prefix_len=12, 4 KB blocks this was ~-0.3% incremental space; at the 8-16 KB
data blocks typical of the space-constrained target (MyRocks/UDB) it is
~0.1-0.2% even with a 16-byte global id -- a fraction of a percent, seen only
by partitioner users. Small block-size workloads are becoming more
niche, and even where they exist, they are also CPU-sensitive (see next
paragraph).
- Real, if small, CPU tax on the *per-block* path. Reconstructing a key from two
non-contiguous parts (file prefix + block residual) cost ~0.5-1.0% additional
retired instructions on the per-block read path (deterministic across gcc and
clang), wall-clock-neutral in cache-resident tests but present even when no
whole-file prefix is configured. Attempts to remove it did not pan out: a
compile-time specialization of the reconstruction *increased* instructions
(+2.9%) by defeating inlining; a per-block combined-prefix buffer trades the
CPU cost for a memory cost on every (default) block; and a per-iterator buffer
was the original larger regression. Only invasive iterator-level
monomorphization would truly zero it, which is not justified on this hot path.
- Decision. A fraction-of-a-percent, partitioner-only space win does not justify
a permanent per-block-read-path CPU tax plus additional on-disk format surface
(a table property that changes whole-file interpretation). Dropped. The generic
`SstPartitioner::ShouldPartitionByPrefix` hook is kept (cheap, and useful
independently), and the reserved `user_key_common_prefix` table property is
removed since no writer will emit it.
### Performance (db_bench, release/native, cache-warm, best-of-3, pinned)
Three arms: (A) fv7 on the base/main branch, (B) fv7 with this change (should be
neutral / byte-identical writing), (C) fv8 with this change and
`optimize_key_common_prefix=enabled` (the optimization is active). Partitioned
index + filters; `compression_type=none`, so SST bytes are the uncompressed
block-cache footprint. `int` = db_bench default keys (variation-early, near worst
case); `pfxN` = an N-byte shared prefix + varying suffix (MyRocks-like).
Setup: `num=3,000,000`, `reads=1,500,000`, single thread,
`metadata_block_size=4096`, cache holds the working set, pinned to one core.
| shape (key/val) | arm | SST bytes | readrandom us/op | seekrandom us/op |
|---|---|---:|---:|---:|
| int (32/8) | A fv7 main | 138,983,658 | 3.484 | 4.995 |
| | B fv7 ours | 138,983,146 | 3.518 | 5.024 |
| | C fv8 ours | 137,681,420 | 3.490 | 4.986 |
| pfx24 (32/8) | A fv7 main | 71,697,531 | 3.417 | 5.561 |
| | B fv7 ours | 71,698,043 | 3.448 | 5.579 |
| | C fv8 ours | 65,578,236 | 3.284 | 5.386 |
| pfx56 (64/16) | A fv7 main | 102,942,266 | 3.624 | 5.830 |
| | B fv7 ours | 102,942,266 | 3.678 | 5.835 |
| | C fv8 ours | 90,554,642 | 3.444 | 5.626 |
fv8 vs fv7-main (C vs A) -- space / readrandom / seekrandom:
- int (32/8): **-0.9%** / ~neutral / ~neutral
- pfx24 (32/8): **-8.5%** / **-3.9%** / **-3.1%**
- pfx56 (64/16): **-12.0%** / **-5.0%** / **-3.5%**
fv7-ours vs fv7-main (B vs A): SST bytes byte-identical (within 512 B; the fv8
paths are gated off), read/seek us/op within run-to-run noise (~1%, best-of-3 on
a shared host). Net: fv8 is a space **and** read/seek-CPU win on prefix-heavy
data and neutral on variation-early (`int`) data, so `kIfFastSeek` is safe on by
default at fv8.
### Space by tier (compression=none, num=3M, mdb=4096; fv7 -> fv8)
Where the win comes from -- the index tiers strip the most:
| shape | data | leaf/partition index | top-level index | filter |
|---|---:|---:|---:|---:|
| int | -0.8% | -29% | -45% | -0.2% |
| pfx24 | -8.3% | -69% | -89% | -0.8% |
| pfx56 | -11.1% | -83% | -96% | -1.6% |
Data bytes dominate the absolute total, but every index tier is stripped:
leaf/partition index -29% to -83% and the partitioned-index top-level -45% to
-96%. Filter *blocks* barely move (-0.2% to -1.6%) because filter bits hash full
user keys; only the small top-level filter index is stripped.
### Point-miss lookups (readmissing)
A partitioned-filter miss short-circuits at the filter (no data-block read), so
it is dominated by the top-level filter-index seek plus the partition bloom. At a
typical `metadata_block_size=4096` (few partitions, tiny top-level filter index)
fv8 is neutral on misses (int +0.9%, pfx24 -1.0%, pfx56 -0.7% us/op -- all
noise). The top-level-filter stripping pays only with many small partitions: at
`metadata_block_size=256` it is pfx24 1.166->1.138 (**-2.4%**) and pfx56
1.273->1.188 (**-6.7%**) us/op.
### Backward / forward compatibility
- Default `format_version` remains 7; the feature is inert unless a CF opts into
fv8. Older RocksDB rejects fv8 files at footer decode (fail-safe).
- Within fv8, the stripped-block set a writer can emit is fixed and every fv8
reader can read it (the `restarts[0]` signal plus uniform reader handling), so
the on-disk surface is self-consistent across builds/downgrade.
### Follow-ups (not in this change)
- Suffix-aware interpolation / hash index seek (currently correct via the
reconstruction fallback, not perf-tuned).
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15117
Test Plan:
- New focused unit tests (`table_test`), run at the full `format_version` matrix
including the now-supported fv8:
`GeneralTableTest.OptimizeKeyCommonPrefix` (data-block space+read+Seek
correctness: existing and missing keys straddling the prefix, before/after-all,
SeekForPrev), `OptimizeKeyCommonPrefixNonBytewise` (reconstruction path), and
`IndexBlockCommonPrefixEquivalence` across `{kBinarySearch,
kBinarySearchWithFirstKey, kHashSearch, kTwoLevelIndexSearch}` x
`{partitioned filters}` x `{bytewise, reverse-bytewise}` (so both top-level
tiers -- partitioned-index and partitioned-filter -- are exercised), asserting
identical reads and a shrunken fv8 index.
- The randomized `ParameterizedHarnessTest` now runs at fv8 (reverse-bytewise,
restart intervals `{1,16,1024}`, special/empty keys fuzzed against the feature).
- `db_crashtest.py` randomizes `format_version` including 8 and
`optimize_key_common_prefix` in {0,1,2}.
- Removed the reserved-property carve-out: dropped the `ReservedUserKeyCommonPrefix`
reader test and the C API getter for the property; C API regenerated via
`tools/c_api_gen/` (`make regen-c-api`), and the intentional public-C-API
removal is recorded in `tools/c_api_gen/api_compatibility_allowlist.json`.
- Java: `BlockBasedTableConfigTest.optimizeKeyCommonPrefix` (default
`kIfFastSeek`, plus set/get of every value).
- `table_test` `FooterTests` updated for the now-published fv8 footer, whose
metaindex-gap encoding stores the metaindex block size in 16 bits (the test
caps that size for fv >= 8).
- Manual verification: the db_bench runs in Performance above (three-arm
space + read/seek CPU, per-tier byte breakdown via `sst_dump`, and
`readmissing`); and `readrandom`/`seekrandom` with
`--block_protection_bytes_per_key=8` find 100% of keys at both fv7 and fv8
(reconstructed keys pass per-key checksum).
Reviewed By: xingbowang
Differential Revision: D116365016
Pulled By: pdillinger
fbshipit-source-id: 66f4b5c60ea4fe0ff48d34c98d191ba8ebdf0665
|
||
|
Xingbo Wang
|
bbdcd2825f |
Add remote storage liveness crash test signal (#15102)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15102 Add a dedicated Skycastle signal that invokes liveness through the existing remote-storage test-driver path. For remote DB runs, double default wrapper/no-progress timeouts while preserving explicit overrides. Default remote liveness `expected_values_dir` to local storage so `UniqueIdVerifier` never treats a remote DB path as a local path; local liveness behavior remains unchanged. Normalize both equals and spaced `--env_uri`/`--fs_uri` forms once so remote detection, workload execution, and cleanup share the same arguments. Track the crash harness and internal release scripts as CI source triggers, scoped so release-script changes select only the remote-storage signal. Add command-level regression coverage and failure-safe expected-values cleanup. Use generic remote DB/storage terminology throughout public RocksDB code and comments. Reviewed By: pdillinger Differential Revision: D115936164 fbshipit-source-id: e4997bf19eec18e4914626d479690b78c6cd3ad2 |
||
|
Peter Dillinger
|
f0c1e3d7d9 |
Fix cleanup-on-failure and flaky liveness check in point_lock_manager_stress_test (#15116)
Summary: The point_lock_manager_stress_test (via PointLockValidationTestRunner) was seen flaky under USE_CLANG=1 COMPILE_WITH_ASAN=1 COMPILE_WITH_UBSAN=1, manifesting as a heap-use-after-free plus "terminate called without an active exception" / SIGABRT. There were two distinct problems. Cleanup on failure: run() uses gtest ASSERT_* macros, which return early from run() on failure. That early return skipped the shutdown flag set and the join of the worker threads, so the worker threads kept running and accessed members of the (now destroyed) test runner object (heap-use-after-free), and destroying the still-joinable threads_ vector called std::terminate. Fixed by adding a destructor plus a StopAndJoinThreads() helper that always signals shutdown and joins the worker threads before any member is destroyed, no matter how run() exits. Copy/move are deleted since the object owns worker threads. Underlying flaky failure: the per-second "No locks were acquired in the last 1 second" liveness assertion was too strict. Under sanitizer builds combined with very short lock timeout/expiration and post-acquisition sleeps, all threads can occasionally fail to acquire a single lock within a given one-second window without the lock manager actually being stuck. The per-thread variant of this check was already disabled for this reason; the aggregate one is now relaxed to tolerate a few consecutive no-progress seconds, which still reliably detects a genuinely stuck (dead/livelocked) lock manager. Also fixed the end-of-run check from `>= 0` (always true) to `> 0` so it actually catches a fully stuck manager. Pull Request resolved: https://github.com/facebook/rocksdb/pull/15116 Test Plan: Verified the cleanup path by temporarily forcing the liveness assertion to fail while worker threads were active: the test now reports a normal gtest failure with clean teardown, with no heap-use-after-free and no std::terminate/SIGABRT. I don't have a consistent reproducer for the failure but ran * 3x `USE_CLANG=1 COMPILE_WITH_ASAN=1 COMPILE_WITH_UBSAN=1 make check` * `USE_CLANG=1 COMPILE_WITH_ASAN=1 COMPILE_WITH_UBSAN=1 rocksptest point_lock_manager_stress_test -r100` Reviewed By: xingbowang Differential Revision: D116349420 Pulled By: pdillinger fbshipit-source-id: 104784c5c59770414224ac4751eabf69f65e0958 |
||
|
Peter Dillinger
|
021374c62c |
Start format_version=8 development toward new features (#15104)
Summary: format_version background: SST files are shared across RocksDB versions. format_version is a critical tool in ensuring prior (and later) versioned readers do not misinterpret a file created by some version. Opt-in features can sometimes rely on previously-rejected corners of the schema space, but not always. This change: lands much of the format_version=8 (fv8) format surface as an internal-only draft -- readable and testable but not user-writable -- so upcoming features can build on a stable base. fv8 is NOT published: kLatestBbtFormatVersion stays 7, and reading or writing fv8 is gated behind the existing TEST-only TEST_AllowUnsupportedFormatVersion() hook. Publishing fv8 (raising the ceiling) is deferred to a later change. The pieces of fv8 added here, and what each unlocks: - Index value-delta escape. Existing behavior: index blocks (format_version >= 4) store each block handle as a delta against the previous entry's handle, and signal "this value is a full handle instead" by reusing the key sentinel `shared == 0`. Because the two are coupled, an index entry whose handle is not contiguous with the previous one must be forced to a restart, costing a full key as well as a full value. fv8 decouples them with an in-value escape codepoint: varint 0 means "a full BlockHandle follows," and genuine size deltas are remapped to zigzag(delta)+1 so that 0 stays free. A non-contiguous entry can then remain an ordinary key-delta-encoded, on-cadence entry. The escape applies to every index-shaped block, which includes the top-level index of index partitions and the index of filter partitions, not just the index over data blocks. Unlocks: the upcoming optimize_key_common_prefix index feature (removes its need to special-case super_block_alignment) and value-delta encoding for embedded-blob SSTs (previously disabled outright because interleaved blob records break handle contiguity). Guarded strictly on fv8; format_version <= 7 encoding and decoding are byte-for-byte unchanged. Note that restart points must not delta-encode, so they still have full values without a sentinel. - Data block footer bit 30 is reserved as an "extended metadata present" escape. Decode rejects a set bit as unsupported/corrupt at all versions. Unlocks: adding block-level metadata in the future through the one shared footer decoder, without another format_version bump or plumbing format_version into every block parser. (Older reader versions relied on this bit being zero for some integrity/consistency checking; format_version bump now makes it safe to use.) - Footer metaindex gap. For fv8, the top 16 bits of the footer's 4-byte metaindex-size field record the gap (in bytes) between the end of the metaindex block and the start of the footer; the low 16 bits hold the metaindex block size. Unlocks: a space very near the end of the file to save full file checksum state, for a "files carry their own checksums" feature that doesn't require SST parsing to extract the checksum states (to run them to completion on the trivial file tail). - Reserved "user_key_common_prefix" table property. The reader returns NotSupported when it is present and non-empty. Unlocks: a future whole-file optimization that stores the common prefix of all user keys once instead of in every restart key; until implemented, fv8 readers refuse such a file rather than mis-read it. - Because of the problem of inferring such a prefix at the start of constructing an SST file, we expect it will rely on SstPartitioner. Here we've added SstPartitioner::ShouldPartitionByPrefix() (not yet used) that will enable this optimization, and avoid the overhead of a virtual callback to SstPartitioner on every key written to an SST file. Supporting cleanups: the Footer's "unset" format_version sentinel is named (kInvalidFormatVersion) and the FormatVersionUses*() predicates assert against it, as each is a `version >= N` test that an unpopulated Footer would silently satisfy. Index block reader entry points (Block::NewIndexIterator() and friends) drop their defaulted trailing arguments so each caller states which codec it expects, and BlockBuilder::Add() takes delta_value by const reference rather than a pointer whose null case no caller used. Pull Request resolved: https://github.com/facebook/rocksdb/pull/15104 Test Plan: New unit tests, all exercising fv8 through TEST_AllowUnsupportedFormatVersion(): - block_test: value-delta escape codec round-trip (delta 0, negative, large, and the escape; asserts format_version <= 7 bytes match the legacy encoding and a legacy 0x00 still decodes as delta 0), and rejection of footer bit 30 at all versions. - db_flush_test: fv7-vs-fv8 read equivalence for super_block_alignment, including the separate_key_value_in_data_block intersection, confirming padding events (hence escape entries) occur and fv8 index values are delta encoded. - sst_file_reader_test: fv7-vs-fv8 read equivalence for embedded-blob SSTs, confirming fv8 (unlike fv7) delta-encodes their index values and actually emits escape entries (counted through a sync point). - block_based_table_reader_test: a builder+reader test that injects a footer metaindex gap via a sync point (verifies the gap bytes are written and the reader locates the metaindex via the recorded gap, and that too large a gap surfaces as a builder error), and that a non-empty reserved user_key_common_prefix property makes the reader return NotSupported. - table_test: two-level index with partitioned filters and index_block_restart_interval > 1, the configuration in which both top-level indexes delta-encode their values; verifies scan and point lookup round trips at each tested format_version including fv8. - (Not strictly fv8) Basic SstPartitionerFixedPrefix tests in slice_test (ShouldPartition, ShouldPartitionByPrefix, their mutual consistency, and CanDoTrivialMove), which had no non-DB coverage before. fv8 is also added to the table_test harness (GenerateArgList) and the shared footer version set (test::kFooterFormatVersionsToTest), so the existing table-level and DB-level property/checksum suites now also run at fv8. And added to crash test (minor revisions to db_stress). ## Performance details Release db_bench (PORTABLE=0, -march=native) A/B/C on the read/write fast path, comparing (a) parent, (b) this commit @ fv7, (c) this commit @ fv8. All DBs on tmpfs, 20M keys, block_size=2048, compression=none. Note index_block_restart_ interval=16 (not the default 1) is required to actually exercise the fv7<->fv8 index value-delta codec -- with restart interval 1 every index entry is a full- handle restart and the changed path never runs (confirmed delta-value=1 via sst_dump). Reads single-thread + core-pinned, warm cache, interleaved/rotated rounds, paired deltas. Reads (readrandom + seekrandom, non-partitioned single ~1.9MB index block and partitioned index+filters): fv7->fv8 in the *same* binary is within noise on all four workloads (largest -1.0%, CIs cross zero). before->after (fv7 path) is also within noise; perf shows a deterministic +0.4-0.6% instructions/op from the added branches (the format_version>=8 value_delta_escape guard + refactored data-block footer decode), which is below the throughput noise floor. Writes (fillseq + compact total CPU): fv7->fv8 null. compact (which rewrites all index blocks) is null before->after. fillseq showed +3.5% CPU before->after, but perf attributes it entirely to cycles/IPC with *identical* instructions/op (+0.01-0.04%) -- a code-layout effect on the untouched memtable-insert loop, not this change. Sanity: any single throughput/CPU delta <~1% is within the build-to-build noise floor, measured directly by comparing two byte-identical binaries (swings up to ~0.9% reads; the pre-native/PORTABLE=1 build showed a spurious -1.85% that vanished under -march=native). Net: no measurable regression from fv8 draft; fv7 encoding/decoding byte-for-byte unchanged, only the noted sub-1% instruction bump on the shared read path. Reviewed By: xingbowang Differential Revision: D116105558 Pulled By: pdillinger fbshipit-source-id: ead776cdd396d497d47872c0956ea8c651c0d13f |
||
|
Josh Kang
|
5dfde1c262 |
Add coroutine reads to stackable RocksDB utilities (#15080)
Summary: ## Summary Pull Request resolved: https://github.com/facebook/rocksdb/pull/15080 Introduce reusable `CoroStackableDBBase` forwarding and public `CoroDB::CoGet()` / `CoMultiGet()` entry points matching the `DB::Get()` / `MultiGet()` overloads. These helpers schedule native coroutine reads on the read executor and use the synchronous DB API when coroutine support or a read executor is unavailable. Use the stackable coroutine hooks for TTL and WritePrepared transaction reads. TTL preserves expiration decoding and timestamp behavior, while WritePrepared preserves transaction visibility. TTL keeps a single `DB` inheritance path while adding `CoroDB`: ``` DBWithTTLImpl `-- CoroStackableDBBase<DBWithTTL> |-- DBWithTTL -> StackableDB -> DB `-- CoroDB ``` `CoroDB::CoGet()` dispatches through the stackable coroutine hook and schedules the operation on the read EventBase. `CoMultiGet()` follows the same path: ``` CoroDB::CoGet(db, ...) |-- db->GetCoroDB() -> DBWithTTLImpl |-- no CoroDB or read executor -> db->Get() `-- schedule on read EventBase `-- virtual CoroDB::GetCoroutine() `-- DBWithTTLImpl::GetCoroutine() |-- CoroStackableDBBase<DBWithTTL>::GetCoroutine() | `-- wrapped CoroDB::GetCoroutine() `-- strip TTL timestamp ``` Reviewed By: xingbowang Differential Revision: D115232510 fbshipit-source-id: 59e5090db28d15bcff1a8b9b8c0b30ca7eeb742d |
||
Darkhan Imangaliyev
|
5fe225357a |
Extend parallel checkpointing and db_stress coverage for column-family-subset checkpoints (#15059)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15059 **What**: 1. Extends the parallel `CheckpointEngine` so it can also checkpoint a subset of column families. 2. Adds `db_stress` and `db_crashtest.py` coverage for subset checkpoints, over both the serial and the parallel API. **How**: - **Engine API.** `CreateCheckpointOptions` gains a `std::vector<ColumnFamilyHandle*> column_families` field (empty, the default, still means all column families). This goes on the existing per-operation options truct rather than adding a second pure virtual to `CheckpointEngineBase`: it avoids overload-resolution hazards with the defaulted `sequence_number_ptr`, and it is the pattern the engine API already uses. `CreateCheckpointOptions` has no users outside the engine API, so adding a field breaks nothing. - **Shared validation.** The handle checks that were inline in `CheckpointImpl::CreateCheckpoint` -- reject null handles, handles belonging to another DB, and handles for already-dropped families; coalesce ids; always add the default CF -- move to `static CheckpointImpl::ResolveIncludeColumnFamilyIds()`, now called by both APIs. Rejecting a foreign handle matters because `handle->GetID()` alone would silently coerce a foreign id onto whichever family in this DB happens to share it. Validation runs before any staging directory is created, so a rejected request leaves nothing behind. - **Almost no new plumbing.** `CheckpointImpl::CreateCheckpointImpl()` already accepted `include_cf_ids` and is shared by the serial and parallel movers, and the MANIFEST `kColumnFamilyDrop` rewrite runs after `mover->Finish()`, so it was already mover-agnostic. The engine change is just resolving the ids and forwarding them. - **db_stress.** New `subset_cf_checkpoint_one_in` knob (mirrors `parallel_checkpoint_one_in`): each checkpoint operation covers a random proper subset with probability 1/N. It is a no-op when the DB has only the default column family, and it is independent of `parallel_checkpoint_one_in`, so subset checkpoints get exercised through both the serial API and the engine. Subset selection includes each non-default family with probability 1/2 and then force-excludes one, so every subset checkpoint exercises the MANIFEST rewrite instead of degenerating into a whole-DB checkpoint; the default handle is passed in only half the time, covering the "default is always included whether or not you pass it" contract on both sides. - **db_stress validation.** `DB::ListColumnFamilies()` on the checkpoint must return exactly the requested set -- the direct check on the MANIFEST post-processing, catching both an excluded family that was not recorded as dropped and a requested family that went missing. The checkpoint is then opened read-only with exactly those descriptors, proving the reduced MANIFEST and reduced file set are self-consistent under the default `paranoid_checks`. The pre-existing per-key comparison against `SharedState::Exists()` still runs, restricted to the families actually present. - **Incompatibility.** `clear_column_family_one_in > 0` is rejected together with the new knob, the same way D113103899 did for `abort_and_resume_cf_compactions_one_in`: a subset checkpoint hands live handles from `column_families_` to the API and validates against `column_family_names_`, and neither is stable while families are dropped and recreated underneath. `db_crashtest.py` already pins `clear_column_family_one_in` to 0. - **Test cleanup.** The subset tests shared a lot of setup and verification, so `PopulateFourCfsForSubsetCheckpoint()`, `VerifyFourCfsIntact()` and `VerifyDefaultAndTwoSubsetCheckpoint()` were extracted into the test file's anonymous namespace and are reused by the pre-existing `CheckpointSubsetOfCF` as well as the four new engine tests. This extends `StressTest::TestCheckpoint` with a mode rather than adding a separate operation, which is how `parallel_checkpoint_one_in` was integrated. Reviewed By: rban1 Differential Revision: D114832267 fbshipit-source-id: f3f843819614477497cc1466910547922cafc51d |
||
|
Peter Dillinger
|
af0eae4de3 |
Rename TransactionLogIterator to WalIterator and document its behavior (#15056)
Summary:
`TransactionLogIterator` iterates the write-ahead log. It does not return transactions, does not require `TransactionDB`, and in fact returns `Status::NotSupported()` for two of the three `TransactionDB` write policies (WritePrepared and WriteUnprepared). The name predates `TransactionDB` and comes from "transaction log" being an old synonym for WAL. Everything else in this header was already renamed away from that terminology -- `LogFile` -> `WalFile`, `VectorLogPtr` -> `VectorWalPtr` -- and the C API has long used `rocksdb_wal_iterator_t` and `rocksdb_wal_iter_*`. This finishes the job.
The public class is now `WalIterator` in `include/rocksdb/wal_iterator.h`. `include/rocksdb/transaction_log.h` remains as a forwarding header and `TransactionLogIterator` remains as an alias, both marked deprecated by comment only (not `[[deprecated]]`) so existing users compiling with -Werror are unaffected, matching how `LogFile` and `VectorLogPtr` were retired. `DB::GetUpdatesSince()` keeps its name.
More importantly, the documentation was incomplete in ways that could cost a user data, so this states plainly what the API has always done:
- It only sees the WAL. Writes made with `WriteOptions::disableWAL`, and sequence numbers consumed by `IngestExternalFile()`, advance the DB sequence number without producing a WAL record. They leave permanent holes that end iteration. This was not documented anywhere, so it was unreasonable to expect a user to know those features silently break WAL tailing.
- The three iterator states, in particular that `!Valid()` with an OK `status()` means caught up rather than finished, and that calling `Next()` again is the intended way to tail a DB. The old comment on `Next()` said "REQUIRES: Valid() to be true", which forbids the primary use case.
- `Status::TryAgain`, which a caller must handle but which appeared nowhere in the public headers.
- That `GetUpdatesSince(seq)` silently starts later when `seq` is unavailable. Combined with the above this is the sharp edge: recovering from a spent iterator means calling `GetUpdatesSince()` again, so the recovery step is exactly where data can be skipped unnoticed. A consumer applying these batches to a copy of the DB diverges permanently and undetectably if it misses one, so the header now says such consumers must verify continuity themselves using `BatchResult::sequence` and `WriteBatch::Count()`.
Almost all of this diff is mechanical: renaming the type and its files, updating `#include` paths, rewriting comments, and updating build lists (`src.mk`, `CMakeLists.txt`, `BUCK`, `java/Makefile`, `java/CMakeLists.txt`). `db/c.cc` is regenerated from `tools/c_api_gen/c_base.cc`. Reviewers should focus on the four places that are not:
1. `WalIteratorImpl::UpdateCurrentWriteBatch()` in `db/wal_iterator_impl.cc` (renamed from `db/transaction_log_impl.cc`) drops dead code. On a sequence discontinuity it set `current_status_` to `NotFound` and then called `SeekToStartSequence()`, which returns immediately when `current_status_` is not OK. That guard and the early return in `Next()` were both added by
|
||
|
Ranjan Banerjee
|
cda4a73a19 |
Fsync destination files in parallel CheckpointEngine copies (#15038)
Summary: Parallel checkpoint copies were never fsynced. `CopyEngine::CopyOrCreateFile` gated the destination sync on `use_fsync` (wired to `db_options.use_fsync`) -- but `use_fsync` means "fsync vs fdatasync", not "whether to sync". With the default `use_fsync=false`, the parallel path skipped the sync, so a copied MANIFEST/SST stayed in the unsynced buffer and was dropped by `TestFSWritableFile::Close` under `sync_fault_injection`, leaving a 0-byte file and checkpoint corruption: Corruption: no log_file_number, next_file_number, last_sequence entry in MANIFEST Corruption: Sst file size mismatch between expected N and file system 0 Serial `CopyFile` always syncs (using `use_fsync` only to pick the flavor), so it never hit this. Fix: split `WorkItem` / `CopyOrCreateFile` into `sync` (whether to sync) and `use_fsync` (fsync vs fdatasync). CheckpointEngine now passes `sync=true, use_fsync=db_options.use_fsync`, matching serial. BackupEngine is unchanged -- `sync=options_.sync` with the new `use_fsync=false` default reproduces the prior behavior. Pull Request resolved: https://github.com/facebook/rocksdb/pull/15038 Reviewed By: xingbowang Differential Revision: D114414620 Pulled By: rban1 fbshipit-source-id: e0f1f3dbadcd48c5a5c2a11111a266dbed623f86 |
||
|
Peter Dillinger
|
afc922bf5f |
Make ReadOptions::table_filter a caller-owned pointer for cheaper copies (#15005)
Summary: ReadOptions is copied frequently on read paths, and its only non-trivially-copyable member, table_filter, is a std::function that is almost always empty yet still gets deep-copied on every ReadOptions copy (and can heap-allocate). This changes table_filter to a pointer to a caller-owned std::function instead of storing one by value: when non-nullptr, the caller is responsible for keeping the pointed-to function alive for the duration of the read (like the other pointer fields such as iterate_upper_bound). With this change every ReadOptions member is trivially copyable, so copying ReadOptions no longer touches a std::function at all. That trivial-copyability is locked in with a static_assert in options_test.cc (kept out of the public header). This is inspired by D113109635, which targeted the same ReadOptions-copy hotspot by relocating table_filter to the end of the struct so the trivial members memcpy in one contiguous block. This is intended to be a superior solution: rather than only defragmenting the copy, it removes the std::function copy entirely. This is a breaking API change, tolerated because table_filter is a relatively obscure option. A nullptr, or a pointer to an empty std::function, means no filtering (preserving the previous empty-function behavior). A shared internal helper HasTableFilter(const ReadOptions&) in options_helper.h centralizes the nullptr + empty-function check used on the read paths. The C API and Java bindings now own the std::function and hand ReadOptions a pointer to it; their public signatures are unchanged, and the C API rocksdb_readoptions_t clears the owned function when the callback is null. Making ReadOptions trivially copyable also let -Werror=unused-but-set-variable fire on a pre-existing dead local in db_bench's MultiScan: a ReadOptions that set async_io/readahead_size but was never passed to NewMultiScan (the warning had been suppressed while ReadOptions had a non-trivial member). That local was in fact inert for multiscan -- BlockBasedTableIterator::Prepare overrides ReadOptions::async_io with MultiScanArgs::use_async_io for the coalesced prefetch, so multiscan async I/O is driven by --multiscan_use_async_io, not ReadOptions::async_io. Removed the dead local; MultiScan now uses read_options_ directly, consistent with MultiScanRandom. Pull Request resolved: https://github.com/facebook/rocksdb/pull/15005 Test Plan: Existing table_filter coverage was adapted to the new pointer API and still exercises the behavior end to end: db_iterator_test.TableFilter (callback invoked once per table; returning false hides a table) and its range-conversion rejection case; db_bloom_filter_test SstQueryFilter/FixedWidthSegments range queries; and trie_index_db_test's range-query iterator (which returns iterators outliving the filter, exercising the caller-owned lifetime). db/c_test covers the C API set/clear/has_table_filter path. options_test asserts std::is_trivially_copyable_v<ReadOptions>. Manually verified the db_bench MultiScan async behavior with a ~200MB DB and --use_direct_reads: with ReadOptions::async_io held true, async reads tracked only --multiscan_use_async_io (rocksdb.read.async.micros count 329511 with use_async_io=true vs 0 with false), confirming ReadOptions::async_io does not drive multiscan async and the removed local had no effect. Reviewed By: anand1976 Differential Revision: D113613525 Pulled By: pdillinger fbshipit-source-id: 38d73b607500a5c7ca2b89dd8f4187ef43940aa4 |
||
|
Darkhan Imangaliyev
|
f9df618e4d |
Api to checkpoint only selected set of column families (#15019)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15019 RocksDB's Checkpoint API currently copies all column families. For use cases that only need a subset, this is wasteful. This diff adds a new `CreateCheckpoint(dir, column_families, ...)` overload that: - Validates input handles (rejects null/foreign handles, coalesces duplicates, always includes default CF). - Filters GetLiveFilesStorageInfo to capture only the requested CFs' table/blob files. - Post-processes the copied MANIFEST by appending kColumnFamilyDrop records for excluded CFs, so the checkpoint opens without referencing missing files. - An empty column_families vector falls through to the existing whole-DB path. Reviewed By: xingbowang, pdillinger Differential Revision: D113963616 fbshipit-source-id: d1becbbb9528dd13c2dc314165880b76123757f0 |
||
|
Ranjan Banerjee
|
f5c0c3e274 |
Record CheckpointEngine copy bytes under checkpoint tickers (#14997)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/14997 `CheckpointEngine`'s shared `CopyEngine` recorded copied bytes under the `BACKUP_READ_BYTES` / `BACKUP_WRITE_BYTES` tickers (the `CopyEngineOptions` defaults), conflating checkpoint I/O with backup I/O in statistics. This adds dedicated `CHECKPOINT_READ_BYTES` / `CHECKPOINT_WRITE_BYTES` tickers and points `CheckpointEngineImpl`'s `CopyEngineOptions` at them, so checkpoint copies report under their own metric names. `BackupEngine` is unchanged (still uses the backup-ticker defaults). Addresses review feedback that the shared `CopyEngine` should not hardcode backup metric names for non-backup callers. The new tickers are intentionally not mirrored in the Java bindings (`TickerType.java` / `portal.h`): the signed-`jbyte` id space there is exhausted, so they remain C++-only, documented inline in `statistics.h`. Reviewed By: xingbowang Differential Revision: D113469742 fbshipit-source-id: 2a6a7ef04c939a1240f1c48c7bcaba8b2e7a1c10 |
||
|
Ranjan Banerjee
|
19482988e3 |
Add parallel CheckpointEngine [4/4] (#14948)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/14948 Adds a reusable `CheckpointEngine` (opened once, works for any DB) that creates checkpoints via the shared `CopyEngine` pool, copying and hard-linking data files across `max_background_operations` threads. `CheckpointImpl::CreateCheckpoint` is refactored into a shared `CreateCheckpointImpl`, with the serial/parallel split behind a `CheckpointFileMover`; files the FileSystem cannot hard-link fall back to copies. The legacy `Checkpoint` API stays serial and unchanged. Reviewed By: xingbowang Differential Revision: D111387993 fbshipit-source-id: 036d375da9c9b8f051cfc3da0551b035e29d80c4 |
||
|
Ranjan Banerjee
|
0c0bfbe8be |
Add hard-link support to CopyEngine [3/4] (#14947)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/14947 Teaches the shared `CopyEngine` to hard-link files through its background pool: adds `WorkItemType::Link` and `CopyEngine::LinkFile(src, dst, dst_env)`, which returns the FileSystem's `NotSupported` when links are unavailable so callers can fall back to copying. This is what CheckpointEngine (next diff) needs for warm storage, where linking is supported but benefits from parallelism. No existing caller links through the engine yet (BackupEngine never hard-links), so no behavior change; the path is exercised by a new `copy_engine_test.cc` (parallel copy, parallel link, single-thread copy, per-item error reporting). Reviewed By: xingbowang Differential Revision: D111387972 fbshipit-source-id: ec0b0e9b80749a3f51e62066ceef2df21a1811d2 |
||
|
Ranjan Banerjee
|
fac47c3072 |
Extract shared CopyEngine [2/4] (#14946) (#14946)
Summary:
Extracts BackupEngine's parallel file-copy/checksum thread pool into a standalone `CopyEngine` (`utilities/copy_engine/copy_engine.{h,cc}`) so `CheckpointEngine` (later in the stack) can reuse it. Behavior-preserving; all `TEST_SYNC_POINT` names are unchanged.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/14946
Reviewed By: xingbowang
Differential Revision: D111387950
Pulled By: rban1
fbshipit-source-id: b5046adb98735a606fd1ba471731fd316ce6f036
|
||
|
Peter Dillinger
|
ffa95117b9 |
Bug fix and clean up error/status handling for wide-column + blob support (#14980)
Summary: An audit of the wide-column blob-reference read paths (from the preceding "multiple backing buffers" change, D112212266) surfaced several inconsistencies in how Status is used: control-flow-via-Status, a status classification that diverged from the rest of the codebase, the same "no blob fetcher" Corruption repeated at many call sites, a swallowed non-OK status (bug), and read statistics advanced on failed reads. This change cleans those up. It is internal only -- no public API or on-disk format change -- and stacks on that diff. Concretely, this resolves two of the intended follow-ups called out in D112212266: * "Resolve FIXME about pre-existing swallowed non-OK status in db_impl.cc" * the error-status/API cleanup (classify an unrecognized wide-column version as Corruption, not NotSupported, at all entry points; replace the NotSupported sentinel from GetValueOfDefaultColumn with an is_blob_reference out-param, removing GetValueOfDefaultColumnResolvingBlobs; and centralize the "blob-backed value without a blob fetcher" Corruption into the resolvers, resolving the ResolveEntityForMerge FIXME, rather than repeating it at each caller). The other two follow-ups from that diff -- the value_size_soft_limit TODO in db_impl.cc and extending the zero-copy optimization to SST embedded-blob references -- are deferred to separate diffs. Unrecognized version -> Corruption. A wide-column version newer than kVersion2 now returns Corruption instead of NotSupported, consistent with how the rest of the codebase classifies an unrecognized serialized version/type (BlobIndex, the blob log header, the table footer). The guard is applied at every entry point (Deserialize, GetValueOfDefaultColumn, HasBlobColumns, ForEachBlobFileNumber); the latter two previously lacked it and would misparse a future/corrupt version. Reject trailing data after an empty entity. Deserialize() previously returned OK early for a zero-column entity, silently ignoring any trailing bytes; it now dispatches zero-column entities to the V1/V2 parser like the non-empty case, so the "exactly one serialized entity" trailing-data check the preceding diff added for non-empty entities now covers the empty case too. Default-column extraction no longer signals via Status. now returns (value, is_blob_reference): for a blob-referenced default column it returns OK with the raw serialized BlobIndex bytes and the flag set, instead of a NotSupported sentinel that every caller had to detect and pair with a second, redundant full deserialization. GetValueOfDefaultColumnResolvingBlobs and the memtable GetDefaultColumnBlobIndexSlice helper are removed; a single ResolveDefaultColumnBlobReference() resolves the raw BlobIndex (handling an inlined index with no fetch). This also fixes a case where a non-default column 0 that happened to be a blob reference was misreported as a blob-backed default. The one remaining NotSupported is the honest, explicit base-DB case: reading a blob-index default column when is_blob_index was not requested. Null-blob-fetcher handling centralized. "A blob-backed value with no fetcher to read it" (reachable via a reader without blob support such as SstFileReader, or corrupt/legacy data) was checked -- and reported as Corruption -- at ~6 call sites with three slightly different messages, forced by the resolvers asserting the fetcher was non-null. The asserts are gone; each resolver now owns the check in one commented place: ResolveEntityBlobColumnsMultiBuffer (whole entity), ResolveDefaultColumnBlobReference (default column), and the MaybeResolveMemtableBlobValue plain-value path (which open-codes its fetch). Callers just propagate the status. A side benefit: an entity/column whose only blob references are inlined now resolves without a fetcher instead of being wrongly rejected. Return Status from the memtable resolve helpers, fixing a swallowed error. MaybeResolveMemtableBlobValue (was bool + Status*) and PostprocessMemtableValue- Read (was void + Status*) now return Status, matching codebase convention; MaybeResolveMemtableBlobValue reports "there was a reference to resolve" via a bool* did_resolve out-param (true means it has finalized value/columns). This resolves a pre-existing bug: on the blob-resolve path a non-OK status was returned early without the output cleanup the plain path performed, leaving a half-populated result behind an error. PostprocessMemtableValueRead now has a single success/error finalization block reached by every path (self-pin on success, clear outputs on any error), and returns the finalized status. Also switched the memtable "no blob fetcher" case from NotSupported to Corruption (an internal helper whose sole caller treats all non-OK statuses identically). Read-only/secondary read stats mirror the primary path on error. DBImpl::GetImpl gates byte/throughput accounting (BYTES_READ, BYTES_PER_READ, get_read_bytes) and merge-operand output on s.ok(); the read-only and secondary Get paths did so unconditionally, logging spurious samples (e.g. a 0-byte BYTES_PER_READ sample) on NotFound/error. They now gate on s.ok() too, while keeping NUMBER_KEYS_READ, MEMTABLE_HIT, and super-version cleanup unconditional. Header docs updated to match (Deserialize/DeserializeSimple return Corruption for an unexpected blob reference; unknown per-column value types are rejected as Corruption). Pull Request resolved: https://github.com/facebook/rocksdb/pull/14980 Test Plan: Extended db/wide/wide_column_serialization_test.cc: DeserializeUnsupportedVersion now expects Corruption; new FutureVersionRejectedConsistently verifies that Deserialize, GetValueOfDefaultColumn, HasBlobColumns, and ForEachBlobFileNumber all reject a future version as Corruption; V2GetValueOfDefaultColumnBlobRef now checks the is_blob_reference flag and raw-BlobIndex bytes and that resolving without a fetcher is Corruption; DeserializeRejectsTrailingData now also covers empty (zero-column) V1 and V2 entities. db/blob/db_blob_index_test.cc: MaybeResolveMemtableBlobValueWithoutFetcherFails- Closed updated to the new signature (asserts did_resolve + Corruption); new PostprocessMemtableValueReadClearsOutputsOnError pins the swallowed-status fix, covering both an already-non-OK incoming status and (the actual FIXME scenario) an OK incoming status whose blob-resolve path fails -- both clear outputs and return the error. The StackableDB-style DBBlobIndexTest.Get contract is unchanged (its NotSupported comes from the base-DB blob-index read path, not these helpers). Reviewed By: xingbowang Differential Revision: D113099114 Pulled By: pdillinger fbshipit-source-id: 4bc374dd74f8ba83bdf49bd8753c1a8d88529cde |
||
|
Peter Dillinger
|
81ea677072 |
PinnableWideColumns support for >1 backing buffer to avoid re-serialization (#14943)
Summary:
Wide-column read paths that materialize blob-referenced columns used to re-serialize the entire entity into a fresh contiguous buffer, copying the (potentially large) resolved column values byte-for-byte. This was forced by PinnableWideColumns holding a single backing buffer with all column Slices pointing into it.
PinnableWideColumns now holds its columns as zero-copy Slices into one or more stable backing buffers, kept in a std::forward_list<PinnableSlice> whose nodes never relocate on move, move-assignment, or splice. Resolved blob columns can be spliced in as additional backing buffers instead of re-serializing the entity.
Using address-stable nodes also cleans up the old Slice-invalidation handling. The previous single-buffer representation had to cope with a self-pinned PinnableSlice whose small (SSO) std::string relocates when the PinnableSlice is moved: on every move it detected the relocated backing pointer and re-derived the column Slices by re-parsing. With forward_list nodes that never move, the column Slices stay valid across move / move-assignment / splice, so that handling is gone and the move operations are now the defaulted member-wise moves.
The multi-buffer plumbing is internal only: a new PinnableWideColumnsHelper (a friend) performs the splice and the little bit of private-member access the read paths need, so there is no new public API surface and the previous `friend class DBImpl` is gone. The public view is unchanged: columns() still returns a single, in-order set.
Read paths converted to resolve without re-serializing:
- GetContext::SaveWideColumnEntityToColumns (eager GetEntity SST reads, separate-blob-file references)
- DBImpl::ResolveDirectWriteWideColumns (direct-write blob references)
- DBImpl::MaybeResolveMemtableBlobValue (blob-backed memtable reads)
Merge output still serializes (the MergeOperator API consumes/produces a serialized entity). Same-file ("embedded") blob columns are resolved -- and still re-serialized -- earlier in the block-based table reader (MaybeResolveEmbeddedValue / ResolveEmbeddedValueForGet) before the entity reaches GetContext, so the GetEntity path above only avoids re-serialization for separate-blob-file references. Compaction's separate entity machinery is unchanged.
API/accounting: deprecated PinnableWideColumns::serialized_size() (comment-only, per repo convention) in favor of the new payload_size() (sum of column name + value sizes; equals the value size for a plain value). Read statistics for entities (BYTES_READ, BYTES_PER_READ, per-key GetEntity/MultiGetEntity byte counts) and the ReadOptions::value_size_soft_limit threshold now measure entities by payload_size() instead of the serialized size. This drops framing bytes (entity measurements are slightly smaller), thus making plain-value and wide-column accounting consistent and reducing unnecessary exposure to implementation details.
Serialization cleanup enabled by the above:
- Consolidated Deserialize/DeserializeV2 into a single Deserialize(const Slice&, WideColumns&, blob_columns*) plus a DeserializeSimple alias for the no-blob-references case; both take const Slice&, require the input to be exactly one serialized entity, and reject any trailing data uniformly (V1 previously ignored trailing bytes). An unexpected blob reference where a fully-resolved entity is required now returns Corruption.
- ResolveEntityBlobColumns (serialized output, used only by merge) is now a thin wrapper over the new multi-buffer resolver plus Serialize, removing the duplicated fetch loop and the SerializeResolvedEntity helper.
- Consistently use `const Slice& input` rather than `Slice& input` for wide column functions expecting an input slice with no trailing data.
Design/polish from human review:
* The initial plan called for more complex handling around a vector<PinnableSlice> backing store (unique_ptr indirection and re-deriving the column Slices on move to cope with reallocation and SSO relocation); switching the backing store to a std::forward_list of address-stable nodes was a human suggestion that removed that complexity and the move-time Slice re-derivation entirely.
* Keep resolve plumbing off the public API via PinnableWideColumnsHelper and removing the DBImpl friend
* Various pieces of cleanup and polish (more left to do)
* The eager-GetEntity optimization
* The memtable inline-copy NOTE/TODO.
Intended follow-up:
* Resolve FIXME about pre-existing swallowed non-OK status in db_impl.cc
* Resolve TODO about value_size_soft_limit in db_impl.cc
* Extend this optimization to SST embedded-blob references
* Error-status/API cleanup: classify an unrecognized wide-column version as Corruption (not NotSupported) at all entry points; replace the NotSupported sentinel from GetValueOfDefaultColumn with an explicit is_blob_reference out-param (removing GetValueOfDefaultColumnResolvingBlobs); and centralize the "blob-backed value without a blob fetcher" Corruption into the resolvers (resolving the ResolveEntityForMerge FIXME) rather than repeating it at each caller
Pull Request resolved: https://github.com/facebook/rocksdb/pull/14943
Test Plan:
Extended db/wide/wide_column_serialization_test.cc with PinnableWideColumns unit tests: set/payload_size for plain, V1, and resolved entities; move and move-assign of a multi-buffer result preserve column data pointers (the node-stability guarantee); Reset; ResolveColumns splices blob buffers in zero-copy (resolved values' data() point into the spliced-in buffers, not a re-serialized entity); serialized_size() == SerializedSizeV1(columns()); and Deserialize rejects trailing data after a serialized entity for both V1 and V2 layouts.
Built and ran (debug): wide_column_serialization_test, db_wide_basic_test, db_wide_blob_direct_write_test, db_blob_basic_test, compaction_iterator_test, db_merge_operator_test, db_iterator_test, udt_util_test, write_batch_with_index_test -- all pass. librocksdb and db_stress build. make format-auto and build_tools/check-sources.sh are clean.
Reviewed By: xingbowang
Differential Revision: D112212266
Pulled By: pdillinger
fbshipit-source-id: 13b89e2bb7d8371f4c1ff18b234a549bea9c9e77
|
||
Laurynas Biveinis
|
d461606f0f |
Make range-lock waits interruptible with SetIsKilledCallback (#14927)
Summary: Add RangeLockManagerHandle::SetIsKilledCallback, a std::function<bool()> predicate mirroring SetEscalationBarrierFunc. When installed, RangeTreeLockManager wakes the toku lock_request::wait loop every 100ms to poll it; if it fires, the wait completes with a new DB_LOCK_INTERRUPTED result mapped to Status::Aborted(), so a blocked GetRangeLock() returns promptly instead of waiting out the lock timeout. Range-lock users that install no callback keep the original wait-until-grant-or-timeout behavior (no extra wakeups). Also remove the unused toku kill_waiter abort path (lock_request and locktree_manager), which had no caller. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14927 Test Plan: `build_tools/rocksptest.sh range_locking_test` passes, including the two new tests. KilledCallbackAbortsWait completes in ~90ms despite a 100s lock timeout, confirming the kill callback aborts the wait promptly; KilledCallbackFalseStillTimesOut confirms an installed-but-false callback still times out. Tested end-to-end together with the MyRocks side in D110890436: with rocksdb_use_range_locking=ON, a KILL QUERY parked on a conflicting range-lock wait is interrupted promptly (rocksdb.range_locking MTR suite) instead of waiting out rocksdb_lock_wait_timeout. Reviewed By: pdillinger Differential Revision: D111663405 Pulled By: laurynas-biveinis fbshipit-source-id: 4819e09bde66f4e932aef74f6adfd14789bb1425 |
||
|
Josh Kang
|
dc72a64a5b |
Flush no-WAL live-file data with LockWAL (#14931)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/14931 The failing crash-test scenario combines `disable_wal=1`, `LockWAL()`, and backup/checkpoint live-file-copy verification. A WAL-disabled write can leave a key only in the memtable. The old live-file path skipped `FlushForGetLiveFiles()` whenever `LockWAL()` was active, so the copied SST/WAL file set could omit that memtable-only key even though the original DB still served it. Restore verification could then report the key as present in the original DB but missing from the restored backup/checkpoint. Fix live-file capture to flush unpersisted WAL-disabled data even while `LockWAL()` is active. If all data is already persisted, the path still skips the flush while WAL is locked. If unpersisted data exists, backup/checkpoint can now block behind `LockWAL()` until another thread calls `UnlockWAL()` and the flush can safely proceed. To avoid self-deadlock, track `LockWAL()` ownership by thread with recursive counts. `UnlockWAL()` now has to be called by the same thread that successfully called `LockWAL()`, and wait=true flushes called by that same owner return `Aborted("Likely deadlock as the same thread called LockWAL()")` instead of waiting forever. Cross-thread flushes remain allowed and wait for all `LockWAL()` owners to unlock. Also audited MyRocks and can confirm they already conform to the Lock/Unlock on same thread semantics. Reviewed By: pdillinger Differential Revision: D111793089 fbshipit-source-id: 1bbb03b25d00d47440126cc6bb4ce7410302e25b |
||
|
Peter Dillinger
|
74aba599ab |
Fix WriteUnprepared transaction rollback after commit write failure (#14920)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/14920 When a WriteUnprepared transaction's commit write fails while two_write_queues is enabled (e.g. due to an I/O error), CommitInternal() unconditionally cleared unprep_seqs_ and the flushed/unflushed save points before returning the error. That left the transaction with no record of its prepared sequence numbers, so a subsequent Rollback() -- the natural way for a caller to recover from a failed commit -- constructed a WriteUnpreparedCommitEntryPreReleaseCallback over an empty unprep_seqs_ and tripped assert(unprep_seqs.size() > 0) in its constructor. In release builds this would instead corrupt commit-cache accounting rather than aborting. Fix: only clear unprep_seqs_ and the save points on the commit success path. On the two-write-queues commit-write failure path we still remove the prepared entry that AddPreparedCallback may have added for commit_batch_seq (that seq is never stored in unprep_seqs_, so a later Rollback cannot clean it up), but unprep_seqs_ is otherwise preserved so the transaction can still be rolled back once the caller resumes the DB from the write error. Originally authored by Devmate trigger Reviewed By: xingbowang Differential Revision: D110271443 fbshipit-source-id: 7d991e01ddc74ccd170cbc25f6af160c5f3d5663 |
||
|
Jay Huh
|
e1ea04d9ef |
Add reverse order support to MultiScan (#14876)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/14876 Add reverse scan support to `MultiScanArgs`/`MultiScan` so callers can iterate bounded scan ranges in descending order. Plumb reverse scan state through `DBIter`, `BlockBasedTableIterator`, and `MultiScanIndexIterator`, including reverse prefetch windowing and async I/O support. Update `db_bench` `multiscanrandom` to honor `--reverse_iterator` and report rows scanned so reverse MultiScan and iterator baselines can be compared on row throughput. Add `db_stress --multiscan_reverse` coverage so the prepared MultiScan path can be stressed with `SeekForPrev()`/`Prev()` against a control iterator. Also fix the table child iterator path to accept reverse `SeekForPrev()` targets above a file's prepared ranges. `MergingIterator` and `LevelIterator` can legally seek every child to the current range limit; a child file may only have lower prepared ranges, so treating that as invalid caused intermittent `SeekForPrev target is outside prepared ranges` exceptions in the benchmark. Reviewed By: xingbowang Differential Revision: D109357730 fbshipit-source-id: eccece5ca7d8f5d6760d51ebe76801a332a49061 |
||
|
Peter Dillinger
|
f6fabdb64f |
Improve build and test experience for Make builds (#14883)
Summary: Local Make builds silently reuse object files even when build parameters (DEBUG_LEVEL, sanitizers, ASSERT_STATUS_CHECKED, RTTI, etc.) change, since all object files share the same paths. This leads to confusing linker errors, sanitizer false negatives, ODR violations, and "phantom" bugs that are easy to miss -- a pitfall for humans and especially for AI agents driving builds non-interactively. This change improves the Make experience on three fronts: 1. Build-parameter change detection in the Makefile. After flags are fully resolved, we hash the effective compile/link inputs (CC|CXX|CFLAGS|CXXFLAGS|LDFLAGS|EXEC_LDFLAGS) into a per-OBJ_DIR .build_signature stamp and compare against the previous build: - Default: stop with a clear error when parameters changed. This is optimized for the expert who might not intend a full rebuild, and for avoiding CI inefficiency. - AUTO_CLEAN=1: automatically run clean-rocks and proceed. - ALLOW_BUILD_PARAMETER_CHANGE=1: skip the check (e.g. intentionally mixing DEBUG_LEVEL=1 and DEBUG_LEVEL=2 objects). The check is skipped for dry runs (-n) and for non-building / self-cleaning targets (clean*, format, check-*, tags, gen-pc, check-progress, watch-log, release, coverage, the asan_*/ubsan_* variants, etc.). make_config.mk is removed only by the top-level `clean` target, not by clean-rocks, so the auto-clean does not delete the make_config.mk already included by the running make. The signature is removed last in clean-rocks so an interrupted clean keeps change detection meaningful. 3. New build_tools/rockstest.sh and build_tools/rocksptest.sh helpers that build and run unit tests in one step. They set AUTO_CLEAN=1 and build with -j<NCORES> (computed like the Makefile). rocksptest.sh runs one or more binaries under the checked-in gtest-parallel (sharing one parallel worker pool); rockstest.sh runs a single binary directly for a small number of cases. Both reject a leading-option first argument; `rockstest.sh install` writes ~/bin/rockstest and ~/bin/rocksptest shims that defer to the in-tree scripts (with a friendly message when run outside a source root). This is a big win for AI-agent workflows: one command instead of a separate `make` then test-run invocation, removing the overhead of monitoring two long commands, while avoiding serial builds and serial test execution. 4. CLAUDE.md guidance updated to recommend AUTO_CLEAN=1 for manual make invocations and the rocks(p)test.sh helpers for running tests Although there might be drive to consolidate around the BUCK build for internal use, I'll note that last I checked it was around ~20 minutes to run all the unit tests under buck and ~2 minutes under `make check`. Bonus: the first revision of this had copyright/license mistakes, and this change corrects similar errors elsewhere, with CLAUDE.md advice updated. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14883 Test Plan: Manual verification of the Makefile change detection: - Confirmed the resolved-flags hash is deterministic across runs for the same goal, and differs between configs (e.g. default/shared vs static_lib, and with/without ASSERT_STATUS_CHECKED). - Unchanged parameters: rebuild does not error. - Changed parameters: errors by default; AUTO_CLEAN=1 runs clean-rocks and proceeds; ALLOW_BUILD_PARAMETER_CHANGE=1 bypasses the check. - Dry runs (make -n) and exempt targets (e.g. `make check-progress`, list_all_tests, gen-pc) do not trip the check and leave the stamp untouched. - Reproduced the ASSERT_STATUS_CHECKED=1 auto-clean path end to end and confirmed it no longer fails with "No rule to make target 'make_config.mk'"; verified `make clean` removes make_config.mk while `clean-rocks` does not, and that the signature is deleted as the final clean-rocks step. Manual verification of the helper scripts: - shellcheck-clean and `bash -n` pass for both scripts. - Usage/guard behavior: missing argument and leading-option first argument are rejected with a clear message. - `rockstest.sh install` (validated against a temp HOME) generates both shims; running a shim outside a source root prints "(Not in a rocksdb source root directory?)" and exits non-zero. - rocksptest.sh argument splitting verified for single binary, multiple binaries, and binaries followed by gtest-parallel/test args. - .gitignore patterns verified with `git check-ignore` for .build_signature (root and jl/jls) and build_tools/__pycache__/. Reviewed By: xingbowang Differential Revision: D109713512 Pulled By: pdillinger fbshipit-source-id: a801d29afa91f53a2dce717db9fe5e5d485cc154 |
||
|
Jay Huh
|
43d60bfe9e |
Revert reverse-comparator handling in range tree lock manager (#14887)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/14887 This reverts two changes to the range-lock endpoint comparator and restores the prior, direction-agnostic behavior: - https://github.com/facebook/rocksdb/pull/14831 ("Fix range lock manager crash with reverse comparator"), which made RangeTreeLockManager::CompareDbtEndpoints reverse-aware, and - https://github.com/facebook/rocksdb/pull/14880 (a follow-up that only corrected the equal-length / point-range branch). Why: #14831 taught the endpoint comparator about reverse-ordered column families by flipping the suffix tie-break based on comparator direction. However, callers that use range locking on a reverse-ordered CF already flip the start/end endpoints themselves before calling lock_range() -- for example, MyRocks does this in ha_rocksdb::set_range_lock() (see its "RangeFlagsShouldBeFlippedForRevCF" comment). The two corrections stack and produce left > right again, which: (a) trips paranoid_invariant(compare(left, right) <= 0) in toku::locktree::try_acquire_lock() and aborts on equality/point locks, and (b) for range scans, mis-orders the endpoints and produces an incorrect lock range, leading to silent correctness issues under concurrency. #14880 only addressed (a) (the equal-length / point-range branch), not (b), so it was an incomplete fix at the wrong layer. Decision: keep CompareDbtEndpoints direction-agnostic and make it an explicit caller requirement to pass a flipped (swapped start/end, with negated infimum/supremum flags) range for range locking to work on a reverse-ordered column family. A function-level comment documenting this contract is added so reverse-awareness is not reintroduced here. Removes the RangeLockWithReverseComparator unit test added by #14831 (it asserts the reverted direction-flipped ordering). Keeps a regression test that a point range lock [infimum(K), supremum(K)] works on a reverse-comparator column family. Reviewed By: xingbowang Differential Revision: D109713989 fbshipit-source-id: 4ea0999c92645e1adc9cd1114d924cad142f2001 |
||
|
Peter Dillinger
|
8c0790bdf8 |
Refresh platform010 build dependencies and remove TBB (#14882)
Summary: Refresh the fbcode platform010 toolchain and library pins to current third-party2 versions, and remove the now-unused Intel TBB dependency. Toolchain / deps: - Fix update_dependencies.sh: the gcc and binutils trees moved from centos8-native to centos9-native, so the old paths resolved to nothing and a re-run emitted empty GCC_BASE/BINUTILS_BASE. Point them at centos9-native. - Bump clang 15 -> 21; binutils 2.37 -> 2.43; libunwind 1.4 -> 1.8; valgrind 3.19 -> 3.22; plus hash-only refreshes for the LATEST-tracked libs. Regenerate dependencies_platform010.sh. - Keep GCC pinned at 11.x. The only newer GCC in third-party2 (13.x) is built for glibc >= 2.35 (its libgcc_s needs _dl_find_object@GLIBC_2.35), but platform010 ships glibc 2.34, so GCC 13 will not link/run here. - zlib stays 1.2.8 (the only version with an x86_64 platform010 build). - Document why GCC/libgcc/glibc/zlib remain pinned in update_dependencies.sh. Remove TBB (no longer used) from the build system: the get_lib_base entry, both fbcode_config*.sh, build_detect_platform detection, the CMake WITH_TBB option / find_dependency, and cmake/modules/FindTBB.cmake. Dockerfiles and the HISTORY.md changelog are left untouched. (TBB was used by the old clock cache, long ago removed.) Although this change was originally motivated by upgrading gcc for its libasan not to hit process lifetime thread limits, upgrading gcc proved impractical under platform010. Bonus: fix USBAN+gcc build by making GetParam() valid by the time it is called in several test class constructors. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14882 Test Plan: a variety of local builds (gcc, clang; various sanitizers) using fbcode tooling No production code changes Reviewed By: xingbowang Differential Revision: D109631787 Pulled By: pdillinger fbshipit-source-id: 9c466c59b039c2a67ee0318c0ccbac02e19f537b |
||
|
Jay Huh
|
b0ecf86f6e |
Fix range lock manager assertion crash on reverse-comparator CF point locks (#14880)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/14880 The RocksDB 11.5.0 reverse-comparator range-lock fix (the HISTORY.md entry "Fixed a bug where the range lock manager would crash with an assertion failure when using a reverse comparator column family") made `RangeTreeLockManager::CompareDbtEndpoints` reverse-aware by wrapping the endpoint-suffix tie-break results in an `is_reverse` flip. The length-disparity (different user key) branches need that flip, but the equal-length / equal-user-key branch was flipped too, and that is incorrect. Range-lock endpoints are encoded as `[suffix_byte][user_key]` where the suffix is `SUFFIX_INFIMUM` (0x0) or `SUFFIX_SUPREMUM` (0x1). When two endpoints share the same user key and differ only in the suffix byte, the suffix is a positional boundary marker for that key, not key content, so the infimum endpoint must sort before the supremum endpoint in any total order, regardless of the column family sort direction. Flipping it for a reverse comparator makes `m_cmp(infimum(K), supremum(K)) > 0`. MyRocks issues an equality lock (e.g. `UPDATE t1 SET a=123 WHERE a=35`) as the range `[infimum(K), supremum(K)]` over the same user key K. On a reverse (`rev:`) column family — which uses the stock `ReverseBytewiseComparator()`, so `is_reverse` is true — the flipped comparator reports `left > right`, tripping `paranoid_invariant(m_cmp(left_key, right_key) <= 0)` in `toku::locktree::try_acquire_lock` and aborting the server with SIGABRT. This broke many MyRocks `---range_locking` MTR tests (partial_index*, bypass_select_range_sk*, alter_table_online_debug, allow_no_primary_key_with_sk) after RocksDB was bumped to 11.5.0 in D108936924. Fix: in the equal-length / equal-user-key branch of `CompareDbtEndpoints`, compare the suffix bytes without the `is_reverse` flip so infimum always precedes supremum. The length-disparity branches keep the reverse-aware flip (still covered by `RangeLockWithReverseComparator`). Note: this lands in `internal_repo_rocksdb/repo`, the RocksDB development tree; it reaches `fbcode/rocksdb/src` (what MyRocks/ZippyDB link) through the RocksDB release sync. Tracked in T276980728. Reviewed By: xingbowang Differential Revision: D109591908 fbshipit-source-id: 078b16e6e51f4f365b4a8d539ac5b968a7461729 |