mirror of
https://github.com/facebook/rocksdb.git
synced 2026-10-23 08:09:02 +03:00
main
981 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Peter Dillinger
|
539532377d |
Remove Solaris support (#15449)
Summary: Solaris support was contributed by outside users starting with |
||
Josh Kang
|
aeebcbded4 |
Add coroutine external table reads (#15291)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15291 Add experimental coroutine interfaces for implementing asynchronous basic and full `ExternalTableReader`s. Implementations opt in through `CoroExternalTableReaderBase<Mode>` by providing `GetCoroutine()` and may override `MultiGetCoroutine()`; the default implementation runs the individual Get operations concurrently. The external-table adapter uses these APIs when supported and otherwise retains the existing synchronous behavior. As a smaller supporting fix, Single Get coroutine reads using non-mmap, non-direct I/O on a kFSBuffer-capable `FileSystem` now use `FSRandomAccessFile::SubmitReadAsync()` instead of falling back to synchronous `Read()`. Reviewed By: xingbowang Differential Revision: D122187491 fbshipit-source-id: 217e2953756938bbd3f30358a997a7ca3ed5543e |
||
zaidoon
|
7a90dc5ba8 |
Add built-in index factory wrappers (#14961)
Summary: Let the built-in binary, hash, and partitioned indexes be created through `IndexFactory`. The wrappers preserve the existing internal-key handling and multi-block finish path. Pass the common-prefix flag to every builder, including both levels of a partitioned index. The regression test checks the prefix encoding and decoded keys with the flag on and off. The factory tests pass on the original PR plus the fix. Focused tests on the full stack also pass with `ASSERT_STATUS_CHECKED=1` and `COERCE_CONTEXT_SWITCH=1`. Full `make check` runs had test timeouts and hit the 30-minute local limit. Part 4 of 9. Previous: https://github.com/facebook/rocksdb/issues/14960. Next: https://github.com/facebook/rocksdb/issues/14962. The first three parts have landed. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14961 Reviewed By: jaykorean Differential Revision: D119493650 Pulled By: xingbowang fbshipit-source-id: 7d43cc456de86e8bd35e4b0689fb3d256608cf2f |
||
|
Josh Kang
|
88752298ff |
Add native coroutine read dispatch to RocksDB (#15073)
Summary: ## Summary Pull Request resolved: https://github.com/facebook/rocksdb/pull/15073 The callback-based async read APIs make coroutine-native implementations and StackableDB composition awkward because wrappers must bridge callbacks back into tasks. This change introduces `CoroDB` as a side capability exposing lazy `GetCoroutine` and `MultiGetCoroutine` tasks. `DBImpl` and `CompactedDBImpl` implement this capability directly, while `DB::GetAsync` and `DB::MultiGetAsync` remain available as compatibility entry points with synchronous fallback. The native interface should also be more efficient because callers directly await the DB coroutine. It avoids the callback/Baton adapter, callback context handoff, and explicit rescheduling required for each request. Native tasks must run on the DB filesystem read executor's EventBase; debug builds verify this affinity. Public API documentation, a release note, and native-coroutine support in `db_bench` and `db_stress` are included. ## Dispatch DB::GetAsync -> GetCoroDB() on the concrete DB -> virtual CoroDB::GetCoroutine(...) -> DBImpl::GetCoroutine or CompactedDBImpl::GetCoroutine -> AsyncCallback::OnComplete(...) Virtual dispatch selects the coroutine implementation belonging to the concrete DB while preserving the existing callback API. ## Statistics Coroutine reads preserve RocksDB's TLS statistics behavior while keeping concurrent requests isolated. Each top-level task copies only the caller's statistics configuration—`PerfLevel`, per-level perf context enablement, and IO statistics enablement—into fresh request-local `PerfContext` and `IOStatsContext` objects. Existing counters are not inherited. If both perf and IO statistics are disabled, no request statistics context is installed. Folly `RequestContext` preserves these contexts across suspension and resumption on the read executor's EventBase. At completion, RocksDB publishes the request's collected values to TLS, allowing the awaiting caller to use `get_perf_context()` and `get_iostats_context()` as after a synchronous read. Callers must consume or copy them before leaving that EventBase. Statistics scopes are installed only by top-level coroutine reads and cannot be nested. ## Benchmarks Compared the previous `GetAsync` callback bridge with native `GetCoroutine`. Both variants used optimized builds, the same compacted DB, a fixed seed, eight executor threads, ten jobs per thread, and 2.56M reads per invocation. Results are the mean of three independent invocations using the default RocksDB block cache. A sequential read warmed the OS page cache before the non-direct-read measurements. | Workload | `GetAsync` bridge | Native `GetCoroutine` | Change | | Direct I/O | 237,710 ops/s | 243,588 ops/s | +2.47% | | Warm OS page cache | 783,859 ops/s | 891,531 ops/s | +13.74% | The direct-I/O difference is within observed run-to-run variation. When reads are served from the OS page cache and API overhead is a larger portion of latency, the native interface improves throughput by 13.74%. Every invocation completed all requested reads. ```bash db_bench \ --benchmarks=readrandomcoroutine \ --db=$CORO_BENCH_DB \ --use_existing_db=true \ --num=2000000 \ --value_size=1024 \ --compression_type=none \ --threads=8 \ --coro_jobs_per_thread=10 \ --reads=320000 \ --use_direct_reads=<true|false> \ --perf_level=1 \ --statistics=true \ --seed=20260807 ``` Reviewed By: xingbowang Differential Revision: D115232511 fbshipit-source-id: 30b48da2a798094d9080178f5430aae26e6c5954 |
||
|
Peter Dillinger
|
fd7399004f |
Add lazy wide-column read API (GetEntityLazy / MultiGetEntityLazy) (#15012)
Summary:
Adds an experimental **lazy wide-column read API** that lets callers enumerate a
wide-column entity and read back only the columns (and, in a later phase, the
byte ranges) they actually need, instead of eagerly materializing every
blob-backed column like `GetEntity()` / `MultiGetEntity()`.
New public entry points (`include/rocksdb/db.h`):
- `DB::GetEntityLazy(read_options, cf, key, LazyWideColumns* result)`
- `DB::MultiGetEntityLazy(read_options, cf, num_keys, keys, LazyWideColumnsBatch* result, statuses, sorted_input)`
Both leave a wide-column entity's **inline** columns materialized up front but
its **blob-backed** columns as *unresolved references*, so blobs for columns
that are never pulled are never read.
### Result types (`include/rocksdb/lazy_wide_columns.h`)
- **`LazyWideColumn`** — the lazy analogue of `WideColumn`: `name()`, `index()`,
`inline_value()` (an `OptSlice`; empty ⇒ unresolved reference),
`is_reference()`, and `logical_size()` (`std::optional<uint64_t>`; known for
inline and uncompressed references, empty for compressed ones). All no-I/O.
- **`LazyWideColumns`** — a move-only result that is **indexable and iterable
like a vector** of `LazyWideColumn` (`size()`, `operator[]`, range `begin/end`).
Unlike `PinnableWideColumns`, it may be used after the producing call returns:
it **pins the SuperVersion** (like an iterator) so referenced blob files / SST
readers stay valid for deferred reads.
- **`LazyWideColumnsBatch`** — the `MultiGetEntityLazy` result; vector-like over
its per-key `LazyWideColumns`.
### Resolution
- **`LazyColumnReadRequest { const LazyWideColumn* column; uint64_t offset;
size_t length; bool force_verify; PinnableSlice* result; Status* status; }`** —
a read names a **typed column pointer**, not an untyped index. Because a
column identifies its owning result (and, for a batch, its owning entity),
`MultiResolve` routes each read correctly and **rejects a null or foreign
column** with `InvalidArgument` on that read's per-read status — cross-result
misuse can't be silently mis-indexed.
- **`LazyWideColumns::MultiResolve` / `LazyWideColumnsBatch::MultiResolve`** take
the same request type (pointer+count and `std::vector` sugar overloads); the
batch form resolves reads spanning any entities of the batch in one call.
- **`ResolveColumn(column, result)` / `ResolveColumnRange(column, offset, length,
result)`** — per-column sugar over a one-entry `MultiResolve`.
- A column's blob is fetched **at most once** over the result's lifetime (bytes
are cached in the result), so repeated reads of the same column reuse that
fetch. Resolution honors the `ReadOptions` captured at `GetEntityLazy()` time
(e.g. `read_tier`, `verify_checksums`); the resolve methods intentionally take
no per-resolve `ReadOptions`.
### Where it works
- Primary, **read-only**, and **secondary** DB instances (each `GetImpl`
override forwards the lazy signal and transfers a SuperVersion pin via the
shared `DBImpl::TransferSuperVersionPin` helper).
- Forwarded through `StackableDB` wrappers (e.g. `TransactionDB`) rather than
falling through to the `NotSupported` base default.
- Requires `max_open_files == -1` (immortal table readers) so same-file
("embedded") references stay resolvable lazily; otherwise the call returns
`InvalidArgument`.
## Implementation notes
- Read path: `GetContext` gains a lazy mode that leaves blob references
unresolved and captures the SST's `SameFileBlobReader`; `Version::Get` forwards
the new (defaulted) `lazy_columns_same_file_reader` out-param.
- `ReadPathBlobResolver` performs on-demand, cached blob resolution bound to the
pinned `Version` + decoded blob references.
- Internal identity is by heap-stable `Rep` pointers, so column/result/batch
pointers stay valid across moves with no move-time fix-ups.
## Scope / current limitations
- Resolves **whole columns**; byte ranges are served by slicing the whole column
for now (so `length`/`offset`/`force_verify` don't yet save I/O or skip
checksums). Single column family. Synchronous. `kBlockCacheTier` on the
originating call yields `Incomplete` on a blob-cache miss.
- User-defined timestamps: entities aren't writable under UDT, so `GetEntityLazy`
is at parity with `GetEntity` (same read-timestamp validation; plain values
return a single default column).
- Planned follow-ups: partial (byte-range) blob reads that skip checksum/cache-
fill, cross-key coalescing, and asynchronous execution.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15012
Test Plan:
- **`db/wide/db_lazy_entity_test.cc` (15 tests, all passing):** enumeration with
no blob I/O; single range and whole-column resolution; `MultiResolve` of many
blobs and mixed ranges/columns (incl. the `std::vector` overload); cross-key
batch resolution; result-outlives-call / pin survives move; unpulled columns
never read; `max_open_files == -1` requirement; `kBlockCacheTier` ⇒
`Incomplete`; foreign/null column ⇒ `InvalidArgument` (single-entity and
batch); UDT parity with `GetEntity`; `StackableDB` forwarding; and read-only
and secondary instances.
- **db_stress:** `GetEntityLazy` / `MultiGetEntityLazy` run as a differential
against eager `GetEntity` / `MultiGetEntity` (gated by `--lazy_entity_read_one_in`,
which only takes effect under `--open_files=-1`), including coverage on
read-only (`--open_read_only_one_in`) and secondary (`--test_secondary`)
instances. Smoke-run (`EXIT=0`, "Verification successful") with integrated blob
files + `--use_put_entity_one_in`.
- `make format-auto` clean; builds `ASSERT_STATUS_CHECKED`-clean.
Reviewed By: xingbowang
Differential Revision: D113792365
Pulled By: pdillinger
fbshipit-source-id: cb73fc845fe0e20ed352d697e552e7b47b2eb9cf
|
||
Xingbo Wang
|
460aea1523 |
Fix C API compatibility baseline selection in CI (#15039)
Summary: The C API compatibility job always fetched `main` as its baseline. On release branches, that can report false API removals for symbols that were added to `main` after the release branch was created. Select the compatibility baseline from the triggering event instead: - Pull requests use the merge target commit. - Pushes use the branch's pre-push commit. - Newly created branches or tags use `HEAD`, since they have no prior ref state to preserve. Fetch the selected commit by SHA, fail explicitly for unsupported events or missing event data, and pass the resolved baseline to `check-c-api-gen`. Also update the Makefile documentation to describe the PR and push behavior. Pull Request resolved: https://github.com/facebook/rocksdb/pull/15039 Test Plan: CI Reviewed By: pdillinger Differential Revision: D114437175 Pulled By: xingbowang fbshipit-source-id: ee66f54bcabba238477d07a69ec7dcb42219c088 |
||
|
Peter Dillinger
|
b7b6d7c8c0 |
Mirror CI third-party build deps on a GitHub release instead of S3 (#15041)
Summary: CI/build downloaded pinned third-party binaries (compression source tarballs, Java test jars, and clang-format-diff.py) from an S3 bucket. Move the mirror to assets on an unlisted "third-party-deps" GitHub release, so the files live alongside the code with no separate bucket to maintain and no per-repo bandwidth quota (unlike Git LFS). Recent releases on the RocksDB release page are not polluted (https://github.com/facebook/rocksdb/releases/, far down the list probably from tagging a very old release). The compression download variables are refactored from *_DOWNLOAD_BASE (a directory URL) to *_DOWNLOAD_PREFIX (directory + "<project>-", or "...archive/v"), so each release asset can carry a project-name prefix in the otherwise flat release namespace while the upstream defaults still resolve to byte-identical URLs. The Java DEPS_URL defaults (Makefile and CMake) and the clang-format-diff.py download are repointed at the release. A comment at each dependency site links to the release edit page for adding/updating assets. Pull Request resolved: https://github.com/facebook/rocksdb/pull/15041 Test Plan: CI and... Downloaded all 11 assets from the release and confirmed their SHA256 sums match the files previously served from S3 (tarball/jar sums are also enforced by the existing Makefile checksum checks). Verified the refactored Makefile upstream defaults expand to the same URLs as before, and that the CI *_DOWNLOAD_PREFIX overrides plus DEPS_URL resolve to the uploaded asset names. Covered by the normal PR/nightly/weekly jobs that build compression from source and run the Java tests (Make and CMake). Reviewed By: xingbowang Differential Revision: D114443886 Pulled By: pdillinger fbshipit-source-id: 8c59435e7064f0d9931cba103c8120fe113b0d6d |
||
|
Peter Dillinger
|
026f63de6c |
Parallelize the non-unit-test checks in make check (#15028)
Summary:
`make check` ran a series of extra checks serially, one after another, after the GNU-parallel unit tests: the Python syntax check, ldb/db_crashtest/ rocksdb_dump tests, and the format/buck/sources/workflow/c-api-gen source checks. These had accumulated into a non-trivial single-threaded tail on every `make check`.
Each extra check is now a make target collected in a single CHECK_EXTRA_TARGETS list, and gen_check_scripts emits a `t/run-<target>` wrapper per enabled check. check_0 already runs every `t/run-*` as a gnu_parallel job, so the checks now run alongside the unit tests, hiding their cost behind the (dominant) test phase. The slow ones (check-c-api-gen, check-format) are front-loaded via slow_test_regexp, and under CI sharding the checks are distributed round-robin so each runs on exactly one shard. The J=1 / no-gnu-parallel fallback runs them serially. New check-all-python and rocksdb_dump_test targets wrap the two steps that were previously inlined; the guards match the old serial tail exactly.
Each wrapper frames its result with a gtest-style
`[ RUN ]` / `[ OK ]` / `[ FAILED ] >>> make <check> <<<` line so a non-unit-test failure is easy to distinguish and re-run.
The wrappers spawn sub-makes concurrently, and each make invocation otherwise re-runs build_detect_platform, which rewrites the shared make_config.mk non-atomically -- concurrently that corrupts it ("missing separator"). The wrappers pass a new ROCKSDB_RECURSIVE_MAKE_SAME_CONFIG=1, which reuses the parent's already-generated make_config.mk (guarded on it existing, so normal builds still detect) and also silences the parse-time banners for clean output.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15028
Test Plan: `make check` runs the same set of auxiliary checks as before, now as parallel jobs. Verified that gen_check_scripts emits one wrapper per enabled check, that the SKIP_FORMAT_BUCK_CHECKS / ASSERT_STATUS_CHECKED / AIX guards still gate the set, that CI_TOTAL_SHARDS distributes the checks one-per-shard (union across shards is the full set), and that many concurrent wrappers no longer corrupt make_config.mk. A seeded check failure surfaces framed as `[ FAILED ] >>> make <check> <<<`.
Reviewed By: xingbowang
Differential Revision: D114245857
Pulled By: pdillinger
fbshipit-source-id: 519f78ccdcfbfdd88509980f78944e527a572331
|
||
Ranjan Banerjee
|
0c0bfbe8be |
Add hard-link support to CopyEngine [3/4] (#14947)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/14947 Teaches the shared `CopyEngine` to hard-link files through its background pool: adds `WorkItemType::Link` and `CopyEngine::LinkFile(src, dst, dst_env)`, which returns the FileSystem's `NotSupported` when links are unavailable so callers can fall back to copying. This is what CheckpointEngine (next diff) needs for warm storage, where linking is supported but benefits from parallelism. No existing caller links through the engine yet (BackupEngine never hard-links), so no behavior change; the path is exercised by a new `copy_engine_test.cc` (parallel copy, parallel link, single-thread copy, per-item error reporting). Reviewed By: xingbowang Differential Revision: D111387972 fbshipit-source-id: ec0b0e9b80749a3f51e62066ceef2df21a1811d2 |
||
|
Peter Dillinger
|
755c104bca |
Unify clang detection for the C API codegen and add make regen-c-api (#14994)
Summary: `USE_CLANG=1 make check-c-api-gen` failed with "clang++ not found" because the staleness check and the Python generator detected clang inconsistently: the Makefile inspected the `$(CXX)` make variable (the fbcode clang path) while the generator read the `CXX` environment variable, which make never exports. More broadly, clang++ was detected in several ad hoc places with fragile heuristics (grepping the compiler path, duplicated fallback lists in shell and Python). Centralize clang++ detection in build_detect_platform as a single `CLANG_CXX`: - `CLANG_CXX` honors `$CXX` when it is itself clang (so CI's `CXX=clang++-21` is used), else takes the first clang++ on PATH (bare, versioned, or the Meta-internal clang++.par). `CXX` (when building with clang) and `CLANG_ANALYZER` now derive from `CLANG_CXX` instead of re-detecting. - Infer `USE_CLANG=1` when the compiler is clang, detected via the compiler's own `__clang__` predefined macro rather than matching the binary path (which false-positives, e.g. a home dir containing "clang"), and emit it so clang-specific Makefile logic is consistent. - Emit `CLANG_CXX` to make_config.mk and pass it explicitly to the codegen, so `make check-c-api-gen` works without requiring `USE_CLANG=1` and the generator never guesses. Add a first-class `make regen-c-api` target -- now the preferred regenerate command referenced by the generated banners and user-facing messages. The Python generators no longer detect clang themselves; `--clang` defaults to `clang++` for manual runs and is overridden by the Makefile with `CLANG_CXX`. Regeneration verifies the clang runs before writing any file, so a missing/broken toolchain aborts cleanly instead of leaving a partially regenerated tree. Make field discovery clang-version-independent: older clang (e.g. 15) spells field types fully qualified (`rocksdb::ReadTier`, `rocksdb::BlockBasedTableOptions::IndexType`) while the fbcode/CI clang (21) uses the bare `ReadTier` / `IndexType` the family tables expect. Strip the ROCKSDB namespace and enclosing-struct qualifiers so both agree; this is a no-op on newer clang, so checked-in output and CI are unaffected. Also escape the literal "generated" in the generator .py sources (they emit the marker but are not themselves generated) so tooling stops treating them as read-only generated files. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14994 Test Plan: Manual verification (not covered by CI): - `make check-c-api-gen` now runs the staleness sub-check (instead of skipping) and passes in a default gcc build as well as with `USE_CLANG=1` on an fbcode host, since `CLANG_CXX` is always populated. - `ROCKSDB_NO_FBCODE=1 make regen-c-api` using a system clang 15 (clang++.par) produces output byte-identical to the checked-in files (generated with clang 21), and `check-c-api-gen` reports "up to date" -- confirming the namespace/struct stripping is version-robust and a no-op on clang 21. - `regen_all.py --clang /nonexistent` exits non-zero with a clear message and leaves the generated files untouched. - `make regen-c-api` is idempotent (no diff on a clean tree). - Confirmed the escaped `\x40generated` still emits `// generated` byte-for-byte in the regenerated c.h/c.cc/.inc. - Reviewed .github/workflows/pr-jobs.yml: the build-linux-clang-21-no_test_run job's `CXX=clang++-21 make check-c-api-gen` resolves `CLANG_CXX` to clang++-21 (honoring `$CXX`) rather than the container's stray bare clang++. Reviewed By: xingbowang Differential Revision: D113447462 Pulled By: pdillinger fbshipit-source-id: 91ca92d21a71a8b92f9ff57500a8cec91a884078 |
||
|
Peter Dillinger
|
a40466d963 |
Three misc CI fixes (#14910)
Summary: Fixes three unrelated nightly/CI failures. Makefile: 'make build_folly' was tripping the build-signature change check, causing the folly-flavored jobs (e.g. nightly clang-21 ASAN+UBSAN+folly, which runs 'make build_folly' then 'USE_FOLLY=1 ... make check') to abort with 'Build parameters changed since the last build'. build_folly only builds the external folly library under third-party/folly and never compiles RocksDB objects into OBJ_DIR, so it should not record or check a build signature; add it to BUILD_SIG_NONBUILD_GOALS alongside checkout_folly. util/rate_limiter_test.cc: RateLimiterTest.Rate's minimum-rate assertion is timing-sensitive and flakes on the small/loaded ARM CI runners, which (unlike x86_64) can't reliably sustain the target rate. Extend the existing CI exemption with an ARM-only GITHUB_ACTIONS branch, preserving the check on x86_64. tools/ldb_test.py: the list_live_files_metadata output includes the full DB path, and the unescaped-dot regex '\d+.sst' could match a <digit><char>sst run inside the random tempfile.mkdtemp suffix (drawn from [a-z0-9_]) that precedes the real filename, collapsing testMap to a single bogus key such as '8asst'. Require a literal '.sst'; temp-dir characters can never contain a dot, so only the real filename matches. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14910 Test Plan: - build_folly: confirmed 'make build_folly' no longer writes .build_signature and that BUILD_SIG_DO_BUILD resolves empty for that goal. - rate_limiter_test: confirmed via the C preprocessor that x86_64 keeps the SANDCASTLE-only exemption while aarch64 selects the new GITHUB_ACTIONS branch. - ldb_test: reproduced the collapse with a poisoned temp suffix (produces {'8asst': ['0', 'mycol2']}) and confirmed the escaped-dot regex yields the correct 4-entry map. Reviewed By: mszeszko-meta Differential Revision: D110405164 Pulled By: pdillinger fbshipit-source-id: a9a73b7a717e726fdf79d7795f085f3dbede940e |
||
|
Peter Dillinger
|
f6fabdb64f |
Improve build and test experience for Make builds (#14883)
Summary: Local Make builds silently reuse object files even when build parameters (DEBUG_LEVEL, sanitizers, ASSERT_STATUS_CHECKED, RTTI, etc.) change, since all object files share the same paths. This leads to confusing linker errors, sanitizer false negatives, ODR violations, and "phantom" bugs that are easy to miss -- a pitfall for humans and especially for AI agents driving builds non-interactively. This change improves the Make experience on three fronts: 1. Build-parameter change detection in the Makefile. After flags are fully resolved, we hash the effective compile/link inputs (CC|CXX|CFLAGS|CXXFLAGS|LDFLAGS|EXEC_LDFLAGS) into a per-OBJ_DIR .build_signature stamp and compare against the previous build: - Default: stop with a clear error when parameters changed. This is optimized for the expert who might not intend a full rebuild, and for avoiding CI inefficiency. - AUTO_CLEAN=1: automatically run clean-rocks and proceed. - ALLOW_BUILD_PARAMETER_CHANGE=1: skip the check (e.g. intentionally mixing DEBUG_LEVEL=1 and DEBUG_LEVEL=2 objects). The check is skipped for dry runs (-n) and for non-building / self-cleaning targets (clean*, format, check-*, tags, gen-pc, check-progress, watch-log, release, coverage, the asan_*/ubsan_* variants, etc.). make_config.mk is removed only by the top-level `clean` target, not by clean-rocks, so the auto-clean does not delete the make_config.mk already included by the running make. The signature is removed last in clean-rocks so an interrupted clean keeps change detection meaningful. 3. New build_tools/rockstest.sh and build_tools/rocksptest.sh helpers that build and run unit tests in one step. They set AUTO_CLEAN=1 and build with -j<NCORES> (computed like the Makefile). rocksptest.sh runs one or more binaries under the checked-in gtest-parallel (sharing one parallel worker pool); rockstest.sh runs a single binary directly for a small number of cases. Both reject a leading-option first argument; `rockstest.sh install` writes ~/bin/rockstest and ~/bin/rocksptest shims that defer to the in-tree scripts (with a friendly message when run outside a source root). This is a big win for AI-agent workflows: one command instead of a separate `make` then test-run invocation, removing the overhead of monitoring two long commands, while avoiding serial builds and serial test execution. 4. CLAUDE.md guidance updated to recommend AUTO_CLEAN=1 for manual make invocations and the rocks(p)test.sh helpers for running tests Although there might be drive to consolidate around the BUCK build for internal use, I'll note that last I checked it was around ~20 minutes to run all the unit tests under buck and ~2 minutes under `make check`. Bonus: the first revision of this had copyright/license mistakes, and this change corrects similar errors elsewhere, with CLAUDE.md advice updated. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14883 Test Plan: Manual verification of the Makefile change detection: - Confirmed the resolved-flags hash is deterministic across runs for the same goal, and differs between configs (e.g. default/shared vs static_lib, and with/without ASSERT_STATUS_CHECKED). - Unchanged parameters: rebuild does not error. - Changed parameters: errors by default; AUTO_CLEAN=1 runs clean-rocks and proceeds; ALLOW_BUILD_PARAMETER_CHANGE=1 bypasses the check. - Dry runs (make -n) and exempt targets (e.g. `make check-progress`, list_all_tests, gen-pc) do not trip the check and leave the stamp untouched. - Reproduced the ASSERT_STATUS_CHECKED=1 auto-clean path end to end and confirmed it no longer fails with "No rule to make target 'make_config.mk'"; verified `make clean` removes make_config.mk while `clean-rocks` does not, and that the signature is deleted as the final clean-rocks step. Manual verification of the helper scripts: - shellcheck-clean and `bash -n` pass for both scripts. - Usage/guard behavior: missing argument and leading-option first argument are rejected with a clear message. - `rockstest.sh install` (validated against a temp HOME) generates both shims; running a shim outside a source root prints "(Not in a rocksdb source root directory?)" and exits non-zero. - rocksptest.sh argument splitting verified for single binary, multiple binaries, and binaries followed by gtest-parallel/test args. - .gitignore patterns verified with `git check-ignore` for .build_signature (root and jl/jls) and build_tools/__pycache__/. Reviewed By: xingbowang Differential Revision: D109713512 Pulled By: pdillinger fbshipit-source-id: a801d29afa91f53a2dce717db9fe5e5d485cc154 |
||
|
zaidoon
|
1cec28d82d |
Finish C API code generation (continues #14572) (#14868)
Summary: This continues and finishes **https://github.com/facebook/rocksdb/issues/14572** ("Add semi-automated code generation for RocksDB C API bindings") by xingbowang. The original author is unavailable to finish it, so I've taken it over. **All 13 of the original commits are preserved** (this branch was created from the PR head and builds on top of it — `git log` shows the original `Xingbo Wang` authorship intact); my follow-up work is in the commits prefixed `C API codegen:`. The underlying design is unchanged and is the original author's: hand-written source templates (`tools/c_api_gen/c_base.h` / `c_base.cc`) plus two generators (auto-discovery from the C++ headers + a spec-driven generator) are inlined into a single, self-contained, `generated` `include/rocksdb/c.h` and `db/c.cc`. This grows the public C API by **668 functions** while keeping `c.h` a single includable header with no `-I` requirement (so `bindgen` and other FFI tools keep working unchanged). This branch reconciles the PR with ~4 months of `main` and addresses the outstanding review feedback (clang-tidy bot, the automated code review, the `c.h` self-containedness discussion, and pdillinger's points about `include/rocksdb` hygiene and `generated` marking). ## What changed on top of the original PR ### Reconciled with current `main` - Merged current `main` (conflicts were confined to the generated/test files) and regenerated. Reconciled the 14 C API functions `main` added since the merge-base (e.g. `rocksdb_set_db_options`, the backup-engine rate limiters, `memtable_batch_lookup_optimization`, `optimize_multiget_for_io`, …) and restored 5 enum constants that upstream had added by hand (`rocksdb_txndb_write_policy_*`, `..._index_block_search_type_auto`, `rocksdb_blob_cache_read_byte`). ### Maintainer feedback (pdillinger) - **No non-user-includable files in `include/rocksdb`.** Moved the hand-written templates out of `include/rocksdb/` and `db/` to `tools/c_api_gen/c_base.{h,cc}`. They were `#include`-ing generated fragments, which broke `make check-headers` and was shipped by `make install`. `include/rocksdb/` now contains only the user-facing, self-contained, `generated` `c.h`. - `c.h` / `c.cc` carry the `// generated` marker. ### Backward compatibility (zero ABI break) - The generator derived each wrapper's C type purely from the C++ field, which had silently changed **5 already-shipped signatures** (e.g. `rocksdb_writeoptions_disable_WAL` `int` → `unsigned char`). Added an ABI type-pinning layer (`tools/c_api_gen/abi_type_overrides.json`) so already-shipped functions keep their historical C signature (the body still casts to the real field type). A repo-wide diff against the merge-base now reports **0 ABI drift**. - New `check_api_compatibility.py` gate (wired into CI + `make`) fails on any removed/changed public function **or** removed enum/typedef symbol, vs a reference revision. Intentional changes go in an allowlist with a reason. ### Correctness (from the automated review) - Restored 5 option setters that were declared in `c.h` but **defined nowhere** (link failure for downstream bindings such as `rust-rocksdb`). Added `check_api_completeness.py` (dependency-free; runs in CI + `make`) asserting every declared function has exactly one definition — this is the gate that would have caught it. - `CopyStringVector` now null-checks `malloc`; the WAL filter `std::move`s the `WriteBatch`; the backup exclude-files callback captures by value instead of the wrapper pointer. ### Build / CI robustness - Removed the dead `C_API_CODEGEN_STAMP` Makefile prerequisite (it was a silent no-op). - The `make check` staleness check is now opt-out-able (behind `SKIP_FORMAT_BUCK_CHECKS`) and skips gracefully when `clang++` is unavailable, so `make check` works without the codegen toolchain. CI remains the authoritative gate. - Pinned `clang-format` consistently through `regen_all.py` / `verify_generated_up_to_date.py` (CI uses clang-format-21) so regeneration is byte-reproducible across environments. - Cleared all 20 `clang-tidy` warnings the bot reported on `db/c.cc` changed lines (fixed in the `c_base.cc` template, not the generated output). - Updated the internal Buck `c_test_bin` wrapper to expose generated `c_api_gen/*.inc` fragments as headers, so sandboxed Buck builds can compile `db/c_test.c` after the generated round-trip tests are included. ### Test coverage - Added `gen_roundtrip_tests.py`, which derives **462 set→get→assert round-trip checks across 25 option objects** from the same generated fragments and wires them into `db/c_test.c`. Coverage now tracks the generated surface automatically. ### Docs - Added the `unreleased_history/public_api_changes` note and fixed `claude_md/add_public_api.md`, which still told contributors to hand-edit the now-`generated` `c.h`/`c.cc`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14868 Test Plan: - `make c_test && ./c_test` — **passes**, including the 462 generated round-trip assertions (a successful link also confirms the API is complete). - `python3 tools/c_api_gen/check_api_completeness.py` — all 1737 declared functions defined exactly once. - `python3 tools/c_api_gen/check_api_compatibility.py --ref <release>` — 1070 reference functions + 229 enum/typedef symbols preserved, 0 removed/changed. - `python3 tools/c_api_gen/verify_generated_up_to_date.py` — generated output is stable. - `include/rocksdb/c.h` confirmed self-contained (only `<stdbool.h>`, `<stddef.h>`, `<stdint.h>`). cc xingbowang - `buck2 build --flagfile fbcode//mode/dev fbcode//internal_repo_rocksdb/repo:c_test_bin` — passes. - `buck2 build --flagfile fbcode//mode/dev --config fbcode.arch=aarch64 fbcode//internal_repo_rocksdb/repo:c_test_bin` — passes. Reviewed By: pdillinger Differential Revision: D109149150 Pulled By: xingbowang fbshipit-source-id: 3417375345f360a4c78bdfe27e9850b89d0a226a |
||
|
Peter Dillinger
|
3883a8d05e |
Rename db_test2 -> db_etc2_test (#14218)
Summary: When searching/grepping through unit tests, it's convenient to use test.cc suffix to match all unit tests, or to exclude test.cc when excluding unit tests from a code search. (I've even seen my AI assistant grep through `*test.cc`.) So I'm renaming db_test2.cc (as planned in https://github.com/facebook/rocksdb/issues/14076) Pull Request resolved: https://github.com/facebook/rocksdb/pull/14218 Test Plan: existing tests + CI Reviewed By: xingbowang Differential Revision: D90140624 Pulled By: pdillinger fbshipit-source-id: 78aa099290670bbb4093b2c7c02bc47ab3bf5f8e |
||
|
Peter Dillinger
|
e82af29adb |
Makefile fix and speed up 'clean' (#14767)
Summary: * Fix Makefile default target (was ordered after a folly target) * Improved the speed of `make clean` by using just one `find` and by pruning "hidden" .* and third-party directories that should not be modified anyway. * Reduce excessive output from `make clean` Pull Request resolved: https://github.com/facebook/rocksdb/pull/14767 Test Plan: manual Reviewed By: mszeszko-meta Differential Revision: D105972958 Pulled By: pdillinger fbshipit-source-id: 2c0f6097c74c3129b815450f23c19ef07bfbe656 |
||
|
Peter Dillinger
|
c48b020e92 |
Speed up parallel 'make check' scheduling (#14745)
Summary: Reduce wall-clock time of parallel 'make check' by improving the scheduling and granularity of slow test binaries. Three Makefile changes: 1) Refresh slow_test_regexp with current observed bottlenecks. Adds binaries (point_lock_manager_stress_test, compaction_service_test, corruption_test, comparator_db_test, external_sst_file_basic_test, rate_limiter_test, db_compaction_test, db_merge_operator_test, db_dynamic_level_test, db_bloom_filter_test, error_handler_fs_test, merge_helper_test, db_kv_checksum_test, inlineskiplist_test) whose shards take >=15s but were not being front-loaded for early queueing. Also drops stale FIXME comments that no longer apply and adds tier annotations + a maintenance recipe. 2) Add SHARD_SIZE_OVERRIDES, a per-binary override of GTEST_SHARD_SIZE, so binaries with slow individual tests (e.g. point_lock_manager_stress_test where each test is ~10s) can be chopped into more, smaller shards. The default of 10 stays for everything else. Each shard's effective size is reported in the 'Generating ... shards for ...' line. 3) Add 'make suggest-slow-tests' to print a per-binary aggregation of the most recent LOG (max single-shard time, total time, shard count) for any binary worth attention. Used to maintain the regex and override list above. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14745 Test Plan: Two runs each of 'make -j166 check', before and after this change (all compilation already finished): Before: 197s and 198s After: 123s and 125s Reduction: 37% Reviewed By: xingbowang Differential Revision: D105332444 Pulled By: pdillinger fbshipit-source-id: 1d1c2f89a32647e6651e2ffeb72da9d51bcc004f |
||
xingbowang
|
224e849e8d |
env: suppress liburing TSAN false positives (#14710)
Summary: - Use the liburing TSAN suppressions in `tools/tsan_suppressions.txt` instead of defining the process-wide `__tsan_default_suppressions()` hook, avoiding conflicts with downstream applications. - Wire RocksDB TSAN make and crash-test flows to use that suppressions file by default without overriding caller-provided `TSAN_OPTIONS`. - Cover direct `db_crashtest.py` launches by passing the default suppressions to `db_stress` subprocesses. - Fix the GCC 16 unity-build warning in `CacheItemHelper` by directly initializing the no-secondary-cache helper fields instead of delegating with `this`. Imported from D101303486. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14710 Test Plan: - `COMPILE_WITH_TSAN=1 make -j128 env_test` - `timeout 60s ./env_test --gtest_filter=EnvPosixTest.IOUringAddressReuseNoTsanFalsePositive` - `python3 tools/db_crashtest_test.py` - `python3 -m py_compile tools/db_crashtest.py tools/db_crashtest_test.py` - CI: `build-linux-unity-and-headers` passed after the `CacheItemHelper` fix Reviewed By: hx235 Differential Revision: D104103800 Pulled By: xingbowang fbshipit-source-id: 6066d9abe02a3c44d75f9ce449889468c927ce56 |
||
|
Peter Dillinger
|
c28b4f0e12 |
Fixes and enhancements to pre-push hook (#14680)
Summary: The previous pre-push hook auto-formatted, committed, and re-pushed on behalf of the user. This was fragile: it's not clear whether the format fix should amend the current commit or create a new one. Adding a commit breaks populating the summary for creating a new PR. Also, the hook's internal re-push would drop flags like --set-upstream from the original command. Replace with a simple check-only approach that blocks the push and tells the user what to fix. Also add a new check for untracked source files (.cc, .h, .py, etc.) in tracked directories (excluding third-party/). These typically indicate files that were forgotten in the commit, which would cause the pushed code to fail to build. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14680 Test Plan: manual Reviewed By: xingbowang Differential Revision: D102855227 Pulled By: pdillinger fbshipit-source-id: 7dc2c4e7a2b2c392bf8da74d7ea43883c8c075a9 |
||
xingbowang
|
df5695cf24 |
Fix workflow YAML parsing and add validation (#14624)
Summary: This fixes the Claude review workflow YAML parse error and adds a CI validation step for GitHub Actions workflow YAML via `make check- workflow-yaml`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14624 Reviewed By: archang19 Differential Revision: D101225058 Pulled By: xingbowang fbshipit-source-id: 0f371dcbd70ece2c25220c96e43305d9665554b1 |
||
|
Xingbo Wang
|
3070f73e97 |
Wide-column blob separation: lazy resolution through read, compaction, and write paths (#14386)
Summary: Wide-column blob separation: lazy resolution through read, compaction, and write paths Extend blob direct write to support wide-column entities (PutEntity), and add lazy blob resolution for wide-column values across all read and compaction paths. **Write path -- PutEntity blob separation:** - BlobWriteBatchTransformer::PutEntityCF now extracts large column values (>= min_blob_size) to blob files and serializes V2 entities with BlobIndex references, matching the existing Put behavior. - Add MaybePreprocessWideColumns() static helper to share blob extraction logic between the WriteBatch transformer and the new PutEntity fast path. - Add PutEntityFastPath() in DBImpl that preprocesses columns (sort, blob extract, serialize) before calling WriteImpl, skipping the redundant WriteBatch transformation pass. Trace batch preserves the original columns. **Read path -- blob resolution for Get/MultiGet/Iterator:** - GetContext::SaveValue resolves V2 entity blob columns eagerly: for value (Get), resolves the default column's blob reference; for columns (GetEntity), resolves all blob columns and re-serializes as V1. - DBIter::SetValueAndColumnsFromEntity detects V2 entities, deserializes with DeserializeV2, and eagerly resolves all blob columns via a new ReadPathBlobResolver. Resolved values are cached in the resolver and wide_columns_ Slices point into the cache, avoiding copies. - Add ReadPathBlobResolver (new file) -- on-demand blob fetcher for the read path with per-column caching, used by both DBIter and GetContext. - BlobFetcher gains allow_write_path_fallback to read from in-flight direct-write blob files not yet visible through Version (pre-flush reads). - Memtable lookups for Get(key) on V2 entities with a blob default column now return the blob index with is_blob_index=true, triggering the existing BDW resolution in MaybeResolveWritePathValue. - MaybeResolveWritePathValue (renamed from MaybeResolveDirectWriteBlobIndex) now also resolves V2 entity blob columns for GetEntity/MultiGetEntity, re-serializing as V1 after resolution. **Compaction path -- filter, GC, and extraction:** - CompactionIterator::InvokeFilterIfNeeded handles V2 entities: FilterV3 gets eagerly-resolved column values for backward compatibility; FilterV4 gets a CompactionBlobResolver for lazy on-demand resolution. - Add CompactionFilter::FilterV4 with WideColumnBlobResolver* parameter and SupportsFilterV4() opt-in. Default delegates to FilterV3. - CompactionBlobResolver (new class) implements WideColumnBlobResolver for the compaction path with stats tracking. - ExtractLargeColumnValuesIfNeeded extracts inline columns to blob files during compaction (entities without existing blob columns only). - GarbageCollectEntityBlobsIfNeeded relocates blob values from old blob files to new ones during compaction GC, with helpers FetchBlobsNeedingGC, RelocateBlobValues, and SerializeEntityAfterGC. - PrepareOutput unified entity deserialization: single DeserializeV2 call reused by both filter and GC/extraction paths via entity_deserialized_ flag, avoiding redundant parsing. **Merge path -- V2 entity base value resolution:** - MergeHelper::MergeUntil, GetContext::MergeWithWideColumnBaseValue, and DBIter::MergeWithWideColumnBaseValue resolve V2 blob columns before calling TimedFullMerge, using ResolveEntityForMerge. **Blob garbage accounting:** - BlobGarbageMeter tracks blob file in/out flow for V2 entity blob columns via ForEachBlobFileNumber, used for accurate GC decisions. - FileMetaData::UpdateBoundaries tracks oldest_blob_file_number for V2 entities, ensuring blob files referenced by entities are not prematurely deleted. **Serialization improvements:** - WideColumnSerialization::SerializeV2Impl allocates serialized_blob_indices only for actual blob columns (not all columns) and uses autovector for name/value sizes. - Add ForEachBlobFileNumber for lightweight blob file number extraction without full deserialization. - Add ResolveEntityForMerge helper for merge-path resolution. - Add section-size validation in DeserializeV2Impl. - Add empty blob index and column type validation. - blob_column_resolver_util.h -- shared helpers (FindBlobColumn, FindInCache, CacheInlinedBlob) used by both ReadPathBlobResolver and CompactionBlobResolver. **Testing:** - db_blob_direct_write_test: end-to-end PutEntity with BDW before/after flush, verifying Get, GetEntity, MultiGetEntity, and Iterator. - db_blob_index_test: ~1550 lines covering V2 entity blob resolution through Get, GetEntity, MultiGet, Iterator, compaction filter (V3 compat and V4 lazy), merge with blob base, and compaction GC/extraction. - compaction_iterator_test: ~950 lines testing entity blob GC, extraction, filter interaction, and combined GC+filter scenarios. - db_wide_basic_test: ~1200 lines for wide-column lazy blob resolution through all read paths plus compaction round-trips. - db_open_with_config_test: ~450 lines for BDW entity config validation. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14386 Reviewed By: anand1976 Differential Revision: D99739701 Pulled By: xingbowang fbshipit-source-id: 6badd89b577f3054802eaaa654738468efb9dbdb |
||
|
Xingbo Wang
|
d6b03d9c96 |
Add pre-push git hook to auto-format code (#14558)
Summary: Add a git pre-push hook that automatically runs `make format-auto` before every push. If formatting issues are found, the hook fixes them, creates a new commit, and retries the push — all in one shot. The hook is auto-configured on the first `make` build via `core.hooksPath`, so there is no manual install step needed. ## Changes **`githooks/pre-push`** — pre-push hook that: 1. Runs `make check-format` to detect formatting issues 2. If issues found, runs `make format-auto` to fix them 3. Creates a new "format" commit with the fixes 4. Retries the push automatically with the format commit included 5. Stashes/restores any uncommitted work so it is not affected 6. Skips in non-interactive environments (CI) unless `ROCKSDB_FORMAT_HOOK` is set 7. Can be skipped with `git push --no-verify` **`Makefile`**: - `setup-hooks` target: auto-sets `core.hooksPath=githooks` in the local repo config. Runs as a dependency of `all`, so any `make` build activates the hook with zero manual setup. - `install-hooks` / `uninstall-hooks` targets: manual copy-based alternative for those who prefer it Pull Request resolved: https://github.com/facebook/rocksdb/pull/14558 Reviewed By: archang19 Differential Revision: D99324330 Pulled By: xingbowang fbshipit-source-id: b824e56d572ad423fab7de7e15ead5fa8fd8e847 |
||
|
Xingbo Wang
|
af4e32945b |
Blob direct write v1: write-path blob separation with partitioned files (reduced scope) (#14535)
Summary: This PR introduces **blob direct write v1**, a reduced-scope write-path optimization where large values (>= `min_blob_size`) are written directly to blob files during `Put()` and replaced in the memtable with compact `BlobIndex` references. This avoids holding full values in memory until flush time. ### Motivation In the existing BlobDB architecture, values are written to the WAL and memtable in their full form and separated into blob files only at flush time. This means: - Large values are held in memory twice (raw in memtable + blob file at flush) - Blob I/O is serialized through a single flush thread per column family Blob direct write addresses both: values leave the write path as small `BlobIndex` references, and multiple **partitions** (configurable via `blob_direct_write_partitions`) allow concurrent blob writes with independent locks. ### Design (v1 — single-writer, WAL-disabled, reduced scope) The v1 design intentionally keeps scope narrow for correctness and reviewability: - **Single writer thread assumption**: no concurrent writes to the same partition file. One logical writer serializes the batch. - **WAL-disabled**: direct-write blob files are only registered in MANIFEST at flush time. WAL replay cannot recover unregistered blob references, so WAL is disabled for this v1. - **Flush-on-write**: each `AddRecord` call flushes to the OS immediately. - **FIFO generation batching**: each memtable switch creates one generation batch. Direct-write files for that memtable are sealed and registered atomically when the batch is flushed to MANIFEST. - **Round-robin partitions**: blob writes are distributed across `blob_direct_write_partitions` files using an atomic counter. ### New components | Component | Description | |---|---| | `BlobFilePartitionManager` | Owns N partition files per CF. Manages open/seal/register lifecycle tied to memtable generations. | | `BlobWriteBatchTransformer` | A `WriteBatch::Handler` that rewrites qualifying `Put` values as `BlobIndex` entries before the batch enters the write group. | ### Write path integration 1. `DBImpl::WriteImpl` calls `BlobWriteBatchTransformer::TransformBatch` before entering the writer group (for default write path), or before joining the batch group (for pipelined/unordered write). 2. Values >= `min_blob_size` are written to a partition file; the key is stored with a `BlobIndex` in the transformed batch. A rollback guard marks blob bytes as initial garbage if the write fails. 3. On `SwitchMemtable`, `RotateCurrentGeneration` moves active partitions into the next immutable batch. 4. `FlushMemTableToOutputFile` / `AtomicFlushMemTablesToOutputFiles` call `PrepareFlushAdditions` to seal partition files and collect `BlobFileAddition` + `BlobFileGarbage` entries registered to MANIFEST alongside the flush. 5. Shutdown paths (`CancelAllBackgroundWork`, `WaitForCompact` with `close_db=true`) force-flush all CFs with active direct-write managers to ensure blob files are registered before close. ### Read path - **Get/MultiGet**: `MaybeResolveBlobForWritePath` resolves `BlobIndex` references found in memtable or immutable memtable via `BlobFilePartitionManager::ResolveBlobDirectWriteIndex`, which first checks manifest-visible state and falls back to direct blob-file reads via `BlobFileCache`. - **Iterator**: `DBIter::BlobReader` is extended with a `BlobFilePartitionManager*` to resolve direct-write blob indexes during iteration. The unified `ResolveBlobDirectWriteIndex` path handles both manifest-visible and not-yet-flushed files. ### New options | Option | Default | Description | |---|---|---| | `enable_blob_direct_write` | `false` | Enable write-path blob separation for this CF. Requires `enable_blob_files = true`. Not dynamically changeable. | | `blob_direct_write_partitions` | `1` | Number of parallel partition files per CF. Not dynamically changeable. | ### Feature incompatibilities (reduced v1 scope) The following features are *not supported* when `enable_blob_direct_write = true`, and are enforced both in `db_stress_tool` validation and `db_crashtest.py` sanitization: **Write model constraints:** - `threads` must be 1 (single writer assumption) - `allow_concurrent_memtable_write` = 0 - `enable_pipelined_write` = 0 (transformation done before batch group, but pipelined path supported with pre-transform) - `two_write_queues` = 0 - `unordered_write` = 0 (transformation done before batch group, but unordered path supported with pre-transform) **WAL and recovery:** - `disable_wal` = 1 (required — WAL replay of unregistered blob files is out of v1 scope) - `best_efforts_recovery` = 0 - `reopen` = 0 (no crash-restart with WAL replay) - All WAL-related stress features disabled: `manual_wal_flush_one_in`, `sync_wal_one_in`, `lock_wal_one_in`, `get_sorted_wal_files_one_in`, `get_current_wal_file_one_in`, `track_and_verify_wals`, `rate_limit_auto_wal_flush`, `recycle_log_file_num` **Blob GC and dynamic options:** - `use_blob_db` = 0 (stacked BlobDB not supported) - `allow_setting_blob_options_dynamically` = 0 - `enable_blob_garbage_collection` = 0 - `blob_compaction_readahead_size` = 0 - `blob_file_starting_level` = 0 **Unsupported value types and APIs:** - Merge (`use_merge`, `use_full_merge_v1`) — merge values pass through untransformed - Entity APIs (`use_put_entity_one_in`, `use_get_entity`, `use_multi_get_entity`, `use_attribute_group`) - `use_timed_put_one_in` - User-defined timestamps (`user_timestamp_size`, `persist_user_defined_timestamps`, `create_timestamped_snapshot_one_in`) - Transactions (`use_txn`, `use_optimistic_txn`, `test_multi_ops_txns`, `commit_bypass_memtable_one_in`) — though `WriteCommittedTxn::CommitInternal` falls back from bypass-memtable to normal path when BDW is active - `IngestWriteBatchWithIndex` returns `NotSupported` - `inplace_update_support` = 0 **Fault injection:** - All write/read/metadata fault injection disabled (`sync_fault_injection`, `write_fault_one_in`, `metadata_write_fault_one_in`, `read_fault_one_in`, `metadata_read_fault_one_in`, `open_*_fault_one_in`) **Infrastructure/snapshot APIs:** - `remote_compaction_worker_threads` = 0 - `test_secondary` = 0 - `backup_one_in` = 0 - `checkpoint_one_in` = 0 - `get_live_files_apis_one_in` = 0 - `ingest_external_file_one_in` = 0 - `ingest_wbwi_one_in` = 0 ### Tests - `db/blob/db_blob_basic_test.cc`: ~660 lines of new direct-write unit tests covering basic put/get, multi-partition, flush/compaction, recovery, and error injection. - `db/blob/blob_file_cache_test.cc`: ~96 lines of new tests for direct-write blob file cache behavior. - `db/write_batch_test.cc`: ~96 lines of tests for WriteBatch with blob index entries. - `utilities/transactions/transaction_test.cc`: verifies transaction commit path falls back correctly with direct write enabled. - `db_stress_tool/`: full stress test support with `--enable_blob_direct_write` and `--blob_direct_write_partitions` flags, integrated into `db_crashtest.py` with 10% random selection alongside regular blob params. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14535 Test Plan: ``` make -j128 db_blob_basic_test && ./db_blob_basic_test make -j128 blob_file_cache_test && ./blob_file_cache_test make -j128 write_batch_test && ./write_batch_test make -j128 transaction_test && ./transaction_test make -j128 check ``` Stress test: ``` python3 tools/db_crashtest.py blackbox --enable_blob_direct_write=1 \ --enable_blob_files=1 --blob_direct_write_partitions=4 \ --disable_wal=1 --threads=1 ``` Reviewed By: pdillinger Differential Revision: D98766843 Pulled By: xingbowang fbshipit-source-id: 1577653826913a59d05680a87bce5534ac5a5e69 |
||
|
Xingbo Wang
|
7832ee886f |
db_stress: fix prefix scan correctness and crashtest stability (#14512)
Summary: This PR fixes several correctness and stability issues in the stress test and crash test infrastructure, plus adds regression coverage for trie UDI iterators. ### db_stress: constrain batched prefix scans `BatchedOpsStressTest::TestPrefixScan` opens 10 iterators with different prefixes and compares results in lockstep. Without `prefix_same_as_start`, unconstrained iterators could return keys beyond their seek prefix, producing more entries than bounded iterators and causing spurious assertion failures. Fix: set `prefix_same_as_start = true` on all iterators so every iterator stops at the seek prefix boundary. ### db_stress: skip invalid cross-prefix iterator checks When prefix iteration is enabled, `ReadOptions` requires that the seek key and `iterate_lower_bound` share the same prefix. Previously, stress test verification could run against configurations where they don't (e.g. when `lower_bound` spans a prefix boundary), producing false positives. Fix: detect this invalid configuration and mark the check as diverged (skip verification) rather than asserting. ### db_crashtest: disable BlobDB in best-efforts recovery BlobDB is not compatible with best-efforts recovery mode. Fix: explicitly disable blob file options when `best_efforts_recovery=True` in db_crashtest.py to avoid spurious failures. ### db_crashtest: preserve expected-state dirs across restarts The expected-state directory was being wiped on certain restart paths, causing verification failures on the next run. Fix: preserve expected-state dirs across db_crashtest restarts. Adds a new `db_crashtest_test.py` test to verify the behavior, runnable via `make db_crashtest_test`. ### Add trie UDI iterator regression coverage Adds regression tests for trie-based UserDefinedIndex (UDI) iterators covering snapshot-based reads, lower/upper bound iteration, `auto_refresh_iterator_with_snapshot`, and multi-version key handling. Also fixes an MSVC C4244 warning (int→char implicit narrowing) in the test. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14512 Reviewed By: hx235 Differential Revision: D98302018 Pulled By: xingbowang fbshipit-source-id: 84f5878665e5aeb61338ec2b6bfb2d2b077bb9f2 |
||
Danny Chen
|
ae322e0a73 |
Add SortedRunBuilder: use RocksDB as an external… (#14499)
Summary: CONTEXT: MyRocks and other third-party users currently need to understand deep RocksDB internals to sort unsorted data into SST files. The existing pattern requires: opening a full DB instance, configuring VectorRepFactory with universal compaction, disabling auto-compaction, writing unsorted data, manually triggering CompactRange with kForceOptimized, collecting output via GetColumnFamilyMetaData, and finally ingesting via IngestExternalFile with allow_db_generated_files. This is error-prone and requires ~30 lines of RocksDB configuration knowledge. WHAT: This adds a new utility class, SortedRunBuilder, that wraps all of the above into a simple Create/Add/Finish API. Callers feed in unsorted key-value pairs (in any order, from any number of threads) and receive sorted SST files with seqno=0 ready for ingestion -- or can iterate sorted output directly. The name SortedRunBuilder was chosen over alternatives like UnsortedSstFileWriter because this utility targets a broad audience: anyone wanting to use RocksDB as an external sort engine, not just users migrating from SstFileWriter. That said, this explicitly removes the sorted-input requirement that SstFileWriter enforces -- callers no longer need to pre-sort their data before writing SST files. KEY DESIGN DECISIONS: - Zero changes to core RocksDB internals. This is a pure utility layer that exclusively calls existing public APIs: * DB::Open(), DB::Put(), DB::Write(), DB::Flush(), DB::CompactRange(), DB::GetColumnFamilyMetaData(), DB::NewIterator(), DestroyDB() * VectorRepFactory (sort-on-flush memtable) * BottommostLevelCompaction::kForceOptimized (zero seqnos) - No modifications to: compaction logic, memtable implementation, SST file format, ingestion logic, or DB open/close paths - All new code lives in utilities/sorted_run_builder/ and the public header include/rocksdb/utilities/sorted_run_builder.h - Build file changes are limited to registering the new source/test files API SURFACE: SortedRunBuilderOptions opts; opts.temp_dir = "/tmp/sort_work"; std::unique_ptr<SortedRunBuilder> builder; SortedRunBuilder::Create(opts, &builder); builder->Add(key, value); // any order, thread-safe builder->AddBatch(&batch); // WriteBatch for throughput builder->Finish(); // flush + compact + collect builder->GetOutputFiles(); // sorted SSTs for ingestion builder->NewIterator(ro); // iterate sorted output FILES CHANGED: New: include/rocksdb/utilities/sorted_run_builder.h (public header) New: utilities/sorted_run_builder/sorted_run_builder.cc (implementation) New: utilities/sorted_run_builder/sorted_run_builder_test.cc (12 tests) New: docs/plans/sorted_run_builder_plan.md (design plan) New: docs/plans/sorted_run_builder_usage_guide.md (usage guide) Modified: src.mk, CMakeLists.txt, Makefile (register new files only) Pull Request resolved: https://github.com/facebook/rocksdb/pull/14499 Test Plan: $ make clean && make -j$(nproc) sorted_run_builder_test $ ./sorted_run_builder_test [==========] Running 12 tests from 1 test case. [ PASSED ] 12 tests. (688 ms total) Tests cover: basic sort correctness, empty builder, WriteBatch path, concurrent multi-threaded writes, ingestion into a target DB, large random dataset (10K keys), entry/size counters, error cases (Add after Finish, iterator before Finish, empty temp_dir), cleanup verification, and duplicate key handling. Reviewed By: xingbowang Differential Revision: D98935972 Pulled By: dannyhchen fbshipit-source-id: e3bb7f1ea5f6004aab697e4da667fa21292ca250 |
||
|
Xingbo Wang
|
ec31d0074f |
Accelerate build and test infrastructure (#14478)
Summary:
Reduce `make check` time from ~9.4 minutes to ~3.7 minutes (2.6x) on a 192-core machine, speed up CI by 1.8x across all jobs, and fix ccache hit rates (macOS cmake: 0.51% → 99.80%).
## Changes
### 1. Auto-detect lld linker (`build_detect_platform`)
- Probe for `lld` at configure time; fall back to default `ld.bfd` if unavailable
- Linux-only guard (`TARGET_OS = Linux`)
- **Measured: 12x faster linking** (7.4s → 0.6s per test binary)
- Add `-L/usr/local/lib` for lld library resolution on CI
- Install lld in CI pre-steps action
- Opt out with `ROCKSDB_NO_FAST_LINKER=1`
### 2. Sharded test execution (`Makefile`)
**Before:** `make check` enumerated every individual gtest case via `--gtest_list_tests`, then created a separate shell script for each case with `--gtest_filter=TestName`. For example, `block_based_table_reader_test` has 9,788 parameterized test cases — each spawned its own process, loading a ~50MB binary, initializing gtest, registering all ~10K tests, running exactly ONE, then tearing down. The per-process overhead was ~1.5s, so 9,788 × 1.5s ≈ 15,000s wasted on overhead alone. Across all 302 test binaries this produced 39,467 individual processes.
**After:** Use gtest's built-in sharding (`GTEST_TOTAL_SHARDS`/`GTEST_SHARD_INDEX`) to group ~10 test cases per process. Each shard loads the binary once and runs multiple tests sequentially — eliminating the per-process overhead. The 9,788 cases in `block_based_table_reader_test` become ~980 shards. Total process count drops from 39,467 to ~4,036. Same tests, same coverage, just fewer process spawns.
The shard count adapts to machine size: `min(ceil(test_count / GTEST_SHARD_SIZE), NCORES * 8)`. This ensures large machines (192 cores) get many small shards for parallelism, while small CI runners (4 cores) get fewer larger shards to avoid excessive overhead. Both `GTEST_SHARD_SIZE` (default 10) and `NCORES` can be overridden.
CI sharding uses round-robin distribution across 3 shards to balance heavy tests (db_test, db_compaction_test, etc.) instead of contiguous alphabetical ranges.
Note: gtest continues running all tests after a failure (`GTEST_THROW_ON_FAILURE=0`), so grouping tests does not mask failures — all failures within a shard are reported.
**Measured: 3.6x CPU reduction** (76,121s → 21,033s)
### 3. Fix SyncPoint leaks for test isolation (5 files)
Sharded execution exposed 43 test files that set SyncPoint callbacks but never clean them up. Stale callbacks with captured local variables cause segfaults or data corruption when subsequent tests in the same process trigger them.
**Systematic fix:** Global gtest `TestEventListener` in `testharness.cc` that calls `SyncPoint::DisableProcessing()` + `ClearAllCallBacks()` + `ClearTrace()` + `LoadDependency({})` after every test case. This cleans up all SyncPoint state: callbacks, dependency maps, cleared points, and the point filter. Registered via static initialization — no changes needed to individual test `main()` functions.
**SyncPoint infrastructure fixes:**
- `DisableProcessing()` now calls `cv_.notify_all()` to wake threads blocked in `Process()`. Previously, threads waiting for predecessor sync points that would never fire (because the test ended) would hang forever.
- `Process()` now rechecks `enabled_` after waking from `cv_.wait()`, so threads exit promptly when processing is disabled instead of looping forever.
**Specific fixes (defense-in-depth):**
- `SstFileReaderTest::VerifyNumEntriesCorruption` — leaked `PropertyBlockBuilder::AddTableProperty:Start` callback that corrupted SST entry counts
- `WritePreparedTransactionTest::CommitAndSnapshotDuringCompaction` — leaked `CompactionIterator:AfterInit` callback causing segfault via dangling pointers
- `TransactionTestBase` destructor — cleanup for 26 SyncPoint uses across transaction tests
- `RetriableLogTest` destructor — cleanup for log reader SyncPoint callbacks
### 4. Fix ccache for reliable cross-run caching (`setup-ccache`, `CMakeLists.txt`)
- **`CCACHE_COMPILERCHECK=content`**: Default `mtime` compared compiler binary modification time, which differs on every fresh CI runner → 0% hit rate. `content` hashes compiler output instead — stable across runner instances
- **`CCACHE_SLOPPINESS`**: CMake generates varying `-MF` paths and timestamps. Without sloppiness settings, ccache treated these as different compilations
- **`CMAKE_C/CXX_COMPILER_LAUNCHER`** instead of `RULE_LAUNCH_COMPILE`: Avoids double-wrapping ccache when also injected via PATH. Removes `RULE_LAUNCH_LINK` since ccache cannot cache link operations
- **macOS cmake hit rate: 0.51% → 99.80%**
### 5. Align GTEST_THROW_ON_FAILURE (`Makefile`)
- Set `GTEST_THROW_ON_FAILURE=0` in Makefile default to match CI's `pre-steps/action.yml`
- `=1` caused `std::terminate` in multi-threaded stress tests (e.g., `point_lock_manager_stress_test`, `rate_limiter_test`) when assertions fail — the gtest exception propagates across thread boundaries, triggering undefined behavior and heap-use-after-free under ASAN
### 6. Portability fixes
- `[[maybe_unused]]` instead of `__attribute__((unused))` for MSVC compatibility
- Remove blanket `-fPIC` from `COMMON_FLAGS` — only apply via `PLATFORM_SHARED_CFLAGS` for shared builds, preserving optimal codegen for release/static builds
- `make -s list_all_tests` to suppress make noise in test enumeration
## Results
### Local (192-core devvm, clean build)
| Metric | Before | After | Improvement |
|--------|--------|-------|-------------|
| Build time (clean) | 2m43s | ~2m | lld linker |
| Test wall clock | 400s | 200s | 2.0x |
| Test CPU total | 76,121s | 21,033s | 3.6x |
| Test failures | 0 | 0 | — |
| **Total make check** | **~9.4min** | **3m41s** | **2.6x** |
### CI (GitHub Actions)
| Metric | Before | After | Improvement |
|--------|--------|-------|-------------|
| Aggregate CI time | 15,220s | 8,425s | 1.8x |
| build-linux-arm | 763s | 221s | 3.5x |
| build-linux-release | 616s | 244s | 2.5x |
| build-windows-vs2022 | 2,868s | 1,205s | 2.4x |
| build-linux-java-static | 858s | 335s | 2.6x |
### ccache hit rates (all jobs)
| Job | Before | After |
|-----|--------|-------|
| macOS cmake (0-3) | 0.51% | **99.80%** |
| build-linux | 99.68% | **99.84%** |
| build-linux-clang18-asan-ubsan | 98.58% | **99.84%** |
| build-linux-clang18-mini-tsan | 98.74% | **99.84%** |
| build-windows-vs2022 | 99.22% | **99.83%** |
| All 27 ccache jobs | — | **>94%** |
## Remaining bottleneck
The critical path is now dominated by genuinely slow integration tests:
- `external_sst_file_test` shards: 97–113s each
- `db_test` shards: 109–124s each
- `db_bloom_filter_test`: 103s (non-parallel)
Further improvement would require test optimization or a `make check-fast` target.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/14478
Test Plan:
- `make check -j192 SKIP_FORMAT_BUCK_CHECKS=1` — all 4,045 shards pass (ASAN+UBSAN), zero failures
- Verified SyncPoint fixes: each crash/corruption reproduces before fix, passes after
- Verified SyncPoint::DisableProcessing wakes blocked threads (previously caused indefinite hangs)
- Verified gtest does NOT stop after failure with `GTEST_THROW_ON_FAILURE=0` — all tests in a shard run and all failures are reported
- CI: all 34 checks pass across two consecutive runs
- ccache: validated hit rates >94% on second run after cache seeding
Reviewed By: hx235
Differential Revision: D97623305
Pulled By: xingbowang
fbshipit-source-id: b299e6d43c8713a9ef2b5659e8bbd74958afe155
|
||
|
Xingbo Wang
|
bcaf2794dc |
Fix TSAN data race in InjectedErrorLog by suppressing benign races (#14467)
Summary:
InjectedErrorLog is a lock-free circular ring buffer designed to be safe to call from signal handlers (which cannot use locks). It has an intentional benign data race between Record() (called by worker threads) and PrintAll() (called by the main thread from a signal/termination handler). The code documents this trade-off in comments, but was missing TSAN suppression annotations.
Simply adding TSAN_SUPPRESSION (__attribute__((no_sanitize("thread")))) is insufficient because TSAN still intercepts libc functions like vsnprintf/snprintf -- accesses through these interceptors are still tracked even when the calling function is annotated.
The fix:
1. Add TSAN_SUPPRESSION to both Record() and PrintAll() to suppress direct field reads/writes in the function body.
2. Restructure both functions to use local stack buffers for vsnprintf/snprintf operations instead of operating directly on shared entry data. This avoids passing shared memory through TSAN-intercepted libc functions.
Also adds fault_injection_fs_test with a ConcurrentRecordAndPrintAll test that exercises the concurrent Record() + PrintAll() pattern and verifies no TSAN race is reported.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/14467
Test Plan:
- fault_injection_fs_test passes under TSAN (buck2 test fbcode//mode/dbg-tsan)
- Reverted fix, re-ran: Fatal (6 TSAN warnings) -- round-trip confirmed
- fault_injection_fs_test passes under debug mode (no regression)
Reviewed By: mszeszko-meta
Differential Revision: D96948483
Pulled By: xingbowang
fbshipit-source-id: efdd5eafa12a5a82f973e40aa327901cc5f95033
|
||
|
zaidoon
|
ec22903914 |
Support all operation types in User Defined Index (UDI) interface (#14399)
Summary: Remove the restriction that limited UDI to ingest-only, Puts-only use cases. This enables UDI plugins (including the trie index from https://github.com/facebook/rocksdb/issues/14310) to work with all operation types: Put, Delete, Merge, SingleDelete, PutEntity, etc. Part of https://github.com/facebook/rocksdb/issues/12396 Pull Request resolved: https://github.com/facebook/rocksdb/pull/14399 Reviewed By: pdillinger Differential Revision: D96392636 Pulled By: xingbowang fbshipit-source-id: 0f1e6c38531fa72539a0e2c6a3dffff333392b4c |
||
|
Josh Kang
|
0c6f741422 |
Rename write_prepared_transaction_test_seqno to write_prepared_transaction_seqno_test (#14453)
Summary: write_prepared_transaction_test_seqno keeps showing up in git changes test binaries need to end with _test so they are ignored by .gitignore Pull Request resolved: https://github.com/facebook/rocksdb/pull/14453 Reviewed By: xingbowang Differential Revision: D96202671 Pulled By: joshkang97 fbshipit-source-id: 91642226bedde37ca72c6af03f300990dca8ff3a |
||
|
Xingbo Wang
|
3b66de3fac |
Fix out of disk unit test issue (#14425)
Summary:
It contains 3 commits.
Commit 1: Fix /dev/shm exhaustion during make check
Fix /dev/shm exhaustion during make check
1. Add space-heavy tests to NON_PARALLEL_TEST: perf_context_test (1GB
write_buffer_size), obsolete_files_test (~1GB), backup_engine_test
(1GB), prefetch_test (1GB), and db_io_failure_test (256MB).
2. Add per-test cleanup on success: each parallel test script now
removes its test directory after passing. Failed test directories
are preserved for debugging.
3. Add age-based stale directory cleanup: at the start of make check,
remove /dev/shm/rocksdb.* directories older than 3 hours from
previous failed runs, using age-based filtering to avoid disturbing
concurrent runs.
Commit 2: Fix SIGSEGV in prefetch_test due to stale SyncPoint callbacks
Both FilePrefetchBufferTest and FSBufferPrefetchTest fixtures set up
SyncPoint callbacks capturing local variables by reference and enable
processing, but neither fixture's TearDown() clears them. When a
subsequent test runs, the stale callbacks fire with dangling
references, causing memory corruption and SIGSEGV.
Fixed by adding DisableProcessing() and ClearAllCallBacks() to both
fixtures' TearDown() methods.
Commit 3: Fix OpenFilesAsyncTest using excessive disk space
Fix OpenFilesAsyncTest using excessive disk space in /dev/shm
OpenFilesAsyncTest::SetupData() set write_buffer_size to SIZE_MAX to
prevent automatic flushes. This caused each of its ~20 parallel
instances to consume 6-43 GB in /dev/shm (validated: a single
Shutdown/3 instance used 43 GB), totaling ~233 GB and filling the
disk. This was the primary cause of "No space left on device" errors
in make check. The sst flushed is small, but it causes issue in WAL
space reservation, which bloated the disk usage.
The SIZE_MAX is unnecessary — the test writes only 4 tiny key-value
pairs and does explicit Flush() after each. The default 64 MB
write_buffer_size will never auto-flush for such small writes.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/14425
Test Plan: Unit Test
Reviewed By: joshkang97
Differential Revision: D95253518
Pulled By: xingbowang
fbshipit-source-id: 03907df59b3d89d90413a0e33996ec205944d48b
|
||
|
zaidoon
|
ce0fff9f41 |
Add trie-based User Defined Index (UDI) plugin (#14310)
Summary: Implement a Fast Succinct Trie (FST) index based on LOUDS encoding as a User Defined Index plugin for RocksDB's block-based tables. First step toward trie-based indexing per https://github.com/facebook/rocksdb/issues/12396. The trie uses hybrid LOUDS-Dense (upper levels, 256-bit bitmaps) + LOUDS-Sparse (lower levels, label arrays) encoding inspired by the [SuRF paper](https://www.pdl.cmu.edu/PDL-FTP/Storage/surf_sigmod18.pdf) (Zhang et al., SIGMOD 2018). The boundary between dense and sparse levels is automatically chosen to minimize total space. Key Components - **Bitvector** with O(1) rank and O(log n) select using a rank LUT sampled every 256 bits with popcount intrinsics. Uses uint32_t rank LUT entries (halving memory vs uint64_t). Includes word-level `AppendWord()` for efficient dense bitmap construction and `AppendMultiple()` optimized at word granularity for bulk bit fills. - **Streaming trie builder** using flat per-level arrays with deferred internal marking, handle migration for prefix keys, and lazy node creation. Infers trie structure directly from sorted keys via LCP analysis in a single pass (no intermediate tree). - **LoudsTrie** for immutable querying with BFS-ordered handle reordering built into single-pass level-by-level serialization. Move-only semantics with correct pointer re-seating after `std::string` move. - **LoudsTrieIterator** with rank-based traversal, key reconstruction from trie path, and stack-based backtracking for `Next()`. Uses packed 8-byte `LevelPos` (is_dense flag in bit 63) and `autovector<LevelPos, 24>` to avoid heap allocation. Key reconstruction uses a raw char buffer allocated once to `MaxDepth()+1` bytes. - **TrieIndexFactory/Builder/Reader/Iterator** implementing the `UserDefinedIndexFactory` interface. - **Zero-copy block handle loading** using two fixed-width uint64_t arrays (offsets + sizes) with 8-byte alignment, enabling O(1) initialization via direct pointer assignment. Seek Hot Path Optimizations - **Fanout-1 sparse fast path**: Most sparse nodes in tries built from zero-padded numeric keys have exactly one child. Detected via `start_pos + 1 == end_pos` and inlined as a single byte comparison, avoiding the full `SparseSeekLabel` call. - **Linear scan for small sparse nodes**: `SparseSeekLabel` uses sequential scan for nodes with ≤16 labels instead of binary search. Faster for common 10-child digit nodes where branch misprediction cost outweighs linear scan cost. - **Rank reuse**: `DenseLeafIndexFromRankAndHasChildRank` and `SparseLeafIndexFromHasChildRank` overloads accept pre-computed `has_child_rank` from the Seek descent, avoiding redundant `Rank1` calls. General Performance Optimizations - **Select-free sparse traversal**: Precomputed child position lookup tables (`s_child_start_pos_`/`s_child_end_pos_`) eliminate `Select1` calls during Seek. Sparse traversal tracks `(start_pos, end_pos)` directly, using only `Rank1` (O(1)) + array lookup (O(1)) for child descent. - **Cached label_rank pattern**: Eliminates redundant `Rank1` calls in hot paths (Seek, Next, Advance all cache and reuse the label_rank computed for has_child checking). - **Leaf index fast path**: When no prefix keys exist (common case), `SparseLeafIndex` and `DenseLeafIndexFromRank` skip the prefix-key `Rank1` calls entirely, reducing from 3 to 1 `Rank1` call. - **Popcount-based Select64** via 6-step binary search within 64-bit words. - MSVC portability using RocksDB's `BitsSetToOne`/`CountTrailingZeroBits`. Benchmark Results Trie Seek at 32K keys (16-byte keys, 5M lookups, median of 5 runs): | Configuration | ns/op | |---|---:| | Trie (optimized) | **118** | | Binary search (native) | 134 | Trie Seek is **~12% faster** than native binary search index at 32K keys per block. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14310 Reviewed By: anand1976 Differential Revision: D93921511 Pulled By: xingbowang fbshipit-source-id: ba18604bc6bd4f575311a1ae3047a541274869b6 |
||
|
xingbowang
|
aa7571c3ad |
Run clang-tidy in github CI (#14347)
Summary: RocksDB has been using clang-tidy for a long time inside Meta. However, it is not efficient for external contributor, as the result from clang-tidy has to be ferried back through internal contributor. This PR added support to run clang-tidy on external github CI. It added .clang-tidy file based on internal version. It run clang-tidy in a separate pr job and a workflow step would post the pr job result to the PR itself. See example below. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14347 Test Plan: Github CI Reviewed By: archang19 Differential Revision: D93862467 Pulled By: xingbowang fbshipit-source-id: bb4330241036894deb619470efd73a7041a8b62f |
||
anand76
|
653fd9c65b |
Bug fix for bg error recovery in TransactionDB (#14313)
Summary: This PR fixes a bug in the interaction between WritePrepared/WriteUnprepared TransactionDB (with two_write_queues=true) and background error recovery. This bug caused crash tests to fail with a "sequence number going backwards" error during DB open. Root Cause ------------ When two_write_queues=true, sequence numbers are allocated via FetchAddLastAllocatedSequence() before a write completes, but are only published via SetLastSequence() after the write succeeds. If a background error occurs (e.g., a MANIFEST write failure during flush), the error recovery path in DBImpl::ResumeImpl creates new memtables and WAL files. The new WAL's starting sequence number is based on LastSequence() (the published value), which can be lower than already-allocated sequence numbers that were written to the old WAL. On subsequent recovery, RocksDB detects that sequence numbers in the new WAL are lower than those in the old WAL and reports a "sequence number going backwards" corruption error, causing the DB to fail to open. Fix --- The fix adds a call to a new VersionSet::SyncLastSequenceWithAllocated() method at the beginning of DBImpl::ResumeImpl, before any new memtables or WALs are created. This method advances last_sequence_ to match last_allocated_sequence_ if the latter is higher, ensuring the new WAL starts with a sequence number that is at least as high as any previously allocated one. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14313 Test Plan: --------- Add new unit tests in write_prepared_transaction_test_seqno Reviewed By: pdillinger Differential Revision: D92746944 Pulled By: anand1976 fbshipit-source-id: 34385fc13fd74435dd1c3283637eb118f45d887e |
||
|
Xingbo Wang
|
92378eb3b8 |
Add CLAUDE.md and optimize tooling for claude code (#14293)
Summary: * Add CLAUDE.md This CLAUDE.md is generated through the analysis of 9,012 commits from the RocksDB repository since 2016. It aggregate the commits into 8 major components based on the component it is changing. It selected top 100 most complex PR based on line of changes to collect the code review feedbacks. For each PR, we collected: • PR title and description to understand the change context • Files changed to identify affected components and measure complexity • Inline code review comments from reviewers • General review summaries and approval/change request feedback The feedback was then categorized by RocksDB component and analyzed for recurring themes, patterns, and best practices. This CLAUDE.md file could be used for guiding code generation and code review. * Optimize tooling for claude code Add make check-progress and format-auto targets for automation Add machine-parseable progress reporting for `make check` to support automated monitoring tools like Claude Code: - Add `build_tools/check_progress.sh` script that outputs JSON progress - Add `make check-progress` target to poll build/test progress - Detects phases: compiling -> linking -> generating -> testing - Reports failed tests with exit codes, signals, and log output - Limits to 10 failures with last 50 lines of output each Also add non-interactive formatting support: - Add `-y` flag to format-diff.sh for auto-apply without prompts - Add `make format-auto` target for CI/automation use Pull Request resolved: https://github.com/facebook/rocksdb/pull/14293 Test Plan: Local build Reviewed By: pdillinger Differential Revision: D92085763 Pulled By: xingbowang fbshipit-source-id: ba122a4ff51087aec5c06bab804edfee34e13880 |
||
|
Xingbo Wang
|
656b734a5f |
Support abort background compaction jobs. (#14227)
Summary: This adds a new public API to allow applications to abort all running compactions and prevent new ones from starting. Unlike DisableManualCompaction() which only pauses manual compactions and waits for them to finish naturally, AbortAllCompactions() actively signals running compactions (both automatic and manual) to terminate early and waits for them to complete before returning. The abort signal is checked periodically during compaction (every 100 keys), so ongoing compactions abort quickly. Any output files from aborted compactions are automatically cleaned up to prevent partial results from being installed. This is useful for scenarios where applications need to quickly stop all compaction activity, such as during graceful shutdown or when performing maintenance operations. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14227 Test Plan: - Unit tests in db_compaction_abort_test.cc cover various abort scenarios including: abort before/during compaction, abort with multiple subcompactions, nested abort/resume calls, abort with CompactFiles API, abort across multiple column families, and timing guarantees - Updated compaction_job_test.cc to include the new parameter Reviewed By: anand1976 Differential Revision: D91480994 Pulled By: xingbowang fbshipit-source-id: 36837971d8a540cd34d3ec28a78bc94b582625b0 |
||
Ryan Hancock
|
2b28885c80 |
Introducing IO Dispatcher (#14135)
Summary: This diff introduces the IO Dispatcher, which will be used to simplify the code path for MultiScan, while further providing a centralized place to enact policy on how MultiScan is done (i.e., limit memory usage and pinned buffers for example). Right now this diff only encapsulates the functionality done during the Prepare of MultiScan. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14135 Reviewed By: anand1976 Differential Revision: D87837261 Pulled By: krhancoc fbshipit-source-id: 2698910ade02bc3d182413ae07ce69fe7abb7ec5 |
||
|
Peter Dillinger
|
7e9f54d56b |
Remove remaining pieces of Lua integration (#14200)
Summary: Going from deprecated and partly removed to fully removed Pull Request resolved: https://github.com/facebook/rocksdb/pull/14200 Test Plan: existing tests Reviewed By: archang19 Differential Revision: D89697543 Pulled By: pdillinger fbshipit-source-id: bbf6161c04322e9756a9479758488cac6d03473a |
||
Xingbo Wang
|
9e14d06143 |
Support ccache in make file (#14123)
Summary: Support ccache in make file Pull Request resolved: https://github.com/facebook/rocksdb/pull/14123 Test Plan: local build Reviewed By: cbi42 Differential Revision: D87332892 Pulled By: xingbowang fbshipit-source-id: 2088bd19bdab1bd7070734c886200be80f1a65af |
||
|
Peter Dillinger
|
37176a4a44 |
Auto-tune manifest file size (#14076)
Summary:
Adds auto-tuning of manifest file size to avoid the need to scale `max_manifest_file_size` in proportion to things like number of SST files to properly balance (a) manifest file write amp and new file creation, vs. (b) manifest file space amp and replay time, including non-incremental space usage in backups. (Manifest file write amp comes from re-writing a "live" record when the manifest file is re-created, or "compacted"; space amp is usage beyond what would be used by a compacted manifest file.) In more detail,
* Add new option `max_manifest_space_amp_pct` with default value of 500, which defaults to 0.2 write amp and up to roughly 5.0 space amp, except `max_manifest_file_size` is treated as the "minimum" size before re-creating ("compacting") the manifest file.
* `max_manifest_file_size` in a way means the same thing, with the same default of 1GB, but in a way has taken on a new role. What is the same is that we do not re-create the manifest file before reaching this size (except for DB re-open), and so users are very unlikely to see a change in default behavior (auto-tuning only kicking in if auto-tuning would exceed 1GB for effective max size for the current manifest file). The new role is as a file size lower bound before auto-tuning kicks in, to minimize churn in files considered "negligibly small." We recommend a new setting of around 1MB or even smaller like 64KB, and expect something like this to become the default soon.
* These two options along with `manifest_preallocation_size` are now mutable with SetDBOptions. The effect is nearly immediate, affecting the next write to the current manifest file.
Also in this PR:
* Refactoring of VersionSet to allow it to get (more) settings from MutableDBOptions. This touches a number of files in not very interesting ways, but notably we have to be careful about thread-safe access to MutableDBOptions fields, and even fields within VersionSet. I have decided to save copies of relevant fields from MutableDBOptions to simplify testing, etc. by not saving a reference to MutableDBOptions but getting notified of updates.
* Updated some logging in VersionSet to provide some basic data about final and compacted manifest sizes (effects of auto-tuning), making sure to avoid I/O while holding DB mutex.
* Added db_etc3_test.cc which is intended as a successor to db_test and db_test2, but having "test.cc" in its name for easier exclusion of test files when using `git grep`. Intended follow-up: rename db_test2 to db_etc2_test
* Moved+updated `ManifestRollOver` test to the new file to be closer to other manifest file rollover testing.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/14076
Test Plan:
As for correctness, new unit test AutoTuneManifestSize is pretty thorough. Some other unit tests updated appropriately. Manual tests in the performance section were also audited for expected behavior based on the new logging in the DB LOG. Example LOG data with -max_manifest_file_size=2048 -max_manifest_space_amp_pct=500:
```
2025/10/24-11:12:48.979472 2150678 [/version_set.cc:5927] Created manifest 5, compacted+appended from 52 to 116
2025/10/24-11:12:49.626441 2150682 [/version_set.cc:5927] Created manifest 24, compacted+appended from 2169 to 1801
2025/10/24-11:12:52.194592 2150682 [/version_set.cc:5927] Created manifest 91, compacted+appended from 10913 to 8707
2025/10/24-11:13:02.969944 2150682 [/version_set.cc:5927] Created manifest 362, compacted+appended from 52259 to 13321
2025/10/24-11:13:18.815120 2150681 [/version_set.cc:5927] Created manifest 765, compacted+appended from 80064 to 13304
2025/10/24-11:13:35.590905 2150681 [/version_set.cc:5927] Created manifest 1167, compacted+appended from 79863 to 13304
```
As you can see, it only took a few iterations of ramp-up to settle on the auto-tuned max manifest size for tracking ~122 live SST files, around 80KB and compacting down to about 13KB. (13KB * (500 + 100) / 100 = 78KB). With the default large setting for max_manifest_file_size, we end up with a 232KB manifest, which is more than 90% wasted space. (A long-running DB would be much worse.)
As for performance, we don't expect a difference, even with TransactionDB because actual writing of the manifest is done without holding the DB mutex. I was not able to see a performance regression using db_bench with FIFO compaction and >1000 ~10MB SST files, including settings of -max_manifest_file_size=2048 -max_manifest_space_amp_pct={500,10,0}. No "hiccups" visible with -histogram either.
I also tried seeding a 1 second delay in writing new manifest files (other than the first). This had no significant effect at -max_manifest_space_amp_pct=500 but at 100 started causing write stalls in my test. In many ways this is kind of a worst case scenario and out-of-proportion test, but gives me more confidence that a higher number like 500 is probably the best balance in general.
Reviewed By: xingbowang
Differential Revision: D85445178
Pulled By: pdillinger
fbshipit-source-id: 1e6e07e89c586762dd65c65bb7cb2b8b719513f9
|
||
ngina
|
7603712a88 |
Introduce tail estimation to prevent oversized compaction files (#14051)
Summary: **Summary:** This change introduces tail size estimation during SST construction to improve compaction file cutting accuracy to prevent oversized files. The BlockBasedTableBuilder now estimates the SST tail size (index and filter blocks) and uses this estimate, in addition to the data size, to determine when to cut files during compaction. **Problem:** Currently, file cutting logic only considers data size when determining where to cut a file, failing to reserve space for index and filter blocks that are added when the file is finalized. This often leads to SST files that exceed target file size limits. **Behavior Change:** Implement size estimation methods for index and filter builders, and integrate these estimates into BlockBasedTableBuilder via a new EstimatedTailSize() method. This method aggregates estimates from all tail components and is used for file cutting decisions during compaction. **Performance Considerations:** To minimize CPU overhead, size estimates are updated when data blocks are finalized rather than on every key add. For index builders, estimates are updated when index entries are added (one per data block). For filter builders, the OnDataBlockFinalized() hook triggers estimate updates when data blocks are cut/finalized. This approach provides: * Minimal impact to compaction hot path (key additions) * Near real-time estimates for file cutting decisions * Meaningful estimate changes only when data blocks are finalized **Usage:** * Set true mutable cf option `compaction_use_tail_size_estimation` to use tail size estimation for compaction file cutting decisions. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14051 Test Plan: * Assert tail size estimate is an overestimate in BlockBasedTableBuilder::Finish * Add new test to verify compaction output file is below target file size **Next steps:** * Enable tail size estimation for compaction file cutting by default (and other improvements) Reviewed By: pdillinger, cbi42 Differential Revision: D84852285 Pulled By: nmk70 fbshipit-source-id: c43cf5dbd2cb2f623a0622591ef24eee30ce0c87 |
||
|
Peter Dillinger
|
d56da8c112 |
More folly build updates (#14099)
Summary: * Fix nightly build-linux-cmake-with-folly-lite-no-test for real this time with correct include directory. (CMakeLists.txt) * Add test runs to that build (and rename) * Improve folly build caching with a folly.mk file with most of the relevant parts of Makefile that contribute to the checkout_folly and build_folly builds. This reduces the risk of false passing of CI job with cache folly build. This caching is still only for folly debug builds, (which is probably OK with just a single nightly build relying on release folly build, which also serves as a rough canary against false passing because of caching). * Use `make VERBOSE=1` after cmake calls for detailed output Pull Request resolved: https://github.com/facebook/rocksdb/pull/14099 Test Plan: temporary CI change to put the relevant parts in pr-jobs, then back to homes including in nightly Reviewed By: mszeszko-meta Differential Revision: D86243363 Pulled By: pdillinger fbshipit-source-id: f7975fa190ef45195c6d0b74417f7886e551516a |
||
|
Peter Dillinger
|
befa6b8050 |
Fix and check for potential ODR violations (#14096)
Summary: ... caused by public headers depending on build parameters (macro definitions). This change also adds a check under 'make check-headers' (already in CI) looking for potential future violations. I've audited the uses of '#if' in public headers and either * Eliminated them * Systematically excluded them because they are intentional or similar (details in comments in check-public-header.sh * Manually excluded them as being ODR-SAFE In the case of ROCKSDB_USING_THREAD_STATUS, there was no good reason for this to appear in public headers so I've replaced it with a static bool ThreadStatus::kEnabled. I considered getting rid of the ability to disable this code but some relatively recent PRs have been submitted for fixing that case. I've added a release note and updated one of the CI jobs to use this build configuration. (I didn't want to combine with some jobs like no_compression and status_checked because the interaction might limit what is checked. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14096 Test Plan: manual 'make check-headers' + manual cmake as in new CI config + CI Reviewed By: jaykorean Differential Revision: D86241864 Pulled By: pdillinger fbshipit-source-id: d16addc9e3480706b174a006720a4def0740bf2e |
||
|
Peter Dillinger
|
9577b92b55 |
Fix ODR violation from open source folly build, update (#14094)
Summary:
Following up on https://github.com/facebook/rocksdb/pull/14071, updating folly to
|
||
|
Peter Dillinger
|
94d91daddb |
Update folly (part way), fix USE_FOLLY_LITE (#14071)
Summary: Resolving this folly upgrade required fixing the FOLLY_LITE build with header include from the 'fmt' library. I was close to timing out on fixing USE_FOLLY_LITE and removing it altogether - it could be considered obsolete and/or not worth the maintenance cost. Follow-up: make the folly build caching more friendly by hashing the relevant makefile parts. Not in this PR because then you wouldn't be able to see what changed in the folly build steps themselves. UPDATE/NOTE: I wasn't able to fully update to latest due to a failure seen in F14, using the next folly commit or later. The source of the bug is likely outside of F14 but investigation is in progress. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14071 Test Plan: CI Reviewed By: jaykorean Differential Revision: D85268833 Pulled By: pdillinger fbshipit-source-id: 1d0a2d61f095524a20e6ec796ef46c02d0696f4e |
||
|
Xingbo Wang
|
a1dad12c8c |
Reduce github CI build time (#14057)
Summary: * Reduce build time of folly from 45m~1hr down to 25m. This is achieved by caching folly build artifact from previous build. * Reduce windows build time of folly from 1hr 15m down to 50m. This is done by increase windows build machine size. * Fix build on macos on other macos target. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14057 Test Plan: github CI Reviewed By: archang19, nmk70 Differential Revision: D84848041 Pulled By: xingbowang fbshipit-source-id: 00306750737070e7e446ee436d607ed6ecae79ae |
||
|
ngina
|
13172e2be3 |
Add method to estimate index size (#14010)
Summary: This method will be used to improve the compaction logic by accounting for the tail size, in addition to the data size, when determining when to cut a file. Problem: Currently the file cutting logic only considers data size when determining where to cut a file, failing to reserve space for index and filter blocks that are added when the file is finalized. Key changes: - Add EstimateCurrentIndexSize() to IndexBuilder interface - Implement in ShortenedIndexBuilder with buffer that accounts for the next index entry. The buffer addresses under-estimation where the current index size doesn't account for the next index entry associated with the data block currently being built. The 2x multiplier bounds the estimate in the right direction and handles outlier cases with large keys. - Add num_index_entries_ member to track added index entries (== data blocks emitted). This is thread-safe since it's updated/read in the serialized emit step. Next steps: - Partitioned index size estimation implementation - Update compaction file cutting logic to consider index size estimation Pull Request resolved: https://github.com/facebook/rocksdb/pull/14010 Test Plan: Added a new test class with unit tests for new builder size estimation across all IndexBuilder implementations. Reviewed By: pdillinger Differential Revision: D83501741 Pulled By: nmk70 fbshipit-source-id: d58fc2a9e92e12a162f6244d4abd707a9c9e1885 |
||
|
Xingbo Wang
|
1aca60c089 |
Improve efficiency in PointLockManager by using separate Condvar (#13731)
Summary: PointLockManager manages point lock per key. The old implementation partition the per key lock into 16 stripes. Each stripe handles the point lock for a subset of keys. Each stripe have only one conditional variable. This conditional variable is used by all the transactions that are waiting for its turn to acquire a lock of a key that belongs to this stripe. In production, we notified that when there are multiple transactions trying to write to the same key, all of them will wait on the same conditional variables. When the previous lock holder released the key, all of the transactions are woken up, but only one of them could proceed, and the rest goes back to sleep. This wasted a lot of CPU cycles. In addition, when there are other keys being locked/unlocked on the same lock stripe, the problem becomes even worse. In order to solve this issue, we implemented a new PerKeyPointLockManager that keeps a transaction waiter queue at per key level. When a transaction could not acquire a lock immediately, it joins the waiter queue of the key and waits on a dedicated conditional variable. When previous lock holder released the lock, it wakes up the next set of transactions that are eligible to acquire the lock from the waiting queue. The queue respect FIFO order, except it prioritizes lock upgrade/downgrade operation. However, this waiter queue change increases the deadlock detection cost, because the transaction waiting in the queue also needs to be considered during deadlock detection. To resolve this issue, a new deadlock_timeout_us (microseconds) configuration is introduced in transaction option. Essentially, when a transaction is waiting on a lock, it will join the wait queue and wait for the duration configured by deadlock_timeout_us without perform deadlock detection. If the transaction didn't get the lock after the deadlock_timeout_us timeout is reached, it will then perform deadlock detection and wait until lock_timeout is reached. This optimization takes the heuristic where majority of the transaction would be able to get the lock without perform deadlock detection. The deadlock_timeout_us configuration needs to be tuned for different workload, if the likelihood of deadlock is very low, the deadlock_timeout_us could be configured close to a big higher than the average transaction execution time, so that majority of the transaction would be able to acquire the lock without performing deadlock detection. If the likelihood of deadlock is high, deadlock_timeout_us could be configured with lower value, so that deadlock would get detected faster. The new PerKeyPointLockManager is disabled by default. It can be enabled by TransactionDBOptions.use_per_key_point_lock_mgr. The deadlock_timeout_us is only effective when PerKeyPointLockManager is used. When deadlock_timeout_us is set to 0, transaction will perform deadlock detection immediately before wait. Pull Request resolved: https://github.com/facebook/rocksdb/pull/13731 Test Plan: Unit test. Stress unit test that validates deadlock detection and exclusive, shared lock guarantee. A new point_lock_bench binary is created to help perform performance test. Reviewed By: pdillinger Differential Revision: D77353607 Pulled By: xingbowang fbshipit-source-id: 21cf93354f9a367a78c8666596ed14013ac7240b |
||
|
Peter Dillinger
|
86bb0c0d1b |
Use C++20 in public API, fix CI (#13915)
Summary: A follow-up to https://github.com/facebook/rocksdb/issues/13904 which was incomplete in updating CI jobs to support C++20 because the C++20 usage was only in tests. Here we add subtle C++20 usage in the public API ("using enum" feature in db.h) to force the issue. A lot of the work for this PR was in updating the Ubuntu22 docker image, for earlier compiler/runtime versions supporting C++20, and generating a new Ubuntu24 docker image, for later compiler/runtime versions. The Ubuntu22 image needed to be updated because there are incompatibilities with clang-13 + c++20 + libstdc++ for gcc 11, seen on these examples ``` #include <chrono> int main(int argc, char *argv[]) { std::chrono::microseconds d = {}; return 0; } ``` and ``` #include <coroutine> int main() { return 0; } ``` The second was causing recurring failures in build-linux-clang-13-asan-ubsan-with-folly, now fixed. So we have to install clang's libc++ to compile with clang-13. I haven't been able to get this to work with some of the libraries like benchmark, glog, and/or gflags, but I'm able to compile core RocksDB with clang-13. On this docker image, an extra compiler parameter is needed to compile with gcc and glog because it's built from source perhaps not perfectly, because the ubuntu package transitively conflicts with libc++. The Ubuntu24 image seems to be low-drama and generally work for testing out newer compiler versions. The mingw build uses Ubuntu24 because the mingw package on Ubuntu22 uses a gcc version that is too old. And the mass of other code changes are trying to work around new warnings, mostly from clang-analyze, which I upgraded to clang-18 in CI. Pull Request resolved: https://github.com/facebook/rocksdb/pull/13915 Test Plan: CI, including temporarily including the nightly jobs in the PR jobs in earlier revisions to test and stabilize Reviewed By: archang19 Differential Revision: D81933067 Pulled By: pdillinger fbshipit-source-id: 7e33823006a79d5f3cf5bc1d625f0a3c08a7d74c |
||
|
Peter Dillinger
|
2950e99219 |
Require C++20 (#13904)
Summary: I am wanting to use std::counting_semaphore for something and the timing seems good to require C++20 support. The internets suggest: * GCC >= 10 is adequate, >= 11 preferred * Clang >= 10 is needed * Visual Studio >= 2019 is adquate And popular linux distributions look like this: * CentOS Stream 9 -> GCC 11.2 (CentOS 8 is EOL) * Ubuntu 22.04 LTS -> GCC 11.x (Ubuntu 20 just ended standard support) * Debian 12 (oldstable) -> GCC 12.2 * (Debian 11 has ended security updates, uses GCC 10.2) This required generating a new docker image based on Ubuntu 22 for CI using gcc. The existing Ubuntu 20 image works for covering appropriate clang versions (though we should maybe add a much later version as well, in the next increment of our Ubuntu 22 image; however the minimum available clang build from apt.llvm.org for Ubuntu 22 is clang 13). Update to SetDumpFilter is to quiet a mysterious gcc-13 warning-as-error. Removed --compile-no-warning-as-error from a cmake command line because cmake in the new docker image is too old for this option. Pull Request resolved: https://github.com/facebook/rocksdb/pull/13904 Test Plan: CI, one minor unit test added to verify std::counting_semaphor works Reviewed By: xingbowang Differential Revision: D81266435 Pulled By: pdillinger fbshipit-source-id: 26040eeccca7004416e29a6ff4f6ea93f2052684 |
||
Changyu Bi
|
13f054febb |
Support DbStressCustomCompressionManager in ldb and sst_dump (#13827)
Summary: while debugging stress test failure, I noticed that sst_dump and ldb do not work if custom db_stress compression manager is used. This PR adds support for it. ``` ./sst_dump --command=raw --show_properties --file=/tmp/rocksdb_crashtest_whitebox4ny5mass/000589.sst options.env is 0x7f2b1f4b9000 Process /tmp/rocksdb_crashtest_whitebox4ny5mass/000589.sst Sst file format: block-based /tmp/rocksdb_crashtest_whitebox4ny5mass/000589.sst: Not implemented: Could not load CompressionManager: DbStressCustom1 /tmp/rocksdb_crashtest_whitebox4ny5mass/000589.sst is not a valid SST file ./ldb idump --db=/tmp/rocksdb_crashtest_whiteboxy_emah11 --ignore_unknown_options --hex >> /tmp/i_dump Failed: Not implemented: Could not load CompressionManager: DbStressCustom1 ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/13827 Test Plan: manually tested that ldb and sst_dump work with DbStressCustomCompressionManager after this PR Reviewed By: pdillinger Differential Revision: D79461175 Pulled By: cbi42 fbshipit-source-id: c8c092b10b4fde3a295b00751057749e8f0cf095 |
||
|
Ryan Hancock
|
7c5c37a1a4 |
IntervalSet Data Structure (#13787)
Summary: This diff introduces the IntervalSet data structure, which will be used to help create sets of non overlapping sets of intervals for MultiScan scan options. Specifically, we add specializations for Slices to assist in this. Pull Request resolved: https://github.com/facebook/rocksdb/pull/13787 Test Plan: Added test to catch various cases within adding intervals. Reviewed By: anand1976 Differential Revision: D78624970 Pulled By: krhancoc fbshipit-source-id: 9a3e4a28738ab8428788467540fc05ab5c1a1b67 |