Summary:
Solaris support was contributed by outside users starting with d78c6b28c (2016), and no project CI has ever run on Solaris. The last Solaris-motivated change was c76688745 (https://github.com/facebook/rocksdb/issues/4012, June 2018). Four months later, d6ec28870 (https://github.com/facebook/rocksdb/issues/4226, 5.18) gave PerfContext a user-declared destructor while the Solaris-only definition was still `__thread PerfContext perf_context_`, which GCC rejects, so the Solaris build has not compiled since. fd00f39f9 (https://github.com/facebook/rocksdb/issues/8117, 2021) then renamed that definition to `perf_context` while keeping
`#define perf_context (*get_perf_context())` in perf_context_imp.h, so the definition expands to `thread_local PerfContext (*get_perf_context());`, an error still present today. The Solaris-only paths are dead and untestable, yet still attract new carve-outs.
Because Solaris support is apparently unused and complicating things, this change removes all OS_SOLARIS / SunOS handling from the build (Make, CMake, build_detect_platform, INSTALL.md), the portability layer and Env (endianness, gethrtime, directio, stack trace), the perf_context macro and destructor carve-out, Solaris-only test workarounds and exclusions, and RocksJava's Environment.isSolaris() and Solaris JNI library naming.
Not Solaris-specific and kept: SPARC unaligned-access handling in util/coding.h, sparc64 in Makefile machine lists (Linux/OpenBSD), and sparcv9 in RocksJava Environment.is64Bit(). Third-party sources are untouched.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15449
Test Plan: On non-Solaris platforms every removed condition was either an OS_SOLARIS-only branch or a `!defined(OS_SOLARIS)` term that was always true, so the code compiled is unchanged and the CI matrix covers it. The Solaris breakage was confirmed by syntax-checking monitoring/perf_context.cc with -DOS_SOLARIS (GCC 11, stub sys/isa_defs.h) at each commit touching the perf_context files from c31979205 through fd00f39f9: clean before d6ec28870, failing at every commit from d6ec28870 on.
Reviewed By: archang19
Differential Revision: D126714985
Pulled By: pdillinger
fbshipit-source-id: 40d22fcd5a462375c9d2c93f52407df651286ef7
Summary:
`USE_CLANG=1 make check-c-api-gen` failed with "clang++ not found" because the staleness check and the Python generator detected clang inconsistently: the Makefile inspected the `$(CXX)` make variable (the fbcode clang path) while the generator read the `CXX` environment variable, which make never exports. More broadly, clang++ was detected in several ad hoc places with fragile heuristics (grepping the compiler path, duplicated fallback lists in shell and Python).
Centralize clang++ detection in build_detect_platform as a single `CLANG_CXX`:
- `CLANG_CXX` honors `$CXX` when it is itself clang (so CI's `CXX=clang++-21` is used), else takes the first clang++ on PATH (bare, versioned, or the Meta-internal clang++.par). `CXX` (when building with clang) and `CLANG_ANALYZER` now derive from `CLANG_CXX` instead of re-detecting.
- Infer `USE_CLANG=1` when the compiler is clang, detected via the compiler's own `__clang__` predefined macro rather than matching the binary path (which false-positives, e.g. a home dir containing "clang"), and emit it so clang-specific Makefile logic is consistent.
- Emit `CLANG_CXX` to make_config.mk and pass it explicitly to the codegen, so `make check-c-api-gen` works without requiring `USE_CLANG=1` and the generator never guesses.
Add a first-class `make regen-c-api` target -- now the preferred regenerate command referenced by the generated banners and user-facing messages. The Python generators no longer detect clang themselves; `--clang` defaults to `clang++` for manual runs and is overridden by the Makefile with `CLANG_CXX`. Regeneration verifies the clang runs before writing any file, so a missing/broken toolchain aborts cleanly instead of leaving a partially regenerated tree.
Make field discovery clang-version-independent: older clang (e.g. 15) spells field types fully qualified (`rocksdb::ReadTier`,
`rocksdb::BlockBasedTableOptions::IndexType`) while the fbcode/CI clang (21) uses the bare `ReadTier` / `IndexType` the family tables expect. Strip the ROCKSDB namespace and enclosing-struct qualifiers so both agree; this is a no-op on newer clang, so checked-in output and CI are unaffected.
Also escape the literal "generated" in the generator .py sources (they emit the marker but are not themselves generated) so tooling stops treating them as read-only generated files.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/14994
Test Plan:
Manual verification (not covered by CI):
- `make check-c-api-gen` now runs the staleness sub-check (instead of skipping) and passes in a default gcc build as well as with `USE_CLANG=1` on an fbcode host, since `CLANG_CXX` is always populated.
- `ROCKSDB_NO_FBCODE=1 make regen-c-api` using a system clang 15 (clang++.par) produces output byte-identical to the checked-in files (generated with clang 21), and `check-c-api-gen` reports "up to date" -- confirming the namespace/struct stripping is version-robust and a no-op on clang 21.
- `regen_all.py --clang /nonexistent` exits non-zero with a clear message and leaves the generated files untouched.
- `make regen-c-api` is idempotent (no diff on a clean tree).
- Confirmed the escaped `\x40generated` still emits `// generated` byte-for-byte in the regenerated c.h/c.cc/.inc.
- Reviewed .github/workflows/pr-jobs.yml: the build-linux-clang-21-no_test_run job's `CXX=clang++-21 make check-c-api-gen` resolves `CLANG_CXX` to clang++-21 (honoring `$CXX`) rather than the container's stray bare clang++.
Reviewed By: xingbowang
Differential Revision: D113447462
Pulled By: pdillinger
fbshipit-source-id: 91ca92d21a71a8b92f9ff57500a8cec91a884078
Summary:
Local Make builds silently reuse object files even when build parameters (DEBUG_LEVEL, sanitizers, ASSERT_STATUS_CHECKED, RTTI, etc.) change, since all object files share the same paths. This leads to confusing linker errors, sanitizer false negatives, ODR violations, and "phantom" bugs that are easy to miss -- a pitfall for humans and especially for AI agents driving builds non-interactively.
This change improves the Make experience on three fronts:
1. Build-parameter change detection in the Makefile. After flags are fully resolved, we hash the effective compile/link inputs (CC|CXX|CFLAGS|CXXFLAGS|LDFLAGS|EXEC_LDFLAGS) into a per-OBJ_DIR .build_signature stamp and compare against the previous build:
- Default: stop with a clear error when parameters changed. This is optimized for the expert who might not intend a full rebuild, and for avoiding CI inefficiency.
- AUTO_CLEAN=1: automatically run clean-rocks and proceed.
- ALLOW_BUILD_PARAMETER_CHANGE=1: skip the check (e.g. intentionally mixing DEBUG_LEVEL=1 and DEBUG_LEVEL=2 objects).
The check is skipped for dry runs (-n) and for non-building / self-cleaning targets (clean*, format, check-*, tags, gen-pc, check-progress, watch-log, release, coverage, the asan_*/ubsan_* variants, etc.). make_config.mk is removed only by the top-level `clean` target, not by clean-rocks, so the auto-clean does not delete the make_config.mk already included by the running make. The signature is removed last in clean-rocks so an interrupted clean keeps change detection meaningful.
3. New build_tools/rockstest.sh and build_tools/rocksptest.sh helpers that build and run unit tests in one step. They set AUTO_CLEAN=1 and build with -j<NCORES> (computed like the Makefile). rocksptest.sh runs one or more binaries under the checked-in gtest-parallel (sharing one parallel worker pool); rockstest.sh runs a single binary directly for a small number of cases. Both reject a leading-option first argument; `rockstest.sh install` writes ~/bin/rockstest and ~/bin/rocksptest shims that defer to the in-tree scripts (with a friendly message when run outside a source root). This is a big win for AI-agent workflows: one command instead of a separate `make` then test-run invocation, removing the overhead of monitoring two long commands, while avoiding serial builds and serial test execution.
4. CLAUDE.md guidance updated to recommend AUTO_CLEAN=1 for manual make invocations and the rocks(p)test.sh helpers for running tests
Although there might be drive to consolidate around the BUCK build for internal use, I'll note that last I checked it was around ~20 minutes to run all the unit tests under buck and ~2 minutes under `make check`.
Bonus: the first revision of this had copyright/license mistakes, and this change corrects similar errors elsewhere, with CLAUDE.md advice updated.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/14883
Test Plan:
Manual verification of the Makefile change detection:
- Confirmed the resolved-flags hash is deterministic across runs for the same goal, and differs between configs (e.g. default/shared vs static_lib, and with/without ASSERT_STATUS_CHECKED).
- Unchanged parameters: rebuild does not error.
- Changed parameters: errors by default; AUTO_CLEAN=1 runs clean-rocks and proceeds; ALLOW_BUILD_PARAMETER_CHANGE=1 bypasses the check.
- Dry runs (make -n) and exempt targets (e.g. `make check-progress`, list_all_tests, gen-pc) do not trip the check and leave the stamp untouched.
- Reproduced the ASSERT_STATUS_CHECKED=1 auto-clean path end to end and confirmed it no longer fails with "No rule to make target 'make_config.mk'"; verified `make clean` removes make_config.mk while `clean-rocks` does not, and that the signature is deleted as the final clean-rocks step.
Manual verification of the helper scripts:
- shellcheck-clean and `bash -n` pass for both scripts.
- Usage/guard behavior: missing argument and leading-option first argument are rejected with a clear message.
- `rockstest.sh install` (validated against a temp HOME) generates both shims; running a shim outside a source root prints "(Not in a rocksdb source root directory?)" and exits non-zero.
- rocksptest.sh argument splitting verified for single binary, multiple binaries, and binaries followed by gtest-parallel/test args.
- .gitignore patterns verified with `git check-ignore` for .build_signature (root and jl/jls) and build_tools/__pycache__/.
Reviewed By: xingbowang
Differential Revision: D109713512
Pulled By: pdillinger
fbshipit-source-id: a801d29afa91f53a2dce717db9fe5e5d485cc154
Summary:
Refresh the fbcode platform010 toolchain and library pins to current third-party2 versions, and remove the now-unused Intel TBB dependency.
Toolchain / deps:
- Fix update_dependencies.sh: the gcc and binutils trees moved from centos8-native to centos9-native, so the old paths resolved to nothing and a re-run emitted empty GCC_BASE/BINUTILS_BASE. Point them at centos9-native.
- Bump clang 15 -> 21; binutils 2.37 -> 2.43; libunwind 1.4 -> 1.8; valgrind 3.19 -> 3.22; plus hash-only refreshes for the LATEST-tracked libs. Regenerate dependencies_platform010.sh.
- Keep GCC pinned at 11.x. The only newer GCC in third-party2 (13.x) is built for glibc >= 2.35 (its libgcc_s needs _dl_find_object@GLIBC_2.35), but platform010 ships glibc 2.34, so GCC 13 will not link/run here.
- zlib stays 1.2.8 (the only version with an x86_64 platform010 build).
- Document why GCC/libgcc/glibc/zlib remain pinned in update_dependencies.sh.
Remove TBB (no longer used) from the build system: the get_lib_base entry, both fbcode_config*.sh, build_detect_platform detection, the CMake WITH_TBB option / find_dependency, and cmake/modules/FindTBB.cmake. Dockerfiles and the HISTORY.md changelog are left untouched. (TBB was used by the old clock cache, long ago removed.)
Although this change was originally motivated by upgrading gcc for its libasan not to hit process lifetime thread limits, upgrading gcc proved impractical under platform010.
Bonus: fix USBAN+gcc build by making GetParam() valid by the time it is called in several test class constructors.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/14882
Test Plan:
a variety of local builds (gcc, clang; various sanitizers) using fbcode tooling
No production code changes
Reviewed By: xingbowang
Differential Revision: D109631787
Pulled By: pdillinger
fbshipit-source-id: 9c466c59b039c2a67ee0318c0ccbac02e19f537b
Summary:
- parse folly getdeps manifests with bare package entries so fallback prefetching actually runs
- validate and remove bad cached/downloaded archives before trying fallback mirrors
- download through temporary files and include libiberty in the GNU toolchain fallback set
Context:
Nightly test failed with dependency download failure in folly.
```
Assessing autoconf...
Download with https://ftpmirror.gnu.org/gnu/autoconf/autoconf-2.69.tar.gz -> /tmp/fbcode_builder_getdeps-Z__wZrocksdbZrocksdbZthird-partyZfollyZbuildZfbcode_builder-root/downloads/autoconf-autoconf-2.69.tar.gz ...
[Complete in 136.616022 seconds]
raise Exception(
Exception: https://ftpmirror.gnu.org/gnu/autoconf/autoconf-2.69.tar.gz: expected sha256 954bd69b391edc12d6a4a51a2dd1476543da5c6bbf05a95b59dc0dd6fd4c2969 but got e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
make: *** [folly.mk:152: build_folly] Error 1
##[error]Process completed with exit code 2.
```
Pull Request resolved: https://github.com/facebook/rocksdb/pull/14763
Test Plan:
1. Connection failure:
- Forced first mirror to http://127.0.0.1:1/...
- It logged connection refused.
- It then tried https://mirrors.kernel.org/gnu/...
- Download succeeded, size 1927468, SHA matched 954bd69b...
2. Empty file / bad hash:
- Ran a local HTTP server returning a zero-byte autoconf-2.69.tar.gz
- Script logged mismatch with actual=e3b0c442... size=0
- It removed the bad download and fell back to mirrors.kernel.org
- Download succeeded with the expected SHA.
3. Existing zero-byte cache:
- Seeded cache with an empty tarball.
- Script removed invalid cache and downloaded a verified copy.
Reviewed By: mszeszko-meta
Differential Revision: D105859558
Pulled By: xingbowang
fbshipit-source-id: ff1f20f87debad561610271ce99b8b8de2d4264f
Summary:
The non-ASCII character check in check-sources.sh used git grep -P (Perl regex), which requires git compiled with PCRE support. On systems without it, the command fails with exit code 128, which is != 1 (no match), so the check always reported a violation -- effectively dead.
Even in CI where git has PCRE2 support, the check was silently broken: git grep -P uses PCRE2 in UTF mode by default, which interprets [\x80-\xFF] as a Unicode codepoint range (U+0080 to U+00FF). Characters like em-dash (U+2014), arrows (U+2192), and math symbols (U+2248, etc.) fall outside that range and were not detected. Only Latin-1 Supplement characters (U+0080-U+00FF) would have been caught.
Replace with LC_ALL=C git grep using bash $'[\x80-\xff]' literal byte range, which works with basic regex in the C locale, and replace all non-ASCII characters in non-excluded source files:
- em-dash to --
- arrow to ->
- math symbols to ASCII equivalents (~=, <=, >=)
- box-drawing characters to ASCII art
Also exclude .github/ from the check, as scripts there can use non-ascii without disrupting RocksDB builds on non-UTF-8 systems.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/14729
Test Plan: manual / CI (make check-sources passes clean)
Reviewed By: hx235
Differential Revision: D104692574
Pulled By: pdillinger
fbshipit-source-id: 1d884c21056dcd83558b825a04b867f1c08e3f45
Summary:
Replace raw owning pointers with std::unique_ptr for StreamingCompress/Uncompress in Create functions and log::Writer/Reader, eliminating manual delete calls in destructors. Also update the StreamingCompressionTest to use unique_ptr for its local StreamingCompress, StreamingUncompress, and MemoryAllocator objects.
Bonus: improve error message for missing/broken ruby (new build requirement)
Pull Request resolved: https://github.com/facebook/rocksdb/pull/14665
Test Plan: ./log_test - all 211 tests pass.
Reviewed By: xingbowang
Differential Revision: D102381893
Pulled By: pdillinger
fbshipit-source-id: 02806b46c8371474a969d6fa3012fff5eb7886c3
Summary:
This fixes the Claude review workflow YAML parse error and adds a CI validation step for GitHub Actions workflow YAML via `make check-
workflow-yaml`.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/14624
Reviewed By: archang19
Differential Revision: D101225058
Pulled By: xingbowang
fbshipit-source-id: 0f371dcbd70ece2c25220c96e43305d9665554b1
Summary:
Upgrade the Ubuntu 24 CI Docker image to include clang-21 (from clang-18), and bump all workflow references to the new image tags. Also adds ccache to both images and renames all clang-18 job references to clang-21.
This is mainly for upgrading the clang version later used for generating C API automatically. See PR https://github.com/facebook/rocksdb/issues/14572
### Changes
**`build_tools/ubuntu24_image/Dockerfile`**
- Add clang-21 installation from LLVM snapshot repo (`apt.llvm.org/noble/llvm-toolchain-noble-21`)
- Add ccache
- Add comment pointing Meta employees to internal devvm build guide
**`build_tools/ubuntu22_image/Dockerfile`**
- Add ccache
- Add comment pointing Meta employees to internal devvm build guide
**`.github/workflows/pr-jobs.yml`**
- Rename `build-linux-clang-18-no_test_run` → `build-linux-clang-21-no_test_run`
- Rename `build-linux-clang18-asan-ubsan` → `build-linux-clang21-asan-ubsan`
- Rename `build-linux-clang18-mini-tsan` → `build-linux-clang21-mini-tsan`
- Update all `clang-18`/`clang++-18` references to `clang-21`/`clang++-21`
- Update ccache key prefixes: `clang18-asan-ubsan` → `clang21-asan-ubsan`, `clang18-tsan` → `clang21-tsan`
- Bump `rocksdb_ubuntu:24.0` → `rocksdb_ubuntu:24.1`
**`.github/workflows/nightly.yml`**
- Rename `build-linux-clang-18-asan-ubsan-with-folly` → `build-linux-clang-21-asan-ubsan-with-folly`
- Update clang-18 → clang-21 compiler references
- Bump `rocksdb_ubuntu:24.0` → `rocksdb_ubuntu:24.1`
**`.github/workflows/clang-tidy.yml`**
- Update clang-18 → clang-21
- Bump `rocksdb_ubuntu:24.0` → `rocksdb_ubuntu:24.1`
### New Docker Images
Both images have been built and tested locally. They will be pushed to `ghcr.io/facebook/rocksdb_ubuntu` before this PR is merged.
- `ghcr.io/facebook/rocksdb_ubuntu:22.2` — adds ccache, adds devvm build note
- `ghcr.io/facebook/rocksdb_ubuntu:24.1` — adds clang-21 from LLVM snapshot repo, adds ccache
Pull Request resolved: https://github.com/facebook/rocksdb/pull/14576
Test Plan: Built both images locally and verified CI job names/compiler flags are consistent.
Reviewed By: joshkang97
Differential Revision: D99694293
Pulled By: xingbowang
fbshipit-source-id: c23c27f5cf870fb2c8b4e3d1cba281d0ce63f9d6
Summary:
Reduce `make check` time from ~9.4 minutes to ~3.7 minutes (2.6x) on a 192-core machine, speed up CI by 1.8x across all jobs, and fix ccache hit rates (macOS cmake: 0.51% → 99.80%).
## Changes
### 1. Auto-detect lld linker (`build_detect_platform`)
- Probe for `lld` at configure time; fall back to default `ld.bfd` if unavailable
- Linux-only guard (`TARGET_OS = Linux`)
- **Measured: 12x faster linking** (7.4s → 0.6s per test binary)
- Add `-L/usr/local/lib` for lld library resolution on CI
- Install lld in CI pre-steps action
- Opt out with `ROCKSDB_NO_FAST_LINKER=1`
### 2. Sharded test execution (`Makefile`)
**Before:** `make check` enumerated every individual gtest case via `--gtest_list_tests`, then created a separate shell script for each case with `--gtest_filter=TestName`. For example, `block_based_table_reader_test` has 9,788 parameterized test cases — each spawned its own process, loading a ~50MB binary, initializing gtest, registering all ~10K tests, running exactly ONE, then tearing down. The per-process overhead was ~1.5s, so 9,788 × 1.5s ≈ 15,000s wasted on overhead alone. Across all 302 test binaries this produced 39,467 individual processes.
**After:** Use gtest's built-in sharding (`GTEST_TOTAL_SHARDS`/`GTEST_SHARD_INDEX`) to group ~10 test cases per process. Each shard loads the binary once and runs multiple tests sequentially — eliminating the per-process overhead. The 9,788 cases in `block_based_table_reader_test` become ~980 shards. Total process count drops from 39,467 to ~4,036. Same tests, same coverage, just fewer process spawns.
The shard count adapts to machine size: `min(ceil(test_count / GTEST_SHARD_SIZE), NCORES * 8)`. This ensures large machines (192 cores) get many small shards for parallelism, while small CI runners (4 cores) get fewer larger shards to avoid excessive overhead. Both `GTEST_SHARD_SIZE` (default 10) and `NCORES` can be overridden.
CI sharding uses round-robin distribution across 3 shards to balance heavy tests (db_test, db_compaction_test, etc.) instead of contiguous alphabetical ranges.
Note: gtest continues running all tests after a failure (`GTEST_THROW_ON_FAILURE=0`), so grouping tests does not mask failures — all failures within a shard are reported.
**Measured: 3.6x CPU reduction** (76,121s → 21,033s)
### 3. Fix SyncPoint leaks for test isolation (5 files)
Sharded execution exposed 43 test files that set SyncPoint callbacks but never clean them up. Stale callbacks with captured local variables cause segfaults or data corruption when subsequent tests in the same process trigger them.
**Systematic fix:** Global gtest `TestEventListener` in `testharness.cc` that calls `SyncPoint::DisableProcessing()` + `ClearAllCallBacks()` + `ClearTrace()` + `LoadDependency({})` after every test case. This cleans up all SyncPoint state: callbacks, dependency maps, cleared points, and the point filter. Registered via static initialization — no changes needed to individual test `main()` functions.
**SyncPoint infrastructure fixes:**
- `DisableProcessing()` now calls `cv_.notify_all()` to wake threads blocked in `Process()`. Previously, threads waiting for predecessor sync points that would never fire (because the test ended) would hang forever.
- `Process()` now rechecks `enabled_` after waking from `cv_.wait()`, so threads exit promptly when processing is disabled instead of looping forever.
**Specific fixes (defense-in-depth):**
- `SstFileReaderTest::VerifyNumEntriesCorruption` — leaked `PropertyBlockBuilder::AddTableProperty:Start` callback that corrupted SST entry counts
- `WritePreparedTransactionTest::CommitAndSnapshotDuringCompaction` — leaked `CompactionIterator:AfterInit` callback causing segfault via dangling pointers
- `TransactionTestBase` destructor — cleanup for 26 SyncPoint uses across transaction tests
- `RetriableLogTest` destructor — cleanup for log reader SyncPoint callbacks
### 4. Fix ccache for reliable cross-run caching (`setup-ccache`, `CMakeLists.txt`)
- **`CCACHE_COMPILERCHECK=content`**: Default `mtime` compared compiler binary modification time, which differs on every fresh CI runner → 0% hit rate. `content` hashes compiler output instead — stable across runner instances
- **`CCACHE_SLOPPINESS`**: CMake generates varying `-MF` paths and timestamps. Without sloppiness settings, ccache treated these as different compilations
- **`CMAKE_C/CXX_COMPILER_LAUNCHER`** instead of `RULE_LAUNCH_COMPILE`: Avoids double-wrapping ccache when also injected via PATH. Removes `RULE_LAUNCH_LINK` since ccache cannot cache link operations
- **macOS cmake hit rate: 0.51% → 99.80%**
### 5. Align GTEST_THROW_ON_FAILURE (`Makefile`)
- Set `GTEST_THROW_ON_FAILURE=0` in Makefile default to match CI's `pre-steps/action.yml`
- `=1` caused `std::terminate` in multi-threaded stress tests (e.g., `point_lock_manager_stress_test`, `rate_limiter_test`) when assertions fail — the gtest exception propagates across thread boundaries, triggering undefined behavior and heap-use-after-free under ASAN
### 6. Portability fixes
- `[[maybe_unused]]` instead of `__attribute__((unused))` for MSVC compatibility
- Remove blanket `-fPIC` from `COMMON_FLAGS` — only apply via `PLATFORM_SHARED_CFLAGS` for shared builds, preserving optimal codegen for release/static builds
- `make -s list_all_tests` to suppress make noise in test enumeration
## Results
### Local (192-core devvm, clean build)
| Metric | Before | After | Improvement |
|--------|--------|-------|-------------|
| Build time (clean) | 2m43s | ~2m | lld linker |
| Test wall clock | 400s | 200s | 2.0x |
| Test CPU total | 76,121s | 21,033s | 3.6x |
| Test failures | 0 | 0 | — |
| **Total make check** | **~9.4min** | **3m41s** | **2.6x** |
### CI (GitHub Actions)
| Metric | Before | After | Improvement |
|--------|--------|-------|-------------|
| Aggregate CI time | 15,220s | 8,425s | 1.8x |
| build-linux-arm | 763s | 221s | 3.5x |
| build-linux-release | 616s | 244s | 2.5x |
| build-windows-vs2022 | 2,868s | 1,205s | 2.4x |
| build-linux-java-static | 858s | 335s | 2.6x |
### ccache hit rates (all jobs)
| Job | Before | After |
|-----|--------|-------|
| macOS cmake (0-3) | 0.51% | **99.80%** |
| build-linux | 99.68% | **99.84%** |
| build-linux-clang18-asan-ubsan | 98.58% | **99.84%** |
| build-linux-clang18-mini-tsan | 98.74% | **99.84%** |
| build-windows-vs2022 | 99.22% | **99.83%** |
| All 27 ccache jobs | — | **>94%** |
## Remaining bottleneck
The critical path is now dominated by genuinely slow integration tests:
- `external_sst_file_test` shards: 97–113s each
- `db_test` shards: 109–124s each
- `db_bloom_filter_test`: 103s (non-parallel)
Further improvement would require test optimization or a `make check-fast` target.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/14478
Test Plan:
- `make check -j192 SKIP_FORMAT_BUCK_CHECKS=1` — all 4,045 shards pass (ASAN+UBSAN), zero failures
- Verified SyncPoint fixes: each crash/corruption reproduces before fix, passes after
- Verified SyncPoint::DisableProcessing wakes blocked threads (previously caused indefinite hangs)
- Verified gtest does NOT stop after failure with `GTEST_THROW_ON_FAILURE=0` — all tests in a shard run and all failures are reported
- CI: all 34 checks pass across two consecutive runs
- ccache: validated hit rates >94% on second run after cache seeding
Reviewed By: hx235
Differential Revision: D97623305
Pulled By: xingbowang
fbshipit-source-id: b299e6d43c8713a9ef2b5659e8bbd74958afe155
Summary:
* Add CLAUDE.md This CLAUDE.md is generated through the analysis of 9,012 commits from the RocksDB repository since 2016. It aggregate the commits into 8 major components based on the component it is changing. It selected top 100 most complex PR based on line of changes to collect the code review feedbacks. For each PR, we collected:
• PR title and description to understand the change context
• Files changed to identify affected components and measure complexity
• Inline code review comments from reviewers
• General review summaries and approval/change request feedback
The feedback was then categorized by RocksDB component and analyzed for recurring themes, patterns, and best practices.
This CLAUDE.md file could be used for guiding code generation and code review.
* Optimize tooling for claude code Add make check-progress and format-auto targets for automation
Add machine-parseable progress reporting for `make check` to support automated monitoring tools like Claude Code:
- Add `build_tools/check_progress.sh` script that outputs JSON progress
- Add `make check-progress` target to poll build/test progress
- Detects phases: compiling -> linking -> generating -> testing
- Reports failed tests with exit codes, signals, and log output
- Limits to 10 failures with last 50 lines of output each
Also add non-interactive formatting support:
- Add `-y` flag to format-diff.sh for auto-apply without prompts
- Add `make format-auto` target for CI/automation use
Pull Request resolved: https://github.com/facebook/rocksdb/pull/14293
Test Plan: Local build
Reviewed By: pdillinger
Differential Revision: D92085763
Pulled By: xingbowang
fbshipit-source-id: ba122a4ff51087aec5c06bab804edfee34e13880
Summary:
Folly download dependencies directly from external source. Sometimes, this could fail due to external website instability. To solve this, we added github cache to cache the dependencies. We also added a python script to try different sources during download to reduce the chance of failure.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/14226
Test Plan: github CI
Reviewed By: krhancoc, archang19
Differential Revision: D90343051
Pulled By: xingbowang
fbshipit-source-id: 3faad6aaa6c1bfd361b9e405c298856cd64bf457
Summary:
Fixing internal validator failure
```
Every project specific source file must contain a doc block with an appropriate copyright header. Unrelated files must be listed as exceptions in the Copyright Headers Exceptions page in the repo dashboard.
A copyright header clearly indicates that the code is owned by Meta. Every open source file must start with a comment containing "Meta Platforms, Inc. and affiliates"
https://github.com/facebook/rocksdb/blob/main/buckifier/targets_cfg.py:
The first 16 lines of 'buckifier/targets_cfg.py' do not contain the patterns:
(Meta Platforms, Inc. and affiliates)|(Facebook, Inc(\.|,)? and its affiliates)|([0-9]{4}-present(\.|,)? Facebook)|([0-9]{4}(\.|,)? Facebook)
```
While fixing the text to pass the linter, I took the opportunity to modify `format-diff.sh` script to add the copyright header automatically if missing in new files.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/14143
Test Plan:
```
$> make format
```
**new python file**
```
build_tools/format-diff.sh
Checking format of uncommitted changes...
Checking for copyright headers in new files...
Added copyright header to build_tools/test.py
Copyright headers were added to new files.
Nothing needs to be reformatted!
```
**new header file**
```
build_tools/format-diff.sh
Checking format of uncommitted changes...
Checking for copyright headers in new files...
Added copyright header to db/db_impl/db_impl_jewoongh.h
Copyright headers were added to new files.
Nothing needs to be reformatted!
```
Reviewed By: hx235
Differential Revision: D87653124
Pulled By: jaykorean
fbshipit-source-id: 164322cfcd2c162bb3b41bb8f3bafefa3f20b695
Summary:
... caused by public headers depending on build parameters (macro definitions). This change also adds a check under 'make check-headers' (already in CI) looking for potential future violations.
I've audited the uses of '#if' in public headers and either
* Eliminated them
* Systematically excluded them because they are intentional or similar (details in comments in check-public-header.sh
* Manually excluded them as being ODR-SAFE
In the case of ROCKSDB_USING_THREAD_STATUS, there was no good reason for this to appear in public headers so I've replaced it with a static bool ThreadStatus::kEnabled. I considered getting rid of the ability to disable this code but some relatively recent PRs have been submitted for fixing that case. I've added a release note and updated one of the CI jobs to use this build configuration. (I didn't want to combine with some jobs like no_compression and status_checked because the interaction might limit what is checked.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/14096
Test Plan: manual 'make check-headers' + manual cmake as in new CI config + CI
Reviewed By: jaykorean
Differential Revision: D86241864
Pulled By: pdillinger
fbshipit-source-id: d16addc9e3480706b174a006720a4def0740bf2e
Summary:
Following up on https://github.com/facebook/rocksdb/pull/14071, updating folly to
8a9fc1e80a or beyond was failing an F14Table assertion for a very subtle reason: ODR violation between the folly build and RocksDB build because folly build was release mode and RocksDB build was debug mode. What was happening was that folly change introduced a dependence on kDebug (whether build is debug) in a hashing implementation in a .h file, and the inconsistency between the inlined implementation during RocksDB build and the linked-to implementation from the folly build was leading to inconsistencies in the data structure.
The primary fix is to ensure we build folly in debug mode for debug mode RocksDB builds. Also,
* Needed to use the `patchelf` tool in `build_folly` to ensure the glog dependency shared library can always find its own gflags dependency. I explored many options for working around this, and this is what would work without reworking folly's own build.
* Updated folly to latest commit.
* Thrown in an ad hoc folly patch to use ftp.gnu.org mirrors (the canonical is super slow)
* Moved the placement of GETDEPS_USE_WGET=1 to apply to local builds also, to avoid the issue of a large download almost reaching completion and then stalling indefinitely.
* Fix failing nightly build-linux-cmake-with-folly-lite-no-test with fmt includes in cmake build (as was done with make build)
* Add a release mode folly+RocksDB to nightly CI, including both cmake and make. This also serves as a non-cached folly build to detect potential problems with PR jobs working from cached folly build.
* Move build-linux-cmake-with-folly to nightly because it's mostly covered by build-linux-cmake-with-folly-coroutines
Intended follow-up:
* folly-lite build with tests
* Make the folly build caching more friendly+accurate by hashing the relevant Makefile parts and tagging whether debug or release. Not in this PR because then you wouldn't be able to see what changed in the folly build steps themselves.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/14094
Test Plan: manual + CI
Reviewed By: mszeszko-meta
Differential Revision: D85864871
Pulled By: pdillinger
fbshipit-source-id: 50009b33422d5781074fcbbdf18089be9e36800d
Summary:
Resolving this folly upgrade required fixing the FOLLY_LITE build with header include from the 'fmt' library.
I was close to timing out on fixing USE_FOLLY_LITE and removing it altogether - it could be considered obsolete and/or not worth the maintenance cost.
Follow-up: make the folly build caching more friendly by hashing the relevant makefile parts. Not in this PR because then you wouldn't be able to see what changed in the folly build steps themselves.
UPDATE/NOTE: I wasn't able to fully update to latest due to a failure seen in F14, using the next folly commit or later. The source of the bug is likely outside of F14 but investigation is in progress.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/14071
Test Plan: CI
Reviewed By: jaykorean
Differential Revision: D85268833
Pulled By: pdillinger
fbshipit-source-id: 1d0a2d61f095524a20e6ec796ef46c02d0696f4e
Summary:
A follow-up to https://github.com/facebook/rocksdb/issues/13904 which was incomplete in updating CI jobs to support C++20 because the C++20 usage was only in tests. Here we add subtle C++20 usage in the public API ("using enum" feature in db.h) to force the issue.
A lot of the work for this PR was in updating the Ubuntu22 docker image, for earlier compiler/runtime versions supporting C++20, and generating a new Ubuntu24 docker image, for later compiler/runtime versions. The Ubuntu22 image needed to be updated because there are incompatibilities with clang-13 + c++20 + libstdc++ for gcc 11, seen on these examples
```
#include <chrono>
int main(int argc, char *argv[]) {
std::chrono::microseconds d = {}; return 0;
}
```
and
```
#include <coroutine>
int main() { return 0; }
```
The second was causing recurring failures in build-linux-clang-13-asan-ubsan-with-folly, now fixed.
So we have to install clang's libc++ to compile with clang-13. I haven't been able to get this to work with some of the libraries like benchmark, glog, and/or gflags, but I'm able to compile core RocksDB with clang-13. On this docker image, an extra compiler parameter is needed to compile with gcc and glog because it's built from source perhaps not perfectly, because the ubuntu package transitively conflicts with libc++.
The Ubuntu24 image seems to be low-drama and generally work for testing out newer compiler versions. The mingw build uses Ubuntu24 because the mingw package on Ubuntu22 uses a gcc version that is too old.
And the mass of other code changes are trying to work around new warnings, mostly from clang-analyze, which I upgraded to clang-18 in CI.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13915
Test Plan: CI, including temporarily including the nightly jobs in the PR jobs in earlier revisions to test and stabilize
Reviewed By: archang19
Differential Revision: D81933067
Pulled By: pdillinger
fbshipit-source-id: 7e33823006a79d5f3cf5bc1d625f0a3c08a7d74c
Summary:
I am wanting to use std::counting_semaphore for something and the timing seems good to require C++20 support. The internets suggest:
* GCC >= 10 is adequate, >= 11 preferred
* Clang >= 10 is needed
* Visual Studio >= 2019 is adquate
And popular linux distributions look like this:
* CentOS Stream 9 -> GCC 11.2 (CentOS 8 is EOL)
* Ubuntu 22.04 LTS -> GCC 11.x (Ubuntu 20 just ended standard support)
* Debian 12 (oldstable) -> GCC 12.2
* (Debian 11 has ended security updates, uses GCC 10.2)
This required generating a new docker image based on Ubuntu 22 for CI using gcc. The existing Ubuntu 20 image works for covering appropriate clang versions (though we should maybe add a much later version as well, in the next increment of our Ubuntu 22 image; however the minimum available clang build from apt.llvm.org for Ubuntu 22 is clang 13).
Update to SetDumpFilter is to quiet a mysterious gcc-13 warning-as-error.
Removed --compile-no-warning-as-error from a cmake command line because cmake in the new docker image is too old for this option.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13904
Test Plan: CI, one minor unit test added to verify std::counting_semaphor works
Reviewed By: xingbowang
Differential Revision: D81266435
Pulled By: pdillinger
fbshipit-source-id: 26040eeccca7004416e29a6ff4f6ea93f2052684
Summary:
We were seeing some internal builds apparently failing the `-d /mnt/gvfs/third-party` check. Although third-party2 is likely a better check (see dependencies_platform010.sh), that would create a big headache with check_format_compatible.sh which has to work across codebase versions.
* Report a WARNING when we detect on a Meta machine but the `-d /mnt/gvfs/third-party` check fails
* Let USE_CLANG influence default compiler choice so that things might still work in that case (e.g. `USE_CLANG=1 make -j24 check`)
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13820
Test Plan: manual, CI
Reviewed By: jaykorean
Differential Revision: D79277197
Pulled By: pdillinger
fbshipit-source-id: 19b2d45ed794f64bbf838f4414568d77ae9ca6f1
Summary:
Some tests were failing due to apparent missing include of iomanip. I suspect this was from a gtest upgrade, because in open source, the include iomanip comes from gtest.h. To ensure we maintain compatibility with older gtest as well as the newer one, I pulled the include iomanip out of the in-repo gtest.h. Note that other places in gtest code only instantiate floating-point related templates with `float` and `double` types.
Also, to avoid `make format` being insanely slow on gtest.h, I've excluded third-party from the formatting check.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13661
Test Plan: make check, internal CI, manually ensure formatting check works outside of third-party/
Reviewed By: jaykorean
Differential Revision: D75963897
Pulled By: pdillinger
fbshipit-source-id: ed5737dd456e74068185f1ac5d57046d7509df7a
Summary:
Leading up to some compression code refactoring, we have a bit of an ifdef nightmare in compression.h relating to zstd support. With the major release RocksDB 10.0.0 coming up, it is a good time to clean up much of this tech debt by requiring zstd >= 1.4.0 (April 2019) if building RocksDB with ZSTD support. For example, Ubuntu 20, the first LTS version to properly support C++17 in its built-in gcc, comes with zstd version 1.4.4. This should not be a significant limitation.
* Almost all of the `ZSTD_VERSION_NUMBER` checks are simplified to just `ZSTD`, though
* `ROCKSDB_ZSTD_DDICT` still needs to be separate because of dependency on `ZSTD_STATIC_LINKING_ONLY` (added to fbcode_config_platform010.sh by the way)
* Similar for ZDICT_finalizeDictionary, which is only generally available in >= 1.4.5
* Eliminate deprecated `kZSTDNotFinalCompression`
* Reduce some cases of unnecessary copying definitions across `#if` branches (e.g. `ZSTDUncompressCachedData`)
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13362
Test Plan:
minor unit test updates. `make check` on several build variants with/without zstd and with/without `ZSTD_STATIC_LINKING_ONLY`
Also deflaked DBTest.DynamicLevelCompressionPerLevel which was flaky before this change but failed once in CI
Reviewed By: cbi42
Differential Revision: D69129453
Pulled By: pdillinger
fbshipit-source-id: ef0cbf9f0fea4e7684fa0999320aa170cfbec233
Summary:
- Updated pinned folly version to the latest
- gcc/g++ 10 is required since https://github.com/facebook/folly/commit/2c1c617e9e so we had to modify the tests using gcc/g++ 7
- libsodium 1.0.17 is no longer downloadable from GitHub so I found it elsewhere. I will submit a PR for that upstream to folly
- USE_FOLLY_LITE changes
- added boost header dependency instead of commenting out the `#include`s since that approach stopped working
- added "folly/lang/Exception.cpp" to the compilation
Pull Request resolved: https://github.com/facebook/rocksdb/pull/12795
Reviewed By: hx235
Differential Revision: D58916693
Pulled By: ajkr
fbshipit-source-id: b5f9bca2d929825846ac898b785972b071db62b1
Summary:
As title
Pull Request resolved: https://github.com/facebook/rocksdb/pull/12563
Test Plan:
```
make -j64 release
```
**Before the fix**
```
$DEBUG_LEVEL is 0, $LIB_MODE is static
Makefile:306: Warning: /mnt/gvfs/third-party2/llvm-fb/1f6edd1ff15c99c861afc8f3cd69054cd974dd64/15/platform010/72a2ff8/../../src/llvm/clang/tools/scan-build/bin/scan-build does not exist
...
```
**After the fix**
```
$DEBUG_LEVEL is 0, $LIB_MODE is static
...
```
Reviewed By: ajkr
Differential Revision: D56318047
Pulled By: jaykorean
fbshipit-source-id: 4a11ad8353fc94aa96676e57c67063d051de5fbc
Summary:
Previously when building with fbcode and having a system install of liburing, it would link liburing from fbcode statically as well as the system library dynamically. That led to the following error:
```
./db_stress: error while loading shared libraries: liburing.so.1: cannot open shared object file: No such file or directory
```
The fix is to skip the feature test for system liburing when `FBCODE_BUILD=true`.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/12525
Test Plan:
- `make clean && make ROCKSDB_NO_FBCODE=1 V=1 -j56 db_stress && ./db_stress`
- `make clean && make V=1 -j56 db_stress && ./db_stress`
Reviewed By: anand1976
Differential Revision: D55997335
Pulled By: ajkr
fbshipit-source-id: 17d8561100f41c6c9ae382a80c6cddc14f050bdc
Summary:
Update llvm-fb to 15 and some other dependency versions.
## Test
Copied over the two script files to tp2 librocksdb source and ran tp2_build, it succeeded.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/12342
Reviewed By: ltamasi
Differential Revision: D53690631
Pulled By: bunnypak
fbshipit-source-id: 68f884b2a565f98bc3510290b411a901ef781adb
Summary:
The original [clang-format-diff.py script](https://raw.githubusercontent.com/llvm/llvm-project/main/clang/tools/clang-format/clang-format-diff.py), referenced in format.sh, exits with a status of 1 at the end after writing diffs to stderr. Consequently, the format.sh script terminates after initializing the 'diffs' variable.
Implemented additional logic in format-diff.sh to ensure continuous execution, even when changes are detected and further formatting is required.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/12329
Reviewed By: pdillinger
Differential Revision: D53483185
Pulled By: cbi42
fbshipit-source-id: b7adff26f129220941258fd6ee83d053fa12b077
Summary:
* Largely based on https://github.com/facebook/rocksdb/issues/12085 but grouped into one large workflow because of bad GHA UI design (see comments).
* Windows job details consolidated into an action file so that those jobs can easily move between per-pr-push and nightly.
* Simplify some handling of "CIRCLECI" environment and add "GITHUB_ACTIONS" in the same places
* For jobs that we want to go in pr-jobs or nightly there are disabled "candidate" workflows with draft versions of those jobs.
* ARM jobs are disabled waiting on full GHA support.
* build-linux-java-static needed some special attention to work, due to GLIBC compatibility issues (see comments).
Pull Request resolved: https://github.com/facebook/rocksdb/pull/12163
Test Plan:
Nightly jobs can be seen passing between these two links:
https://github.com/facebook/rocksdb/actions/runs/7266835435/job/19799390061?pr=12163https://github.com/facebook/rocksdb/actions/runs/7269697823/job/19807724471?pr=12163
And per-PR jobs of course passing on this PR.
Reviewed By: hx235
Differential Revision: D52335810
Pulled By: pdillinger
fbshipit-source-id: bbb95196f33eabad8cddf3c6b52f4413c80e034d
Summary: Python3 makes the use of `(object)` in class inheritance unnecessary. Let's modernize our code by eliminating this.
Reviewed By: itamaro
Differential Revision: D48673915
fbshipit-source-id: a1a6ae8572271eb2898b748c8216ea68e362f06a
Summary:
fixes https://github.com/facebook/rocksdb/issues/11220
fixes https://github.com/facebook/rocksdb/issues/11594
CXX is not initialized early enough in Makefile.
On OpenBSD its value is `g++` at first, and this results in several `command not found`, notably during the tests for HAVE_POWER8 and HAS_ALTIVEC which results in the build problem mentionned in https://github.com/facebook/rocksdb/issues/11594
reordering the Makefile fixes the issue, by placing the creation of make_config.mk and its import before any use of `$(CXX)`
Also, fixes the platofrm version for macos. it must be 10.14 now that rocksdb is using the C++17 standard
Pull Request resolved: https://github.com/facebook/rocksdb/pull/11675
Reviewed By: cbi42
Differential Revision: D48101615
Pulled By: ajkr
fbshipit-source-id: 1f1b4d4604480b31675140b92c6fe97dc55b8c75
Summary:
jemalloc was not building on M1 Macs. This makes it work.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/11257
Reviewed By: anand1976
Differential Revision: D45959570
Pulled By: ajkr
fbshipit-source-id: 08c2b81b399f5003a2c159d037f9bcc5d0059556
Summary:
**Background** - runtime detection of certain x86 CPU features was added for optimizing CRC32c checksums, where performance is dramatically affected by the availability of certain CPU instructions and code using intrinsics for those instructions. And Java builds with native library try to be broadly compatible but performant.
What has changed is that CRC32c is no longer the most efficient cheecksum on contemporary x86_64 hardware, nor the default checksum. XXH3 is generally faster and not as dramatically impacted by the availability of certain CPU instructions. For example, on my Skylake system using db_bench (similar on an older Skylake system without AVX512):
PORTABLE=1 empty USE_SSE : xxh3->8 GB/s crc32c->0.8 GB/s (no SSE4.2 nor AVX2 instructions)
PORTABLE=1 USE_SSE=1 : xxh3->19 GB/s crc32c->16 GB/s (with SSE4.2 and AVX2)
PORTABLE=0 USE_SSE ignored: xxh3->28 GB/s crc32c->16 GB/s (also some AVX512)
Testing a ~10 year old system, with SSE4.2 but without AVX2, crc32c is a similar speed to the new systems but xxh3 is only about half that speed, also 8GB/s like the non-AVX2 compile above. Given that xxh3 has specific optimization for AVX2, I think we can infer that that crc32c is only fastest for that ~2008-2013 period when SSE4.2 was included but not AVX2. And given that xxh3 is only about 2x slower on these systems (not like >10x slower for unoptimized crc32c), I don't think we need to invest too much in optimally adapting to these old cases.
x86 hardware that doesn't support fast CRC32c is now extremely rare, so requiring a custom build to support such hardware is fine IMHO.
**This change** does two related things:
* Remove runtime CPU detection for optimizing CRC32c on x86. Maintaining this code is non-zero work, and compiling special code that doesn't work on the configured target instruction set for code generation is always dubious. (On the one hand we have to ensure the CRC32c code uses SSE4.2 but on the other hand we have to ensure nothing else does.)
* Detect CPU features in source code, not in build scripts. Although there are some hypothetical advantages to detectiong in build scripts (compiler generality), RocksDB supports at least three build systems: make, cmake, and buck. It's not practical to support feature detection on all three, and we have suffered from missed optimization opportunities by relying on missing or incomplete detection in cmake and buck. We also depend on some components like xxhash that do source code detection anyway.
**In more detail:**
* `HAVE_SSE42`, `HAVE_AVX2`, and `HAVE_PCLMUL` replaced by standard macros `__SSE4_2__`, `__AVX2__`, and `__PCLMUL__`.
* MSVC does not provide high fidelity defines for SSE, PCLMUL, or POPCNT, but we can infer those from `__AVX__` or `__AVX2__` in a compatibility header. In rare cases of false negative or false positive feature detection, a build engineer should be able to set defines to work around the issue.
* `__POPCNT__` is another standard define, but we happen to only need it on MSVC, where it is set by that compatibility header, or can be set by the build engineer.
* `PORTABLE` can be set to a CPU type, e.g. "haswell", to compile for that CPU type.
* `USE_SSE` is deprecated, now equivalent to PORTABLE=haswell, which roughly approximates its old behavior.
Notably, this change should enable more builds to use the AVX2-optimized Bloom filter implementation.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/11419
Test Plan:
existing tests, CI
Manual performance tests after the change match the before above (none expected with make build).
We also see AVX2 optimized Bloom filter code enabled when expected, by injecting a compiler error. (Performance difference is not big on my current CPU.)
Reviewed By: ajkr
Differential Revision: D45489041
Pulled By: pdillinger
fbshipit-source-id: 60ceb0dd2aa3b365c99ed08a8b2a087a9abb6a70
Summary:
We haven't been actively mantaining RocksDB LITE recently and the size must have been gone up significantly. We are removing the support.
Most of changes were done through following comments:
unifdef -m -UROCKSDB_LITE `git grep -l ROCKSDB_LITE | egrep '[.](cc|h)'`
by Peter Dillinger. Others changes were manually applied to build scripts, CircleCI manifests, ROCKSDB_LITE is used in an expression and file db_stress_test_base.cc.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/11147
Test Plan: See CI
Reviewed By: pdillinger
Differential Revision: D42796341
fbshipit-source-id: 4920e15fc2060c2cd2221330a6d0e5e65d4b7fe2
Summary:
Instead of existing calls to ps from gnu_parallel, call a new wrapper that does ps, looks for unit test like processes, and uses pstack or gdb to print thread stack traces. Also, using `ps -wwf` instead of `ps -wf` ensures output is not cut off.
For security, CircleCI runs with security restrictions on ptrace (/proc/sys/kernel/yama/ptrace_scope = 1), and this change adds a work-around to `InstallStackTraceHandler()` (only used by testing tools) to allow any process from the same user to debug it. (I've also touched >100 files to ensure all the unit tests call this function.)
Pull Request resolved: https://github.com/facebook/rocksdb/pull/10828
Test Plan: local manual + temporary infinite loop in a unit test to observe in CircleCI
Reviewed By: hx235
Differential Revision: D40447634
Pulled By: pdillinger
fbshipit-source-id: 718a4c4a5b54fa0f9af2d01a446162b45e5e84e1
Summary:
As title
Pull Request resolved: https://github.com/facebook/rocksdb/pull/10717
Test Plan:
Unit Tests
CI
Reviewed By: riversand963
Differential Revision: D39700707
Pulled By: gitbw95
fbshipit-source-id: 54de27e695535a50159f5f6467da36aaf21bebae
Summary:
The current integration with folly requires cherry-picking folly source files to include in RocksDB for external CI builds. Its not scaleable as we depend on more features in folly, such as coroutines. This PR adds a dependency from RocksDB to the folly library when ```USE_FOLLY``` or ```USE_COROUTINES``` are set. We build folly using the build scripts in ```third-party/folly```, relying on it to download and build its dependencies. A new ```Makefile``` target, ```build_folly```, is provided to make building folly easier.
A new option, ```USE_FOLLY_LITE``` is added to retain the old model of compiling selected folly sources with RocksDB. This might be useful for short-term development.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/10103
Reviewed By: pdillinger
Differential Revision: D38426787
Pulled By: anand1976
fbshipit-source-id: 33bc84abd9fdc7e2567749f02aa1b2494eb62b2f
Summary:
Moved linux builds to using docker to avoid CI instability caused by dependency installation site down.
Added the `Dockerfile` which is used to build the image.
The build time is also significantly reduced, because no dependencies installation and with using 2xlarge+ instance for slow build (like tsan test).
Also fixed a few issues detected while building this:
* `DestoryDB()` Status not checked for a few tests
* nullptr might be used in `inlineskiplist.cc`
Pull Request resolved: https://github.com/facebook/rocksdb/pull/10496
Test Plan: CI
Reviewed By: ajkr
Differential Revision: D38554200
Pulled By: jay-zhuang
fbshipit-source-id: 16e8fb2bf07b9c84bb27fb18421c4d54f2f248fd
Summary:
Travis CI is depreciated and haven't been maintained for some time.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/10407
Reviewed By: ajkr
Differential Revision: D38078382
Pulled By: jay-zhuang
fbshipit-source-id: f42057f2f41f722bdce56bf195f67a94835191fb
