Summary:
`USE_CLANG=1 make check-c-api-gen` failed with "clang++ not found" because the staleness check and the Python generator detected clang inconsistently: the Makefile inspected the `$(CXX)` make variable (the fbcode clang path) while the generator read the `CXX` environment variable, which make never exports. More broadly, clang++ was detected in several ad hoc places with fragile heuristics (grepping the compiler path, duplicated fallback lists in shell and Python).
Centralize clang++ detection in build_detect_platform as a single `CLANG_CXX`:
- `CLANG_CXX` honors `$CXX` when it is itself clang (so CI's `CXX=clang++-21` is used), else takes the first clang++ on PATH (bare, versioned, or the Meta-internal clang++.par). `CXX` (when building with clang) and `CLANG_ANALYZER` now derive from `CLANG_CXX` instead of re-detecting.
- Infer `USE_CLANG=1` when the compiler is clang, detected via the compiler's own `__clang__` predefined macro rather than matching the binary path (which false-positives, e.g. a home dir containing "clang"), and emit it so clang-specific Makefile logic is consistent.
- Emit `CLANG_CXX` to make_config.mk and pass it explicitly to the codegen, so `make check-c-api-gen` works without requiring `USE_CLANG=1` and the generator never guesses.
Add a first-class `make regen-c-api` target -- now the preferred regenerate command referenced by the generated banners and user-facing messages. The Python generators no longer detect clang themselves; `--clang` defaults to `clang++` for manual runs and is overridden by the Makefile with `CLANG_CXX`. Regeneration verifies the clang runs before writing any file, so a missing/broken toolchain aborts cleanly instead of leaving a partially regenerated tree.
Make field discovery clang-version-independent: older clang (e.g. 15) spells field types fully qualified (`rocksdb::ReadTier`,
`rocksdb::BlockBasedTableOptions::IndexType`) while the fbcode/CI clang (21) uses the bare `ReadTier` / `IndexType` the family tables expect. Strip the ROCKSDB namespace and enclosing-struct qualifiers so both agree; this is a no-op on newer clang, so checked-in output and CI are unaffected.
Also escape the literal "generated" in the generator .py sources (they emit the marker but are not themselves generated) so tooling stops treating them as read-only generated files.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/14994
Test Plan:
Manual verification (not covered by CI):
- `make check-c-api-gen` now runs the staleness sub-check (instead of skipping) and passes in a default gcc build as well as with `USE_CLANG=1` on an fbcode host, since `CLANG_CXX` is always populated.
- `ROCKSDB_NO_FBCODE=1 make regen-c-api` using a system clang 15 (clang++.par) produces output byte-identical to the checked-in files (generated with clang 21), and `check-c-api-gen` reports "up to date" -- confirming the namespace/struct stripping is version-robust and a no-op on clang 21.
- `regen_all.py --clang /nonexistent` exits non-zero with a clear message and leaves the generated files untouched.
- `make regen-c-api` is idempotent (no diff on a clean tree).
- Confirmed the escaped `\x40generated` still emits `// generated` byte-for-byte in the regenerated c.h/c.cc/.inc.
- Reviewed .github/workflows/pr-jobs.yml: the build-linux-clang-21-no_test_run job's `CXX=clang++-21 make check-c-api-gen` resolves `CLANG_CXX` to clang++-21 (honoring `$CXX`) rather than the container's stray bare clang++.
Reviewed By: xingbowang
Differential Revision: D113447462
Pulled By: pdillinger
fbshipit-source-id: 91ca92d21a71a8b92f9ff57500a8cec91a884078
Summary:
Refresh the fbcode platform010 toolchain and library pins to current third-party2 versions, and remove the now-unused Intel TBB dependency.
Toolchain / deps:
- Fix update_dependencies.sh: the gcc and binutils trees moved from centos8-native to centos9-native, so the old paths resolved to nothing and a re-run emitted empty GCC_BASE/BINUTILS_BASE. Point them at centos9-native.
- Bump clang 15 -> 21; binutils 2.37 -> 2.43; libunwind 1.4 -> 1.8; valgrind 3.19 -> 3.22; plus hash-only refreshes for the LATEST-tracked libs. Regenerate dependencies_platform010.sh.
- Keep GCC pinned at 11.x. The only newer GCC in third-party2 (13.x) is built for glibc >= 2.35 (its libgcc_s needs _dl_find_object@GLIBC_2.35), but platform010 ships glibc 2.34, so GCC 13 will not link/run here.
- zlib stays 1.2.8 (the only version with an x86_64 platform010 build).
- Document why GCC/libgcc/glibc/zlib remain pinned in update_dependencies.sh.
Remove TBB (no longer used) from the build system: the get_lib_base entry, both fbcode_config*.sh, build_detect_platform detection, the CMake WITH_TBB option / find_dependency, and cmake/modules/FindTBB.cmake. Dockerfiles and the HISTORY.md changelog are left untouched. (TBB was used by the old clock cache, long ago removed.)
Although this change was originally motivated by upgrading gcc for its libasan not to hit process lifetime thread limits, upgrading gcc proved impractical under platform010.
Bonus: fix USBAN+gcc build by making GetParam() valid by the time it is called in several test class constructors.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/14882
Test Plan:
a variety of local builds (gcc, clang; various sanitizers) using fbcode tooling
No production code changes
Reviewed By: xingbowang
Differential Revision: D109631787
Pulled By: pdillinger
fbshipit-source-id: 9c466c59b039c2a67ee0318c0ccbac02e19f537b
Summary:
Leading up to some compression code refactoring, we have a bit of an ifdef nightmare in compression.h relating to zstd support. With the major release RocksDB 10.0.0 coming up, it is a good time to clean up much of this tech debt by requiring zstd >= 1.4.0 (April 2019) if building RocksDB with ZSTD support. For example, Ubuntu 20, the first LTS version to properly support C++17 in its built-in gcc, comes with zstd version 1.4.4. This should not be a significant limitation.
* Almost all of the `ZSTD_VERSION_NUMBER` checks are simplified to just `ZSTD`, though
* `ROCKSDB_ZSTD_DDICT` still needs to be separate because of dependency on `ZSTD_STATIC_LINKING_ONLY` (added to fbcode_config_platform010.sh by the way)
* Similar for ZDICT_finalizeDictionary, which is only generally available in >= 1.4.5
* Eliminate deprecated `kZSTDNotFinalCompression`
* Reduce some cases of unnecessary copying definitions across `#if` branches (e.g. `ZSTDUncompressCachedData`)
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13362
Test Plan:
minor unit test updates. `make check` on several build variants with/without zstd and with/without `ZSTD_STATIC_LINKING_ONLY`
Also deflaked DBTest.DynamicLevelCompressionPerLevel which was flaky before this change but failed once in CI
Reviewed By: cbi42
Differential Revision: D69129453
Pulled By: pdillinger
fbshipit-source-id: ef0cbf9f0fea4e7684fa0999320aa170cfbec233
Summary:
As title
Pull Request resolved: https://github.com/facebook/rocksdb/pull/12563
Test Plan:
```
make -j64 release
```
**Before the fix**
```
$DEBUG_LEVEL is 0, $LIB_MODE is static
Makefile:306: Warning: /mnt/gvfs/third-party2/llvm-fb/1f6edd1ff15c99c861afc8f3cd69054cd974dd64/15/platform010/72a2ff8/../../src/llvm/clang/tools/scan-build/bin/scan-build does not exist
...
```
**After the fix**
```
$DEBUG_LEVEL is 0, $LIB_MODE is static
...
```
Reviewed By: ajkr
Differential Revision: D56318047
Pulled By: jaykorean
fbshipit-source-id: 4a11ad8353fc94aa96676e57c67063d051de5fbc
Summary:
**Background** - runtime detection of certain x86 CPU features was added for optimizing CRC32c checksums, where performance is dramatically affected by the availability of certain CPU instructions and code using intrinsics for those instructions. And Java builds with native library try to be broadly compatible but performant.
What has changed is that CRC32c is no longer the most efficient cheecksum on contemporary x86_64 hardware, nor the default checksum. XXH3 is generally faster and not as dramatically impacted by the availability of certain CPU instructions. For example, on my Skylake system using db_bench (similar on an older Skylake system without AVX512):
PORTABLE=1 empty USE_SSE : xxh3->8 GB/s crc32c->0.8 GB/s (no SSE4.2 nor AVX2 instructions)
PORTABLE=1 USE_SSE=1 : xxh3->19 GB/s crc32c->16 GB/s (with SSE4.2 and AVX2)
PORTABLE=0 USE_SSE ignored: xxh3->28 GB/s crc32c->16 GB/s (also some AVX512)
Testing a ~10 year old system, with SSE4.2 but without AVX2, crc32c is a similar speed to the new systems but xxh3 is only about half that speed, also 8GB/s like the non-AVX2 compile above. Given that xxh3 has specific optimization for AVX2, I think we can infer that that crc32c is only fastest for that ~2008-2013 period when SSE4.2 was included but not AVX2. And given that xxh3 is only about 2x slower on these systems (not like >10x slower for unoptimized crc32c), I don't think we need to invest too much in optimally adapting to these old cases.
x86 hardware that doesn't support fast CRC32c is now extremely rare, so requiring a custom build to support such hardware is fine IMHO.
**This change** does two related things:
* Remove runtime CPU detection for optimizing CRC32c on x86. Maintaining this code is non-zero work, and compiling special code that doesn't work on the configured target instruction set for code generation is always dubious. (On the one hand we have to ensure the CRC32c code uses SSE4.2 but on the other hand we have to ensure nothing else does.)
* Detect CPU features in source code, not in build scripts. Although there are some hypothetical advantages to detectiong in build scripts (compiler generality), RocksDB supports at least three build systems: make, cmake, and buck. It's not practical to support feature detection on all three, and we have suffered from missed optimization opportunities by relying on missing or incomplete detection in cmake and buck. We also depend on some components like xxhash that do source code detection anyway.
**In more detail:**
* `HAVE_SSE42`, `HAVE_AVX2`, and `HAVE_PCLMUL` replaced by standard macros `__SSE4_2__`, `__AVX2__`, and `__PCLMUL__`.
* MSVC does not provide high fidelity defines for SSE, PCLMUL, or POPCNT, but we can infer those from `__AVX__` or `__AVX2__` in a compatibility header. In rare cases of false negative or false positive feature detection, a build engineer should be able to set defines to work around the issue.
* `__POPCNT__` is another standard define, but we happen to only need it on MSVC, where it is set by that compatibility header, or can be set by the build engineer.
* `PORTABLE` can be set to a CPU type, e.g. "haswell", to compile for that CPU type.
* `USE_SSE` is deprecated, now equivalent to PORTABLE=haswell, which roughly approximates its old behavior.
Notably, this change should enable more builds to use the AVX2-optimized Bloom filter implementation.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/11419
Test Plan:
existing tests, CI
Manual performance tests after the change match the before above (none expected with make build).
We also see AVX2 optimized Bloom filter code enabled when expected, by injecting a compiler error. (Performance difference is not big on my current CPU.)
Reviewed By: ajkr
Differential Revision: D45489041
Pulled By: pdillinger
fbshipit-source-id: 60ceb0dd2aa3b365c99ed08a8b2a087a9abb6a70
Summary:
ROCKSDB_SUPPORT_THREAD_LOCAL definition has been removed.
`__thread`(#define) has been replaced with `thread_local`(C++ keyword) across the code base.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/10015
Reviewed By: siying
Differential Revision: D36485491
Pulled By: pdillinger
fbshipit-source-id: 6522d212514ee190b90b4e2750c80c7e34013c78
Summary:
Since they operate at distinct abstraction layers, I thought it
was prudent to combine with EncryptedEnv CI test for each PR, for efficiency
in testing. Also added supported compressions to sst_dump --help output
so that CI job can verify no compiled-in compression support.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/9850
Test Plan: CI, some manual stuff
Reviewed By: riversand963
Differential Revision: D35682346
Pulled By: pdillinger
fbshipit-source-id: be9879c1533fed304ee32c89fd9ba4b07c2b90cc
Summary:
It's to support Meta's internal environment platform010. Gcc still doesn't work but USE_CLANG=1 should work.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/9843
Test Plan: Try to make and ROCKSDB_FBCODE_BUILD_WITH_PLATFORM010=1 USE_CLANG=1 make
Reviewed By: pdillinger
Differential Revision: D35652507
fbshipit-source-id: a4a14b2fa4a2d6ca6fbf1b65060e81c39f079363
