mirror of
https://github.com/facebook/rocksdb.git
synced 2026-10-23 08:09:02 +03:00
Summary: `USE_CLANG=1 make check-c-api-gen` failed with "clang++ not found" because the staleness check and the Python generator detected clang inconsistently: the Makefile inspected the `$(CXX)` make variable (the fbcode clang path) while the generator read the `CXX` environment variable, which make never exports. More broadly, clang++ was detected in several ad hoc places with fragile heuristics (grepping the compiler path, duplicated fallback lists in shell and Python). Centralize clang++ detection in build_detect_platform as a single `CLANG_CXX`: - `CLANG_CXX` honors `$CXX` when it is itself clang (so CI's `CXX=clang++-21` is used), else takes the first clang++ on PATH (bare, versioned, or the Meta-internal clang++.par). `CXX` (when building with clang) and `CLANG_ANALYZER` now derive from `CLANG_CXX` instead of re-detecting. - Infer `USE_CLANG=1` when the compiler is clang, detected via the compiler's own `__clang__` predefined macro rather than matching the binary path (which false-positives, e.g. a home dir containing "clang"), and emit it so clang-specific Makefile logic is consistent. - Emit `CLANG_CXX` to make_config.mk and pass it explicitly to the codegen, so `make check-c-api-gen` works without requiring `USE_CLANG=1` and the generator never guesses. Add a first-class `make regen-c-api` target -- now the preferred regenerate command referenced by the generated banners and user-facing messages. The Python generators no longer detect clang themselves; `--clang` defaults to `clang++` for manual runs and is overridden by the Makefile with `CLANG_CXX`. Regeneration verifies the clang runs before writing any file, so a missing/broken toolchain aborts cleanly instead of leaving a partially regenerated tree. Make field discovery clang-version-independent: older clang (e.g. 15) spells field types fully qualified (`rocksdb::ReadTier`, `rocksdb::BlockBasedTableOptions::IndexType`) while the fbcode/CI clang (21) uses the bare `ReadTier` / `IndexType` the family tables expect. Strip the ROCKSDB namespace and enclosing-struct qualifiers so both agree; this is a no-op on newer clang, so checked-in output and CI are unaffected. Also escape the literal "generated" in the generator .py sources (they emit the marker but are not themselves generated) so tooling stops treating them as read-only generated files. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14994 Test Plan: Manual verification (not covered by CI): - `make check-c-api-gen` now runs the staleness sub-check (instead of skipping) and passes in a default gcc build as well as with `USE_CLANG=1` on an fbcode host, since `CLANG_CXX` is always populated. - `ROCKSDB_NO_FBCODE=1 make regen-c-api` using a system clang 15 (clang++.par) produces output byte-identical to the checked-in files (generated with clang 21), and `check-c-api-gen` reports "up to date" -- confirming the namespace/struct stripping is version-robust and a no-op on clang 21. - `regen_all.py --clang /nonexistent` exits non-zero with a clear message and leaves the generated files untouched. - `make regen-c-api` is idempotent (no diff on a clean tree). - Confirmed the escaped `\x40generated` still emits `// generated` byte-for-byte in the regenerated c.h/c.cc/.inc. - Reviewed .github/workflows/pr-jobs.yml: the build-linux-clang-21-no_test_run job's `CXX=clang++-21 make check-c-api-gen` resolves `CLANG_CXX` to clang++-21 (honoring `$CXX`) rather than the container's stray bare clang++. Reviewed By: xingbowang Differential Revision: D113447462 Pulled By: pdillinger fbshipit-source-id: 91ca92d21a71a8b92f9ff57500a8cec91a884078
175 lines
5.8 KiB
Bash
175 lines
5.8 KiB
Bash
#!/bin/sh
|
|
# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved.
|
|
#
|
|
# Set environment variables so that we can compile rocksdb using
|
|
# fbcode settings. It uses the latest g++ and clang compilers and also
|
|
# uses jemalloc
|
|
# Environment variables that change the behavior of this script:
|
|
# PIC_BUILD -- if true, it will only take pic versions of libraries from fbcode. libraries that don't have pic variant will not be included
|
|
|
|
|
|
BASEDIR=`dirname $BASH_SOURCE`
|
|
source "$BASEDIR/dependencies_platform010.sh"
|
|
|
|
# Disallow using libraries from default locations as they might not be compatible with platform010 libraries.
|
|
CFLAGS=" --sysroot=/DOES/NOT/EXIST"
|
|
|
|
# libgcc
|
|
LIBGCC_INCLUDE="$LIBGCC_BASE/include/c++/trunk"
|
|
LIBGCC_LIBS=" -L $LIBGCC_BASE/lib -B$LIBGCC_BASE/lib/gcc/x86_64-facebook-linux/trunk/"
|
|
|
|
# glibc
|
|
GLIBC_INCLUDE="$GLIBC_BASE/include"
|
|
GLIBC_LIBS=" -L $GLIBC_BASE/lib"
|
|
GLIBC_LIBS+=" -B$GLIBC_BASE/lib"
|
|
|
|
if test -z $PIC_BUILD; then
|
|
MAYBE_PIC=
|
|
else
|
|
MAYBE_PIC=_pic
|
|
fi
|
|
|
|
if ! test $ROCKSDB_DISABLE_SNAPPY; then
|
|
# snappy
|
|
SNAPPY_INCLUDE=" -I $SNAPPY_BASE/include/"
|
|
SNAPPY_LIBS=" $SNAPPY_BASE/lib/libsnappy${MAYBE_PIC}.a"
|
|
CFLAGS+=" -DSNAPPY"
|
|
fi
|
|
|
|
if ! test $ROCKSDB_DISABLE_ZLIB; then
|
|
# location of zlib headers and libraries
|
|
ZLIB_INCLUDE=" -I $ZLIB_BASE/include/"
|
|
ZLIB_LIBS=" $ZLIB_BASE/lib/libz${MAYBE_PIC}.a"
|
|
CFLAGS+=" -DZLIB"
|
|
fi
|
|
|
|
if ! test $ROCKSDB_DISABLE_BZIP; then
|
|
# location of bzip headers and libraries
|
|
BZIP_INCLUDE=" -I $BZIP2_BASE/include/"
|
|
BZIP_LIBS=" $BZIP2_BASE/lib/libbz2${MAYBE_PIC}.a"
|
|
CFLAGS+=" -DBZIP2"
|
|
fi
|
|
|
|
if ! test $ROCKSDB_DISABLE_LZ4; then
|
|
LZ4_INCLUDE=" -I $LZ4_BASE/include/"
|
|
LZ4_LIBS=" $LZ4_BASE/lib/liblz4${MAYBE_PIC}.a"
|
|
CFLAGS+=" -DLZ4"
|
|
fi
|
|
|
|
if ! test $ROCKSDB_DISABLE_ZSTD; then
|
|
ZSTD_INCLUDE=" -I $ZSTD_BASE/include/"
|
|
ZSTD_LIBS=" $ZSTD_BASE/lib/libzstd${MAYBE_PIC}.a"
|
|
CFLAGS+=" -DZSTD -DZSTD_STATIC_LINKING_ONLY"
|
|
fi
|
|
|
|
# location of gflags headers and libraries
|
|
GFLAGS_INCLUDE=" -I $GFLAGS_BASE/include/"
|
|
GFLAGS_LIBS=" $GFLAGS_BASE/lib/libgflags${MAYBE_PIC}.a"
|
|
CFLAGS+=" -DGFLAGS=gflags"
|
|
|
|
BENCHMARK_INCLUDE=" -I $BENCHMARK_BASE/include/"
|
|
BENCHMARK_LIBS=" $BENCHMARK_BASE/lib/libbenchmark${MAYBE_PIC}.a"
|
|
|
|
# location of jemalloc
|
|
JEMALLOC_INCLUDE=" -I $JEMALLOC_BASE/include/"
|
|
JEMALLOC_LIB=" $JEMALLOC_BASE/lib/libjemalloc${MAYBE_PIC}.a"
|
|
|
|
# location of numa
|
|
NUMA_INCLUDE=" -I $NUMA_BASE/include/"
|
|
NUMA_LIB=" $NUMA_BASE/lib/libnuma${MAYBE_PIC}.a"
|
|
CFLAGS+=" -DNUMA"
|
|
|
|
# location of libunwind
|
|
LIBUNWIND="$LIBUNWIND_BASE/lib/libunwind${MAYBE_PIC}.a"
|
|
|
|
# location of LIBURING
|
|
LIBURING_INCLUDE=" -isystem $LIBURING_BASE/include/"
|
|
LIBURING_LIBS="$LIBURING_BASE/lib/liburing${MAYBE_PIC}.a"
|
|
CFLAGS+=" -DLIBURING"
|
|
|
|
test "$USE_SSE" || USE_SSE=1
|
|
export USE_SSE
|
|
test "$PORTABLE" || PORTABLE=1
|
|
export PORTABLE
|
|
|
|
BINUTILS="$BINUTILS_BASE/bin"
|
|
AR="$BINUTILS/ar"
|
|
AS="$BINUTILS/as"
|
|
|
|
DEPS_INCLUDE="$SNAPPY_INCLUDE $ZLIB_INCLUDE $BZIP_INCLUDE $LZ4_INCLUDE $ZSTD_INCLUDE $GFLAGS_INCLUDE $NUMA_INCLUDE $LIBURING_INCLUDE $BENCHMARK_INCLUDE"
|
|
|
|
STDLIBS="-L $GCC_BASE/lib64"
|
|
|
|
CLANG_BIN="$CLANG_BASE/bin"
|
|
CLANG_LIB="$CLANG_BASE/lib"
|
|
CLANG_SRC="$CLANG_BASE/../../src"
|
|
|
|
# A clang++ is always available in the fbcode toolchain regardless of whether
|
|
# the build itself uses gcc or clang. CLANG_CXX exposes it for tooling (e.g. the
|
|
# C API codegen, which parses C++ ASTs) that needs clang even in a gcc build.
|
|
CLANG_CXX="$CLANG_BIN/clang++"
|
|
CLANG_ANALYZER="$CLANG_BIN/clang++"
|
|
CLANG_SCAN_BUILD="$CLANG_BIN/scan-build"
|
|
|
|
if [ -z "$USE_CLANG" ]; then
|
|
# gcc
|
|
CC="$GCC_BASE/bin/gcc"
|
|
CXX="$GCC_BASE/bin/g++"
|
|
AR="$GCC_BASE/bin/gcc-ar"
|
|
|
|
CFLAGS+=" -B$BINUTILS -nostdinc -nostdlib"
|
|
CFLAGS+=" -I$GCC_BASE/include"
|
|
CFLAGS+=" -isystem $GCC_BASE/lib/gcc/x86_64-redhat-linux-gnu/11.2.1/include"
|
|
CFLAGS+=" -isystem $GCC_BASE/lib/gcc/x86_64-redhat-linux-gnu/11.2.1/install-tools/include"
|
|
CFLAGS+=" -isystem $GCC_BASE/lib/gcc/x86_64-redhat-linux-gnu/11.2.1/include-fixed/"
|
|
CFLAGS+=" -isystem $LIBGCC_INCLUDE"
|
|
CFLAGS+=" -isystem $GLIBC_INCLUDE"
|
|
CFLAGS+=" -I$GLIBC_INCLUDE"
|
|
CFLAGS+=" -I$LIBGCC_BASE/include"
|
|
CFLAGS+=" -I$LIBGCC_BASE/include/c++/11.x/"
|
|
CFLAGS+=" -I$LIBGCC_BASE/include/c++/11.x/x86_64-facebook-linux/"
|
|
CFLAGS+=" -I$LIBGCC_BASE/include/c++/11.x/backward"
|
|
CFLAGS+=" -isystem $GLIBC_INCLUDE -I$GLIBC_INCLUDE"
|
|
JEMALLOC=1
|
|
else
|
|
# clang
|
|
CLANG_INCLUDE="$CLANG_LIB/clang/stable/include"
|
|
CC="$CLANG_BIN/clang"
|
|
CXX="$CLANG_BIN/clang++"
|
|
AR="$CLANG_BIN/llvm-ar"
|
|
|
|
CFLAGS+=" -B$BINUTILS -nostdinc -nostdlib"
|
|
CFLAGS+=" -isystem $LIBGCC_BASE/include/c++/trunk "
|
|
CFLAGS+=" -isystem $LIBGCC_BASE/include/c++/trunk/x86_64-facebook-linux "
|
|
CFLAGS+=" -isystem $GLIBC_INCLUDE"
|
|
CFLAGS+=" -isystem $LIBGCC_INCLUDE"
|
|
CFLAGS+=" -isystem $CLANG_INCLUDE"
|
|
CFLAGS+=" -Wno-expansion-to-defined "
|
|
CXXFLAGS="-nostdinc++"
|
|
fi
|
|
|
|
KERNEL_HEADERS_INCLUDE="$KERNEL_HEADERS_BASE/include"
|
|
CFLAGS+=" -isystem $KERNEL_HEADERS_INCLUDE/linux "
|
|
CFLAGS+=" -isystem $KERNEL_HEADERS_INCLUDE "
|
|
|
|
CFLAGS+=" $DEPS_INCLUDE"
|
|
CFLAGS+=" -DROCKSDB_PLATFORM_POSIX -DROCKSDB_LIB_IO_POSIX -DROCKSDB_FALLOCATE_PRESENT -DROCKSDB_MALLOC_USABLE_SIZE -DROCKSDB_RANGESYNC_PRESENT -DROCKSDB_SCHED_GETCPU_PRESENT -DROCKSDB_IOURING_PRESENT"
|
|
CXXFLAGS+=" $CFLAGS"
|
|
|
|
EXEC_LDFLAGS=" $SNAPPY_LIBS $ZLIB_LIBS $BZIP_LIBS $LZ4_LIBS $ZSTD_LIBS $GFLAGS_LIBS $NUMA_LIB $LIBURING_LIBS $BENCHMARK_LIBS"
|
|
EXEC_LDFLAGS+=" -Wl,--dynamic-linker,/usr/local/fbcode/platform010/lib/ld.so"
|
|
EXEC_LDFLAGS+=" $LIBUNWIND"
|
|
EXEC_LDFLAGS+=" -Wl,-rpath=/usr/local/fbcode/platform010/lib"
|
|
EXEC_LDFLAGS+=" -Wl,-rpath=$GCC_BASE/lib64"
|
|
# dynamic loading
|
|
EXEC_LDFLAGS+=" -ldl"
|
|
|
|
PLATFORM_LDFLAGS="$LIBGCC_LIBS $GLIBC_LIBS $STDLIBS -lgcc -lstdc++"
|
|
PLATFORM_LDFLAGS+=" -B$BINUTILS"
|
|
|
|
EXEC_LDFLAGS_SHARED="$SNAPPY_LIBS $ZLIB_LIBS $BZIP_LIBS $LZ4_LIBS $ZSTD_LIBS $GFLAGS_LIBS $LIBURING_LIBS $BENCHMARK_LIBS"
|
|
|
|
VALGRIND_VER="$VALGRIND_BASE/bin/"
|
|
|
|
export CC CXX AR AS CFLAGS CXXFLAGS EXEC_LDFLAGS EXEC_LDFLAGS_SHARED VALGRIND_VER JEMALLOC_LIB JEMALLOC_INCLUDE CLANG_CXX CLANG_ANALYZER CLANG_SCAN_BUILD
|