Summary: GitHub-hosted Ubuntu runners expose kernel-specific bpftool packaging that can lag the active Azure kernel. Follow the portable Katran approach by adding bpftool as a pinned getdeps source dependency, installing it into the getdeps bin directory, and removing distro-specific bpftool packages. Regenerate the socket-lb workflow so GitHub Actions explicitly fetches the new dependency.
Reviewed By: kainanpeace666
Differential Revision: D122838798
fbshipit-source-id: 5778f0621ce75c8ca237dfe306888b0f78950de3
Summary:
Memory-capped builds stopped sccache and let the first parallel compiler clients restart it lazily.
Under real `-j36` compiler fanout this exhausted the container PID cgroup, causing intermittent startup timeouts and `unable to fork (EAGAIN)`.
Restart sccache serially under the same `preexec_fn` memory cap before launching CMake. Preserve the uncapped, and missing-sccache behavior, and fail early if the replacement server cannot start.
Reviewed By: srikrishnagopu
Differential Revision: D122615787
fbshipit-source-id: 26120f150e12c8611a4213dad3df585249d5cf10
Summary:
Bump `fbcode/opensource/fbcode_builder/manifests/c-ares` to the 1.34.7
release archive so ShipIt-exported Proxygen builds fetch the patched
version. URL, SHA-256 and `subdir` change; `patchfile =
c-ares-mintimeout.patch` is deliberately left as-is.
Split out of D121051347 because getdeps is a different build system from
buck -- nothing under `third-party-buck` or `tools/build_defs` references
this manifest, so it neither needs nor benefits from landing atomically
with the buck dependency flip, and its verification is entirely separate
(archive digest plus patch reapplication, not a buck build).
The retained `patchfile` line is the part worth a reviewer's attention:
the Meta `ARES_OPT_MINTIMEOUTMS` patch is applied by getdeps at build time
against the downloaded archive, so the upgrade silently loses the
configured DNS timeout floor if that patch stops applying. Verified below
that it still does.
Reviewed By: oargon
Differential Revision: D121207734
fbshipit-source-id: 885c4b15f9f75cfe703c7eb072e6c5ae0a67f818
Summary: The comment near the top of the packsim getdeps manifest explains why it has no keys for building the project out of a monorepo checkout. It now describes those keys (the export project name, its path map and its strip list) instead of quoting them. The manifest's settings are unchanged, so no build changes.
Differential Revision: D122093195
fbshipit-source-id: f338cf7d508db35ccf25eecc3c1c2ce333c1ac73
Summary:
getdeps caps each project's `cmake --build` with an `RLIMIT_AS` of `job_weight_mib * 10` MiB. With `sccache` as the launcher, compiles run inside the long-lived sccache server rather than under ninja, so a project inherits the cap of whichever earlier build started the server (5, 10 or 15 GiB for dependencies) instead of its own.
FBOSS's largest FSDB instantiation TUs now peak at up to 17.2 GiB, so local and GitHub Actions OSS builds fail with `LLVM ERROR: out of memory` depending on which dependency last started the server, even with plenty of host memory free. This is the FSDB OOM targeted by D120549290's stack, whose ninja pool bounds concurrency rather than the per-process cap.
Stopping the server before each capped build makes the next compile respawn it under the current project's cap. Sandcastle builds skip sccache and are unchanged.
Reviewed By: AarjunC
Differential Revision: D121882709
fbshipit-source-id: 0a6d7a1cf309c8bf4bbf23d6f5fb3b5ed6dae2fe
Summary:
- getdeps.py vendor wrote fetcher.hash() next to each vendored project,
which for a git project without a pinned rev is the branch name, so a
project that ships no build/deps/github_hashes (mcrouter) got 'folly
main' recorded, identifying nothing. Ask the checkout for HEAD instead.
- Also record a version as a third field where one can be determined:
for a git checkout the nearest tag (git describe, deepening the
shallow clone as needed), as the tag itself when HEAD is tagged or
<tag>^<distance>.<commit> otherwise; for a downloaded archive the
version in its file name. Distributions bundling the vendored tree
need 'Provides: bundled(<name>) = <version>' (Fedora guidelines).
- Internal adjustments on import: BLACK/import sorting via arc lint -a,
E731 lambda fix in the new test.
Reviewed By: bigfootjon
Differential Revision: D121677377
fbshipit-source-id: f4a8a01925b8c8f9f431fd44ba348fb7bdfb4b7a
Summary:
When `--allow-system-packages` is enabled on Ubuntu, Katran’s `[debs]` entry makes getdeps treat the project itself as satisfied by `libssl-dev`. As a result, dependent projects do not fetch or build Katran and cannot find `katranConfig.cmake`.
Remove the misleading package replacement. Folly’s OpenSSL dependency continues to pull in `libssl-dev` transitively.
Reviewed By: p-shah256
Differential Revision: D121691384
fbshipit-source-id: f68680312e5c6f2211c1e28699d9aae2dafeb12d
Summary:
Why?
Four statements the export makes to an external reader are wrong or
incomplete, and nothing internal checks any of them — no test reads the
READMEs, no job runs the GitHub workflow, and `oss_check`'s content lint
only looks for internal tokens.
1. `.github/ISSUE_TEMPLATE/bug_report.md` points at `examples/incast-2host.json`.
The exported path is `packsim/examples/incast-2host.json`; a bare
`examples/` resolves to a root directory that does not exist. The same
file gets the sibling reference right two comments later, so this is an
inconsistency inside one file rather than a convention.
2. `packsim/README.md` tells a maintainer to regenerate the GitHub workflow
with a command that does not reproduce it. `--free-up-disk` and
`--free-up-disk-before-build` both default to `False`
(`getdeps/workflow_generator.py`), and the three steps they gate —
deleting the Android SDK, dotnet, powershell and ghcup — are load-bearing
on a job that builds boost, openssl, folly and fbthrift from source and
then compiles every unit-test executable. Following the documented command
silently drops them and yields a disk-exhaustion failure with nothing in
the diff to explain it.
3. `packsim/README.md` names `libdwarf`, `libaio` and `libiberty` as
"copyleft" without saying what that asks of the reader, or that all three
arrive through folly rather than through PackSim. It also does not
mention that the CMake path against pre-installed dependencies avoids
them entirely. Note `libunwind` is in the same folly Linux dependency
block but is MIT, not copyleft, so it is deliberately NOT added here.
4. `CONTRIBUTING.md` sends security reports to `facebook.com/whitehat`
while `bug_report.md` and the shared `SECURITY.md` both use
`bugbounty.meta.com`.
How?
Fixes all four, and adds two things the export was missing:
- `.github/ISSUE_TEMPLATE/config.yml`, which turns off the blank-issue link
so the templates are the only route in and the security notice cannot be
bypassed, and adds a contact link for security reports. No Configerator
change was needed — `oss_check classify` confirms the existing
`ISSUE_TEMPLATE/*.(md|ya?ml)` exception already admits it.
- A Contributing section on the root `README.md`. Neither `CONTRIBUTING.md`
nor `CODE_OF_CONDUCT.md` was linked from the front page.
Two internal-only corrections ride along because they are the same class of
staleness and are cheaper to fix than to explain:
- `CMakeLists.txt` said its test sources are globbed out of 23 `oss_tests/`
directories. There are 24. The count is dropped rather than incremented,
since the sentence reads identically without it and cannot then drift.
- The getdeps manifest hard-coded "187 test executables" in a comment that
ships publicly. Replaced with a pointer at the figure the configure step
prints live.
- `.llms/rules/packsim-oss-build.md` claimed five files ship from this
directory and that nothing under `.github/` but a workflow does. The
contributor templates have shipped since D120640286. Its `apply_to_path`
is widened to match, so the rule now actually fires on the files it
describes.
Differential Revision: D121131700
fbshipit-source-id: 75439e492a46616443c4b1ab56285158f5e4b955
Summary: Ubuntu exposes `bpftool` as a virtual APT package, which makes getdeps fail because APT cannot choose a provider. Select `linux-tools-common` for Ubuntu while retaining `bpftool` for non-Ubuntu environments.
Reviewed By: tagrawal03
Differential Revision: D121095850
fbshipit-source-id: 170d670d725da798fcc1feed925408d54a1115d9
Summary:
`mcrouter`'s buck targets, sources, and CMake `find_package` list never reference `mvfst`. The single hit is a test string literal (`set_umember3() = "mvfst"`) exercising serialization round-trips — pure test data, no headers used, no libraries linked, and it would pass identically with any other string.
So the manifest's direct `mvfst` entry is dropped. Note: `mvfst` stays in the resolved closure transitively via `fbthrift`; this changes declared direct deps only, not what gets built.
Reviewed By: bigfootjon
Differential Revision: D121020246
fbshipit-source-id: d9fe8f7934af7cdcc2d8be271c4151f246ba75c0
Summary:
Bump the OSS/getdeps default SAI version to 1.19.0 (libsai manifest, CMakeLists, run-getdeps.py, build-helper.py, fboss_oss_build.bzl, container docs).
Sites updated:
- opensource/fbcode_builder/manifests/libsai: url, sha256, subdir
- fboss/github/CMakeLists.txt: default SAI_VERSION + status message
- fboss/oss/scripts/run-getdeps.py: new SAI_VERSION_SHAS entry
- fboss/oss/scripts/build-helper.py: url + sha tables, choices, const/default, help
- fboss/facebook/oss_ci/fboss_oss_build.bzl: DEFAULT_SAI_VERSION + its docstring
- fboss/github/docs/static/code_snips/build_helper.sh: default text + supported list
The v1.19.0 tarball sha256 (199d2cd32408b6470ce0c43beee660cc468e29e72921f1e93e64d4a8eb80246a)
was obtained by downloading the release tarball and hashing it; it is used
identically in all three places that record it.
Building_FBOSS_on_containers.md carries no SAI version reference, so it needed
no change despite being listed in the integration runbook.
Reviewed By: ashutoshgrewal, shri-khare
Differential Revision: D120953509
fbshipit-source-id: 01072090c5844062d31f4ec2ab4aff6ed421d9b9
Summary:
## What
Adds a `[github.actions]` section to `opensource/fbcode_builder/manifests/packsim`
setting `timeout_minutes = 180`, and regenerates
`ai_simulation/.github/workflows/getdeps_linux.yml` from it.
## Why
The job had 60 minutes, which is the workflow generator's fallback
(`workflow_generator.py`: `timeout_minutes = gh("timeout_minutes") or "60"`) and
not a figure anyone picked for this project. One run has to build boost,
openssl, folly and fbthrift from source, then compile 187 test executables, then
run them -- and each test process defaults to 4 PDES worker threads in the
open-source build while ctest is already running jobs in parallel, so the test
phase oversubscribes a standard two-core runner several times over.
180 is a ceiling, not an estimate, and the manifest comment says so. Nothing is
known about the real wall clock because the workflow has never executed: nothing
has been pushed to the repository yet, and it only triggers on push and pull
request against `main`. Its first run will be its first measurement. A job that
is killed at 60 minutes reports as a red badge on the repository's front page
and tells you nothing about why; one that finishes in 70 tells you the number to
set. Lower it once a green run has reported one.
Deliberately not changed: the runner size. `--cpu-cores` would also help the
oversubscription, but that is a standing cost decision rather than a one-off
ceiling, and it belongs with whoever owns the project's CI budget.
## Why the manifest and not the workflow
The workflow file carries an `generated by getdeps.py` header and is rebuilt by
`opensource/fbcode_builder/getdeps/facebook/update-all-github-actions.sh`, which
already has an entry for packsim. Editing the YAML directly would be reverted by
the next sweep; `[github.actions]` keys in the manifest are the supported way to
change it, and they survive.
Differential Revision: D120642627
fbshipit-source-id: 4ea8d4a5bcc8533e6eec0cfb41dde63871c72f9b
Summary:
Enable getdeps-based builds on every Enterprise Linux distro (RHEL, AlmaLinux, Rocky Linux, CentOS Stream) by selecting EL-specific manifest sections with distro_family instead of individual distro names.
The manifests that mean "any Enterprise Linux 9" (or 8) spell it as distro=centos_stream, because CentOS Stream is what CI runs. That never matched AlmaLinux or Rocky, and after the previous change gave RHEL its own distro name it would not match RHEL either. The per-distro enumeration is the bug class itself: each file was incomplete in a different way (libaio hand-listed alma+rocky but missed RHEL; everything else listed only centos_stream), so the fix has to apply everywhere the pattern occurs -- splitting it per project would just re-create the drift. Every project touched here builds against the same Folly-stack toolchain (gcc toolsets via the shared gcc12/gcc14 manifests, clang19, python3.12 for sapling), so distro coverage has to move together. The most consequential case is cachelib's gcc-toolset-14: without it an EL 9 build uses the default GCC 11 and fails to compile C++20 code although the toolset is right there in the repositories.
Rewrite those sections with the distro_family variable introduced in the previous change, preserving each selector's meaning while extending it to the whole family:
```
- all(distro=centos_stream,distro_vers=9) -> all(distro_family=rhel,distro_vers=9)
cachelib, gcc12, gcc14, openr, python, python-setuptools,
rebalancer, rebalancer-explorer
- all(distro=centos_stream,distro_vers=8) -> all(distro_family=rhel,distro_vers=8)
boost, boost-python
- not(all(any(distro=centos_stream,distro=centos),distro_vers=8))
-> not(all(distro_family=rhel,distro_vers=8))
libmnl
- not(any(all(distro=centos,distro_vers=8),all(distro=centos_stream,distro_vers=9)))
-> not(all(distro_family=rhel,any(distro_vers=8,distro_vers=9)))
lz4, lz4-python
- rpms.any(distro=centos_stream,...,distro=fedora) -> rpms
libaio, numa (every rpm distro ships libaio-devel and numactl-devel)
- rpms.distro=centos_stream -> rpms.distro_family=rhel
libaio-python
```
"centos" was the old CentOS Linux 8 name and folds into the family. The wider exclusions this produces (e.g. lz4-static now also excluded on centos_stream 8 and centos 9) only touch EOL or practically unused combos. No manifest still selects on an individual EL distro name. libaio-devel and numactl-devel are confirmed present on RHEL, CentOS and Fedora, the full rpm universe getdeps targets.
This must land after the distro_family change: the expression parser rejects unknown selector variables, so on a getdeps without it these manifests fail to load.
Tested with both changes applied: every manifest passes validate-manifest, and `install-system-deps --recursive --dry-run --os-type linux --distro rhel --distro-version 9 cachelib` now yields the same package set as --distro centos_stream, including gcc-toolset-14, libaio-devel and numactl-devel, where before it lacked all three. The openr, rebalancer and rebalancer-explorer hunks are the same mechanical rewrite but those projects were not built here.
**Depends on https://github.com/facebook/CacheLib/issues/495** (`distro_family`): must land after it, since the expression parser rejects unknown selector variables. Submitted separately so each can be reviewed on its own; the two are otherwise independent of https://github.com/facebook/CacheLib/issues/494.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
X-link: https://github.com/facebook/CacheLib/pull/496
Reviewed By: likewhatevs
Differential Revision: D120501944
Pulled By: michel-slm
fbshipit-source-id: fdfc8b65770736ed190223df5e4dc2d0b2b417ba
Summary: Export public build scaffolding at the repository root and place the implementation under `socket-lb/`, matching Katran and other Meta OSS projects. Keep the top-level CMake and build scripts usable from both the internal and exported layouts, and update public build instructions accordingly.
Reviewed By: p-shah256
Differential Revision: D118875318
fbshipit-source-id: 996c0735d98fa093aa01bf08fce590e9e072d8f5
Summary:
Fedora retired liboqs in Fedora 45 ("Obsoleted by openssl 3.5": OpenSSL 3.5 ships ML-KEM natively), so the `[rpms.distro=fedora]` mapping added in facebook/CacheLib#488 only resolves on Fedora 44 and older. On Rawhide a build with --allow-system-packages that generates its BuildRequires from the manifests fails at dependency resolution with "No match for argument: liboqs-devel". fizz still hard-depends on liboqs, so let getdeps build it from source (or the vendor tree) on Fedora as it does everywhere else.
Found while building the Fedora cachelib package on Rawhide with BuildRequires generated from the manifests; the mapping was added in https://github.com/facebook/CacheLib/issues/488 and only checked against Fedora 44 at the time. CentOS Stream 10 does ship liboqs-devel, so a `distro_family=rhel` mapping for EL 10 could be considered separately once https://github.com/facebook/CacheLib/issues/495 lands.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
X-link: https://github.com/facebook/CacheLib/pull/497
Reviewed By: likewhatevs
Differential Revision: D120517415
Pulled By: michel-slm
fbshipit-source-id: 1386a4418536d59a36feffb24f85f48cf19f11f3
Summary:
get_linux_type() derives the distro name from NAME in /etc/os-release, which on Red Hat Enterprise Linux yields "red_hat_enterprise". Nothing knows that name: HostType.get_package_manager() returns None for it, so --allow-system-packages silently satisfies nothing and every dependency is built from source, and the manifests with distro=centos_stream selectors do not apply either. RHEL's VERSION_ID also carries a minor ("9.8"), as do AlmaLinux's and Rocky's, so even a matching distro would miss the distro_vers=9 selectors written for CentOS Stream.
This matters for EPEL: Fedora's Koji builds EPEL 9 packages in a buildroot with redhat-release (real RHEL), while EPEL 10 buildroots and the local mock configs use centos-stream-release. A getdeps-based EPEL 9 build therefore behaves differently in Koji than in local mock. Nobody noticed because getdeps has not been used for distro packaging before and RHEL is rare among people building from source.
Rather than pretend RHEL is CentOS Stream, model what the manifests actually mean. Following Chef's platform_family, introduce a distro_family alongside distro: "rhel" for rhel, centos, centos_stream, alma and rocky; "fedora"; "debian" for debian, ubuntu, pop!_os and mint; "arch". Fedora is deliberately its own family, unlike Ansible's os_family which folds it into RedHat: its base repositories carry many dependencies that the EL family only has in EPEL or not at all (glog, gflags, fmt, fast_float, googletest, benchmark, liboqs, jq), zlib is zlib-ng-compat there, and CentOS Stream 9 needs gcc-toolset; nineteen manifest sections already select on exactly that difference.
Concretely: - RHEL reports distro "rhel", the same short name as its os-release ID; - within the rhel family distro_vers is the major version only, so RHEL 9.8, Alma 9.6 and Rocky 9.6 all match distro_vers=9 as CentOS Stream 9 does; Fedora and Ubuntu keep their full versions; - distro_family is available to manifest selectors and drives get_package_manager(), which now returns rpm for the whole rhel family; - `install-system-deps --distro` accepts rhel; - os-release parsing moves into parse_os_release() so it can be unit tested.
Existing selectors keep their meaning: distro=centos_stream still matches only CentOS Stream. A follow-up will switch the sections that mean "any EL 9" to distro_family=rhel so that RHEL, Alma and Rocky pick up gcc-toolset, libaio and numactl too. Until then a RHEL host already gains every plain [rpms] mapping and the not(distro=fedora) ones.
Tested: `install-system-deps --recursive --dry-run --os-type linux --distro rhel --distro-version 9 cachelib` now produces a dnf command (previously "I don't know how to install any packages on this system"), and unit tests cover the os-release parsing for Fedora, CentOS Stream, RHEL, Alma, Rocky and Ubuntu plus the family and package-manager mapping. Verified against synthetic os-release contents, not a live RHEL host.
A follow-up PR will switch the manifest selectors that mean "any EL 9" (gcc-toolset, libaio, numactl and friends) to `distro_family=rhel`; it must land after this one because the expression parser rejects unknown selector variables.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
X-link: https://github.com/facebook/CacheLib/pull/495
Reviewed By: likewhatevs
Differential Revision: D120501703
Pulled By: michel-slm
fbshipit-source-id: f24b01e46207a08ca09b9f09500ba0fb78a502fa
Summary:
Fedora and EPEL package sparse-map as tsl-sparse-map-devel: 0.7.0 on F45 and Rawhide (stable since 2026-09-17), 0.6.2 on F44 and EPEL 9, and an EPEL 10 build is under way. With --allow-system-packages this takes sparsemap out of the set of projects built from source, leaving only the Meta stack (folly, fizz, wangle, mvfst, fbthrift) for a vendored distro build. A plain `[rpms]` section is used, as for libsodium, xxhash and snappy: where the package is not installed (CentOS Stream without EPEL, say) getdeps simply falls back to building sparsemap itself.
The F44 version is older than the 0.7.0 the manifest pins, so the difference was checked rather than assumed: the public API of sparse_map and sparse_set is identical between 0.6.2 and 0.7.0 down to template defaults, the tsl::detail_sparse_hash::sparse_array template cachelib reaches into has the same parameters, and the serialization format is unchanged. 0.7.0's only compatibility change is requiring C++17, irrelevant to cachelib. cachelib locates the headers by path (cmake/FindSparsemap.cmake: find_path tsl/sparse_map.h), which the package satisfies at /usr/include/tsl.
Tested on Fedora 44 with tsl-sparse-map-devel 0.6.2 installed: `getdeps.py --allow-system-packages build --no-tests --src-dir=. cachelib` no longer fetches or builds sparsemap (no Assessing/Building step for it in the log), cachelib configures with SPARSEMAP_INCLUDE_DIRS=/usr/include, SparseMapIndex.cpp compiles against the system header with -Werror on, cachebench links, and the build exits 0 after 58 minutes at two jobs.
Manifest-only follow-up to https://github.com/facebook/CacheLib/issues/488 and https://github.com/facebook/CacheLib/issues/493. RHEL hosts only benefit once https://github.com/facebook/CacheLib/issues/495 (distro_family) lands, since getdeps does not currently recognise RHEL as an rpm distro; the two are independent otherwise.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
X-link: https://github.com/facebook/CacheLib/pull/494
Reviewed By: likewhatevs
Differential Revision: D120501594
Pulled By: michel-slm
fbshipit-source-id: db5b7ad6514b56984aa3d92a24dbde1614ff1ed4
Summary: Convert os.path joins, basenames, dirnames, and exists/isfile/isdir/islink checks in the getdeps test suite to pathlib with os.fspath at the str boundaries, matching slices 1-3b. Deliberately kept: os.path.abspath (no exact Path equivalent; still used inside Path() for the fixtures/manifests dirs), os.path.realpath replaced only where resolve() is exactly equivalent, os.unlink/symlink/chmod/listdir (not os.path), and vendor_test.py (owned by unlanded D120204081). Verified: arc lint clean, pyre clean, buck test 113/113 pass.
Reviewed By: vitaut
Differential Revision: D120383078
fbshipit-source-id: 94b41927b849bd1667793648f5351476fa49fa98
Summary: Convert remaining os.path joins, basenames, dirnames, and exists/isfile checks to pathlib with os.fspath at the str-contract boundaries, matching slices 1-3a. Deliberately kept: os.path.normcase/relpath (no Path equivalent), os.fsdecode, os.path.sep, listdir/lstat traversal, os.walk in include_rewriter (rglob would change prune semantics), the abspath site in workflow_generator, and all test files (no mock seams affected; vendor_test stays owned by D120204081). Verified: arc lint clean, pyre clean, buck test 113/113 pass.
Reviewed By: bigfootjon
Differential Revision: D120383077
fbshipit-source-id: 37741cd8a8a042e40589b2925b08885f5bca8e7c
Summary: fetcher.py path manipulation now uses pathlib (joins, exists, makedirs, archive validation via resolve()+relative_to instead of a raw startswith prefix check). get_src_dir() stays -> str and constructor path fields stay str (builders/loaders consume them as str; retyped in the core slice). ShipitPathMap is deliberately untouched: its posix-matching semantics are load-bearing per the comment on add_mapping. retry_test now neutralizes Path.mkdir instead of os.makedirs (the seam _download_dir uses). New LocalDirFetcher/copy_if_different Path-input tests.
Reviewed By: likewhatevs
Differential Revision: D120383075
fbshipit-source-id: 8908611c0f79d853b09dcc86f9da09e4782e389b
Summary: Adds lint_cmake_floors.py, an arc lint wrapper holding the single cmake_minimum_required floor (3.14, from folly) for the folly-family OSS CMake projects, registered in fbsource-lint-engine.toml as OSSCMAKEFLOOR. It checks 13 effective roots (both sides of fbcode<->xplat mirrors, so skew is caught too) with at-or-above semantics so fbthrift 3.24 and mcrouter 3.16 stay legal, and offers the bump as an arc lint -a autofix. Any future dip below the floor fails lint on the offending diff instead of shipping a deprecation warning (or worse) to OSS consumers. Related: facebook/folly#2703.
Reviewed By: likewhatevs
Differential Revision: D120432856
fbshipit-source-id: 74895b760e4e73238ae0321f6e7eb5f966e39215
Summary:
job_weight_mib is the per-compiler memory estimate getdeps divides available RAM by to pick the build concurrency, and the base of the RLIMIT_AS safety net it puts on each build process. cachelib and fbthrift declared 2048 MiB, which two measured builds show is low.
Peak resident memory of every cc1plus process was sampled every two seconds during full builds on Fedora 44 (aarch64, gcc 15), once at four jobs and once at two jobs with tests disabled; 1182 compiler processes were observed across the fbthrift and cachelib phases of the second run:
| percentile | peak RSS |
|------------|----------|
| p50 | 489 MiB |
| p90 | 1321 MiB |
| p95 | 1626 MiB |
| p99 | 2580 MiB |
| max | 4027 MiB |
24 translation units exceeded 2048 MiB and 4 exceeded 3072 MiB. The heaviest were cachebench's Stressor.cpp (4.0 GiB) and fbthrift's ThriftServer.cpp (3.8 GiB), ThriftQuicServer.cpp (3.1 GiB), ThriftRocketServerHandler.cpp and Cpp2Connection.cpp (2.8 GiB each).
3072 MiB matches what proxygen, fboss, openr, bgp and moxygen already declare and keeps a builder sized for N jobs from over-committing when several of the heavy units land together, which on a machine whose scratch dir sits on a tmpfs was enough to push it into swap. The weight is a typical-case budget rather than a ceiling, so the remaining 4 GiB outliers are left to the 10x RLIMIT_AS margin. fbthrift-python builds the same sources and moves with fbthrift.
Follow-up to https://github.com/facebook/CacheLib/issues/488/#490/https://github.com/facebook/CacheLib/issues/491, measured on the same host; manifest-only change.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
X-link: https://github.com/facebook/CacheLib/pull/492
Reviewed By: likewhatevs
Differential Revision: D120414253
Pulled By: michel-slm
fbshipit-source-id: f93eab02ecb3d3aa6a5b4e1a45a31cd7b376c366
Summary:
sparse-map 0.6.2 declares cmake_minimum_required(VERSION 3.1). CMake 4 removed compatibility with anything below 3.5, so on hosts whose system CMake is 4.x (Fedora 44, Homebrew, the GitHub macOS runners, and soon Ubuntu 26.04 and Debian 14) the sparsemap step fails to configure:
```
CMake Error at CMakeLists.txt:1 (cmake_minimum_required):
Compatibility with CMake < 3.5 has been removed from CMake.
```
Upstream fixed this in 0.7.0 (released 2025-11-02), which requires CMake 3.10, so take the release rather than carry a patch. 0.7.0 also moves to C++17, fixes iterator invalidation on a non-inserting insert and the serialization of small hashes, and adds a no-exceptions mode; the headers install to the same include/tsl layout. cachelib is the only manifest that depends on sparsemap, and the one internal type it reaches into, tsl::detail_sparse_hash::sparse_array, keeps the same template parameters.
Tested on Fedora 44 with system CMake 4.3.0 and gcc 15, no CMAKE_POLICY_VERSION_MINIMUM anywhere on the command line: `getdeps.py --allow-system-packages build --no-tests --src-dir=. cachelib` configured and installed sparsemap 0.7.0 with no CMake error (the only CMake output of note being the < 3.10 deprecation warnings from fizz and wangle, tracked in facebook/folly#2703), compiled SparseMapIndex.cpp against the new headers with -Werror on, and linked cachebench; 51 minutes at two jobs.
Manifest-only change; follow-up to https://github.com/facebook/CacheLib/issues/488/#490/https://github.com/facebook/CacheLib/issues/491. The CMake version matrix behind the platform list is in facebook/folly#2703.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
X-link: https://github.com/facebook/CacheLib/pull/493
Reviewed By: bigfootjon
Differential Revision: D120425085
Pulled By: michel-slm
fbshipit-source-id: 9802415458f4d6ba20fab21630ec9047d7ba45d5
Summary: katran_tester is moving to a CLI11 command interface, so the getdeps build needs CLI11 installed and KatranTesterCli.cpp compiled. Nothing calls it yet; the next diff adds the dispatch. Split out because edits under fbcode/opensource trigger every open source build.
Reviewed By: tagrawal03
Differential Revision: D119598499
fbshipit-source-id: 9db46020fc36e3e00f828a4b35ba725fd1a149eb
Summary: copytree, envfuncs.path_search internals, and runcmd cwd/log_file now operate on pathlib.Path, accepting str | Path. Env values, argv lists, and hash inputs stay str by necessity (subprocess + project hashes); path_search documents its str return for the same reason, to be revisited when core builders are retyped. No caller changes needed. New tests/test_path_inputs.py locks the widened contracts.
Reviewed By: bigfootjon
Differential Revision: D120383074
fbshipit-source-id: b4145f56d906f58dc8f71d7ad96a08b419f19796
Summary: getdeps already requires 3.6+ in practice (interpolated f-strings throughout), but nothing stated or enforced it: pre-3.6 interpreters died with a bare SyntaxError on the first f-string. README now documents the floor, getdeps.py exits early with a clear message below 3.6, and the dead sys.version_info < (3, 2) shim in manifest_test.py is gone.
Reviewed By: likewhatevs
Differential Revision: D120383073
fbshipit-source-id: 394c5757b3de1262e81356ca71ed4eb64f3f3709
Summary:
Three commits, best read in order. Together they give getdeps the equivalent of `cargo vendor` plus `--offline`, which is what Fedora's vendored-dependency guidelines need for packaging cachelib without maintaining separate folly/fizz/wangle/mvfst/fbthrift packages in lockstep.
## getdeps: add a vendor subcommand
`getdeps.py vendor --output-dir DIR project` fetches every third-party dependency of the project and copies its source tree to `DIR/<name>`, skipping deps satisfied by system packages and the project itself. It also writes DIR/getdeps-vendor.txt listing each vendored project with its fetcher hash (the pinned git revision or the archive sha256).
This is the getdeps analogue of `cargo vendor` / `go mod vendor`: distro packagers can ship the result as a vendor tarball next to the project source, as Fedora's vendored-dependency guidelines require, and a follow-up change will let `build` consume such a directory offline.
Extracted trees rather than the original archives and clones are vendored so the result is self-contained, reviewable and license-scannable; .git directories are dropped and symlinks are followed for the same reason (subproject fetchers symlink into the scratch dir, which does not exist on an offline builder).
Tested on Fedora 44: `getdeps.py --allow-system-packages vendor --no-tests --output-dir /var/tmp/cachelib-vendor cachelib` produced eleven trees (about 330 MB, fbthrift being 216 MB of that) plus getdeps-vendor.txt, with no .git directories or symlinks left behind. Once the Fedora rpm mappings from facebook/CacheLib#488 land that drops to the seven projects Fedora does not package.
## getdeps: add --vendor-dir to build from vendored sources offline
`getdeps.py --vendor-dir DIR build project` takes every third-party dependency from `DIR/<project>`, as populated by `getdeps.py vendor`, instead of cloning or downloading it. The lookup sits in the manifest's fetcher selection after the system-package check, so a dependency that --allow-system-packages resolves to an installed package still wins, and anything not found in DIR raises an error naming the missing project rather than falling back to the network. That single check is the offline guarantee: no GitFetcher or ArchiveFetcher is ever constructed.
The vendored tree is wrapped in the existing LocalDirFetcher, the same mechanism --src-dir uses, so no fetcher code changes. Its hash is fixed and it always reports the sources as changed, which means repeated builds against a vendor dir reconfigure their dependencies each time; that is acceptable for the one-shot distro builds this is meant for.
--free-up-disk only removed build trees whose sources live under the scratch dir, a guard meant to protect a user's own --src-dir checkout. Vendored sources are ours to clean up after, so the guard now also accepts the vendor dir (stored realpath'd so the prefix comparison matches LocalDirFetcher's realpath'd source paths).
Known limitation: patchfiles are applied with `git apply` from the enclosing git top-level, so a vendor dir placed inside another git checkout would mis-apply patches for the (few) manifests that carry them. A distro build directory is not a git checkout, so this does not affect the intended use; making the patch step independent of the surrounding repository is left for a follow-up.
Tested on Fedora 44 (aarch64, 4 cores) on a tree that also carried facebook/CacheLib#488, https://github.com/facebook/CacheLib/issues/489 and https://github.com/facebook/CacheLib/issues/490, against the seven-project vendor dir produced by the previous commit (magic_enum, sparsemap, folly, fizz, wangle, mvfst, fbthrift; 276 MB):
```
unshare -rn python3 build/fbcode_builder/getdeps.py \
--allow-system-packages --vendor-dir /var/tmp/cachelib-vendor \
--scratch-path /var/tmp/getdeps-offline-scratch --num-jobs 2 \
--extra-cmake-defines '{"CMAKE_POLICY_VERSION_MINIMUM":"3.5"}' \
build --free-up-disk --no-tests --src-dir=. cachelib
```
`unshare -rn` puts the build in its own network namespace. Inspected from outside while it ran, the build process was in net:[4026532485] versus the shell's net:[4026531833]; `nsenter -n ip -brief link` inside it showed only `lo` DOWN, `getent hosts github.com` failed (exit 2), and `ss -tunap` listed no sockets. The build finished in 72 minutes with exit 0, all eight projects installed, no "Download with" or "Cloning" line in the log, and cachebench linking the system glog, liboqs, libaio and libnuma. Requesting a project that is neither vendored nor allowed from system packages fails with the new error, and `show-source-dir --recursive` resolves every vendored project into the vendor dir. The --free-up-disk fix was verified separately: an offline sparsemap build with the flag left no build tree behind, where the 72-minute run (made before the fix) had left 11 GB.
## getdeps: add tests for vendor and --vendor-dir
`getdeps/test/vendor_test.py`, in the style of the existing builder tests (MagicMock loader and build options, real `ManifestParser` objects), 5 tests: `vendor` copies a non-system dependency's tree, skips a dependency that resolves to a `SystemPackageFetcher`, skips the project itself, drops `.git`, materialises symlinks and writes `getdeps-vendor.txt`; `vendor` replaces a stale tree in the output dir; `--vendor-dir` resolves a download-URL manifest to a `LocalDirFetcher` on `<vendor-dir>/<name>`; a project missing from the vendor dir fails naming the project rather than constructing a network fetcher; and without `--vendor-dir` the normal `ArchiveFetcher` is still chosen. Run from `build/fbcode_builder` with `python3 -m unittest getdeps.test.vendor_test`.
Verified together with https://github.com/facebook/CacheLib/issues/488, https://github.com/facebook/CacheLib/issues/489 and https://github.com/facebook/CacheLib/issues/490 applied; each of those stands alone and none of them is required for this change to apply.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
X-link: https://github.com/facebook/CacheLib/pull/491
Reviewed By: aleivag, likewhatevs
Differential Revision: D120204081
Pulled By: michel-slm
fbshipit-source-id: f877310e93bcddbf6ff6f81f72e2738e23ecdfa5
Summary:
`FindGlog` creates the `glog::glog` imported target whether or not it found anything. When glog is missing the target is there all the same, pointing at `GLOG_LIBRARY-NOTFOUND`, so nothing goes wrong until the link step reports `ld: library 'GLOG_LIBRARY-NOTFOUND' not found`. The same block also runs `find_package(Gflags)` and `find_package(LibUnwind)` to fill in glog's own dependencies, two searches that have nothing to contribute when there is no glog to attach them to.
Guarding the block on `GLOG_FOUND`, the way `FindGflags` already guards its `gflags` target, makes a failed search report itself at configure time and leaves the namespace clear.
That second part matters to folly. A fetched glog declares `glog::glog` as an `ALIAS`, and CMake rejects an alias whose name is already taken, so the previous diff had to run the module's library search by hand and pick between find and fetch before `FindGlog` could get to it. That workaround goes away and the block reads as `find_package` plus a fallback again, like every other dependency in the file.
Reaching the module in the not-found case does leave two of its variables to account for on folly's side. `GLOG_LIBRARIES` is named in the `EXPORTED_DEPS` of 42 of folly's granular libraries, so the fetch branch points it at `glog::glog`. `GLOG_INCLUDE_DIR` only means anything when glog was found, so appending it and the `glog/export.h` check that goes with it move into that branch.
Reviewed By: bigfootjon
Differential Revision: D120177659
fbshipit-source-id: ee65caed9aa25b4517b8b952b826b451b462f96b
Summary:
Fedora is already detected by getdeps_platform.py and mapped to the rpm package manager, but `install-system-deps --distro` only accepted ubuntu and centos_stream, so the Fedora package list could not be produced from another host.
Also let the remaining cachelib deps that Fedora packages be satisfied from the system with --allow-system-packages instead of being built from source. libaio and numa already had rpm entries scoped to CentOS Stream (and Alma/Rocky for libaio); those selectors now include Fedora. The other two gain new [rpms.distro=fedora] sections. Package names and versions were checked against the Fedora 44 repositories:
| manifest | Fedora package | Fedora | manifest |
|-----------|------------------------|---------|----------|
| benchmark | google-benchmark-devel | 1.9.5 | 1.8.0 |
| libaio | libaio-devel | 0.3.111 | 0.3.113 |
| liboqs | liboqs-devel | 0.12.0 | 0.12.0 |
| numa | numactl-devel | 2.0.19 | 2.0.19 |
magic_enum is deliberately not mapped: Fedora ships 0.9.5, which installs `<magic_enum.hpp>` at the include root, while cachelib includes the `<magic_enum/magic_enum.hpp>` layout introduced in 0.9.6.
Tested on Fedora 44 (linux-fedora-44 host type): all changed manifests pass validate-manifest, `install-system-deps --recursive --dry-run --distro fedora cachelib` lists the four packages above, and a full `build --allow-system-packages cachelib` succeeds with fizz linking the system liboqs and cachebench linking the system libaio and libnuma.
X-link: https://github.com/facebook/CacheLib/pull/488
Reviewed By: likewhatevs
Differential Revision: D120159819
Pulled By: michel-slm
fbshipit-source-id: 8fcee0d3f90447209f259df7ba8a0a41aae4a18c
Summary:
cachelib's CMakeLists defaults BUILD_TESTS to ON and the manifest never overrode it, so `getdeps.py build --no-tests cachelib` still compiled every test binary. Wire the getdeps test context to BUILD_TESTS the same way the folly and fizz manifests do.
With tests off the cachelib build on Fedora 44 drops from 497 to 110 ninja steps, which matters for memory- and CPU-constrained distro builders.
Tested on Fedora 44 in a single `getdeps.py --allow-system-packages build --no-tests cachelib` run together with the Fedora rpm mappings from facebook/CacheLib#488 and the FindGlog config-mode fix; the resulting CMake cache has BUILD_TESTS:BOOL=OFF and the build succeeds.
X-link: https://github.com/facebook/CacheLib/pull/490
Reviewed By: likewhatevs
Differential Revision: D120177373
Pulled By: michel-slm
fbshipit-source-id: 8b079e9878b945f778c882f84dbb4433660b181d
Summary:
With fboss off it in the parent diff, no getdeps manifest depends on fatal, so the manifest builds a library nothing asks for. Delete it.
This does not affect the GitHub repo: `github.com/facebook/fatal` is exported by ShipIt from `configerator/source/opensource/shipit_config/facebook/fatal.cconf`, which is independent of this manifest, and the repo builds through its own scripts rather than getdeps. fatal itself stays in fbcode, where roughly 350 include sites across 59 projects still use it.
Reviewed By: bigfootjon
Differential Revision: D120061602
fbshipit-source-id: 1e06ba4212a17b98973941485e46150da74a6e6b
Summary:
Every OSS build of fboss fetches and installs fatal, and nothing in fboss includes it: there is no `#include <fatal/` anywhere under `fbcode/fboss`. The include path is a leftover from when Thrift's cpp2 reflection emitted fatal-based headers.
Drop the `find_path`/`include_directories` pair from the GitHub CMakeLists and remove `fatal` from the getdeps manifest.
The `reflection` entries still passed in `OPTIONS` to `add_fbthrift_cpp_library` are inert: the current Thrift compiler does not know that option and emits no reflection or fatal headers for it. Left alone here to keep this diff to the dependency.
Reviewed By: Mizuchi
Differential Revision: D120061693
fbshipit-source-id: 500c1de8e2db2fe9f8cac582621a7a8fe131df7e
Summary:
Two changes to the block that sets up tests.
**Rename `enable_tests` to `THRIFT_TESTS`, and declare it.** It was the only knob in this file never passed to `option()`, so despite being documented in the README it did not show up in `cmake -LH` or `ccmake`. The name was also the odd one out next to `THRIFT_RPC` and `THRIFT_BENCHMARKS`. Default stays `OFF`, matching the previous behaviour of an unset variable.
**Drop three dead lines from the block:**
- `include_directories(${GTEST_INCLUDE_DIRS} ${GMOCK_INCLUDE_DIRS})`. Both test directories link `GTest::gtest_main`, which carries its own include directories, so this is a directory-wide copy nobody reads.
- `find_package(GMock REQUIRED)`. Nothing in the CMake build mentions gmock outside this line; the tests use plain gtest.
- `find_package(PythonInterp REQUIRED)`. No `PYTHON_EXECUTABLE` reference exists anywhere in the CMake build, and CMake 4 reports the module as removed under CMP0148, so it buys a deprecation warning and nothing else.
`include(GoogleTest)` is added in their place, and the copy in `thrift/lib/cpp2/protocol/test` removed. It defines `gtest_discover_tests()`, which both test directories call. Today that works without it, because CMake's `FindGTest.cmake` ends with an unconditional `include(GoogleTest)` and nothing on our module path overrides it. That is a side effect rather than a contract: setting `CMAKE_FIND_PACKAGE_PREFER_CONFIG` bypasses the Find module and the command disappears, and only the cpp2 test directory includes the module itself.
Reviewed By: iahs
Differential Revision: D120047192
fbshipit-source-id: 80c94c36a0d4445c6929943656e9831db6a598c8
Summary:
Follows the parent diff, which turned `thriftpy` into `THRIFT_PY_DEPRECATED`.
The two options left in the old lowercase style get the same treatment:
| Before | After |
| --- | --- |
| `thriftpy3` | `THRIFT_PY3_DEPRECATED` |
| `thrift_python` | `THRIFT_PYTHON` |
Both are plain renames -- each was already an `option()` defaulting to `OFF`,
so unlike `thriftpy` there is no tri-state to unpick and no default to preserve
by hand. Every `THRIFT_`-prefixed option in the file is now uppercase.
`THRIFT_PYTHON` has one caller outside the repo's own CMake: the
`fbthrift-python` getdeps manifest sets it under `[cmake.defines.os=linux]`,
and that line is renamed with it. Missing it would not have failed the build --
the option would simply have stayed `OFF` and the Python bindings would have
gone quietly unbuilt. `THRIFT_PY3_DEPRECATED` has no such caller.
Of the two, only `THRIFT_PYTHON` is interpolated into
`FBThriftConfig.cmake.in`, as `FBThrift_python_FOUND`; there is no
`FBThrift_py3_FOUND`.
## Grouping
The five `option()` calls were declared in two places: the Python ones near the
top, before the PIC block that has to precede the fetch blocks, and
`THRIFT_RPC` and `THRIFT_BENCHMARKS` a hundred lines further down, after
`find_package(OpenSSL)`. Neither of the latter two depends on anything between,
so they move up and all five now sit together, ahead of the first thing that
reads one.
## Descriptions
The three Python options now describe themselves in one shape, naming each
runtime the way the docs do rather than by its path in this repo:
```
"Enable the deprecated thrift-py library; use thrift-python instead"
"Enable the deprecated thrift-py3 library; use thrift-python instead"
"Enable the thrift-python library and extensions (requires Cython)"
```
The old py text, "Install the thrift/lib/py library as an FB Python archive
manifest", described a mechanism rather than a choice and did not say the
runtime is deprecated -- the thing a reader of `cmake-gui` most needs to know.
The py3 text loses "requires Cython" in the trade for that deprecation notice.
`THRIFT_PYTHON` still carries it, and the requirement is enforced either way by
the `find_package(Cython 0.28 REQUIRED)` inside the `if (THRIFT_PY3_DEPRECATED)`
block, which fails with a clear message.
The "use thrift-python instead" in the deprecated descriptions, and the two
`# DEPRECATED:` comments in `thrift/lib`, name the runtime rather than the
option, so they use the hyphenated spelling and did not become `THRIFT_PYTHON`.
Reviewed By: createdbysk
Differential Revision: D119666961
fbshipit-source-id: da3b412274f4200dbfd0b8fc5ae2e540310c4bbe
Summary:
Follow-up to D119583892, which moved the last two manifests that used
`builder = boost` (`boost` and `boost-python`) onto cmake. Nothing selects the
b2 builder any more, so delete it:
- `class Boost` in `getdeps/builder.py`, which bootstrapped and ran `b2`.
- The `builder == "boost"` branch in `getdeps/manifest.py`.
- The `b2.args` and `bootstrap.args` manifest sections. Both fed only the b2
builder — `bootstrap.args` went to boost's `bootstrap.sh` / `bootstrap.bat` —
and no manifest declares either one now.
- `--extra-b2-args` in `getdeps/cli.py`, plus its `extra_b2_args` plumbing
through `ManifestParser.create_builder()`.
Two scripts pass `--extra-b2-args "cflags=-fPIC" --extra-b2-args
"cxxflags=-fPIC"` and would die with `unrecognized arguments` once the flag is
gone, so they drop it: `fbcode/ai_infra/rsc/build/airstore_client_cmake_build.sh`
and `fbcode/warm_storage/experimental/ws_airstore/tools/ws_airstore_client_cmake_build.sh`.
Both already pass `CMAKE_POSITION_INDEPENDENT_CODE=ON` in
`--extra-cmake-defines`, which is what the cmake boost build uses for PIC, so
their boost stays relocatable.
Not touched, but worth knowing:
- `fbcode/hphp/public_tld/build/oss_build.sh` splices `--user-config` and
`toolset=` into the boost manifest after a `[b2.args.os=linux]` line and
aborts with "boost manifest has no Linux b2 args section" when that line is
missing. D119583892 already removed the section, so that script needs a cmake
equivalent regardless of this diff.
- `fbcode/gen_ai/unified_parallelism/build/fbcode_builder/manifests/boost` still
says `builder = boost`, but it is a frozen 2024 snapshot of the manifests
directory that this getdeps never loads.
Reviewed By: bigfootjon
Differential Revision: D119706069
fbshipit-source-id: 4d01c8627d8385de2bee2e03ffc3923805011770
Summary:
Switch the `boost` and `boost-python` manifests from `builder = boost` (b2) to
`builder = cmake`, so the bespoke b2 support in getdeps can be deleted.
Boost ships a CMake superproject, but only in the GitHub release archives. The
`archives.boost.io` tarballs are packaged for b2 and carry no `CMakeLists.txt`
anywhere, so the download URLs move to
`github.com/boostorg/boost/releases/download/boost-1.83.0/` and `subdir` becomes
`boost-1.83.0`. Those archives lay headers out per library under
`libs/<lib>/include/boost/...`, so `patches/boost_1_83_0.patch` needs the longer
path. The patchfile also drops its two b2-only hunks (`tools/build/src/tools/msvc.jam`
MSVC 14.4 detection and `tools/build/src/tools/clang.jam` cross-compile flags),
keeping only the `strong_typedef.hpp` source fix.
`CMAKE_POSITION_INDEPENDENT_CODE=ON` replaces the `cxxflags=-fPIC` that the
deleted `[b2.args.os=linux]` block set; getdeps only turns PIC on by itself for
`--shared-lib` builds.
`fbcode/tools/lfs/.lfs-pointers` gains the new archives (already uploaded to the
manifold bucket with the `manifold` CLI). Without the pointers getdeps downloads
from GitHub and then fails mirroring the result: `lfs.py`'s single HTTP PUT caps
at 82,681,580 bytes and the tarball is 123,695,390.
Other behavior changes:
- Windows no longer gets a shared boost variant. b2 built `link=static` plus
`shared`; the cmake path builds static only.
- `BOOST_ENABLE_MPI=OFF` in both manifests. Boost.MPI needs an MPI toolchain our
build hosts do not have, and Boost.Graph_Parallel needs Boost.MPI.
- `BOOST_ENABLE_PYTHON` is off in `boost` and on in `boost-python`, matching the
split the two manifests already had.
- No `BOOST_INCLUDE_LIBRARIES` list: Boost's cmake build only installs the
libraries it is asked for, and consumers pull headers from all over Boost.
Follow-up (not in this diff): delete `class Boost` from `getdeps/builder.py`, the
`builder == "boost"` branch in `getdeps/manifest.py`, and `--extra-b2-args` in
`getdeps/cli.py`.
Reviewed By: genevievehelsel
Differential Revision: D119583892
fbshipit-source-id: be1ecaa7b5f556279f6c439c85568df6d61b37a9
Summary:
## What
Three changes, all consequences of `add_fbcode_builder` (D119431762) making
`fbcode/opensource/fbcode_builder/` part of the exported repository:
- **`packsim/BUCK`** — add `fbcode/opensource/fbcode_builder/manifests/packsim`
to `PACKSIM_OSS_CI_SRCS`, so a change to packsim's own getdeps manifest
schedules the export gate. The comment records why the *rest* of
`fbcode_builder/` is deliberately left out.
- **`scripts/oss_check.py`** — `check_placement` now builds the shipit path from
the repository root instead of the packsim root, so `classify` answers for
every mapped root rather than hard-erroring on all but one. It still refuses a
path outside the checkout, and marker linting stays packsim-scoped.
- **`opensource/fbcode_builder/manifests/packsim`** — reword the header comment,
which ships to GitHub and told the reader to run a `buck2` command against an
fbcode path.
## Why
221 of the exported repository's 647 files now come from
`fbcode/opensource/fbcode_builder/`, and every boundary packsim's tooling
assumes is still drawn at `fbcode/ai_simulation/packsim/`:
- `PACKSIM_OSS_CI_SRCS` matched nothing there, so a diff to packsim's getdeps
manifest — the file that declares the dependency set the open-source build
resolves — scheduled no packsim signal at all, on diff or on trunk. This is the
same gap `PacksimOssExportCheck` (D119234561) closed on the Configerator side,
reopened in fbsource.
- `oss_check classify` is the tool the `.llms` rules point at for "does this
ship?", and it answered `outside ai_simulation/packsim, so the export rules say
nothing about it` for files `materialize` writes into the export. Two halves of
one tool disagreeing about the export's boundary.
The rest of `fbcode_builder/` is left out on purpose, and the BUCK comment says
so: it belongs to the open_source team, and neither tier would earn its keep —
`--fast` replays `mappingTestCases` and lints packsim's markers, neither of which
reads fbcode_builder's contents, and the full check builds the internal Buck
graph, which fbcode_builder is not part of. Widening the trigger would bill
another team twenty minutes a diff for a signal that cannot see their change.
Marker linting is deliberately NOT widened: `getdeps/test/strip_marker_test.py`
carries `oss-disable` strings as test data, and flagging those would be a false
finding on another team's file. `marker_syntaxes_for` is unchanged and the
`check_placement` docstring records the asymmetry.
Differential Revision: D119582014
fbshipit-source-id: 29889b5f4cc1d2320132b6fb7546e9ca14b2290d
Summary:
X-link: https://github.com/facebookexperimental/moxygen/pull/229
The gperf manifest downloads its tarball from `ftpmirror.gnu.org`, which
redirects to a randomly chosen GNU mirror. Version 3.3 is recent enough that
many mirrors have not picked it up yet, so the download either 404s or returns
a stale file that fails the recorded checksum. Point the manifest at
`ftp.gnu.org`, the canonical host, which always has the release. The `sha256`
is unchanged since it is the same tarball.
Reviewed By: jbeshay
Differential Revision: D119370094
fbshipit-source-id: e29c50ca8d35f3acb4efbb3cc3150900fd100765
Summary:
The install RPATH was an absolute path baked in at configure time:
```cmake
set(CMAKE_INSTALL_RPATH "${CMAKE_INSTALL_PREFIX}/${LIB_INSTALL_DIR}")
set(CMAKE_INSTALL_RPATH_USE_LINK_PATH TRUE)
```
That makes the installation non-relocatable — move the prefix and installed
binaries still point at the old one — and it embedded every linked dependency
directory into every artifact.
Installed artifacts now find fbthrift's own shared libraries relative to
themselves, via two entries appended to `CMAKE_INSTALL_RPATH`:
`$ORIGIN` and `$ORIGIN/../<libdir>` on Linux, `loader_path` and
`loader_path/../<libdir>` on macOS. The hop is computed from
`CMAKE_INSTALL_BINDIR` and `CMAKE_INSTALL_LIBDIR` rather than hard-coding
`lib`, so it follows `lib64`. Windows is untouched; it has no RPATH.
Both entries go on every artifact rather than being split per target type.
Libraries resolve through the first and executables in bindir through the
second; the unused entry in each case is a directory that holds no libraries,
so it costs one empty search path and nothing else. That keeps this to a
single list at the top level instead of a helper plus a call next to every
installed executable.
Two details worth attention:
- The list is appended to, not assigned. fbcode_builder passes
`-DCMAKE_INSTALL_RPATH=<translated DYLD_LIBRARY_PATH>` on macOS, and the
old unconditional `set()` created a normal variable that shadowed it,
silently discarding what getdeps asked for. Appending honors it and adds
ours. Same for a user-supplied `-DCMAKE_INSTALL_RPATH=...`.
- `CMAKE_INSTALL_RPATH_USE_LINK_PATH TRUE` is removed from the CMake build
and set in the two getdeps manifests instead. getdeps installs each
dependency to its own prefix, so it genuinely needs those directories
recorded — but that is a getdeps requirement, not something every consumer
of the OSS build should inherit.
Reviewed By: iahs
Differential Revision: D119211405
fbshipit-source-id: 6c913dd62a36b5e0ffb4aa9e0768ac8aa4ec475e
Summary:
## What
Adds the open-source build for `github.com/facebook/ai_simulation`:
- **`fbcode/ai_simulation/CMakeLists.txt`** -- the whole build. 16 thrift
codegen steps, one static library over the 83 shipping non-test sources, the
`packsim` / `packsim-replay` / `coro_demo` binaries, and 163 test executables
(one per test source, matching the internal one-`.cpp`-per-`cpp_unittest`
shape). It is hand-written against the EXPORT's layout, not generated from
the Buck graph.
- **`opensource/fbcode_builder/manifests/packsim`** -- the getdeps manifest
declaring the dependency set.
- **`oss_check materialize <dir>`** -- writes the export to a directory laid out
as the GitHub repo is. Without it the CMake could not be run at all, let alone
tested; see *Why `materialize`* below.
- **`.llms/rules/oss-export-compliance.md`** -- documents both, and corrects the
now-false claim that the export has no build system.
Sources are globbed, not enumerated: what ships is the shipit config's decision,
and a list here could only ever disagree with it. The one thing that must be
enumerated is the IDL set, because each `.thrift` gets its own codegen step --
so the build fails at configure time, naming both lists, if
`PACKSIM_THRIFT_FILES` drifts from `packsim/if/*.thrift`.
**Companion Configerator diff: D119058158, which must land with this one.**
`fbcode/ai_simulation/CMakeLists.txt` does not ship without it -- the sibling
rule strips everything at that level that is not `packsim/`. Neither diff does
anything useful alone.
## Why
The export is 415 sources with no way to compile them. Every `BUCK`, `PACKAGE`
and `.bzl` file is removed by ShipIt's shared strip list, so an external user
who cloned the repo today could read packsim but not build it. This is the long
pole on the open-sourcing plan (P2493484069, item 2).
Shipping the Buck files instead is not an option and is not the goal: they load
`fbcode_macros//...` and `fbsource//tools/...`, which have no open-source
prelude equivalent.
### The dependency set is seven libraries, not four
The plan recorded "no includes outside packsim + folly/fmt/fbthrift/gtest".
Enumerating every angled include in the exported content, it is:
| Dependency | Distinct headers | Include sites |
|---|---|---|
| folly | 43 | 272 |
| gtest | 2 | 173 |
| fmt | 3 | 78 |
| gmock | 2 | 54 |
| fbthrift runtime | 3 | 34 |
| gflags | 1 | 9 |
| boost (header-only) | 2 | 2 |
| magic_enum | 1 | 1 |
| POSIX (`fcntl.h`, `unistd.h`, `sys/resource.h`) | 3 | 3 |
All seven are permissively licensed and compatible with the Apache-2.0 LICENSE
the repo ships. packsim declares no thrift services, so the manifest asks for
`fbthrift = !rpc, !benchmark` and the build links
`thriftprotocol`/`thriftmetadata`/`thrifttype` rather than `thriftcpp2` --
keeping fizz, wangle and mvfst out of the graph. Precedent:
`manifests/rebalancer`, which does the same for the same reason.
### Why `materialize`
`transform` and the new `materialize` apply different halves of the export.
`transform` rewrites marked lines in place and deliberately leaves paths alone,
because the internal Buck graph has to keep resolving -- that is what `check`
builds. The CMake build needs the other half too: the repository layout, where
packsim sits under `packsim/`, and the `cppIncludeMappings` rewrite that makes
the sources' `#include`s agree with it. Doing that in place would rewrite every
include in the working copy, so it happens out of tree.
Like everything else in `oss_check`, it reads the live config on every run and
records no copy of what ships. `_export_inputs` is driven by `pathMappings`
rather than by walking packsim, which is what lets it pick up a repo-level file
the root mapping also covers.
Differential Revision: D119058074
fbshipit-source-id: 570f98fdda30c39cf222a2b9342dcc876f6a08ef
Summary:
Context:
The BGP OSS manifest stopped stripping the complete BB platform directory so `PlatformConstant.h` could support the BB CMake build. With no narrower rule, any future file added to that directory would also ship to facebook/BGP.
Motivation:
The OSS export should include only the BB platform file required by `BgpBB.cmake`. A fail-closed rule prevents unrelated future files from entering the public repository automatically.
This diff:
- allows `PlatformConstant.h` under `cpp/common/platform/bb`
- strips every other current or future file in that directory
- keeps `manifests/bgp` aligned with the production `BGP.cconf` rule
Reviewed By: jaiharil
Differential Revision: D118808795
fbshipit-source-id: e965cae3294e6199cd7387d53edb8cf796835d85
Summary:
D117475730 added the `gcc12` getdeps dependency to support the CentOS Stream 9 Open/R build. Although its compiler overrides were correctly limited to CentOS Stream 9, the dependency itself was unconditional. Ubuntu 24.04 GitHub Actions therefore tried to resolve a `gcc12` manifest with no matching Ubuntu package and failed during `query-paths`, before compilation.
Scope `gcc12` to CentOS Stream 9 alongside the existing compiler definitions and regenerate the OpenR Linux workflow fixture. This preserves the intended CentOS GCC 12 toolchain from D117475730 while removing the invalid Ubuntu dependency.
Differential Revision: D118736179
fbshipit-source-id: af500b1e3f256223ff4c252c3b5839d4758499e1
Summary:
`oss-glean-linux-getdeps` and `oss-hsthrift-linux-getdeps` still fail
100% of the time, now at `make cabal-update`:
```
Writing default configuration to /var/twsvcscm/.cabal/config
<repo>/root.json does not have enough signatures signed with the appropriate keys
make: *** [Makefile:114: cabal-update] Error 1
```
Hackage rotated its TUF root keys. `manifests/cabal` pins
`cabal-install-3.6.2.0` (December 2021), whose bootstrap key set predates
the rotation, so it rejects the current `root.json` and cannot run
`cabal update` at all.
This is the same class of problem as the `index-state` divergence: the
internal getdeps build runs a different toolchain from GitHub CI, which
does `ghcup install cabal 3.10` (`.github/workflows/ci.yml`). Bump the
manifest to 3.10.3.0, which is what `3.10` currently resolves to
(3.10.1.0, 3.10.2.0 and 3.10.3.0 are the published 3.10.x releases).
Staying on the `x86_64-linux-deb10` variant, as before.
## Blast radius
`manifests/cabal` is a shared manifest, but in practice it has exactly two
consumers. Searching every file under `manifests/` for `cabal` as a
dependency entry (`^\s*cabal\s*$`) matches only:
* `manifests/glean`
* `manifests/hsthrift`
which are the two projects this is meant to fix. Both also pull `ghc`.
Worth being aware that getdeps has no per-consumer version override, so a
future third consumer needing a different cabal would be a genuine
conflict — there is no such consumer today.
Every other Haskell build path at Meta gets cabal from ghcup rather than
from this manifest, so none of them change:
| path | cabal source |
| --- | --- |
| Glean GitHub CI x86 (`ci.yml`) | `ghcup install cabal 3.10` |
| Glean GitHub CI aarch64 (`ci-aarch64.yml`) | `ghcup install cabal --set` |
| hsthrift GitHub CI (`ci-getdeps.yml`) | `ghcup install cabal --set` |
| `spidermate/repomate` Haskell docker builder | `ghcup install cabal 3.10.3.0` |
The last row is independent corroboration of the version chosen here — an
unrelated Meta system already pins exactly 3.10.3.0.
Not fixed here, but flagged: `glean/github/tld/Dockerfile` installs
`ghc-8.10.2` from apt on an old base image and then runs a bare
`cabal update`, with no ghcup. That is very likely broken by the same key
rotation. This is inferred from the toolchain age, not from a failing run.
## LFS
Also adds the `.lfs-pointers` entry for the new archive, which the
"Warn about missing LFS updates to getdeps manifest diffs" CI check asks
for on any getdeps manifest bump:
```
1d7a7131402295b01f25be5373fde095a404c45f9b5a5508fb7474bb0d3d057a 4920976 cabal-cabal-install-3.10.3.0-x86_64-linux-deb10.tar.xz
```
The name follows `ArchiveFetcher`'s convention of `<manifest>-<url
basename>` (`getdeps/fetcher.py:1062`), matching the existing 3.6.2.0 row,
which is left in place — the file already keeps superseded entries (for
example both cmake 3.20.2 and 3.20.4).
Without this the bump would work but would never use the cache:
`LFSCachingArchiveFetcher` calls `lfs_upload` after a public-URL fallback,
but `lfs.py upload` records the pointer in the *worker's* checkout of
`fbcode/tools/lfs/.lfs-pointers`, which is discarded at teardown. Every
build would re-fetch the archive from `downloads.haskell.org`, leaving CI
coupled to that host staying up and staying in the `lego` fwdproxy
allowlist.
Reviewed By: jjuliamolin
Differential Revision: D118644355
fbshipit-source-id: 2dae8a3fb050253d3fda97ce94aa147cc282455e
Summary: Add the socket-lb getdeps manifest, generated Linux GitHub Actions workflow, and automatic Sandcastle ownership. Map the workflow to repository-root `.github`, strip internal `BUCK` and `PACKAGE` metadata, and cover socket-lb dependency determination in getdeps tests. This diff is ordered before the socket-lb CMake change so that change automatically triggers the new job.
Reviewed By: avasylev
Differential Revision: D117744481
fbshipit-source-id: 64cc871fd8c9a6f2222058b6ae00b5a49da8b0c2
Summary:
Context:
The OSS CMake build now has independent `dc` and `bb` variants. Each variant uses a different FBOSS dependency profile, but neither path had end-to-end OSS tests or continuous build coverage.
Motivation:
CI should verify each variant against only its intended dependency surface so a DC dependency cannot accidentally leak into BB and a passing build of one daemon cannot hide a failure in the other.
This diff:
- adds GoogleTest and `BUILD_TESTING` manifest support
- registers BB-only CTests from `BgpBB.cmake` for platform constants, statistics, and the `bgp_bb --version` smoke test
- keeps the OSS source-boundary test in the common CMake graph so both variants run it
- validates required public BB files, license headers, and forbidden internal or copied-source paths across every CMake fragment
- extends the internal OSS build rule to pass `BGP_BUILD_VARIANT=dc|bb`, build named targets, and run CTest
- adds a DC job using the FBOSS `fsdb_client` profile and a BB job using `exported_libraries`
- converts the GitHub workflow to the same DC and BB matrix
The two jobs use separate build directories and dependency profiles; each configures only its selected daemon.
Reviewed By: xiangxu1121
Differential Revision: D117102134
fbshipit-source-id: 8a1374613c55f9f8fe71d3f382361f27ec6f6c9a
Summary:
Context:
The BB service, health, peer, and FIB implementations live under internal-only paths even though their dependencies are available through the OSS build surface.
Motivation:
The OSS build must own and compile the BB runtime libraries before a standalone daemon can be added. Moving the sources with their build ownership keeps every exported source represented in both Buck and CMake.
This diff:
- moves `BgpServiceBB`, `HealthValidatorBB`, `PeerManagerBB`, and `FibEbb` into OSS-visible paths with Apache license headers
- retains the existing internal Buck target names as compatibility aliases
- updates internal callers and tests to use the shared paths and targets
- adds CMake targets for the BB stats, netlink, FIB, RIB, peer, health, and service libraries
- installs the newly exported BB headers and libraries
- updates ShipIt and CMake-source validation for the exported sources
- preserves the existing Open/R coroutine client calls
- keeps runtime logic unchanged; source edits are limited to relocation-related license and include path updates
This diff compiles the BB libraries but does not add a BB executable or profile switch.
Reviewed By: xiangxu1121
Differential Revision: D118168786
fbshipit-source-id: 56c88ffa6e2e420d565538df8b3afdea90ccfb29
Summary:
Context:
D117100427 adds a supported Open/R `exported_libraries` profile containing the headers, generated Thrift libraries, and component libraries used by BGP++. The BGP OSS build still generates a copied Open/R Thrift surface and compiles copied utility sources from `public_tld/openr`.
Motivation:
BGP++ should consume Open/R through its supported package boundary. Removing the copied source slice prevents the BGP tree from drifting from Open/R and makes the dependency explicit in getdeps.
This diff:
- adds the Open/R `exported_libraries` feature to the BGP manifest
- resolves installed Open/R headers, component libraries, and generated Thrift libraries from one Open/R installation
- constrains every Open/R archive lookup to the library directory paired with the discovered Open/R headers, preventing collisions with generic Thrift archive names from other dependencies
- removes the local Open/R Thrift generation and copied utility targets
- deletes the copied Network, Platform, NetworkUtil, and SystemMetrics sources
- stages the current Open/R checkout in the internal OSS build
- makes stable-commit and dependency-cache guards inspect the complete local stack
The existing BGP source and coroutine API calls remain unchanged. The DC OSS binary remains the only CMake executable in this diff.
Reviewed By: xiangxu1121
Differential Revision: D118168784
fbshipit-source-id: 3074c918dda7b98a0d06776032597be1ebb551d3
Summary:
Context:
BGP++ consumes four Open/R component libraries and seven generated Thrift libraries. The Open/R getdeps dependency currently builds and installs the full daemon, tools, and test graph.
Motivation:
An explicit `exported_libraries` profile gives downstream projects a supported narrow build surface while preserving the existing full Open/R build and install behavior.
This diff:
- adds `OPENR_BUILD_PROFILE=exported_libraries` and documents selection with `!default, !full, exported_libraries`
- follows the established FBOSS profile pattern for feature selection, environment override, install interception, deferred target pruning, and an aggregate exported-libraries target
- documents each component and generated Thrift library installed by the profile
- keeps one `OPENR_EXPORTED_TARGETS` list for build pruning, aggregate-target dependencies, and installation
- builds the selected targets and their transitive dependencies without the Open/R daemon, tools, tests, or unrelated libraries
- builds exported static archives as position-independent code so downstream PIE executables and shared libraries can link them
- installs the selected libraries and their generated/public headers only for `exported_libraries`
- keeps the default `full` profile on C++20 with its existing install behavior
The optional fbthrift benchmark-pruning change was removed so this diff does not alter the shared fb303 dependency profile.
Reviewed By: xiangxu1121
Differential Revision: D117100427
fbshipit-source-id: 415e9b0cd23d0155682716be4d2ed38078ccb8d5
Summary:
Context:
BGP++ uses address conversion helpers from fboss/agent/AddressUtil.h, which couples otherwise portable code to the FBOSS agent target.
Motivation:
common/network is shared fbsource source. The OSS ShipIt mapping copies only AddressUtil.h and Address.thrift into the BGP checkout, so the OSS build remains self-contained.
This diff:
- switches BGP++ address conversion users to common/network/AddressUtil.h
- exports only AddressUtil.h and its Address.thrift schema into the OSS common/ tree
- keeps the OSS build self-contained without adding an FBOSS package dependency
Reviewed By: xiangxu1121
Differential Revision: D117097866
fbshipit-source-id: b3abdc2e91a546c4b5d8d41ba1de950f95c8b4d7
