Summary:
Same attribution for HTTP/3, and the largest gap of the three.
`HQFramedCodec::checkConnectionError` built every frame-layer `HTTPException`
from the hardcoded literal `Connection error`, so `HQControlCodec`'s five
indistinguishable `HTTP_FRAME_UNEXPECTED` returns and `HQFramer`'s message-less
returns all collapsed into one string. Only the numeric HTTP/3 code
differentiated them.
Connection error [File]=HQControlCodec.cpp
[Context]=settings-not-first-frame
The frame loop lives in the base class while the checks live in the derived
codecs, so `setParseErrorContext` takes the caller's file and the origin points
at the real rejecting file rather than the base. The base clears the context
before `checkFrameAllowed` and falls back to a generic reason only when the
derived codec named none, so a specific attribution is never overwritten.
Instrumented `HQFramedCodec`, `HQControlCodec` and `HQStreamCodec`.
Reviewed By: afrind
Differential Revision: D114943895
fbshipit-source-id: 47123a6e08d1ecca411f505f500f3bee3f285680
Summary:
`mcrouter`'s buck targets, sources, and CMake `find_package` list never reference `mvfst`. The single hit is a test string literal (`set_umember3() = "mvfst"`) exercising serialization round-trips — pure test data, no headers used, no libraries linked, and it would pass identically with any other string.
So the manifest's direct `mvfst` entry is dropped. Note: `mvfst` stays in the resolved closure transitively via `fbthrift`; this changes declared direct deps only, not what gets built.
Reviewed By: bigfootjon
Differential Revision: D121020246
fbshipit-source-id: d9fe8f7934af7cdcc2d8be271c4151f246ba75c0
The internal and external repositories are out of sync. This Pull Request attempts to brings them back in sync by patching the GitHub repository. Please carefully review this patch. You must disable ShipIt for your project in order to merge this pull request. DO NOT IMPORT this pull request. Instead, merge it directly on GitHub using the MERGE BUTTON. Re-enable ShipIt after merging.
fbshipit-source-id: e24e32453662b79b80308343ceefcb628225c94c
Co-authored-by: Facebook GitHub Bot <facebook-github-bot@users.noreply.github.com>
Summary:
Bump the OSS/getdeps default SAI version to 1.19.0 (libsai manifest, CMakeLists, run-getdeps.py, build-helper.py, fboss_oss_build.bzl, container docs).
Sites updated:
- opensource/fbcode_builder/manifests/libsai: url, sha256, subdir
- fboss/github/CMakeLists.txt: default SAI_VERSION + status message
- fboss/oss/scripts/run-getdeps.py: new SAI_VERSION_SHAS entry
- fboss/oss/scripts/build-helper.py: url + sha tables, choices, const/default, help
- fboss/facebook/oss_ci/fboss_oss_build.bzl: DEFAULT_SAI_VERSION + its docstring
- fboss/github/docs/static/code_snips/build_helper.sh: default text + supported list
The v1.19.0 tarball sha256 (199d2cd32408b6470ce0c43beee660cc468e29e72921f1e93e64d4a8eb80246a)
was obtained by downloading the release tarball and hashing it; it is used
identically in all three places that record it.
Building_FBOSS_on_containers.md carries no SAI version reference, so it needed
no change despite being listed in the integration runbook.
Reviewed By: ashutoshgrewal, shri-khare
Differential Revision: D120953509
fbshipit-source-id: 01072090c5844062d31f4ec2ab4aff6ed421d9b9
Summary:
Same attribution for HTTP/2. `checkConnectionError` reported every framer
failure as the single literal `Connection error`, so roughly thirty distinct
rejection paths in `HTTP2Framer` were indistinguishable in `debug_data`.
Frame parsing now records a reason via `setParseErrorContext`:
GOAWAY error: excessively large frame len=32768
[File]=HTTP2Codec.cpp [Context]=frame-exceeds-max-size
Reviewed By: afrind
Differential Revision: D114943825
fbshipit-source-id: e2e0ae29fed562b96e8fabaeb38641eef558cc62
Summary: Change `HTTPCoroSession` so it only enables push if that was specified in the construction settings.
Reviewed By: lnicco, hanidamlaj
Differential Revision: D120591826
fbshipit-source-id: 911e12cce94640e14059f64d62599f4d3b4bd5da
Summary:
## What
Adds a `[github.actions]` section to `opensource/fbcode_builder/manifests/packsim`
setting `timeout_minutes = 180`, and regenerates
`ai_simulation/.github/workflows/getdeps_linux.yml` from it.
## Why
The job had 60 minutes, which is the workflow generator's fallback
(`workflow_generator.py`: `timeout_minutes = gh("timeout_minutes") or "60"`) and
not a figure anyone picked for this project. One run has to build boost,
openssl, folly and fbthrift from source, then compile 187 test executables, then
run them -- and each test process defaults to 4 PDES worker threads in the
open-source build while ctest is already running jobs in parallel, so the test
phase oversubscribes a standard two-core runner several times over.
180 is a ceiling, not an estimate, and the manifest comment says so. Nothing is
known about the real wall clock because the workflow has never executed: nothing
has been pushed to the repository yet, and it only triggers on push and pull
request against `main`. Its first run will be its first measurement. A job that
is killed at 60 minutes reports as a red badge on the repository's front page
and tells you nothing about why; one that finishes in 70 tells you the number to
set. Lower it once a green run has reported one.
Deliberately not changed: the runner size. `--cpu-cores` would also help the
oversubscription, but that is a standing cost decision rather than a one-off
ceiling, and it belongs with whoever owns the project's CI budget.
## Why the manifest and not the workflow
The workflow file carries an `generated by getdeps.py` header and is rebuilt by
`opensource/fbcode_builder/getdeps/facebook/update-all-github-actions.sh`, which
already has an entry for packsim. Editing the YAML directly would be reverted by
the next sweep; `[github.actions]` keys in the manifest are the supported way to
change it, and they survive.
Differential Revision: D120642627
fbshipit-source-id: 4ea8d4a5bcc8533e6eec0cfb41dde63871c72f9b
Summary:
a test was recently added, this simply refactors it to use the test fixture to simplify code
___
Differential Revision: D120731929
fbshipit-source-id: 5f0910f7ab6ee9d32168a3b3b9c371c7754f3caf
Summary:
Revproxy codec errors reach the `proxygen_codec_errors` Scuba dataset via
`CodecErrorFilter`, whose `debug_data` column is `HTTPException::what()`.
For HTTP/1, the generic `http_parser` callback failure does not identify which
branch rejected the message.
Record a static reason for ingress parser callback failures and append it to
the parser error. Header-value failures also use
`CodecUtil::validateHeaderValueDetail` from the parent diff to include the
header name and rejected validation rule.
Reviewed By: hanidamlaj
Differential Revision: D120020224
fbshipit-source-id: 0664905ae48263552cde1ed33ddadc29463287d2
Summary:
Enable getdeps-based builds on every Enterprise Linux distro (RHEL, AlmaLinux, Rocky Linux, CentOS Stream) by selecting EL-specific manifest sections with distro_family instead of individual distro names.
The manifests that mean "any Enterprise Linux 9" (or 8) spell it as distro=centos_stream, because CentOS Stream is what CI runs. That never matched AlmaLinux or Rocky, and after the previous change gave RHEL its own distro name it would not match RHEL either. The per-distro enumeration is the bug class itself: each file was incomplete in a different way (libaio hand-listed alma+rocky but missed RHEL; everything else listed only centos_stream), so the fix has to apply everywhere the pattern occurs -- splitting it per project would just re-create the drift. Every project touched here builds against the same Folly-stack toolchain (gcc toolsets via the shared gcc12/gcc14 manifests, clang19, python3.12 for sapling), so distro coverage has to move together. The most consequential case is cachelib's gcc-toolset-14: without it an EL 9 build uses the default GCC 11 and fails to compile C++20 code although the toolset is right there in the repositories.
Rewrite those sections with the distro_family variable introduced in the previous change, preserving each selector's meaning while extending it to the whole family:
```
- all(distro=centos_stream,distro_vers=9) -> all(distro_family=rhel,distro_vers=9)
cachelib, gcc12, gcc14, openr, python, python-setuptools,
rebalancer, rebalancer-explorer
- all(distro=centos_stream,distro_vers=8) -> all(distro_family=rhel,distro_vers=8)
boost, boost-python
- not(all(any(distro=centos_stream,distro=centos),distro_vers=8))
-> not(all(distro_family=rhel,distro_vers=8))
libmnl
- not(any(all(distro=centos,distro_vers=8),all(distro=centos_stream,distro_vers=9)))
-> not(all(distro_family=rhel,any(distro_vers=8,distro_vers=9)))
lz4, lz4-python
- rpms.any(distro=centos_stream,...,distro=fedora) -> rpms
libaio, numa (every rpm distro ships libaio-devel and numactl-devel)
- rpms.distro=centos_stream -> rpms.distro_family=rhel
libaio-python
```
"centos" was the old CentOS Linux 8 name and folds into the family. The wider exclusions this produces (e.g. lz4-static now also excluded on centos_stream 8 and centos 9) only touch EOL or practically unused combos. No manifest still selects on an individual EL distro name. libaio-devel and numactl-devel are confirmed present on RHEL, CentOS and Fedora, the full rpm universe getdeps targets.
This must land after the distro_family change: the expression parser rejects unknown selector variables, so on a getdeps without it these manifests fail to load.
Tested with both changes applied: every manifest passes validate-manifest, and `install-system-deps --recursive --dry-run --os-type linux --distro rhel --distro-version 9 cachelib` now yields the same package set as --distro centos_stream, including gcc-toolset-14, libaio-devel and numactl-devel, where before it lacked all three. The openr, rebalancer and rebalancer-explorer hunks are the same mechanical rewrite but those projects were not built here.
**Depends on https://github.com/facebook/CacheLib/issues/495** (`distro_family`): must land after it, since the expression parser rejects unknown selector variables. Submitted separately so each can be reviewed on its own; the two are otherwise independent of https://github.com/facebook/CacheLib/issues/494.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
X-link: https://github.com/facebook/CacheLib/pull/496
Reviewed By: likewhatevs
Differential Revision: D120501944
Pulled By: michel-slm
fbshipit-source-id: fdfc8b65770736ed190223df5e4dc2d0b2b417ba
Summary:
# Context
`CoroSSLTransport::writeImpl` assumes a successful `SSL_write` consumes the entire input buffer. `SSL_MODE_ENABLE_PARTIAL_WRITE` permits positive partial returns, so writes larger than one TLS record can hit a fatal `XCHECK`.
# Mitigation
Advance the input range after each successful write and continue until all bytes are consumed. Preserve the original size for `WriteInfo` and add a regression covering a 64 KiB write with partial-write mode enabled.
Reviewed By: hanidamlaj, sandarsh
Differential Revision: D120634805
fbshipit-source-id: 3d0daf65f924c26d3c388d68503471c5105c67fe
Summary:
The goal here is just safety, when the WriteHandle is done we should unconditionally erase the handle from WritableStreams in ::finish
This is similar to what we currently do with connFcBlockedStreams & finOnlyStreams
Reviewed By: SUJP123
Differential Revision: D120439884
fbshipit-source-id: b29723ff99446789152037b93ec1cb3cbea3fcc5
Summary: Export public build scaffolding at the repository root and place the implementation under `socket-lb/`, matching Katran and other Meta OSS projects. Keep the top-level CMake and build scripts usable from both the internal and exported layouts, and update public build instructions accordingly.
Reviewed By: p-shah256
Differential Revision: D118875318
fbshipit-source-id: 996c0735d98fa093aa01bf08fce590e9e072d8f5
Summary:
Fedora retired liboqs in Fedora 45 ("Obsoleted by openssl 3.5": OpenSSL 3.5 ships ML-KEM natively), so the `[rpms.distro=fedora]` mapping added in facebook/CacheLib#488 only resolves on Fedora 44 and older. On Rawhide a build with --allow-system-packages that generates its BuildRequires from the manifests fails at dependency resolution with "No match for argument: liboqs-devel". fizz still hard-depends on liboqs, so let getdeps build it from source (or the vendor tree) on Fedora as it does everywhere else.
Found while building the Fedora cachelib package on Rawhide with BuildRequires generated from the manifests; the mapping was added in https://github.com/facebook/CacheLib/issues/488 and only checked against Fedora 44 at the time. CentOS Stream 10 does ship liboqs-devel, so a `distro_family=rhel` mapping for EL 10 could be considered separately once https://github.com/facebook/CacheLib/issues/495 lands.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
X-link: https://github.com/facebook/CacheLib/pull/497
Reviewed By: likewhatevs
Differential Revision: D120517415
Pulled By: michel-slm
fbshipit-source-id: 1386a4418536d59a36feffb24f85f48cf19f11f3
Summary:
get_linux_type() derives the distro name from NAME in /etc/os-release, which on Red Hat Enterprise Linux yields "red_hat_enterprise". Nothing knows that name: HostType.get_package_manager() returns None for it, so --allow-system-packages silently satisfies nothing and every dependency is built from source, and the manifests with distro=centos_stream selectors do not apply either. RHEL's VERSION_ID also carries a minor ("9.8"), as do AlmaLinux's and Rocky's, so even a matching distro would miss the distro_vers=9 selectors written for CentOS Stream.
This matters for EPEL: Fedora's Koji builds EPEL 9 packages in a buildroot with redhat-release (real RHEL), while EPEL 10 buildroots and the local mock configs use centos-stream-release. A getdeps-based EPEL 9 build therefore behaves differently in Koji than in local mock. Nobody noticed because getdeps has not been used for distro packaging before and RHEL is rare among people building from source.
Rather than pretend RHEL is CentOS Stream, model what the manifests actually mean. Following Chef's platform_family, introduce a distro_family alongside distro: "rhel" for rhel, centos, centos_stream, alma and rocky; "fedora"; "debian" for debian, ubuntu, pop!_os and mint; "arch". Fedora is deliberately its own family, unlike Ansible's os_family which folds it into RedHat: its base repositories carry many dependencies that the EL family only has in EPEL or not at all (glog, gflags, fmt, fast_float, googletest, benchmark, liboqs, jq), zlib is zlib-ng-compat there, and CentOS Stream 9 needs gcc-toolset; nineteen manifest sections already select on exactly that difference.
Concretely: - RHEL reports distro "rhel", the same short name as its os-release ID; - within the rhel family distro_vers is the major version only, so RHEL 9.8, Alma 9.6 and Rocky 9.6 all match distro_vers=9 as CentOS Stream 9 does; Fedora and Ubuntu keep their full versions; - distro_family is available to manifest selectors and drives get_package_manager(), which now returns rpm for the whole rhel family; - `install-system-deps --distro` accepts rhel; - os-release parsing moves into parse_os_release() so it can be unit tested.
Existing selectors keep their meaning: distro=centos_stream still matches only CentOS Stream. A follow-up will switch the sections that mean "any EL 9" to distro_family=rhel so that RHEL, Alma and Rocky pick up gcc-toolset, libaio and numactl too. Until then a RHEL host already gains every plain [rpms] mapping and the not(distro=fedora) ones.
Tested: `install-system-deps --recursive --dry-run --os-type linux --distro rhel --distro-version 9 cachelib` now produces a dnf command (previously "I don't know how to install any packages on this system"), and unit tests cover the os-release parsing for Fedora, CentOS Stream, RHEL, Alma, Rocky and Ubuntu plus the family and package-manager mapping. Verified against synthetic os-release contents, not a live RHEL host.
A follow-up PR will switch the manifest selectors that mean "any EL 9" (gcc-toolset, libaio, numactl and friends) to `distro_family=rhel`; it must land after this one because the expression parser rejects unknown selector variables.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
X-link: https://github.com/facebook/CacheLib/pull/495
Reviewed By: likewhatevs
Differential Revision: D120501703
Pulled By: michel-slm
fbshipit-source-id: f24b01e46207a08ca09b9f09500ba0fb78a502fa
Summary:
Fedora and EPEL package sparse-map as tsl-sparse-map-devel: 0.7.0 on F45 and Rawhide (stable since 2026-09-17), 0.6.2 on F44 and EPEL 9, and an EPEL 10 build is under way. With --allow-system-packages this takes sparsemap out of the set of projects built from source, leaving only the Meta stack (folly, fizz, wangle, mvfst, fbthrift) for a vendored distro build. A plain `[rpms]` section is used, as for libsodium, xxhash and snappy: where the package is not installed (CentOS Stream without EPEL, say) getdeps simply falls back to building sparsemap itself.
The F44 version is older than the 0.7.0 the manifest pins, so the difference was checked rather than assumed: the public API of sparse_map and sparse_set is identical between 0.6.2 and 0.7.0 down to template defaults, the tsl::detail_sparse_hash::sparse_array template cachelib reaches into has the same parameters, and the serialization format is unchanged. 0.7.0's only compatibility change is requiring C++17, irrelevant to cachelib. cachelib locates the headers by path (cmake/FindSparsemap.cmake: find_path tsl/sparse_map.h), which the package satisfies at /usr/include/tsl.
Tested on Fedora 44 with tsl-sparse-map-devel 0.6.2 installed: `getdeps.py --allow-system-packages build --no-tests --src-dir=. cachelib` no longer fetches or builds sparsemap (no Assessing/Building step for it in the log), cachelib configures with SPARSEMAP_INCLUDE_DIRS=/usr/include, SparseMapIndex.cpp compiles against the system header with -Werror on, cachebench links, and the build exits 0 after 58 minutes at two jobs.
Manifest-only follow-up to https://github.com/facebook/CacheLib/issues/488 and https://github.com/facebook/CacheLib/issues/493. RHEL hosts only benefit once https://github.com/facebook/CacheLib/issues/495 (distro_family) lands, since getdeps does not currently recognise RHEL as an rpm distro; the two are independent otherwise.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
X-link: https://github.com/facebook/CacheLib/pull/494
Reviewed By: likewhatevs
Differential Revision: D120501594
Pulled By: michel-slm
fbshipit-source-id: db5b7ad6514b56984aa3d92a24dbde1614ff1ed4
Summary: Convert os.path joins, basenames, dirnames, and exists/isfile/isdir/islink checks in the getdeps test suite to pathlib with os.fspath at the str boundaries, matching slices 1-3b. Deliberately kept: os.path.abspath (no exact Path equivalent; still used inside Path() for the fixtures/manifests dirs), os.path.realpath replaced only where resolve() is exactly equivalent, os.unlink/symlink/chmod/listdir (not os.path), and vendor_test.py (owned by unlanded D120204081). Verified: arc lint clean, pyre clean, buck test 113/113 pass.
Reviewed By: vitaut
Differential Revision: D120383078
fbshipit-source-id: 94b41927b849bd1667793648f5351476fa49fa98
Summary: Convert remaining os.path joins, basenames, dirnames, and exists/isfile checks to pathlib with os.fspath at the str-contract boundaries, matching slices 1-3a. Deliberately kept: os.path.normcase/relpath (no Path equivalent), os.fsdecode, os.path.sep, listdir/lstat traversal, os.walk in include_rewriter (rglob would change prune semantics), the abspath site in workflow_generator, and all test files (no mock seams affected; vendor_test stays owned by D120204081). Verified: arc lint clean, pyre clean, buck test 113/113 pass.
Reviewed By: bigfootjon
Differential Revision: D120383077
fbshipit-source-id: 37741cd8a8a042e40589b2925b08885f5bca8e7c
Summary: fetcher.py path manipulation now uses pathlib (joins, exists, makedirs, archive validation via resolve()+relative_to instead of a raw startswith prefix check). get_src_dir() stays -> str and constructor path fields stay str (builders/loaders consume them as str; retyped in the core slice). ShipitPathMap is deliberately untouched: its posix-matching semantics are load-bearing per the comment on add_mapping. retry_test now neutralizes Path.mkdir instead of os.makedirs (the seam _download_dir uses). New LocalDirFetcher/copy_if_different Path-input tests.
Reviewed By: likewhatevs
Differential Revision: D120383075
fbshipit-source-id: 8908611c0f79d853b09dcc86f9da09e4782e389b
Summary: Adds lint_cmake_floors.py, an arc lint wrapper holding the single cmake_minimum_required floor (3.14, from folly) for the folly-family OSS CMake projects, registered in fbsource-lint-engine.toml as OSSCMAKEFLOOR. It checks 13 effective roots (both sides of fbcode<->xplat mirrors, so skew is caught too) with at-or-above semantics so fbthrift 3.24 and mcrouter 3.16 stay legal, and offers the bump as an arc lint -a autofix. Any future dip below the floor fails lint on the offending diff instead of shipping a deprecation warning (or worse) to OSS consumers. Related: facebook/folly#2703.
Reviewed By: likewhatevs
Differential Revision: D120432856
fbshipit-source-id: 74895b760e4e73238ae0321f6e7eb5f966e39215
Summary: CMake 4.x removed compatibility with minimums below 3.5 and deprecates minimums below 3.10 ahead of removing that too. folly already requires 3.14, which is the effective floor for every folly-dependent OSS project: declaring lower no longer excludes any supported consumer (lowest CMake in any supported config is 3.22) and only buys deprecation warnings today plus hard configure failures once the next removal lands. Aligned here: fizz 3.5, wangle 3.5, mvfst 3.10, proxygen 3.10, cachelib 3.12, fb303 3.8. Left alone above the floor: fbthrift 3.24, mcrouter 3.16; folly itself already 3.14. Related: facebook/folly#2703.
Reviewed By: likewhatevs
Differential Revision: D120432855
fbshipit-source-id: 0734e2b4e4e0c8866de52bc8657d4bf0d81e7c57
Summary:
job_weight_mib is the per-compiler memory estimate getdeps divides available RAM by to pick the build concurrency, and the base of the RLIMIT_AS safety net it puts on each build process. cachelib and fbthrift declared 2048 MiB, which two measured builds show is low.
Peak resident memory of every cc1plus process was sampled every two seconds during full builds on Fedora 44 (aarch64, gcc 15), once at four jobs and once at two jobs with tests disabled; 1182 compiler processes were observed across the fbthrift and cachelib phases of the second run:
| percentile | peak RSS |
|------------|----------|
| p50 | 489 MiB |
| p90 | 1321 MiB |
| p95 | 1626 MiB |
| p99 | 2580 MiB |
| max | 4027 MiB |
24 translation units exceeded 2048 MiB and 4 exceeded 3072 MiB. The heaviest were cachebench's Stressor.cpp (4.0 GiB) and fbthrift's ThriftServer.cpp (3.8 GiB), ThriftQuicServer.cpp (3.1 GiB), ThriftRocketServerHandler.cpp and Cpp2Connection.cpp (2.8 GiB each).
3072 MiB matches what proxygen, fboss, openr, bgp and moxygen already declare and keeps a builder sized for N jobs from over-committing when several of the heavy units land together, which on a machine whose scratch dir sits on a tmpfs was enough to push it into swap. The weight is a typical-case budget rather than a ceiling, so the remaining 4 GiB outliers are left to the 10x RLIMIT_AS margin. fbthrift-python builds the same sources and moves with fbthrift.
Follow-up to https://github.com/facebook/CacheLib/issues/488/#490/https://github.com/facebook/CacheLib/issues/491, measured on the same host; manifest-only change.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
X-link: https://github.com/facebook/CacheLib/pull/492
Reviewed By: likewhatevs
Differential Revision: D120414253
Pulled By: michel-slm
fbshipit-source-id: f93eab02ecb3d3aa6a5b4e1a45a31cd7b376c366
Summary:
sparse-map 0.6.2 declares cmake_minimum_required(VERSION 3.1). CMake 4 removed compatibility with anything below 3.5, so on hosts whose system CMake is 4.x (Fedora 44, Homebrew, the GitHub macOS runners, and soon Ubuntu 26.04 and Debian 14) the sparsemap step fails to configure:
```
CMake Error at CMakeLists.txt:1 (cmake_minimum_required):
Compatibility with CMake < 3.5 has been removed from CMake.
```
Upstream fixed this in 0.7.0 (released 2025-11-02), which requires CMake 3.10, so take the release rather than carry a patch. 0.7.0 also moves to C++17, fixes iterator invalidation on a non-inserting insert and the serialization of small hashes, and adds a no-exceptions mode; the headers install to the same include/tsl layout. cachelib is the only manifest that depends on sparsemap, and the one internal type it reaches into, tsl::detail_sparse_hash::sparse_array, keeps the same template parameters.
Tested on Fedora 44 with system CMake 4.3.0 and gcc 15, no CMAKE_POLICY_VERSION_MINIMUM anywhere on the command line: `getdeps.py --allow-system-packages build --no-tests --src-dir=. cachelib` configured and installed sparsemap 0.7.0 with no CMake error (the only CMake output of note being the < 3.10 deprecation warnings from fizz and wangle, tracked in facebook/folly#2703), compiled SparseMapIndex.cpp against the new headers with -Werror on, and linked cachebench; 51 minutes at two jobs.
Manifest-only change; follow-up to https://github.com/facebook/CacheLib/issues/488/#490/https://github.com/facebook/CacheLib/issues/491. The CMake version matrix behind the platform list is in facebook/folly#2703.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
X-link: https://github.com/facebook/CacheLib/pull/493
Reviewed By: bigfootjon
Differential Revision: D120425085
Pulled By: michel-slm
fbshipit-source-id: 9802415458f4d6ba20fab21630ec9047d7ba45d5
Summary:
https://fb.workplace.com/groups/proxygendevelopers/posts/3993434290963932
A handler that sets its own Content-Length and then sends a complete response silently loses compression due to duplicate Content-Length.
Full chain of events:
(1) The handler sets Content-Length on the response.
(2) On the non-chunked path ResponseBuilder::send() appends a second one instead of replacing it, so the message now carries two values.
https://www.internalfb.com/code/fbsource/fbcode/proxygen/httpserver/ResponseBuilder.h?lines=161
(3) HTTPHeaders::getSingleOrEmpty returns an empty string for a multi-valued header, so isMinimumCompressibleSize reads the length as 0.
https://www.internalfb.com/code/fbsource/fbcode/proxygen/lib/utils/CompressionFilterUtils.h?lines=96-106
(4) The size test therefore fails and shouldCompress returns false, whatever the real body size.
https://www.internalfb.com/code/fbsource/fbcode/proxygen/lib/utils/CompressionFilterUtils.h?lines=91-92
(5) The response goes out uncompressed. The codec collapses the duplicate to a single value on the wire, so the response looks correct to the client and nothing is logged.
The chunked path never hits this, because ResponseBuilder sets no Content-Length there and the getIsChunked() arm short-circuits the size test. That is why the parent diff is still required: disabling it makes the chunked case regress immediately.
Fix is to treat the field as a singleton.
Reviewed By: hanidamlaj
Differential Revision: D120387888
fbshipit-source-id: 0c7e89df5deb02f9a4346c7bf1e01a8d45f478b4
Summary: katran_tester is moving to a CLI11 command interface, so the getdeps build needs CLI11 installed and KatranTesterCli.cpp compiled. Nothing calls it yet; the next diff adds the dispatch. Split out because edits under fbcode/opensource trigger every open source build.
Reviewed By: tagrawal03
Differential Revision: D119598499
fbshipit-source-id: 9db46020fc36e3e00f828a4b35ba725fd1a149eb
Summary: copytree, envfuncs.path_search internals, and runcmd cwd/log_file now operate on pathlib.Path, accepting str | Path. Env values, argv lists, and hash inputs stay str by necessity (subprocess + project hashes); path_search documents its str return for the same reason, to be revisited when core builders are retyped. No caller changes needed. New tests/test_path_inputs.py locks the widened contracts.
Reviewed By: bigfootjon
Differential Revision: D120383074
fbshipit-source-id: b4145f56d906f58dc8f71d7ad96a08b419f19796
Summary: getdeps already requires 3.6+ in practice (interpolated f-strings throughout), but nothing stated or enforced it: pre-3.6 interpreters died with a bare SyntaxError on the first f-string. README now documents the floor, getdeps.py exits early with a clear message below 3.6, and the dead sys.version_info < (3, 2) shim in manifest_test.py is gone.
Reviewed By: likewhatevs
Differential Revision: D120383073
fbshipit-source-id: 394c5757b3de1262e81356ca71ed4eb64f3f3709
Summary:
small refactor for WtStreamManagerTest by creating a fixture to reduce a bunch of duplicated declarations for each test
___
Differential Revision: D120421514
fbshipit-source-id: 03892d3d7a3a0bab6e731047472e4af00b0343f9
Summary:
Three commits, best read in order. Together they give getdeps the equivalent of `cargo vendor` plus `--offline`, which is what Fedora's vendored-dependency guidelines need for packaging cachelib without maintaining separate folly/fizz/wangle/mvfst/fbthrift packages in lockstep.
## getdeps: add a vendor subcommand
`getdeps.py vendor --output-dir DIR project` fetches every third-party dependency of the project and copies its source tree to `DIR/<name>`, skipping deps satisfied by system packages and the project itself. It also writes DIR/getdeps-vendor.txt listing each vendored project with its fetcher hash (the pinned git revision or the archive sha256).
This is the getdeps analogue of `cargo vendor` / `go mod vendor`: distro packagers can ship the result as a vendor tarball next to the project source, as Fedora's vendored-dependency guidelines require, and a follow-up change will let `build` consume such a directory offline.
Extracted trees rather than the original archives and clones are vendored so the result is self-contained, reviewable and license-scannable; .git directories are dropped and symlinks are followed for the same reason (subproject fetchers symlink into the scratch dir, which does not exist on an offline builder).
Tested on Fedora 44: `getdeps.py --allow-system-packages vendor --no-tests --output-dir /var/tmp/cachelib-vendor cachelib` produced eleven trees (about 330 MB, fbthrift being 216 MB of that) plus getdeps-vendor.txt, with no .git directories or symlinks left behind. Once the Fedora rpm mappings from facebook/CacheLib#488 land that drops to the seven projects Fedora does not package.
## getdeps: add --vendor-dir to build from vendored sources offline
`getdeps.py --vendor-dir DIR build project` takes every third-party dependency from `DIR/<project>`, as populated by `getdeps.py vendor`, instead of cloning or downloading it. The lookup sits in the manifest's fetcher selection after the system-package check, so a dependency that --allow-system-packages resolves to an installed package still wins, and anything not found in DIR raises an error naming the missing project rather than falling back to the network. That single check is the offline guarantee: no GitFetcher or ArchiveFetcher is ever constructed.
The vendored tree is wrapped in the existing LocalDirFetcher, the same mechanism --src-dir uses, so no fetcher code changes. Its hash is fixed and it always reports the sources as changed, which means repeated builds against a vendor dir reconfigure their dependencies each time; that is acceptable for the one-shot distro builds this is meant for.
--free-up-disk only removed build trees whose sources live under the scratch dir, a guard meant to protect a user's own --src-dir checkout. Vendored sources are ours to clean up after, so the guard now also accepts the vendor dir (stored realpath'd so the prefix comparison matches LocalDirFetcher's realpath'd source paths).
Known limitation: patchfiles are applied with `git apply` from the enclosing git top-level, so a vendor dir placed inside another git checkout would mis-apply patches for the (few) manifests that carry them. A distro build directory is not a git checkout, so this does not affect the intended use; making the patch step independent of the surrounding repository is left for a follow-up.
Tested on Fedora 44 (aarch64, 4 cores) on a tree that also carried facebook/CacheLib#488, https://github.com/facebook/CacheLib/issues/489 and https://github.com/facebook/CacheLib/issues/490, against the seven-project vendor dir produced by the previous commit (magic_enum, sparsemap, folly, fizz, wangle, mvfst, fbthrift; 276 MB):
```
unshare -rn python3 build/fbcode_builder/getdeps.py \
--allow-system-packages --vendor-dir /var/tmp/cachelib-vendor \
--scratch-path /var/tmp/getdeps-offline-scratch --num-jobs 2 \
--extra-cmake-defines '{"CMAKE_POLICY_VERSION_MINIMUM":"3.5"}' \
build --free-up-disk --no-tests --src-dir=. cachelib
```
`unshare -rn` puts the build in its own network namespace. Inspected from outside while it ran, the build process was in net:[4026532485] versus the shell's net:[4026531833]; `nsenter -n ip -brief link` inside it showed only `lo` DOWN, `getent hosts github.com` failed (exit 2), and `ss -tunap` listed no sockets. The build finished in 72 minutes with exit 0, all eight projects installed, no "Download with" or "Cloning" line in the log, and cachebench linking the system glog, liboqs, libaio and libnuma. Requesting a project that is neither vendored nor allowed from system packages fails with the new error, and `show-source-dir --recursive` resolves every vendored project into the vendor dir. The --free-up-disk fix was verified separately: an offline sparsemap build with the flag left no build tree behind, where the 72-minute run (made before the fix) had left 11 GB.
## getdeps: add tests for vendor and --vendor-dir
`getdeps/test/vendor_test.py`, in the style of the existing builder tests (MagicMock loader and build options, real `ManifestParser` objects), 5 tests: `vendor` copies a non-system dependency's tree, skips a dependency that resolves to a `SystemPackageFetcher`, skips the project itself, drops `.git`, materialises symlinks and writes `getdeps-vendor.txt`; `vendor` replaces a stale tree in the output dir; `--vendor-dir` resolves a download-URL manifest to a `LocalDirFetcher` on `<vendor-dir>/<name>`; a project missing from the vendor dir fails naming the project rather than constructing a network fetcher; and without `--vendor-dir` the normal `ArchiveFetcher` is still chosen. Run from `build/fbcode_builder` with `python3 -m unittest getdeps.test.vendor_test`.
Verified together with https://github.com/facebook/CacheLib/issues/488, https://github.com/facebook/CacheLib/issues/489 and https://github.com/facebook/CacheLib/issues/490 applied; each of those stands alone and none of them is required for this change to apply.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
X-link: https://github.com/facebook/CacheLib/pull/491
Reviewed By: aleivag, likewhatevs
Differential Revision: D120204081
Pulled By: michel-slm
fbshipit-source-id: f877310e93bcddbf6ff6f81f72e2738e23ecdfa5
Summary:
right now if we fail to parse any part of the message prior to an error (either upstream or downstream error prior to onHeadersComplete) – the HTTPMessage will neither be of type `Type::Request` or `Type::Response` which can be very error prone for the callback / consumer of the message
This ensures we only attach the partialMsg on parse error via ::onParserError if it's downstream
Reviewed By: haydn111, afrind
Differential Revision: D120194535
fbshipit-source-id: 7450959ded7179d93b1b5af7c80cccf4d663bd99
Summary:
`FindGlog` creates the `glog::glog` imported target whether or not it found anything. When glog is missing the target is there all the same, pointing at `GLOG_LIBRARY-NOTFOUND`, so nothing goes wrong until the link step reports `ld: library 'GLOG_LIBRARY-NOTFOUND' not found`. The same block also runs `find_package(Gflags)` and `find_package(LibUnwind)` to fill in glog's own dependencies, two searches that have nothing to contribute when there is no glog to attach them to.
Guarding the block on `GLOG_FOUND`, the way `FindGflags` already guards its `gflags` target, makes a failed search report itself at configure time and leaves the namespace clear.
That second part matters to folly. A fetched glog declares `glog::glog` as an `ALIAS`, and CMake rejects an alias whose name is already taken, so the previous diff had to run the module's library search by hand and pick between find and fetch before `FindGlog` could get to it. That workaround goes away and the block reads as `find_package` plus a fallback again, like every other dependency in the file.
Reaching the module in the not-found case does leave two of its variables to account for on folly's side. `GLOG_LIBRARIES` is named in the `EXPORTED_DEPS` of 42 of folly's granular libraries, so the fetch branch points it at `glog::glog`. `GLOG_INCLUDE_DIR` only means anything when glog was found, so appending it and the `glog/export.h` check that goes with it move into that branch.
Reviewed By: bigfootjon
Differential Revision: D120177659
fbshipit-source-id: ee65caed9aa25b4517b8b952b826b451b462f96b
Summary:
Remove completed `WtWriteHandle`s from `connFcBlockedStreams_` before deletion.
- Clears `connFcBlockedStreams_` during a shutdown() call.
- Erases wh from `connFcBlockedStreams_` anytime `finish(True)` is called
Reviewed By: hanidamlaj
Differential Revision: D119586489
fbshipit-source-id: bbe15c7f7b39e74299b0ba55906884500f269466
Summary:
Fedora is already detected by getdeps_platform.py and mapped to the rpm package manager, but `install-system-deps --distro` only accepted ubuntu and centos_stream, so the Fedora package list could not be produced from another host.
Also let the remaining cachelib deps that Fedora packages be satisfied from the system with --allow-system-packages instead of being built from source. libaio and numa already had rpm entries scoped to CentOS Stream (and Alma/Rocky for libaio); those selectors now include Fedora. The other two gain new [rpms.distro=fedora] sections. Package names and versions were checked against the Fedora 44 repositories:
| manifest | Fedora package | Fedora | manifest |
|-----------|------------------------|---------|----------|
| benchmark | google-benchmark-devel | 1.9.5 | 1.8.0 |
| libaio | libaio-devel | 0.3.111 | 0.3.113 |
| liboqs | liboqs-devel | 0.12.0 | 0.12.0 |
| numa | numactl-devel | 2.0.19 | 2.0.19 |
magic_enum is deliberately not mapped: Fedora ships 0.9.5, which installs `<magic_enum.hpp>` at the include root, while cachelib includes the `<magic_enum/magic_enum.hpp>` layout introduced in 0.9.6.
Tested on Fedora 44 (linux-fedora-44 host type): all changed manifests pass validate-manifest, `install-system-deps --recursive --dry-run --distro fedora cachelib` lists the four packages above, and a full `build --allow-system-packages cachelib` succeeds with fizz linking the system liboqs and cachebench linking the system libaio and libnuma.
X-link: https://github.com/facebook/CacheLib/pull/488
Reviewed By: likewhatevs
Differential Revision: D120159819
Pulled By: michel-slm
fbshipit-source-id: 8fcee0d3f90447209f259df7ba8a0a41aae4a18c
Summary:
https://fb.workplace.com/groups/proxygendevelopers/posts/3993434290963932
A handler that sets `Content-Length` then streams its body leaves that length on a chunked message. `CompressionFilter::sendHeaders` added `Content-Encoding` and forwarded it unchanged, so the recipient expected the uncompressed size, got the compressed bytes, and waited out an idle timeout with no error raised.
Remove the length there. Chunks frame the message, and the non-chunked path already recomputes it after compressing.
Reviewed By: hanidamlaj
Differential Revision: D120282282
fbshipit-source-id: 6588cebce4c943e485677663c6229e3f9ae39a6a
Summary: In QuicWtSession it would be helpful to elide a stream lookup to query both the FlowControlState and the MaxWritable in one api. I think we can cut over all (few) invocations to getMax(Stream|Conn)Writable to ::get(Stream|Conn)FlowControl and remove the ::maxWritable apis
Reviewed By: jbeshay
Differential Revision: D120168940
fbshipit-source-id: e1e008cc1c576aa1a30530fa0318d24c3adec877
Summary:
Add `HeaderValueError` and `validateHeaderValueDetail` so callers can identify which rule rejected a header value while preserving the existing boolean `validateHeaderValue` API.
Add unit coverage for each validation result and a benchmark target covering representative accepted, folded, and rejected values. The benchmark confirms that detailed error classification stays off the successful hot path.
Reviewed By: hanidamlaj
Differential Revision: D114943767
fbshipit-source-id: b8154bb2b2234b552777a225a2c69de7530a6d2e
Summary:
cachelib's CMakeLists defaults BUILD_TESTS to ON and the manifest never overrode it, so `getdeps.py build --no-tests cachelib` still compiled every test binary. Wire the getdeps test context to BUILD_TESTS the same way the folly and fizz manifests do.
With tests off the cachelib build on Fedora 44 drops from 497 to 110 ninja steps, which matters for memory- and CPU-constrained distro builders.
Tested on Fedora 44 in a single `getdeps.py --allow-system-packages build --no-tests cachelib` run together with the Fedora rpm mappings from facebook/CacheLib#488 and the FindGlog config-mode fix; the resulting CMake cache has BUILD_TESTS:BOOL=OFF and the build succeeds.
X-link: https://github.com/facebook/CacheLib/pull/490
Reviewed By: likewhatevs
Differential Revision: D120177373
Pulled By: michel-slm
fbshipit-source-id: 8b079e9878b945f778c882f84dbb4433660b181d
Summary:
With fboss off it in the parent diff, no getdeps manifest depends on fatal, so the manifest builds a library nothing asks for. Delete it.
This does not affect the GitHub repo: `github.com/facebook/fatal` is exported by ShipIt from `configerator/source/opensource/shipit_config/facebook/fatal.cconf`, which is independent of this manifest, and the repo builds through its own scripts rather than getdeps. fatal itself stays in fbcode, where roughly 350 include sites across 59 projects still use it.
Reviewed By: bigfootjon
Differential Revision: D120061602
fbshipit-source-id: 1e06ba4212a17b98973941485e46150da74a6e6b
