Summary: The files in `fboss/oss` are mainly just config files, docker files, and other things that our on diff jobs don't test anyway. By preventing them from running on changes made in this directory, we can save hours of test time on each relevant diff, as the on-diff jobs always have to wait in long queues.
Reviewed By: joseph5wu
Differential Revision: D81816095
fbshipit-source-id: d3c7e9dee96328c9364e69bc9111c610890fd7f2
Summary: When making changes to docs, target determinator still spawns the on-diff jobs which makes getting changes in take hours. This adds a shipit.strip section to the fboss manifest so that the determinator will ignore changes made in the docs directory.
Reviewed By: shiva-menta
Differential Revision: D81794034
fbshipit-source-id: 1bdb068a665f93f33800044248f161b9e0fcd2d5
Summary: We want getdeps to be able to use gcc12 without having to build it from source. This adds it as a manifest file and depndency of the fboss project, instructing getdeps.py to fetch the rpm and then point to the path containing gcc12.2.
Reviewed By: huruinan
Differential Revision: D81594108
fbshipit-source-id: 7406524beb5a1fa8d2f452712f13cce77b30fa35
Summary:
clearly made a mistake here, lack of cmake experience (thanks mingtaoy):
c-ares ships a CMake package (c-ares-config.cmake) that we should use instead of our bespoke FindCares.cmake
Reviewed By: afrind
Differential Revision: D81248145
fbshipit-source-id: 15b5595bac25f08dcf5faa7c3fb2efb10da3fc7b
Summary:
We would like to build a version of torchcomms that has minimal dependencies on dynamic libraries. I saw that there is an option `USE_STATIC_DEPS_ON_UNIX` introduced in D17228181. However, currently when we enable `USE_STATIC_DEPS_ON_UNIX=ON` to build folly, it will fail by
```
-- Could NOT find LIBUNWIND (missing: LIBUNWIND_LIBRARY)
CMake Error at build/fbcode_builder/CMake/FindLibEvent.cmake:68 (message):
Could NOT find libevent.
Call Stack (most recent call first):
CMake/folly-deps.cmake:73 (find_package)
CMakeLists.txt:145 (include)
-- Configuring incomplete, errors occurred!
```
That is because we only set ".a" to `CMAKE_FIND_LIBRARY_SUFFIXES` and there exists a special library libunwind that does not provide static linking library.
To set `CMAKE_FIND_LIBRARY_SUFFIXES = ".a" ".so"`, we can let cmake first look for `libname.a` and then `libname.so` if `.a` is not found.
Reviewed By: d4l3k
Differential Revision: D81062853
fbshipit-source-id: ce901b03473c82032c75742302ee3ed875d6af49
Summary: We're working on open sourcing our new `proxygen::coro` library. `proxygen::coro` has a dependency on c-ares, so this diff adds the c-ares manifest file to support oss builds
Reviewed By: mjoras
Differential Revision: D80456516
fbshipit-source-id: 66542834ae41553d875017aaf40238d53715af32
Summary: Now moxygen can be built using getdeps
Reviewed By: jordicenzano
Differential Revision: D67601759
fbshipit-source-id: 808cb95f3f968b3948429f3fbcc9c2c096b25fcb
Summary:
Refactor the include rewriter feature to be a build option rather than a separate builder type, and use the existing `shipit.pathmap` section instead of creating a new `include.pathmap` section.
**Before:**
```
[build]
builder = include_rewriting_cmake
[include.pathmap]
fbcode/your/project = your_project
```
**After:**
```
[build]
builder = cmake
rewrite_includes = true
[shipit.pathmap]
fbcode/your/project = your_project
```
This approach is more flexible and reuses the existing pathmap functionality rather than creating a duplicate configuration section.
**Changes made:**
- Added `rewrite_includes` as optional field in build section schema
- Modified `CMakeBuilder._build()` to check for `rewrite_includes = true` and run include rewriter when enabled
- Updated include rewriter to use existing `shipit.pathmap` section instead of new `include.pathmap`
- Removed `include.pathmap` configuration option from manifest schema
- Removed `IncludeRewritingCMakeBuilder` class entirely
- Removed `include_rewriting_cmake` builder mapping
- Removed non-existent `has_section` check that was causing issues
- Added logic to strip `fbcode/` and `xplat/` prefixes from shipit.pathmap source paths since these prefixes don't appear in `#include` statements
**Prefix Handling:**
The shipit.pathmap may contain entries like:
```
[shipit.pathmap]
fbcode/proxygen/lib = proxygen
xplat/folly = folly
```
But `#include` statements don't use these prefixes:
```cpp
#include "proxygen/HTTPMessage.h" // not "fbcode/proxygen/HTTPMessage.h"
#include "folly/String.h" // not "xplat/folly/String.h"
```
The include rewriter now automatically strips these prefixes before pattern matching.
The include rewriting functionality remains unchanged - it still uses the same `include_rewriter.py` module but now uses the existing `shipit.pathmap` configuration with proper prefix handling.
Reviewed By: bigfootjon
Differential Revision: D79846033
fbshipit-source-id: c8a70c44198119a2195c66f3d676b8268e3ed9ed
Summary: `${service}_processmap_serialization.cpp` was mistakenly added to source files in D80006546 while we do not generate this file. Fix it to unbreak fboss and other users of `add_fbthrift_cpp_library`.
Reviewed By: tanquer
Differential Revision: D80189355
fbshipit-source-id: e24f3d38ca8f1762c2636c6c4e4bc5e80aa0699d
Summary: Update the OSS configs to use `<module>_types_serialization.cpp` introduced in D79106423 and remove the `__FBTHRIFT_SEPARATE_SERIALIZATION` workaround.
Reviewed By: iahs
Differential Revision: D80006546
fbshipit-source-id: cdffcef00baff6bb2109a10f59e4cc34439669c7
Summary:
Unfortunately we can't just use `git.rev` in manifests to specify branches as we clone using `--depth`. This means `rev-parse` on a remote branch name will fail.
This diff adds `git.branch` which allows us to be explicit when we want a branch and so check this out in the first place. By doing this future uses of `rev-parse` will also understand the branch name.
Reviewed By: martindemello
Differential Revision: D79660472
fbshipit-source-id: 6b04158bdd72c28864322be28d88ee4e41f54bc8
Summary:
getdeps has been using CMake 3.20.2 since December 2021 (D32805140).
CMake 3.20.2 maps `CMAKE_CXX_STANDARD 20` on MSVC to `/std:c++latest`.
CMake 3.20.4 includes [this change](3aaf1d91bf), which maps `CMAKE_CXX_STANDARD 20` on newer versions of MSVC to `/std:c++20`.
This change allows OSS builds to build with `CMAKE_CXX_STANDARD 20` without getting dumped into `/std:c++latest`, which may jump to even newer versions than desired.
---
This will allow us to build `fbthrift` OSS using C++ 20, and avoid OSS build breakages due to the discrepancy in C++ build standards (C++ 20 internally via Buck, C++ 17 in OSS via CMake) - e.g. D79365997.
Reviewed By: chadaustin
Differential Revision: D79590917
fbshipit-source-id: 552a68ae6854ff8482222b2fc164c7d6408e777a
Summary: There are probably some other rules too, although this is the only one I know of.
Reviewed By: martindemello
Differential Revision: D79474227
fbshipit-source-id: 175f266cb43f671da241e2d2ca289328b9e413d0
Summary: Add a "manifest" for CinderX allowing it and its dependencies to be built and tested with `getdeps.py`. There was no pre-existing "builder" that was appropriate for building a Python extension from source, but rather than adding one that's specific to Cinderx I've tried to write a general builder that uses `setup.py` to install the build into the Python install.
Reviewed By: alexmalyshev
Differential Revision: D79287319
fbshipit-source-id: d302bea10c6a79cdedc08cd93b0362259dea522b
Summary:
This will be used to support the OSS build of CinderX with getdeps.py in upcoming diffs.
To make this work I had to teach getdeps.py how to pass `--external-branch` to `codesync`.
Reviewed By: alexmalyshev
Differential Revision: D79286425
fbshipit-source-id: f4b1066f8736fdb407f5bd75d3e2cc47b9e7d592
Summary: We want to make use of this for our OSS build of CinderX (see diffs further up the stack). However, the older version of setuptools seems to break in really weird ways I can't find work arounds for. Upgrading it seems to make most of the problems go away or avoidable so if nobody complains I'd rather fix things by just doing this.
Reviewed By: alexmalyshev
Differential Revision: D79195099
fbshipit-source-id: 2d65e8960e435536dc7dd7e6a142f1bd2df89a41
Summary:
Fix some issues with rust/shed that are causing problems with the getdeps builds.
Specifically:
- Disable autoexamples for `quickcheck_arbitrary_derive` as they clash with the manually defined example.
- Fix the fbcode_builder path for fb303.
- Fix import in `buffered_weight` memory bound code by avoiding use of `anyhow::Ok`.
- Fix unused item warnings in non-linux builds.
- Upgrade the version of proc-macro-crate that non-buck builds use to one that is available in the vendored packages.
- Fix documentation in sampling and bounded_traversal.
Reviewed By: RajivTS
Differential Revision: D78992994
fbshipit-source-id: cc19878b5ae8ed7cbd884805849be797a96d3fba
Summary:
The *string* `"false"`, which was being returned by default, will be interpreted as truthy. So we were never getting the `-j` arg on the `make` command.
Should be a nice little speed boost for some builds.
Reviewed By: chadaustin
Differential Revision: D78819581
fbshipit-source-id: a1b57f17185d189df25e913e06bcf71f8f540053
Summary: D78165684 added a new phony target that needs to be run before everything else
Reviewed By: kbojarczuk
Differential Revision: D78331749
fbshipit-source-id: d338d0628cc4432a6e02fb97e0f43bbffce70efa
Summary:
X-link: https://github.com/facebookincubator/fizz/pull/163
The github [windows-2019 actions image was retired by github](https://github.com/actions/runner-images/issues/12045), so all jobs on it fail
Update to windows-2022 to get them running again
windows-2022 has [different tools and versions than windows-2019](https://github.com/actions/runner-images/issues/3949). Notably it moves from Visual Studio 2019 (aka msvc 16.x) to Visual Studio 2022 (aka msvc 17.x), hence the update to buildopts.py discovery
In the course of regenerating the github actions I also fixed a couple of issues that stopped regeneration matching repo contents
* a few workflows were using workflow_dispatch, added support
* there were a trailing and double spaces for project_prefix, fixed (use ignore whitespace to remove this from review!)
Reviewed By: bigfootjon, yfeldblum
Differential Revision: D78019509
fbshipit-source-id: f8b0e9438bfc6b481b4207ad82bc1002e496a2d9
Summary:
This change causes fbcode_builder (getdeps.py) to fail immediately on most non-zero error codes from delegated commands.
While internal tools may be able to surface failures in a build that doesn't fail fast, the debug cycle is difficult externally -- this makes failures pop out more easily and has been instrumental in debugging OSS build issues.
X-link: https://github.com/facebook/sapling/pull/1099
Differential Revision: D77608371
Pulled By: quark-zju
fbshipit-source-id: e9fc00a574bc64fbc165a22e51406e85d015e2ae
Summary:
Fixes integration test failure due to lack of a functioning
python-psutil (on Linux, this requires the inclusion of native code
which is not permitted in a zipapp).
X-link: https://github.com/facebook/sapling/pull/1100
Reviewed By: zzl0
Differential Revision: D77674102
Pulled By: quark-zju
fbshipit-source-id: 11ac197d8c4082eaf16e0e28bc1a45c67f7dbb07
Summary:
When building from scratch folly on python 3.13, here is the issue you will encounter:
```
Extract /private/tmp/follyt/downloads/ninja-v1.10.2.tar.gz -> /private/tmp/follyt/extracted/ninja-v1.10.2.tar.gz
Building ninja...
---
+ cd /private/tmp/follyt/extracted/ninja-v1.10.2.tar.gz/ninja-1.10.2 && \
+ /opt/homebrew/opt/python@3.13/bin/python3.13 \
+ configure.py \
+ --bootstrap
Traceback (most recent call last):
File "/private/tmp/follyt/extracted/ninja-v1.10.2.tar.gz/ninja-1.10.2/configure.py", line 26, in <module>
import pipes
ModuleNotFoundError: No module named 'pipes'
Command '['/opt/homebrew/opt/python@3.13/bin/python3.13', 'configure.py', '--bootstrap']' returned non-zero exit status 1.
!! Failed
```
(`python3.13 ./build/fbcode_builder/getdeps.py build --scratch-path /tmp/follyt`)
This is because the version `1.10.2` of ninja uses `pipes`, which has been deprecated since python 3.11 and removed in python 3.13. [^1]
This PR changes the ninja dependency from 1.10.2 to 1.12.1.
[^1]: https://docs.python.org/3/library/pipes.html
X-link: https://github.com/facebook/folly/pull/2392
Reviewed By: yfeldblum
Differential Revision: D76523565
Pulled By: Orvid
fbshipit-source-id: 8ba5b96b3a01334ab9b04e62c80588868273bb25
Summary:
Redesigned the schema and rewrote the indexer. Compared with the previous indexer:
* this captures a lot more xrefs (e.g. local variables)
* it has more information (distinguishes functions/classes/constructors etc.)
* it is much simpler and probably faster, because it doesn't go via hiedb, it reads .hie files directly.
* it is probably more correct, I fixed a lot of things.
The schema is carefully designed so that a Name uniquely identifies an entity and corresponds fairly closely to GHC's Name, including OccName. The main difference is we don't store Uniques, instead we distinguish local Names by including their ByteSpan.
There are a couple of snapshot tests, one for the plain indexer and one for the codemarkup layer, and a Glass regression test.
Not done yet:
* we can extract types from the .hie file too, and provide type hovers in Glass. That wouldn't be too hard.
* extracting more structure so that we can reconstruct data/class decls should be possible but it's not straightforward using .hie. I'm still thinking about how best to do that.
X-link: https://github.com/facebookincubator/Glean/pull/511
Reviewed By: rubmary
Differential Revision: D74400980
Pulled By: jjuliamolin
fbshipit-source-id: 6cb183b96ef1c7030c8b7278434f84a4d72ceb28
Summary: fbthrift has moved compact/binary protocol instantiation outside _types.cpp (to _types_compact.cpp and _types_binary.cpp). We need to add these two files to the build system.
Reviewed By: vitaut
Differential Revision: D74768484
fbshipit-source-id: a45c5893489083f98832ff4eb79aee4f568b2dc2
Summary:
X-link: https://github.com/facebook/fboss/pull/433
X-link: https://github.com/facebook/fboss/pull/432
We currently download the libnl tarball from infradead.org, but it has had outages of variying timeframes. It interrupts our github workflows for too long, so we should download from the github repo instead.
Reviewed By: srikrishnagopu
Differential Revision: D74664457
fbshipit-source-id: af06c7f83624b88136ac5e133b4d1d34a201783f
Summary:
X-link: https://github.com/facebook/fboss/pull/387
We want to copy libraries identified by ldd rather than the hardcoded mapping we have right now in package-fboss.py. These need to be packaged alongside the binaries as they are dynamically linked.
Reviewed By: paulcruz74
Differential Revision: D72731449
fbshipit-source-id: 79748615b7b0970269a5663b17f2102c6f537ffa
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/1266
ZStrong is moving away from fbcode_builder to use vanilla CMake. Removing this manifest so getdeps can stop considering the repo.
Reviewed By: terrelln
Differential Revision: D72397364
fbshipit-source-id: 5cbbd6edebe46a29aeddd93d848ad5a45e8d0aad
Summary:
Fix fb303 OSS builds on macOS that failing to find xxhash.h.
Problem was that thrift cpp lib now needs xxhash but it wasn't declared in the cmake rules, which shows up as a problem when system xxhash is not installed.
CustomCompressorRegistry.cpp was also missing from cmake build resulting in link errors once the xxhash.h discovery was fixed.
Reviewed By: markbt
Differential Revision: D72452460
fbshipit-source-id: 6bdb8e0e3961529f04f9d9d29ba0daeff2e7ff2a
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/1228
ubuntu 24.04 fast float is too old and causes folly build to fail, so exclude it from system packages and build from source.
ubuntu 25.04 plucky is due to get an updated version so not removing the deb mapping entirely: https://launchpad.net/ubuntu/plucky/+source/fast-float
Reviewed By: bigfootjon
Differential Revision: D71213548
fbshipit-source-id: b1fd7c28360a476c766da1099e56fd2aa5df3d55
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/1227
Add retries to ArchiveFetcher when downloading fails. There will be 4 retries, with backoff and jitter. The max delay is capped at 10 seconds.
Reviewed By: srikrishnagopu
Differential Revision: D71167342
fbshipit-source-id: d927a639cf99185c5a04d063400bdab874dfddfe
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/1216
`str_to_floating_fast_float_from_chars` can be simplified using the new upstream option `allow_leading_plus`. It does mean that we also support parsing `+nan` and `+infinity` which we previously didn't.
Mapping to `Inf` is also handled by the parsing logic, so this custom branch can be removed.
Reviewed By: Gownta
Differential Revision: D70392258
fbshipit-source-id: 62972a6cadd1547d8ff0a3162510ae5e979d459f
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/1211
* Seems like rust-script is picky with the comment format for cargo dependencies. Switching from /* */ to //! allows is to correctly build the script.
* Remove Ubuntu 20.04 (gcc too old)
* Add Ubuntu 24.04 (latest LTS)
* Remove Fedora 36/37/38 (38 support ended 2024-05-21)
* Add Fedora 40/41/42 (42 to be released 2025-04-22)
X-link: https://github.com/facebook/watchman/pull/1275
Reviewed By: chadaustin
Differential Revision: D70350468
fbshipit-source-id: f5a29743da5b381fadeba2ed35a440b4054ca453
Summary:
X-link: https://github.com/facebookincubator/fizz/pull/160
X-link: https://github.com/facebookincubator/zstrong/pull/1192
Fix case when testing a github repo checkout, wth tpx on path, where --no-testpilot not specified.
As workaround before this lands, if in this situation pass --no-testpilot
Reviewed By: bigfootjon
Differential Revision: D69852662
fbshipit-source-id: 5065cdf3acae3bc9c90df89ed96eab3fc3e19906
Summary:
X-link: https://github.com/facebook/folly/pull/2379
This commit addresses a CMake warning due to inconsistent capitalization of the Zstd package name in FindZstd.cmake. This fixes issues with finding and using the Zstd library.
Tested on Cmake 3.28.3
X-link: https://github.com/facebookincubator/zstrong/pull/1170
Reviewed By: terrelln
Differential Revision: D69284782
fbshipit-source-id: 4acc32705ae937f075e1531cae3449cfae0d84ec
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/1175
Instead of having fbthrift download and build Python 3.8 (which is ancient), users should be able to build fbthrift against the target Python environment. (This is consistent with how Python packages would normally be built/installed.)
Note that users can specify their target version of Python via the `PYTHON_LIBRARY` and `PYTHON_INCLUDE_DIR` CMake variables, as demonstrated in the command in the test plan.
Reviewed By: vladmihailescu
Differential Revision: D68510340
fbshipit-source-id: f94805de4220b2c107ed0aa0d57fd23b65b29bd2
