Summary: The files in `fboss/oss` are mainly just config files, docker files, and other things that our on diff jobs don't test anyway. By preventing them from running on changes made in this directory, we can save hours of test time on each relevant diff, as the on-diff jobs always have to wait in long queues.
Reviewed By: joseph5wu
Differential Revision: D81816095
fbshipit-source-id: d3c7e9dee96328c9364e69bc9111c610890fd7f2
Summary: When making changes to docs, target determinator still spawns the on-diff jobs which makes getting changes in take hours. This adds a shipit.strip section to the fboss manifest so that the determinator will ignore changes made in the docs directory.
Reviewed By: shiva-menta
Differential Revision: D81794034
fbshipit-source-id: 1bdb068a665f93f33800044248f161b9e0fcd2d5
Summary: We want getdeps to be able to use gcc12 without having to build it from source. This adds it as a manifest file and depndency of the fboss project, instructing getdeps.py to fetch the rpm and then point to the path containing gcc12.2.
Reviewed By: huruinan
Differential Revision: D81594108
fbshipit-source-id: 7406524beb5a1fa8d2f452712f13cce77b30fa35
Summary: We're working on open sourcing our new `proxygen::coro` library. `proxygen::coro` has a dependency on c-ares, so this diff adds the c-ares manifest file to support oss builds
Reviewed By: mjoras
Differential Revision: D80456516
fbshipit-source-id: 66542834ae41553d875017aaf40238d53715af32
Summary: Now moxygen can be built using getdeps
Reviewed By: jordicenzano
Differential Revision: D67601759
fbshipit-source-id: 808cb95f3f968b3948429f3fbcc9c2c096b25fcb
Summary:
Unfortunately we can't just use `git.rev` in manifests to specify branches as we clone using `--depth`. This means `rev-parse` on a remote branch name will fail.
This diff adds `git.branch` which allows us to be explicit when we want a branch and so check this out in the first place. By doing this future uses of `rev-parse` will also understand the branch name.
Reviewed By: martindemello
Differential Revision: D79660472
fbshipit-source-id: 6b04158bdd72c28864322be28d88ee4e41f54bc8
Summary:
getdeps has been using CMake 3.20.2 since December 2021 (D32805140).
CMake 3.20.2 maps `CMAKE_CXX_STANDARD 20` on MSVC to `/std:c++latest`.
CMake 3.20.4 includes [this change](3aaf1d91bf), which maps `CMAKE_CXX_STANDARD 20` on newer versions of MSVC to `/std:c++20`.
This change allows OSS builds to build with `CMAKE_CXX_STANDARD 20` without getting dumped into `/std:c++latest`, which may jump to even newer versions than desired.
---
This will allow us to build `fbthrift` OSS using C++ 20, and avoid OSS build breakages due to the discrepancy in C++ build standards (C++ 20 internally via Buck, C++ 17 in OSS via CMake) - e.g. D79365997.
Reviewed By: chadaustin
Differential Revision: D79590917
fbshipit-source-id: 552a68ae6854ff8482222b2fc164c7d6408e777a
Summary: There are probably some other rules too, although this is the only one I know of.
Reviewed By: martindemello
Differential Revision: D79474227
fbshipit-source-id: 175f266cb43f671da241e2d2ca289328b9e413d0
Summary: Add a "manifest" for CinderX allowing it and its dependencies to be built and tested with `getdeps.py`. There was no pre-existing "builder" that was appropriate for building a Python extension from source, but rather than adding one that's specific to Cinderx I've tried to write a general builder that uses `setup.py` to install the build into the Python install.
Reviewed By: alexmalyshev
Differential Revision: D79287319
fbshipit-source-id: d302bea10c6a79cdedc08cd93b0362259dea522b
Summary:
This will be used to support the OSS build of CinderX with getdeps.py in upcoming diffs.
To make this work I had to teach getdeps.py how to pass `--external-branch` to `codesync`.
Reviewed By: alexmalyshev
Differential Revision: D79286425
fbshipit-source-id: f4b1066f8736fdb407f5bd75d3e2cc47b9e7d592
Summary: We want to make use of this for our OSS build of CinderX (see diffs further up the stack). However, the older version of setuptools seems to break in really weird ways I can't find work arounds for. Upgrading it seems to make most of the problems go away or avoidable so if nobody complains I'd rather fix things by just doing this.
Reviewed By: alexmalyshev
Differential Revision: D79195099
fbshipit-source-id: 2d65e8960e435536dc7dd7e6a142f1bd2df89a41
Summary:
Fix some issues with rust/shed that are causing problems with the getdeps builds.
Specifically:
- Disable autoexamples for `quickcheck_arbitrary_derive` as they clash with the manually defined example.
- Fix the fbcode_builder path for fb303.
- Fix import in `buffered_weight` memory bound code by avoiding use of `anyhow::Ok`.
- Fix unused item warnings in non-linux builds.
- Upgrade the version of proc-macro-crate that non-buck builds use to one that is available in the vendored packages.
- Fix documentation in sampling and bounded_traversal.
Reviewed By: RajivTS
Differential Revision: D78992994
fbshipit-source-id: cc19878b5ae8ed7cbd884805849be797a96d3fba
Summary: D78165684 added a new phony target that needs to be run before everything else
Reviewed By: kbojarczuk
Differential Revision: D78331749
fbshipit-source-id: d338d0628cc4432a6e02fb97e0f43bbffce70efa
Summary:
X-link: https://github.com/facebookincubator/fizz/pull/163
The github [windows-2019 actions image was retired by github](https://github.com/actions/runner-images/issues/12045), so all jobs on it fail
Update to windows-2022 to get them running again
windows-2022 has [different tools and versions than windows-2019](https://github.com/actions/runner-images/issues/3949). Notably it moves from Visual Studio 2019 (aka msvc 16.x) to Visual Studio 2022 (aka msvc 17.x), hence the update to buildopts.py discovery
In the course of regenerating the github actions I also fixed a couple of issues that stopped regeneration matching repo contents
* a few workflows were using workflow_dispatch, added support
* there were a trailing and double spaces for project_prefix, fixed (use ignore whitespace to remove this from review!)
Reviewed By: bigfootjon, yfeldblum
Differential Revision: D78019509
fbshipit-source-id: f8b0e9438bfc6b481b4207ad82bc1002e496a2d9
Summary:
When building from scratch folly on python 3.13, here is the issue you will encounter:
```
Extract /private/tmp/follyt/downloads/ninja-v1.10.2.tar.gz -> /private/tmp/follyt/extracted/ninja-v1.10.2.tar.gz
Building ninja...
---
+ cd /private/tmp/follyt/extracted/ninja-v1.10.2.tar.gz/ninja-1.10.2 && \
+ /opt/homebrew/opt/python@3.13/bin/python3.13 \
+ configure.py \
+ --bootstrap
Traceback (most recent call last):
File "/private/tmp/follyt/extracted/ninja-v1.10.2.tar.gz/ninja-1.10.2/configure.py", line 26, in <module>
import pipes
ModuleNotFoundError: No module named 'pipes'
Command '['/opt/homebrew/opt/python@3.13/bin/python3.13', 'configure.py', '--bootstrap']' returned non-zero exit status 1.
!! Failed
```
(`python3.13 ./build/fbcode_builder/getdeps.py build --scratch-path /tmp/follyt`)
This is because the version `1.10.2` of ninja uses `pipes`, which has been deprecated since python 3.11 and removed in python 3.13. [^1]
This PR changes the ninja dependency from 1.10.2 to 1.12.1.
[^1]: https://docs.python.org/3/library/pipes.html
X-link: https://github.com/facebook/folly/pull/2392
Reviewed By: yfeldblum
Differential Revision: D76523565
Pulled By: Orvid
fbshipit-source-id: 8ba5b96b3a01334ab9b04e62c80588868273bb25
Summary:
Redesigned the schema and rewrote the indexer. Compared with the previous indexer:
* this captures a lot more xrefs (e.g. local variables)
* it has more information (distinguishes functions/classes/constructors etc.)
* it is much simpler and probably faster, because it doesn't go via hiedb, it reads .hie files directly.
* it is probably more correct, I fixed a lot of things.
The schema is carefully designed so that a Name uniquely identifies an entity and corresponds fairly closely to GHC's Name, including OccName. The main difference is we don't store Uniques, instead we distinguish local Names by including their ByteSpan.
There are a couple of snapshot tests, one for the plain indexer and one for the codemarkup layer, and a Glass regression test.
Not done yet:
* we can extract types from the .hie file too, and provide type hovers in Glass. That wouldn't be too hard.
* extracting more structure so that we can reconstruct data/class decls should be possible but it's not straightforward using .hie. I'm still thinking about how best to do that.
X-link: https://github.com/facebookincubator/Glean/pull/511
Reviewed By: rubmary
Differential Revision: D74400980
Pulled By: jjuliamolin
fbshipit-source-id: 6cb183b96ef1c7030c8b7278434f84a4d72ceb28
Summary:
X-link: https://github.com/facebook/fboss/pull/433
X-link: https://github.com/facebook/fboss/pull/432
We currently download the libnl tarball from infradead.org, but it has had outages of variying timeframes. It interrupts our github workflows for too long, so we should download from the github repo instead.
Reviewed By: srikrishnagopu
Differential Revision: D74664457
fbshipit-source-id: af06c7f83624b88136ac5e133b4d1d34a201783f
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/1266
ZStrong is moving away from fbcode_builder to use vanilla CMake. Removing this manifest so getdeps can stop considering the repo.
Reviewed By: terrelln
Differential Revision: D72397364
fbshipit-source-id: 5cbbd6edebe46a29aeddd93d848ad5a45e8d0aad
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/1228
ubuntu 24.04 fast float is too old and causes folly build to fail, so exclude it from system packages and build from source.
ubuntu 25.04 plucky is due to get an updated version so not removing the deb mapping entirely: https://launchpad.net/ubuntu/plucky/+source/fast-float
Reviewed By: bigfootjon
Differential Revision: D71213548
fbshipit-source-id: b1fd7c28360a476c766da1099e56fd2aa5df3d55
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/1216
`str_to_floating_fast_float_from_chars` can be simplified using the new upstream option `allow_leading_plus`. It does mean that we also support parsing `+nan` and `+infinity` which we previously didn't.
Mapping to `Inf` is also handled by the parsing logic, so this custom branch can be removed.
Reviewed By: Gownta
Differential Revision: D70392258
fbshipit-source-id: 62972a6cadd1547d8ff0a3162510ae5e979d459f
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/1175
Instead of having fbthrift download and build Python 3.8 (which is ancient), users should be able to build fbthrift against the target Python environment. (This is consistent with how Python packages would normally be built/installed.)
Note that users can specify their target version of Python via the `PYTHON_LIBRARY` and `PYTHON_INCLUDE_DIR` CMake variables, as demonstrated in the command in the test plan.
Reviewed By: vladmihailescu
Differential Revision: D68510340
fbshipit-source-id: f94805de4220b2c107ed0aa0d57fd23b65b29bd2
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/1145
This diff contains a few changes to the xz and libiberty getdeps builds that are required
in order for folly to build as a shared library, which in turn is needed for folly's Python
extensions to build correctly.
Reviewed By: hyuen
Differential Revision: D68252093
fbshipit-source-id: 036bc4a0e7bf9a11f0a4aa6ec0014f7935afbb5b
Summary:
Demonstrate feasibility of merging fbcode and xplat copies of thrift on a single target, `thrift/compiler/detail/mustache:mustache`. To this end:
* Switch fbcode_builder from fbcode to xplat.
* Map dependencies to the correct variants for fbcode.
* Exclude mustache directory from target remapping. Once the merge is complete remapping rules will no longer be necessary at all, further reducing maintenance burden and complexity.
* Remove one copy of mustache (~2.6kLOC).
This also demonstrates that autodeps are working as expected after the fix in D67919546.
Reviewed By: yoney
Differential Revision: D67676056
fbshipit-source-id: ba3854c0997c4dd10f47a5e623381c3dadef4ecc
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/1099
Fizz requires liboqs, if present, to be version >=0.11.0
Reviewed By: frqiu
Differential Revision: D67806173
fbshipit-source-id: 2480028c023269ae5ff196a1aa102cb32677b2c9
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/1103
CacheLib OSS builds are failing because getdeps assumes libaio is already installed but for navy is a requirement for us to compile folly successfully.
Reviewed By: fanzeyi
Differential Revision: D67814070
fbshipit-source-id: 29d9b7038178bffa459fc7b983bf0eb7f3a86598
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/1098
We have been manually syncing our builds when there is a change in builds of our dependencies (folly, thrift). This has been one of the major source of work in OSS maintenance. Migrating to getdeps will automatically sync the dependencies which means we only have to manage our own builds.
NOTE: There is a dependency of getdeps on zlib which requires us to first run sudo dnf install -y zlib-devel before we successfully run getdeps. I don't think this should affect the OSS build as it is a getdeps dependency.
Reviewed By: haowu14
Differential Revision: D65844211
fbshipit-source-id: 8e89e670cdec4a21ca7aba48ae58b5b72ddbf832
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/1084
openssl build on windows is slow due to nmake being single threaded
fortunately the Qt developers had the same problem and produced jom - a nmake compatible make that adds the /j<parallelism> flag
add a jom manifest and use it for the openssl build on windows
Reviewed By: bigfootjon
Differential Revision: D66818562
fbshipit-source-id: 88938dbc862da18ae7f75df51aa99bb669aae71a
Summary:
X-link: https://github.com/facebookincubator/fizz/pull/152
X-link: https://github.com/facebookincubator/zstrong/pull/1073
On linux getdeps uses system openssl, on macOS we mostly use the homebrew one on github CI. Both of these are openssl3.x, however the fallback build from source version is still set as openssl1.1. This can be confusing, giving the impression getdeps based builds need openssl1.1
* Update the openssl manifest source url to openssl-3.0.15, this version picked to match the ubuntu 22.04 LTS system packages.
The other potentially confusing part was that the openssl source tarball was downloaded even when on on a platform like Linux where openssl is always taken from the system packages.
* Make the download configurable so that does nothing if satisfied from system packages, and no need to check the cache.
Reviewed By: ckwalsh
Differential Revision: D66495352
fbshipit-source-id: 4d24bb82bfabe44c7764b819de7f4a05f80daed1
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/1072
watchman oss tests on ubuntu are [hanging 2 hours on github and then failing](https://github.com/facebook/watchman/actions/runs/11989130985/job/33425158244), and locally show errors.
One clue was that tests worked locally on centos stream 9 and fedora 40, pointing to one or more of the ubuntu system packages being the trigger.
Turns out that having the ubuntu xz aka liblzma-dev system packages linked is triggering the issue on both ubuntu 22.04 and 24.04. Disabled system packages for xz and for glog and libunwind that also bring in the xz system packages on ubuntu.
Why ubuntu's liblzma triggers this problem is unknown at this time. This change seems to be an improvement in that we get test results, with two tests showing intermittent failutes on ubuntu-22.04 runs.
Reviewed By: ckwalsh
Differential Revision: D66446570
fbshipit-source-id: 8a631e51012c7b90e867268451fd023304039589
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/1066
Changes to enable building systemd libs from source (rather than relying on
system packages i.e. systemd, systemd-devel).
From the FBOSS perspective, this is desirable because statically linking the dependencies into the binaries makes them easier to package without relying on similar state between build and runtime environments.
Reviewed By: somasun
Differential Revision:
D65827936
Privacy Context Container: L1125642
fbshipit-source-id: c0ba2f0690466a969bb4d9a4db664b9a5b3d3d79
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/1036
X-link: https://github.com/facebook/fboss/pull/285
__Why__
1. This is pre-req for other services to validate "PM generated data". e.g `platform_manager::ConfigValidator().isValidSlotPath(...)`
2. Beefing up SlotPath validation. See T205471819 more details.
__What__
1. Dropped ConfigValidator call in `Utils::parseDevicePath` because we can assume that config paths are valid. If ill-input is provided, unexpected behaviour.
2. Added additional regex for group capturing because it doesn't seem to work with a single regex for some reason...
3. Added topological validation on slot path.
__Next__
1. DeviceName validation. basically does device definition exist in the slot path?
2. Suppose VersionedPmUnit...
Reviewed By: somasun
Differential Revision: D64798493
fbshipit-source-id: 9ca1936bf81250bda1fc6b9d800221055b6b2f48
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/1034
Add systemd lib as a dependency for fboss via getdeps manifest in preparation for D64922139.
This is just to unblock the above diff. A future fix will be needed to add the build mechanism for systemd libs from source.
Reviewed By: kevin645
Differential Revision: D65299072
fbshipit-source-id: 65c4df639a5119f1af58b3c48b612a9da7999182
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/1019
Enable coroutines on GCC for fb303 and eden OSS cmake builds to match folly and [fbthrift](197890bbed/CMakeLists.txt (L64-L75)). This stops the eden tests from immediatedly core dumping when they try to open up the thrift server
So can check if core dumps:
* fix to eden main.py to stub par_telemetry in OSS where its not available
* add the missing getdeps dependency from eden to sapling for tests (it needs the sapling binaries for the tests torun)
Reviewed By: jdelliot
Differential Revision: D64911998
fbshipit-source-id: f6316908314bd821dd8c0e5afb5fe4584f5be23e
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/1018
* test-cross-repo-mononoke-git-sot.t deleted, remove from exclusion list
* exclude flaky integration tests
changes done to make this easier:
* python-click wasn't found in sapling dir, added package mappings for rpm and deb so that we can pick up the installed version
* add a bit more info to the "unknown python exception" message from hg
* reduce paths set by getdeps, no point listing non-existing dir, makes it easier to see what is happening
X-link: https://github.com/facebook/sapling/pull/974
Reviewed By: quark-zju
Differential Revision: D64827811
Pulled By: ahornby
fbshipit-source-id: 82b4fa224d8ce957ef85e5ecf1e220f71c93e6c4
Summary:
X-link: https://github.com/facebook/sapling/pull/973
X-link: https://github.com/facebookincubator/zstrong/pull/1017
* disable flaky tests as per comment in Makefile
* fix Makefiles JOBS sed expression to be macOS make compatible
* remove Makefile comment in multiline shell, macOS make didn't like it
* remove hexdump from deps for macOS where it is a system util
Reviewed By: quark-zju
Differential Revision: D64781284
fbshipit-source-id: a7f52c67d430219dcf7173b8d03cafe32fe41989
Summary:
Ubuntu 24.04 and Fedora 40 are both python 3.12 based. Update sapling's OSS getdeps build to allow build/run/test on python 3.12 while keeping 3.10 support.
* distutils is deprecated in python 3.12 stdlib, but fortunately setuptools is pretty similar and includes a vendored distutils, so:
* added a manifest so we get python3-setuptools installed.
* setup.py: removed deprecated usage like `find_executable` in favor of `shutil.which()`, also removed some super old xcode 4/5.1 detection I found
* util.py makedate(): datetime.utcfromtimestamp is deprecated in 3.12, so updated to use non-deprecated methods. Tested with test-command-template.t which shows tz offset.
* Makefile: disabled test-eager-exchange.t on 3.12 where updating the expectation to run on 3.12 and 3.10 is tricky (debug output differs)
* tests modified to run on 3.12 and 3.10:
* test-import-eol.t: updated for invalid escape sequence warnings
* test-install.t: updated to filter out message that appears in different order on 3.12 vs earlier versions
* test-merge-driver2.t: match different debug output in 3.12, remove check for mercurial package
* test-sign-commit.t: updated to add --yes for newer gpg that python 3.12 using distros have
* found some tests needed bunzip2, so update manifest for sapling and bz2 to install it
X-link: https://github.com/facebook/sapling/pull/964
X-link: https://github.com/facebookincubator/zstrong/pull/1005
Reviewed By: quark-zju, singhsrb
Differential Revision: D63958742
Pulled By: ahornby
fbshipit-source-id: 460c42eb1315f2e1631c74d356e4976469104c1b
Summary:
Connect up the sapling *.t tests so that github CI has visible cli test status.
To get them to run:
* some needed fbpython on path. I included a shim for that in the test makefile target.
* test-rust-hooks: Command not found message, added a glob
* test-identity.t: add a glob for the sapling version
* test-eolfilename.t: output order was unstable, added sorts to make stable
* helpers-testrepo.sh: fix assumption that system hg would be able to read test repo, check if its Sapling first.
* added a manifest for the hexdump utility some of the tests required
* excluded a few remaining tests (see comments in Makefile for reason)
* fixed getdeps support for generating actions steps for test only dependencies
NB the tests run as "hg". The expectations would need to be updated if we were to run as "sl"
X-link: https://github.com/facebook/sapling/pull/963
X-link: https://github.com/facebookincubator/zstrong/pull/1004
Reviewed By: quark-zju
Differential Revision: D63958737
Pulled By: ahornby
fbshipit-source-id: 75c0d39258c320100d8d02b31390994bc2f3a3ce
