Summary: There are probably some other rules too, although this is the only one I know of.
Reviewed By: martindemello
Differential Revision: D79474227
fbshipit-source-id: 175f266cb43f671da241e2d2ca289328b9e413d0
Summary: Add a "manifest" for CinderX allowing it and its dependencies to be built and tested with `getdeps.py`. There was no pre-existing "builder" that was appropriate for building a Python extension from source, but rather than adding one that's specific to Cinderx I've tried to write a general builder that uses `setup.py` to install the build into the Python install.
Reviewed By: alexmalyshev
Differential Revision: D79287319
fbshipit-source-id: d302bea10c6a79cdedc08cd93b0362259dea522b
Summary:
This will be used to support the OSS build of CinderX with getdeps.py in upcoming diffs.
To make this work I had to teach getdeps.py how to pass `--external-branch` to `codesync`.
Reviewed By: alexmalyshev
Differential Revision: D79286425
fbshipit-source-id: f4b1066f8736fdb407f5bd75d3e2cc47b9e7d592
Summary:
X-link: https://github.com/facebookincubator/fizz/pull/152
X-link: https://github.com/facebookincubator/zstrong/pull/1073
On linux getdeps uses system openssl, on macOS we mostly use the homebrew one on github CI. Both of these are openssl3.x, however the fallback build from source version is still set as openssl1.1. This can be confusing, giving the impression getdeps based builds need openssl1.1
* Update the openssl manifest source url to openssl-3.0.15, this version picked to match the ubuntu 22.04 LTS system packages.
The other potentially confusing part was that the openssl source tarball was downloaded even when on on a platform like Linux where openssl is always taken from the system packages.
* Make the download configurable so that does nothing if satisfied from system packages, and no need to check the cache.
Reviewed By: ckwalsh
Differential Revision: D66495352
fbshipit-source-id: 4d24bb82bfabe44c7764b819de7f4a05f80daed1
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/1066
Changes to enable building systemd libs from source (rather than relying on
system packages i.e. systemd, systemd-devel).
From the FBOSS perspective, this is desirable because statically linking the dependencies into the binaries makes them easier to package without relying on similar state between build and runtime environments.
Reviewed By: somasun
Differential Revision:
D65827936
Privacy Context Container: L1125642
fbshipit-source-id: c0ba2f0690466a969bb4d9a4db664b9a5b3d3d79
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/1025
Add getdeps.py `env` subcommand to get the environment from getdeps in a sourcable form so that its easier to run/test/debug the binaries
There is an existing `debug` subcommand, but it starts an interactive shell inside getdeps, which isn't so useful for programatic use.
To get the output clean enough to source I switched getdeps print() logging to go to stderr.
example usage:
```
$ (source <(./build/fbcode_builder/getdeps.py --allow-system-packages env mononoke_integration); which mononoke)
```
Reviewed By: jdelliot
Differential Revision: D64982397
fbshipit-source-id: 65212936d42185e4d395557b56d3dba499caa128
Summary:
Connect up the sapling *.t tests so that github CI has visible cli test status.
To get them to run:
* some needed fbpython on path. I included a shim for that in the test makefile target.
* test-rust-hooks: Command not found message, added a glob
* test-identity.t: add a glob for the sapling version
* test-eolfilename.t: output order was unstable, added sorts to make stable
* helpers-testrepo.sh: fix assumption that system hg would be able to read test repo, check if its Sapling first.
* added a manifest for the hexdump utility some of the tests required
* excluded a few remaining tests (see comments in Makefile for reason)
* fixed getdeps support for generating actions steps for test only dependencies
NB the tests run as "hg". The expectations would need to be updated if we were to run as "sl"
X-link: https://github.com/facebook/sapling/pull/963
X-link: https://github.com/facebookincubator/zstrong/pull/1004
Reviewed By: quark-zju
Differential Revision: D63958737
Pulled By: ahornby
fbshipit-source-id: 75c0d39258c320100d8d02b31390994bc2f3a3ce
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/907
I don't sure that I make all in accordance with the your contribution pipeline, so please correct me If there is needed.
So, I've made some changes to support the `pacman` package manager in your `getdeps.py` script. In `manifests` I'm also duplicated some packages from `debs` and `rpms` sections and create a new `pps` sections with according packages for `pacman`.
Issue: https://github.com/facebook/folly/issues/1701
X-link: https://github.com/facebook/folly/pull/1702
Reviewed By: yfeldblum
Differential Revision: D33514769
Pulled By: Orvid
fbshipit-source-id: a081c3a5bcb7f7cdde3a4a91c0d15517c0a171b3
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/897
Builder refactoring: instead of providing `install_dirs` to `build()`, `test()` etc., provide `loader` and `dep_manifests` when creating the builder. This is a cleaner API because we were computing `install_dirs` in multiple places before.
Furthermore this lets us do things that need to see the manifests of the dependencies, not just the list of `install_dirs`, such as treating direct dependencies differently from indirect dependencies (see D58244928).
Reviewed By: chadaustin
Differential Revision: D58200528
fbshipit-source-id: e52d35e84161b83ab49ab43099c3e3b9bb03f36e
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/857
Where one project should be checked out in a subdirectory of another
project. Like git submodules.
This is how the Glean build currently works: hsthrift is a separate
git repo, but Glean builds with hsthrift checked out in a
subdirectory.
Reviewed By: chadaustin
Differential Revision: D58055066
fbshipit-source-id: 1a22abaa8c5261c40b752d685a03d01625215b12
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/853
For projects that can't use the simple shipit transformer (e.g. hsthrift) and projects where it would be a pain to duplicate the shipit config (e.g. Glean).
Reviewed By: pepeiborra
Differential Revision: D58055453
fbshipit-source-id: f693a320f42cfccd9808306b4ad8b1f31ce00f97
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/849
Needed for installing ghc binaries, which have an autoconf-based
distribution that only supports `make install`, not `make`.
Reviewed By: chadaustin
Differential Revision: D58011865
fbshipit-source-id: db7c3cc45701201466a6b2853c5f3515b178fa74
Summary: `.cargo/config.toml` is the modern, editor-friendly name preferred since Rust 1.39. See https://doc.rust-lang.org/1.71.0/cargo/reference/config.html.
Reviewed By: shayne-fletcher
Differential Revision: D48341886
fbshipit-source-id: dc4390f2b6a974e113e1d4e01d3a3b287a05e785
Summary:
Applies new import merging and sorting from µsort v1.0.
When merging imports, µsort will make a best-effort to move associated
comments to match merged elements, but there are known limitations due to
the diynamic nature of Python and developer tooling. These changes should
not produce any dangerous runtime changes, but may require touch-ups to
satisfy linters and other tooling.
Note that µsort uses case-insensitive, lexicographical sorting, which
results in a different ordering compared to isort. This provides a more
consistent sorting order, matching the case-insensitive order used when
sorting import statements by module name, and ensures that "frog", "FROG",
and "Frog" always sort next to each other.
For details on µsort's sorting and merging semantics, see the user guide:
https://usort.readthedocs.io/en/stable/guide.html#sorting
Reviewed By: lisroach
Differential Revision: D36402214
fbshipit-source-id: b641bfa9d46242188524d4ae2c44998922a62b4c
Summary:
This adds the ability to specify a patch file to be applied on top of any library in getdeps.
For example, zlib doesn't support static linking with CMake, but there's a small patch that can be applied to support static linking. This is the specific use case this diff is intended to support.
Reviewed By: bigfootjon
Differential Revision: D35410512
fbshipit-source-id: d1af0ddf9ec45ef28aa902c06735af86817ac194
Summary: Since Bistro is slated to be deprecated, let us turn off OSS CI for it.
Reviewed By: AnonymousMonkey2021
Differential Revision: D35347845
fbshipit-source-id: be23bcfaa098f77162f27ecfed6469b8db7d3401
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/110
Pull Request resolved: https://github.com/facebookexperimental/rust-shed/pull/27
Make it so that changes to rust-shed or other common rust source are used locally vendored, so they don't need to be pushed to github before they are visible in a build.
There was already some support for cargo vendoring in getdeps, but it was limited to dependencies between manifests built with cargo builder. This wasn't enough to build something like eden (cmake is main entry point, with later calls cargo) or eden_scm (make is main entry point, with later calls to cargo), so this diff adds a cargo prepare step for getdeps other primary build systems.
The cargo vendoring is done by using a cargo config file to point to the source files used by getdeps. It has two modes:
1. per crate, existing mode which is already automatic for cargo to cargo manifest dependencies. To use it for a non cargo build manifest, add crate.pathmap
2. per git url, existing mode which was only use for crates.io third-party crates, now can be enabled by setting cargo.cargo_config_file
Reviewed By: yancouto
Differential Revision: D33895469
fbshipit-source-id: 7b13c0b679532492a336ce217de875c25fe1be90
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/106
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/107
Summary
* Add EdenFS builds on external CI now EdenSCM is good
* Mac builds on github actions by using brew for system dependencies
To make this work had to fix some path ordering issues with install directories for Linux and Mac, and generalise the homebrew path fixups we were doing for bison to all the used homebrew packages.
Previously Installed packages were being added after system paths, so our own installed thing might be ignored. On github these meant system python 3.9 was being used for hg tests rather than our specified 3.8 (this showed we have some test fails on python 3.9 with "SystemError: deallocated bytearray object has exported buffers", that are beyond the scope of this diff to fix)
Also needed to include the getdeps generated python into the generated edenscmdeps3.zip archive setup.py produces otherwise EdenFS tests failed to import thrift.Thrift
Eden tests are hanging when run externally about half way through, so disable them on github actions for now as this PR is already fairly large. They work when run locally on an internal devserver, so probably some bit of environment necessary is not defined in the test runner
Reviewed By: chadaustin
Differential Revision: D34116505
fbshipit-source-id: d0d628db5daabc28d0bd8997cd5c1bc885ed1e73
Summary:
Watchman advertises support for FreeBSD but it's regressed a couple
ways. Ensure it builds on FreeBSD again with a Vagrantfile.
Reviewed By: ahornby
Differential Revision: D33989289
fbshipit-source-id: ff906bc219974bafbc349740ce6bddfe5bd2908f
Summary:
In order to speed up build times on a mac, add homebrew support to
getdeps.
Homebrew packages can be declared in a manifest using the `homebrew` header.
Futher, ahornby has added manifest entries for homebrew packages which are
included in this diff and also included a change to use the correct version of
openssl. Without this openssl change, homebrew cmake configure finds an old
openssl 1.0.2 install.
This diff provides a 2x speed up for building folly:
Timings for clean getdeps folly build on mid-2018 2.9Ghz i9 6 core intel macbook pro with 32GB RAM:
With new homebrew system deps:
```
rm -rf /Users/ahornby/.scratch/UsersZahornbyZfbsource/fbcode_builder_getdeps/
time ./opensource/fbcode_builder/getdeps.py build --allow-system-packages folly
real 17m39.329s
user 76m10.317s
sys 5m50.163s
```
Without:
```
rm -rf /Users/ahornby/.scratch/UsersZahornbyZfbsource/fbcode_builder_getdeps/
time ./opensource/fbcode_builder/getdeps.py build folly
real 32m10.344s
user 105m53.448s
sys 15m57.858s
```
Reviewed By: ahornby
Differential Revision: D33842632
fbshipit-source-id: ac785d4a8dcfa31b77292bddd9e747022ac36e3b
Summary: Add expressions for shipit entries so that we can map different directories if needed
Differential Revision: D33947996
fbshipit-source-id: e765769ae5a6ab90055b63284e538b965d6071b3
Summary: Create a new fbsource property for manifests so we can still use monorepo rust toolchain when running builds with --no-facebook-internal
Differential Revision: D33945559
fbshipit-source-id: 606af62e03d09c3529018af4478bcafecd0cd640
Summary: Boost 1.69 does not build on Fedora 35, so upgrade to Boost 1.78.
Reviewed By: ahornby
Differential Revision: D33855670
fbshipit-source-id: 85469a835a1dab1a7d5222413b1f1349bdcff280
Summary:
A long time ago, getdeps scheduled each build up to the number of hardware threads. For some builds, that was too heavy, so it got throttled back to only ever use half the hardware threads. This left parallelism on the table for CPU-bound projects that don't use much RAM per compilation.
This commit makes better use of the hardware with finer-grained logic that allows each manifest to specify a `job_weight_mib` estimate in MiB, and limit concurrency to `available_ram / job_weight`.
Reviewed By: ahornby
Differential Revision: D33754018
fbshipit-source-id: 785bed6c6cfe3c473244e0806a77cec1fc119e1f
Summary:
Move cargo logic to separate file as builder.py was getting a bit large
It's just a move of source file for CargoBuilder. Diff looks a bit big as I did hg cp to preserve history
Differential Revision: D33888925
fbshipit-source-id: 2d57343535ab087e09876edba4d00a6f0234fcd0
Summary:
Now that we have some type annotations, we might as well support
running the type checker.
Reviewed By: xavierd
Differential Revision: D33715757
fbshipit-source-id: baf693e4b2415e0e1aa50b569b744ca0cfb91337
Summary:
This diffs adds
* `--shared-libs` command line argument to the `build` command which enables building shared libraries for supported projects (cmake-based projects, boost, libevent)
* this flag overrides `BUILD_SHARED_LIBS` cmake flags in manifest files and from `--extra-cmake-defines`
* adds `shared_libs=on` expression support in manifest files
* for boost, the flag enables building shared libraries **in addition to** statically linked ones
Reviewed By: simonmar
Differential Revision: D27462289
fbshipit-source-id: d22ab434f7228c30472611bc323830d88efba0a5
Summary:
A number of places were extracting dependencies from manifests, but only one was adding in the implicit dependencies for build tools.
Extract the logic to one place and use so that a change in a tool like cmake will now correctly affect all tools using cmake, as it will be taken into account as a dependency hash when the manifest's hash is computed.
Tests for this change revealed that install_dirs needed to be populated in reverse order from the manifest topo-sort, so have also addressed that
Reviewed By: wittgenst
Differential Revision: D32730717
fbshipit-source-id: 1b2a25e460de6085d274c99acfd391b3bd259264
Summary: The system packages necessary on linux can vary even within .deb or .rpm space. Add a getdeps.py feature to allow this, and use it in boost to mark that only centos_stream 8 has/needs the special boost169 packages
Reviewed By: HarveyHunt
Differential Revision: D32987046
fbshipit-source-id: 37f771159128e3a61fe4d6bb3d1444e95efd8a62
Summary:
Update to a newer python that builds on macOS Big Sur, make eden_scm depend on it and use it from PATH
python 3.8 requires libffi which is reference via its pkg-config setup, however python's pkg-config libffi detection is broken (https://bugs.python.org/issue34823) with the documented workaround requiring an environment variable to be passed into its ./configure step, which is why this change also adds a feature to AutoconfBuilder
With the updated python in place I was able to remove disable_env_override_pkgconfig = 1
disable_env_override_path = 1 from the eden_scm config so that it actually uses the pkg-configs, PATHs and other settings from its dependencies. This should make future python and other dependency upgrades much simpler.
Reviewed By: HarveyHunt
Differential Revision: D32231261
fbshipit-source-id: a2b6addbe22f38e3d71618c802d2c6f836fdd86c
Summary:
rpm names needed updating as boost-contract wasn't found.
The rpm names now include the boost version which should be handy when boost dependency is updated next
Unfortunately the centos boost 1.69 rpms aren't found automatically by the centos cmake boost rules, so this also adds builder.py logic to point to it using BOOST_INCLUDEDIR and BOOST_LIBRARYDIR
Reviewed By: mzr
Differential Revision: D32140834
fbshipit-source-id: 3e2dd822613957ae4d7be5b73623ff581f11d02b
Summary:
Description:
Prior to this patch it was not possible to run open/r on linux running on ARM due to cmake and openssl being harded to download/compile for the x86_64 arch.
In order to prevent such problem this patch actively compiles cmake instead of using pre-compiled binaries and also allows the OpenSSLBuilder to provide the correct build args to openssl, thus not trying to compile or run x86_64 software.
Pull Request resolved: https://github.com/facebook/openr/pull/95
Test Plan: * built the project by using ./build/build_openr.sh
Reviewed By: wez
Differential Revision: D28224684
Pulled By: cooperlees
fbshipit-source-id: 9de61dc6d7dcf7116ec5c67f3f165cd4a4bb5e5c
Summary:
This applies the formatting changes from black v21.4b2 to all covered
projects in fbsource. Most changes are to single line docstrings, as black
will now remove leading and trailing whitespace to match PEP8. Any other
formatting changes are likely due to files that landed without formatting,
or files that previously triggered errors in black.
Any changes to code should be AST identical. Any test failures are likely
due to bad tests, or testing against the output of pyfmt.
Reviewed By: thatch
Differential Revision: D28204910
fbshipit-source-id: 804725bcd14f763e90c5ddff1d0418117c15809a
Summary:
This took a lot of cmake fuckery. I am guessing it will break the current travis-ci.org based build, since it no longer downloads gflags.
As I put this diff up, mvfst (a dependency of quic (a dependency of proxygen)) is not building right now due to another diff - a fix is in progress but isn't landed yet. I applied the fix locally for the test plan, but didn't include it in the diff.
Reviewed By: snarkmaster
Differential Revision: D25651851
fbshipit-source-id: b4f15ac7fa5293123710fe367c20d355c0970c42
Summary:
`extra_cmake_defines` are extra flags that are passed to cmake when
compiling each one of the dependencies. For instance:
```
$ ./opensource/fbcode_builder/getdeps.py build f4d --extra-cmake-define='{"CMAKE_CXX_FLAGS": "-mavx2 -mfma -mavx -mf16c -march=native"}'
```
It takes a json map as input, which can take a list of defines (key value
pairs).
Reviewed By: wez
Differential Revision: D25855781
fbshipit-source-id: 7f4fef2c66f4d12f23c8d7086d6a4f24fcc01ff7
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/67
With this change it will be possible to build dependencies of and run integration tests using getdeps.py.
This is the first goal of Q4 as per https://fb.quip.com/v8YzAYNSYgot: "Get Open Source version of integration tests running on Legocastle".
Before this diff:
The OSS integration tests run now on GitHub by:
- Building some test dependencies with getdeps.py
- Building some test dependencies with homebrew/apt-get
- Running tests via python script
The OSS integration tests were not running on Sandcastle.
After this diff:
The OSS integration tests run on Github by:
- Building and executing tests via getdeps.py (execution of tests happens by getdeps.py calling Make calling python script)
The OSS integration tests run on Sandcastle using the same getdeps.py setup as Github.
Reviewed By: krallin
Differential Revision: D24253268
fbshipit-source-id: cae249b72d076222673b8bbe4ec21866dcdbb253
Summary:
This diff adds all third party dependencies that are required by getdeps to be able to build and runn Mononoke's integration tests.
Also add a stub Makefile with no-op steps that will be filled in next diff.
Reviewed By: ahornby
Differential Revision: D24251894
fbshipit-source-id: 67384ecfd0ced6762dddc3c6e61feb1240b1162d
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/51
This diff extends capabilities of CargoBuilder in getdeps so that individual manifests can be build even without workspaces. Thanks to that a build for edenapi/tools can be made and its artifacts can be used in mononoke integration tests.
Reviewed By: StanislavGlebik
Differential Revision: D23574887
fbshipit-source-id: 8a974a6b5235d36a44fe082aad55cd380d84dd09
Summary:
- Replace deprecated method
- Stops spam to my console as I run with show deprecation warnings
Reviewed By: yi-xian
Differential Revision: D22776372
fbshipit-source-id: d29193e4c4c26d7facfabf9038dcb33c1af92434
Summary:
In order to do what the title says, this diff does:
1. Add the `eden/oss/.../third-party/rust/.../Cargo.toml` files. As mentioned in the previous diff, those are required by GitHub so that the third party dependencies that are local in fbsource are properly defined with a "git" dependency in order for Cargo to "link" crates properly.
2. Changes to `eden/scm/Makefile` to add build/install commands for getdeps to invoke. Those command knowing that they are called from withing getdeps context they link the dependencies brought by getdeps into their proper places that match their folder layout in fbsource. Those Makefile commands also pass a GETDEPS_BUILD env to the setup.py invocations so that it knows it is being called withing a getdeps build.
3. Changes to `eden/scm/setup.py` that add "thriftasset" that makes use of the getdeps.py provided "thrift" binary to build .py files out of thrift files.
4. Changes to `distutils_rust` to use the vendored crates dir provided by getdeps.
5. Changes to `getdeps/builder.py` and `getdeps/manifest.py` that enable more fine-grained configuratior of how Makefile builds are invoked.
6. Changes to `getdeps/buildopts.py` and `getdeps/manifest.py` to disable overriding PATH and pkgconfig env, so that "eden/scm" builds in getdeps using system libraries rather than getdeps-provided ones (NOTE: I've tried to use getdeps provided libraries, but the trickiest bit was that Rust links with Python, which is currently not providable by getdeps, so if you try to build everything the system provided Python libraries will collide with getdeps provided ones)
7. Added `opensource/fbcode_builder/manifests/eden_scm` for the getdeps build.
Reviewed By: quark-zju
Differential Revision: D22336485
fbshipit-source-id: 244d10c9e06ee83de61e97e62a1f2a2184d2312f
