Summary:
X-link: https://github.com/facebookincubator/fizz/pull/163
The github [windows-2019 actions image was retired by github](https://github.com/actions/runner-images/issues/12045), so all jobs on it fail
Update to windows-2022 to get them running again
windows-2022 has [different tools and versions than windows-2019](https://github.com/actions/runner-images/issues/3949). Notably it moves from Visual Studio 2019 (aka msvc 16.x) to Visual Studio 2022 (aka msvc 17.x), hence the update to buildopts.py discovery
In the course of regenerating the github actions I also fixed a couple of issues that stopped regeneration matching repo contents
* a few workflows were using workflow_dispatch, added support
* there were a trailing and double spaces for project_prefix, fixed (use ignore whitespace to remove this from review!)
Reviewed By: bigfootjon, yfeldblum
Differential Revision: D78019509
fbshipit-source-id: f8b0e9438bfc6b481b4207ad82bc1002e496a2d9
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/1018
* test-cross-repo-mononoke-git-sot.t deleted, remove from exclusion list
* exclude flaky integration tests
changes done to make this easier:
* python-click wasn't found in sapling dir, added package mappings for rpm and deb so that we can pick up the installed version
* add a bit more info to the "unknown python exception" message from hg
* reduce paths set by getdeps, no point listing non-existing dir, makes it easier to see what is happening
X-link: https://github.com/facebook/sapling/pull/974
Reviewed By: quark-zju
Differential Revision: D64827811
Pulled By: ahornby
fbshipit-source-id: 82b4fa224d8ce957ef85e5ecf1e220f71c93e6c4
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/899
You shouldn't be able to depend on a library unless it is in your direct dependencies, also this shortens the massive GETDEPS_CABAL_FLAGS to something more sensible.
Reviewed By: chadaustin
Differential Revision: D58244928
fbshipit-source-id: 3e93f26ef197252cd723a65c1752dad53b5327b6
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/871
The problem I need to solve is that projects without a pkg-config
can't be found by Cabal. I need to pass extra flags to Cabal for it to
find the includes and libraries for these projects. So here I'm
creating a `GETDEPS_CABAL_FLAGS` env with all the necessary flags.
It's a bit horrible. Really I want to do this only for the direct
deps, but where the env is being setup we don't have access to the
direct vs. non-direct deps currently, only the install_dirs.
Reviewed By: josefs
Differential Revision: D58200841
fbshipit-source-id: 03f8630610691485561438d69fe8e1182396cd04
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/709
The thrift compiler includes some additional thrift files for built-in features, e.g. annotations. Unlike other thrift libraries, these thrift files are installed directly to `include` instead of `include/thrift-files`.
If we detect `include/thrift`, then consider that to be an indication that we should add `include` to the `THRIFT_INCLUDE_PATH`, so that these can be picked up by things that depend on the built-in thrift files.
Reviewed By: mitrandir77
Differential Revision:
D53943880
Privacy Context Container: L1122763
fbshipit-source-id: d30c3e2598ca8904c1b054965aca2057733ecd20
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/653
getdeps: add --build-type option to build in Debug or RelWithDebInfo mode
Adds a --build-type option so one can force RelWithDebInfo or Debug explicity
Default remains RelWithDebInfo for cmake.
cargo default is updated to --release to match cmake more closely, if you don't want release use --build-type Debug.
If you want to run github CI in Debug mode (faster build, but tests will run slower), then can pass --build-type Debug to getdeps.py generate-github-actions
X-link: https://github.com/facebook/sapling/pull/786
Reviewed By: mitrandir77
Differential Revision: D51564770
Pulled By: bigfootjon
fbshipit-source-id: ef30332ca193d9805bce005d12b5dbc9f58fcafc
Summary:
X-link: https://github.com/facebookincubator/velox/pull/6927
allow getdeps github actions to free up disk
Allow getdeps to free up some disk from the runner and intermediate build steps as some runs (notably the linux eden and mononoke ones) are hitting disk space limits
X-link: https://github.com/facebook/sapling/pull/689
Reviewed By: sggutier
Differential Revision: D49875256
Pulled By: genevievehelsel
fbshipit-source-id: b85b6b2f11857670915b64f47d3c0abd4ca8ca31
Summary:
Watchman's vagrant images rely on mapping a watchman source directory
fetched from fbsource into the VM. Thus, they sometimes look like
fbsource, and that's largely okay. However, the vendored cargo support
breaks. Instead, explicitly skip vendored cargo when there is no
fbsource dir.
Reviewed By: xavierd
Differential Revision: D36386674
fbshipit-source-id: 61f2af19507fecd2342cfc94bbb7120ab91c33b4
Summary:
Applies new import merging and sorting from µsort v1.0.
When merging imports, µsort will make a best-effort to move associated
comments to match merged elements, but there are known limitations due to
the diynamic nature of Python and developer tooling. These changes should
not produce any dangerous runtime changes, but may require touch-ups to
satisfy linters and other tooling.
Note that µsort uses case-insensitive, lexicographical sorting, which
results in a different ordering compared to isort. This provides a more
consistent sorting order, matching the case-insensitive order used when
sorting import statements by module name, and ensures that "frog", "FROG",
and "Frog" always sort next to each other.
For details on µsort's sorting and merging semantics, see the user guide:
https://usort.readthedocs.io/en/stable/guide.html#sorting
Reviewed By: lisroach
Differential Revision: D36402214
fbshipit-source-id: b641bfa9d46242188524d4ae2c44998922a62b4c
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/106
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/107
Summary
* Add EdenFS builds on external CI now EdenSCM is good
* Mac builds on github actions by using brew for system dependencies
To make this work had to fix some path ordering issues with install directories for Linux and Mac, and generalise the homebrew path fixups we were doing for bison to all the used homebrew packages.
Previously Installed packages were being added after system paths, so our own installed thing might be ignored. On github these meant system python 3.9 was being used for hg tests rather than our specified 3.8 (this showed we have some test fails on python 3.9 with "SystemError: deallocated bytearray object has exported buffers", that are beyond the scope of this diff to fix)
Also needed to include the getdeps generated python into the generated edenscmdeps3.zip archive setup.py produces otherwise EdenFS tests failed to import thrift.Thrift
Eden tests are hanging when run externally about half way through, so disable them on github actions for now as this PR is already fairly large. They work when run locally on an internal devserver, so probably some bit of environment necessary is not defined in the test runner
Reviewed By: chadaustin
Differential Revision: D34116505
fbshipit-source-id: d0d628db5daabc28d0bd8997cd5c1bc885ed1e73
Summary:
Pull Request resolved: https://github.com/facebook/watchman/pull/998
A detailed description of this change is included in the new comment.
Reviewed By: fanzeyi
Differential Revision: D34124030
fbshipit-source-id: f37c458bae135357e16b256e09f0d6eb8f1f996d
Summary: Add some basic support for FreeBSD to getdeps.
Reviewed By: ahornby
Differential Revision: D33989129
fbshipit-source-id: 42ff5f160b7e19c12196bb2e52a726f7815487bd
Summary: Make the --facebook-internal flag value available from BuildOptions so can have jobs of both values where required
Reviewed By: xavierd
Differential Revision: D33945588
fbshipit-source-id: 5d0e1b42973bd3e411a4a71075915bdb2755f0eb
Summary:
In order to speed up build times on a mac, add homebrew support to
getdeps.
Homebrew packages can be declared in a manifest using the `homebrew` header.
Futher, ahornby has added manifest entries for homebrew packages which are
included in this diff and also included a change to use the correct version of
openssl. Without this openssl change, homebrew cmake configure finds an old
openssl 1.0.2 install.
This diff provides a 2x speed up for building folly:
Timings for clean getdeps folly build on mid-2018 2.9Ghz i9 6 core intel macbook pro with 32GB RAM:
With new homebrew system deps:
```
rm -rf /Users/ahornby/.scratch/UsersZahornbyZfbsource/fbcode_builder_getdeps/
time ./opensource/fbcode_builder/getdeps.py build --allow-system-packages folly
real 17m39.329s
user 76m10.317s
sys 5m50.163s
```
Without:
```
rm -rf /Users/ahornby/.scratch/UsersZahornbyZfbsource/fbcode_builder_getdeps/
time ./opensource/fbcode_builder/getdeps.py build folly
real 32m10.344s
user 105m53.448s
sys 15m57.858s
```
Reviewed By: ahornby
Differential Revision: D33842632
fbshipit-source-id: ac785d4a8dcfa31b77292bddd9e747022ac36e3b
Summary: Create a new fbsource property for manifests so we can still use monorepo rust toolchain when running builds with --no-facebook-internal
Differential Revision: D33945559
fbshipit-source-id: 606af62e03d09c3529018af4478bcafecd0cd640
Summary:
A long time ago, getdeps scheduled each build up to the number of hardware threads. For some builds, that was too heavy, so it got throttled back to only ever use half the hardware threads. This left parallelism on the table for CPU-bound projects that don't use much RAM per compilation.
This commit makes better use of the hardware with finer-grained logic that allows each manifest to specify a `job_weight_mib` estimate in MiB, and limit concurrency to `available_ram / job_weight`.
Reviewed By: ahornby
Differential Revision: D33754018
fbshipit-source-id: 785bed6c6cfe3c473244e0806a77cec1fc119e1f
Summary:
Populate the new environment variable if relevant path for it found.
For the OSS getdeps builds of things depending on fb303 this will allow them to be able to be able to find the fb303 thrift files from the declared fb303 dependency, rather than needing duplicates/stubs inside each package.
Also adds python as dependency to fbthrift manifest, found it needed it when testing
Reviewed By: HarveyHunt
Differential Revision: D33771979
fbshipit-source-id: 9635221c0e6a2ac225d7d0babe76f1c919b583de
Summary:
On a single-core Ubuntu VM, getdeps attempts to pass -j0 to Boost Jam,
which fails.
Reviewed By: fanzeyi
Differential Revision: D33675531
fbshipit-source-id: 629ee41448941213aadec10658583d8f984b13c2
Summary:
This diffs adds
* `--shared-libs` command line argument to the `build` command which enables building shared libraries for supported projects (cmake-based projects, boost, libevent)
* this flag overrides `BUILD_SHARED_LIBS` cmake flags in manifest files and from `--extra-cmake-defines`
* adds `shared_libs=on` expression support in manifest files
* for boost, the flag enables building shared libraries **in addition to** statically linked ones
Reviewed By: simonmar
Differential Revision: D27462289
fbshipit-source-id: d22ab434f7228c30472611bc323830d88efba0a5
Summary:
Update to a newer python that builds on macOS Big Sur, make eden_scm depend on it and use it from PATH
python 3.8 requires libffi which is reference via its pkg-config setup, however python's pkg-config libffi detection is broken (https://bugs.python.org/issue34823) with the documented workaround requiring an environment variable to be passed into its ./configure step, which is why this change also adds a feature to AutoconfBuilder
With the updated python in place I was able to remove disable_env_override_pkgconfig = 1
disable_env_override_path = 1 from the eden_scm config so that it actually uses the pkg-configs, PATHs and other settings from its dependencies. This should make future python and other dependency upgrades much simpler.
Reviewed By: HarveyHunt
Differential Revision: D32231261
fbshipit-source-id: a2b6addbe22f38e3d71618c802d2c6f836fdd86c
Summary: This fixes some missing cert errors when running eden_scm tests against our own build of openssl on macOS
Reviewed By: HarveyHunt
Differential Revision: D32617808
fbshipit-source-id: efc34341c2c9b578f9c1c92e0678dfb16ef9112e
Summary:
Allow a Facebook-specific vcvarsall to be used. The getdeps default of
MSVC 2017 had at least one annoying bug fixed in 2019.
Reviewed By: chadaustin
Differential Revision: D31188039
fbshipit-source-id: 1f569fe1c5f4ceb68f0ddd339e6de3c8ca07588b
Summary:
Description:
Prior to this patch it was not possible to run open/r on linux running on ARM due to cmake and openssl being harded to download/compile for the x86_64 arch.
In order to prevent such problem this patch actively compiles cmake instead of using pre-compiled binaries and also allows the OpenSSLBuilder to provide the correct build args to openssl, thus not trying to compile or run x86_64 software.
Pull Request resolved: https://github.com/facebook/openr/pull/95
Test Plan: * built the project by using ./build/build_openr.sh
Reviewed By: wez
Differential Revision: D28224684
Pulled By: cooperlees
fbshipit-source-id: 9de61dc6d7dcf7116ec5c67f3f165cd4a4bb5e5c
Summary:
This applies the formatting changes from black v21.4b2 to all covered
projects in fbsource. Most changes are to single line docstrings, as black
will now remove leading and trailing whitespace to match PEP8. Any other
formatting changes are likely due to files that landed without formatting,
or files that previously triggered errors in black.
Any changes to code should be AST identical. Any test failures are likely
due to bad tests, or testing against the output of pyfmt.
Reviewed By: thatch
Differential Revision: D28204910
fbshipit-source-id: 804725bcd14f763e90c5ddff1d0418117c15809a
Summary:
`extra_cmake_defines` are extra flags that are passed to cmake when
compiling each one of the dependencies. For instance:
```
$ ./opensource/fbcode_builder/getdeps.py build f4d --extra-cmake-define='{"CMAKE_CXX_FLAGS": "-mavx2 -mfma -mavx -mf16c -march=native"}'
```
It takes a json map as input, which can take a list of defines (key value
pairs).
Reviewed By: wez
Differential Revision: D25855781
fbshipit-source-id: 7f4fef2c66f4d12f23c8d7086d6a4f24fcc01ff7
Summary: During intensive builds using all available cores tends to crash the machine. Cutting default number of jobs by half prevents that.
Reviewed By: wez
Differential Revision: D25027757
fbshipit-source-id: 5ea6c77fe4363b9ee09e50de1a5cc33281872e1d
Summary:
Pull Request resolved: https://github.com/facebookexperimental/rust-shed/pull/12
The OpenSSL version on Mac doesn't work well with EdenSCM and Mononoke integration, just use the one from getdeps/brew.
Also remove the now redundant "DEVELOPER_DIR" since the modern XCode version works.
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/63
Reviewed By: StanislavGlebik
Differential Revision: D23927022
Pulled By: lukaspiatkowski
fbshipit-source-id: 6b6b3baa33d49b567b9aa6178cbd20b7ae9edc89
Summary: `SDKROOT` is a requirement if we manually specify the location of the compiler on macOS. Otherwise it wouldn't be able to find the system libraries headers.
Reviewed By: wez
Differential Revision: D22577887
fbshipit-source-id: 98140e6f9e564d665db085d21023986b240b3732
Summary:
Fixes include:
1. Passing "GETDEPS_BUILD_DIR" and "GETDEPS_INSTALL_DIR" env variable and using them in eden/scm/Makefile rather than assuming the source code is always in the same place regardless getdeps arguments (it isn't).
2. Added "fbthrift-source" and "fb303-source" to avoid unnecessary compilation (at least of fb303) and to put fbthrift and fb303 source code in an easy to locate place inside getdeps' "installed" folder.
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/25
Test Plan: sandcastle, check oss-eden_scm-darwin-getdeps
Reviewed By: farnz
Differential Revision: D22431872
Pulled By: lukaspiatkowski
fbshipit-source-id: 8ccbb090713ec085a5dd56df509eb58ab6fb9e34
Summary:
In order to do what the title says, this diff does:
1. Add the `eden/oss/.../third-party/rust/.../Cargo.toml` files. As mentioned in the previous diff, those are required by GitHub so that the third party dependencies that are local in fbsource are properly defined with a "git" dependency in order for Cargo to "link" crates properly.
2. Changes to `eden/scm/Makefile` to add build/install commands for getdeps to invoke. Those command knowing that they are called from withing getdeps context they link the dependencies brought by getdeps into their proper places that match their folder layout in fbsource. Those Makefile commands also pass a GETDEPS_BUILD env to the setup.py invocations so that it knows it is being called withing a getdeps build.
3. Changes to `eden/scm/setup.py` that add "thriftasset" that makes use of the getdeps.py provided "thrift" binary to build .py files out of thrift files.
4. Changes to `distutils_rust` to use the vendored crates dir provided by getdeps.
5. Changes to `getdeps/builder.py` and `getdeps/manifest.py` that enable more fine-grained configuratior of how Makefile builds are invoked.
6. Changes to `getdeps/buildopts.py` and `getdeps/manifest.py` to disable overriding PATH and pkgconfig env, so that "eden/scm" builds in getdeps using system libraries rather than getdeps-provided ones (NOTE: I've tried to use getdeps provided libraries, but the trickiest bit was that Rust links with Python, which is currently not providable by getdeps, so if you try to build everything the system provided Python libraries will collide with getdeps provided ones)
7. Added `opensource/fbcode_builder/manifests/eden_scm` for the getdeps build.
Reviewed By: quark-zju
Differential Revision: D22336485
fbshipit-source-id: 244d10c9e06ee83de61e97e62a1f2a2184d2312f
Summary:
We use drive letter substitutions to workaround Windows
filename length limitations. On my personal Windows system running
python 3.8.2 realpath inside an ssh session manages to resolve to
the full filename which causes the boost build to fail.
Avoid that!
Reviewed By: chadaustin
Differential Revision: D21507851
fbshipit-source-id: 1220c1c85d2124ddc51f42cefff2ce00e10c55c9
Summary:
From the outset, we wanted to be sure that getdeps was able
to source and build the dependencies so that we knew that we'd have
a repeatable build. This came at the cost of build times: having
to build boost on each CI run is a bit of a chore.
This commit adds three new elements to the manifest files:
* `rpms` - a list of RPM names that are all required to be present
in order to consider the dependency satisfied
* `debs` - like `rpms` above, but scoped to debian package names
* `preinstalled.env` - a list of environment variables that if they
are all set and non-empty will satisfy the dependency.
A new `--allow-system-packages` option to getdeps enables the new
logic that looks for system packages; it is off by default, but
enabled in the generated GitHub Actions workflows.
A new `install-system-deps` subcommand is provided that will attempt
to install the system packages needed to satisfy the build. This
typically needs to be run via sudo and is thus broken out separately
from the main getdeps build flow.
I made a pass over the manifest files and added package names that
satisfy the build on ubuntu-18 and fedora-31.
shri-khare: I renamed the `Python3.7.6` manifest to just `python` as
part of this change; the version of python that it pulls in through
the normal build is the same and I believe that an equal or newer
version of python3 is available in the GH actions builder.
The `preinstalled.env` is used only by the boost manifest: it references
the name of an environment variable that is set by the github
windows hosts and that points to a pre-built and pre-installed
copy of boost. Since there is no package manager that we can
easily query for this sort of thing, probing from the environment
seems like a reasonable and fast way to check for this. We
may need to evolve this over time to become more feature rich,
but this seems like a good starting point.
This commit has the potential to save 20 minutes of build time
from each public CI build just due to the boost dependency alone!
Refs: https://github.com/facebook/watchman/pull/797
Reviewed By: yfeldblum
Differential Revision: D20740410
fbshipit-source-id: 6c38019449c54465127656c3d18a6ff1f30adaea
Summary:
I noticed that copytree was taking forever and realized
that it wasn't issuing a prefetch call so I started looking in here;
this commit teaches getdeps how to recognize and EdenFS repo on
Windows but skips calling prefetch on Windows.
Currently the prefetch implementation triggers some very slow
processing in mercurial that is slower to start than just
enumerating the files in the opensource build.
It turned out that my original problem was just that my credentials
had expired and we weren't surfacing that error on Windows yet.
Reviewed By: simpkins
Differential Revision: D20755905
fbshipit-source-id: 8d3695cdd1f04199d1d409895482b8c706285d5f
Summary:
This diff extracts the fbsource commit hash and the date of that
commit and maintains that in place of just the commit hash that
we were previously extracting.
This data is exported into the environment that we pass on to
builders so that it is available if they choose to use it.
In a follow on diff I'll use this to default a version number
in the watchman project.
Reviewed By: fanzeyi
Differential Revision: D20949666
fbshipit-source-id: dc12bffe5f0efc4297b15ba0140c4c67a23ab0fd
Summary:
This takes some pressure off both cpu and memory
on a laptop.
Reviewed By: pkaush
Differential Revision: D20562474
fbshipit-source-id: a058c71c47f25c3a2b3c1e34a0d0caf83e642021
Summary:
Ideally we'd be using mkscratch, but this still isn't shipped
to our Windows systems.
Pick a path that is more friendly to our corp windows environment by default.
Reviewed By: pkaush
Differential Revision: D20342277
fbshipit-source-id: c85bccee6701adc03b26c92ba217b18bd684257a
Summary:
Automatically detect the `--facebook-internal` flag based on the current
repository project name.
Reviewed By: wez
Differential Revision: D18621358
fbshipit-source-id: f2b3018169b151811eec455863a8bfc17667d4d8
