Summary: Add a "manifest" for CinderX allowing it and its dependencies to be built and tested with `getdeps.py`. There was no pre-existing "builder" that was appropriate for building a Python extension from source, but rather than adding one that's specific to Cinderx I've tried to write a general builder that uses `setup.py` to install the build into the Python install.
Reviewed By: alexmalyshev
Differential Revision: D79287319
fbshipit-source-id: d302bea10c6a79cdedc08cd93b0362259dea522b
Summary:
The *string* `"false"`, which was being returned by default, will be interpreted as truthy. So we were never getting the `-j` arg on the `make` command.
Should be a nice little speed boost for some builds.
Reviewed By: chadaustin
Differential Revision: D78819581
fbshipit-source-id: a1b57f17185d189df25e913e06bcf71f8f540053
Summary:
This change causes fbcode_builder (getdeps.py) to fail immediately on most non-zero error codes from delegated commands.
While internal tools may be able to surface failures in a build that doesn't fail fast, the debug cycle is difficult externally -- this makes failures pop out more easily and has been instrumental in debugging OSS build issues.
X-link: https://github.com/facebook/sapling/pull/1099
Differential Revision: D77608371
Pulled By: quark-zju
fbshipit-source-id: e9fc00a574bc64fbc165a22e51406e85d015e2ae
Summary:
X-link: https://github.com/facebookincubator/fizz/pull/160
X-link: https://github.com/facebookincubator/zstrong/pull/1192
Fix case when testing a github repo checkout, wth tpx on path, where --no-testpilot not specified.
As workaround before this lands, if in this situation pass --no-testpilot
Reviewed By: bigfootjon
Differential Revision: D69852662
fbshipit-source-id: 5065cdf3acae3bc9c90df89ed96eab3fc3e19906
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/1084
openssl build on windows is slow due to nmake being single threaded
fortunately the Qt developers had the same problem and produced jom - a nmake compatible make that adds the /j<parallelism> flag
add a jom manifest and use it for the openssl build on windows
Reviewed By: bigfootjon
Differential Revision: D66818562
fbshipit-source-id: 88938dbc862da18ae7f75df51aa99bb669aae71a
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/1066
Changes to enable building systemd libs from source (rather than relying on
system packages i.e. systemd, systemd-devel).
From the FBOSS perspective, this is desirable because statically linking the dependencies into the binaries makes them easier to package without relying on similar state between build and runtime environments.
Reviewed By: somasun
Differential Revision:
D65827936
Privacy Context Container: L1125642
fbshipit-source-id: c0ba2f0690466a969bb4d9a4db664b9a5b3d3d79
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/1025
Add getdeps.py `env` subcommand to get the environment from getdeps in a sourcable form so that its easier to run/test/debug the binaries
There is an existing `debug` subcommand, but it starts an interactive shell inside getdeps, which isn't so useful for programatic use.
To get the output clean enough to source I switched getdeps print() logging to go to stderr.
example usage:
```
$ (source <(./build/fbcode_builder/getdeps.py --allow-system-packages env mononoke_integration); which mononoke)
```
Reviewed By: jdelliot
Differential Revision: D64982397
fbshipit-source-id: 65212936d42185e4d395557b56d3dba499caa128
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/995
Bring back mononoke getdeps integration tests. This is a refesh of the previously working export-D34186407 branch from 2022
Main changes since:
* depend on restored sapling manifest from previous commit
* bring back selected manifests needed by mononoke that were deleted in D51869247. I added the sqlite binary packages from this to the sqlite3 manifest rather than bring back the sqllite3-bin manifest
* add manifests for new tools used: ripgrep, git-lfs, and zstd cli (we already had the zstd libs)
* fix a few test expecations that where too closely tied to git cli or TLS version etc (ubuntu 22.04 is on older version)
* getdeps MakefileBuilder.run_tests() improvements
* fix error status reporting, was not failing if tests failed
* pass of --num-jobs to tests to stop it OOMing my machine
* pass of --filter to tests so can iterate on one test more easily
Can to iterate local execution for one test with:
```
python3 ./build/fbcode_builder/getdeps.py test --num-jobs 4 --allow-system-packages --no-facebook-internal --src-dir=. mononoke_integration --retry 0 --filter server/test-gettreepack.t
```
X-link: https://github.com/facebook/sapling/pull/951
Reviewed By: quark-zju
Differential Revision: D62978526
Pulled By: ahornby
fbshipit-source-id: 0070a67d798bb23ee9e78e1a5149ba5364d548c9
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/899
You shouldn't be able to depend on a library unless it is in your direct dependencies, also this shortens the massive GETDEPS_CABAL_FLAGS to something more sensible.
Reviewed By: chadaustin
Differential Revision: D58244928
fbshipit-source-id: 3e93f26ef197252cd723a65c1752dad53b5327b6
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/897
Builder refactoring: instead of providing `install_dirs` to `build()`, `test()` etc., provide `loader` and `dep_manifests` when creating the builder. This is a cleaner API because we were computing `install_dirs` in multiple places before.
Furthermore this lets us do things that need to see the manifests of the dependencies, not just the list of `install_dirs`, such as treating direct dependencies differently from indirect dependencies (see D58244928).
Reviewed By: chadaustin
Differential Revision: D58200528
fbshipit-source-id: e52d35e84161b83ab49ab43099c3e3b9bb03f36e
Summary:
Starts a shell in the build dir, with the environment set up in the
same way as for the build. Useful for experimenting and testing.
Reviewed By: chadaustin
Differential Revision: D58082246
fbshipit-source-id: 82b275401528d7616c2560d80b4c187de67f6032
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/849
Needed for installing ghc binaries, which have an autoconf-based
distribution that only supports `make install`, not `make`.
Reviewed By: chadaustin
Differential Revision: D58011865
fbshipit-source-id: db7c3cc45701201466a6b2853c5f3515b178fa74
Summary:
getdeps can generate a windows wrapper script that can be used to run build artifacts from build directory.
In github actions for large projects we set a getdeps option delete the build dir as soon as we've successfully installed artefacts to save disk space. This option was enabled for windows in D56165825. Turns out that didn't work, this diff adds the missing conditional so that it should.
Reviewed By: vitaut
Differential Revision: D56930778
fbshipit-source-id: 0cb9ac94ef9b39f4e33af8fb91098dc0d833731b
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/720
When running getdeps as part of Github CI, the toolchains employed are found in "C:\Program Files" or "C:\Program Files(x86)" - both requiring quoting to execute.
Lets add quotes and fix tha.
Reviewed By: vitaut
Differential Revision: D54433098
fbshipit-source-id: d3570486a2a2b12f4649becc987f9da1ca65c796
Summary:
An update to the Windows toolchain broke OSS getdeps builds. This was caused by the execution of `vcvarsall.bat` returning an ERRORLEVEL=1 when any extension (optional tools) was absent. Given we don't install many extensions this meant that the setup script was failing. The resultant behavior was to fail all build steps.
The fix was to wrap the invocation of `vcvarsall.bat` in a different batch file that always returns ERRORLEVEL=0. This should be OK as any real build failures will come by running the actual build scripts.
NOTE: There are other known failures (i.e. folly not building due to new compiler) that will be exposed after this change. They will not be causing any new job failures, but should be addressed as well.
Reviewed By: chadaustin
Differential Revision: D54280190
fbshipit-source-id: 7bf38bb2cb084cf5c4cd5650b5f0f06bb1dbcd9b
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/653
getdeps: add --build-type option to build in Debug or RelWithDebInfo mode
Adds a --build-type option so one can force RelWithDebInfo or Debug explicity
Default remains RelWithDebInfo for cmake.
cargo default is updated to --release to match cmake more closely, if you don't want release use --build-type Debug.
If you want to run github CI in Debug mode (faster build, but tests will run slower), then can pass --build-type Debug to getdeps.py generate-github-actions
X-link: https://github.com/facebook/sapling/pull/786
Reviewed By: mitrandir77
Differential Revision: D51564770
Pulled By: bigfootjon
fbshipit-source-id: ef30332ca193d9805bce005d12b5dbc9f58fcafc
Summary: As titled, ignore space when applying oss patches. Certain source code will use tabs instead of spaces (for eg, iproute2) and the patch fails when applying;
Reviewed By: shri-khare
Differential Revision: D51971636
fbshipit-source-id: 094983d142a039428da4cd9f980d6f30ca3e50fa
Summary:
As titled, remove stdint include from iproute2 patch from iproute2 builder. Also this include is no longer neccessary as 4.12 tc_core.c contains stdint already.
Code pointer for stdint: https://github.com/iproute2/iproute2/blob/v4.12.0/tc/tc_core.c#L15
Reviewed By: shri-khare
Differential Revision: D51971637
fbshipit-source-id: b9b7a2a1fe24fc93b04c932b3fe4ae6080aeab08
Summary:
X-link: https://github.com/facebookincubator/velox/pull/6927
allow getdeps github actions to free up disk
Allow getdeps to free up some disk from the runner and intermediate build steps as some runs (notably the linux eden and mononoke ones) are hitting disk space limits
X-link: https://github.com/facebook/sapling/pull/689
Reviewed By: sggutier
Differential Revision: D49875256
Pulled By: genevievehelsel
fbshipit-source-id: b85b6b2f11857670915b64f47d3c0abd4ca8ca31
Summary:
A request to record_results failed, which caused a folly gcc 8 build
regression to slip in, which broke the Watchman and EdenFS builds.
Instead, turn record_results requests into a logged warning.
Reviewed By: xavierd
Differential Revision: D38170729
fbshipit-source-id: d59ae5fa48b5cbcf649281108d958ff95e319e35
Summary:
TPX is now available on Windows, the testpilot code is thus no longer
necessary.
Reviewed By: fanzeyi
Differential Revision: D37476157
fbshipit-source-id: 885881de2e06eab4990c6c96dcef8f51614a17f5
Summary:
TPX on Windows is being worked on, but not yet ready for prime time. Let's make
sure that getdeps isn't trying to use TPX on Windows for now.
Reviewed By: splhack
Differential Revision: D37474152
fbshipit-source-id: e636b0b68f6f73472f28295c280803ac44fafebf
Summary:
Fix local test runs. Previously, they were silently passing, because
they skipped tests that were unknown to tpx. Instead, treat local runs
separately from CI diff runs.
Reviewed By: xavierd
Differential Revision: D37287032
fbshipit-source-id: a3a88e117cd0e078e18c0f795bae467768d812a5
Summary:
Before the change:
```
% uname -m -s
Darwin arm64
% ./autogen.sh
...
ld: warning: ignoring file /private/var/.../installed/openssl-OovnrH2WrHG18y7xi0irAegZLOlNsDyeOGb8BBEcoGw/lib/libssl.dylib, building for macOS-arm64 but attempting to link with file built for macOS-x86_64
ld: warning: ignoring file /private/var/.../installed/openssl-OovnrH2WrHG18y7xi0irAegZLOlNsDyeOGb8BBEcoGw/lib/libcrypto.dylib, building for macOS-arm64 but attempting to link with file built for macOS-x86_64
...
```
After the change build succeeds.
X-link: https://github.com/facebook/watchman/pull/1032
Reviewed By: fanzeyi
Differential Revision: D37137060
Pulled By: genevievehelsel
fbshipit-source-id: a8381c39821bbd29ddea1d1009b94f5ed6cacb21
Summary: Adding patching support for all platforms using `git apply`
Reviewed By: xavierd
Differential Revision: D36939728
fbshipit-source-id: ccf6e4ccd30257c6928ba804a76882d4d2f71700
Summary:
Without ctest installed, `getdeps.py test $project` would produce a
bad error message about failing to run a `''` command. Instead,
explicitly error early when a command is required.
Reviewed By: genevievehelsel
Differential Revision: D36535929
fbshipit-source-id: d300c6b1b0c124b56dffffae0a71bee9b7f12fe7
Summary:
Watchman's vagrant images rely on mapping a watchman source directory
fetched from fbsource into the VM. Thus, they sometimes look like
fbsource, and that's largely okay. However, the vendored cargo support
breaks. Instead, explicitly skip vendored cargo when there is no
fbsource dir.
Reviewed By: xavierd
Differential Revision: D36386674
fbshipit-source-id: 61f2af19507fecd2342cfc94bbb7120ab91c33b4
Summary:
Applies new import merging and sorting from µsort v1.0.
When merging imports, µsort will make a best-effort to move associated
comments to match merged elements, but there are known limitations due to
the diynamic nature of Python and developer tooling. These changes should
not produce any dangerous runtime changes, but may require touch-ups to
satisfy linters and other tooling.
Note that µsort uses case-insensitive, lexicographical sorting, which
results in a different ordering compared to isort. This provides a more
consistent sorting order, matching the case-insensitive order used when
sorting import statements by module name, and ensures that "frog", "FROG",
and "Frog" always sort next to each other.
For details on µsort's sorting and merging semantics, see the user guide:
https://usort.readthedocs.io/en/stable/guide.html#sorting
Reviewed By: lisroach
Differential Revision: D36402214
fbshipit-source-id: b641bfa9d46242188524d4ae2c44998922a62b4c
Summary:
This adds the ability to specify a patch file to be applied on top of any library in getdeps.
For example, zlib doesn't support static linking with CMake, but there's a small patch that can be applied to support static linking. This is the specific use case this diff is intended to support.
Reviewed By: bigfootjon
Differential Revision: D35410512
fbshipit-source-id: d1af0ddf9ec45ef28aa902c06735af86817ac194
Summary: Since Bistro is slated to be deprecated, let us turn off OSS CI for it.
Reviewed By: AnonymousMonkey2021
Differential Revision: D35347845
fbshipit-source-id: be23bcfaa098f77162f27ecfed6469b8db7d3401
Summary:
If getdeps tests run from fbcode, tpx guesses that the test type is FBCODE rather than FBTEST. This makes UTF ignore the buildsystem=getdeps config, which leads tests to be ignored.
This diff sets the test type explicitly.
More context in [this post](https://fb.workplace.com/groups/tpx.users/permalink/1309289352870499/).
Reviewed By: sandeepkumarpani888
Differential Revision: D34271402
fbshipit-source-id: 2c6c37d706f22d214cd2f7622a7124b4c3f92ba0
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/110
Pull Request resolved: https://github.com/facebookexperimental/rust-shed/pull/27
Make it so that changes to rust-shed or other common rust source are used locally vendored, so they don't need to be pushed to github before they are visible in a build.
There was already some support for cargo vendoring in getdeps, but it was limited to dependencies between manifests built with cargo builder. This wasn't enough to build something like eden (cmake is main entry point, with later calls cargo) or eden_scm (make is main entry point, with later calls to cargo), so this diff adds a cargo prepare step for getdeps other primary build systems.
The cargo vendoring is done by using a cargo config file to point to the source files used by getdeps. It has two modes:
1. per crate, existing mode which is already automatic for cargo to cargo manifest dependencies. To use it for a non cargo build manifest, add crate.pathmap
2. per git url, existing mode which was only use for crates.io third-party crates, now can be enabled by setting cargo.cargo_config_file
Reviewed By: yancouto
Differential Revision: D33895469
fbshipit-source-id: 7b13c0b679532492a336ce217de875c25fe1be90
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/106
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/107
Summary
* Add EdenFS builds on external CI now EdenSCM is good
* Mac builds on github actions by using brew for system dependencies
To make this work had to fix some path ordering issues with install directories for Linux and Mac, and generalise the homebrew path fixups we were doing for bison to all the used homebrew packages.
Previously Installed packages were being added after system paths, so our own installed thing might be ignored. On github these meant system python 3.9 was being used for hg tests rather than our specified 3.8 (this showed we have some test fails on python 3.9 with "SystemError: deallocated bytearray object has exported buffers", that are beyond the scope of this diff to fix)
Also needed to include the getdeps generated python into the generated edenscmdeps3.zip archive setup.py produces otherwise EdenFS tests failed to import thrift.Thrift
Eden tests are hanging when run externally about half way through, so disable them on github actions for now as this PR is already fairly large. They work when run locally on an internal devserver, so probably some bit of environment necessary is not defined in the test runner
Reviewed By: chadaustin
Differential Revision: D34116505
fbshipit-source-id: d0d628db5daabc28d0bd8997cd5c1bc885ed1e73
