Summary:
An update to the Windows toolchain broke OSS getdeps builds. This was caused by the execution of `vcvarsall.bat` returning an ERRORLEVEL=1 when any extension (optional tools) was absent. Given we don't install many extensions this meant that the setup script was failing. The resultant behavior was to fail all build steps.
The fix was to wrap the invocation of `vcvarsall.bat` in a different batch file that always returns ERRORLEVEL=0. This should be OK as any real build failures will come by running the actual build scripts.
NOTE: There are other known failures (i.e. folly not building due to new compiler) that will be exposed after this change. They will not be causing any new job failures, but should be addressed as well.
Reviewed By: chadaustin
Differential Revision: D54280190
fbshipit-source-id: 7bf38bb2cb084cf5c4cd5650b5f0f06bb1dbcd9b
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/653
getdeps: add --build-type option to build in Debug or RelWithDebInfo mode
Adds a --build-type option so one can force RelWithDebInfo or Debug explicity
Default remains RelWithDebInfo for cmake.
cargo default is updated to --release to match cmake more closely, if you don't want release use --build-type Debug.
If you want to run github CI in Debug mode (faster build, but tests will run slower), then can pass --build-type Debug to getdeps.py generate-github-actions
X-link: https://github.com/facebook/sapling/pull/786
Reviewed By: mitrandir77
Differential Revision: D51564770
Pulled By: bigfootjon
fbshipit-source-id: ef30332ca193d9805bce005d12b5dbc9f58fcafc
Summary: As titled, ignore space when applying oss patches. Certain source code will use tabs instead of spaces (for eg, iproute2) and the patch fails when applying;
Reviewed By: shri-khare
Differential Revision: D51971636
fbshipit-source-id: 094983d142a039428da4cd9f980d6f30ca3e50fa
Summary:
As titled, remove stdint include from iproute2 patch from iproute2 builder. Also this include is no longer neccessary as 4.12 tc_core.c contains stdint already.
Code pointer for stdint: https://github.com/iproute2/iproute2/blob/v4.12.0/tc/tc_core.c#L15
Reviewed By: shri-khare
Differential Revision: D51971637
fbshipit-source-id: b9b7a2a1fe24fc93b04c932b3fe4ae6080aeab08
Summary:
X-link: https://github.com/facebookincubator/velox/pull/6927
allow getdeps github actions to free up disk
Allow getdeps to free up some disk from the runner and intermediate build steps as some runs (notably the linux eden and mononoke ones) are hitting disk space limits
X-link: https://github.com/facebook/sapling/pull/689
Reviewed By: sggutier
Differential Revision: D49875256
Pulled By: genevievehelsel
fbshipit-source-id: b85b6b2f11857670915b64f47d3c0abd4ca8ca31
Summary:
A request to record_results failed, which caused a folly gcc 8 build
regression to slip in, which broke the Watchman and EdenFS builds.
Instead, turn record_results requests into a logged warning.
Reviewed By: xavierd
Differential Revision: D38170729
fbshipit-source-id: d59ae5fa48b5cbcf649281108d958ff95e319e35
Summary:
TPX is now available on Windows, the testpilot code is thus no longer
necessary.
Reviewed By: fanzeyi
Differential Revision: D37476157
fbshipit-source-id: 885881de2e06eab4990c6c96dcef8f51614a17f5
Summary:
TPX on Windows is being worked on, but not yet ready for prime time. Let's make
sure that getdeps isn't trying to use TPX on Windows for now.
Reviewed By: splhack
Differential Revision: D37474152
fbshipit-source-id: e636b0b68f6f73472f28295c280803ac44fafebf
Summary:
Fix local test runs. Previously, they were silently passing, because
they skipped tests that were unknown to tpx. Instead, treat local runs
separately from CI diff runs.
Reviewed By: xavierd
Differential Revision: D37287032
fbshipit-source-id: a3a88e117cd0e078e18c0f795bae467768d812a5
Summary:
Before the change:
```
% uname -m -s
Darwin arm64
% ./autogen.sh
...
ld: warning: ignoring file /private/var/.../installed/openssl-OovnrH2WrHG18y7xi0irAegZLOlNsDyeOGb8BBEcoGw/lib/libssl.dylib, building for macOS-arm64 but attempting to link with file built for macOS-x86_64
ld: warning: ignoring file /private/var/.../installed/openssl-OovnrH2WrHG18y7xi0irAegZLOlNsDyeOGb8BBEcoGw/lib/libcrypto.dylib, building for macOS-arm64 but attempting to link with file built for macOS-x86_64
...
```
After the change build succeeds.
X-link: https://github.com/facebook/watchman/pull/1032
Reviewed By: fanzeyi
Differential Revision: D37137060
Pulled By: genevievehelsel
fbshipit-source-id: a8381c39821bbd29ddea1d1009b94f5ed6cacb21
Summary: Adding patching support for all platforms using `git apply`
Reviewed By: xavierd
Differential Revision: D36939728
fbshipit-source-id: ccf6e4ccd30257c6928ba804a76882d4d2f71700
Summary:
Without ctest installed, `getdeps.py test $project` would produce a
bad error message about failing to run a `''` command. Instead,
explicitly error early when a command is required.
Reviewed By: genevievehelsel
Differential Revision: D36535929
fbshipit-source-id: d300c6b1b0c124b56dffffae0a71bee9b7f12fe7
Summary:
Watchman's vagrant images rely on mapping a watchman source directory
fetched from fbsource into the VM. Thus, they sometimes look like
fbsource, and that's largely okay. However, the vendored cargo support
breaks. Instead, explicitly skip vendored cargo when there is no
fbsource dir.
Reviewed By: xavierd
Differential Revision: D36386674
fbshipit-source-id: 61f2af19507fecd2342cfc94bbb7120ab91c33b4
Summary:
Applies new import merging and sorting from µsort v1.0.
When merging imports, µsort will make a best-effort to move associated
comments to match merged elements, but there are known limitations due to
the diynamic nature of Python and developer tooling. These changes should
not produce any dangerous runtime changes, but may require touch-ups to
satisfy linters and other tooling.
Note that µsort uses case-insensitive, lexicographical sorting, which
results in a different ordering compared to isort. This provides a more
consistent sorting order, matching the case-insensitive order used when
sorting import statements by module name, and ensures that "frog", "FROG",
and "Frog" always sort next to each other.
For details on µsort's sorting and merging semantics, see the user guide:
https://usort.readthedocs.io/en/stable/guide.html#sorting
Reviewed By: lisroach
Differential Revision: D36402214
fbshipit-source-id: b641bfa9d46242188524d4ae2c44998922a62b4c
Summary:
This adds the ability to specify a patch file to be applied on top of any library in getdeps.
For example, zlib doesn't support static linking with CMake, but there's a small patch that can be applied to support static linking. This is the specific use case this diff is intended to support.
Reviewed By: bigfootjon
Differential Revision: D35410512
fbshipit-source-id: d1af0ddf9ec45ef28aa902c06735af86817ac194
Summary: Since Bistro is slated to be deprecated, let us turn off OSS CI for it.
Reviewed By: AnonymousMonkey2021
Differential Revision: D35347845
fbshipit-source-id: be23bcfaa098f77162f27ecfed6469b8db7d3401
Summary:
If getdeps tests run from fbcode, tpx guesses that the test type is FBCODE rather than FBTEST. This makes UTF ignore the buildsystem=getdeps config, which leads tests to be ignored.
This diff sets the test type explicitly.
More context in [this post](https://fb.workplace.com/groups/tpx.users/permalink/1309289352870499/).
Reviewed By: sandeepkumarpani888
Differential Revision: D34271402
fbshipit-source-id: 2c6c37d706f22d214cd2f7622a7124b4c3f92ba0
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/110
Pull Request resolved: https://github.com/facebookexperimental/rust-shed/pull/27
Make it so that changes to rust-shed or other common rust source are used locally vendored, so they don't need to be pushed to github before they are visible in a build.
There was already some support for cargo vendoring in getdeps, but it was limited to dependencies between manifests built with cargo builder. This wasn't enough to build something like eden (cmake is main entry point, with later calls cargo) or eden_scm (make is main entry point, with later calls to cargo), so this diff adds a cargo prepare step for getdeps other primary build systems.
The cargo vendoring is done by using a cargo config file to point to the source files used by getdeps. It has two modes:
1. per crate, existing mode which is already automatic for cargo to cargo manifest dependencies. To use it for a non cargo build manifest, add crate.pathmap
2. per git url, existing mode which was only use for crates.io third-party crates, now can be enabled by setting cargo.cargo_config_file
Reviewed By: yancouto
Differential Revision: D33895469
fbshipit-source-id: 7b13c0b679532492a336ce217de875c25fe1be90
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/106
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/107
Summary
* Add EdenFS builds on external CI now EdenSCM is good
* Mac builds on github actions by using brew for system dependencies
To make this work had to fix some path ordering issues with install directories for Linux and Mac, and generalise the homebrew path fixups we were doing for bison to all the used homebrew packages.
Previously Installed packages were being added after system paths, so our own installed thing might be ignored. On github these meant system python 3.9 was being used for hg tests rather than our specified 3.8 (this showed we have some test fails on python 3.9 with "SystemError: deallocated bytearray object has exported buffers", that are beyond the scope of this diff to fix)
Also needed to include the getdeps generated python into the generated edenscmdeps3.zip archive setup.py produces otherwise EdenFS tests failed to import thrift.Thrift
Eden tests are hanging when run externally about half way through, so disable them on github actions for now as this PR is already fairly large. They work when run locally on an internal devserver, so probably some bit of environment necessary is not defined in the test runner
Reviewed By: chadaustin
Differential Revision: D34116505
fbshipit-source-id: d0d628db5daabc28d0bd8997cd5c1bc885ed1e73
Summary:
Watchman advertises support for FreeBSD but it's regressed a couple
ways. Ensure it builds on FreeBSD again with a Vagrantfile.
Reviewed By: ahornby
Differential Revision: D33989289
fbshipit-source-id: ff906bc219974bafbc349740ce6bddfe5bd2908f
Summary: Add some basic support for FreeBSD to getdeps.
Reviewed By: ahornby
Differential Revision: D33989129
fbshipit-source-id: 42ff5f160b7e19c12196bb2e52a726f7815487bd
Summary:
The new boost has an MSVC detection algorithm that 1) I don't
understand and 2) does not even seem deterministic. Since our internal
builds are pinned to vc142, force that in the manifest.
Reviewed By: fanzeyi
Differential Revision: D34085081
fbshipit-source-id: d8ce4d8e28aad14f6fa60f9227dafb546ce321e5
Summary: Boost 1.69 does not build on Fedora 35, so upgrade to Boost 1.78.
Reviewed By: ahornby
Differential Revision: D33855670
fbshipit-source-id: 85469a835a1dab1a7d5222413b1f1349bdcff280
Summary:
A long time ago, getdeps scheduled each build up to the number of hardware threads. For some builds, that was too heavy, so it got throttled back to only ever use half the hardware threads. This left parallelism on the table for CPU-bound projects that don't use much RAM per compilation.
This commit makes better use of the hardware with finer-grained logic that allows each manifest to specify a `job_weight_mib` estimate in MiB, and limit concurrency to `available_ram / job_weight`.
Reviewed By: ahornby
Differential Revision: D33754018
fbshipit-source-id: 785bed6c6cfe3c473244e0806a77cec1fc119e1f
Summary:
Move cargo logic to separate file as builder.py was getting a bit large
It's just a move of source file for CargoBuilder. Diff looks a bit big as I did hg cp to preserve history
Differential Revision: D33888925
fbshipit-source-id: 2d57343535ab087e09876edba4d00a6f0234fcd0
Summary:
Now that we have some type annotations, we might as well support
running the type checker.
Reviewed By: xavierd
Differential Revision: D33715757
fbshipit-source-id: baf693e4b2415e0e1aa50b569b744ca0cfb91337
Summary: Use the new tpx option to run tests locally on the OSS CI hosts
Reviewed By: bigfootjon
Differential Revision: D33454014
fbshipit-source-id: dcf49dcebcd476c68b3eb46eb013079c30db52c7
Summary:
This diffs adds
* `--shared-libs` command line argument to the `build` command which enables building shared libraries for supported projects (cmake-based projects, boost, libevent)
* this flag overrides `BUILD_SHARED_LIBS` cmake flags in manifest files and from `--extra-cmake-defines`
* adds `shared_libs=on` expression support in manifest files
* for boost, the flag enables building shared libraries **in addition to** statically linked ones
Reviewed By: simonmar
Differential Revision: D27462289
fbshipit-source-id: d22ab434f7228c30472611bc323830d88efba0a5
Summary:
A number of places were extracting dependencies from manifests, but only one was adding in the implicit dependencies for build tools.
Extract the logic to one place and use so that a change in a tool like cmake will now correctly affect all tools using cmake, as it will be taken into account as a dependency hash when the manifest's hash is computed.
Tests for this change revealed that install_dirs needed to be populated in reverse order from the manifest topo-sort, so have also addressed that
Reviewed By: wittgenst
Differential Revision: D32730717
fbshipit-source-id: 1b2a25e460de6085d274c99acfd391b3bd259264
Summary:
Update to a newer python that builds on macOS Big Sur, make eden_scm depend on it and use it from PATH
python 3.8 requires libffi which is reference via its pkg-config setup, however python's pkg-config libffi detection is broken (https://bugs.python.org/issue34823) with the documented workaround requiring an environment variable to be passed into its ./configure step, which is why this change also adds a feature to AutoconfBuilder
With the updated python in place I was able to remove disable_env_override_pkgconfig = 1
disable_env_override_path = 1 from the eden_scm config so that it actually uses the pkg-configs, PATHs and other settings from its dependencies. This should make future python and other dependency upgrades much simpler.
Reviewed By: HarveyHunt
Differential Revision: D32231261
fbshipit-source-id: a2b6addbe22f38e3d71618c802d2c6f836fdd86c
Summary: Make it faster, the build can run in parallel. Unfortunately install can't but its pretty quick anyway.
Reviewed By: Croohand
Differential Revision: D32608049
fbshipit-source-id: 6c86184993a065e29f95df658f1b50ba563a5b14
Summary:
rpm names needed updating as boost-contract wasn't found.
The rpm names now include the boost version which should be handy when boost dependency is updated next
Unfortunately the centos boost 1.69 rpms aren't found automatically by the centos cmake boost rules, so this also adds builder.py logic to point to it using BOOST_INCLUDEDIR and BOOST_LIBRARYDIR
Reviewed By: mzr
Differential Revision: D32140834
fbshipit-source-id: 3e2dd822613957ae4d7be5b73623ff581f11d02b
Summary:
For whatever reason, sc_testpilot default to --return-zero-on-failures, which
means that test failures are simply not reported properly to the signal box.
Fix that by explicitely passing --return-nonzero-on-failures
Reviewed By: fanzeyi
Differential Revision: D29509158
fbshipit-source-id: ef991f91df48e99769f96ffd8d7015cdf507c723
Summary: Hack to use parexec and the par directly.
Reviewed By: chadaustin
Differential Revision: D29461444
fbshipit-source-id: 9ac6c1aa43728782b8de0a71446109f7fd5dab1d
