Summary: Since Bistro is slated to be deprecated, let us turn off OSS CI for it.
Reviewed By: AnonymousMonkey2021
Differential Revision: D35347845
fbshipit-source-id: be23bcfaa098f77162f27ecfed6469b8db7d3401
Summary:
If getdeps tests run from fbcode, tpx guesses that the test type is FBCODE rather than FBTEST. This makes UTF ignore the buildsystem=getdeps config, which leads tests to be ignored.
This diff sets the test type explicitly.
More context in [this post](https://fb.workplace.com/groups/tpx.users/permalink/1309289352870499/).
Reviewed By: sandeepkumarpani888
Differential Revision: D34271402
fbshipit-source-id: 2c6c37d706f22d214cd2f7622a7124b4c3f92ba0
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/110
Pull Request resolved: https://github.com/facebookexperimental/rust-shed/pull/27
Make it so that changes to rust-shed or other common rust source are used locally vendored, so they don't need to be pushed to github before they are visible in a build.
There was already some support for cargo vendoring in getdeps, but it was limited to dependencies between manifests built with cargo builder. This wasn't enough to build something like eden (cmake is main entry point, with later calls cargo) or eden_scm (make is main entry point, with later calls to cargo), so this diff adds a cargo prepare step for getdeps other primary build systems.
The cargo vendoring is done by using a cargo config file to point to the source files used by getdeps. It has two modes:
1. per crate, existing mode which is already automatic for cargo to cargo manifest dependencies. To use it for a non cargo build manifest, add crate.pathmap
2. per git url, existing mode which was only use for crates.io third-party crates, now can be enabled by setting cargo.cargo_config_file
Reviewed By: yancouto
Differential Revision: D33895469
fbshipit-source-id: 7b13c0b679532492a336ce217de875c25fe1be90
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/106
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/107
Summary
* Add EdenFS builds on external CI now EdenSCM is good
* Mac builds on github actions by using brew for system dependencies
To make this work had to fix some path ordering issues with install directories for Linux and Mac, and generalise the homebrew path fixups we were doing for bison to all the used homebrew packages.
Previously Installed packages were being added after system paths, so our own installed thing might be ignored. On github these meant system python 3.9 was being used for hg tests rather than our specified 3.8 (this showed we have some test fails on python 3.9 with "SystemError: deallocated bytearray object has exported buffers", that are beyond the scope of this diff to fix)
Also needed to include the getdeps generated python into the generated edenscmdeps3.zip archive setup.py produces otherwise EdenFS tests failed to import thrift.Thrift
Eden tests are hanging when run externally about half way through, so disable them on github actions for now as this PR is already fairly large. They work when run locally on an internal devserver, so probably some bit of environment necessary is not defined in the test runner
Reviewed By: chadaustin
Differential Revision: D34116505
fbshipit-source-id: d0d628db5daabc28d0bd8997cd5c1bc885ed1e73
Summary:
Watchman advertises support for FreeBSD but it's regressed a couple
ways. Ensure it builds on FreeBSD again with a Vagrantfile.
Reviewed By: ahornby
Differential Revision: D33989289
fbshipit-source-id: ff906bc219974bafbc349740ce6bddfe5bd2908f
Summary: Add some basic support for FreeBSD to getdeps.
Reviewed By: ahornby
Differential Revision: D33989129
fbshipit-source-id: 42ff5f160b7e19c12196bb2e52a726f7815487bd
Summary:
The new boost has an MSVC detection algorithm that 1) I don't
understand and 2) does not even seem deterministic. Since our internal
builds are pinned to vc142, force that in the manifest.
Reviewed By: fanzeyi
Differential Revision: D34085081
fbshipit-source-id: d8ce4d8e28aad14f6fa60f9227dafb546ce321e5
Summary: Boost 1.69 does not build on Fedora 35, so upgrade to Boost 1.78.
Reviewed By: ahornby
Differential Revision: D33855670
fbshipit-source-id: 85469a835a1dab1a7d5222413b1f1349bdcff280
Summary:
A long time ago, getdeps scheduled each build up to the number of hardware threads. For some builds, that was too heavy, so it got throttled back to only ever use half the hardware threads. This left parallelism on the table for CPU-bound projects that don't use much RAM per compilation.
This commit makes better use of the hardware with finer-grained logic that allows each manifest to specify a `job_weight_mib` estimate in MiB, and limit concurrency to `available_ram / job_weight`.
Reviewed By: ahornby
Differential Revision: D33754018
fbshipit-source-id: 785bed6c6cfe3c473244e0806a77cec1fc119e1f
Summary:
Move cargo logic to separate file as builder.py was getting a bit large
It's just a move of source file for CargoBuilder. Diff looks a bit big as I did hg cp to preserve history
Differential Revision: D33888925
fbshipit-source-id: 2d57343535ab087e09876edba4d00a6f0234fcd0
Summary:
Now that we have some type annotations, we might as well support
running the type checker.
Reviewed By: xavierd
Differential Revision: D33715757
fbshipit-source-id: baf693e4b2415e0e1aa50b569b744ca0cfb91337
Summary: Use the new tpx option to run tests locally on the OSS CI hosts
Reviewed By: bigfootjon
Differential Revision: D33454014
fbshipit-source-id: dcf49dcebcd476c68b3eb46eb013079c30db52c7
Summary:
This diffs adds
* `--shared-libs` command line argument to the `build` command which enables building shared libraries for supported projects (cmake-based projects, boost, libevent)
* this flag overrides `BUILD_SHARED_LIBS` cmake flags in manifest files and from `--extra-cmake-defines`
* adds `shared_libs=on` expression support in manifest files
* for boost, the flag enables building shared libraries **in addition to** statically linked ones
Reviewed By: simonmar
Differential Revision: D27462289
fbshipit-source-id: d22ab434f7228c30472611bc323830d88efba0a5
Summary:
A number of places were extracting dependencies from manifests, but only one was adding in the implicit dependencies for build tools.
Extract the logic to one place and use so that a change in a tool like cmake will now correctly affect all tools using cmake, as it will be taken into account as a dependency hash when the manifest's hash is computed.
Tests for this change revealed that install_dirs needed to be populated in reverse order from the manifest topo-sort, so have also addressed that
Reviewed By: wittgenst
Differential Revision: D32730717
fbshipit-source-id: 1b2a25e460de6085d274c99acfd391b3bd259264
Summary:
Update to a newer python that builds on macOS Big Sur, make eden_scm depend on it and use it from PATH
python 3.8 requires libffi which is reference via its pkg-config setup, however python's pkg-config libffi detection is broken (https://bugs.python.org/issue34823) with the documented workaround requiring an environment variable to be passed into its ./configure step, which is why this change also adds a feature to AutoconfBuilder
With the updated python in place I was able to remove disable_env_override_pkgconfig = 1
disable_env_override_path = 1 from the eden_scm config so that it actually uses the pkg-configs, PATHs and other settings from its dependencies. This should make future python and other dependency upgrades much simpler.
Reviewed By: HarveyHunt
Differential Revision: D32231261
fbshipit-source-id: a2b6addbe22f38e3d71618c802d2c6f836fdd86c
Summary: Make it faster, the build can run in parallel. Unfortunately install can't but its pretty quick anyway.
Reviewed By: Croohand
Differential Revision: D32608049
fbshipit-source-id: 6c86184993a065e29f95df658f1b50ba563a5b14
Summary:
rpm names needed updating as boost-contract wasn't found.
The rpm names now include the boost version which should be handy when boost dependency is updated next
Unfortunately the centos boost 1.69 rpms aren't found automatically by the centos cmake boost rules, so this also adds builder.py logic to point to it using BOOST_INCLUDEDIR and BOOST_LIBRARYDIR
Reviewed By: mzr
Differential Revision: D32140834
fbshipit-source-id: 3e2dd822613957ae4d7be5b73623ff581f11d02b
Summary:
For whatever reason, sc_testpilot default to --return-zero-on-failures, which
means that test failures are simply not reported properly to the signal box.
Fix that by explicitely passing --return-nonzero-on-failures
Reviewed By: fanzeyi
Differential Revision: D29509158
fbshipit-source-id: ef991f91df48e99769f96ffd8d7015cdf507c723
Summary: Hack to use parexec and the par directly.
Reviewed By: chadaustin
Differential Revision: D29461444
fbshipit-source-id: 9ac6c1aa43728782b8de0a71446109f7fd5dab1d
Summary:
The testpilot command line getdeps generated was not suitable for tpx
when filtering. tpx would consider the passed test filter an
environment variable.
Reorder a few things to at least make filtering work.
Reviewed By: fanzeyi
Differential Revision: D28976061
fbshipit-source-id: 21c45b3a4a59711a2a5da36a1bd11b7b4871ec5d
Summary:
Description:
Prior to this patch it was not possible to run open/r on linux running on ARM due to cmake and openssl being harded to download/compile for the x86_64 arch.
In order to prevent such problem this patch actively compiles cmake instead of using pre-compiled binaries and also allows the OpenSSLBuilder to provide the correct build args to openssl, thus not trying to compile or run x86_64 software.
Pull Request resolved: https://github.com/facebook/openr/pull/95
Test Plan: * built the project by using ./build/build_openr.sh
Reviewed By: wez
Differential Revision: D28224684
Pulled By: cooperlees
fbshipit-source-id: 9de61dc6d7dcf7116ec5c67f3f165cd4a4bb5e5c
Summary:
The migration to tpx broke Watchman's tests because test_bser relies on YARN_YARN_OFFLINE_MIRROR being set in the environment and tpx doesn't forward environment variables into the test.
Explicitly forward them ourselves.
Reviewed By: fanzeyi
Differential Revision: D27897172
fbshipit-source-id: 16c8017a89979802bd9d443825ed4e22cb6ff6c9
Summary:
Tpx works well for all OSS jobs, we should migrate to Tpx instead of
relying testpilot.
Reviewed By: marcelogomez
Differential Revision: D27396539
fbshipit-source-id: dd1994be0eb3b4396764c1f3e68ddd6763b60139
Summary:
This took a lot of cmake fuckery. I am guessing it will break the current travis-ci.org based build, since it no longer downloads gflags.
As I put this diff up, mvfst (a dependency of quic (a dependency of proxygen)) is not building right now due to another diff - a fix is in progress but isn't landed yet. I applied the fix locally for the test plan, but didn't include it in the diff.
Reviewed By: snarkmaster
Differential Revision: D25651851
fbshipit-source-id: b4f15ac7fa5293123710fe367c20d355c0970c42
Summary:
`extra_cmake_defines` are extra flags that are passed to cmake when
compiling each one of the dependencies. For instance:
```
$ ./opensource/fbcode_builder/getdeps.py build f4d --extra-cmake-define='{"CMAKE_CXX_FLAGS": "-mavx2 -mfma -mavx -mf16c -march=native"}'
```
It takes a json map as input, which can take a list of defines (key value
pairs).
Reviewed By: wez
Differential Revision: D25855781
fbshipit-source-id: 7f4fef2c66f4d12f23c8d7086d6a4f24fcc01ff7
Summary: During intensive builds using all available cores tends to crash the machine. Cutting default number of jobs by half prevents that.
Reviewed By: wez
Differential Revision: D25027757
fbshipit-source-id: 5ea6c77fe4363b9ee09e50de1a5cc33281872e1d
Summary: As part of the effort to deprecate futures 0.1 in favor of 0.3 I want to create a new futures_ext crate that will contain some of the extensions that are applicable from the futures_01_ext. But first I need to reclame this crate name by renaming the old futures_ext crate. This will also make it easier to track which parts of codebase still use the old futures.
Reviewed By: farnz
Differential Revision: D24725776
fbshipit-source-id: 3574d2a0790f8212f6fad4106655cd41836ff74d
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/67
With this change it will be possible to build dependencies of and run integration tests using getdeps.py.
This is the first goal of Q4 as per https://fb.quip.com/v8YzAYNSYgot: "Get Open Source version of integration tests running on Legocastle".
Before this diff:
The OSS integration tests run now on GitHub by:
- Building some test dependencies with getdeps.py
- Building some test dependencies with homebrew/apt-get
- Running tests via python script
The OSS integration tests were not running on Sandcastle.
After this diff:
The OSS integration tests run on Github by:
- Building and executing tests via getdeps.py (execution of tests happens by getdeps.py calling Make calling python script)
The OSS integration tests run on Sandcastle using the same getdeps.py setup as Github.
Reviewed By: krallin
Differential Revision: D24253268
fbshipit-source-id: cae249b72d076222673b8bbe4ec21866dcdbb253
Summary:
This diff adds all third party dependencies that are required by getdeps to be able to build and runn Mononoke's integration tests.
Also add a stub Makefile with no-op steps that will be filled in next diff.
Reviewed By: ahornby
Differential Revision: D24251894
fbshipit-source-id: 67384ecfd0ced6762dddc3c6e61feb1240b1162d
Summary: If retries is 0, then `0 < 0` is false, meaning we will skip the `while` loop completely and just try to read retcode, which was never assigned.
Reviewed By: fanzeyi
Differential Revision: D23999523
fbshipit-source-id: fac4a1104eba3585fb52fc8d83163cb1a87b8fee
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/51
This diff extends capabilities of CargoBuilder in getdeps so that individual manifests can be build even without workspaces. Thanks to that a build for edenapi/tools can be made and its artifacts can be used in mononoke integration tests.
Reviewed By: StanislavGlebik
Differential Revision: D23574887
fbshipit-source-id: 8a974a6b5235d36a44fe082aad55cd380d84dd09
Summary:
- Added a commandline flag to ignore testpilot even when it's available
- Fixed an oversight that caused getdeps to return 0 even if ctest tests have failed.
Reviewed By: wez
Differential Revision: D23161362
fbshipit-source-id: 1ed97c481315e2b39f1128680386390930194970
Summary: `SDKROOT` is a requirement if we manually specify the location of the compiler on macOS. Otherwise it wouldn't be able to find the system libraries headers.
Reviewed By: wez
Differential Revision: D22577887
fbshipit-source-id: 98140e6f9e564d665db085d21023986b240b3732
Summary:
This diff adds a minimal workflow for running integrations tests for Mononoke. Currently only one test is run and it fails.
This also splits the regular Mononoke CI into separate files for Linux and Mac to match the current style in Eden repo.
There are the "scopeguard::defer" fixes here that somehow escaped the CI tests.
Some tweaks have been made to "integration_runner_real.py" to make it runnable outside FB context.
Lastly the change from using "[[ -v ... ]" to "[[ -n "${...:-}" ]]; in "library.sh" was made because the former is not supported by the default Bash version preinstalled on modern MacOS.
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/26
Reviewed By: krallin
Differential Revision: D22541344
Pulled By: lukaspiatkowski
fbshipit-source-id: 5023d147823166a8754be852c29b1e7b0e6d9f5f
Summary:
Fixes include:
1. Passing "GETDEPS_BUILD_DIR" and "GETDEPS_INSTALL_DIR" env variable and using them in eden/scm/Makefile rather than assuming the source code is always in the same place regardless getdeps arguments (it isn't).
2. Added "fbthrift-source" and "fb303-source" to avoid unnecessary compilation (at least of fb303) and to put fbthrift and fb303 source code in an easy to locate place inside getdeps' "installed" folder.
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/25
Test Plan: sandcastle, check oss-eden_scm-darwin-getdeps
Reviewed By: farnz
Differential Revision: D22431872
Pulled By: lukaspiatkowski
fbshipit-source-id: 8ccbb090713ec085a5dd56df509eb58ab6fb9e34
Summary: This commit adds a flag `--retry` to getdeps and teach it to run retry failed test. This allows us to still pass the tests when there are some flaky tests presents.
Reviewed By: wez
Differential Revision: D22291063
fbshipit-source-id: 572af48a52ceb4a9abbf530cc0154ded0120c0de
Summary:
In order to do what the title says, this diff does:
1. Add the `eden/oss/.../third-party/rust/.../Cargo.toml` files. As mentioned in the previous diff, those are required by GitHub so that the third party dependencies that are local in fbsource are properly defined with a "git" dependency in order for Cargo to "link" crates properly.
2. Changes to `eden/scm/Makefile` to add build/install commands for getdeps to invoke. Those command knowing that they are called from withing getdeps context they link the dependencies brought by getdeps into their proper places that match their folder layout in fbsource. Those Makefile commands also pass a GETDEPS_BUILD env to the setup.py invocations so that it knows it is being called withing a getdeps build.
3. Changes to `eden/scm/setup.py` that add "thriftasset" that makes use of the getdeps.py provided "thrift" binary to build .py files out of thrift files.
4. Changes to `distutils_rust` to use the vendored crates dir provided by getdeps.
5. Changes to `getdeps/builder.py` and `getdeps/manifest.py` that enable more fine-grained configuratior of how Makefile builds are invoked.
6. Changes to `getdeps/buildopts.py` and `getdeps/manifest.py` to disable overriding PATH and pkgconfig env, so that "eden/scm" builds in getdeps using system libraries rather than getdeps-provided ones (NOTE: I've tried to use getdeps provided libraries, but the trickiest bit was that Rust links with Python, which is currently not providable by getdeps, so if you try to build everything the system provided Python libraries will collide with getdeps provided ones)
7. Added `opensource/fbcode_builder/manifests/eden_scm` for the getdeps build.
Reviewed By: quark-zju
Differential Revision: D22336485
fbshipit-source-id: 244d10c9e06ee83de61e97e62a1f2a2184d2312f
