Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/995
Bring back mononoke getdeps integration tests. This is a refesh of the previously working export-D34186407 branch from 2022
Main changes since:
* depend on restored sapling manifest from previous commit
* bring back selected manifests needed by mononoke that were deleted in D51869247. I added the sqlite binary packages from this to the sqlite3 manifest rather than bring back the sqllite3-bin manifest
* add manifests for new tools used: ripgrep, git-lfs, and zstd cli (we already had the zstd libs)
* fix a few test expecations that where too closely tied to git cli or TLS version etc (ubuntu 22.04 is on older version)
* getdeps MakefileBuilder.run_tests() improvements
* fix error status reporting, was not failing if tests failed
* pass of --num-jobs to tests to stop it OOMing my machine
* pass of --filter to tests so can iterate on one test more easily
Can to iterate local execution for one test with:
```
python3 ./build/fbcode_builder/getdeps.py test --num-jobs 4 --allow-system-packages --no-facebook-internal --src-dir=. mononoke_integration --retry 0 --filter server/test-gettreepack.t
```
X-link: https://github.com/facebook/sapling/pull/951
Reviewed By: quark-zju
Differential Revision: D62978526
Pulled By: ahornby
fbshipit-source-id: 0070a67d798bb23ee9e78e1a5149ba5364d548c9
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/907
I don't sure that I make all in accordance with the your contribution pipeline, so please correct me If there is needed.
So, I've made some changes to support the `pacman` package manager in your `getdeps.py` script. In `manifests` I'm also duplicated some packages from `debs` and `rpms` sections and create a new `pps` sections with according packages for `pacman`.
Issue: https://github.com/facebook/folly/issues/1701
X-link: https://github.com/facebook/folly/pull/1702
Reviewed By: yfeldblum
Differential Revision: D33514769
Pulled By: Orvid
fbshipit-source-id: a081c3a5bcb7f7cdde3a4a91c0d15517c0a171b3
Summary:
In order to speed up build times on a mac, add homebrew support to
getdeps.
Homebrew packages can be declared in a manifest using the `homebrew` header.
Futher, ahornby has added manifest entries for homebrew packages which are
included in this diff and also included a change to use the correct version of
openssl. Without this openssl change, homebrew cmake configure finds an old
openssl 1.0.2 install.
This diff provides a 2x speed up for building folly:
Timings for clean getdeps folly build on mid-2018 2.9Ghz i9 6 core intel macbook pro with 32GB RAM:
With new homebrew system deps:
```
rm -rf /Users/ahornby/.scratch/UsersZahornbyZfbsource/fbcode_builder_getdeps/
time ./opensource/fbcode_builder/getdeps.py build --allow-system-packages folly
real 17m39.329s
user 76m10.317s
sys 5m50.163s
```
Without:
```
rm -rf /Users/ahornby/.scratch/UsersZahornbyZfbsource/fbcode_builder_getdeps/
time ./opensource/fbcode_builder/getdeps.py build folly
real 32m10.344s
user 105m53.448s
sys 15m57.858s
```
Reviewed By: ahornby
Differential Revision: D33842632
fbshipit-source-id: ac785d4a8dcfa31b77292bddd9e747022ac36e3b
Summary:
From the outset, we wanted to be sure that getdeps was able
to source and build the dependencies so that we knew that we'd have
a repeatable build. This came at the cost of build times: having
to build boost on each CI run is a bit of a chore.
This commit adds three new elements to the manifest files:
* `rpms` - a list of RPM names that are all required to be present
in order to consider the dependency satisfied
* `debs` - like `rpms` above, but scoped to debian package names
* `preinstalled.env` - a list of environment variables that if they
are all set and non-empty will satisfy the dependency.
A new `--allow-system-packages` option to getdeps enables the new
logic that looks for system packages; it is off by default, but
enabled in the generated GitHub Actions workflows.
A new `install-system-deps` subcommand is provided that will attempt
to install the system packages needed to satisfy the build. This
typically needs to be run via sudo and is thus broken out separately
from the main getdeps build flow.
I made a pass over the manifest files and added package names that
satisfy the build on ubuntu-18 and fedora-31.
shri-khare: I renamed the `Python3.7.6` manifest to just `python` as
part of this change; the version of python that it pulls in through
the normal build is the same and I believe that an equal or newer
version of python3 is available in the GH actions builder.
The `preinstalled.env` is used only by the boost manifest: it references
the name of an environment variable that is set by the github
windows hosts and that points to a pre-built and pre-installed
copy of boost. Since there is no package manager that we can
easily query for this sort of thing, probing from the environment
seems like a reasonable and fast way to check for this. We
may need to evolve this over time to become more feature rich,
but this seems like a good starting point.
This commit has the potential to save 20 minutes of build time
from each public CI build just due to the boost dependency alone!
Refs: https://github.com/facebook/watchman/pull/797
Reviewed By: yfeldblum
Differential Revision: D20740410
fbshipit-source-id: 6c38019449c54465127656c3d18a6ff1f30adaea
Summary:
Previously, we were using autoconf to build sqlite,
but that isn't available on Windows. Instead, here's a builder
that generates a little cmake configuration for building and
installing sqlite.
Using cmake for this means that we can test the same builder
on all platforms that need to pull in sqlite.
Reviewed By: simpkins
Differential Revision: D15179387
fbshipit-source-id: fccf969182565982bd5be55545a2d2625aa99124
Summary:
These are ported over from the logic in the watchman and eden getdeps
scripts, with additions to help bootstrap a build environment.
These are sufficient to build watchman with thrift support on windows, mac and
linux, and eden on mac and linux when combined with the getdeps code that
follows in later diffs in this stack.
Reviewed By: simpkins
Differential Revision: D14691005
fbshipit-source-id: 7f8b02fedcdc020e2d0e758c466959d8161d4587
