Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/907
I don't sure that I make all in accordance with the your contribution pipeline, so please correct me If there is needed.
So, I've made some changes to support the `pacman` package manager in your `getdeps.py` script. In `manifests` I'm also duplicated some packages from `debs` and `rpms` sections and create a new `pps` sections with according packages for `pacman`.
Issue: https://github.com/facebook/folly/issues/1701
X-link: https://github.com/facebook/folly/pull/1702
Reviewed By: yfeldblum
Differential Revision: D33514769
Pulled By: Orvid
fbshipit-source-id: a081c3a5bcb7f7cdde3a4a91c0d15517c0a171b3
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/909
Linux Mint is based on Ubuntu and installs correctly with the same options. This adds it to the supported versions so that running `sudo ./install-system-packages.sh` succeeds.
X-link: https://github.com/facebook/watchman/pull/1234
Reviewed By: chadaustin
Differential Revision: D59666084
Pulled By: genevievehelsel
fbshipit-source-id: cfdee239be6358d4c8e2f1154391159a40ef91f4
Summary:
M1 macbooks appear to have a default system Python3 version of 3.8.x, with a "universal" executable that incorrectly reports the platform (`platform.machine()`) as `x86_64`. This causes Watchman's `./autogen.sh` which uses `getdeps.py build` to try to build for `x86_64`, which eventually causes linker errors with pre-installed dependencies eg `building for macOS-x86_64 but attempting to link with file built for macOS-arm64`. See also https://github.com/pyinstaller/pyinstaller/issues/5315
PR: https://github.com/facebook/watchman/pull/1081 attempted to resolve by noting a higher python version where platform.machine() does the right thing.
We can work around this though by using `os.uname()` which correctly
distinguishes intel and M1.
```
> python3
>>> platform.machine()
'x86_64'
>>> os.uname().version
'Darwin Kernel Version 21.6.0: Thu Sep 29 20:13:56 PDT 2022; root:xnu-8020.240.7~1/RELEASE_ARM64_T6000'
```
Reviewed By: mshroyer
Differential Revision: D42182936
fbshipit-source-id: 0d0484a86600da102450e9035c621c01d250cbf7
Summary:
Occasionally we see reports [1] that /proc/meminfo is missing or has
unexpected values on Linux. Instead of failing the build, guess a
reasonable value, at the risk that this will unnecessarily limit the
build's concurrency.
[1] https://github.com/facebook/watchman/issues/1040
Reviewed By: genevievehelsel
Differential Revision: D38126612
fbshipit-source-id: 9d9d9f6003703acf6dffcdd5b2022f0f7b3aa691
Summary:
This function is expected to return a tuple of 3 elements. This doesn't happen when it fails opening the `/etc/os-release` file.
X-link: https://github.com/facebook/watchman/pull/1025
Reviewed By: fanzeyi
Differential Revision: D36736667
Pulled By: chadaustin
fbshipit-source-id: a6839235d6c5ebb50267cd78f1c9590b2fb180d7
Summary: Add some basic support for FreeBSD to getdeps.
Reviewed By: ahornby
Differential Revision: D33989129
fbshipit-source-id: 42ff5f160b7e19c12196bb2e52a726f7815487bd
Summary:
In order to speed up build times on a mac, add homebrew support to
getdeps.
Homebrew packages can be declared in a manifest using the `homebrew` header.
Futher, ahornby has added manifest entries for homebrew packages which are
included in this diff and also included a change to use the correct version of
openssl. Without this openssl change, homebrew cmake configure finds an old
openssl 1.0.2 install.
This diff provides a 2x speed up for building folly:
Timings for clean getdeps folly build on mid-2018 2.9Ghz i9 6 core intel macbook pro with 32GB RAM:
With new homebrew system deps:
```
rm -rf /Users/ahornby/.scratch/UsersZahornbyZfbsource/fbcode_builder_getdeps/
time ./opensource/fbcode_builder/getdeps.py build --allow-system-packages folly
real 17m39.329s
user 76m10.317s
sys 5m50.163s
```
Without:
```
rm -rf /Users/ahornby/.scratch/UsersZahornbyZfbsource/fbcode_builder_getdeps/
time ./opensource/fbcode_builder/getdeps.py build folly
real 32m10.344s
user 105m53.448s
sys 15m57.858s
```
Reviewed By: ahornby
Differential Revision: D33842632
fbshipit-source-id: ac785d4a8dcfa31b77292bddd9e747022ac36e3b
Summary:
A long time ago, getdeps scheduled each build up to the number of hardware threads. For some builds, that was too heavy, so it got throttled back to only ever use half the hardware threads. This left parallelism on the table for CPU-bound projects that don't use much RAM per compilation.
This commit makes better use of the hardware with finer-grained logic that allows each manifest to specify a `job_weight_mib` estimate in MiB, and limit concurrency to `available_ram / job_weight`.
Reviewed By: ahornby
Differential Revision: D33754018
fbshipit-source-id: 785bed6c6cfe3c473244e0806a77cec1fc119e1f
Summary: Missed this when switching from spaces to underscores in distro names
Reviewed By: krallin
Differential Revision: D32693527
fbshipit-source-id: 658504d76c96defd3f1a217ada8b05f99c841ede
Summary: I don't think the spaces were breaking anything, but many years on unix means they set my spidey sense tingling when looking at the build logs.
Reviewed By: StanislavGlebik
Differential Revision: D32619583
fbshipit-source-id: 8d5bc35f085faced20457645f1c1430371a3c578
Summary:
Distro name wasn't matching and thus no rpms being installed or used
Also improved the error message for unknown systems
Reviewed By: farnz
Differential Revision: D32139850
fbshipit-source-id: 438278094159f732f6b1de506490b89d2c9cd3f7
Summary:
Some changes that had to be done in Debian for the build to work:
* Installation of system deps failed as it was not detecting the OS correctly
* Updated the CMake manifest to uncomment the deb dependency as the CMake versions are newer
* fb303 was not getting cloned as it was looking for a non-existent master branch
Pull Request resolved: https://github.com/facebook/watchman/pull/943
Differential Revision: D30700180
Pulled By: fanzeyi
fbshipit-source-id: 62eb9f8ba57510eb98d6bd56e40384bf38a74d8c
Summary:
Description:
Prior to this patch it was not possible to run open/r on linux running on ARM due to cmake and openssl being harded to download/compile for the x86_64 arch.
In order to prevent such problem this patch actively compiles cmake instead of using pre-compiled binaries and also allows the OpenSSLBuilder to provide the correct build args to openssl, thus not trying to compile or run x86_64 software.
Pull Request resolved: https://github.com/facebook/openr/pull/95
Test Plan: * built the project by using ./build/build_openr.sh
Reviewed By: wez
Differential Revision: D28224684
Pulled By: cooperlees
fbshipit-source-id: 9de61dc6d7dcf7116ec5c67f3f165cd4a4bb5e5c
Summary:
From the outset, we wanted to be sure that getdeps was able
to source and build the dependencies so that we knew that we'd have
a repeatable build. This came at the cost of build times: having
to build boost on each CI run is a bit of a chore.
This commit adds three new elements to the manifest files:
* `rpms` - a list of RPM names that are all required to be present
in order to consider the dependency satisfied
* `debs` - like `rpms` above, but scoped to debian package names
* `preinstalled.env` - a list of environment variables that if they
are all set and non-empty will satisfy the dependency.
A new `--allow-system-packages` option to getdeps enables the new
logic that looks for system packages; it is off by default, but
enabled in the generated GitHub Actions workflows.
A new `install-system-deps` subcommand is provided that will attempt
to install the system packages needed to satisfy the build. This
typically needs to be run via sudo and is thus broken out separately
from the main getdeps build flow.
I made a pass over the manifest files and added package names that
satisfy the build on ubuntu-18 and fedora-31.
shri-khare: I renamed the `Python3.7.6` manifest to just `python` as
part of this change; the version of python that it pulls in through
the normal build is the same and I believe that an equal or newer
version of python3 is available in the GH actions builder.
The `preinstalled.env` is used only by the boost manifest: it references
the name of an environment variable that is set by the github
windows hosts and that points to a pre-built and pre-installed
copy of boost. Since there is no package manager that we can
easily query for this sort of thing, probing from the environment
seems like a reasonable and fast way to check for this. We
may need to evolve this over time to become more feature rich,
but this seems like a good starting point.
This commit has the potential to save 20 minutes of build time
from each public CI build just due to the boost dependency alone!
Refs: https://github.com/facebook/watchman/pull/797
Reviewed By: yfeldblum
Differential Revision: D20740410
fbshipit-source-id: 6c38019449c54465127656c3d18a6ff1f30adaea
Summary:
Add a ContextGenerator class so that we actually use the correct per-project
context when loading projects and computing dependencies.
Previously commands like `build` and `test` would change the contexts for each
project as they iterated through and performed the build. However, they did
not do this when first loading the projects. This could cause them to use
different context values when loading dependencies than when performing the
build. For instance, this could cause issues if a project depends on
`googletest` only when testing is enabled, as the code previously did not set
the "test" parameter when evaluating dependencies.
Reviewed By: pkaush
Differential Revision: D16477396
fbshipit-source-id: c1e055f07de1cb960861d19594e3bda20a2ccd87
Summary:
with the reshuffling around getdeps.py, and changes in folly/thrift/wangle,
our internal package stopped building. This diff brings it back to life. It is
a little bit interwined:
* Introduce a `--facebook-internal` switch to getdeps that causes `fb` in the
build context to be set to `on` rather than `off`. This allows specifying
options in the manifest that are FB specific.
* Introduce cmake options to control the WATCHMAN_STATE_DIR and whether cmake
should attempt to create WATCHMAN_STATE_DIR when it is installed.
* For FB specific builds on macOS, set the state dir to match our existing location
and disable creation of WATCHMAN_STATE_DIR because that will ultimately be
handling during package installation and we don't have privs for that on
the machine generating the package.
* Adjust Facebook CI scripts to use getdeps for building and testing watchman
Reviewed By: simpkins
Differential Revision: D15337970
fbshipit-source-id: 8ab3c1f810decf5f0b16337dc1cbe6dc73d26426
Summary:
While the command isn't necessarily super useful
on its own, it does show that the plumbing for walking the
deps is functioning, and that is important when it comes
to building.
The output lists the projects in the order that they
would be built.
The `fetch` command has been augmented to add a `--recursive`
flag that uses the same mechanism to recursively fetch
the dependencies.
Reviewed By: simpkins
Differential Revision: D14691004
fbshipit-source-id: b00bf6ad4742f8bb0a70698f71a5fe03d6a1f453
Summary:
This will feed into the manifest context for system
dependent manifest sections. This is essentially the same
code borrowed from the watchman and eden getdeps.py.
Reviewed By: simpkins
Differential Revision: D14690997
fbshipit-source-id: 3d3ae146237a2cd49609aaa2bd0e785ebe21f02c
