Summary: Fix the getdeps.py generation sub command and regenerate github actions before making changes to support actions generation for eden_scm
Reviewed By: fanzeyi
Differential Revision: D34044243
fbshipit-source-id: 0039d04f25af4c842145dc142dae6b9996fc8046
Summary: Make the --facebook-internal flag value available from BuildOptions so can have jobs of both values where required
Reviewed By: xavierd
Differential Revision: D33945588
fbshipit-source-id: 5d0e1b42973bd3e411a4a71075915bdb2755f0eb
Summary:
In order to speed up build times on a mac, add homebrew support to
getdeps.
Homebrew packages can be declared in a manifest using the `homebrew` header.
Futher, ahornby has added manifest entries for homebrew packages which are
included in this diff and also included a change to use the correct version of
openssl. Without this openssl change, homebrew cmake configure finds an old
openssl 1.0.2 install.
This diff provides a 2x speed up for building folly:
Timings for clean getdeps folly build on mid-2018 2.9Ghz i9 6 core intel macbook pro with 32GB RAM:
With new homebrew system deps:
```
rm -rf /Users/ahornby/.scratch/UsersZahornbyZfbsource/fbcode_builder_getdeps/
time ./opensource/fbcode_builder/getdeps.py build --allow-system-packages folly
real 17m39.329s
user 76m10.317s
sys 5m50.163s
```
Without:
```
rm -rf /Users/ahornby/.scratch/UsersZahornbyZfbsource/fbcode_builder_getdeps/
time ./opensource/fbcode_builder/getdeps.py build folly
real 32m10.344s
user 105m53.448s
sys 15m57.858s
```
Reviewed By: ahornby
Differential Revision: D33842632
fbshipit-source-id: ac785d4a8dcfa31b77292bddd9e747022ac36e3b
Summary: Boost 1.69 does not build on Fedora 35, so upgrade to Boost 1.78.
Reviewed By: ahornby
Differential Revision: D33855670
fbshipit-source-id: 85469a835a1dab1a7d5222413b1f1349bdcff280
Summary:
checkout@v2 is the current action version, and the old one may have
security issues. Let's just run with the latest version.
Reviewed By: ahornby
Differential Revision: D33834847
fbshipit-source-id: fba0733577bf23c208c9824aa3815bd01a813d9a
Summary:
Add support for overriding os, distro and distro version to command line when inspecting system packages so one can requested see ubuntu 18.04 package from other OS. Makes testing easier
Used shlex to shell unquote the value to be tested in the getdeps expression evaluator. getdeps expression parser didn't tolerate 18.04 as . is special char to getdeps expressions, needed to be "18.04"
Reviewed By: quark-zju
Differential Revision: D33741323
fbshipit-source-id: d83397c7fb5180a4d985d0d8ae7b3ff33b72f828
Summary:
This diffs adds
* `--shared-libs` command line argument to the `build` command which enables building shared libraries for supported projects (cmake-based projects, boost, libevent)
* this flag overrides `BUILD_SHARED_LIBS` cmake flags in manifest files and from `--extra-cmake-defines`
* adds `shared_libs=on` expression support in manifest files
* for boost, the flag enables building shared libraries **in addition to** statically linked ones
Reviewed By: simonmar
Differential Revision: D27462289
fbshipit-source-id: d22ab434f7228c30472611bc323830d88efba0a5
Summary:
A number of places were extracting dependencies from manifests, but only one was adding in the implicit dependencies for build tools.
Extract the logic to one place and use so that a change in a tool like cmake will now correctly affect all tools using cmake, as it will be taken into account as a dependency hash when the manifest's hash is computed.
Tests for this change revealed that install_dirs needed to be populated in reverse order from the manifest topo-sort, so have also addressed that
Reviewed By: wittgenst
Differential Revision: D32730717
fbshipit-source-id: 1b2a25e460de6085d274c99acfd391b3bd259264
Summary: Useful to be able to see what packages would be installed without doing it, and to see rpm vs deb packages
Reviewed By: Croohand
Differential Revision: D32693528
fbshipit-source-id: a01257e7192f2f0299d57f6f8f7ee93452a6f3e4
Summary: Update to new version of upload-artifact without branch name as that's a separate repo that our dynamic branch settings should not apply to.
Reviewed By: zpao
Differential Revision: D32717494
fbshipit-source-id: 968908d1072d6d4550705e7df0b3828528e057dc
Summary: Branches updated some time ago, update this to match
Reviewed By: zpao
Differential Revision: D32693531
fbshipit-source-id: 02da79e3fb5fcd3b204be4605521a151b977ce2c
Summary: I found this useful to see where manifests resolve to when they are already built
Reviewed By: HarveyHunt
Differential Revision: D32644308
fbshipit-source-id: b796cb67f154177fbac1093290d97fe5aab6b20c
Summary:
Distro name wasn't matching and thus no rpms being installed or used
Also improved the error message for unknown systems
Reviewed By: farnz
Differential Revision: D32139850
fbshipit-source-id: 438278094159f732f6b1de506490b89d2c9cd3f7
Summary:
Need to fix branch name for OSS mvfst so CI continues on.
Ran opensource/fbcode_builder/getdeps/facebook/update-all-github-actions.sh
Reviewed By: lnicco
Differential Revision: D32041656
fbshipit-source-id: f72aab91a3084814a5d3fa8643ae5f405908c313
Summary: This diff adds support to customize main branch name when generating GitHub Actions.
Differential Revision: D30679305
fbshipit-source-id: 0fc7eb1c97c27e2b42e60cc1ab69a48ab93b93fa
Summary:
This applies the formatting changes from black v21.4b2 to all covered
projects in fbsource. Most changes are to single line docstrings, as black
will now remove leading and trailing whitespace to match PEP8. Any other
formatting changes are likely due to files that landed without formatting,
or files that previously triggered errors in black.
Any changes to code should be AST identical. Any test failures are likely
due to bad tests, or testing against the output of pyfmt.
Reviewed By: thatch
Differential Revision: D28204910
fbshipit-source-id: 804725bcd14f763e90c5ddff1d0418117c15809a
Summary: A show-build-dir subcommand. Possibly useful for building a project and then running only one test or benchmark binary directly.
Reviewed By: wez
Differential Revision: D26502869
fbshipit-source-id: 0d8f6c42a2bbeb2440503a39c7fc770e3b3e4fff
Summary:
`extra_cmake_defines` are extra flags that are passed to cmake when
compiling each one of the dependencies. For instance:
```
$ ./opensource/fbcode_builder/getdeps.py build f4d --extra-cmake-define='{"CMAKE_CXX_FLAGS": "-mavx2 -mfma -mavx -mf16c -march=native"}'
```
It takes a json map as input, which can take a list of defines (key value
pairs).
Reviewed By: wez
Differential Revision: D25855781
fbshipit-source-id: 7f4fef2c66f4d12f23c8d7086d6a4f24fcc01ff7
Summary:
Recently I found that its impossible to access elevated Watchman daemon from non-elevated process on Windows.
```
events.js:174
throw er; // Unhandled 'error' event
^
Error: connect EPERM \\.\pipe\watchman
at PipeConnectWrap.afterConnect [as oncomplete] (net.js:1106:14)
Emitted 'error' event at:
at Socket.<anonymous> (C:\open\ovrsource\unity\socialvr\_js\node_modules\fb-watchman\index.js:118:12)
at Socket.emit (events.js:198:13)
at emitErrorNT (internal/streams/destroy.js:91:8)
at emitErrorAndCloseNT (internal/streams/destroy.js:59:3)
at process._tickCallback (internal/process/next_tick.js:63:19)
```
To fix this, it was suggested by wez to use [his library](https://github.com/wez/EleDo) to force Watchman daemon always start in normal mode on Windows. In this stack of commits I did integrated his library into project and used it to force daemon restart in normal mode when spawned from elevated terminal.
To make it happen, I checked-in library sources and created proxy project which depends on the initial library and contains header bindings and cmake configuration. I did copy pasted Rust cmake macroses from another facebook project - eden, and also created analogue of autogen.sh but for Windows - autogen.cmd.
Pull Request resolved: https://github.com/facebook/watchman/pull/878
Test Plan:
Launch elevated terminal
Start watchman.exe produced from sources
Observe daemon starting and answering
In process monitor, observe watchman.exe process running under user group
Reviewed By: fanzeyi
Differential Revision: D25595879
Pulled By: arhelmus
fbshipit-source-id: 15eb29adcf5bd4a5708b6533a1b2bacbf13f431c
Summary:
- Added a commandline flag to ignore testpilot even when it's available
- Fixed an oversight that caused getdeps to return 0 even if ctest tests have failed.
Reviewed By: wez
Differential Revision: D23161362
fbshipit-source-id: 1ed97c481315e2b39f1128680386390930194970
Summary:
- I find the ability to make a branch and play with CI on GitHub super handy
- Removing the `- master` limitation gives me this ability
Reviewed By: yi-xian
Differential Revision: D22771835
fbshipit-source-id: 8e8839cb860ab4d1dfa0dda590afaf165127f60d
Summary:
- Make OpenR build all deps from source until we remove fbzmq as a dep
- Allow a project to move it's Open Source CI forward to an alternative version of ubuntu via a new `--ubuntu-version` parameter
Reviewed By: wez
Differential Revision: D22768987
fbshipit-source-id: 07205efbd0c87a702638cf30b84a2850d064fa8e
Summary: This commit adds a flag `--retry` to getdeps and teach it to run retry failed test. This allows us to still pass the tests when there are some flaky tests presents.
Reviewed By: wez
Differential Revision: D22291063
fbshipit-source-id: 572af48a52ceb4a9abbf530cc0154ded0120c0de
Summary:
For large projects, with lots of tests, running all the tests can take a lot
of time, but for quick development iteration, only a subset of the tests may
be needed to run.
On non-Windows platforms, this can be easily achieved by manually executing
the individual tests binaries and use the builtin filtering mechanism of that
test binary to achieve the goal. On Windows, this can quickly become
impossible as DLLs might not be available, and the right PATH would need to
be manually specified by hand to execute the tests binaries[0].
To solve this, let's simply provide a unified way of running specific tests
by passing in a regexp. Both testpilot and CTest do support regex to execute
specific tests. My understanding is that cargo doesn't yet allows regex, but
will in the future.
[0]: And a missing DLLs would produce no output when executed from
PowerShell, which makes this very confusing.
Reviewed By: wez
Differential Revision: D21484774
fbshipit-source-id: ee32e950e25bb2a498a2b364a447955a917b0590
Summary:
The breakage has been fixed, so bring back the manifest, but only the Linux one, because the Mac version is failing due to another issue.
Also to make it easier to debug issues on GitHub Actions separate out the dependencies build from Mononoke and rust-shed builds.
Reviewed By: krallin
Differential Revision: D21448412
fbshipit-source-id: 68d89c858d1692727a7fd66bca114920e6dfb4dc
Summary:
D21364132 accidentally broke this; we can't run the fetcher
for projects for which we pulled the build out of cache, because there
is no source to update in that case.
This commit adjusts the logic so that we write out a marker file
to indicate that we installed a build from cache and to look for
that file being present to gate the new update logic.
Reviewed By: lnicco
Differential Revision: D21419122
fbshipit-source-id: 304670848add22531d88549d66f22c40ff255140
Summary:
When the commit hash changed in fbsource, we would correctly decide
that we'd need to rebuild first-party projects but we would incorrectly skip
running the fetcher.update method. This would mean that we'd not perform the
shipit run and that our shipit tree would diverge from the source tree.
This commit resolves this by performing the fetcher.update but ignoring
the source update status in this case.
Reviewed By: xavierd
Differential Revision: D21364131
fbshipit-source-id: b4001e549c7d3f27aa4a21b19893c9bb7c0f6d1f
Summary:
We didn't do a great job of recognizing that we'd need to
build a project when one of its dependencies had changed: we relied
chiefly on the dependency hash for this and could fail to handle
changes in individual source files.
This commit helps to improve this situation by checking to see if
any installed files in the dependencies of a manifest are newer than
the most recent built time of a given manifest. If so, we'll trigger
a build. We try to be reasonably smart about deciding when to trigger
a cmake reconfigure if it looks like cmake files in the deps have
been changed.
Reviewed By: xavierd
Differential Revision: D21364132
fbshipit-source-id: 7534496e10d1f532aa9cf865900ace84a8785327
Summary:
Our linter really wants to include formatting changes unrelated
to my diff stack.
This is a formatting only change to avoid clouding my diffs; no functional
effect.
Reviewed By: xavierd
Differential Revision: D21364519
fbshipit-source-id: 7670dd4154e788f593f256aabdfdeef6d17aeec4
Summary:
On Linux the debug info sections in projects downstream from folly and
thrift are huge due to a lot of C++ template usage.
We build in RelWithDebInfo mode as that is most useful when debugging
locally and because the build times are long, but most casual consumers
of the binaries via GH actions really don't care about debug info,
and this should save ~180MB of size in (for example) the watchman
executables.
Pull Request resolved: https://github.com/facebook/watchman/pull/809
Refs: https://github.com/facebook/watchman/issues/804
Test Plan:
Review the artifact size on the linux build in this PR (see
https://github.com/facebook/watchman/actions/runs/91688952) and confirm that it
is a bit smaller than 180MB; now under 3 MB.
Reviewed By: simpkins
Differential Revision: D21318302
Pulled By: wez
fbshipit-source-id: f78bc5e735dd78771e0604abae64c0a23cf9158d
Summary:
Rather than have a single main.yml file containing all off the different
builds, split that up so that we have one file per build environment
(linux, mac, windows).
This has a couple of advantages:
* It is quicker to see the status of just one of the platforms
* Artifact collection for one platform is not blocked pending completion
of the builds for all systems
* It's a little easier to understand what is happening for a single platform
To support having multiple files I've changed the output-file option to
be an output-dir.
I've included the rm of main.yml in this commit. Once this gets
imported back to the FB internal system I'll amend in an update to
the helper script that updates all of our opensource builds and run
and amend that.
Pull Request resolved: https://github.com/facebook/folly/pull/1360
Test Plan:
the GH action status on this PR should show three different
actions running, one for each platform.
I updated and ran
`fbcode/opensource/fbcode_builder/getdeps/facebook/update-all-github-actions.sh`
to regenerate all the actions files for FB.
Reviewed By: yfeldblum
Differential Revision: D21310991
Pulled By: wez
fbshipit-source-id: 604ef652c8f746781a4b410c6b996cdee4524e0d
Summary:
The environment changed since I tested D20740410 and now it appears
that we'll need to re-export a versioned variable in order for cmake
to detect boost on the GH actions hosts.
Pull Request resolved: https://github.com/facebook/folly/pull/1359
Test Plan:
the GH actions status of this diff.
The workflow was updated via:
```
python3 build/fbcode_builder/getdeps.py generate-github-actions folly --output-file .github/workflows/main.yml
```
Reviewed By: yfeldblum
Differential Revision: D21307640
Pulled By: yfeldblum
fbshipit-source-id: 1555cbcade822775379cd9054be37fdbc17b4d93
Summary:
From the outset, we wanted to be sure that getdeps was able
to source and build the dependencies so that we knew that we'd have
a repeatable build. This came at the cost of build times: having
to build boost on each CI run is a bit of a chore.
This commit adds three new elements to the manifest files:
* `rpms` - a list of RPM names that are all required to be present
in order to consider the dependency satisfied
* `debs` - like `rpms` above, but scoped to debian package names
* `preinstalled.env` - a list of environment variables that if they
are all set and non-empty will satisfy the dependency.
A new `--allow-system-packages` option to getdeps enables the new
logic that looks for system packages; it is off by default, but
enabled in the generated GitHub Actions workflows.
A new `install-system-deps` subcommand is provided that will attempt
to install the system packages needed to satisfy the build. This
typically needs to be run via sudo and is thus broken out separately
from the main getdeps build flow.
I made a pass over the manifest files and added package names that
satisfy the build on ubuntu-18 and fedora-31.
shri-khare: I renamed the `Python3.7.6` manifest to just `python` as
part of this change; the version of python that it pulls in through
the normal build is the same and I believe that an equal or newer
version of python3 is available in the GH actions builder.
The `preinstalled.env` is used only by the boost manifest: it references
the name of an environment variable that is set by the github
windows hosts and that points to a pre-built and pre-installed
copy of boost. Since there is no package manager that we can
easily query for this sort of thing, probing from the environment
seems like a reasonable and fast way to check for this. We
may need to evolve this over time to become more feature rich,
but this seems like a good starting point.
This commit has the potential to save 20 minutes of build time
from each public CI build just due to the boost dependency alone!
Refs: https://github.com/facebook/watchman/pull/797
Reviewed By: yfeldblum
Differential Revision: D20740410
fbshipit-source-id: 6c38019449c54465127656c3d18a6ff1f30adaea
Summary:
We have a global `--install-prefix` argument that can be used to set
the prefix for all projects, but that is only suitable if you are running with
sufficient privileges to install each of the deps to that location during the
build. Cmake dependency resolution won't work from the build directory in that
situation; it can only see the final installed location and it will error out
if those files are not present, or link against the currently installed version
instead of the version we just built; not great!
This commit adds a project specific `--project-install-prefix` that can be used
on just the leaf project in a set of deps. That sidesteps the dependency
concern because only the last stage is built in that mode. This option
can technically be applied to an arbitrary set of projects, but in light
of the above, in practice it only makes sense to use it for the final
cmake project. Only the CMakeBuilder respects this option.
In the watchman repo, this commit adjusts the autogen.sh script to allow
specifying the installation prefix; it defaults to `/usr/local` as you
might expect.
refs: https://github.com/facebook/watchman/issues/760
Reviewed By: yfeldblum
Differential Revision: D20674439
fbshipit-source-id: 52799dbd47f3c295e2d6469ee2b74cedeaa20138
Summary:
My recent change to ensure that we were using python3 to launch everything failed on windows: the GH actions environment has `python.exe` in the path and it is python version 3. There is no such thing as `python3` in that environment, although there is a `python2`.
Refs: https://github.com/facebook/watchman/pull/797
Reviewed By: chadaustin
Differential Revision: D20740411
fbshipit-source-id: 0e40590ccedc18e327ebb84901e2509588fdd0ff
Summary:
The GH actions defaults for git prevent it from being able
to checkout the fbthrift repo due the length of the java related
files in the fbthrift repo.
This commit tells git to use long filenames and allows the checkout
to succeed.
Reviewed By: fanzeyi
Differential Revision: D20659750
fbshipit-source-id: 060b36d312d52a0769cf2f2dd9af60f7446f94a8
Summary:
Ensure that we are referencing python3 in the paths
that we generate for the github actions workflows, and remove
some shebangs that influence how our internal linters process
the python code.
Reviewed By: fanzeyi
Differential Revision: D20659747
fbshipit-source-id: 6f300f8e91edf7701bb27babc7b1418958cf0a10
Summary:
With this change the getdeps' CargoBuilder will support depencies between cargo builds.
The way how it works is documented in the code and required few assumptions about how a cargo project has to be defined in order to support this.
This change also adds the "mononoke" manifest and few Cargo.toml files to the mononoke project to prove that this new feature works.
Reviewed By: farnz
Differential Revision: D19468912
fbshipit-source-id: f299733cbbc2ec7bca399c898ec8d710334d0fa9
Summary:
Automatically detect the `--facebook-internal` flag based on the current
repository project name.
Reviewed By: wez
Differential Revision: D18621358
fbshipit-source-id: f2b3018169b151811eec455863a8bfc17667d4d8
