Summary:
Applies new import merging and sorting from µsort v1.0.
When merging imports, µsort will make a best-effort to move associated
comments to match merged elements, but there are known limitations due to
the diynamic nature of Python and developer tooling. These changes should
not produce any dangerous runtime changes, but may require touch-ups to
satisfy linters and other tooling.
Note that µsort uses case-insensitive, lexicographical sorting, which
results in a different ordering compared to isort. This provides a more
consistent sorting order, matching the case-insensitive order used when
sorting import statements by module name, and ensures that "frog", "FROG",
and "Frog" always sort next to each other.
For details on µsort's sorting and merging semantics, see the user guide:
https://usort.readthedocs.io/en/stable/guide.html#sorting
Reviewed By: lisroach
Differential Revision: D36402214
fbshipit-source-id: b641bfa9d46242188524d4ae2c44998922a62b4c
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/113
GitHub Actions was failing at apt-get stage, error message suggested adding an apt-get update as remedy. Added in this diff and it indeed works.
The actions build fails later on missing mio::net module when building mysql async, but that's unrelated to this change.
Reviewed By: farnz
Differential Revision: D34368662
fbshipit-source-id: f0a00da3ee740ae4443a328616e792ea615c922c
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/110
Pull Request resolved: https://github.com/facebookexperimental/rust-shed/pull/27
Make it so that changes to rust-shed or other common rust source are used locally vendored, so they don't need to be pushed to github before they are visible in a build.
There was already some support for cargo vendoring in getdeps, but it was limited to dependencies between manifests built with cargo builder. This wasn't enough to build something like eden (cmake is main entry point, with later calls cargo) or eden_scm (make is main entry point, with later calls to cargo), so this diff adds a cargo prepare step for getdeps other primary build systems.
The cargo vendoring is done by using a cargo config file to point to the source files used by getdeps. It has two modes:
1. per crate, existing mode which is already automatic for cargo to cargo manifest dependencies. To use it for a non cargo build manifest, add crate.pathmap
2. per git url, existing mode which was only use for crates.io third-party crates, now can be enabled by setting cargo.cargo_config_file
Reviewed By: yancouto
Differential Revision: D33895469
fbshipit-source-id: 7b13c0b679532492a336ce217de875c25fe1be90
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/106
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/107
Summary
* Add EdenFS builds on external CI now EdenSCM is good
* Mac builds on github actions by using brew for system dependencies
To make this work had to fix some path ordering issues with install directories for Linux and Mac, and generalise the homebrew path fixups we were doing for bison to all the used homebrew packages.
Previously Installed packages were being added after system paths, so our own installed thing might be ignored. On github these meant system python 3.9 was being used for hg tests rather than our specified 3.8 (this showed we have some test fails on python 3.9 with "SystemError: deallocated bytearray object has exported buffers", that are beyond the scope of this diff to fix)
Also needed to include the getdeps generated python into the generated edenscmdeps3.zip archive setup.py produces otherwise EdenFS tests failed to import thrift.Thrift
Eden tests are hanging when run externally about half way through, so disable them on github actions for now as this PR is already fairly large. They work when run locally on an internal devserver, so probably some bit of environment necessary is not defined in the test runner
Reviewed By: chadaustin
Differential Revision: D34116505
fbshipit-source-id: d0d628db5daabc28d0bd8997cd5c1bc885ed1e73
Summary:
Pull Request resolved: https://github.com/facebook/folly/pull/1714
windows-2016 will be removed in March, and now that we've updated
Boost, we can upgrade to windows-2019.
Reviewed By: genevievehelsel
Differential Revision: D34119171
fbshipit-source-id: caad432bebf6b8282844bf79f82b1fbc34f2df9e
Summary: Add some basic support for FreeBSD to getdeps.
Reviewed By: ahornby
Differential Revision: D33989129
fbshipit-source-id: 42ff5f160b7e19c12196bb2e52a726f7815487bd
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/103
Automate maintenance of the edenscm_* github actions yamls
Add job file and name options and support for the Rust install section
Reviewed By: fanzeyi
Differential Revision: D34044422
fbshipit-source-id: 7d5f07d37bab1eff5de30a88e710dbf7479ca192
Summary: Fix the getdeps.py generation sub command and regenerate github actions before making changes to support actions generation for eden_scm
Reviewed By: fanzeyi
Differential Revision: D34044243
fbshipit-source-id: 0039d04f25af4c842145dc142dae6b9996fc8046
Summary: Make the --facebook-internal flag value available from BuildOptions so can have jobs of both values where required
Reviewed By: xavierd
Differential Revision: D33945588
fbshipit-source-id: 5d0e1b42973bd3e411a4a71075915bdb2755f0eb
Summary:
In order to speed up build times on a mac, add homebrew support to
getdeps.
Homebrew packages can be declared in a manifest using the `homebrew` header.
Futher, ahornby has added manifest entries for homebrew packages which are
included in this diff and also included a change to use the correct version of
openssl. Without this openssl change, homebrew cmake configure finds an old
openssl 1.0.2 install.
This diff provides a 2x speed up for building folly:
Timings for clean getdeps folly build on mid-2018 2.9Ghz i9 6 core intel macbook pro with 32GB RAM:
With new homebrew system deps:
```
rm -rf /Users/ahornby/.scratch/UsersZahornbyZfbsource/fbcode_builder_getdeps/
time ./opensource/fbcode_builder/getdeps.py build --allow-system-packages folly
real 17m39.329s
user 76m10.317s
sys 5m50.163s
```
Without:
```
rm -rf /Users/ahornby/.scratch/UsersZahornbyZfbsource/fbcode_builder_getdeps/
time ./opensource/fbcode_builder/getdeps.py build folly
real 32m10.344s
user 105m53.448s
sys 15m57.858s
```
Reviewed By: ahornby
Differential Revision: D33842632
fbshipit-source-id: ac785d4a8dcfa31b77292bddd9e747022ac36e3b
Summary: Boost 1.69 does not build on Fedora 35, so upgrade to Boost 1.78.
Reviewed By: ahornby
Differential Revision: D33855670
fbshipit-source-id: 85469a835a1dab1a7d5222413b1f1349bdcff280
Summary:
checkout@v2 is the current action version, and the old one may have
security issues. Let's just run with the latest version.
Reviewed By: ahornby
Differential Revision: D33834847
fbshipit-source-id: fba0733577bf23c208c9824aa3815bd01a813d9a
Summary:
Add support for overriding os, distro and distro version to command line when inspecting system packages so one can requested see ubuntu 18.04 package from other OS. Makes testing easier
Used shlex to shell unquote the value to be tested in the getdeps expression evaluator. getdeps expression parser didn't tolerate 18.04 as . is special char to getdeps expressions, needed to be "18.04"
Reviewed By: quark-zju
Differential Revision: D33741323
fbshipit-source-id: d83397c7fb5180a4d985d0d8ae7b3ff33b72f828
Summary:
This diffs adds
* `--shared-libs` command line argument to the `build` command which enables building shared libraries for supported projects (cmake-based projects, boost, libevent)
* this flag overrides `BUILD_SHARED_LIBS` cmake flags in manifest files and from `--extra-cmake-defines`
* adds `shared_libs=on` expression support in manifest files
* for boost, the flag enables building shared libraries **in addition to** statically linked ones
Reviewed By: simonmar
Differential Revision: D27462289
fbshipit-source-id: d22ab434f7228c30472611bc323830d88efba0a5
Summary:
A number of places were extracting dependencies from manifests, but only one was adding in the implicit dependencies for build tools.
Extract the logic to one place and use so that a change in a tool like cmake will now correctly affect all tools using cmake, as it will be taken into account as a dependency hash when the manifest's hash is computed.
Tests for this change revealed that install_dirs needed to be populated in reverse order from the manifest topo-sort, so have also addressed that
Reviewed By: wittgenst
Differential Revision: D32730717
fbshipit-source-id: 1b2a25e460de6085d274c99acfd391b3bd259264
Summary: Useful to be able to see what packages would be installed without doing it, and to see rpm vs deb packages
Reviewed By: Croohand
Differential Revision: D32693528
fbshipit-source-id: a01257e7192f2f0299d57f6f8f7ee93452a6f3e4
Summary: Update to new version of upload-artifact without branch name as that's a separate repo that our dynamic branch settings should not apply to.
Reviewed By: zpao
Differential Revision: D32717494
fbshipit-source-id: 968908d1072d6d4550705e7df0b3828528e057dc
Summary: Branches updated some time ago, update this to match
Reviewed By: zpao
Differential Revision: D32693531
fbshipit-source-id: 02da79e3fb5fcd3b204be4605521a151b977ce2c
Summary: I found this useful to see where manifests resolve to when they are already built
Reviewed By: HarveyHunt
Differential Revision: D32644308
fbshipit-source-id: b796cb67f154177fbac1093290d97fe5aab6b20c
Summary:
Distro name wasn't matching and thus no rpms being installed or used
Also improved the error message for unknown systems
Reviewed By: farnz
Differential Revision: D32139850
fbshipit-source-id: 438278094159f732f6b1de506490b89d2c9cd3f7
Summary:
Need to fix branch name for OSS mvfst so CI continues on.
Ran opensource/fbcode_builder/getdeps/facebook/update-all-github-actions.sh
Reviewed By: lnicco
Differential Revision: D32041656
fbshipit-source-id: f72aab91a3084814a5d3fa8643ae5f405908c313
Summary: This diff adds support to customize main branch name when generating GitHub Actions.
Differential Revision: D30679305
fbshipit-source-id: 0fc7eb1c97c27e2b42e60cc1ab69a48ab93b93fa
Summary:
This applies the formatting changes from black v21.4b2 to all covered
projects in fbsource. Most changes are to single line docstrings, as black
will now remove leading and trailing whitespace to match PEP8. Any other
formatting changes are likely due to files that landed without formatting,
or files that previously triggered errors in black.
Any changes to code should be AST identical. Any test failures are likely
due to bad tests, or testing against the output of pyfmt.
Reviewed By: thatch
Differential Revision: D28204910
fbshipit-source-id: 804725bcd14f763e90c5ddff1d0418117c15809a
Summary: A show-build-dir subcommand. Possibly useful for building a project and then running only one test or benchmark binary directly.
Reviewed By: wez
Differential Revision: D26502869
fbshipit-source-id: 0d8f6c42a2bbeb2440503a39c7fc770e3b3e4fff
Summary:
`extra_cmake_defines` are extra flags that are passed to cmake when
compiling each one of the dependencies. For instance:
```
$ ./opensource/fbcode_builder/getdeps.py build f4d --extra-cmake-define='{"CMAKE_CXX_FLAGS": "-mavx2 -mfma -mavx -mf16c -march=native"}'
```
It takes a json map as input, which can take a list of defines (key value
pairs).
Reviewed By: wez
Differential Revision: D25855781
fbshipit-source-id: 7f4fef2c66f4d12f23c8d7086d6a4f24fcc01ff7
Summary:
Recently I found that its impossible to access elevated Watchman daemon from non-elevated process on Windows.
```
events.js:174
throw er; // Unhandled 'error' event
^
Error: connect EPERM \\.\pipe\watchman
at PipeConnectWrap.afterConnect [as oncomplete] (net.js:1106:14)
Emitted 'error' event at:
at Socket.<anonymous> (C:\open\ovrsource\unity\socialvr\_js\node_modules\fb-watchman\index.js:118:12)
at Socket.emit (events.js:198:13)
at emitErrorNT (internal/streams/destroy.js:91:8)
at emitErrorAndCloseNT (internal/streams/destroy.js:59:3)
at process._tickCallback (internal/process/next_tick.js:63:19)
```
To fix this, it was suggested by wez to use [his library](https://github.com/wez/EleDo) to force Watchman daemon always start in normal mode on Windows. In this stack of commits I did integrated his library into project and used it to force daemon restart in normal mode when spawned from elevated terminal.
To make it happen, I checked-in library sources and created proxy project which depends on the initial library and contains header bindings and cmake configuration. I did copy pasted Rust cmake macroses from another facebook project - eden, and also created analogue of autogen.sh but for Windows - autogen.cmd.
Pull Request resolved: https://github.com/facebook/watchman/pull/878
Test Plan:
Launch elevated terminal
Start watchman.exe produced from sources
Observe daemon starting and answering
In process monitor, observe watchman.exe process running under user group
Reviewed By: fanzeyi
Differential Revision: D25595879
Pulled By: arhelmus
fbshipit-source-id: 15eb29adcf5bd4a5708b6533a1b2bacbf13f431c
Summary:
- Added a commandline flag to ignore testpilot even when it's available
- Fixed an oversight that caused getdeps to return 0 even if ctest tests have failed.
Reviewed By: wez
Differential Revision: D23161362
fbshipit-source-id: 1ed97c481315e2b39f1128680386390930194970
Summary:
- I find the ability to make a branch and play with CI on GitHub super handy
- Removing the `- master` limitation gives me this ability
Reviewed By: yi-xian
Differential Revision: D22771835
fbshipit-source-id: 8e8839cb860ab4d1dfa0dda590afaf165127f60d
Summary:
- Make OpenR build all deps from source until we remove fbzmq as a dep
- Allow a project to move it's Open Source CI forward to an alternative version of ubuntu via a new `--ubuntu-version` parameter
Reviewed By: wez
Differential Revision: D22768987
fbshipit-source-id: 07205efbd0c87a702638cf30b84a2850d064fa8e
Summary: This commit adds a flag `--retry` to getdeps and teach it to run retry failed test. This allows us to still pass the tests when there are some flaky tests presents.
Reviewed By: wez
Differential Revision: D22291063
fbshipit-source-id: 572af48a52ceb4a9abbf530cc0154ded0120c0de
Summary:
For large projects, with lots of tests, running all the tests can take a lot
of time, but for quick development iteration, only a subset of the tests may
be needed to run.
On non-Windows platforms, this can be easily achieved by manually executing
the individual tests binaries and use the builtin filtering mechanism of that
test binary to achieve the goal. On Windows, this can quickly become
impossible as DLLs might not be available, and the right PATH would need to
be manually specified by hand to execute the tests binaries[0].
To solve this, let's simply provide a unified way of running specific tests
by passing in a regexp. Both testpilot and CTest do support regex to execute
specific tests. My understanding is that cargo doesn't yet allows regex, but
will in the future.
[0]: And a missing DLLs would produce no output when executed from
PowerShell, which makes this very confusing.
Reviewed By: wez
Differential Revision: D21484774
fbshipit-source-id: ee32e950e25bb2a498a2b364a447955a917b0590
Summary:
The breakage has been fixed, so bring back the manifest, but only the Linux one, because the Mac version is failing due to another issue.
Also to make it easier to debug issues on GitHub Actions separate out the dependencies build from Mononoke and rust-shed builds.
Reviewed By: krallin
Differential Revision: D21448412
fbshipit-source-id: 68d89c858d1692727a7fd66bca114920e6dfb4dc
Summary:
D21364132 accidentally broke this; we can't run the fetcher
for projects for which we pulled the build out of cache, because there
is no source to update in that case.
This commit adjusts the logic so that we write out a marker file
to indicate that we installed a build from cache and to look for
that file being present to gate the new update logic.
Reviewed By: lnicco
Differential Revision: D21419122
fbshipit-source-id: 304670848add22531d88549d66f22c40ff255140
Summary:
When the commit hash changed in fbsource, we would correctly decide
that we'd need to rebuild first-party projects but we would incorrectly skip
running the fetcher.update method. This would mean that we'd not perform the
shipit run and that our shipit tree would diverge from the source tree.
This commit resolves this by performing the fetcher.update but ignoring
the source update status in this case.
Reviewed By: xavierd
Differential Revision: D21364131
fbshipit-source-id: b4001e549c7d3f27aa4a21b19893c9bb7c0f6d1f
Summary:
We didn't do a great job of recognizing that we'd need to
build a project when one of its dependencies had changed: we relied
chiefly on the dependency hash for this and could fail to handle
changes in individual source files.
This commit helps to improve this situation by checking to see if
any installed files in the dependencies of a manifest are newer than
the most recent built time of a given manifest. If so, we'll trigger
a build. We try to be reasonably smart about deciding when to trigger
a cmake reconfigure if it looks like cmake files in the deps have
been changed.
Reviewed By: xavierd
Differential Revision: D21364132
fbshipit-source-id: 7534496e10d1f532aa9cf865900ace84a8785327
Summary:
Our linter really wants to include formatting changes unrelated
to my diff stack.
This is a formatting only change to avoid clouding my diffs; no functional
effect.
Reviewed By: xavierd
Differential Revision: D21364519
fbshipit-source-id: 7670dd4154e788f593f256aabdfdeef6d17aeec4
Summary:
On Linux the debug info sections in projects downstream from folly and
thrift are huge due to a lot of C++ template usage.
We build in RelWithDebInfo mode as that is most useful when debugging
locally and because the build times are long, but most casual consumers
of the binaries via GH actions really don't care about debug info,
and this should save ~180MB of size in (for example) the watchman
executables.
Pull Request resolved: https://github.com/facebook/watchman/pull/809
Refs: https://github.com/facebook/watchman/issues/804
Test Plan:
Review the artifact size on the linux build in this PR (see
https://github.com/facebook/watchman/actions/runs/91688952) and confirm that it
is a bit smaller than 180MB; now under 3 MB.
Reviewed By: simpkins
Differential Revision: D21318302
Pulled By: wez
fbshipit-source-id: f78bc5e735dd78771e0604abae64c0a23cf9158d
Summary:
Rather than have a single main.yml file containing all off the different
builds, split that up so that we have one file per build environment
(linux, mac, windows).
This has a couple of advantages:
* It is quicker to see the status of just one of the platforms
* Artifact collection for one platform is not blocked pending completion
of the builds for all systems
* It's a little easier to understand what is happening for a single platform
To support having multiple files I've changed the output-file option to
be an output-dir.
I've included the rm of main.yml in this commit. Once this gets
imported back to the FB internal system I'll amend in an update to
the helper script that updates all of our opensource builds and run
and amend that.
Pull Request resolved: https://github.com/facebook/folly/pull/1360
Test Plan:
the GH action status on this PR should show three different
actions running, one for each platform.
I updated and ran
`fbcode/opensource/fbcode_builder/getdeps/facebook/update-all-github-actions.sh`
to regenerate all the actions files for FB.
Reviewed By: yfeldblum
Differential Revision: D21310991
Pulled By: wez
fbshipit-source-id: 604ef652c8f746781a4b410c6b996cdee4524e0d
Summary:
The environment changed since I tested D20740410 and now it appears
that we'll need to re-export a versioned variable in order for cmake
to detect boost on the GH actions hosts.
Pull Request resolved: https://github.com/facebook/folly/pull/1359
Test Plan:
the GH actions status of this diff.
The workflow was updated via:
```
python3 build/fbcode_builder/getdeps.py generate-github-actions folly --output-file .github/workflows/main.yml
```
Reviewed By: yfeldblum
Differential Revision: D21307640
Pulled By: yfeldblum
fbshipit-source-id: 1555cbcade822775379cd9054be37fdbc17b4d93
