Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/653
getdeps: add --build-type option to build in Debug or RelWithDebInfo mode
Adds a --build-type option so one can force RelWithDebInfo or Debug explicity
Default remains RelWithDebInfo for cmake.
cargo default is updated to --release to match cmake more closely, if you don't want release use --build-type Debug.
If you want to run github CI in Debug mode (faster build, but tests will run slower), then can pass --build-type Debug to getdeps.py generate-github-actions
X-link: https://github.com/facebook/sapling/pull/786
Reviewed By: mitrandir77
Differential Revision: D51564770
Pulled By: bigfootjon
fbshipit-source-id: ef30332ca193d9805bce005d12b5dbc9f58fcafc
Summary:
X-link: https://github.com/facebookincubator/velox/pull/7446
Add a new command to show the location of the scratch dir that getdeps
will use.
Reviewed By: xavierd
Differential Revision: D51027807
fbshipit-source-id: 80a7cfc04320002588d5ec8964fc88914771c6c7
Summary:
X-link: https://github.com/facebookincubator/velox/pull/7072
use system patchelf in eden fs build
Saves us from doing a fetch and build of autoconf, libtool, automake and patchelf during the arfifacts part of CI
X-link: https://github.com/facebook/sapling/pull/750
Reviewed By: sggutier
Differential Revision: D50313417
Pulled By: genevievehelsel
fbshipit-source-id: 7c585357c848c15a65c5797d6c8750d1119b6efd
Summary:
X-link: https://github.com/facebookincubator/velox/pull/7071
fix CI warning about node12 by updating actions/checkout version
Update the actions/checkout version to fix build warning
X-link: https://github.com/facebook/sapling/pull/749
Reviewed By: sggutier
Differential Revision: D50313430
Pulled By: genevievehelsel
fbshipit-source-id: 4d55d36f7509be51bb4e26728ddbce8e45faba40
Summary:
X-link: https://github.com/facebookincubator/velox/pull/6943
Update to boost 1.83. This addresses a folly build failure on Xcode 15, due to
boost 1.78 referring to the removed std::unary_function.
Reviewed By: jdelliot
Differential Revision: D49972186
fbshipit-source-id: 8d9fdd27c24ccc5a072b6973b86e0c8ed5b77ac3
Summary:
X-link: https://github.com/facebookincubator/velox/pull/6924
fix getdeps actions generation for rust toolchain
Generated actions files created from getdeps.py were missing rust toolchain which meant annoying manual edits
This change also makes the github actions generation honor the --no-tests argument, which is useful when regenerating the edenfs actions
X-link: https://github.com/facebook/sapling/pull/682
Reviewed By: sggutier
Differential Revision: D49875258
Pulled By: genevievehelsel
fbshipit-source-id: 173f86083ba92bab4063813d3e392df428b9ffe4
Summary:
X-link: https://github.com/facebookincubator/velox/pull/6927
allow getdeps github actions to free up disk
Allow getdeps to free up some disk from the runner and intermediate build steps as some runs (notably the linux eden and mononoke ones) are hitting disk space limits
X-link: https://github.com/facebook/sapling/pull/689
Reviewed By: sggutier
Differential Revision: D49875256
Pulled By: genevievehelsel
fbshipit-source-id: b85b6b2f11857670915b64f47d3c0abd4ca8ca31
Summary:
X-link: https://github.com/facebookincubator/velox/pull/4796
`brew` no longer supports running as root:
```
Error: Running Homebrew as root is extremely dangerous and no longer supported.
As Homebrew does not drop privileges on installation you would be giving all
build scripts full access to your system.
Command '['brew', 'install', 'autoconf', 'automake', 'boost', 'cmake', 'double-conversion', 'icu4c', 'libevent', 'libsodium', 'libtool', 'lz4', 'ninja', 'openssl@1.1', 'xz', 'zlib', 'zstd']' returned non-zero exit status 1.
!! Failed
```
And as a result the recommended install instructions no longer work:
```
sudo ./build/fbcode_builder/getdeps.py install-system-deps ...
```
and cannot be made to work portably across platforms because Linux requires `sudo` be there while macOS requires `sudo` not be there.
To fix this move `sudo` to Linux system dependency installation only.
Also update `apt-get` to `apt` while at it.
Resolves https://github.com/facebook/fbthrift/issues/545.
Reviewed By: somasun
Differential Revision: D45371004
fbshipit-source-id: 75334db22226efc961e7d4d6c6eca911086b97ba
Summary:
X-link: https://github.com/facebookincubator/velox/pull/4087
Under some circumstances, getdeps can cache a project artifact
where there is no built_marker, forcing the script to re-build the artifact
from source. After D43260530 (47dde7c249) this no longer causes a build failure, but it can
still make builds take longer than they should.
This ensures we only cache artifacts with a built_marker, so that they aren't
overwritten by artifacts without.
Reviewed By: genevievehelsel
Differential Revision: D43405855
fbshipit-source-id: b1b9e194e642fa820d0fbc54a7c15ff81fc90a82
Summary:
Automation should use `apt-get` instead of `apt`, as `apt` does not
have a stable interface.
This fixes a warning when running getdeps.py on recent Ubuntu:
```
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
```
Reviewed By: fanzeyi
Differential Revision: D39031719
fbshipit-source-id: 54a0fe51ce159174b089fc446e99a4ab5d69d626
Summary:
In D36905191 (6e6bf12f62), getdeps gained the ability to apply patches on Windows.
However, on GitHub Action workers, `core.autocrlf` is turned on by default. This means the patch files we checked in will end up having CRLF endings while the source code we downloaded are still in LF endings, and `git apply` doesn't like that.
This diff teaches getdeps to disable `core.autocrlf` in GitHub Action Windows workers.
Reviewed By: vitaut
Differential Revision: D37008387
fbshipit-source-id: 34c9f89e8783d0613040d190c4ad3477bd7bfd53
Summary:
Applies new import merging and sorting from µsort v1.0.
When merging imports, µsort will make a best-effort to move associated
comments to match merged elements, but there are known limitations due to
the diynamic nature of Python and developer tooling. These changes should
not produce any dangerous runtime changes, but may require touch-ups to
satisfy linters and other tooling.
Note that µsort uses case-insensitive, lexicographical sorting, which
results in a different ordering compared to isort. This provides a more
consistent sorting order, matching the case-insensitive order used when
sorting import statements by module name, and ensures that "frog", "FROG",
and "Frog" always sort next to each other.
For details on µsort's sorting and merging semantics, see the user guide:
https://usort.readthedocs.io/en/stable/guide.html#sorting
Reviewed By: lisroach
Differential Revision: D36402214
fbshipit-source-id: b641bfa9d46242188524d4ae2c44998922a62b4c
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/113
GitHub Actions was failing at apt-get stage, error message suggested adding an apt-get update as remedy. Added in this diff and it indeed works.
The actions build fails later on missing mio::net module when building mysql async, but that's unrelated to this change.
Reviewed By: farnz
Differential Revision: D34368662
fbshipit-source-id: f0a00da3ee740ae4443a328616e792ea615c922c
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/110
Pull Request resolved: https://github.com/facebookexperimental/rust-shed/pull/27
Make it so that changes to rust-shed or other common rust source are used locally vendored, so they don't need to be pushed to github before they are visible in a build.
There was already some support for cargo vendoring in getdeps, but it was limited to dependencies between manifests built with cargo builder. This wasn't enough to build something like eden (cmake is main entry point, with later calls cargo) or eden_scm (make is main entry point, with later calls to cargo), so this diff adds a cargo prepare step for getdeps other primary build systems.
The cargo vendoring is done by using a cargo config file to point to the source files used by getdeps. It has two modes:
1. per crate, existing mode which is already automatic for cargo to cargo manifest dependencies. To use it for a non cargo build manifest, add crate.pathmap
2. per git url, existing mode which was only use for crates.io third-party crates, now can be enabled by setting cargo.cargo_config_file
Reviewed By: yancouto
Differential Revision: D33895469
fbshipit-source-id: 7b13c0b679532492a336ce217de875c25fe1be90
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/106
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/107
Summary
* Add EdenFS builds on external CI now EdenSCM is good
* Mac builds on github actions by using brew for system dependencies
To make this work had to fix some path ordering issues with install directories for Linux and Mac, and generalise the homebrew path fixups we were doing for bison to all the used homebrew packages.
Previously Installed packages were being added after system paths, so our own installed thing might be ignored. On github these meant system python 3.9 was being used for hg tests rather than our specified 3.8 (this showed we have some test fails on python 3.9 with "SystemError: deallocated bytearray object has exported buffers", that are beyond the scope of this diff to fix)
Also needed to include the getdeps generated python into the generated edenscmdeps3.zip archive setup.py produces otherwise EdenFS tests failed to import thrift.Thrift
Eden tests are hanging when run externally about half way through, so disable them on github actions for now as this PR is already fairly large. They work when run locally on an internal devserver, so probably some bit of environment necessary is not defined in the test runner
Reviewed By: chadaustin
Differential Revision: D34116505
fbshipit-source-id: d0d628db5daabc28d0bd8997cd5c1bc885ed1e73
Summary:
Pull Request resolved: https://github.com/facebook/folly/pull/1714
windows-2016 will be removed in March, and now that we've updated
Boost, we can upgrade to windows-2019.
Reviewed By: genevievehelsel
Differential Revision: D34119171
fbshipit-source-id: caad432bebf6b8282844bf79f82b1fbc34f2df9e
Summary: Add some basic support for FreeBSD to getdeps.
Reviewed By: ahornby
Differential Revision: D33989129
fbshipit-source-id: 42ff5f160b7e19c12196bb2e52a726f7815487bd
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/103
Automate maintenance of the edenscm_* github actions yamls
Add job file and name options and support for the Rust install section
Reviewed By: fanzeyi
Differential Revision: D34044422
fbshipit-source-id: 7d5f07d37bab1eff5de30a88e710dbf7479ca192
Summary: Fix the getdeps.py generation sub command and regenerate github actions before making changes to support actions generation for eden_scm
Reviewed By: fanzeyi
Differential Revision: D34044243
fbshipit-source-id: 0039d04f25af4c842145dc142dae6b9996fc8046
Summary: Make the --facebook-internal flag value available from BuildOptions so can have jobs of both values where required
Reviewed By: xavierd
Differential Revision: D33945588
fbshipit-source-id: 5d0e1b42973bd3e411a4a71075915bdb2755f0eb
Summary:
In order to speed up build times on a mac, add homebrew support to
getdeps.
Homebrew packages can be declared in a manifest using the `homebrew` header.
Futher, ahornby has added manifest entries for homebrew packages which are
included in this diff and also included a change to use the correct version of
openssl. Without this openssl change, homebrew cmake configure finds an old
openssl 1.0.2 install.
This diff provides a 2x speed up for building folly:
Timings for clean getdeps folly build on mid-2018 2.9Ghz i9 6 core intel macbook pro with 32GB RAM:
With new homebrew system deps:
```
rm -rf /Users/ahornby/.scratch/UsersZahornbyZfbsource/fbcode_builder_getdeps/
time ./opensource/fbcode_builder/getdeps.py build --allow-system-packages folly
real 17m39.329s
user 76m10.317s
sys 5m50.163s
```
Without:
```
rm -rf /Users/ahornby/.scratch/UsersZahornbyZfbsource/fbcode_builder_getdeps/
time ./opensource/fbcode_builder/getdeps.py build folly
real 32m10.344s
user 105m53.448s
sys 15m57.858s
```
Reviewed By: ahornby
Differential Revision: D33842632
fbshipit-source-id: ac785d4a8dcfa31b77292bddd9e747022ac36e3b
Summary: Boost 1.69 does not build on Fedora 35, so upgrade to Boost 1.78.
Reviewed By: ahornby
Differential Revision: D33855670
fbshipit-source-id: 85469a835a1dab1a7d5222413b1f1349bdcff280
Summary:
checkout@v2 is the current action version, and the old one may have
security issues. Let's just run with the latest version.
Reviewed By: ahornby
Differential Revision: D33834847
fbshipit-source-id: fba0733577bf23c208c9824aa3815bd01a813d9a
Summary:
Add support for overriding os, distro and distro version to command line when inspecting system packages so one can requested see ubuntu 18.04 package from other OS. Makes testing easier
Used shlex to shell unquote the value to be tested in the getdeps expression evaluator. getdeps expression parser didn't tolerate 18.04 as . is special char to getdeps expressions, needed to be "18.04"
Reviewed By: quark-zju
Differential Revision: D33741323
fbshipit-source-id: d83397c7fb5180a4d985d0d8ae7b3ff33b72f828
Summary:
This diffs adds
* `--shared-libs` command line argument to the `build` command which enables building shared libraries for supported projects (cmake-based projects, boost, libevent)
* this flag overrides `BUILD_SHARED_LIBS` cmake flags in manifest files and from `--extra-cmake-defines`
* adds `shared_libs=on` expression support in manifest files
* for boost, the flag enables building shared libraries **in addition to** statically linked ones
Reviewed By: simonmar
Differential Revision: D27462289
fbshipit-source-id: d22ab434f7228c30472611bc323830d88efba0a5
Summary:
A number of places were extracting dependencies from manifests, but only one was adding in the implicit dependencies for build tools.
Extract the logic to one place and use so that a change in a tool like cmake will now correctly affect all tools using cmake, as it will be taken into account as a dependency hash when the manifest's hash is computed.
Tests for this change revealed that install_dirs needed to be populated in reverse order from the manifest topo-sort, so have also addressed that
Reviewed By: wittgenst
Differential Revision: D32730717
fbshipit-source-id: 1b2a25e460de6085d274c99acfd391b3bd259264
Summary: Useful to be able to see what packages would be installed without doing it, and to see rpm vs deb packages
Reviewed By: Croohand
Differential Revision: D32693528
fbshipit-source-id: a01257e7192f2f0299d57f6f8f7ee93452a6f3e4
Summary: Update to new version of upload-artifact without branch name as that's a separate repo that our dynamic branch settings should not apply to.
Reviewed By: zpao
Differential Revision: D32717494
fbshipit-source-id: 968908d1072d6d4550705e7df0b3828528e057dc
Summary: Branches updated some time ago, update this to match
Reviewed By: zpao
Differential Revision: D32693531
fbshipit-source-id: 02da79e3fb5fcd3b204be4605521a151b977ce2c
Summary: I found this useful to see where manifests resolve to when they are already built
Reviewed By: HarveyHunt
Differential Revision: D32644308
fbshipit-source-id: b796cb67f154177fbac1093290d97fe5aab6b20c
Summary:
Distro name wasn't matching and thus no rpms being installed or used
Also improved the error message for unknown systems
Reviewed By: farnz
Differential Revision: D32139850
fbshipit-source-id: 438278094159f732f6b1de506490b89d2c9cd3f7
Summary:
Need to fix branch name for OSS mvfst so CI continues on.
Ran opensource/fbcode_builder/getdeps/facebook/update-all-github-actions.sh
Reviewed By: lnicco
Differential Revision: D32041656
fbshipit-source-id: f72aab91a3084814a5d3fa8643ae5f405908c313
Summary: This diff adds support to customize main branch name when generating GitHub Actions.
Differential Revision: D30679305
fbshipit-source-id: 0fc7eb1c97c27e2b42e60cc1ab69a48ab93b93fa
