Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/907
I don't sure that I make all in accordance with the your contribution pipeline, so please correct me If there is needed.
So, I've made some changes to support the `pacman` package manager in your `getdeps.py` script. In `manifests` I'm also duplicated some packages from `debs` and `rpms` sections and create a new `pps` sections with according packages for `pacman`.
Issue: https://github.com/facebook/folly/issues/1701
X-link: https://github.com/facebook/folly/pull/1702
Reviewed By: yfeldblum
Differential Revision: D33514769
Pulled By: Orvid
fbshipit-source-id: a081c3a5bcb7f7cdde3a4a91c0d15517c0a171b3
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/897
Builder refactoring: instead of providing `install_dirs` to `build()`, `test()` etc., provide `loader` and `dep_manifests` when creating the builder. This is a cleaner API because we were computing `install_dirs` in multiple places before.
Furthermore this lets us do things that need to see the manifests of the dependencies, not just the list of `install_dirs`, such as treating direct dependencies differently from indirect dependencies (see D58244928).
Reviewed By: chadaustin
Differential Revision: D58200528
fbshipit-source-id: e52d35e84161b83ab49ab43099c3e3b9bb03f36e
Summary:
Starts a shell in the build dir, with the environment set up in the
same way as for the build. Useful for experimenting and testing.
Reviewed By: chadaustin
Differential Revision: D58082246
fbshipit-source-id: 82b275401528d7616c2560d80b4c187de67f6032
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/858
Because it needs to be passed to other commands that use project
hashes, such as "test".
(in general there are probably a lot more of these, just fixing the
ones I've ran into so far)
Reviewed By: genevievehelsel
Differential Revision: D58082245
fbshipit-source-id: 09fa6b5ce4cc4b3ae7ecfb34ac83681cecb36e3c
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/848
Because it should work with `show-inst-dir`, otherwise we can't
calculate the right project hash.
Reviewed By: chadaustin
Differential Revision: D58011867
fbshipit-source-id: d8960b4a993efbada8e27584e56976279fcd6b43
Summary: this runs on github... but also not what i really care about
Reviewed By: terrelln
Differential Revision: D57125977
fbshipit-source-id: 39d7fd319a0ce45a36ba11c4301f5a27356b6c9f
Summary:
there is a bunch of scripts that uses python3 to run, this means that they are at the mercy of whoever controll their path, in linux, this can be system python3, but can also be platform python (and probably it is both), and in windows in particular it abuses c:\tools\fb-python\python3.exe. fbpython is the universal way to run python at the company that chooses the right platform python, but also provides monitoring and observability.
this scripts generates jobs like
https://www.internalfb.com/sandcastle/job/18014399781484898/
that will stop working once we remove c:\tools\fb-python\python3*.exe
Reviewed By: fried
Differential Revision: D56896564
fbshipit-source-id: d911fdaf6750635adb05b096f0522603baf47bcc
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/749
Updating generated workflow files to account for recent changes. Most notably, this updates the checkout action to v4 and sets an explicit read-only permission.
This also adds support for `--cron` in the codegen to account for only running CI on a schedule (useful for managing costs).
Reviewed By: ahornby
Differential Revision: D56165825
fbshipit-source-id: 298b16effefb6b8a2dc6cbcf07d4ec4a61f48364
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/653
getdeps: add --build-type option to build in Debug or RelWithDebInfo mode
Adds a --build-type option so one can force RelWithDebInfo or Debug explicity
Default remains RelWithDebInfo for cmake.
cargo default is updated to --release to match cmake more closely, if you don't want release use --build-type Debug.
If you want to run github CI in Debug mode (faster build, but tests will run slower), then can pass --build-type Debug to getdeps.py generate-github-actions
X-link: https://github.com/facebook/sapling/pull/786
Reviewed By: mitrandir77
Differential Revision: D51564770
Pulled By: bigfootjon
fbshipit-source-id: ef30332ca193d9805bce005d12b5dbc9f58fcafc
Summary:
X-link: https://github.com/facebookincubator/velox/pull/7446
Add a new command to show the location of the scratch dir that getdeps
will use.
Reviewed By: xavierd
Differential Revision: D51027807
fbshipit-source-id: 80a7cfc04320002588d5ec8964fc88914771c6c7
Summary:
X-link: https://github.com/facebookincubator/velox/pull/7072
use system patchelf in eden fs build
Saves us from doing a fetch and build of autoconf, libtool, automake and patchelf during the arfifacts part of CI
X-link: https://github.com/facebook/sapling/pull/750
Reviewed By: sggutier
Differential Revision: D50313417
Pulled By: genevievehelsel
fbshipit-source-id: 7c585357c848c15a65c5797d6c8750d1119b6efd
Summary:
X-link: https://github.com/facebookincubator/velox/pull/7071
fix CI warning about node12 by updating actions/checkout version
Update the actions/checkout version to fix build warning
X-link: https://github.com/facebook/sapling/pull/749
Reviewed By: sggutier
Differential Revision: D50313430
Pulled By: genevievehelsel
fbshipit-source-id: 4d55d36f7509be51bb4e26728ddbce8e45faba40
Summary:
X-link: https://github.com/facebookincubator/velox/pull/6943
Update to boost 1.83. This addresses a folly build failure on Xcode 15, due to
boost 1.78 referring to the removed std::unary_function.
Reviewed By: jdelliot
Differential Revision: D49972186
fbshipit-source-id: 8d9fdd27c24ccc5a072b6973b86e0c8ed5b77ac3
Summary:
X-link: https://github.com/facebookincubator/velox/pull/6924
fix getdeps actions generation for rust toolchain
Generated actions files created from getdeps.py were missing rust toolchain which meant annoying manual edits
This change also makes the github actions generation honor the --no-tests argument, which is useful when regenerating the edenfs actions
X-link: https://github.com/facebook/sapling/pull/682
Reviewed By: sggutier
Differential Revision: D49875258
Pulled By: genevievehelsel
fbshipit-source-id: 173f86083ba92bab4063813d3e392df428b9ffe4
Summary:
X-link: https://github.com/facebookincubator/velox/pull/6927
allow getdeps github actions to free up disk
Allow getdeps to free up some disk from the runner and intermediate build steps as some runs (notably the linux eden and mononoke ones) are hitting disk space limits
X-link: https://github.com/facebook/sapling/pull/689
Reviewed By: sggutier
Differential Revision: D49875256
Pulled By: genevievehelsel
fbshipit-source-id: b85b6b2f11857670915b64f47d3c0abd4ca8ca31
Summary:
X-link: https://github.com/facebookincubator/velox/pull/4796
`brew` no longer supports running as root:
```
Error: Running Homebrew as root is extremely dangerous and no longer supported.
As Homebrew does not drop privileges on installation you would be giving all
build scripts full access to your system.
Command '['brew', 'install', 'autoconf', 'automake', 'boost', 'cmake', 'double-conversion', 'icu4c', 'libevent', 'libsodium', 'libtool', 'lz4', 'ninja', 'openssl@1.1', 'xz', 'zlib', 'zstd']' returned non-zero exit status 1.
!! Failed
```
And as a result the recommended install instructions no longer work:
```
sudo ./build/fbcode_builder/getdeps.py install-system-deps ...
```
and cannot be made to work portably across platforms because Linux requires `sudo` be there while macOS requires `sudo` not be there.
To fix this move `sudo` to Linux system dependency installation only.
Also update `apt-get` to `apt` while at it.
Resolves https://github.com/facebook/fbthrift/issues/545.
Reviewed By: somasun
Differential Revision: D45371004
fbshipit-source-id: 75334db22226efc961e7d4d6c6eca911086b97ba
Summary:
X-link: https://github.com/facebookincubator/velox/pull/4087
Under some circumstances, getdeps can cache a project artifact
where there is no built_marker, forcing the script to re-build the artifact
from source. After D43260530 (47dde7c249) this no longer causes a build failure, but it can
still make builds take longer than they should.
This ensures we only cache artifacts with a built_marker, so that they aren't
overwritten by artifacts without.
Reviewed By: genevievehelsel
Differential Revision: D43405855
fbshipit-source-id: b1b9e194e642fa820d0fbc54a7c15ff81fc90a82
Summary:
Automation should use `apt-get` instead of `apt`, as `apt` does not
have a stable interface.
This fixes a warning when running getdeps.py on recent Ubuntu:
```
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
```
Reviewed By: fanzeyi
Differential Revision: D39031719
fbshipit-source-id: 54a0fe51ce159174b089fc446e99a4ab5d69d626
Summary:
In D36905191 (6e6bf12f62), getdeps gained the ability to apply patches on Windows.
However, on GitHub Action workers, `core.autocrlf` is turned on by default. This means the patch files we checked in will end up having CRLF endings while the source code we downloaded are still in LF endings, and `git apply` doesn't like that.
This diff teaches getdeps to disable `core.autocrlf` in GitHub Action Windows workers.
Reviewed By: vitaut
Differential Revision: D37008387
fbshipit-source-id: 34c9f89e8783d0613040d190c4ad3477bd7bfd53
Summary:
Applies new import merging and sorting from µsort v1.0.
When merging imports, µsort will make a best-effort to move associated
comments to match merged elements, but there are known limitations due to
the diynamic nature of Python and developer tooling. These changes should
not produce any dangerous runtime changes, but may require touch-ups to
satisfy linters and other tooling.
Note that µsort uses case-insensitive, lexicographical sorting, which
results in a different ordering compared to isort. This provides a more
consistent sorting order, matching the case-insensitive order used when
sorting import statements by module name, and ensures that "frog", "FROG",
and "Frog" always sort next to each other.
For details on µsort's sorting and merging semantics, see the user guide:
https://usort.readthedocs.io/en/stable/guide.html#sorting
Reviewed By: lisroach
Differential Revision: D36402214
fbshipit-source-id: b641bfa9d46242188524d4ae2c44998922a62b4c
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/113
GitHub Actions was failing at apt-get stage, error message suggested adding an apt-get update as remedy. Added in this diff and it indeed works.
The actions build fails later on missing mio::net module when building mysql async, but that's unrelated to this change.
Reviewed By: farnz
Differential Revision: D34368662
fbshipit-source-id: f0a00da3ee740ae4443a328616e792ea615c922c
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/110
Pull Request resolved: https://github.com/facebookexperimental/rust-shed/pull/27
Make it so that changes to rust-shed or other common rust source are used locally vendored, so they don't need to be pushed to github before they are visible in a build.
There was already some support for cargo vendoring in getdeps, but it was limited to dependencies between manifests built with cargo builder. This wasn't enough to build something like eden (cmake is main entry point, with later calls cargo) or eden_scm (make is main entry point, with later calls to cargo), so this diff adds a cargo prepare step for getdeps other primary build systems.
The cargo vendoring is done by using a cargo config file to point to the source files used by getdeps. It has two modes:
1. per crate, existing mode which is already automatic for cargo to cargo manifest dependencies. To use it for a non cargo build manifest, add crate.pathmap
2. per git url, existing mode which was only use for crates.io third-party crates, now can be enabled by setting cargo.cargo_config_file
Reviewed By: yancouto
Differential Revision: D33895469
fbshipit-source-id: 7b13c0b679532492a336ce217de875c25fe1be90
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/106
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/107
Summary
* Add EdenFS builds on external CI now EdenSCM is good
* Mac builds on github actions by using brew for system dependencies
To make this work had to fix some path ordering issues with install directories for Linux and Mac, and generalise the homebrew path fixups we were doing for bison to all the used homebrew packages.
Previously Installed packages were being added after system paths, so our own installed thing might be ignored. On github these meant system python 3.9 was being used for hg tests rather than our specified 3.8 (this showed we have some test fails on python 3.9 with "SystemError: deallocated bytearray object has exported buffers", that are beyond the scope of this diff to fix)
Also needed to include the getdeps generated python into the generated edenscmdeps3.zip archive setup.py produces otherwise EdenFS tests failed to import thrift.Thrift
Eden tests are hanging when run externally about half way through, so disable them on github actions for now as this PR is already fairly large. They work when run locally on an internal devserver, so probably some bit of environment necessary is not defined in the test runner
Reviewed By: chadaustin
Differential Revision: D34116505
fbshipit-source-id: d0d628db5daabc28d0bd8997cd5c1bc885ed1e73
Summary:
Pull Request resolved: https://github.com/facebook/folly/pull/1714
windows-2016 will be removed in March, and now that we've updated
Boost, we can upgrade to windows-2019.
Reviewed By: genevievehelsel
Differential Revision: D34119171
fbshipit-source-id: caad432bebf6b8282844bf79f82b1fbc34f2df9e
Summary: Add some basic support for FreeBSD to getdeps.
Reviewed By: ahornby
Differential Revision: D33989129
fbshipit-source-id: 42ff5f160b7e19c12196bb2e52a726f7815487bd
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/103
Automate maintenance of the edenscm_* github actions yamls
Add job file and name options and support for the Rust install section
Reviewed By: fanzeyi
Differential Revision: D34044422
fbshipit-source-id: 7d5f07d37bab1eff5de30a88e710dbf7479ca192
Summary: Fix the getdeps.py generation sub command and regenerate github actions before making changes to support actions generation for eden_scm
Reviewed By: fanzeyi
Differential Revision: D34044243
fbshipit-source-id: 0039d04f25af4c842145dc142dae6b9996fc8046
Summary: Make the --facebook-internal flag value available from BuildOptions so can have jobs of both values where required
Reviewed By: xavierd
Differential Revision: D33945588
fbshipit-source-id: 5d0e1b42973bd3e411a4a71075915bdb2755f0eb
Summary:
In order to speed up build times on a mac, add homebrew support to
getdeps.
Homebrew packages can be declared in a manifest using the `homebrew` header.
Futher, ahornby has added manifest entries for homebrew packages which are
included in this diff and also included a change to use the correct version of
openssl. Without this openssl change, homebrew cmake configure finds an old
openssl 1.0.2 install.
This diff provides a 2x speed up for building folly:
Timings for clean getdeps folly build on mid-2018 2.9Ghz i9 6 core intel macbook pro with 32GB RAM:
With new homebrew system deps:
```
rm -rf /Users/ahornby/.scratch/UsersZahornbyZfbsource/fbcode_builder_getdeps/
time ./opensource/fbcode_builder/getdeps.py build --allow-system-packages folly
real 17m39.329s
user 76m10.317s
sys 5m50.163s
```
Without:
```
rm -rf /Users/ahornby/.scratch/UsersZahornbyZfbsource/fbcode_builder_getdeps/
time ./opensource/fbcode_builder/getdeps.py build folly
real 32m10.344s
user 105m53.448s
sys 15m57.858s
```
Reviewed By: ahornby
Differential Revision: D33842632
fbshipit-source-id: ac785d4a8dcfa31b77292bddd9e747022ac36e3b
Summary: Boost 1.69 does not build on Fedora 35, so upgrade to Boost 1.78.
Reviewed By: ahornby
Differential Revision: D33855670
fbshipit-source-id: 85469a835a1dab1a7d5222413b1f1349bdcff280
Summary:
checkout@v2 is the current action version, and the old one may have
security issues. Let's just run with the latest version.
Reviewed By: ahornby
Differential Revision: D33834847
fbshipit-source-id: fba0733577bf23c208c9824aa3815bd01a813d9a
Summary:
Add support for overriding os, distro and distro version to command line when inspecting system packages so one can requested see ubuntu 18.04 package from other OS. Makes testing easier
Used shlex to shell unquote the value to be tested in the getdeps expression evaluator. getdeps expression parser didn't tolerate 18.04 as . is special char to getdeps expressions, needed to be "18.04"
Reviewed By: quark-zju
Differential Revision: D33741323
fbshipit-source-id: d83397c7fb5180a4d985d0d8ae7b3ff33b72f828
