Summary:
X-link: https://github.com/facebookincubator/fizz/pull/163
The github [windows-2019 actions image was retired by github](https://github.com/actions/runner-images/issues/12045), so all jobs on it fail
Update to windows-2022 to get them running again
windows-2022 has [different tools and versions than windows-2019](https://github.com/actions/runner-images/issues/3949). Notably it moves from Visual Studio 2019 (aka msvc 16.x) to Visual Studio 2022 (aka msvc 17.x), hence the update to buildopts.py discovery
In the course of regenerating the github actions I also fixed a couple of issues that stopped regeneration matching repo contents
* a few workflows were using workflow_dispatch, added support
* there were a trailing and double spaces for project_prefix, fixed (use ignore whitespace to remove this from review!)
Reviewed By: bigfootjon, yfeldblum
Differential Revision: D78019509
fbshipit-source-id: f8b0e9438bfc6b481b4207ad82bc1002e496a2d9
Summary:
This change causes fbcode_builder (getdeps.py) to fail immediately on most non-zero error codes from delegated commands.
While internal tools may be able to surface failures in a build that doesn't fail fast, the debug cycle is difficult externally -- this makes failures pop out more easily and has been instrumental in debugging OSS build issues.
X-link: https://github.com/facebook/sapling/pull/1099
Differential Revision: D77608371
Pulled By: quark-zju
fbshipit-source-id: e9fc00a574bc64fbc165a22e51406e85d015e2ae
Summary:
Fixes integration test failure due to lack of a functioning
python-psutil (on Linux, this requires the inclusion of native code
which is not permitted in a zipapp).
X-link: https://github.com/facebook/sapling/pull/1100
Reviewed By: zzl0
Differential Revision: D77674102
Pulled By: quark-zju
fbshipit-source-id: 11ac197d8c4082eaf16e0e28bc1a45c67f7dbb07
Summary:
X-link: https://github.com/facebook/fboss/pull/387
We want to copy libraries identified by ldd rather than the hardcoded mapping we have right now in package-fboss.py. These need to be packaged alongside the binaries as they are dynamically linked.
Reviewed By: paulcruz74
Differential Revision: D72731449
fbshipit-source-id: 79748615b7b0970269a5663b17f2102c6f537ffa
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/1102
make it possible to run the generated github actions locally with [act](https://github.com/nektos/act) when behind a proxy by preserving http_proxy in generated github actions sudo invocatons
also updated to consistently write `apt-get` instead of the interactive `apt` (we had a mix)
example usage (substitute myproxy for your actual proxy):
```
mkdir $HOME/act-artifacts
cd ~/fizz
time act -r -j build -W .github/workflows/getdeps_linux.yml --artifact-server-path=$HOME/act-artifacts --artifact-server-addr=127.0.0.1 --cache-server-addr=127.0.0.1 --env http_proxy=http://myproxy:8080 --env https_proxy=myproxy:8080 --action-offline-mode
```
Reviewed By: bigfootjon
Differential Revision: D67864689
fbshipit-source-id: 45695675ca5672e56104c8c33803afe004ff98da
Summary:
X-link: https://github.com/facebook/sapling/pull/1005
Speed up github builds with github actions CI caching based on the cache-key mechanism already present in getdeps.
Cached fizz windows build completed in 22% of original time, linux build in 54% of the original time, and mac in 72% of original time. This saves a lot of waiting around for PR builds on OSS changes as windows signal is usually the lagging one. Speed ups will vary across the different Meta OSS projects.
Windows benefits most from caching dependencies as we don't use system packages there at all, linux next as its default runner is slower than mac, and then finaly mac (the strongest hardware of the default github runners).
Github allows [up to 10GB cache per repo](https://github.blog/changelog/2021-11-23-github-actions-cache-size-is-now-increased-to-10gb-per-repository/), expiring data over capacity. You can see the size of the caches generated in the [caches view of githhub actions UX](https://github.com/ahornby/fizz/actions/caches?query=sort%3Asize-desc), looks like our usage is pretty small so far.
More background:
Github actions caching decides its own compression format (currently it prefers zstd), but they are free to change that in future, hence no .zstd suffix or similar on the cache keys.
Github actions caches from main are used from feature branches but not vice versa, hence a PR can't pollute cache for the trunk build. This allows us to benefit from caching on main and PRs where dependency versions match.
The final item being built, and dependencies inside the same repo as the final are not cached. A different mechanism would be necessary for those (e.g. using a tool like ccache/sccache or caching cargo/buck2 output). This means that when building EdenFS, sapling could not be cached (its in the same repo), but folly could be as we have a key for it based on the folly-rev.txt file.
When there is a cache hit the build step is skipped using extension of the conditionals introduced in previous diff D67839708
Reviewed By: bigfootjon
Differential Revision: D67839730
fbshipit-source-id: c384a216eb27ccd3f816e3c31b167232bda571a6
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/1100
Update generated github actions to only run the fetch and and build steps when there are sources expected for a manifest
For local github actions testing using `act` this speeds up the test runs, and in real github CI it makes it clearer which steps are actually doing something on the given runner (we don't know exactly what it has installed beforehand)
Also set the windows git config the same as on internal CI
Reviewed By: bigfootjon
Differential Revision: D67839708
fbshipit-source-id: 0a60c6fc89e8c6abb2464f879459aa23d5aec969
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/1086
Discussed with Opensource team further. 4-core-ubuntu-22.04 label does not work since it does not exist and runners can work only with the allocated list of labels.
Opensource team has now added necessary label, is added for 8-core as requested. So, now making code change for good, hopefully!
https://fb.workplace.com/groups/osssupport/permalink/27367197886235467/
Reviewed By: xiangxu1121
Differential Revision: D66487987
fbshipit-source-id: 62fa2b60b5613876aaa1f0c1a47dde8826746926
Summary:
X-link: https://github.com/facebookincubator/fizz/pull/152
X-link: https://github.com/facebookincubator/zstrong/pull/1073
On linux getdeps uses system openssl, on macOS we mostly use the homebrew one on github CI. Both of these are openssl3.x, however the fallback build from source version is still set as openssl1.1. This can be confusing, giving the impression getdeps based builds need openssl1.1
* Update the openssl manifest source url to openssl-3.0.15, this version picked to match the ubuntu 22.04 LTS system packages.
The other potentially confusing part was that the openssl source tarball was downloaded even when on on a platform like Linux where openssl is always taken from the system packages.
* Make the download configurable so that does nothing if satisfied from system packages, and no need to check the cache.
Reviewed By: ckwalsh
Differential Revision: D66495352
fbshipit-source-id: 4d24bb82bfabe44c7764b819de7f4a05f80daed1
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/1025
Add getdeps.py `env` subcommand to get the environment from getdeps in a sourcable form so that its easier to run/test/debug the binaries
There is an existing `debug` subcommand, but it starts an interactive shell inside getdeps, which isn't so useful for programatic use.
To get the output clean enough to source I switched getdeps print() logging to go to stderr.
example usage:
```
$ (source <(./build/fbcode_builder/getdeps.py --allow-system-packages env mononoke_integration); which mononoke)
```
Reviewed By: jdelliot
Differential Revision: D64982397
fbshipit-source-id: 65212936d42185e4d395557b56d3dba499caa128
Summary:
Connect up the sapling *.t tests so that github CI has visible cli test status.
To get them to run:
* some needed fbpython on path. I included a shim for that in the test makefile target.
* test-rust-hooks: Command not found message, added a glob
* test-identity.t: add a glob for the sapling version
* test-eolfilename.t: output order was unstable, added sorts to make stable
* helpers-testrepo.sh: fix assumption that system hg would be able to read test repo, check if its Sapling first.
* added a manifest for the hexdump utility some of the tests required
* excluded a few remaining tests (see comments in Makefile for reason)
* fixed getdeps support for generating actions steps for test only dependencies
NB the tests run as "hg". The expectations would need to be updated if we were to run as "sl"
X-link: https://github.com/facebook/sapling/pull/963
X-link: https://github.com/facebookincubator/zstrong/pull/1004
Reviewed By: quark-zju
Differential Revision: D63958737
Pulled By: ahornby
fbshipit-source-id: 75c0d39258c320100d8d02b31390994bc2f3a3ce
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/995
Bring back mononoke getdeps integration tests. This is a refesh of the previously working export-D34186407 branch from 2022
Main changes since:
* depend on restored sapling manifest from previous commit
* bring back selected manifests needed by mononoke that were deleted in D51869247. I added the sqlite binary packages from this to the sqlite3 manifest rather than bring back the sqllite3-bin manifest
* add manifests for new tools used: ripgrep, git-lfs, and zstd cli (we already had the zstd libs)
* fix a few test expecations that where too closely tied to git cli or TLS version etc (ubuntu 22.04 is on older version)
* getdeps MakefileBuilder.run_tests() improvements
* fix error status reporting, was not failing if tests failed
* pass of --num-jobs to tests to stop it OOMing my machine
* pass of --filter to tests so can iterate on one test more easily
Can to iterate local execution for one test with:
```
python3 ./build/fbcode_builder/getdeps.py test --num-jobs 4 --allow-system-packages --no-facebook-internal --src-dir=. mononoke_integration --retry 0 --filter server/test-gettreepack.t
```
X-link: https://github.com/facebook/sapling/pull/951
Reviewed By: quark-zju
Differential Revision: D62978526
Pulled By: ahornby
fbshipit-source-id: 0070a67d798bb23ee9e78e1a5149ba5364d548c9
Summary:
X-link: https://github.com/facebook/sapling/pull/952
fix getdeps generate-github-actions by adding missing process_project_dir_arguments. Without this couldn't specify any project_install_prefix other that the default /usr/local
mononoke installs directly to getdeps inst dir so needs a project_install_prefix of / for the artifact discovery to work
regenerated the gh actions with `python3 fbcode/opensource/fbcode_builder/getdeps.py generate-github-actions mononoke --os-type=linux --allow-system-packages --output-dir fbcode/eden/oss/.github/workflows --job-file-prefix mononoke_ --job-name-prefix "Mononoke " --free-up-disk --project-install-prefix mononoke:/`
Resolves https://github.com/facebook/sapling/issues/922
Reviewed By: bigfootjon
Differential Revision: D63031271
fbshipit-source-id: 768a4fbd7617c6061eca66b4d916ac25a4b66be9
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/907
I don't sure that I make all in accordance with the your contribution pipeline, so please correct me If there is needed.
So, I've made some changes to support the `pacman` package manager in your `getdeps.py` script. In `manifests` I'm also duplicated some packages from `debs` and `rpms` sections and create a new `pps` sections with according packages for `pacman`.
Issue: https://github.com/facebook/folly/issues/1701
X-link: https://github.com/facebook/folly/pull/1702
Reviewed By: yfeldblum
Differential Revision: D33514769
Pulled By: Orvid
fbshipit-source-id: a081c3a5bcb7f7cdde3a4a91c0d15517c0a171b3
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/897
Builder refactoring: instead of providing `install_dirs` to `build()`, `test()` etc., provide `loader` and `dep_manifests` when creating the builder. This is a cleaner API because we were computing `install_dirs` in multiple places before.
Furthermore this lets us do things that need to see the manifests of the dependencies, not just the list of `install_dirs`, such as treating direct dependencies differently from indirect dependencies (see D58244928).
Reviewed By: chadaustin
Differential Revision: D58200528
fbshipit-source-id: e52d35e84161b83ab49ab43099c3e3b9bb03f36e
Summary:
Starts a shell in the build dir, with the environment set up in the
same way as for the build. Useful for experimenting and testing.
Reviewed By: chadaustin
Differential Revision: D58082246
fbshipit-source-id: 82b275401528d7616c2560d80b4c187de67f6032
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/858
Because it needs to be passed to other commands that use project
hashes, such as "test".
(in general there are probably a lot more of these, just fixing the
ones I've ran into so far)
Reviewed By: genevievehelsel
Differential Revision: D58082245
fbshipit-source-id: 09fa6b5ce4cc4b3ae7ecfb34ac83681cecb36e3c
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/848
Because it should work with `show-inst-dir`, otherwise we can't
calculate the right project hash.
Reviewed By: chadaustin
Differential Revision: D58011867
fbshipit-source-id: d8960b4a993efbada8e27584e56976279fcd6b43
Summary: this runs on github... but also not what i really care about
Reviewed By: terrelln
Differential Revision: D57125977
fbshipit-source-id: 39d7fd319a0ce45a36ba11c4301f5a27356b6c9f
Summary:
there is a bunch of scripts that uses python3 to run, this means that they are at the mercy of whoever controll their path, in linux, this can be system python3, but can also be platform python (and probably it is both), and in windows in particular it abuses c:\tools\fb-python\python3.exe. fbpython is the universal way to run python at the company that chooses the right platform python, but also provides monitoring and observability.
this scripts generates jobs like
https://www.internalfb.com/sandcastle/job/18014399781484898/
that will stop working once we remove c:\tools\fb-python\python3*.exe
Reviewed By: fried
Differential Revision: D56896564
fbshipit-source-id: d911fdaf6750635adb05b096f0522603baf47bcc
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/749
Updating generated workflow files to account for recent changes. Most notably, this updates the checkout action to v4 and sets an explicit read-only permission.
This also adds support for `--cron` in the codegen to account for only running CI on a schedule (useful for managing costs).
Reviewed By: ahornby
Differential Revision: D56165825
fbshipit-source-id: 298b16effefb6b8a2dc6cbcf07d4ec4a61f48364
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/653
getdeps: add --build-type option to build in Debug or RelWithDebInfo mode
Adds a --build-type option so one can force RelWithDebInfo or Debug explicity
Default remains RelWithDebInfo for cmake.
cargo default is updated to --release to match cmake more closely, if you don't want release use --build-type Debug.
If you want to run github CI in Debug mode (faster build, but tests will run slower), then can pass --build-type Debug to getdeps.py generate-github-actions
X-link: https://github.com/facebook/sapling/pull/786
Reviewed By: mitrandir77
Differential Revision: D51564770
Pulled By: bigfootjon
fbshipit-source-id: ef30332ca193d9805bce005d12b5dbc9f58fcafc
Summary:
X-link: https://github.com/facebookincubator/velox/pull/7446
Add a new command to show the location of the scratch dir that getdeps
will use.
Reviewed By: xavierd
Differential Revision: D51027807
fbshipit-source-id: 80a7cfc04320002588d5ec8964fc88914771c6c7
Summary:
X-link: https://github.com/facebookincubator/velox/pull/7072
use system patchelf in eden fs build
Saves us from doing a fetch and build of autoconf, libtool, automake and patchelf during the arfifacts part of CI
X-link: https://github.com/facebook/sapling/pull/750
Reviewed By: sggutier
Differential Revision: D50313417
Pulled By: genevievehelsel
fbshipit-source-id: 7c585357c848c15a65c5797d6c8750d1119b6efd
Summary:
X-link: https://github.com/facebookincubator/velox/pull/7071
fix CI warning about node12 by updating actions/checkout version
Update the actions/checkout version to fix build warning
X-link: https://github.com/facebook/sapling/pull/749
Reviewed By: sggutier
Differential Revision: D50313430
Pulled By: genevievehelsel
fbshipit-source-id: 4d55d36f7509be51bb4e26728ddbce8e45faba40
Summary:
X-link: https://github.com/facebookincubator/velox/pull/6943
Update to boost 1.83. This addresses a folly build failure on Xcode 15, due to
boost 1.78 referring to the removed std::unary_function.
Reviewed By: jdelliot
Differential Revision: D49972186
fbshipit-source-id: 8d9fdd27c24ccc5a072b6973b86e0c8ed5b77ac3
Summary:
X-link: https://github.com/facebookincubator/velox/pull/6924
fix getdeps actions generation for rust toolchain
Generated actions files created from getdeps.py were missing rust toolchain which meant annoying manual edits
This change also makes the github actions generation honor the --no-tests argument, which is useful when regenerating the edenfs actions
X-link: https://github.com/facebook/sapling/pull/682
Reviewed By: sggutier
Differential Revision: D49875258
Pulled By: genevievehelsel
fbshipit-source-id: 173f86083ba92bab4063813d3e392df428b9ffe4
Summary:
X-link: https://github.com/facebookincubator/velox/pull/6927
allow getdeps github actions to free up disk
Allow getdeps to free up some disk from the runner and intermediate build steps as some runs (notably the linux eden and mononoke ones) are hitting disk space limits
X-link: https://github.com/facebook/sapling/pull/689
Reviewed By: sggutier
Differential Revision: D49875256
Pulled By: genevievehelsel
fbshipit-source-id: b85b6b2f11857670915b64f47d3c0abd4ca8ca31
Summary:
X-link: https://github.com/facebookincubator/velox/pull/4796
`brew` no longer supports running as root:
```
Error: Running Homebrew as root is extremely dangerous and no longer supported.
As Homebrew does not drop privileges on installation you would be giving all
build scripts full access to your system.
Command '['brew', 'install', 'autoconf', 'automake', 'boost', 'cmake', 'double-conversion', 'icu4c', 'libevent', 'libsodium', 'libtool', 'lz4', 'ninja', 'openssl@1.1', 'xz', 'zlib', 'zstd']' returned non-zero exit status 1.
!! Failed
```
And as a result the recommended install instructions no longer work:
```
sudo ./build/fbcode_builder/getdeps.py install-system-deps ...
```
and cannot be made to work portably across platforms because Linux requires `sudo` be there while macOS requires `sudo` not be there.
To fix this move `sudo` to Linux system dependency installation only.
Also update `apt-get` to `apt` while at it.
Resolves https://github.com/facebook/fbthrift/issues/545.
Reviewed By: somasun
Differential Revision: D45371004
fbshipit-source-id: 75334db22226efc961e7d4d6c6eca911086b97ba
Summary:
X-link: https://github.com/facebookincubator/velox/pull/4087
Under some circumstances, getdeps can cache a project artifact
where there is no built_marker, forcing the script to re-build the artifact
from source. After D43260530 (47dde7c249) this no longer causes a build failure, but it can
still make builds take longer than they should.
This ensures we only cache artifacts with a built_marker, so that they aren't
overwritten by artifacts without.
Reviewed By: genevievehelsel
Differential Revision: D43405855
fbshipit-source-id: b1b9e194e642fa820d0fbc54a7c15ff81fc90a82
Summary:
Automation should use `apt-get` instead of `apt`, as `apt` does not
have a stable interface.
This fixes a warning when running getdeps.py on recent Ubuntu:
```
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
```
Reviewed By: fanzeyi
Differential Revision: D39031719
fbshipit-source-id: 54a0fe51ce159174b089fc446e99a4ab5d69d626
Summary:
In D36905191 (6e6bf12f62), getdeps gained the ability to apply patches on Windows.
However, on GitHub Action workers, `core.autocrlf` is turned on by default. This means the patch files we checked in will end up having CRLF endings while the source code we downloaded are still in LF endings, and `git apply` doesn't like that.
This diff teaches getdeps to disable `core.autocrlf` in GitHub Action Windows workers.
Reviewed By: vitaut
Differential Revision: D37008387
fbshipit-source-id: 34c9f89e8783d0613040d190c4ad3477bd7bfd53
