Summary:
This change causes fbcode_builder (getdeps.py) to fail immediately on most non-zero error codes from delegated commands.
While internal tools may be able to surface failures in a build that doesn't fail fast, the debug cycle is difficult externally -- this makes failures pop out more easily and has been instrumental in debugging OSS build issues.
X-link: https://github.com/facebook/sapling/pull/1099
Differential Revision: D77608371
Pulled By: quark-zju
fbshipit-source-id: e9fc00a574bc64fbc165a22e51406e85d015e2ae
Summary:
X-link: https://github.com/facebook/sapling/pull/1005
Speed up github builds with github actions CI caching based on the cache-key mechanism already present in getdeps.
Cached fizz windows build completed in 22% of original time, linux build in 54% of the original time, and mac in 72% of original time. This saves a lot of waiting around for PR builds on OSS changes as windows signal is usually the lagging one. Speed ups will vary across the different Meta OSS projects.
Windows benefits most from caching dependencies as we don't use system packages there at all, linux next as its default runner is slower than mac, and then finaly mac (the strongest hardware of the default github runners).
Github allows [up to 10GB cache per repo](https://github.blog/changelog/2021-11-23-github-actions-cache-size-is-now-increased-to-10gb-per-repository/), expiring data over capacity. You can see the size of the caches generated in the [caches view of githhub actions UX](https://github.com/ahornby/fizz/actions/caches?query=sort%3Asize-desc), looks like our usage is pretty small so far.
More background:
Github actions caching decides its own compression format (currently it prefers zstd), but they are free to change that in future, hence no .zstd suffix or similar on the cache keys.
Github actions caches from main are used from feature branches but not vice versa, hence a PR can't pollute cache for the trunk build. This allows us to benefit from caching on main and PRs where dependency versions match.
The final item being built, and dependencies inside the same repo as the final are not cached. A different mechanism would be necessary for those (e.g. using a tool like ccache/sccache or caching cargo/buck2 output). This means that when building EdenFS, sapling could not be cached (its in the same repo), but folly could be as we have a key for it based on the folly-rev.txt file.
When there is a cache hit the build step is skipped using extension of the conditionals introduced in previous diff D67839708
Reviewed By: bigfootjon
Differential Revision: D67839730
fbshipit-source-id: c384a216eb27ccd3f816e3c31b167232bda571a6
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/1025
Add getdeps.py `env` subcommand to get the environment from getdeps in a sourcable form so that its easier to run/test/debug the binaries
There is an existing `debug` subcommand, but it starts an interactive shell inside getdeps, which isn't so useful for programatic use.
To get the output clean enough to source I switched getdeps print() logging to go to stderr.
example usage:
```
$ (source <(./build/fbcode_builder/getdeps.py --allow-system-packages env mononoke_integration); which mononoke)
```
Reviewed By: jdelliot
Differential Revision: D64982397
fbshipit-source-id: 65212936d42185e4d395557b56d3dba499caa128
Summary:
The cargo config was being unconditionally added, resutling in "duplicate key `crates-io` in table `source`" error
check if snippet is present before writing
Reviewed By: mzr
Differential Revision: D59803494
fbshipit-source-id: 701773d604c91fd6724069b0451b212cf0018251
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/897
Builder refactoring: instead of providing `install_dirs` to `build()`, `test()` etc., provide `loader` and `dep_manifests` when creating the builder. This is a cleaner API because we were computing `install_dirs` in multiple places before.
Furthermore this lets us do things that need to see the manifests of the dependencies, not just the list of `install_dirs`, such as treating direct dependencies differently from indirect dependencies (see D58244928).
Reviewed By: chadaustin
Differential Revision: D58200528
fbshipit-source-id: e52d35e84161b83ab49ab43099c3e3b9bb03f36e
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/653
getdeps: add --build-type option to build in Debug or RelWithDebInfo mode
Adds a --build-type option so one can force RelWithDebInfo or Debug explicity
Default remains RelWithDebInfo for cmake.
cargo default is updated to --release to match cmake more closely, if you don't want release use --build-type Debug.
If you want to run github CI in Debug mode (faster build, but tests will run slower), then can pass --build-type Debug to getdeps.py generate-github-actions
X-link: https://github.com/facebook/sapling/pull/786
Reviewed By: mitrandir77
Differential Revision: D51564770
Pulled By: bigfootjon
fbshipit-source-id: ef30332ca193d9805bce005d12b5dbc9f58fcafc
Summary:
X-link: https://github.com/facebookincubator/velox/pull/6923
fix mononoke linux local cargo build with lld
During my recent reviewstack changes I noticed that mononoke links fine on macOS, but github PRs are failing on linux with duplicate zstd symbols during linking.
To fix this, switch to lld for the final link on linux by setting the cargo options, which needs a getdeps cargo.py change to ensure they aren't overwritten
It works locally but looks like its running our of disk space on github. Next change in stack addresses github CI diskspace
X-link: https://github.com/facebook/sapling/pull/697
Reviewed By: sggutier
Differential Revision: D49875277
Pulled By: genevievehelsel
fbshipit-source-id: 998f525fd71333b31747d0ad0e0dda3ebe2b0796
Summary:
Context: https://fb.workplace.com/groups/rust.foundation.team/posts/1613252192522755
In D43983384 it was observed that "Build rust-shed" fails in `oss-mononoke-linux-getdeps` if you vendor a particular commit from https://github.com/jimblandy/perf-event.git. See this job on V7 of that diff: https://www.internalfb.com/intern/sandcastle/job/18014399413710280/insights
```
Updating git repository `https://github.com/jimblandy/perf-event.git`
error: failed to load source for dependency `perf-event`
Caused by:
Unable to update https://github.com/jimblandy/perf-event.git?rev=f15d5fa9c82a59d6134ca706b047a6baa2c24e58
Caused by:
revspec 'f15d5fa9c82a59d6134ca706b047a6baa2c24e58' not found; class=Reference (4); code=NotFound (-3)
```
In D44095532, I confirmed that without `git-fetch-with-cli = true` in getdeps the same code succeeds the "Build rust-shed" step.
https://www.internalfb.com/intern/sandcastle/job/906715711/insights
The only difference between V7 of krallin's diff and V1 of my working diff is:
```
$ hg diff -r 14b25d1e64dbbf6d627eb8243e2c9c9146ebb75f -r 5320eb7617dc4367184508058386020c5ea52eb7
diff --git a/fbcode/opensource/fbcode_builder/getdeps/cargo.py b/fbcode/opensource/fbcode_builder/getdeps/cargo.py
--- a/fbcode/opensource/fbcode_builder/getdeps/cargo.py
+++ b/fbcode/opensource/fbcode_builder/getdeps/cargo.py
@@ -45,6 +45,8 @@
# Enable using nightly features with stable compiler
env["RUSTC_BOOTSTRAP"] = "1"
env["LIBZ_SYS_STATIC"] = "1"
+ cmd = ["cargo", "--version"]
+ self._run_cmd(cmd, cwd=self.workspace_dir(), env=env)
cmd = [
"cargo",
operation,
@@ -88,9 +90,6 @@
[build]
target-dir = '''{}'''
-[net]
-git-fetch-with-cli = true
-
[profile.dev]
debug = false
incremental = false
```
Ultimately I guess the `git` CLI on the Sandcastle host must be ancient/problematic because I tested that Cargo is able to handle the following Cargo.toml even with `[net] get-fetch-with-cli = true` in .cargo/config.toml, with both git 2.34.1 (on my devserver) and git 2.39.2 (on my laptop).
```
[package]
name = "repro"
version = "0.0.0"
[dependencies]
perf-event = { git = "https://github.com/jimblandy/perf-event.git", rev = "f15d5fa9c82a59d6134ca706b047a6baa2c24e58", version = "0.4" }
```
I don't know why `[net] git-fetch-with-cli = true` is being used by getdeps. It showed up 3 years ago in D19193954 with no discussion.
Reviewed By: zertosh
Differential Revision: D44098634
fbshipit-source-id: 41b3fd81ed0ce71c31ed8199fed7864215f7a485
Summary:
Watchman's vagrant images rely on mapping a watchman source directory
fetched from fbsource into the VM. Thus, they sometimes look like
fbsource, and that's largely okay. However, the vendored cargo support
breaks. Instead, explicitly skip vendored cargo when there is no
fbsource dir.
Reviewed By: xavierd
Differential Revision: D36386674
fbshipit-source-id: 61f2af19507fecd2342cfc94bbb7120ab91c33b4
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/110
Pull Request resolved: https://github.com/facebookexperimental/rust-shed/pull/27
Make it so that changes to rust-shed or other common rust source are used locally vendored, so they don't need to be pushed to github before they are visible in a build.
There was already some support for cargo vendoring in getdeps, but it was limited to dependencies between manifests built with cargo builder. This wasn't enough to build something like eden (cmake is main entry point, with later calls cargo) or eden_scm (make is main entry point, with later calls to cargo), so this diff adds a cargo prepare step for getdeps other primary build systems.
The cargo vendoring is done by using a cargo config file to point to the source files used by getdeps. It has two modes:
1. per crate, existing mode which is already automatic for cargo to cargo manifest dependencies. To use it for a non cargo build manifest, add crate.pathmap
2. per git url, existing mode which was only use for crates.io third-party crates, now can be enabled by setting cargo.cargo_config_file
Reviewed By: yancouto
Differential Revision: D33895469
fbshipit-source-id: 7b13c0b679532492a336ce217de875c25fe1be90
Summary:
A long time ago, getdeps scheduled each build up to the number of hardware threads. For some builds, that was too heavy, so it got throttled back to only ever use half the hardware threads. This left parallelism on the table for CPU-bound projects that don't use much RAM per compilation.
This commit makes better use of the hardware with finer-grained logic that allows each manifest to specify a `job_weight_mib` estimate in MiB, and limit concurrency to `available_ram / job_weight`.
Reviewed By: ahornby
Differential Revision: D33754018
fbshipit-source-id: 785bed6c6cfe3c473244e0806a77cec1fc119e1f
Summary:
Move cargo logic to separate file as builder.py was getting a bit large
It's just a move of source file for CargoBuilder. Diff looks a bit big as I did hg cp to preserve history
Differential Revision: D33888925
fbshipit-source-id: 2d57343535ab087e09876edba4d00a6f0234fcd0
