Summary:
This change causes fbcode_builder (getdeps.py) to fail immediately on most non-zero error codes from delegated commands.
While internal tools may be able to surface failures in a build that doesn't fail fast, the debug cycle is difficult externally -- this makes failures pop out more easily and has been instrumental in debugging OSS build issues.
X-link: https://github.com/facebook/sapling/pull/1099
Differential Revision: D77608371
Pulled By: quark-zju
fbshipit-source-id: e9fc00a574bc64fbc165a22e51406e85d015e2ae
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/1025
Add getdeps.py `env` subcommand to get the environment from getdeps in a sourcable form so that its easier to run/test/debug the binaries
There is an existing `debug` subcommand, but it starts an interactive shell inside getdeps, which isn't so useful for programatic use.
To get the output clean enough to source I switched getdeps print() logging to go to stderr.
example usage:
```
$ (source <(./build/fbcode_builder/getdeps.py --allow-system-packages env mononoke_integration); which mononoke)
```
Reviewed By: jdelliot
Differential Revision: D64982397
fbshipit-source-id: 65212936d42185e4d395557b56d3dba499caa128
Summary:
Applies the black-fbsource codemod with the new build of pyfmt.
paintitblack
Reviewed By: lisroach
Differential Revision: D36324783
fbshipit-source-id: 280c09e88257e5e569ab729691165d8dedd767bc
Summary:
Pull Request resolved: https://github.com/facebookincubator/resctl/pull/8076
D26025779 (2b96889a0f) may have broken the getdeps build when logging
surrogates. Use errors=surrogateescape to try to avoid that.
Reviewed By: danobi
Differential Revision: D26079717
fbshipit-source-id: 98d938bfced417e0b39a6dcddb241225ffb69c06
Summary:
The log file should be opened with a defined encoding rather than
whatever the platform default is. On Windows, that might be cp1252,
which throws an encoding error when the text is unicode.
Reviewed By: genevievehelsel
Differential Revision: D26025779
fbshipit-source-id: 0102b2555e91812dfe94a7c332eb896ca2c9ef7e
Summary:
Now that the subprocess output is no longer piped, we can see a weird situation
where the command being run is displayed after the compilation step:
+ cd /data/users/xavierd/scratch/dataZusersZxavierdZfbsource/fbcode_builder_getdeps/build/eden && \
[1/13] rust_job_pool
Finished release [optimized] target(s) in 0.75s
[12/13] Install the project...
-- Install configuration: "RelWithDebInfo"
-- Installing: /data/users/xavierd/scratch/dataZusersZxavierdZfbsource/fbcode_builder_getdeps/installed/eden/bin/edenfs
-- Set runtime path of "/data/users/xavierd/scratch/dataZusersZxavierdZfbsource/fbcode_builder_getdeps/installed/eden/bin/edenfs" to ""
-- Up-to-date: /data/users/xavierd/scratch/dataZusersZxavierdZfbsource/fbcode_builder_getdeps/installed/eden/bin/edenfsctl
-- Up-to-date: /data/users/xavierd/scratch/dataZusersZxavierdZfbsource/fbcode_builder_getdeps/installed/eden/lib/libbackingstore_rs.a
-- Up-to-date: /data/users/xavierd/scratch/dataZusersZxavierdZfbsource/fbcode_builder_getdeps/installed/eden/lib/libbackingstore.a
-- Up-to-date: /data/users/xavierd/scratch/dataZusersZxavierdZfbsource/fbcode_builder_getdeps/installed/eden/include/eden/scm/lib/backingstore/c_api/HgNativeBackingStore.h
-- Up-to-date: /data/users/xavierd/scratch/dataZusersZxavierdZfbsource/fbcode_builder_getdeps/installed/eden/include/eden/scm/lib/backingstore/c_api/RustBackingStore.h
+ /data/users/xavierd/scratch/dataZusersZxavierdZfbsource/fbcode_builder_getdeps/installed/cmake-Ncng4tsJb6gdOu40ggy14-YtgNQD43 (f2db269ce0)k5ev0n-FXq99I/bin/cmake \
+ --build \
+ /data/users/xavierd/scratch/dataZusersZxavierdZfbsource/fbcode_builder_getdeps/build/eden \
+ --target \
+ install \
+ --config \
+ Release \
+ -j \
+ 24
This is a bit awkward. Flushing stdout's buffer allows for the ordering to be
correct.
Reviewed By: wez
Differential Revision: D23079405
fbshipit-source-id: e2bf25b098d6ab4a788a5ec07deb635a42cae18c
Summary:
Redirecting stdout means that ninja/cmake won't act as if it's invoked
interactively, ie: it will buffer the output, show every single files being
compiled (instead of just a line or progress), etc. This results in a fairly
janky UX when getdeps is used on the command line. By not redirecting stdout,
we get immediate feedback about the tests being run, and the files being
compiled.
Reviewed By: wez
Differential Revision: D22967815
fbshipit-source-id: 872ddbf421065686c384a3a876d0acb8832baf2e
Summary:
Fixes include:
1. Passing "GETDEPS_BUILD_DIR" and "GETDEPS_INSTALL_DIR" env variable and using them in eden/scm/Makefile rather than assuming the source code is always in the same place regardless getdeps arguments (it isn't).
2. Added "fbthrift-source" and "fb303-source" to avoid unnecessary compilation (at least of fb303) and to put fbthrift and fb303 source code in an easy to locate place inside getdeps' "installed" folder.
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/25
Test Plan: sandcastle, check oss-eden_scm-darwin-getdeps
Reviewed By: farnz
Differential Revision: D22431872
Pulled By: lukaspiatkowski
fbshipit-source-id: 8ccbb090713ec085a5dd56df509eb58ab6fb9e34
Summary: Currently output from build command is decoded with "surrogateescape" error handler, but when writing to log files/stdout we don't specify error handlers to be also "surrogateescape" according to https://docs.python.org/3/library/codecs.html#error-handlers, which could cause exception when there's surrogate characters logged in message.
Reviewed By: yfeldblum
Differential Revision: D21850411
fbshipit-source-id: 21c51d1ab2132171ae29f2d1fbe42655ebee94c5
Summary:
From the outset, we wanted to be sure that getdeps was able
to source and build the dependencies so that we knew that we'd have
a repeatable build. This came at the cost of build times: having
to build boost on each CI run is a bit of a chore.
This commit adds three new elements to the manifest files:
* `rpms` - a list of RPM names that are all required to be present
in order to consider the dependency satisfied
* `debs` - like `rpms` above, but scoped to debian package names
* `preinstalled.env` - a list of environment variables that if they
are all set and non-empty will satisfy the dependency.
A new `--allow-system-packages` option to getdeps enables the new
logic that looks for system packages; it is off by default, but
enabled in the generated GitHub Actions workflows.
A new `install-system-deps` subcommand is provided that will attempt
to install the system packages needed to satisfy the build. This
typically needs to be run via sudo and is thus broken out separately
from the main getdeps build flow.
I made a pass over the manifest files and added package names that
satisfy the build on ubuntu-18 and fedora-31.
shri-khare: I renamed the `Python3.7.6` manifest to just `python` as
part of this change; the version of python that it pulls in through
the normal build is the same and I believe that an equal or newer
version of python3 is available in the GH actions builder.
The `preinstalled.env` is used only by the boost manifest: it references
the name of an environment variable that is set by the github
windows hosts and that points to a pre-built and pre-installed
copy of boost. Since there is no package manager that we can
easily query for this sort of thing, probing from the environment
seems like a reasonable and fast way to check for this. We
may need to evolve this over time to become more feature rich,
but this seems like a good starting point.
This commit has the potential to save 20 minutes of build time
from each public CI build just due to the boost dependency alone!
Refs: https://github.com/facebook/watchman/pull/797
Reviewed By: yfeldblum
Differential Revision: D20740410
fbshipit-source-id: 6c38019449c54465127656c3d18a6ff1f30adaea
Summary:
Update the getdeps builders to save all command output to a log file in the
build directory. This makes it easier to go back after the fact and look at
the config options selected by the build run.
Reviewed By: wez
Differential Revision: D18777059
fbshipit-source-id: 86d3ffa48ec992fe90c59a3835e18f08310ed88c
Summary:
This runs a command, raising an exception if it exits with a non-zero error status.
It prints out the arguments in a mostly copy-and-pasteable form, with PATH-like
env vars pretty printed to make it easier to see what is being invoked; here's
an example of how cmake is being invoked later in this stack:
```
---
+ CMAKE_PREFIX_PATH=\
+ /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/installed/ninja-5d7ec7:\
+ /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/installed/cmake-91dc9a:\
+ PKG_CONFIG_PATH=\
+ /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/installed/ninja-5d7ec7/lib/pkgconfig:\
+ /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/installed/cmake-91dc9a/lib/pkgconfig:\
+ cd /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/build/zstd-470344 && \
+ cmake configure /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/repos/github.com-facebook-zstd.git/build/cmake -DCMAKE_INST
ALL_PREFIX=/data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/installed/zstd-470344 -DBUILD_SHARED_LIBS=OFF -DCMAKE_BUILD_TYPE=R
elWithDebInfo -G Ninja
```
Reviewed By: simpkins
Differential Revision: D14690999
fbshipit-source-id: cdb0c681c7dfdfdc6e8c96bf4830bfbcf666411b
