Summary: This helps to squash out some flakiness
Reviewed By: pkaush
Differential Revision: D17804696
fbshipit-source-id: decd8e5dd37d802c62cae1168c4f4d72c0fc5c83
Summary:
Update the generated `run_cmake.py` script to use `subprocess.run()` instead
of `os.execve()`. The `os.execve()` call doesn't really do what we want on
Windows: this causes the script to exit while CMake is still running,
resulting in confusing output. During the build step it also did not work
correctly with `vcvarsall.bat`, and using `subprocess` also solves this.
Reviewed By: wez
Differential Revision: D17493897
fbshipit-source-id: e0477627fc1824b0efcb1fa5a782d207853bcae8
Summary:
Add a new builder that can extract Python wheel files, and re-package them
for consumption by our add_fb_python_library() and add_fb_python_executable()
CMake functions. This is useful for dependencies on packages from PyPI.
At the moment this code only handles architecture-independent pure-Python
packages. It shouldn't be too hard to extend this to handle more complex
wheels, but for now I only need to use it for some pure-Python wheels and so I
haven't tested with more complex wheel files.
This also includes two new manifests for python-six and python-toml that take
use this new builder.
Reviewed By: wez
Differential Revision: D17401216
fbshipit-source-id: d6f74565887c3f004e1c06503dc9ec81599dd697
Summary:
while testing https://github.com/facebook/watchman/pull/743 I
noticed that the cmake builds were picking up the installed mingw GCC
compiler rather than the MSVC compiler. That would be fine except that
boost is built with MSVC and its generated libraries cannot be subsequently
found by a cmake gcc build that uses FindBoost.
This commit forces cmake to pick cl.exe rather than gcc. This is probably
fine to do unconditionally on windows, but since I've only observed this
particular problem with GitHub Actions I'm keeping it constrained to that
environment for now.
Reviewed By: simpkins
Differential Revision: D17385050
fbshipit-source-id: 90bef898b138e5d4bbd28a155ed1bd468acee9de
Summary:
This should make it easier to eg: enable fbzmq on macos at a later
time, and also makes things more explicit about what is being built, and
importantly, by listing these in the manifest ensures that we have a
hash change if we change this list; we wouldn't trigger such a change
if the list were encoded solely in builder.py
Reviewed By: chadaustin
Differential Revision: D17133149
fbshipit-source-id: caf0dd45e262188eeefafe0868ef95ad257a4950
Summary:
Feed the first-party oncall data through to the testpilot
invocation. This will set the owner of the test in continuous runs.
The oncall is passed through via the sandcastle module, which means
that you will have to manually pass it through for local testing
(see test plan), but it should automatically get picked up when
sandcastle schedules the `test` step of the job.
Reviewed By: chadaustin
Differential Revision: D17146802
fbshipit-source-id: a4f0e65853a46ed8709594c96db859ede2530b00
Summary:
Update the generated `run_cmake.py` script to use `os.execve()` rather than
`subprocess.call()`, so that it now propagates CMake's return code back to its
caller.
Reviewed By: chadaustin
Differential Revision: D17089206
fbshipit-source-id: e01f05f492ccb842d4967e59fd0bc9a3e59b8a42
Summary:
In the Facebook infrastructure the testpilot runner talks
to a service to determine whether a given test should be run locally.
The remote service was returning an error for some tests because they
didn't have enough configuration specified.
This diff associates some configuration with the tests based on the
host type and makes things happier.
Reviewed By: Ben0mega
Differential Revision: D16894682
fbshipit-source-id: b6f59b112991160ef26cc126e98041d8908a6273
Summary:
Update the generated `run_cmake.py` script to allow the caller to specify that
they just want to run a build without the install step.
Reviewed By: wez
Differential Revision: D16778007
fbshipit-source-id: 1859aca2b80fa7b099b4790682a6508e0185f2a0
Summary:
This cleans up how the `CMAKE_ENV` and `CMAKE_DEFINE_ARGS` variables are
written in the generated `run_cmake.py` script that we emit for CMake-based
projects.
We now emit each entry in these variables on separate lines, just to improve
readability. (Both of these variables tend to have a number of entries and
are very long if emitted on a single line.)
This also replaces the `-DCMAKE_INSTALL_PREFIX` entry in `CMAKE_DEFINE_ARGS`
to have it correctly honor the `INSTALL_DIR` variable defined in
`run_cmake.py`. This makes `run_cmake.py` still do the right thing if someone
manually edits it to change the `INSTALL_DIR` value.
Reviewed By: wez
Differential Revision: D16778006
fbshipit-source-id: fee5d25748b87b5d9c57ee2edf8de5e586e872ee
Summary:
While developing on a project it is often convenient to be able to invoke its
build manually, rather than always needing to re-run `getdeps.py`. This
updates the CMakeBuilder to also emit a script that can be used to manually
run CMake outside of `getdeps.py`.
The CMakeBuilder is the only builder that this really matters for right now,
as pretty much all of the projects where we do first-party development use
CMake for their build system.
Reviewed By: pkaush
Differential Revision: D16477399
fbshipit-source-id: c8a14af158af7b32d6c799ef685b037e68b748ff
Summary:
If `path_search()` returns that CMake is not available, raise an Exception and
fail the build. This makes the failure somewhat easier to to identify.
Without this the code would continue and would try to invoke `subprocess` with
a value of `None` in the command argument list. This error is slightly harder
to debug, since it isn't clear which command argument or environment variable
is not a string.
Reviewed By: chadaustin
Differential Revision: D16354623
fbshipit-source-id: be972b02cb13bc70db0f867da70e5bf4c6cec46d
Summary: I want to use this logic outside of a builder implementation
Reviewed By: pkaush
Differential Revision: D16101914
fbshipit-source-id: db3c9ac6c84a92ab84a18dddb931953b0a51f127
Summary:
We're seeing a failure on macOS in a CI environment but don't have much context:
```
Stderr:
+ GETDEPS=opensource/fbcode_builder/getdeps.py
+ opensource/fbcode_builder/getdeps.py test --facebook-internal watchman
Traceback (most recent call last):
File "opensource/fbcode_builder/getdeps.py", line 436, in <module>
sys.exit(main())
File "opensource/fbcode_builder/getdeps.py", line 422, in main
return args.func(args)
File "opensource/fbcode_builder/getdeps.py", line 342, in run
builder.run_tests(install_dirs, schedule_type=args.schedule_type)
File "/data/sandcastle/boxes/trunk-hg-fbcode-fbsource/fbcode/opensource/fbcode_builder/getdeps/builder.py", line 372, in run_tests
buck_test_info = list_tests()
File "/data/sandcastle/boxes/trunk-hg-fbcode-fbsource/fbcode/opensource/fbcode_builder/getdeps/builder.py", line 352, in list_tests
data = json.loads(output.decode("utf-8"))
File "/opt/homebrew/Cellar/python27/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "/opt/homebrew/Cellar/python27/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/opt/homebrew/Cellar/python27/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 384, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
```
Add some more context to that error message so we can learn more.
Reviewed By: pkaush
Differential Revision: D16061528
fbshipit-source-id: 23603a5d18651d20641ef1987b7094e73a9b1dbe
Summary: On Windows "--vcvars-path" can be passed to point to the toolchain we want to use.
Reviewed By: wez
Differential Revision: D15926044
fbshipit-source-id: 2b0cde793f7c7f8473b78afde8794640bae351f3
Summary:
D15588809 introduced manifest and customer builder for iproute2. While the
Iproute2Builder copies include/* from build to install directory, it does not
copy lib/* which contains the libnetlink.a
As a result, the build system looks up libnetlink.a on local machine (which it
may not find).
Fix it by explicitly coping lib/* as well.
Reviewed By: wez
Differential Revision: D15708835
fbshipit-source-id: 3f5906eb48ce982b6e887cca0d3c11314425beab
Summary:
We sometimes see some flakeyness in our internal CI, so we can live
without it there.
Reviewed By: pkaush
Differential Revision: D15695124
fbshipit-source-id: 1d76ae89e245d9c95937e8818826c544c0ae8fc3
Summary:
This is towards getting open source FBOSS to build using fbcode_builder.
iproute2 is one of the dependencies for FBOSS. This patch adds a manifest file
to build the specific version of iproute2 needed for FBOSS.
Additionally, the default git clone depth of 100 is insufficient for the
version of iproute2 FBOSS depends on. Thus, this patch extends the git SCHEMA
to add optional argument depth. The default remains 100.
The usual /configure --prefix does not work for iproute2. Thus, we need to add
a custom builder that:
- copies sources to build directory, builds, and
- installs to installed directory using DEST_DIR.
- it must also explicitly copy include from build dir to install dir
Reviewed By: wez
Differential Revision: D15588809
fbshipit-source-id: ac5eab24134e078d88b85b4be433c78b05ef8ce5
Summary:
This should enable test pilot to skip broken/flakey tests.
The `--tag-new-tests` flag is only appropriate for code that has been
landed on master and is used by the FB infra to classify new tests
appropriately.
For continuous builds we use a test collection with different parameters
from the normal developer facing flow so that the infra can re-assess
their status and enable/disable high-signal/noisy tests.
Depends: D15495344
Reviewed By: Ben0mega
Differential Revision: D15500955
fbshipit-source-id: f3b7976cec6a5cf70f5d128b38bde11620b26918
Summary:
The schedule type is used to differentiate between a build run during
code review (and thus runs code that isn't yet in the master branch),
from continuous or other types of run that operate on landed code.
This doesn't change any behavior yet; this diff just adds
the plumbing to pass down an optional arbitrary schedule type string
from the CI system.
In the future, we'll use the schedule type to influence the behavior
of running tests.
Reviewed By: strager
Differential Revision: D15300120
fbshipit-source-id: 3b46afef2ff171b3fa095763dd5006a54ea328b8
Summary:
if we find ccache in the path, then we instruct cmake to use
it as the compiler launcher.
Reviewed By: pkaush
Differential Revision: D15375441
fbshipit-source-id: 602fe54742a5ec07b0533bd2cf63cb905b051e85
Summary:
the cmake `add_test` and related functions allow specifying
the WORKING_DIRECTORY to use for tests. We weren't respecting this
value, so this diff looks up the WORKING_DIRECTORY from the ctest
json info and adjusts the buck test info json blob that we pass
on the testpilot.
Since that interface only allows passing an argv array, we use
the `cmake -E chdir` command to run a command in a specified
directory in a portable manner.
Reviewed By: strager
Differential Revision: D15274012
fbshipit-source-id: 1f02d461d73178745794703d455494e31c2e09ed
Summary:
Previously, we were using autoconf to build sqlite,
but that isn't available on Windows. Instead, here's a builder
that generates a little cmake configuration for building and
installing sqlite.
Using cmake for this means that we can test the same builder
on all platforms that need to pull in sqlite.
Reviewed By: simpkins
Differential Revision: D15179387
fbshipit-source-id: fccf969182565982bd5be55545a2d2625aa99124
Summary:
If `testpilot` is available, generate a buck compatible json file describing the available test binaries and feed that to testpilot to have it run the tests.
A later (yet to be written) diff will be able to pass appropriate flags down to testpilot in continuous runs and that will allow testpilot to auto-disable and file tasks for tests in the opensource builds.
Reviewed By: simpkins
Differential Revision: D14766856
fbshipit-source-id: 4e144ff18f6788cf5e830d29788eabd2dbbae46a
Summary:
Adds a `test` subcommand that runs the tests for project.
We're mostly interested in the 1st party facebook projects for this.
The `sandcastle` flow will run the `test` subcommand just for the "leaf" project--the one named on the command line.
Reviewed By: simpkins
Differential Revision: D14710331
fbshipit-source-id: 7d04a46cfd723894d61018de2f230140b52285ac
Summary:
this could do with a better name; the NopBuilder doesn't actually
build anything, but instead copies some files to the destination location.
This is used together with eg: cmake to install pre-built binaries downloaded
from a tarball.
Reviewed By: simpkins
Differential Revision: D14691015
fbshipit-source-id: a938e977aa4ec5a664bdb8085ff708319a204594
Summary:
the boost builder knows how to perform the non-standard
configure and build for boost.
Ideally we'd just build this statically and be happy but there are
some nuances I've observed while building on different platforms:
* One of our projects (thrift or wangle) explicitly requests static
boost linkage for reasons unspecified
* on darwin the install_name is broken when building dynamic libs
For the sake of expediency in getting getdeps up and running, the
solution for the moment is to build static on posix systems and
build both static and shared on windows systems.
Reviewed By: simpkins
Differential Revision: D14691009
fbshipit-source-id: 634770a6f53c3ada42d1877cc6c3dacc6eed7d18
Summary:
the openssl builder knows how to perform the non-standard
configuration and build steps to build openssl.
On Linux systems the manifests for our projects don't mention
openssl, causing them to pick up the system openssl.
On Mac, apple don't ship openssl headers so we need to build our own.
On Windows there is no standard openssl installation so we also need
to build our own.
As a result, this builder only works on windows and mac.
Reviewed By: simpkins
Differential Revision: D14691010
fbshipit-source-id: 9f8979f9eaeb5209c290cf4f43c97c0cb43d13a2
Summary:
this builder is used to bootstrap the ninja build tool.
On Windows and mac the manifest for ninja is set to download a pre-built executable.
While pre-built executables are available for linux they aren't portable enough
for our purposes so we need to be able to build it for ourselves.
Reviewed By: simpkins
Differential Revision: D14690992
fbshipit-source-id: b60fd02ad04f58dc7c2931280341791270609737
Summary:
the cmake builder knows how to use cmake to configure a build
for (preferably) and out-of-src build. The `cmake.defines` section of
the manifest is used to pass `-Dkey=value` options to the cmake configure
command line.
We prefer to use `ninja` to execute the build so that we can use more
cores than 1 on Windows and just for consistency across platforms
with mac and linux.
Reviewed By: simpkins
Differential Revision: D14690998
fbshipit-source-id: 8102e8b4a47da515ca001772788ed0e5f2645ad7
Summary:
the autoconf builder performs an out-of-source build using
the autoconf suite to configure the build rules.
The `autoconf.args` section from the manifest is passed to the `./configure`
command line.
If an `autogen.sh` script is present then it will be used to regenerate
a missing `configure` script, otherwise we'll try `autoreconf`.
Reviewed By: simpkins
Differential Revision: D14691002
fbshipit-source-id: ab8cceafb833dab513d5a50c65f4c895a4f40047
Summary:
the make builder runs `make` in the source directory.
The `make.args` section from the manifest is passed as arguments
to the `make` invocation.
Reviewed By: simpkins
Differential Revision: D14690996
fbshipit-source-id: 180d657ad05f0c0266a8c1d30979d8d1473958c9
Summary:
a builder knows how to build and install a project.
Later diffs add concrete implementations of the BuilderBase
Reviewed By: simpkins
Differential Revision: D14691018
fbshipit-source-id: 89b14614b5160353cd7e59f27037afcdf6229eb7
