Summary:
In order to do what the title says, this diff does:
1. Add the `eden/oss/.../third-party/rust/.../Cargo.toml` files. As mentioned in the previous diff, those are required by GitHub so that the third party dependencies that are local in fbsource are properly defined with a "git" dependency in order for Cargo to "link" crates properly.
2. Changes to `eden/scm/Makefile` to add build/install commands for getdeps to invoke. Those command knowing that they are called from withing getdeps context they link the dependencies brought by getdeps into their proper places that match their folder layout in fbsource. Those Makefile commands also pass a GETDEPS_BUILD env to the setup.py invocations so that it knows it is being called withing a getdeps build.
3. Changes to `eden/scm/setup.py` that add "thriftasset" that makes use of the getdeps.py provided "thrift" binary to build .py files out of thrift files.
4. Changes to `distutils_rust` to use the vendored crates dir provided by getdeps.
5. Changes to `getdeps/builder.py` and `getdeps/manifest.py` that enable more fine-grained configuratior of how Makefile builds are invoked.
6. Changes to `getdeps/buildopts.py` and `getdeps/manifest.py` to disable overriding PATH and pkgconfig env, so that "eden/scm" builds in getdeps using system libraries rather than getdeps-provided ones (NOTE: I've tried to use getdeps provided libraries, but the trickiest bit was that Rust links with Python, which is currently not providable by getdeps, so if you try to build everything the system provided Python libraries will collide with getdeps provided ones)
7. Added `opensource/fbcode_builder/manifests/eden_scm` for the getdeps build.
Reviewed By: quark-zju
Differential Revision: D22336485
fbshipit-source-id: 244d10c9e06ee83de61e97e62a1f2a2184d2312f
Summary: This bug can be triggered when your computer name contains emoji. getdeps.py will fail to create this file due to Python attempts to write the file as cp1252 (Windows's default encoding)
Reviewed By: wez
Differential Revision: D22171935
fbshipit-source-id: fc3be2d1050c17ddbe05a0fc91d6613865f092ce
Summary:
As per https://github.com/actions/virtual-environments/issues/709 there started to be some issies with Ubuntu envs running out of space. This should fix it.
Also our Cargo builds use a lot of space, changing them to be non-incremental and removing debug symbols keeps the build fast, but greatly reduces the disk space usage leaving us enough space on GitHub Actions virtual machines.
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/23
Reviewed By: farnz
Differential Revision: D22160020
Pulled By: lukaspiatkowski
fbshipit-source-id: c23393e310c15ebf5a18b80f0bb5f1f894d24849
Summary:
Two changes here:
1. The `[patch.crates-io]` section of `third-party/rust/Cargo.toml` is being now copied over to workspaces generated by autocargo for OSS and in the runtime generated Cargo.toml file for cargo-fbcode builds. Without that some projects could be buildable in Buck internally, but not externally on GitHub due to missing patches.
2. If a `[workspace]` Cargo.toml file is being generated and there is already a generated Cargo.toml file in the same directory then instead of overriding that file the `[workspace]` (and `[patch]`) sections are appended to that Cargo.toml file.
Reviewed By: farnz
Differential Revision: D22023144
fbshipit-source-id: dec54491c36c2ee0ab29eefb722b3eceaef6ffe1
Summary:
For large projects, with lots of tests, running all the tests can take a lot
of time, but for quick development iteration, only a subset of the tests may
be needed to run.
On non-Windows platforms, this can be easily achieved by manually executing
the individual tests binaries and use the builtin filtering mechanism of that
test binary to achieve the goal. On Windows, this can quickly become
impossible as DLLs might not be available, and the right PATH would need to
be manually specified by hand to execute the tests binaries[0].
To solve this, let's simply provide a unified way of running specific tests
by passing in a regexp. Both testpilot and CTest do support regex to execute
specific tests. My understanding is that cargo doesn't yet allows regex, but
will in the future.
[0]: And a missing DLLs would produce no output when executed from
PowerShell, which makes this very confusing.
Reviewed By: wez
Differential Revision: D21484774
fbshipit-source-id: ee32e950e25bb2a498a2b364a447955a917b0590
Summary:
Previously getdeps would remove the entire top-level `CMakeFiles` directory
from the build output when it wanted to invalidate the CMake cache. This
directory is used to keep all of the compiled object files for any libraries
or executables defined in the top-level CMakeLists.txt file. Blowing away
this directory forces all of these sources to be re-compiled, even if this was
not necessary. This is particularly problematic for folly, which compiles all
of its source files via rules in the top-level CMakeLists.txt target file.
I did have the code still blow away the CMake error and output logs in this
directory: in the past I have seen situations where CMake would not update
these files on new CMake runs if they already existed.
Reviewed By: wez
Differential Revision: D21360668
fbshipit-source-id: 6fcd1a8e371d756114fbab60d8636be8cd5f8978
Summary:
In the initial stages of the windows port we had
problems building rocksdb on windows, so we disabled it.
These days we're able to build it and detect it--we even
require it in the cmake code, but hadn't gotten around
to telling the rest of the code that we can use it.
This commit re-enables it in the build but leaves sqlite
as the default engine until we're able to perform some
benchmarking.
Rocksdb itself has some build issues on Windows; it doesn't
use cmake to locate dependencies, so even though we built
snappy it doesn't know how to find it without modifying the
source:
https://github.com/facebook/rocksdb/blob/master/thirdparty.inc#L4
For that reason, we disable the use of Snappy in the Windows build.
However, in the version of rocksdb that we were using, it would
default to trying to use Snappy even though it wasn't compiled in
and throw an exception.
I've upgraded to a newer version of rocksdb that will simply not
use compression if no compression was enabled at build time.
Given that we mostly store relatively small objects, I'm assuming
that the lack of compression is fine for now.
Reviewed By: xavierd
Differential Revision: D21319896
fbshipit-source-id: 2a2d06d4bd5382706e9220f9b4a2de99dc18311d
Summary:
Update the generated `run_cmake.py` script to tell ctest to print the test
output on failure. Also pass in a `-j` flag to run tests in parallel by
default.
These flags are already passed in by default when running `getdeps.py test`;
this simply updates this developer utility script to do the same.
Reviewed By: wez
Differential Revision: D21307806
fbshipit-source-id: 42045b0f9362494042c79bc946a1004ff8ad98b6
Summary:
On Windows the build artifacts cannot be easily run directly from the build
output directory without installing them. The `$PATH` environment variable
needs to be set correctly so that the runtime library dependencies can be
found.
This updates the builder code to emit a `run.ps1` wrapper script in the build
output directory that sets `$PATH` to support running build artifacts directly
from the build directory.
Additionally, this updates the CMake-specific builder to set properly when
running the tests with `ctest`.
Reviewed By: wez
Differential Revision: D20688290
fbshipit-source-id: 5d0f4d685692bca7e37370bd88309cf7634d87f0
Summary:
We have a global `--install-prefix` argument that can be used to set
the prefix for all projects, but that is only suitable if you are running with
sufficient privileges to install each of the deps to that location during the
build. Cmake dependency resolution won't work from the build directory in that
situation; it can only see the final installed location and it will error out
if those files are not present, or link against the currently installed version
instead of the version we just built; not great!
This commit adds a project specific `--project-install-prefix` that can be used
on just the leaf project in a set of deps. That sidesteps the dependency
concern because only the last stage is built in that mode. This option
can technically be applied to an arbitrary set of projects, but in light
of the above, in practice it only makes sense to use it for the final
cmake project. Only the CMakeBuilder respects this option.
In the watchman repo, this commit adjusts the autogen.sh script to allow
specifying the installation prefix; it defaults to `/usr/local` as you
might expect.
refs: https://github.com/facebook/watchman/issues/760
Reviewed By: yfeldblum
Differential Revision: D20674439
fbshipit-source-id: 52799dbd47f3c295e2d6469ee2b74cedeaa20138
Summary:
Take the README.md from
7ead0e29e4/README.md
and apply it on Eden repo.
Re-add the Cargo.toml file that declares Cargo workspace.
Re-add fbcode_builder/getdeps manifest for Mononoke
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/13
Test Plan:
./build/fbcode_builder/getdeps.py build mononoke
./build/fbcode_builder/getdeps.py test mononoke
Reviewed By: ahornby
Differential Revision: D19833059
Pulled By: lukaspiatkowski
fbshipit-source-id: fb37e13306c0b9969a7c4e52b05e1a66a577022f
Summary:
Instead of fetching a tarball for the vendored Rust crates, let's just use
what's present in third-party/rust.
Reviewed By: wez, fanzeyi
Differential Revision: D19770580
fbshipit-source-id: ca6615a5f5c672d9a5527ad47feab6bdd33a39db
Summary:
With this change the getdeps' CargoBuilder will support depencies between cargo builds.
The way how it works is documented in the code and required few assumptions about how a cargo project has to be defined in order to support this.
This change also adds the "mononoke" manifest and few Cargo.toml files to the mononoke project to prove that this new feature works.
Reviewed By: farnz
Differential Revision: D19468912
fbshipit-source-id: f299733cbbc2ec7bca399c898ec8d710334d0fa9
Summary:
Broadcom provides this library and they decided to rename it from OpenBCM to
OpenNSA. Thus, rename corresponding fbcode_builder code.
Reviewed By: wez
Differential Revision: D19396687
fbshipit-source-id: 8233dbf4de9342b5a0e54ae275d6c73d43abe6d0
Summary:
Update the getdeps builders to save all command output to a log file in the
build directory. This makes it easier to go back after the fact and look at
the config options selected by the build run.
Reviewed By: wez
Differential Revision: D18777059
fbshipit-source-id: 86d3ffa48ec992fe90c59a3835e18f08310ed88c
Summary:
OpenBCM libraries are stored with git LFS. As a result, fetcher fetches LFS pointers and not the contents. Use git-lfs to pull the real contents before copying to install dir using NoopBuilder.
In future, if more builders require git-lfs, we would consider installing
git-lfs as part of the sandcastle infra as against repeating similar
logic for each builder that requires git-lfs.
Reviewed By: wez
Differential Revision: D18759806
fbshipit-source-id: f988a0460107bc0685e7aba107daba9ed88f71e7
Summary: The cargo builder will be used to verify if an opensource Rust project passes Cargo build, test and (optionally) documentation build.
Reviewed By: markbt
Differential Revision: D18636934
fbshipit-source-id: e982e6a017eb32913e2994e7457c8add2e9d6b95
Summary: Ask testpilot to include more output in test failures.
Reviewed By: fanzeyi
Differential Revision: D18061772
fbshipit-source-id: 0c14092557c21396c877d3b1776c5707437a117c
Summary: testpilot's defaults assume a bigger machine than some of our laptops
Reviewed By: fanzeyi
Differential Revision: D17878120
fbshipit-source-id: e01f1f9c77a4f5f011051c9c642dbe934c66bc0b
Summary: This helps to squash out some flakiness
Reviewed By: pkaush
Differential Revision: D17804696
fbshipit-source-id: decd8e5dd37d802c62cae1168c4f4d72c0fc5c83
Summary:
Update the generated `run_cmake.py` script to use `subprocess.run()` instead
of `os.execve()`. The `os.execve()` call doesn't really do what we want on
Windows: this causes the script to exit while CMake is still running,
resulting in confusing output. During the build step it also did not work
correctly with `vcvarsall.bat`, and using `subprocess` also solves this.
Reviewed By: wez
Differential Revision: D17493897
fbshipit-source-id: e0477627fc1824b0efcb1fa5a782d207853bcae8
Summary:
Add a new builder that can extract Python wheel files, and re-package them
for consumption by our add_fb_python_library() and add_fb_python_executable()
CMake functions. This is useful for dependencies on packages from PyPI.
At the moment this code only handles architecture-independent pure-Python
packages. It shouldn't be too hard to extend this to handle more complex
wheels, but for now I only need to use it for some pure-Python wheels and so I
haven't tested with more complex wheel files.
This also includes two new manifests for python-six and python-toml that take
use this new builder.
Reviewed By: wez
Differential Revision: D17401216
fbshipit-source-id: d6f74565887c3f004e1c06503dc9ec81599dd697
Summary:
while testing https://github.com/facebook/watchman/pull/743 I
noticed that the cmake builds were picking up the installed mingw GCC
compiler rather than the MSVC compiler. That would be fine except that
boost is built with MSVC and its generated libraries cannot be subsequently
found by a cmake gcc build that uses FindBoost.
This commit forces cmake to pick cl.exe rather than gcc. This is probably
fine to do unconditionally on windows, but since I've only observed this
particular problem with GitHub Actions I'm keeping it constrained to that
environment for now.
Reviewed By: simpkins
Differential Revision: D17385050
fbshipit-source-id: 90bef898b138e5d4bbd28a155ed1bd468acee9de
Summary:
This should make it easier to eg: enable fbzmq on macos at a later
time, and also makes things more explicit about what is being built, and
importantly, by listing these in the manifest ensures that we have a
hash change if we change this list; we wouldn't trigger such a change
if the list were encoded solely in builder.py
Reviewed By: chadaustin
Differential Revision: D17133149
fbshipit-source-id: caf0dd45e262188eeefafe0868ef95ad257a4950
Summary:
Feed the first-party oncall data through to the testpilot
invocation. This will set the owner of the test in continuous runs.
The oncall is passed through via the sandcastle module, which means
that you will have to manually pass it through for local testing
(see test plan), but it should automatically get picked up when
sandcastle schedules the `test` step of the job.
Reviewed By: chadaustin
Differential Revision: D17146802
fbshipit-source-id: a4f0e65853a46ed8709594c96db859ede2530b00
Summary:
Update the generated `run_cmake.py` script to use `os.execve()` rather than
`subprocess.call()`, so that it now propagates CMake's return code back to its
caller.
Reviewed By: chadaustin
Differential Revision: D17089206
fbshipit-source-id: e01f05f492ccb842d4967e59fd0bc9a3e59b8a42
Summary:
In the Facebook infrastructure the testpilot runner talks
to a service to determine whether a given test should be run locally.
The remote service was returning an error for some tests because they
didn't have enough configuration specified.
This diff associates some configuration with the tests based on the
host type and makes things happier.
Reviewed By: Ben0mega
Differential Revision: D16894682
fbshipit-source-id: b6f59b112991160ef26cc126e98041d8908a6273
Summary:
Update the generated `run_cmake.py` script to allow the caller to specify that
they just want to run a build without the install step.
Reviewed By: wez
Differential Revision: D16778007
fbshipit-source-id: 1859aca2b80fa7b099b4790682a6508e0185f2a0
Summary:
This cleans up how the `CMAKE_ENV` and `CMAKE_DEFINE_ARGS` variables are
written in the generated `run_cmake.py` script that we emit for CMake-based
projects.
We now emit each entry in these variables on separate lines, just to improve
readability. (Both of these variables tend to have a number of entries and
are very long if emitted on a single line.)
This also replaces the `-DCMAKE_INSTALL_PREFIX` entry in `CMAKE_DEFINE_ARGS`
to have it correctly honor the `INSTALL_DIR` variable defined in
`run_cmake.py`. This makes `run_cmake.py` still do the right thing if someone
manually edits it to change the `INSTALL_DIR` value.
Reviewed By: wez
Differential Revision: D16778006
fbshipit-source-id: fee5d25748b87b5d9c57ee2edf8de5e586e872ee
Summary:
While developing on a project it is often convenient to be able to invoke its
build manually, rather than always needing to re-run `getdeps.py`. This
updates the CMakeBuilder to also emit a script that can be used to manually
run CMake outside of `getdeps.py`.
The CMakeBuilder is the only builder that this really matters for right now,
as pretty much all of the projects where we do first-party development use
CMake for their build system.
Reviewed By: pkaush
Differential Revision: D16477399
fbshipit-source-id: c8a14af158af7b32d6c799ef685b037e68b748ff
Summary:
If `path_search()` returns that CMake is not available, raise an Exception and
fail the build. This makes the failure somewhat easier to to identify.
Without this the code would continue and would try to invoke `subprocess` with
a value of `None` in the command argument list. This error is slightly harder
to debug, since it isn't clear which command argument or environment variable
is not a string.
Reviewed By: chadaustin
Differential Revision: D16354623
fbshipit-source-id: be972b02cb13bc70db0f867da70e5bf4c6cec46d
Summary: I want to use this logic outside of a builder implementation
Reviewed By: pkaush
Differential Revision: D16101914
fbshipit-source-id: db3c9ac6c84a92ab84a18dddb931953b0a51f127
Summary:
We're seeing a failure on macOS in a CI environment but don't have much context:
```
Stderr:
+ GETDEPS=opensource/fbcode_builder/getdeps.py
+ opensource/fbcode_builder/getdeps.py test --facebook-internal watchman
Traceback (most recent call last):
File "opensource/fbcode_builder/getdeps.py", line 436, in <module>
sys.exit(main())
File "opensource/fbcode_builder/getdeps.py", line 422, in main
return args.func(args)
File "opensource/fbcode_builder/getdeps.py", line 342, in run
builder.run_tests(install_dirs, schedule_type=args.schedule_type)
File "/data/sandcastle/boxes/trunk-hg-fbcode-fbsource/fbcode/opensource/fbcode_builder/getdeps/builder.py", line 372, in run_tests
buck_test_info = list_tests()
File "/data/sandcastle/boxes/trunk-hg-fbcode-fbsource/fbcode/opensource/fbcode_builder/getdeps/builder.py", line 352, in list_tests
data = json.loads(output.decode("utf-8"))
File "/opt/homebrew/Cellar/python27/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "/opt/homebrew/Cellar/python27/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/opt/homebrew/Cellar/python27/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 384, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
```
Add some more context to that error message so we can learn more.
Reviewed By: pkaush
Differential Revision: D16061528
fbshipit-source-id: 23603a5d18651d20641ef1987b7094e73a9b1dbe
Summary: On Windows "--vcvars-path" can be passed to point to the toolchain we want to use.
Reviewed By: wez
Differential Revision: D15926044
fbshipit-source-id: 2b0cde793f7c7f8473b78afde8794640bae351f3
Summary:
D15588809 introduced manifest and customer builder for iproute2. While the
Iproute2Builder copies include/* from build to install directory, it does not
copy lib/* which contains the libnetlink.a
As a result, the build system looks up libnetlink.a on local machine (which it
may not find).
Fix it by explicitly coping lib/* as well.
Reviewed By: wez
Differential Revision: D15708835
fbshipit-source-id: 3f5906eb48ce982b6e887cca0d3c11314425beab
Summary:
We sometimes see some flakeyness in our internal CI, so we can live
without it there.
Reviewed By: pkaush
Differential Revision: D15695124
fbshipit-source-id: 1d76ae89e245d9c95937e8818826c544c0ae8fc3
Summary:
This is towards getting open source FBOSS to build using fbcode_builder.
iproute2 is one of the dependencies for FBOSS. This patch adds a manifest file
to build the specific version of iproute2 needed for FBOSS.
Additionally, the default git clone depth of 100 is insufficient for the
version of iproute2 FBOSS depends on. Thus, this patch extends the git SCHEMA
to add optional argument depth. The default remains 100.
The usual /configure --prefix does not work for iproute2. Thus, we need to add
a custom builder that:
- copies sources to build directory, builds, and
- installs to installed directory using DEST_DIR.
- it must also explicitly copy include from build dir to install dir
Reviewed By: wez
Differential Revision: D15588809
fbshipit-source-id: ac5eab24134e078d88b85b4be433c78b05ef8ce5
Summary:
This should enable test pilot to skip broken/flakey tests.
The `--tag-new-tests` flag is only appropriate for code that has been
landed on master and is used by the FB infra to classify new tests
appropriately.
For continuous builds we use a test collection with different parameters
from the normal developer facing flow so that the infra can re-assess
their status and enable/disable high-signal/noisy tests.
Depends: D15495344
Reviewed By: Ben0mega
Differential Revision: D15500955
fbshipit-source-id: f3b7976cec6a5cf70f5d128b38bde11620b26918
Summary:
The schedule type is used to differentiate between a build run during
code review (and thus runs code that isn't yet in the master branch),
from continuous or other types of run that operate on landed code.
This doesn't change any behavior yet; this diff just adds
the plumbing to pass down an optional arbitrary schedule type string
from the CI system.
In the future, we'll use the schedule type to influence the behavior
of running tests.
Reviewed By: strager
Differential Revision: D15300120
fbshipit-source-id: 3b46afef2ff171b3fa095763dd5006a54ea328b8
