Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/897
Builder refactoring: instead of providing `install_dirs` to `build()`, `test()` etc., provide `loader` and `dep_manifests` when creating the builder. This is a cleaner API because we were computing `install_dirs` in multiple places before.
Furthermore this lets us do things that need to see the manifests of the dependencies, not just the list of `install_dirs`, such as treating direct dependencies differently from indirect dependencies (see D58244928).
Reviewed By: chadaustin
Differential Revision: D58200528
fbshipit-source-id: e52d35e84161b83ab49ab43099c3e3b9bb03f36e
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/857
Where one project should be checked out in a subdirectory of another
project. Like git submodules.
This is how the Glean build currently works: hsthrift is a separate
git repo, but Glean builds with hsthrift checked out in a
subdirectory.
Reviewed By: chadaustin
Differential Revision: D58055066
fbshipit-source-id: 1a22abaa8c5261c40b752d685a03d01625215b12
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/106
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/107
Summary
* Add EdenFS builds on external CI now EdenSCM is good
* Mac builds on github actions by using brew for system dependencies
To make this work had to fix some path ordering issues with install directories for Linux and Mac, and generalise the homebrew path fixups we were doing for bison to all the used homebrew packages.
Previously Installed packages were being added after system paths, so our own installed thing might be ignored. On github these meant system python 3.9 was being used for hg tests rather than our specified 3.8 (this showed we have some test fails on python 3.9 with "SystemError: deallocated bytearray object has exported buffers", that are beyond the scope of this diff to fix)
Also needed to include the getdeps generated python into the generated edenscmdeps3.zip archive setup.py produces otherwise EdenFS tests failed to import thrift.Thrift
Eden tests are hanging when run externally about half way through, so disable them on github actions for now as this PR is already fairly large. They work when run locally on an internal devserver, so probably some bit of environment necessary is not defined in the test runner
Reviewed By: chadaustin
Differential Revision: D34116505
fbshipit-source-id: d0d628db5daabc28d0bd8997cd5c1bc885ed1e73
Summary:
This diffs adds
* `--shared-libs` command line argument to the `build` command which enables building shared libraries for supported projects (cmake-based projects, boost, libevent)
* this flag overrides `BUILD_SHARED_LIBS` cmake flags in manifest files and from `--extra-cmake-defines`
* adds `shared_libs=on` expression support in manifest files
* for boost, the flag enables building shared libraries **in addition to** statically linked ones
Reviewed By: simonmar
Differential Revision: D27462289
fbshipit-source-id: d22ab434f7228c30472611bc323830d88efba0a5
Summary:
A number of places were extracting dependencies from manifests, but only one was adding in the implicit dependencies for build tools.
Extract the logic to one place and use so that a change in a tool like cmake will now correctly affect all tools using cmake, as it will be taken into account as a dependency hash when the manifest's hash is computed.
Tests for this change revealed that install_dirs needed to be populated in reverse order from the manifest topo-sort, so have also addressed that
Reviewed By: wittgenst
Differential Revision: D32730717
fbshipit-source-id: 1b2a25e460de6085d274c99acfd391b3bd259264
Summary:
This applies the formatting changes from black v21.4b2 to all covered
projects in fbsource. Most changes are to single line docstrings, as black
will now remove leading and trailing whitespace to match PEP8. Any other
formatting changes are likely due to files that landed without formatting,
or files that previously triggered errors in black.
Any changes to code should be AST identical. Any test failures are likely
due to bad tests, or testing against the output of pyfmt.
Reviewed By: thatch
Differential Revision: D28204910
fbshipit-source-id: 804725bcd14f763e90c5ddff1d0418117c15809a
Summary:
`extra_cmake_defines` are extra flags that are passed to cmake when
compiling each one of the dependencies. For instance:
```
$ ./opensource/fbcode_builder/getdeps.py build f4d --extra-cmake-define='{"CMAKE_CXX_FLAGS": "-mavx2 -mfma -mavx -mf16c -march=native"}'
```
It takes a json map as input, which can take a list of defines (key value
pairs).
Reviewed By: wez
Differential Revision: D25855781
fbshipit-source-id: 7f4fef2c66f4d12f23c8d7086d6a4f24fcc01ff7
Summary:
I was testing vs2019 vs vs2017 and realized that
we weren't reconfiguring when the toolchain was changed;
this resolves that.
Reviewed By: genevievehelsel
Differential Revision: D20795118
fbshipit-source-id: db80f090367cacfcc6b53887b77cf949f9cef0e6
Summary:
We have a global `--install-prefix` argument that can be used to set
the prefix for all projects, but that is only suitable if you are running with
sufficient privileges to install each of the deps to that location during the
build. Cmake dependency resolution won't work from the build directory in that
situation; it can only see the final installed location and it will error out
if those files are not present, or link against the currently installed version
instead of the version we just built; not great!
This commit adds a project specific `--project-install-prefix` that can be used
on just the leaf project in a set of deps. That sidesteps the dependency
concern because only the last stage is built in that mode. This option
can technically be applied to an arbitrary set of projects, but in light
of the above, in practice it only makes sense to use it for the final
cmake project. Only the CMakeBuilder respects this option.
In the watchman repo, this commit adjusts the autogen.sh script to allow
specifying the installation prefix; it defaults to `/usr/local` as you
might expect.
refs: https://github.com/facebook/watchman/issues/760
Reviewed By: yfeldblum
Differential Revision: D20674439
fbshipit-source-id: 52799dbd47f3c295e2d6469ee2b74cedeaa20138
Summary:
This diff allows passing a watchman version number override
via the environment as well as via the cmake `WATCHMAN_VERSION_OVERRIDE`
option.
To help invalidate the build I've added a new section to the manifest
files that allows listing out additional env vars that the project
hashes should be sensitive to. The effect of this is that we'll
re-run the cmake configure step if the listed env vars are changed.
Reviewed By: Ben0mega
Differential Revision: D17865896
fbshipit-source-id: 8ea5572b0b9b7af95ec5c310e494cb17a139ced4
Summary:
Throws an exception when:
* The name specified in the manifest mismatches the filename
* Duplicated manifest -- with the sub-directory support it is now able to have multiple manifest files with the same name
Reviewed By: chadaustin
Differential Revision: D17438460
fbshipit-source-id: ac7ad0b701beb15f0e91bb05cd1ec8fe378ad5b6
Summary: Make getdeps to look for subdirectories for manifest files.
Reviewed By: simpkins
Differential Revision: D17222388
fbshipit-source-id: e13503beccd9edf6d80f78fbc3238b2a8d2053dd
Summary:
Add a new builder that can extract Python wheel files, and re-package them
for consumption by our add_fb_python_library() and add_fb_python_executable()
CMake functions. This is useful for dependencies on packages from PyPI.
At the moment this code only handles architecture-independent pure-Python
packages. It shouldn't be too hard to extend this to handle more complex
wheels, but for now I only need to use it for some pure-Python wheels and so I
haven't tested with more complex wheel files.
This also includes two new manifests for python-six and python-toml that take
use this new builder.
Reviewed By: wez
Differential Revision: D17401216
fbshipit-source-id: d6f74565887c3f004e1c06503dc9ec81599dd697
Summary:
Add arguments to getdeps.py to allow overriding the source, build, and install
directories a per-project basis. The arguments take the form `[PROJECT:]PATH`
If the `PROJECT` portion is omitted, it defaults to the current project being
built.
In particular this makes it possible to specify `--src-dir .` to tell
getdeps.py to find the project sources from the current directory rather than
downloading them.
Reviewed By: wez
Differential Revision: D16778011
fbshipit-source-id: f33b87213ace04abb66334f588babdf59df91964
Summary:
In response to review feedback for D16477400 and D16477401, update
`ManifestLoader.load_all_manifests()` to only update its data for projects
that have not previously been loaded. This helps ensure that code using a
single `ManifestLoader` object cannot have two in-memory `Manifest` objects
for the same project, and that existing data (such as project hashes) can't be
invalidated if a manifest is later loaded from updated on-disk data.
Reviewed By: pkaush
Differential Revision: D16586682
fbshipit-source-id: 50b1979ec55f2ad6901629cd852293a8f6ca903f
Summary:
Move code that computes project hashes to ManifestLoader. ManifestLoader is
the only class that has all of the information necessary to compute the
project hashes correctly. The ManifestLoader object can also cache previously
computed hashes, so that we don't have to keep computing hashes for projects
over and over again. Previously the `BuildOptions.compute_dirs()` function
would end up re-computing hashes for all dependencies each time it was called.
Reviewed By: strager
Differential Revision: D16477401
fbshipit-source-id: ce03642114f91ce4f859f612e6b2e747cf1653be
Summary:
The ManifestLoader contains all of the state needed to create a fetcher
object, so define a helper method on this object to create a fetcher.
Reviewed By: strager
Differential Revision: D16477395
fbshipit-source-id: 6de0942fe6b8de26c18c82bf99343f5467dc006a
Summary:
Add a new ManifestLoader class to handle loading manifests and computing
dependencies.
For now the main thing this class does is maintain the `manifest_by_name`
mapping. In subsequent diffs we should be able to move some additional logic
into this class, which will help clean up the code and eliminate some redudant
work. In particular, we can have this class cache project hashes, which will
avoid re-computing hashes over and over again for the same projects as we do
in many cases today. We should also be able to save and re-use some of the
project dependency ordering computation in some cases as well.
Reviewed By: strager
Differential Revision: D16477400
fbshipit-source-id: f06f62f77d8443fccaa69fe4c1306e39c395b325
Summary:
Add a ContextGenerator class so that we actually use the correct per-project
context when loading projects and computing dependencies.
Previously commands like `build` and `test` would change the contexts for each
project as they iterated through and performed the build. However, they did
not do this when first loading the projects. This could cause them to use
different context values when loading dependencies than when performing the
build. For instance, this could cause issues if a project depends on
`googletest` only when testing is enabled, as the code previously did not set
the "test" parameter when evaluating dependencies.
Reviewed By: pkaush
Differential Revision: D16477396
fbshipit-source-id: c1e055f07de1cb960861d19594e3bda20a2ccd87
Summary:
The loader makes it possible to monkey patch the functions
that are responsible for loading manifests. It is intended to be use
in tests that are run in bucks sandboxed environment and that don't
have direct access to the manifest files on disk.
Reviewed By: simpkins
Differential Revision: D14781326
fbshipit-source-id: 18f69f8ce5768dc605b1a9388a80b7b7b9ffe0f4
Summary:
While the command isn't necessarily super useful
on its own, it does show that the plumbing for walking the
deps is functioning, and that is important when it comes
to building.
The output lists the projects in the order that they
would be built.
The `fetch` command has been augmented to add a `--recursive`
flag that uses the same mechanism to recursively fetch
the dependencies.
Reviewed By: simpkins
Differential Revision: D14691004
fbshipit-source-id: b00bf6ad4742f8bb0a70698f71a5fe03d6a1f453
Summary:
Adds a command that can be used to trigger a fetch for a project
```
$ ./opensource/fbcode_builder/getdeps.py fetch zstd
Cloning https://github.com/facebook/zstd.git...
---
+ git clone --depth=100 https://github.com/facebook/zstd.git /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/repos/github.com-facebook-zstd.git
Cloning into '/data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/repos/github.com-facebook-zstd.git'...
remote: Enumerating objects: 3816, done.
remote: Counting objects: 100% (3816/3816), done.
remote: Compressing objects: 100% (1415/1415), done.
remote: Total 3816 (delta 2556), reused 3312 (delta 2288), pack-reused 0
Receiving objects: 100% (3816/3816), 2.93 MiB | 9.59 MiB/s, done.
Resolving deltas: 100% (2556/2556), done.
Updating /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/repos/github.com-facebook-zstd.git -> v1.3.8
---
+ git -C /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/repos/github.com-facebook-zstd.git fetch origin v1.3.8
From https://github.com/facebook/zstd
* tag v1.3.8 -> FETCH_HEAD
---
+ git -C /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/repos/github.com-facebook-zstd.git checkout FETCH_HEAD
Note: checking out 'FETCH_HEAD'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b <new-branch-name>
HEAD is now at 470344d Merge pull request #1479 from facebook/visualTest
---
+ git -C /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/repos/github.com-facebook-zstd.git submodule update --init
```
Reviewed By: simpkins
Differential Revision: D14691008
fbshipit-source-id: 3afa391360518a08ebd6ff97f5b8b4993f10c4e8
