Summary:
When we detect that we are building out of the fbsource
monorepo internal to FB, pre-set some environment variables to help
projects detect our internally pinned versions of node and yarn.
Previously, the detection logic was deciding that the `yarn` shell
script was the executable to invoke, but since that is a bash script
it would fail to spawn.
Reviewed By: simpkins
Differential Revision: D18523871
fbshipit-source-id: b932d2b0ccd7b79d6f9cd74d363bc426c288e38f
Summary:
Our CI environment cannot directly connect to the internet,
and even if it could, doing so is undesirable to fetch javascript deps.
We maintain an offline mirror of packages used by the build(s) so that
we don't have to go out to the internet.
When running in fbsource, configure the environment to use that offline
mirror.
Reviewed By: chadaustin
Differential Revision: D18061773
fbshipit-source-id: 1a5e112f23c1baaedfb3dff0c4c2a1641f6bb9a1
Summary:
We were troubleshooting an issue with the eden tests on windows
where the boost dlls where not being found during gtest discovery.
When we compute the environment, we were only including INST/bin in the
PATH on windows. On Windows, the dlls are searched for in the PATH, and
since boost installs those into its `lib` dir we were missing those.
This diff causes `lib` dirs to get added to PATH on windows in the same
manner that we would add them to `LD_LIBRARY_PATH` on linux.
Reviewed By: pkaush
Differential Revision: D17694542
fbshipit-source-id: 143a907e6d30d8c12360caa43c8d9c26ff8c88c6
Summary:
This updates fbcode_builder to try and automatically detect the current
repository's project name by looking for a `.projectid` file in the repository
root. If the project name can be detected:
- The current repository will automatically be used as the source directory
for this project (instead of fetching the sources into the scratch
directory).
- If an explicit project name was not specified on the command line, the
current project will be built by default.
This also changes the repository detection logic to use the current working
directory, rather than the directory where the fbcode_builder code lives.
This will allow this logic to work even if we move fbcode_builder into its own
repository in the future, and have projects depend on it using git submodules.
This does mean that callers need to invoke fbcode_builder.py from inside the
repository.
Reviewed By: wez
Differential Revision: D17088938
fbshipit-source-id: f14d28fdcfaa330ff837ea52b8bdd4e358b81c61
Summary: Make getdeps to look for subdirectories for manifest files.
Reviewed By: simpkins
Differential Revision: D17222388
fbshipit-source-id: e13503beccd9edf6d80f78fbc3238b2a8d2053dd
Summary:
GitHub Actions CI `windows-latest` environment has only VS 2019
installed, so we need to expand our logic to be able to locate it.
Note that Boost 1.69 doesn't know how to locate VS 2019 so we are effectively
tied to VS 2017 at the moment; the search order in this diff reflects that.
(That means that we can't target `windows-latest` on GitHub Actions, but that
is really a concern for a later diff in this stack)
Reviewed By: simpkins
Differential Revision: D17385052
fbshipit-source-id: 9bb0612154f42d425a625406488f39bb4ec3d8ae
Summary:
Fix the BuildOptions class to correctly honor the `host_type` parameter that
it was constructed with when constructing the manifest evaluation context.
I accidentally broke this behavior in D16477396, and incorrectly had this code
path default to using the current host system rather than the value passed in
from the command line.
Reviewed By: wez
Differential Revision: D16779579
fbshipit-source-id: de911daaa643f6303fd35149775ab25d3f64d34f
Summary:
While developing on a project it is often convenient to be able to invoke its
build manually, rather than always needing to re-run `getdeps.py`. This
updates the CMakeBuilder to also emit a script that can be used to manually
run CMake outside of `getdeps.py`.
The CMakeBuilder is the only builder that this really matters for right now,
as pretty much all of the projects where we do first-party development use
CMake for their build system.
Reviewed By: pkaush
Differential Revision: D16477399
fbshipit-source-id: c8a14af158af7b32d6c799ef685b037e68b748ff
Summary:
Move code that computes project hashes to ManifestLoader. ManifestLoader is
the only class that has all of the information necessary to compute the
project hashes correctly. The ManifestLoader object can also cache previously
computed hashes, so that we don't have to keep computing hashes for projects
over and over again. Previously the `BuildOptions.compute_dirs()` function
would end up re-computing hashes for all dependencies each time it was called.
Reviewed By: strager
Differential Revision: D16477401
fbshipit-source-id: ce03642114f91ce4f859f612e6b2e747cf1653be
Summary:
The ManifestLoader contains all of the state needed to create a fetcher
object, so define a helper method on this object to create a fetcher.
Reviewed By: strager
Differential Revision: D16477395
fbshipit-source-id: 6de0942fe6b8de26c18c82bf99343f5467dc006a
Summary:
Update `BuildOpts.compute_dirs()` to use the correct project-specific manifest
context when computing project hashes. Previously it was incorrectly using
the initial project's context when evaluating all dependencies. This would
result in some projects potentially seeing the wrong values for variables that
may change from project to project (like `test`).
Reviewed By: pkaush
Differential Revision: D16477398
fbshipit-source-id: 6c23f5e5e19b2402000a138b3920b79044446041
Summary:
Add a ContextGenerator class so that we actually use the correct per-project
context when loading projects and computing dependencies.
Previously commands like `build` and `test` would change the contexts for each
project as they iterated through and performed the build. However, they did
not do this when first loading the projects. This could cause them to use
different context values when loading dependencies than when performing the
build. For instance, this could cause issues if a project depends on
`googletest` only when testing is enabled, as the code previously did not set
the "test" parameter when evaluating dependencies.
Reviewed By: pkaush
Differential Revision: D16477396
fbshipit-source-id: c1e055f07de1cb960861d19594e3bda20a2ccd87
Summary:
flake8 (in Python 2.7 mode) complains that `typing` is mentioned in type annotations but is not defined:
```
fbcode_builder/getdeps/buildopts.py:251:21: F821 undefined name 'typing'
subst_mapping, # type: typing.Mapping[str, str]
^
fbcode_builder/getdeps/buildopts.py:253:5: F821 undefined name 'typing'
# type: (...) -> typing.Optional[str]
^
2 F821 undefined name 'typing'
2
```
Import `typing` explicitly to silence this warning.
Because `typing` may be unavailable, import it conditionally. (Because it's only referenced in comments, failing to import `typing` should have no effect at run time.)
Reviewed By: snarkmaster
Differential Revision: D16435696
fbshipit-source-id: 78a4a7b07acc46aa998f02b54b1a6e52c1daafde
Summary:
The scratch path is used as part of the hash computation for each project. We
need to make sure this path is always normalized to ensure that we compute the
hashes consistently.
Reviewed By: chadaustin
Differential Revision: D16354624
fbshipit-source-id: 39b5362620bdc247cd7e7f1333dac319b354dc6f
Summary:
Mapping scratch dir <from> -> <to> show up in the output of all the show dir commands on Windows, so removing it.
example: getdeps.py show-inst-dir eden
Reviewed By: wez
Differential Revision: D16092494
fbshipit-source-id: 910288a8d23c1d68c5e70b7b2dbb36ef53a326fc
Summary: I want to use this logic outside of a builder implementation
Reviewed By: pkaush
Differential Revision: D16101914
fbshipit-source-id: db3c9ac6c84a92ab84a18dddb931953b0a51f127
Summary: On Windows "--vcvars-path" can be passed to point to the toolchain we want to use.
Reviewed By: wez
Differential Revision: D15926044
fbshipit-source-id: 2b0cde793f7c7f8473b78afde8794640bae351f3
Summary:
getdeps.py's find_existing_win32_subst_for_path function tries to reuse an existing build directory alias. (On Windows, the build directory is aliased to a drive letter to shorten paths.) If this function does not find and existing build directory alias, getdeps.py invalidates many of its caches.
On my Windows machine, find_existing_win32_subst_for_path always fails, so all of my builds are super slow. This happens because find_existing_win32_subst_for_path is given a path with a lower-case drive letter ("c:\users\..."), but the subst command returns paths with an upper-case drive letter ("C:\users\...").
When comparing paths, use ntpath.normpath. This makes the comparison case-insensitive for drive letters. (It also makes the comparison case-insensitive for other path components.)
On Linux and macOS, this diff should not change behavior.
Reviewed By: wez
Differential Revision: D15466096
fbshipit-source-id: 1669aa0a6eaeb6012154f3a9e24eba3f835262c6
Summary:
We were computing `SCRATCH/install` and hashing based on that value,
but the build stuff was later computing `SCRATCH/installed` and passing that
to the builders.
Fixup the mismatch.
Reviewed By: simpkins
Differential Revision: D15337969
fbshipit-source-id: 70288f2d9286aaacf4c1f6e0dac4680a55edac6d
Summary:
previously, a relatively lame hash was computed to use
for the build directory based on some hash of the source directory.
That was good enough to get things off the ground, but in the
interest of being able to cache the build outputs and safely
invalidate them we need a slightly more rigorous implementation.
This diff computes a hash based on the manifest contents and
relevant environmental factors.
The hash is used to name the build directory which will ultimately
be cached eg: using the travis/appveyor cache directory configuration
and some other means for the FB internal CI.
The hash needs to be sufficient that we change the hash when
the manifests change. We can tolerate a false positive change
in hash (it just means that a build will take longer), but
cannot tolerate a false negative (which would result in an
incorrect build).
Reviewed By: simpkins
Differential Revision: D14710332
fbshipit-source-id: ebc2e74eafc6f3305d4412a82195bc9fb9dfa615
Summary:
Adds a `test` subcommand that runs the tests for project.
We're mostly interested in the 1st party facebook projects for this.
The `sandcastle` flow will run the `test` subcommand just for the "leaf" project--the one named on the command line.
Reviewed By: simpkins
Differential Revision: D14710331
fbshipit-source-id: 7d04a46cfd723894d61018de2f230140b52285ac
Summary:
This fixes a TODO; in our CI environment we want to use the
real shipit, so we'll use this flag to make that happen.
Reviewed By: simpkins
Differential Revision: D14695576
fbshipit-source-id: 64ee72c210e2472d295dcbd39c86549273b68452
Summary:
The build options class contains some environmental and
build related information that will be passed down to fetcher
and builder objects that will be introduced in later diffs.
Reviewed By: simpkins
Differential Revision: D14691007
fbshipit-source-id: e8fe7322f590667ac28a5a3925a072056df0b3e3
