Summary:
This will be used to support the OSS build of CinderX with getdeps.py in upcoming diffs.
To make this work I had to teach getdeps.py how to pass `--external-branch` to `codesync`.
Reviewed By: alexmalyshev
Differential Revision: D79286425
fbshipit-source-id: f4b1066f8736fdb407f5bd75d3e2cc47b9e7d592
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/1227
Add retries to ArchiveFetcher when downloading fails. There will be 4 retries, with backoff and jitter. The max delay is capped at 10 seconds.
Reviewed By: srikrishnagopu
Differential Revision: D71167342
fbshipit-source-id: d927a639cf99185c5a04d063400bdab874dfddfe
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/1100
Update generated github actions to only run the fetch and and build steps when there are sources expected for a manifest
For local github actions testing using `act` this speeds up the test runs, and in real github CI it makes it clearer which steps are actually doing something on the given runner (we don't know exactly what it has installed beforehand)
Also set the windows git config the same as on internal CI
Reviewed By: bigfootjon
Differential Revision: D67839708
fbshipit-source-id: 0a60c6fc89e8c6abb2464f879459aa23d5aec969
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/857
Where one project should be checked out in a subdirectory of another
project. Like git submodules.
This is how the Glean build currently works: hsthrift is a separate
git repo, but Glean builds with hsthrift checked out in a
subdirectory.
Reviewed By: chadaustin
Differential Revision: D58055066
fbshipit-source-id: 1a22abaa8c5261c40b752d685a03d01625215b12
Summary:
X-link: https://github.com/facebookincubator/zstrong/pull/852
It was using the old PHP shipit script
Reviewed By: chadaustin
Differential Revision: D58013009
fbshipit-source-id: 97f0c31b897c04217ca5dd62fbe6cc2a55d280c8
Summary:
Neither the default "urlretrieve" nor "pycurl" works fine for all of FBOSS'
dependencies. One or other fails (opennsa, libmnl, exprtk, etc...) with these
two options. So, adding support for "wget".
Tried to use python's wget.download() but this also internally uses
"urlretrieve" and hence it fails for some of the FBOSS' dependencies.
So, using wget command directly instead.
Reviewed By: shri-khare
Differential Revision: D40841867
fbshipit-source-id: 229b3064f3faad4b32cf8a3c43fef53f2bb001fd
Summary:
X-link: https://github.com/facebookincubator/katran/pull/180
The urlretrieve method originally doesn't add this header, and in katran we noticed that the libmnl wasn't fetched anymore. After investigation, the request is blocked with error 403 because this header is missing. Adding it to the request.
Reviewed By: avasylev
Differential Revision: D40386108
fbshipit-source-id: a9b4b937ca12a04bc701d3f945681914151c0110
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/106
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/107
Summary
* Add EdenFS builds on external CI now EdenSCM is good
* Mac builds on github actions by using brew for system dependencies
To make this work had to fix some path ordering issues with install directories for Linux and Mac, and generalise the homebrew path fixups we were doing for bison to all the used homebrew packages.
Previously Installed packages were being added after system paths, so our own installed thing might be ignored. On github these meant system python 3.9 was being used for hg tests rather than our specified 3.8 (this showed we have some test fails on python 3.9 with "SystemError: deallocated bytearray object has exported buffers", that are beyond the scope of this diff to fix)
Also needed to include the getdeps generated python into the generated edenscmdeps3.zip archive setup.py produces otherwise EdenFS tests failed to import thrift.Thrift
Eden tests are hanging when run externally about half way through, so disable them on github actions for now as this PR is already fairly large. They work when run locally on an internal devserver, so probably some bit of environment necessary is not defined in the test runner
Reviewed By: chadaustin
Differential Revision: D34116505
fbshipit-source-id: d0d628db5daabc28d0bd8997cd5c1bc885ed1e73
Summary:
In order to speed up build times on a mac, add homebrew support to
getdeps.
Homebrew packages can be declared in a manifest using the `homebrew` header.
Futher, ahornby has added manifest entries for homebrew packages which are
included in this diff and also included a change to use the correct version of
openssl. Without this openssl change, homebrew cmake configure finds an old
openssl 1.0.2 install.
This diff provides a 2x speed up for building folly:
Timings for clean getdeps folly build on mid-2018 2.9Ghz i9 6 core intel macbook pro with 32GB RAM:
With new homebrew system deps:
```
rm -rf /Users/ahornby/.scratch/UsersZahornbyZfbsource/fbcode_builder_getdeps/
time ./opensource/fbcode_builder/getdeps.py build --allow-system-packages folly
real 17m39.329s
user 76m10.317s
sys 5m50.163s
```
Without:
```
rm -rf /Users/ahornby/.scratch/UsersZahornbyZfbsource/fbcode_builder_getdeps/
time ./opensource/fbcode_builder/getdeps.py build folly
real 32m10.344s
user 105m53.448s
sys 15m57.858s
```
Reviewed By: ahornby
Differential Revision: D33842632
fbshipit-source-id: ac785d4a8dcfa31b77292bddd9e747022ac36e3b
Summary: Add expressions for shipit entries so that we can map different directories if needed
Differential Revision: D33947996
fbshipit-source-id: e765769ae5a6ab90055b63284e538b965d6071b3
Summary:
If a getdeps project creates a bind mount in a subdirectory, we don't
want the shipit transformer to copy all of the bind mount's contents
to the ship-transformed output.
This manifested when I was using Vagrant inside of Watchman and
`getdeps.py fetch watchman` would copy all of the Vagrant VMs too.
Reviewed By: ahornby
Differential Revision: D33855553
fbshipit-source-id: e2bc50ecfe0d067490c95be05d074d6a844f30d5
Summary:
Now that we have some type annotations, we might as well support
running the type checker.
Reviewed By: xavierd
Differential Revision: D33715757
fbshipit-source-id: baf693e4b2415e0e1aa50b569b744ca0cfb91337
Summary: The getdeps fetch step runs early on system python, so it has to work with it. Subprocess capture_output was from 3.7 only, so let's switch to the alternate form that also works on ubuntu 18.04's python 3.6 install
Reviewed By: Croohand
Differential Revision: D32836861
fbshipit-source-id: bd492c0ee90edadbc9124d5820ddfe515fa46f43
Summary: Using system packages changes the binaries, so include the used packages in the cache key when present
Reviewed By: krallin
Differential Revision: D32693534
fbshipit-source-id: c0494667b57040dde680cea3e8e00d77ef48403b
Summary:
Most referenced repos have moved from master to main as default branch, so flip the default in getdeps.
As well as tidying things up, this means more manifests will be able to use the pinned git commit hash for a dependency that shipit generates (e.g. build/deps/github_hashes/facebook/folly-rev.txt). This is because a "rev = " setting in the manifest is treated as a dynamic override in GitFetcher.__init__ and this diff reduces the number of such overrides.
Reviewed By: StanislavGlebik
Differential Revision: D32567564
fbshipit-source-id: 3f1b734eb1bb0f5dbafb01ceb935a3d51b8dcc9a
Summary:
This applies the formatting changes from black v21.4b2 to all covered
projects in fbsource. Most changes are to single line docstrings, as black
will now remove leading and trailing whitespace to match PEP8. Any other
formatting changes are likely due to files that landed without formatting,
or files that previously triggered errors in black.
Any changes to code should be AST identical. Any test failures are likely
due to bad tests, or testing against the output of pyfmt.
Reviewed By: thatch
Differential Revision: D28204910
fbshipit-source-id: 804725bcd14f763e90c5ddff1d0418117c15809a
Summary:
This commit takes advantage of git 2.5.0 being able to fetch a
requested revision rather than relying on the desired revision being within the
depth limited fetch.
This relies on having git 2.5.0 on the server which is true for all
of the projects we have manifests for; this shows zero matches:
```
$ rg repo_url opensource/fbcode_builder/manifests | grep -v github
```
We've had a couple of situations recently where folks have run into issues with
the commit rate in folly being higher than then fetch depth, so this should
address that.
Refs: https://github.com/facebook/watchman/issues/866
Reviewed By: fanzeyi
Differential Revision: D24747992
fbshipit-source-id: e9b67c61dddc9f55e05d8984e8d210e7d2faabcb
Summary: This doesn't do anything anymore and is going to be removed in D23993306, let's remove it here first.
Reviewed By: yns88
Differential Revision: D23993954
fbshipit-source-id: 4d7dd5f992e34be7a0da16ce7cf59810407649c4
Summary:
We didn't do a great job of recognizing that we'd need to
build a project when one of its dependencies had changed: we relied
chiefly on the dependency hash for this and could fail to handle
changes in individual source files.
This commit helps to improve this situation by checking to see if
any installed files in the dependencies of a manifest are newer than
the most recent built time of a given manifest. If so, we'll trigger
a build. We try to be reasonably smart about deciding when to trigger
a cmake reconfigure if it looks like cmake files in the deps have
been changed.
Reviewed By: xavierd
Differential Revision: D21364132
fbshipit-source-id: 7534496e10d1f532aa9cf865900ace84a8785327
Summary:
Only run cmake reconfigure for .cmake, .cmake.in and CMakeLists.txt
files changes.
Previously we would reconfigure for any change to a file with a path that
matched `cmake` which could result in false positives in cases where
you may be iterating on .py or .c files in shared cmake directories.
This also reclassifies non-cmake files under fbcode_builder/CMake as source
files so that we run cmake for those; previously they would cause a
reconfigure and build, now they just cause a build.
Reviewed By: xavierd
Differential Revision: D21364133
fbshipit-source-id: a1231f657d6c6056b269656c677d3449d8715cf6
Summary:
Our linter really wants to include formatting changes unrelated
to my diff stack.
This is a formatting only change to avoid clouding my diffs; no functional
effect.
Reviewed By: xavierd
Differential Revision: D21364519
fbshipit-source-id: 7670dd4154e788f593f256aabdfdeef6d17aeec4
Summary:
From the outset, we wanted to be sure that getdeps was able
to source and build the dependencies so that we knew that we'd have
a repeatable build. This came at the cost of build times: having
to build boost on each CI run is a bit of a chore.
This commit adds three new elements to the manifest files:
* `rpms` - a list of RPM names that are all required to be present
in order to consider the dependency satisfied
* `debs` - like `rpms` above, but scoped to debian package names
* `preinstalled.env` - a list of environment variables that if they
are all set and non-empty will satisfy the dependency.
A new `--allow-system-packages` option to getdeps enables the new
logic that looks for system packages; it is off by default, but
enabled in the generated GitHub Actions workflows.
A new `install-system-deps` subcommand is provided that will attempt
to install the system packages needed to satisfy the build. This
typically needs to be run via sudo and is thus broken out separately
from the main getdeps build flow.
I made a pass over the manifest files and added package names that
satisfy the build on ubuntu-18 and fedora-31.
shri-khare: I renamed the `Python3.7.6` manifest to just `python` as
part of this change; the version of python that it pulls in through
the normal build is the same and I believe that an equal or newer
version of python3 is available in the GH actions builder.
The `preinstalled.env` is used only by the boost manifest: it references
the name of an environment variable that is set by the github
windows hosts and that points to a pre-built and pre-installed
copy of boost. Since there is no package manager that we can
easily query for this sort of thing, probing from the environment
seems like a reasonable and fast way to check for this. We
may need to evolve this over time to become more feature rich,
but this seems like a good starting point.
This commit has the potential to save 20 minutes of build time
from each public CI build just due to the boost dependency alone!
Refs: https://github.com/facebook/watchman/pull/797
Reviewed By: yfeldblum
Differential Revision: D20740410
fbshipit-source-id: 6c38019449c54465127656c3d18a6ff1f30adaea
Summary:
This diff extracts the fbsource commit hash and the date of that
commit and maintains that in place of just the commit hash that
we were previously extracting.
This data is exported into the environment that we pass on to
builders so that it is available if they choose to use it.
In a follow on diff I'll use this to default a version number
in the watchman project.
Reviewed By: fanzeyi
Differential Revision: D20949666
fbshipit-source-id: dc12bffe5f0efc4297b15ba0140c4c67a23ab0fd
Summary:
This avoids invalidating the entire build in response
to just running `hg amend`, which is frustrating and slow.
Reviewed By: chadaustin
Differential Revision: D18005409
fbshipit-source-id: ef93313859919298be78204046eb08bcadc5398e
Summary:
Add arguments to getdeps.py to allow overriding the source, build, and install
directories a per-project basis. The arguments take the form `[PROJECT:]PATH`
If the `PROJECT` portion is omitted, it defaults to the current project being
built.
In particular this makes it possible to specify `--src-dir .` to tell
getdeps.py to find the project sources from the current directory rather than
downloading them.
Reviewed By: wez
Differential Revision: D16778011
fbshipit-source-id: f33b87213ace04abb66334f588babdf59df91964
Summary:
This makes getdeps.py no longer crash if used in the folly repository.
Folly does not depend on any other Facebook projects, so it does not
include a `build/deps` directory.
Reviewed By: wez
Differential Revision: D16778009
fbshipit-source-id: 41be53d862f41b62154b081eb90ddba8742658fe
Summary:
This is towards getting open source FBOSS to build using fbcode_builder.
iproute2 is one of the dependencies for FBOSS. This patch adds a manifest file
to build the specific version of iproute2 needed for FBOSS.
Additionally, the default git clone depth of 100 is insufficient for the
version of iproute2 FBOSS depends on. Thus, this patch extends the git SCHEMA
to add optional argument depth. The default remains 100.
The usual /configure --prefix does not work for iproute2. Thus, we need to add
a custom builder that:
- copies sources to build directory, builds, and
- installs to installed directory using DEST_DIR.
- it must also explicitly copy include from build dir to install dir
Reviewed By: wez
Differential Revision: D15588809
fbshipit-source-id: ac5eab24134e078d88b85b4be433c78b05ef8ce5
Summary:
On Windows, the writing operation would fail with:
```
Traceback (most recent call last):
File ".\opensource\fbcode_builder\getdeps.py", line 400, in <module>
sys.exit(main())
File ".\opensource\fbcode_builder\getdeps.py", line 393, in main
return args.func(args)
File ".\opensource\fbcode_builder\getdeps.py", line 236, in run
change_status = fetcher.update()
File "C:\open\fbsource\fbcode\opensource\fbcode_builder\getdeps\fetcher.py", line 451, in update
return mapping.mirror(self.build_options.fbsource_dir, self.repo_dir)
File "C:\open\fbsource\fbcode\opensource\fbcode_builder\getdeps\fetcher.py", line 400, in mirror
f.write(name + "\n")
File "C:\Python36\lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 104-105: character maps to <undefined>
```
and this is caused by a file in libgit2: https://github.com/libgit2/libgit2/blob/master/tests/resources/status/%E8%BF%99, which is intended to test handling non-ASCII path.
Python on Windows will write file in cp1252 encoding by default, which does not contain that Chines character. (Caveat: that file on my system doesn't have the correct file name as well, it is being encoded in IBM861 for some reason. However the characters in IBM861 does not exist in CP1252 either)
Reviewed By: wez
Differential Revision: D15281521
fbshipit-source-id: 8a75e32bc1042167c945d67e26b549fda83b6b41
Summary:
The goal is to return an error code > 127 in the case of a
transient, retryable, infrastructure error. This diff generates
those in the case of failure in downloading a URL or from interacting
with LFS.
Reviewed By: strager
Differential Revision: D15266838
fbshipit-source-id: 4f52a791320123968869032c37912dded464a86e
Summary:
When running in FB infra, prefer to download from our local LFS
server rather than going out to the internet.
Fall back to a normal internet download if the LFS get fails for some reason.
Upload to LFS after successfully verifying the hash for the downloaded archive.
Add a subcommand that performs a fetch for all possible platforms so that it
is easier to ensure that the lfs-pointers file is up to date.
Reviewed By: simpkins
Differential Revision: D14978660
fbshipit-source-id: 240fc32fc7003d1e06c88b80d85054dae36e2f31
Summary:
previously, a relatively lame hash was computed to use
for the build directory based on some hash of the source directory.
That was good enough to get things off the ground, but in the
interest of being able to cache the build outputs and safely
invalidate them we need a slightly more rigorous implementation.
This diff computes a hash based on the manifest contents and
relevant environmental factors.
The hash is used to name the build directory which will ultimately
be cached eg: using the travis/appveyor cache directory configuration
and some other means for the FB internal CI.
The hash needs to be sufficient that we change the hash when
the manifests change. We can tolerate a false positive change
in hash (it just means that a build will take longer), but
cannot tolerate a false negative (which would result in an
incorrect build).
Reviewed By: simpkins
Differential Revision: D14710332
fbshipit-source-id: ebc2e74eafc6f3305d4412a82195bc9fb9dfa615
Summary:
This fetcher knows how to transform a 1st party project
from fbsource into approximately the same shape as ShipIt produces
for the github repo mirrors. It does this by reading shipit
mapping information from the manifest file for the project.
Since this fetcher uses data in the manifest and is implemented
directly in the getdeps codebase, it is suitable for iterating
on the opensource builds directly out of fbsource on both devservers
and laptops inside FB.
Reviewed By: simpkins
Differential Revision: D14691012
fbshipit-source-id: 05f68a7be64a2e465937b24b8825d25d3348ed13
Summary:
This fetcher knows how to take a 1st party FB project
from fbsource and transform it to the same shape as our github
repos using ShipIt. The transformation creates a transformed
mirror of the code in scratch space (rather than mutating fbsource
directly).
This can only be used in environments where shipit is available.
A later diff implements an alternative that works in more environments.
Reviewed By: simpkins
Differential Revision: D14691013
fbshipit-source-id: 539e307755c9fc0a098a235868ab622652061494
