Summary:
This is towards getting open source FBOSS to build using fbcode_builder.
iproute2 is one of the dependencies for FBOSS. This patch adds a manifest file
to build the specific version of iproute2 needed for FBOSS.
Additionally, the default git clone depth of 100 is insufficient for the
version of iproute2 FBOSS depends on. Thus, this patch extends the git SCHEMA
to add optional argument depth. The default remains 100.
The usual /configure --prefix does not work for iproute2. Thus, we need to add
a custom builder that:
- copies sources to build directory, builds, and
- installs to installed directory using DEST_DIR.
- it must also explicitly copy include from build dir to install dir
Reviewed By: wez
Differential Revision: D15588809
fbshipit-source-id: ac5eab24134e078d88b85b4be433c78b05ef8ce5
Summary:
On Windows, the writing operation would fail with:
```
Traceback (most recent call last):
File ".\opensource\fbcode_builder\getdeps.py", line 400, in <module>
sys.exit(main())
File ".\opensource\fbcode_builder\getdeps.py", line 393, in main
return args.func(args)
File ".\opensource\fbcode_builder\getdeps.py", line 236, in run
change_status = fetcher.update()
File "C:\open\fbsource\fbcode\opensource\fbcode_builder\getdeps\fetcher.py", line 451, in update
return mapping.mirror(self.build_options.fbsource_dir, self.repo_dir)
File "C:\open\fbsource\fbcode\opensource\fbcode_builder\getdeps\fetcher.py", line 400, in mirror
f.write(name + "\n")
File "C:\Python36\lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 104-105: character maps to <undefined>
```
and this is caused by a file in libgit2: https://github.com/libgit2/libgit2/blob/master/tests/resources/status/%E8%BF%99, which is intended to test handling non-ASCII path.
Python on Windows will write file in cp1252 encoding by default, which does not contain that Chines character. (Caveat: that file on my system doesn't have the correct file name as well, it is being encoded in IBM861 for some reason. However the characters in IBM861 does not exist in CP1252 either)
Reviewed By: wez
Differential Revision: D15281521
fbshipit-source-id: 8a75e32bc1042167c945d67e26b549fda83b6b41
Summary:
The goal is to return an error code > 127 in the case of a
transient, retryable, infrastructure error. This diff generates
those in the case of failure in downloading a URL or from interacting
with LFS.
Reviewed By: strager
Differential Revision: D15266838
fbshipit-source-id: 4f52a791320123968869032c37912dded464a86e
Summary:
When running in FB infra, prefer to download from our local LFS
server rather than going out to the internet.
Fall back to a normal internet download if the LFS get fails for some reason.
Upload to LFS after successfully verifying the hash for the downloaded archive.
Add a subcommand that performs a fetch for all possible platforms so that it
is easier to ensure that the lfs-pointers file is up to date.
Reviewed By: simpkins
Differential Revision: D14978660
fbshipit-source-id: 240fc32fc7003d1e06c88b80d85054dae36e2f31
Summary:
previously, a relatively lame hash was computed to use
for the build directory based on some hash of the source directory.
That was good enough to get things off the ground, but in the
interest of being able to cache the build outputs and safely
invalidate them we need a slightly more rigorous implementation.
This diff computes a hash based on the manifest contents and
relevant environmental factors.
The hash is used to name the build directory which will ultimately
be cached eg: using the travis/appveyor cache directory configuration
and some other means for the FB internal CI.
The hash needs to be sufficient that we change the hash when
the manifests change. We can tolerate a false positive change
in hash (it just means that a build will take longer), but
cannot tolerate a false negative (which would result in an
incorrect build).
Reviewed By: simpkins
Differential Revision: D14710332
fbshipit-source-id: ebc2e74eafc6f3305d4412a82195bc9fb9dfa615
Summary:
This fetcher knows how to transform a 1st party project
from fbsource into approximately the same shape as ShipIt produces
for the github repo mirrors. It does this by reading shipit
mapping information from the manifest file for the project.
Since this fetcher uses data in the manifest and is implemented
directly in the getdeps codebase, it is suitable for iterating
on the opensource builds directly out of fbsource on both devservers
and laptops inside FB.
Reviewed By: simpkins
Differential Revision: D14691012
fbshipit-source-id: 05f68a7be64a2e465937b24b8825d25d3348ed13
Summary:
This fetcher knows how to take a 1st party FB project
from fbsource and transform it to the same shape as our github
repos using ShipIt. The transformation creates a transformed
mirror of the code in scratch space (rather than mutating fbsource
directly).
This can only be used in environments where shipit is available.
A later diff implements an alternative that works in more environments.
Reviewed By: simpkins
Differential Revision: D14691013
fbshipit-source-id: 539e307755c9fc0a098a235868ab622652061494
Summary:
this fetcher knows how to download a URL that references
an archive and extract the sources from it. Compressed tarballs
and zipfiles are supported.
Reviewed By: simpkins
Differential Revision: D14690994
fbshipit-source-id: ffbbd1d9479b0d5aa3e5cd13fe11a560e9d01c2a
Summary: this fetcher knows how to check out sources via git.
Reviewed By: simpkins
Differential Revision: D14691000
fbshipit-source-id: 60f1ffbfed7b32a019aef6aa70ae0903f2782451
Summary:
Fetchers are used to fetch a source directory from a
source defined in a manifest file.
More details can be found in comments on the various methods.
The Manifest class offers a create_fetcher method for constructing
an appropriate fetcher. This is just a stub in this commit, but
will expand over the course of the next few diffs as concrete
fetcher instances are added.
Reviewed By: simpkins
Differential Revision: D14691001
fbshipit-source-id: 8c9038eacae3345e9403d5d1304bf979a9ee1555
