Summary:
This should fix the Travis CI builds. It adds rust toolchain support inside docker and sets the required THRIFT env variable.
Pull Request resolved: https://github.com/facebookexperimental/rust-shed/pull/3
Reviewed By: krallin
Differential Revision: D18905608
Pulled By: lukaspiatkowski
fbshipit-source-id: 5db1eff6f215a6617d8acaa0c99a62d45225956b
Summary:
This is an update to the fbcode_builder codebase to allow setting up the python virtualenv with python dependencies installed. I've included wheel and cython (with a pinned version to 0.28.6 which is the only version that works with thriftpy3 at the moment, due to https://github.com/cython/cython/issues/2985) as standard packages since these are required by some of our top-level dependencies (folly and thrift)
As far as I know, there are no other projects that use PYTHON_VENV at the moment except LogDevice so the impact should be minimal.
Reviewed By: lucaspmelo
Differential Revision: D18758383
fbshipit-source-id: 264941311c5e3a19dc4ef2bb78c9a1baa34dfd8c
Summary:
Needs to be enabled by option PYTHON_VENV in the config.
shell_builder.py sets up the venv and uses it once; calling activate
For docker we set ENV; resulting in the virtual environment being present
when the resulting container is run as well as at build time. This is also cleaner
and easier to follow than re-asserting on each RUN step.
For Lego builder we need to source activate on each command as environment
will not persist between commands.
While man on the posts say it makes no sense to use virtualenv within docker
container, this method simplifies the process considerably as we can rely on the
name pip being valid and we don't need to either ensure we are root or pass the
--user flag to pip and setuptools.
Pull Request resolved: https://github.com/facebookincubator/LogDevice/pull/76
Reviewed By: wez
Differential Revision: D14875633
Pulled By: calebmarchent
fbshipit-source-id: aabbcdd509d2a59fa36f8004032a052f014ce1ba
Summary:
openr would like to run our tests within the docker container, but
certain netlink tests require root privilege. Hence this change.
Reviewed By: saifhhasan
Differential Revision: D6415295
fbshipit-source-id: 51608401067df5cc54fa566d8d7e0aec987a5ee1
Summary:
When I was testing ccache in detail, I actually bootstrapped with a 1.8GB max cache size for Bistro. It looked like we were using only 1.2-1.3 GB uncompressed, so I figured that adaptively setting the maximum to be 30% more would be fine.
My two mistakes were:
- ccache uses decimal megabytes, so 30% became 25%.
- ccache's LRU is kind of lossy so your max cache size has to be a LOT larger than the real thing to avoid inducing significant misses from cache cleanups.
I should note that ccache still did speed up both the Bistro and the Proxygen builds somewhat, but cache miss rates seem too high, especially for builds that are not changing any headers. However, the current speedups are more like 20-30% rather than 3x.
This is an attempt to decrease cache miss rates to 0 for no-source-change rebuilds.
I'm not too worried about cache bloat, since (a) even 400MB tarballs should still work fine for our caches, (b) I have no evidence that ccache would actually ever get to its max.
I also noticed & fixed that my `ccache.log` in the Docker container was empty due to permissions issues.
Reviewed By: zpao
Differential Revision: D5687199
fbshipit-source-id: a500c90f0c74797e3e4c8dc17de44977d86b1b70
Summary:
Unfortunately, Travis is constantly in a capacity crunch. Plus, building `folly` + `wangle` + `proxygen` + `thrift` takes upwards of 40 minutes on Travis hosts, and the timeout for builds is 50 minutes. That creates several problems:
- Builds tend to get queued up, and sometimes take a long time to produce signal.
- Adding more platform, or project, or dependency coverage to our Travis builds would just make the problem worse (so I have not pushed much on the adoption of `fbcode_builder`).
- Building a service of any complexity will likely pull in all of the above dependencies, and then it gets 10 minutes for build & tests. So higher-up-the-foodchain projects like `bistro` just time out.
To help accelerate builds, Travis supports caching build artifacts between runs: https://docs.travis-ci.com/user/caching/#Arbitrary-directories
Each build variant gets its own cache, as one would want: https://docs.travis-ci.com/user/caching/#Caches-and-build-matrices
This is great for us because -- especially with all the `Updating submodules` commits -- we actually keep rebuilding nearly the same tree, most of the time.
Better yet, https://ccache.samba.org provides transparent compile caching -- just set your cache directory (and a few options), and use `ccache {gcc,g++}` instead of the vanilla compiler. It does the rest, and is quite robust -- the only issues I had were:
- different platforms have different versions, so one has to target ccache 3.1-ish,
- the max cache size started out too small, which tended to eliminate all speedup.
In this diff, I enable `ccache` for all projects using the default `.travis.yml` (jstrizich, you'll need to patch the custom `.travis.yml` files of `openr` and `fbzmq` if you want to try caching).
Notably, even if a project would time out on Travis without `ccache`, the caching accumulates across minor revisions, and should keep re-build times quite short for most changes.
This also significantly improves the experience of iterating on a Docker-based build on your own machine -- thoughtful use of Docker layer caching & ccache dramatically speeds up rebuilds after certain changes.
It works like this:
- Start with an empty `ccache.tgz`, or a pre-existing one provided by Travis from its cache.
- The `Dockerfile` extracts it into the Docker image under `/ccache`, and sets some environment variables to enable ccache for C & C++ builds.
- The build runs as usual.
- The build either finishes successfully, or it fails, or the new option of `docker_build_timeout` kills the container about 10 minutes before Travis would time out the build.
- A small script runs inside the container to prepare the updated `ccache.tgz`. This includes automatic cache sizing based on the working set size of a full successful build.
- Travis uploads `ccache.tgz` to S3 for the next build.
The rest is plumbing, CLI tweaks, and documentation to make manual iteration using `ccache` fairly pleasant.
The overhead of my `ccache` implementation is hard to measure exactly, but based on my indirect measurements, it's on the order of 2 minutes for `bistro`, which would otherwise take 52+ minutes to build on Travis.
Reviewed By: zpao
Differential Revision: D5648461
fbshipit-source-id: 207b621ba6bc6f827b20a5746b4dc4f28c1c96e2
Summary: We just changed the ShipIt rules, which requires this manual commit. There were also a couple of files that got out-of-sync a long time ago. Fix that.
