Summary:
Add Dockerfile build for openr.thrift python module.
The python module is built by:
1. Building and installing Facebook libraries with fbcode_builder
2. Building Open/R
3. Generating Cython files from thrift files with the FB thrift compiler
4. Generating C++ files from the Cython modules with the Cython compiler
5. Compiling the C++ modules into shared objects
Future work for building and distributing Breeze:
- Fix the hacks in build_breeze.sh, see comments therein
- Use a staged Dockerfile build for the Open/R and Breeze build
- Install openr.thrift. The openr.thrift shared objects are build and
stored in the Docker image generated by Dockerfile, but are unused.
- Install all the openr python submodules in a single openr site-package
- Add cross-compilation to the openr.thrift build. This is needed for
Terragraph
- Upload the openr python package to PyPi
Reviewed By: saifhhasan
Differential Revision: D28614443
fbshipit-source-id: 38b7e7c5594fd4bb5a338f19c69e5fc3b3b95863
Summary:
This should fix the Travis CI builds. It adds rust toolchain support inside docker and sets the required THRIFT env variable.
Pull Request resolved: https://github.com/facebookexperimental/rust-shed/pull/3
Reviewed By: krallin
Differential Revision: D18905608
Pulled By: lukaspiatkowski
fbshipit-source-id: 5db1eff6f215a6617d8acaa0c99a62d45225956b
Summary:
This is an update to the fbcode_builder codebase to allow setting up the python virtualenv with python dependencies installed. I've included wheel and cython (with a pinned version to 0.28.6 which is the only version that works with thriftpy3 at the moment, due to https://github.com/cython/cython/issues/2985) as standard packages since these are required by some of our top-level dependencies (folly and thrift)
As far as I know, there are no other projects that use PYTHON_VENV at the moment except LogDevice so the impact should be minimal.
Reviewed By: lucaspmelo
Differential Revision: D18758383
fbshipit-source-id: 264941311c5e3a19dc4ef2bb78c9a1baa34dfd8c
Summary:
Simply linking `GMOCK_LIBS` into a binary was not telling CMake that the binary depends on `gmock` being built. So, let's add that dependency explicitly.
This wasn't breaking in production because we typically build with `-j 4`, and `gmock` was getting built before the first dependent binary would attempt to link.
Also, since `rsocket` bundles its own `gmock`, it is just a waste of time to compile a system-level gmock. It's not a real dependency.
NB: The change in `fbcode_builder.py` is needed because now that `rsocket` no longer depends on anything on Github, driver programs that were unconditionally setting `projects_dir` started to fail to build `rsocket`.
Reviewed By: simpkins
Differential Revision: D16461572
fbshipit-source-id: 1e95654e96256e7ed37d42e702b5433bf2fe5328
Summary: We need this for dependencies that require submodules
Reviewed By: saifhhasan, GirasoleY
Differential Revision: D15282792
fbshipit-source-id: b0cc8d645e73668252409934fd6741fb211e30ae
Summary:
Needs to be enabled by option PYTHON_VENV in the config.
shell_builder.py sets up the venv and uses it once; calling activate
For docker we set ENV; resulting in the virtual environment being present
when the resulting container is run as well as at build time. This is also cleaner
and easier to follow than re-asserting on each RUN step.
For Lego builder we need to source activate on each command as environment
will not persist between commands.
While man on the posts say it makes no sense to use virtualenv within docker
container, this method simplifies the process considerably as we can rely on the
name pip being valid and we don't need to either ensure we are root or pass the
--user flag to pip and setuptools.
Pull Request resolved: https://github.com/facebookincubator/LogDevice/pull/76
Reviewed By: wez
Differential Revision: D14875633
Pulled By: calebmarchent
fbshipit-source-id: aabbcdd509d2a59fa36f8004032a052f014ce1ba
Summary: This makes it easier to debug and diagnose build problems
Reviewed By: simpkins
Differential Revision: D13831342
fbshipit-source-id: 3921a05715fb00264b2e1a6e134686d68aea804d
Summary:
I want to access the list of deps without the install
instructions attached later in this stack of diffs in `shell_builder.py`.
Refs: https://github.com/facebook/watchman/pull/639
Reviewed By: snarkmaster
Differential Revision: D9552419
fbshipit-source-id: 333cfa0c33bc1aa623de7ac3546a8f1785869a4f
Summary:
I noticed this while testing some changes; the comment
doesn't seem to match up to the `raise` statement, and attempting
to throw a list fails in python.
Reviewed By: strager
Differential Revision: D9566018
fbshipit-source-id: 563e43dfd30867993cc2a0f816db3b835a52e6df
Summary:
When running on macOS there is no /etc/issue. When running
with `set -e` enabled, we don't want this to break the build.
Neither of these is possible today, but will be later in this diff stack.
While I'm in here, also print out the cmake version.
Refs: https://github.com/facebook/watchman/pull/639
Reviewed By: snarkmaster
Differential Revision: D9552416
fbshipit-source-id: 432b5b70678908de9f79787d37c3615a7f8649d3
Summary:
folly, and therefore most of our other open source projects, no longer support
gcc-4.9. Drop Ubuntu 14.04 and Debian 8.6 from our CI platforms list, since
the these images only support gcc 4.9.
We should ideally add support for Ubuntu 18.04 soon, but I'll leave that for a
later diff. Removing the older platforms is somewhat higher priority for now
to get our Travis CI builds green again.
Reviewed By: snarkmaster
Differential Revision: D9258797
fbshipit-source-id: 3cab47a6c51b2dbe63214034240f844c85963c3d
Summary: Currently, fbcode_builder assumes you're building code living in the facebook github org. If you try to do travis builds using a facebookincubator repo, an error occurs due to hard-coded assumptions about the org. This fixes that
Reviewed By: knekritz
Differential Revision: D9183307
fbshipit-source-id: 0a1ac399953485e84f0534cc0616765227088c80
Summary:
Update `cmake_configure()`, `cmake_install()`, and `fb_github_cmake_install()`
to support specifying the directory where CMakeLists.txt is found, relative to
the directory where the build is being performed. Previously these functions
where hardcoded to assume that CMakeLists.txt was always found at '..'
Reviewed By: snarkmaster
Differential Revision: D7540689
fbshipit-source-id: efd3d044345fadc0346e436c01d0a247e1b6fd70
Summary: Debian 8.6 comes with a CMake version that is too old for folly. Add commands to update cmake.
Reviewed By: snarkmaster
Differential Revision: D6842797
fbshipit-source-id: 22f4c407f82a92c3fa90a5c2de334cf8b92e584b
Summary:
Some of our tests require sudo and the docker image doesn't have it by
default: https://travis-ci.org/facebook/openr/builds/306157670.
Reviewed By: saifhhasan
Differential Revision: D6406044
fbshipit-source-id: 6ec07fcc6b4eabce408d50f5656f9e391e4bd883
Summary: Removing git patch logic from fbcode builder and into the openr config. Making the patch available for travis build via curl.
Reviewed By: saifhhasan
Differential Revision: D6394924
fbshipit-source-id: 008f58bdf9d10fc730368bc23e1c49a43afc06e1
Summary: trying to make travis find the libnl patch. this should work
Reviewed By: saifhhasan
Differential Revision: D6262615
fbshipit-source-id: 431786ed20622b0b6d85df095f9a65a93a514f5c
Summary: travis ci machines do not have libnl installed, this installs it.
Reviewed By: saifhhasan
Differential Revision: D6026428
fbshipit-source-id: 70b00a854ab437b69e93ee6b53da83ba58b4d8a2
Summary: This updates fbcode_builder to build fbthrift with cmake instead of autoconf.
Reviewed By: yfeldblum
Differential Revision: D5896957
fbshipit-source-id: 00b7688195348934f113bd778e41870eaf971fcb
Summary:
Unfortunately, Travis is constantly in a capacity crunch. Plus, building `folly` + `wangle` + `proxygen` + `thrift` takes upwards of 40 minutes on Travis hosts, and the timeout for builds is 50 minutes. That creates several problems:
- Builds tend to get queued up, and sometimes take a long time to produce signal.
- Adding more platform, or project, or dependency coverage to our Travis builds would just make the problem worse (so I have not pushed much on the adoption of `fbcode_builder`).
- Building a service of any complexity will likely pull in all of the above dependencies, and then it gets 10 minutes for build & tests. So higher-up-the-foodchain projects like `bistro` just time out.
To help accelerate builds, Travis supports caching build artifacts between runs: https://docs.travis-ci.com/user/caching/#Arbitrary-directories
Each build variant gets its own cache, as one would want: https://docs.travis-ci.com/user/caching/#Caches-and-build-matrices
This is great for us because -- especially with all the `Updating submodules` commits -- we actually keep rebuilding nearly the same tree, most of the time.
Better yet, https://ccache.samba.org provides transparent compile caching -- just set your cache directory (and a few options), and use `ccache {gcc,g++}` instead of the vanilla compiler. It does the rest, and is quite robust -- the only issues I had were:
- different platforms have different versions, so one has to target ccache 3.1-ish,
- the max cache size started out too small, which tended to eliminate all speedup.
In this diff, I enable `ccache` for all projects using the default `.travis.yml` (jstrizich, you'll need to patch the custom `.travis.yml` files of `openr` and `fbzmq` if you want to try caching).
Notably, even if a project would time out on Travis without `ccache`, the caching accumulates across minor revisions, and should keep re-build times quite short for most changes.
This also significantly improves the experience of iterating on a Docker-based build on your own machine -- thoughtful use of Docker layer caching & ccache dramatically speeds up rebuilds after certain changes.
It works like this:
- Start with an empty `ccache.tgz`, or a pre-existing one provided by Travis from its cache.
- The `Dockerfile` extracts it into the Docker image under `/ccache`, and sets some environment variables to enable ccache for C & C++ builds.
- The build runs as usual.
- The build either finishes successfully, or it fails, or the new option of `docker_build_timeout` kills the container about 10 minutes before Travis would time out the build.
- A small script runs inside the container to prepare the updated `ccache.tgz`. This includes automatic cache sizing based on the working set size of a full successful build.
- Travis uploads `ccache.tgz` to S3 for the next build.
The rest is plumbing, CLI tweaks, and documentation to make manual iteration using `ccache` fairly pleasant.
The overhead of my `ccache` implementation is hard to measure exactly, but based on my indirect measurements, it's on the order of 2 minutes for `bistro`, which would otherwise take 52+ minutes to build on Travis.
Reviewed By: zpao
Differential Revision: D5648461
fbshipit-source-id: 207b621ba6bc6f827b20a5746b4dc4f28c1c96e2
Summary:
Previously, when building e.g. Proxygen, we would check out the master branch of dependencies like Folly. This could result in version shear, and the odds of it would go up as you iterated on your branch, while the dependencies moved forward. This degrades our developer experience.
The solution is to actually build from the same point in Facebook's monorepo.
As implemented, this is still imperfect, because the in-tree dependency hashes under `build/deps/github_hashes` are not currently updated atomically. However, on balance, this predictability should still be better -- and if I recall sdwilsh's explanation correctly, the atomicity of dependency commits could be fixed if it really impacts many people.
This will also improve cache coherence when you're using ccache for iteration, see D5648461.
(Also includes some extra "ideas" comments, and a whitespace fix.)
Reviewed By: sdwilsh
Differential Revision: D5648388
fbshipit-source-id: 1baa68ff3ef2e204a6cf5f5297e52514a3b62d21
Summary:
this diff provides ci builds for our two open source projects, openr and fbzmq, by leveraging snarkmaster 's awesome tool, fbcode builder. this will run our external cmake build process on lego-linux sandcastle boxes and also generates scripts for building on travis with docker externally.
this diff is modeled on D4441467. it also includes some changes to fbcode builder itself and some minor changes in our cmake files.
snarkmaster , please take a look at my fbcode builder changes. I needed to make some modifications in order to build some of our dependencies which have slightly non-standard build steps and to run tests. I split up the configure step, and I addressed an issue around workdir with cmake. the last workdir was just a relative path, `build`, so it was hard to get back to it in a future step to run tests.
Reviewed By: saifhhasan
Differential Revision: D5141184
fbshipit-source-id: 94cacab807a3a0da4d0d81016d7f36f37656145d
Summary: We just changed the ShipIt rules, which requires this manual commit. There were also a couple of files that got out-of-sync a long time ago. Fix that.
