From 7463b5ca63b45e37cbd7e12519706cda380b08b6 Mon Sep 17 00:00:00 2001 From: Alexey Spiridonov Date: Wed, 23 Aug 2017 22:08:47 -0700 Subject: [PATCH] Easy: Improve ccache sizing Summary: When I was testing ccache in detail, I actually bootstrapped with a 1.8GB max cache size for Bistro. It looked like we were using only 1.2-1.3 GB uncompressed, so I figured that adaptively setting the maximum to be 30% more would be fine. My two mistakes were: - ccache uses decimal megabytes, so 30% became 25%. - ccache's LRU is kind of lossy so your max cache size has to be a LOT larger than the real thing to avoid inducing significant misses from cache cleanups. I should note that ccache still did speed up both the Bistro and the Proxygen builds somewhat, but cache miss rates seem too high, especially for builds that are not changing any headers. However, the current speedups are more like 20-30% rather than 3x. This is an attempt to decrease cache miss rates to 0 for no-source-change rebuilds. I'm not too worried about cache bloat, since (a) even 400MB tarballs should still work fine for our caches, (b) I have no evidence that ccache would actually ever get to its max. I also noticed & fixed that my `ccache.log` in the Docker container was empty due to permissions issues. Reviewed By: zpao Differential Revision: D5687199 fbshipit-source-id: a500c90f0c74797e3e4c8dc17de44977d86b1b70 --- build/fbcode_builder/docker_build_with_ccache.sh | 11 ++++++++--- build/fbcode_builder/docker_builder.py | 4 +++- 2 files changed, 11 insertions(+), 4 deletions(-) diff --git a/build/fbcode_builder/docker_build_with_ccache.sh b/build/fbcode_builder/docker_build_with_ccache.sh index f080e6c22..3f9826af2 100755 --- a/build/fbcode_builder/docker_build_with_ccache.sh +++ b/build/fbcode_builder/docker_build_with_ccache.sh @@ -155,6 +155,10 @@ echo "$img" # CAUTION: The inner bash runs without -uex, so code accordingly. docker run --user root --name "$container_name" "$img" /bin/bash -c ' build_exit_code='"$build_exit_code"' + + # Might be useful if debugging whether max cache size is too small? + grep " Cleaning up cache directory " /tmp/ccache.log + export CCACHE_DIR=/ccache ccache -s @@ -170,13 +174,14 @@ echo "$img" ) echo "$used_bytes" - # Goal: set the max cache to 30% over the usage of a successful build. - desired_mb=$(( $used_bytes / 806597 )) # 130% in MB: 1024*1024/1.3 + # Goal: set the max cache to twice the usage of a successful build. + # If this is too small, it takes too long to get to a fully warm cache. + desired_mb=$(( $used_bytes / 500000 )) # 200% in decimal MB: 1e6/2 if [[ "$build_exit_code" != "0" ]] ; then # For a bad build, disallow shrinking the max cache size. Instead of # the max cache size, we use on-disk size, which ccache keeps ~10% # under the actual max size, hence the 1.15 safety factor. - cur_max_mb=$(( $total_bytes / 911805 )) # 115% in MB: 1024*1024/1.15 + cur_max_mb=$(( $total_bytes / 869565 )) # 115% in decimal MB: 1e6/1.15 if [[ "$desired_mb" -le "$cur_max_mb" ]] ; then desired_mb="" fi diff --git a/build/fbcode_builder/docker_builder.py b/build/fbcode_builder/docker_builder.py index c053f99f1..7b97721df 100644 --- a/build/fbcode_builder/docker_builder.py +++ b/build/fbcode_builder/docker_builder.py @@ -162,6 +162,8 @@ class DockerFBCodeBuilder(FBCodeBuilder): # Record the current time to let travis_build.sh figure out # the number of bytes in the cache that are actually used -- # this is crucial for tuning the maximum cache size. - 'date +%s > /FBCODE_BUILDER_CCACHE_START_TIME' + 'date +%s > /FBCODE_BUILDER_CCACHE_START_TIME && ' + # The build running as `nobody` should be able to write here + 'chown nobody /tmp/ccache.log' )), ]