3a91b50be1
Fix issue 5763 - Discourage `--oci-worker-no-process-sandbox`, due to the leakage of the processes (by design). Instead, encourage setting `systempaths=unconfined` in `docker run`. This corresponds to `securityContext.procMount: Unmasked` in Kubernetes, however, the configuration is hard on Kubernetes, as it has to be used in conjunction with `hostUsers: false`. - Remove `--device /dev/fuse`, as fuse-overlayfs is no longer used typically. - Use the new Kubernetes struct for AppArmor - Add a hint about `kernel.apparmor_restrict_unprivileged_userns` - Remove `$` from command snippets for ease of copypasting - Make `job.*.yaml` more practical - Add `*.userns.yaml`. Needs `UserNamespaceSupport` feature gate to be enabled. Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
Kubernetes manifests for BuildKit
This directory contains Kubernetes manifests for Pod, Deployment (with Service), StatefulSet, and Job.
Pod: good for quick-startDeployment+Service: good for random load balancing with registry-side cacheStateFulset: good for client-side load balancing, without registry-side cacheJob: good if you don't want to have daemon pods
Variants
*.privileged.yaml: Launches the Pod as the fully privileged root user.*.rootless.yaml: Launches the Pod as a non-root user, whose UID is 1000.*.userns.yaml: Launches the Pod as a non-root user. The UID is determined by kubelet. Needs kubelet and kube-apiserver to be reconfigured to enable theUserNamespacesSupportfeature gate.
It is recommended to use *.rootless.yaml to minimize the chance of container breakout attacks.
See also:
../../docs/rootless.md.- "Building Images Efficiently And Securely On Kubernetes With BuildKit" (KubeCon EU 2019).
Pod
kubectl apply -f pod.rootless.yaml
buildctl \
--addr kube-pod://buildkitd \
build --frontend dockerfile.v0 --local context=/path/to/dir --local dockerfile=/path/to/dir
If rootless mode doesn't work, try pod.privileged.yaml.
⚠️ kube-pod:// connection helper requires Kubernetes role that can access pods/exec resources. If pods/exec is not accessible, use Service instead (See below).
Deployment + Service
Setting up mTLS is highly recommended.
./create-certs.sh SAN [SAN...] can be used for creating certificates.
./create-certs.sh 127.0.0.1
The daemon certificates is created as Secret manifest named buildkit-daemon-certs.
kubectl apply -f .certs/buildkit-daemon-certs.yaml
Apply the Deployment and Service manifest:
kubectl apply -f deployment+service.rootless.yaml
kubectl scale --replicas=10 deployment/buildkitd
Run buildctl with TLS client certificates:
kubectl port-forward service/buildkitd 1234
buildctl \
--addr tcp://127.0.0.1:1234 \
--tlscacert .certs/client/ca.pem \
--tlscert .certs/client/cert.pem \
--tlskey .certs/client/key.pem \
build --frontend dockerfile.v0 --local context=/path/to/dir --local dockerfile=/path/to/dir
StatefulSet
StatefulSet is useful for consistent hash mode.
kubectl apply -f statefulset.rootless.yaml
kubectl scale --replicas=10 statefulset/buildkitd
buildctl \
--addr kube-pod://buildkitd-4 \
build --frontend dockerfile.v0 --local context=/path/to/dir --local dockerfile=/path/to/dir
See ./consistenthash for how to use consistent hashing.
Job
kubectl apply -f job.rootless.yaml
To push the image to the registry, you also need to mount ~/.docker/config.json
and set $DOCKER_CONFIG to /path/to/.docker directory.