mobi/buildkit
1
0
Fork 0
mirror of https://github.com/moby/buildkit.git synced 2025-07-18 21:42:13 +03:00
Code Activity

cache: support nydus compression type

Browse Source 
Nydus image is a container accelerated image format provided by the
Dragonfly image-service project, which offers the ability to pull
image data on demand, without waiting for the entire image pull to
complete and then start the container. It has been put in production
usage and shown vast improvements over the old OCI image format in
terms of container launching speed, image space, and network bandwidth
efficiency, as well as data integrity. Nydus image can be flexibly
configured as a FUSE-based user-space filesystem or in-kernel
EROFS (from Linux kernel v5.16) with Nydus daemon in user-space,
integrating with VM-based container runtime like KataContainers
is much easier.

Nydus has provided a conversion tool Nydusify for converting OCIv1
image to Nydus image and integrated into Harbor Acceld as a conversion
driver, which assumes that the OCI image is already available in the
registry, but a better way would be to build the Nydus images directly
from the build system instead of using the conversion tool, which would
increase the speed of the image export, so we experimentally integrated
the Nydus export in Buildkit.

Unlike other compression formats (gzip, estargz, etc.) in OCI image,
nydus is divided into two types of layer, blob, and bootstrap, where
blob serves as the data part of each layer of the image, and bootstrap
serves as the metadata of the whole image, the bootstrap is equivalent
to the view of the whole image filesystem after all layers overlay. For
example, for an OCI image with 3 layers, the corresponding nydus image
is 4 layers (3 layers of blob + 1 layer of bootstrap).

The nydus-snapshotter project provides a package to do the actual layer
compression, this commit imports the package to implement the export of
nydus compression type.

Signed-off-by: Yan Song <imeoer@linux.alibaba.com>
This commit is contained in:
Yan Song
2022-10-26 09:26:59 +00:00
parent 357281b177
commit cf70ab6723
20 changed files with 493 additions and 23 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
cache/blobs_linux.go vendored
View File

@ -58,11 +58,14 @@ func (sr *immutableRef) tryComputeOverlayBlob(ctx context.Context, lower, upper
if err != nil {
return emptyDesc, false, errors.Wrap(err, "failed to get compressed stream")
}
err = overlay.WriteUpperdir(ctx, io.MultiWriter(compressed, dgstr.Hash()), upperdir, lower)
compressed.Close()
if err != nil {
// Close ensure compressorFunc does some finalization works.
defer compressed.Close()
if err := overlay.WriteUpperdir(ctx, io.MultiWriter(compressed, dgstr.Hash()), upperdir, lower); err != nil {
return emptyDesc, false, errors.Wrap(err, "failed to write compressed diff")
}
if err := compressed.Close(); err != nil {
return emptyDesc, false, errors.Wrap(err, "failed to close compressed diff writer")
}
if labels == nil {
labels = map[string]string{}
}