containerd: images overridden by a build are kept dangling #49702

jsternberg · 2025-03-25T15:35:37Z

The build exporter now clears the image tags and always exported to a
dangling image. It then uses the image tagger to perform the tagging
which causes the dangling image to be removed and the naming message to
be sent correctly.

An additional progress message is sent to indicate the renaming.

Fixes the build portion of #48907.

containerd image store: Fix `docker build` not persisting overridden images as dangling.

vvoland · 2025-04-23T13:33:47Z

daemon/build.go

+	c8dexporter "github.com/docker/docker/builder/builder-next/exporter"
+	"github.com/moby/buildkit/exporter"
+	"github.com/moby/buildkit/exporter/containerimage/exptypes"


I think we should try to keep as much buildkit stuff inside the builder package and avoid any dependencies on the buildkit package inside daemon package.

vvoland · 2025-04-23T13:38:32Z

builder/builder-next/exporter/wrapper.go

+	var exported ExportedCallback
+	if i.callbacks.Exported != nil {
+		exported = i.callbacks.Exported(ctx, src, i.Attrs())
+	}
+


I'm thinking whether we could avoid leaking buildkit stuff into the daemon and having to extend the ImageService.

I think we could also make it work similar to how it works with pull:

moby/daemon/containerd/image_pull.go

Lines 96 to 112 in 0451e4f

if oldImage.Target.Digest != "" {

err = i.leaseContent(ctx, i.content, oldImage.Target)

if err != nil {

return errdefs.System(fmt.Errorf("failed to lease content: %w", err))

}

// If the pulled image is different than the old image, we will keep the old image as a dangling image.

defer func() {

if outNewImg != nil {

if outNewImg.Target().Digest != oldImage.Target.Digest {

if err := i.ensureDanglingImage(ctx, oldImage); err != nil {

log.G(ctx).WithError(err).Warn("failed to keep the previous image as dangling")

}

}

}

}()

}

Basically:

Before build, lease all content targetted by the image that can possibly be overriden (so it doesn't get GC'd)

Invoke build

Check if the images changed, if they did, tag dangling images and then remove leases

It looks like there aren't currently any parts of the buildkit builder that currently interact with the ImageService and the ensureDanglingImage method is only part of the containerd image service so I think I'll need to add a new method to that interface.

Hmm right. Looking at these callbacks in daemon basically call the image service again.

Perhaps it would make sense to move them out of the daemon to the image.
Then we could encapsulate the dangling image logic in the image service completely.

We could replace the ImageExportedByBuildkit with two callbacks like:

BuildInitiated - this would handle leasing the content

BuildFinished - this would handle removing the lease and tagging dangling images

WDYT?

I figured out a way to reuse the existing ImageTagger. The exporter now gets passed the image tagger. When the exporter runs, it clears the name so buildkit won't tag it by name. Instead, buildkit will always tag it as a dangling image and then we can perform the naming ourselves. TagImage already handles deleting the dangling image name when the first tag happens.

The only downside is this won't support keeping a name that is from the frontend. It's possible for buildkit frontends to set image.name and force the image to a specific name as a default. We aren't able to figure out that name until the frontend tells us anyway so there's no way to get that name without teaching the buildkit exporter how to do the dangling image dance.

This wouldn't have worked for the previous implementation and the leaseContent method also wouldn't handle this since it all happens within buildkit's direct interaction with the containerd image store. At the same time, I think this is a rare enough feature that we can just let it go. The default frontends for buildkit don't utilize this feature.

Here's some sample output:

=> => exporting layers 4.6s => => exporting manifest sha256:c010c84be19e970bdca89f47460f4279404d60ead05aa6806782e36cedf4096f 0.0s => => exporting config sha256:7aa7c843e893dee344f9aa5b3a572f3399c6daa9438ff1b1a65c1d7d6ff1c5de 0.0s => => exporting attestation manifest sha256:7fe4b993a97506d594ec18a46e673505b235fb2562fb6e2ada0ce5852a323e3c 0.0s => => exporting manifest list sha256:620031442269282d8570775a6a631bf36127ddd2a8856bebd9b72bf6906a9739 0.0s => => naming to moby-dangling@sha256:620031442269282d8570775a6a631bf36127ddd2a8856bebd9b72bf6906a9739 0.0s => => unpacking to moby-dangling@sha256:620031442269282d8570775a6a631bf36127ddd2a8856bebd9b72bf6906a9739 0.9s => => naming to docker.io/moby/buildkit:local

Still need to do some cleanup but I think this will keep things more maintainable until we can remove dangling images entirely.

vvoland

LGTM

vvoland · 2025-04-25T08:39:02Z

builder/builder-next/exporter/wrapper.go

-	exporterAttrs[string(exptypes.OptKeyName)] = strings.Join(reposAndTags, ",")
+
+	// Force the exporter to not use a name so it always creates a dangling image.
+	exporterAttrs[string(exptypes.OptKeyName)] = ""


Since name will always be empty now - can we make use of the OptKeyDanglingEmptyOnly now?

What do you mean? We can potentially remove it since name will always be empty but it also doesn't harm anything to keep it.

Yeah I meant not forcing OptKeyDanglingEmptyOnly = true (the default is false).

It doesn't harm, but if the buildkit default settings make sense, we should try to stick to them to make the wrapper as minimal as possible.

If it's ok, I can leave this for cleanup. CI is passing and I'd prefer not to trigger CI again.

CI

vvoland · 2025-04-25T08:41:42Z

Ah, the build test failure seems to be related:

       	            				/src/util/testutil/integration/run.go:245
        	Error:      	Not equal: 
        	            	expected: "moby-dangling@sha256:080d92f6b60c05117739c8e7f12a86b9766c014b22fcce9b07ceffc99f8a94fb"
        	            	actual  : "localhost:41079/buildkit/build/exporter:image,localhost:41079/buildkit/build/alternative:image"

The build exporter now clears the image tags and always exported to a dangling image. It then uses the image tagger to perform the tagging which causes the dangling image to be removed and the naming message to be sent correctly. An additional progress message is sent to indicate the renaming. Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>

jsternberg · 2025-04-25T15:06:42Z

I needed to fix up the response metadata. Since buildkit won't set the image name anymore, the image name showed the dangling instead of the proper name. We'll fix it up on our side.

The test also needs to be updated. It has expected and actual wrong so the error message is backwards. That's a buildkit fix though.

thaJeztah · 2025-04-30T12:51:00Z

Did some quick tests with this;

With containerd enabled

printf 'FROM alpine\nRUN echo foo > /hello' | docker build -t myimage -
# ...
=> exporting to image                                                                                                                                 0.1s
 => => exporting layers                                                                                                                                0.0s
 => => exporting manifest sha256:96389b8445f0637428ec374a3d3bedae2ddd578d72e6423e50e71b42562f5aff                                                      0.0s
 => => exporting config sha256:c05cbcf8619b9648264d7cf08a75b6cad1f470846418581e71682872ad9a3909                                                        0.0s
 => => exporting attestation manifest sha256:37f46c7ed95622f2dd5fc8edb5c735f1478bed243619d37e79d16748f7477ed2                                          0.0s
 => => exporting manifest list sha256:58b1c6a60d7b868b4e726414b9285f7cebfb4e2cc439ce74af8dc44db3e3b867                                                 0.0s
 => => naming to moby-dangling@sha256:58b1c6a60d7b868b4e726414b9285f7cebfb4e2cc439ce74af8dc44db3e3b867                                                 0.0s
 => => unpacking to moby-dangling@sha256:58b1c6a60d7b868b4e726414b9285f7cebfb4e2cc439ce74af8dc44db3e3b867                                              0.0s
 => => naming to docker.io/library/myimage:latest                                                                                                      0.0s


docker run --rm myimage@sha256:58b1c6a60d7b868b4e726414b9285f7cebfb4e2cc439ce74af8dc44db3e3b867 cat /hello
foo

Rebuild the image (causes the tag to be pointing to the new image);

printf 'FROM alpine\nRUN echo world > /hello' | docker build -t myimage -
# ...
=> exporting to image                                                                                                                                 0.1s
 => => exporting layers                                                                                                                                0.0s
 => => exporting manifest sha256:6a5da425bc6ecd14160167b145b4469806d18a2305b5f0f4e855518ba27ef6b7                                                      0.0s
 => => exporting config sha256:e887e3ffb647e15ac187568d490cc32e6eac68ecbdd5ffbd8ae9364b1f6126d4                                                        0.0s
 => => exporting attestation manifest sha256:99cccb8e07ee0a762d716b1e1fed1e72e0ef630996ba82c3a024244fe4e7fdae                                          0.0s
 => => exporting manifest list sha256:8edf361df74ae843a48a7ea1f0ab9c1965f6c7c2145cf7c4db13ea7776dbe248                                                 0.0s
 => => naming to moby-dangling@sha256:8edf361df74ae843a48a7ea1f0ab9c1965f6c7c2145cf7c4db13ea7776dbe248                                                 0.0s
 => => unpacking to moby-dangling@sha256:8edf361df74ae843a48a7ea1f0ab9c1965f6c7c2145cf7c4db13ea7776dbe248                                              0.0s
 => => naming to docker.io/library/myimage:latest                                                                                                      0.0s

docker run --rm myimage@sha256:8edf361df74ae843a48a7ea1f0ab9c1965f6c7c2145cf7c4db13ea7776dbe248 cat /hello
world

Try running the previous build;

docker run --rm 58b1c6a60d7b cat /hello
foo

docker run --rm 58b1c6a60d7b868b4e726414b9285f7cebfb4e2cc439ce74af8dc44db3e3b867 cat /hello
foo

Fails with image@digest notation (but ... to be looked at; more on that in the "non-containerd" case)

docker run --rm myimage@sha256:58b1c6a60d7b868b4e726414b9285f7cebfb4e2cc439ce74af8dc44db3e3b867 cat /hello
Unable to find image 'myimage@sha256:58b1c6a60d7b868b4e726414b9285f7cebfb4e2cc439ce74af8dc44db3e3b867' locally
docker: Error response from daemon: pull access denied for myimage, repository does not exist or may require 'docker login'

Run 'docker run --help' for more information

Without containerd enabled

printf 'FROM alpine\nRUN echo foo > /hello' | docker build -t myimage -
# ...
=> exporting to image                                                                                                                                 0.0s
 => => exporting layers                                                                                                                                0.0s
 => => writing image sha256:0255fea9a8e199d42523fc5a609f487184630e1e126c087b2f9691a83fc5ea73                                                           0.0s
 => => naming to docker.io/library/myimage                                                                                                             0.0s

docker run --rm myimage@sha256:0255fea9a8e199d42523fc5a609f487184630e1e126c087b2f9691a83fc5ea73 cat /hello
Unable to find image 'myimage@sha256:0255fea9a8e199d42523fc5a609f487184630e1e126c087b2f9691a83fc5ea73' locally
docker: Error response from daemon: pull access denied for myimage, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

Run 'docker run --help' for more information

☝️ I wonder if that's a regression; ISTR we allowed image@digest format for that in the past? But maybe I'm mistaken 🤔

Short, long, and (with algorithm) ID works though;

docker image ls
REPOSITORY   TAG       IMAGE ID       CREATED              SIZE
myimage      latest    0255fea9a8e1   About a minute ago   8.17MB

docker image ls --no-trunc
REPOSITORY   TAG       IMAGE ID                                                                  CREATED         SIZE
myimage      latest    sha256:0255fea9a8e199d42523fc5a609f487184630e1e126c087b2f9691a83fc5ea73   4 minutes ago   8.17MB

docker run --rm 0255fea9a8e1 cat /hello
foo

docker run --rm sha256:0255fea9a8e199d42523fc5a609f487184630e1e126c087b2f9691a83fc5ea73 cat /hello
foo

docker run --rm 0255fea9a8e199d42523fc5a609f487184630e1e126c087b2f9691a83fc5ea73 cat /hello
foo

Rebuild the image (causes the tag to be pointing to the new image);

printf 'FROM alpine\nRUN echo world > /hello' | docker build -t myimage -
# ...
=> exporting to image                                                                                                                                 0.0s
 => => exporting layers                                                                                                                                0.0s
 => => writing image sha256:3bbefa0e77d736edbd3f22badfd31a19dea16fe3ba60674b5eea2e720411f849                                                           0.0s
 => => naming to docker.io/library/myimage                                                                                                             0.0s


docker run --rm 3bbefa0e77d736edbd3f22badfd31a19dea16fe3ba60674b5eea2e720411f849 cat /hello
world

Try running the previous build;

docker run --rm 0255fea9a8e1 cat /hello
foo

docker run --rm 0255fea9a8e199d42523fc5a609f487184630e1e126c087b2f9691a83fc5ea73 cat /hello
foo

tianon · 2025-05-01T20:25:37Z

Without containerd enabled

...
☝️ I wonder if that's a regression; ISTR we allowed image@digest format for that in the past? But maybe I'm mistaken 🤔

image@digest with the manifest digest would possibly work, but the digest you're getting in the output there is the config digest, which wouldn't make sense via image@digest

One of the things we discussed on today's call is that docker run manifest-digest should be made to work on the graph drivers via RepoDigests (because we have enough data to make that work, in theory, thanks to docker run image@manifest-digest already working), and that docker run config-digest on containerd should also work on a best-effort basis for backwards compat (so new buildx can just always output manifest digest but old buildx will continue to work while outputting config digest).

vvoland · 2025-05-14T17:35:23Z

In this case it's because the dangling image is not attached to a repository anymore so repo@digest doesn't work where just digest does.

We could make it work though, but that's out of scope of this PR.

thaJeztah · 2025-05-30T12:16:55Z

This was (temporarily) reverted in Revert "containerd: images overridden by a build are kept dangling" #50105

jsternberg requested a review from tonistiigi as a code owner March 25, 2025 15:35

jsternberg force-pushed the containerd-dangling-build-image branch from f4c0462 to 14cba58 Compare March 25, 2025 15:56

thompson-shaun added this to the 28.2.0 milestone Apr 10, 2025

jsternberg added this to 🔦 Maintainer spotlight Apr 10, 2025

jsternberg moved this to Up next in 🔦 Maintainer spotlight Apr 10, 2025

jsternberg self-assigned this Apr 10, 2025

thompson-shaun moved this from Up next to Complete in 🔦 Maintainer spotlight Apr 10, 2025

thompson-shaun linked an issue Apr 18, 2025 that may be closed by this pull request

c8d: docker build shows "naming to moby-dangling@sha256...." when not tagging image #46116

Closed

vvoland requested changes Apr 23, 2025

View reviewed changes

vvoland added kind/bugfix PR's that fix bugs containerd-integration Issues and PRs related to containerd integration impact/changelog area/builder area/images area/builder/buildkit Issues affecting buildkit labels Apr 23, 2025

jsternberg force-pushed the containerd-dangling-build-image branch 2 times, most recently from d4e09b7 to 5e9a13c Compare April 24, 2025 20:06

jsternberg requested a review from vvoland April 24, 2025 20:06

vvoland previously approved these changes Apr 25, 2025

View reviewed changes

jsternberg force-pushed the containerd-dangling-build-image branch from 5e9a13c to 50a8561 Compare April 25, 2025 14:32

jsternberg mentioned this pull request Apr 25, 2025

client: reorder arguments for integration test moby/buildkit#5945

Merged

jsternberg requested a review from vvoland April 26, 2025 02:09

vvoland approved these changes Apr 28, 2025

View reviewed changes

vvoland requested review from dmcgowan and thaJeztah April 28, 2025 14:35

thompson-shaun assigned dmcgowan Apr 29, 2025

thompson-shaun assigned vvoland Apr 29, 2025

dmcgowan approved these changes Apr 30, 2025

View reviewed changes

vvoland merged commit 2e25c2b into moby:master May 14, 2025
159 checks passed

jsternberg deleted the containerd-dangling-build-image branch May 14, 2025 19:55

jsternberg mentioned this pull request May 20, 2025

c8d: buildx build, load, and import don't preserve dangling image #48907

Open

thaJeztah mentioned this pull request May 29, 2025

Revert "containerd: images overridden by a build are kept dangling" #50105

Merged

	if oldImage.Target.Digest != "" {
	err = i.leaseContent(ctx, i.content, oldImage.Target)
	if err != nil {
	return errdefs.System(fmt.Errorf("failed to lease content: %w", err))
	}

	// If the pulled image is different than the old image, we will keep the old image as a dangling image.
	defer func() {
	if outNewImg != nil {
	if outNewImg.Target().Digest != oldImage.Target.Digest {
	if err := i.ensureDanglingImage(ctx, oldImage); err != nil {
	log.G(ctx).WithError(err).Warn("failed to keep the previous image as dangling")
	}
	}
	}
	}()
	}

containerd: images overridden by a build are kept dangling #49702

containerd: images overridden by a build are kept dangling #49702

Uh oh!

Conversation

jsternberg commented Mar 25, 2025 • edited by vvoland Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vvoland left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vvoland commented Apr 25, 2025

Uh oh!

jsternberg commented Apr 25, 2025

Uh oh!

thaJeztah commented Apr 30, 2025

With containerd enabled

Without containerd enabled

Uh oh!

tianon commented May 1, 2025

Without containerd enabled

Uh oh!

vvoland commented May 14, 2025

Uh oh!

Uh oh!

thaJeztah commented May 30, 2025

Uh oh!

Uh oh!

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

jsternberg commented Mar 25, 2025 •

edited by vvoland

Loading