Docker Engine 29.9.1 fixes a docker save bug that left deleted layers on disk

I run docker save sometimes to move an image between two machines that don't share a registry. I never thought about what happens to the layers afterwards once I docker rmi the image. Docker Engine 29.9.1, released on 14 October, fixes a bug where the answer was: nothing. They stay on disk, forever, and no command can see them.

The fix is one line in the 29.9.1 release notes: "docker save no longer leaves layers on disk after image removal when an image repeats a layer." I pulled the moby pull request that made the fix (#53927) to see what "repeats a layer" meant, then built both the broken and fixed engine to check it myself.

The bug

The root cause is a reference count that never gets released. When docker save writes out an image's config and layers, it takes a reference on each layer so it isn't garbage collected mid-write. The code released that reference in a deferred call, but only after checking whether the layer's diff ID had already been saved once in the same run. If an image lists the same diff ID twice in its RootFS.Layers — two layers with byte-identical content — the second occurrence hit the "already saved" branch and returned before the deferred release ran. One layer walks away with a reference nobody ever drops.

That's a strange precondition — why would an image list the same layer twice? It happens if a Dockerfile copies the same file into the image at two different steps. The content is identical, so the diff tar it produces is identical, so the layer digest is identical. I built one on purpose:

FROM busybox:1.36.1
COPY f /f
COPY f /f

f was a few hundred bytes of text. docker inspect showed exactly what the bug needs:

["sha256:97e4ece8f8a4d01b66df2c55c1f1e8a2860597b54dffbbc0df2f9942062cb34d",
 "sha256:5a0a403383bddeef1a6e1d06106756361edc2cf35a15397ecdad4f2d10cc62d7",
 "sha256:5a0a403383bddeef1a6e1d06106756361edc2cf35a15397ecdad4f2d10cc62d7"]

The second and third layers are the same digest.

Reproducing it

I downloaded the static binaries for Docker Engine 29.8.2 (the version right before the fix) and 29.9.1 from download.docker.com, and ran each with its own --data-root so they couldn't interfere with each other. Both used the classic overlay2 graph driver, which the release notes specifically name as the affected store — I had to pass "features": {"containerd-snapshotter": false} in daemon.json to get that, since newer Docker defaults to the containerd image store.

First, a control run on 29.8.2 with no docker save involved at all:

$ docker rmi repeat-layer-test busybox:1.36.1
Untagged: repeat-layer-test:latest
Deleted: sha256:d4ec96e71ec1...
Deleted: sha256:93763116f792...
Deleted: sha256:7698c5712b92...
Deleted: sha256:35dc035c6e26...
Untagged: busybox:1.36.1
Deleted: sha256:b116e1550744...
Deleted: sha256:97e4ece8f8a4...

/var/lib/docker/image/overlay2/layerdb/sha256 went from 3 entries to 0. Clean removal, as expected.

Then the same build, this time with docker save in between:

$ docker save repeat-layer-test -o /tmp/repeat-layer-test.tar
$ docker rmi repeat-layer-test busybox:1.36.1
$ ls /var/lib/docker/image/overlay2/layerdb/sha256 | wc -l
3
$ du -sh /var/lib/docker/overlay2
4.5M    /var/lib/docker/overlay2

The tarball from docker save was 4.5MB. After removing both images, /var/lib/docker/overlay2 was still 4.5MB. Nothing was freed. The three layer directories just sat there, owned by nothing.

scenario (29.8.2, overlay2)	layerdb entries after rmi	disk left after rmi
no docker save	0	0
docker save then docker rmi	3	4.5MB

I ran the identical build, save, and remove sequence against 29.9.1 in its own data root:

$ docker save repeat-layer-test -o /tmp/repeat-layer-test-291.tar
$ docker rmi repeat-layer-test busybox:1.36.1
$ ls /var/lib/docker/image/overlay2/layerdb/sha256 | wc -l
0
$ du -sh /var/lib/docker/overlay2
8.0K    /var/lib/docker/overlay2

Zero entries, 8KB left, which is just empty directory structure. The fix works. The moby PR's own author noted they hadn't rerun the end-to-end reproduction against a patched daemon — as far as I can tell, this is that confirmation.

What it costs, and what doesn't show it

The part that bothered me more than the leak itself was that nothing surfaces it. On the broken 29.8.2 daemon, after the leak:

$ docker system df -v
Images space usage:

REPOSITORY   TAG   IMAGE ID   CREATED   SIZE   SHARED SIZE   UNIQUE SIZE   CONTAINERS

Containers space usage:
...
Build cache usage: 0B

Zero images, zero containers, zero build cache. The 4.5MB sitting in overlay2 doesn't appear in any column. I tried both prune commands docker ships:

$ docker system prune -af
Total reclaimed space: 0B
$ docker builder prune -af
Total:  0B

Neither sees it, because both work from the same image and layer records that no longer reference the orphaned layers. I restarted the daemon to rule out an in-memory cache masking the problem; the 4.5MB and the three directories were still there afterwards. On an affected engine version, there's no documented command that gets the space back. The only fix is the upgrade.

The bug is specific to one storage backend

The release notes say this only affects "the graph-driver image store," which on current Docker means you have to be on an upgraded host still running overlay2 rather than the containerd image store that's been the default for fresh installs since Docker Engine 29.0. I checked this rather than taking it on faith, running the same build-save-remove sequence on 29.8.2 with the containerd image store enabled instead:

$ docker system df -v
REPOSITORY          TAG       IMAGE ID       SIZE      SHARED SIZE   UNIQUE SIZE
repeat-layer-test    latest    1e9550303c9d   6.77MB    6.757MB       9.87kB
busybox              1.36.1    73aaf090f3d8   6.76MB    6.757MB       3.821kB

$ docker rmi repeat-layer-test busybox:1.36.1
$ docker system df -v
(nothing listed)
$ du -sh /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.overlayfs
56K

56KB of directory overhead, no leak. The claim held up: a fresh Docker install, or any host that's already switched to the containerd store, was never exposed to this regardless of engine version. It's specifically upgraded hosts that stayed on the legacy driver — which, going by Docker's own migration notes about upgraded hosts keeping overlay2 by default, is still a lot of production fleets.

The leak doesn't multiply with more repeats

My next guess was that an image with more repeated layers would leak proportionally more. I built a Dockerfile with the same file copied five times instead of two, saved it, and removed it on the broken engine:

$ docker save repeat5-test -o /tmp/repeat5.tar
$ du -h /tmp/repeat5.tar
4.5M    /tmp/repeat5.tar
$ docker rmi repeat5-test busybox:1.36.1
$ ls /var/lib/docker/image/overlay2/layerdb/sha256 | wc -l
6
$ du -sh /var/lib/docker/overlay2
4.5M    /var/lib/docker/overlay2

The tarball was the same 4.5MB as the two-copy version, and the disk usage after the leak was the same 4.5MB too. The layerdb entry count went from 3 to 6, because overlay2 tracks a distinct chain ID for each position in the layer stack even when the underlying content is identical, but the actual layer bytes are stored once and shared across those positions. The leak is bounded to the size of the unique data, not multiplied by how many times it's repeated. That's the one number here that argues against the headline: this bug is real and the fix matters, but it isn't the kind of leak that gets worse the more you push it. A one-off accidental duplicate costs you one layer's worth of space, not five.

What I got wrong on the way

My first attempt to run 29.8.2 with the classic graph driver just added --storage-driver overlay2 to the dockerd command line, which failed immediately:

failed to start daemon: configured driver "overlay2" not available: unavailable

I'd assumed the flag alone would turn off the containerd image store. It doesn't — Docker 29 defaults to containerd snapshotters regardless of --storage-driver, and I had to explicitly set "features": {"containerd-snapshotter": false} in a daemon.json passed via --config-file before the classic driver would start at all. Small thing, but it meant my first "reproduction" of the bug was actually running against the containerd store the whole time, and obviously showed no leak.

I also expected three RUN true instructions to produce three identical empty layers and hit the same bug through a more ordinary-looking Dockerfile. They didn't — the legacy builder recognised the empty diffs and didn't create layers for them at all, so docker inspect showed only the base image's single layer. That path doesn't reproduce this bug; you need an actual non-empty duplicate, such as a repeated COPY.

Run it yourself

This needs two Docker Engine builds. Static binaries are at download.docker.com; no package manager install is required.

# get both engine versions
mkdir -p /opt/docker-282 /opt/docker-291
curl -sL -o /tmp/d282.tgz https://download.docker.com/linux/static/stable/x86_64/docker-29.8.2.tgz
tar -xzf /tmp/d282.tgz -C /opt/docker-282 --strip-components=1

# force the classic graph driver (containerd snapshotter is the default otherwise)
echo '{"features": {"containerd-snapshotter": false}}' > /tmp/daemon.json
mkdir -p /data/docker-282
/opt/docker-282/dockerd --data-root /data/docker-282 \
  --host unix:///var/run/docker.sock \
  --config-file /tmp/daemon.json --storage-driver overlay2 &

# build an image with a genuinely repeated layer
mkdir /tmp/ctx && cd /tmp/ctx
echo hello > f
printf 'FROM busybox:1.36.1\nCOPY f /f\nCOPY f /f\n' > Dockerfile
DOCKER_BUILDKIT=0 /opt/docker-282/docker build -t repeat-test .

# save, then remove, then check what's left
/opt/docker-282/docker save repeat-test -o /tmp/repeat.tar
/opt/docker-282/docker rmi repeat-test busybox:1.36.1
ls /data/docker-282/image/overlay2/layerdb/sha256 | wc -l   # 3 on 29.8.2, 0 on 29.9.1
du -sh /data/docker-282/overlay2

Swap /opt/docker-282 for a 29.9.1 build with a separate --data-root to see the difference.

What to do about it

Run docker info | grep "Storage Driver" on whatever host actually runs your docker save step. Mine said overlayfs with driver-type: io.containerd.snapshotter.v1 once I enabled the newer store, and plain overlay2 on the legacy one — that one word is the whole difference between leaking 4.5MB per affected image and leaking nothing.

The hosts most likely to still show overlay2 are the ones nobody has touched the storage config on since before Docker Engine 29.0, because that's what upgrading in place leaves you with. If that's what you find, and a backup or image-migration script on that box calls docker save, the 4.5MB I measured here won't show up in docker system df, won't show up in docker system prune -af, and won't go away on its own. I only got rid of mine by installing 29.9.1. There isn't a second option, and I looked.