Docker Engine 29's default image store lets --storage-opt size fail silently

Docker Engine 29 switches new installs from the classic overlay2 graphdriver to the containerd snapshotter as the default image store. I wanted to know what actually changes when that flag flips, so I ran the same host, the same images, and the same flags against both backends and compared what came out.

The interesting result wasn't a speed number, although I got one of those too. It was a container disk quota that stopped being enforced and didn't tell me.

The setup

I have root on a fresh Docker 29.3.1 install (overlay2 root filesystem is plain ext4, no project quota mount option). I can flip the backend with a daemon flag and restart:

{ "features": { "containerd-snapshotter": false } }

docker info confirms which one is active:

Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1

versus

Storage Driver: overlay2

Finding 1: the quota flag stops erroring, and stops working

--storage-opt size=<limit> is supposed to cap a container's writable layer. It has a well-known precondition: it only works on overlay2 over XFS with the pquota mount option. My ext4 root doesn't have that. Under the classic graphdriver, Docker knows this and refuses:

$ docker run --rm --storage-opt size=100M alpine:3.20 echo ok
docker: Error response from daemon: --storage-opt is supported only for overlay over xfs with 'pquota' mount option

Exit code 125, every time, three runs in a row. The container never starts. That's a correct, loud failure on a host that can't honour the option.

Under the containerd snapshotter — the Docker 29 default — the exact same command and the exact same filesystem behave differently:

$ docker run --rm --storage-opt size=100M alpine:3.20 sh -c \
    "dd if=/dev/zero of=/bigfile bs=1M count=300; ls -lh /bigfile"
300+0 records in
300+0 records out
314572800 bytes (300.0MB) copied, 1.81 s, 165.5MB/s
-rw-r--r--  1 root root  300.0M Sep 13 05:06 /bigfile

Exit code 0. I asked for a 100MB limit and wrote 300MB with no resistance, no warning, no log line. I repeated it three times to be sure it wasn't a one-off race:

Backend	--storage-opt size=100M, write 300MB	Exit code
overlay2 (graphdriver)	refuses to start	125, ×3
containerd snapshotter	writes all 300MB	0, ×3

It gets worse. docker inspect still reports the option as if it applied:

$ docker inspect quotatest --format '{{.HostConfig.StorageOpt}}'
map[size:100M]

If you have anything that checks container config to confirm a quota is set — a compliance script, an audit tool, a human reading docker inspect output — it will tell you the limit is in place. Nothing enforces it. This is the kind of thing that looks fine in every dashboard until a runaway container fills the disk that three other containers are also writing to.

I don't think this is a deliberate design choice so much as an unimplemented check: the containerd image store path doesn't currently validate that the backing filesystem supports the quota before accepting the option, where the graphdriver path does. Whatever the cause, the behaviour changed under a default flip, and nothing in the CLI tells you.

Finding 2: pulls are genuinely faster, independent of why

I pulled node:22 (8 layers, 1.64GB unpacked) from a clean state, alternating backends, always removing the local image first:

Backend	Run 1	Run 2	Run 3	Average
containerd snapshotter	19.9s	17.7s	16.4s	18.0s
overlay2 (graphdriver)	28.9s	28.5s	28.1s	28.5s

That's roughly 37% faster on the new default, consistently, and the graphdriver runs happened later in my test sequence, so it isn't warm-cache bias in the snapshotter's favour — if anything a registry or proxy cache would have made the later runs faster, and they were the slow ones.

Finding 3: the concurrency knob barely matters, which wasn't what I expected

Docker 29's own release notes mention a bug, fixed in 29.7.0, where concurrent pull limits weren't honoured on the snapshotter path. I'm running 29.3.1, so I expected to see it: set max-concurrent-downloads to 1 and watch pulls slow down a lot more under the snapshotter than under the graphdriver.

That's not what happened. Setting the limit to 1 versus 8 moved the snapshotter average from 18.0s to 15.0s — about 17%. Under the graphdriver it moved from 28.5s to 27.5s — about 3.5%, close to noise:

Backend	max-concurrent-downloads=1	max-concurrent-downloads=8
containerd snapshotter	18.0s	15.0s
overlay2 (graphdriver)	28.5s	27.5s

Neither backend shows the dramatic serialization I was looking for. My conclusion, from these numbers, is that unpacking the layers into the filesystem dominates the wall-clock time for a pull far more than the number of concurrent HTTP downloads does, at least on this network path. The backend-versus-backend gap in finding 2 is real and reproducible; the concurrency-setting gap inside either backend is not where the time goes.

Finding 4: the new mount type does what it says, mostly

Docker 29 also graduates --mount type=image out of experimental status — in 29.7.0, according to the release notes. On 29.3.1 it's still flagged:

$ docker run --rm --mount type=image,source=alpine:3.20,target=/mnt \
    busybox:latest ls /mnt
WARNING: Image mount is an experimental feature
bin  dev  etc  home  lib  ...

It correctly refuses writes:

$ docker run --rm --mount type=image,source=alpine:3.20,target=/mnt \
    busybox:latest touch /mnt/testfile
touch: /mnt/testfile: Read-only file system

Exit 1, as it should be — this is the one place where the new feature enforced a restriction I expected and the quota flag didn't. It also fails cleanly on a bad source image (No such image, exit 125) and on a missing target (field Target must not be empty). What I couldn't get working on this build was a subpath option to mount only part of an image; every spelling I tried (subpath=, src.subpath=) came back as an unrecognised option. That may simply not exist yet in 29.3.1's experimental implementation — I'm reporting what this specific version accepts, not what later docs promise.

Finding 5: dedup accounting still works

One thing I checked because I assumed it might have regressed, and it hadn't: docker system df -v still correctly attributes shared base layers under the snapshotter. python:3.12-slim and python:3.13-slim share a Debian base:

REPOSITORY   TAG         SIZE     SHARED SIZE   UNIQUE SIZE
python       3.12-slim   190MB    87.45MB       102.9MB
python       3.13-slim   189MB    87.45MB       101.7MB

87.45MB shared, matching on both images, which is what you'd want to see if you're trying to estimate real disk cost from a fleet of related images. This is the one finding in this post that's a non-finding — I looked for a problem and didn't find one.

What I got wrong on the way

I set out to reproduce the "concurrent download limits not honoured" bug mentioned in the 29.7.0 changelog, expecting my max-concurrent-downloads=1 runs to be dramatically slower than =8 on the snapshotter. When the difference came out small on both backends, my first instinct was that my harness was broken — maybe the proxy in front of my registry was capping throughput regardless of client-side concurrency, hiding the effect. I re-ran with a completely fresh image (python:3.13-slim, untouched by earlier tests) to rule out any registry-side caching of my specific pull pattern, and got the same shape of result. The actual finding wasn't the one I went looking for: the backend swap (finding 2) explains far more of the wall-clock time than the concurrency setting does (finding 3). I'd rather report that than force a bug I couldn't actually reproduce on this version.

Run it yourself

# check which image store is active
docker info | grep -i "storage driver"

# flip to the classic graphdriver (needs root, restarts the daemon)
echo '{"features": {"containerd-snapshotter": false}}' | sudo tee /etc/docker/daemon.json
sudo pkill dockerd && sudo dockerd &

# the quota test — compare exit codes between backends
docker run --rm --storage-opt size=100M alpine:3.20 \
  sh -c "dd if=/dev/zero of=/bigfile bs=1M count=300; echo EXIT=\$?"

# flip back to the containerd snapshotter (Docker 29 default)
echo '{"features": {"containerd-snapshotter": true}}' | sudo tee /etc/docker/daemon.json
sudo pkill dockerd && sudo dockerd &

# pull timing, repeat 3x per backend, always from a clean state
docker rmi -f node:22
start=$(date +%s.%N); docker pull node:22 >/dev/null; end=$(date +%s.%N)
echo "$end - $start" | bc

# the read-only mount check
docker run --rm --mount type=image,source=alpine:3.20,target=/mnt \
  busybox:latest touch /mnt/testfile

If you're running Docker 29 on ext4 or any filesystem without project quota support, and you rely on --storage-opt size anywhere — a shared build host, a CI runner, a multi-tenant container platform — go check which image store you're actually running, and don't trust docker inspect to tell you whether the limit is real. Run the write test above against your own host. It takes ten seconds and it either fills up or it doesn't.