When a container comes up, most of the time goes into one job: downloading the image. The application itself starts in milliseconds, but gigabytes of layers have to be pulled and unpacked first. On a workload that scales, this decides the answer to "when does the new pod take traffic".
The annoying part is that most of the downloaded data is never read. There's a study behind that intuition, and it's the starting point for the lazy-pulling projects: as the SOCI snapshotter's documentation relays, Harter et al's 2016 FAST '16 work found that on a representative suite of images, image download accounts for 76% of container startup time, while on average only 6.4% of the fetched data is actually needed for the container to start doing useful work. Carry those numbers into 2026 with care: the images of that era aren't today's multi-gigabyte model images, and networks got faster. The direction holds; the magnitude depends on your image.
Lazy pulling builds on that observation: start the container without downloading the whole image, and fetch files in the background as they're read. The idea isn't new, but today it has several mature implementations: eStargz, zstd:chunked, SOCI and nydus. All of them work with standard registries; this article looks at the first three.
What I want to convey here is as much about which guarantee you keep in which stack as about the mechanism itself. Because "lazy pulling" isn't one thing: two implementations of the same idea come with two different security positions, and the difference hides in the name of a configuration option that's easy to skim past.
What happens on my own server
First, the ordinary case:
$ docker info | grep -E "Server Version|Storage Driver"
Server Version: 29.4.3
Storage Driver: overlayfs
Right now this machine pulls images the classic way: all layers come down, get unpacked, and then the container starts. But the overlayfs detail in that output matters — that's not the old overlay2 graph driver, it's the containerd snapshotter. The machine is already running on the containerd image store.
That difference retires a widely repeated claim: "Docker can't do lazy pulling" is no longer true. The stargz snapshotter's install documentation has a dedicated section for Docker (Moby); after a change that landed in Docker v24, writing this into daemon.json and running the snapshotter service is enough:
{
"features": { "containerd-snapshotter": true },
"storage-driver": "stargz"
}
So the side doing lazy pulling is still the snapshotter beneath the runtime; this isn't an image format change, it's a runtime configuration job. It's just that you can now do that configuration in Docker too.
eStargz: prioritised files and a TOC
What the stargz snapshotter does is fetch the necessary chunks of the image on demand: the container runs while the download continues in the background.
The format that makes this possible is eStargz. In the project's own words it's a lazily-pullable image format, and two of its properties matter. First, compatibility: it's compatible with OCI/Docker images, so it can be pushed to standard registries and still runs on eStargz-agnostic runtimes, Docker included. Second, prioritisation: the snapshotter prefetches and caches the files likely to be accessed while the container runs.
Its third property — the most important one for this article — is content verification. In the eStargz documentation's words, the goal is to ensure the downloaded metadata and the contents of all files are the expected ones, based on the calculated digests. Since the layer is pulled lazily, verification happens piece by piece: on mounting a layer the snapshotter fetches the TOC from the registry, recalculates its digest and compares it against the TOC digest in the verified manifest, then re-verifies each fetched chunk against the digest in its entry. So the equation "lazy pulling means no verification" is wrong.
The requirements are documented too: containerd newer than v1.4.2, a stargz snapshotter daemon on the nodes, configuration changes on the containerd side, and a tool like BuildKit v0.10+ to convert the image. On CRI-O and Podman the stargz snapshotter can't be used directly; the documentation points to a separate additional layer store (Stargz Store) for those.
zstd:chunked: skippable frames and range requests
The second approach comes from the Podman/containers-storage side, and its mechanism approaches the problem differently.
The classic layer format is tar+gzip; the standard was later augmented with tar+zstd. zstd:chunked is a variant of that zstd layer: it uses zstd's skippable frames to embed extra metadata inside the layer. The core of that metadata is a table of contents — the SHA-256 of each file plus the offsets of its chunks. On top of that, chunks are compressed separately.
The result: a client can dynamically fetch only the content it doesn't already have, using HTTP range requests. It doesn't have to download the whole layer.
One implementation detail matters here: each layer keeps metadata called chunked-manifest-cache, and when a new image is pulled the other layers on the system are scanned through it. If a file with a matching digest is found it isn't downloaded again — depending on configuration it's shared via a hard link or a reflink, and if the filesystem doesn't support that, a full physical copy is made. Where configured, there's also a best-effort attempt to enable fs-verity on the file.
So zstd:chunked doesn't just mean "download less", it means "don't download what you already have". Given how many libraries different images share, the saving moves from layer granularity down to file granularity.
SOCI: lazy loading without conversion
There's a third approach whose design choice differs from the others. AWS's SOCI ("Seekable OCI") snapshotter aims to lazily load standard OCI images without requiring a build-time conversion step.
The rationale is operational: existing lazy-loading snapshotters rely on a conversion step that produces a new image artifact, which is a problem for teams that can't change their CI/CD pipeline or don't want to keep the same image in two formats. And — critically for security — the conversion step invalidates signatures created against the original OCI image.
SOCI's answer is to leave the image alone and place a separate index artifact (the "SOCI index") next to it in the registry, queried at container launch through the OCI reference types mechanism.
A note: the project's own documentation says this applies to the first version, that adding or removing an index on a widely deployed image can cause performance changes in production scenarios, and that the second version went back to a build-time conversion step. So "lazy loading without conversion" produced its own price in practice.
The price: which guarantee, in which stack?
Now the main point — and here you must separate two stacks, because the same words describe different things.
On the containerd + stargz snapshotter path, "off" isn't a security setting but an installation state: without the snapshotter installed and configured, there's no lazy pulling. Content verification, as described above, is built into the format.
On the Podman/containers-storage path, it's a configuration matter. The enable_partial_images option enables partial pulls, and its default in the code is false:
{&res.enablePartialImages, "enable_partial_images", false},
A little documentation archaeology is needed here: the project's zstd:chunked design document says that "at the time of this writing, support for this is enabled by default in the code", and the document is dated August 2024. That sentence was true then; the default was switched on in February 2024 and switched back off in November 2024. So the documents don't contradict each other — one is stale. On your own system, check the behaviour rather than the commented-out line in storage.conf.
The genuinely interesting constraint lives in the option whose name says it all: insecure_allow_unpredictable_image_contents. The documentation's first sentence is that it "should almost never be set". What it does is allow partial pulls without guaranteeing that partial and non-partial pulls produce the same image contents. The side effect that rarely gets quoted is more concrete: with it enabled, image IDs become unpredictable — usually not equal to the traditional value matching the config digest.
So when is the flag needed? The source code answers clearly: zstd:chunked layers without tar-split data fall back to the ordinary download path because consistency with non-partial pulls can't be guaranteed — unless the flag is set. eStargz layers likewise use the traditional path on the Podman side without it. Today's tar-split-carrying zstd:chunked images are pulled partially with the guarantee intact; the flag is for the older format and for eStargz.
My takeaway: what's sacrificed here isn't content verification but predictability — the guarantee that partial and full pulls produce the same storage representation and the same image ID. It looks like a performance setting, but the consequences land on the supply chain side.
If you handle image security with a CI gate like the one in my article on container image scanning with Trivy, you have to ask which identity that gate scanned: is the image ID you scanned the one you run? With signatures the question is sharper still, because format conversion invalidates the signature.
On the Podman side there's an official way out of this dilemma too: the convert_images option converts images to zstd:chunked while pulling. The documentation notes it's off by default because it's an expensive operation — but you get to choose between "forcing the old format with a flag" and "converting locally".
Where does it actually pay off?
The size of the gain depends on a single ratio: how much of the image is really read? The smaller that ratio, the bigger the difference lazy pulling makes.
The places where the ratio is typically small are predictable: huge machine-learning images (CUDA layers, model weights), application images carrying a full language runtime, "just in case" debug images stuffed with tools. By contrast, a carefully built 20 MB distroless image holding a Go binary has little to gain — everything downloaded is read anyway.
On the scenario side, three stand out: workloads that scale to zero (cold start latency lands directly on the user), CI runners (the same enormous image on every job), and Kubernetes clusters with many nodes (the same image pulled hundreds of times).
When you do the maths, remember: lazy pulling doesn't remove the download, it spreads it over time. When the application takes its first request, missing files may still be arriving; on a slow network the latency of those first seconds comes not from the download but from on-demand reads. Measure to the moment "it served the first request at normal latency", not "the container started".
Growth on disk is a separate matter; I wrote about how layers behave on write in the OverlayFS copy_up article. Lazy pulling changes the read side; the write-side mechanics stay the same.
Checklist before you try it
- Identify your stack: containerd (including Docker v24+ in
containerd-snapshottermode) with a stargz snapshotter is one path, Podman/containers-storage another. On CRI-O and Podman you need a separate additional layer store (Stargz Store) rather than the stargz snapshotter. -
enable_partial_imagesis off by default today; verify that from behaviour, not from the documents. - Before switching on
insecure_allow_unpredictable_image_contents, answer this: does every image that could ever reach this machine come from a build system you trust not to attack image integrity? - Move image conversion into CI; an eStargz image built with BuildKit v0.10+ also runs on classic runtimes, so the migration can be gradual.
- Confirm your registry supports HTTP range requests; all three approaches rest on them.
- Watch the snapshotter cache: in an on-demand read setup, cache or daemon problems can surface as file read errors while the container still looks healthy. Test your readiness probe with a real file read.
- On the storage side, check
use_hard_linksand reflink support; that's where file-level sharing pays.
Pointing instead of copying
Lazy pulling reminded me what container images actually are: copying a filesystem snapshot over the network. The copy model is simple and durable, but as you scale, the assumption "move everything" gets expensive.
The alternative model is to point rather than copy: share a content-addressable table and fetch a chunk when it's needed. The same idea already lives in git, in registries, and in content-addressable storage; lazy pulling carries it into the runtime.
So the question for your own setup: how much of the image you pull most often actually gets executed? If the answer is "less than half", these mechanisms solve your problem — as long as you hand over the guarantee knowingly.
Top comments (0)