Containerd 2.4's switch to klauspost's gzip decoder speeds up real image layers by 18%

Containerd 2.4.0 shipped a one-line-sounding change: layer decompression now uses klauspost/compress/gzip instead of the standard library's compress/gzip. The pull request claimed 1.4 to 1.5 times faster decompression, measured on a 3.4GiB CUDA image and a 7GiB PyTorch image, on arm64. I run neither of those images, and my container host is an Intel Xeon, not an Apple Silicon or Graviton box, so I wanted to know what this actually buys an ordinary python:3.12 pull on ordinary x86_64 hardware.

It buys about 18%, not 40-50%. The allocation story, on the other hand, held up completely.

What actually changed

The change is PR #13560 against containerd/containerd, merged 30 June 2026 and released in containerd 2.4.0 (16 September 2026). It touches exactly one function, gzipDecompress in pkg/archive/compression/compression.go, and only the decoder. Compression still goes through the standard library. The PR description is explicit that this "affects only gzip layer decompression" — nothing about zstd layers changes, because containerd was already using klauspost/compress for those.

My environment runs this code today. Docker Engine here bundles containerd v2.4.2, and strings /usr/bin/containerd | grep klauspost/compress/gzip finds the vendored package compiled into the binary I actually have running.

Getting real layers to test against

Benchmarking synthetic data would have missed the point — gzip's throughput depends heavily on what you feed it, and a container image layer is mostly binaries, shared libraries, and Python wheels, not random bytes or highly repetitive text. I pulled python:3.12 with docker pull, which populates containerd's own content-addressable store, then read the real compressed layer blobs straight out of that store:

ctr -n moby content get sha256:<layer-digest> > layer.tar.gz

That gave me the amd64 manifest's seven layers, 392.8MiB compressed, 1081.3MiB decompressed once I ran them through gzip -t to confirm they were intact. The smallest is a 249-byte metadata layer; the largest is 237MB compressed, 670MB decompressed.

I wrote a small Go program that decompresses each blob with compress/gzip and with github.com/klauspost/compress/gzip in turn, three times each, keeping the fastest run, and reading runtime.MemStats around the final run to get allocation counts.

The headline number, and the one that argues against it

	stdlib compress/gzip	klauspost/compress/gzip
Total time (7 layers, 1081.3MiB)	5.85s	4.98s
Throughput	184.8 MiB/s	217.3 MiB/s
Speedup	—	1.18x

Per layer, the speedup ranged from 1.12x to 1.23x, with the 249-byte layer too small to measure meaningfully either way. That's a real, repeatable improvement — I ran the full suite twice on separate invocations and got 1.18x and 1.19x — but it is well under half of what the PR's own numbers suggested.

I don't think the PR's authors were wrong; I think the gap is the workload. Their test images were dominated by large, highly compressible binary blobs typical of CUDA and PyTorch wheels, run on Apple Silicon or Graviton cores with different memory and cache characteristics than my Xeon. A python:3.12 base image is a more ordinary mix of shared libraries, .pyc files, and documentation, and on this specific box the win is real but modest. If your images look more like a typical web service than a GPU training image, expect something closer to my number than theirs.

The allocation claim held up exactly

The PR's second claim was about memory pressure, not speed, and this one matched almost precisely on the largest layer in my set:

	stdlib	klauspost
Mallocs (237MB compressed layer)	425,133	1,246
Bytes allocated	66.6 MiB	0.15 MiB

That's 341 times fewer allocations and about 444 times less memory churned through the allocator, for decompressing the same bytes into the same output. The PR quoted similar ratios on its own test images (1.91 million mallocs down to 1,948, 305MiB down to 0.18MiB), and my numbers land in the same range even though the raw layer sizes don't match. This is the part of the change that's architecture- and workload-independent: fewer, larger buffer allocations instead of one small allocation per inflate call, regardless of what's inside the layer.

Under concurrency, the gap didn't widen the way I expected

Containerd unpacks the layers of a pulled image in parallel, so I decompressed all seven layers from four goroutines worth of CPU at once, three times, keeping the best run:

	stdlib	klauspost
Wall time, 7 layers concurrently	3.485s	3.007s
Speedup	—	1.16x

I expected concurrency to widen klauspost's advantage, on the theory that stdlib's heavier allocation rate would cause more garbage-collector pressure under parallel load, and that pressure would compound. It didn't: 1.16x concurrent against 1.18x serial, within noise of each other. On four cores, both versions are simply CPU-bound by the cost of running DEFLATE, and the allocator wasn't the bottleneck at this scale. The allocation win is real, but on this machine it didn't translate into an extra concurrency dividend.

A claim from the PR discussion I couldn't reproduce

During review, a maintainer reported that a fuzzing job slowed from roughly 1.59 million executions per second to about 26,000 after this change, and the author attributed it to klauspost's reader allocating its buffers eagerly on construction, even when the fuzz harness never reads from it. I tested exactly that: calling NewReader on a real 237MB layer and immediately closing it, 2,000 times, with no read in between.

stdlib     total=25ms per-call=12.719µs
klauspost  total=19ms per-call=9.598µs

Klauspost's reader was not slower to construct — it was marginally faster, on both runs I did. I think the fuzzer's reported 60x slowdown is specific to the fuzz harness's own execution pattern (likely many small or malformed inputs, triggering different code paths than a NewReader call against a valid, cached file), not something that shows up on a plain construction-only benchmark against real data. I can't confirm the fuzzer's number was wrong; I can only confirm I didn't see anything like it under a more ordinary access pattern, and anyone deciding whether this matters for their own fuzzing setup should test their own harness rather than extrapolate from either of our numbers.

What it refuses, and a surprise about when it refuses

I corrupted a copy of the largest layer two ways: truncated to 50 bytes, truncated to 1MB, and with a single byte flipped at offset 5,000,000 in the full 237MB file. Both decoders gave identical errors on the same inputs:

=== 50 bytes ===
stdlib:    err=unexpected EOF
klauspost: err=unexpected EOF

=== 1MB of 237MB ===
stdlib:    bytes=3757688 err=unexpected EOF
klauspost: bytes=3757688 err=unexpected EOF

=== full file, one byte flipped ===
stdlib:    bytes=669852201 err=gzip: invalid checksum
klauspost: bytes=669852201 err=gzip: invalid checksum

The last one is the interesting case. Flipping a single byte deep inside a 237MB compressed stream didn't produce an error at the point of corruption. Both decoders happily decompressed all 670MB of output — garbage from the flip point onward, but no error — and only failed at the very end, when the gzip trailer's CRC32 checksum didn't match what had actually come out. Neither library changed this behaviour; it's inherent to the gzip format, which checks integrity once, after the fact, rather than per block. If you're pulling a large layer over a flaky connection, this is the shape of failure to expect: you pay for decompressing the entire thing before being told it was corrupt, independent of which decoder unpacked it.

What I got wrong on the way

My working assumption going in was that klauspost's allocation efficiency would show its biggest advantage exactly where allocation-heavy code usually suffers most: under concurrent load, where garbage collection has to compete with multiple goroutines doing real work. I built the concurrent test expecting to report a bigger number than the serial one. It came back smaller — 1.16x against 1.18x, a difference inside the run-to-run noise I was seeing anyway. Four cores running seven goroutines' worth of DEFLATE is simply CPU-bound on the arithmetic, and reducing allocator traffic that wasn't the bottleneck doesn't buy anything extra. I'd have reported this wrong if I'd gone in assuming the theory was obviously true and skipped actually running the concurrent case.

Run it yourself

This needs Go 1.24+ and either Docker with a running daemon, or direct access to a containerd content store.

docker pull python:3.12
SOCK=/var/run/docker/containerd/containerd.sock
mkdir layers
# get the amd64 manifest digest from the image index, then for each layer:
ctr -a $SOCK -n moby content get sha256:<layer-digest> > layers/<digest>.tar.gz

mkdir gzipbench && cd gzipbench
go mod init gzipbench
go get github.com/klauspost/compress@v1.18.0

package main

import (
    "fmt"
    "io"
    "os"
    "path/filepath"
    "time"

    stdgzip "compress/gzip"
    kgzip "github.com/klauspost/compress/gzip"
)

func decompress(path string, open func(io.Reader) (io.ReadCloser, error)) (int64, time.Duration) {
    f, _ := os.Open(path)
    defer f.Close()
    start := time.Now()
    r, err := open(f)
    if err != nil {
        panic(err)
    }
    defer r.Close()
    n, _ := io.Copy(io.Discard, r)
    return n, time.Since(start)
}

func main() {
    files, _ := filepath.Glob(filepath.Join(os.Args[1], "*.tar.gz"))
    for _, f := range files {
        _, std := decompress(f, func(r io.Reader) (io.ReadCloser, error) { return stdgzip.NewReader(r) })
        _, kp := decompress(f, func(r io.Reader) (io.ReadCloser, error) { return kgzip.NewReader(r) })
        fmt.Printf("%-20s std=%s klauspost=%s speedup=%.2fx\n",
            filepath.Base(f), std, kp, float64(std)/float64(kp))
    }
}

go run . ../layers

I ran every command above as part of this post, on the same host and image as the numbers reported.

What to do with this

If you're already on containerd 2.4.0 or later — check with ctr --version, or docker version --format '{{.Server.Components}}' if Docker bundles it — this change is already active and there is no flag to turn it on or off; the maintainers declined to add one. Don't expect it to move your docker pull wall-clock time by much on an ordinary network connection, where transfer time usually dwarfs decompression time; it matters most when you're pulling from a fast local registry mirror or unpacking already-cached layers, where decompression is a bigger share of the total. And if you maintain anything that fuzzes gzip decoding against containerd's dependencies, check your own executions-per-second before assuming either my number or the one from the PR discussion applies to your harness.