Layer Caching to Speed Up Rebuilds of an AI Docker Image

#ai #llm #machinelearning #programming

The symptom is familiar: change one line in a handler, rebuild, and watch pip reinstall a gigabyte of CUDA wheels. Nothing is broken. The builder is following a simple rule, and the rule can be predicted exactly once you know what it hashes.

What actually invalidates a layer

The builder walks your instructions in order, computing a cache key for each from the parent image’s key plus something about the instruction. The first instruction whose key does not match a cached result is a miss — and every instruction after it is unconditionally re-executed, because each depends on the filesystem the previous one produced. There is no partial reuse further down.

What goes into the key differs by instruction, and this is the part worth being precise about:

For RUN, the command string. Only the text. The builder does not execute the command to see whether the result would differ, which is why RUN apt-get update can serve a cached layer with a package index from six weeks ago, and why RUN pip install -r requirements.txt is a hit even when the index has newer versions.
For COPY and ADD, the contents. A checksum of every file being copied, plus its metadata. Change one byte in one copied file and the layer misses. This is the mechanism the ordering rule exploits.
For ENV, ARG and the rest, the literal instruction text. A build-arg whose value changes invalidates from the point it is used onwards, which is how a CACHEBUST argument works and how a version-stamping ARG placed too early accidentally destroys every cache in the file.

The ordering rule

Order instructions from least to most frequently changing. In practice that means one thing: copy the dependency manifest by itself, install from it, and copy the application source afterwards.

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY src/ ./src/

Now a source change misses only the last COPY. The install layer is keyed on the contents of requirements.txt, which did not change, so it is reused. The inverse — COPY . . followed by the install — keys the install on every file in the context, which is why it reinstalls when you edit a README.

Two refinements matter on an AI image specifically. Split the heavy, rarely-changing framework install from the light, frequently-churning application dependencies into two instructions with two files, so adding a small library does not re-resolve the framework. And pin versions: an unpinned requirements file makes the layer reproducible in cache terms but not in content terms, so the cached layer and a fresh build can contain different package versions with the same key. That is a correctness problem wearing a caching costume.

Cache mounts, which are a different mechanism

Layer caching reuses a whole layer or none of it. A BuildKit cache mount is orthogonal: it attaches a persistent directory to a single RUN, and that directory survives across builds without ever becoming part of any layer.

# syntax=docker/dockerfile:1
RUN --mount=type=cache,target=/root/.cache/pip \
    pip install -r requirements.txt

The difference in behaviour is the whole point. When the layer cache misses — because you added one package — the install runs again, but pip finds every previously downloaded wheel already in its cache directory and only fetches the new one. Without the mount, a miss means downloading everything from scratch, and on a dependency tree that includes CUDA wheels that is most of the build’s wall-clock time.

Two conditions. The syntax directive must be the first line of the Dockerfile, and BuildKit must be the builder — it has been the default since Docker Engine 23.0, and before that needed DOCKER_BUILDKIT=1. Note also that a cache mount and --no-cache-dir are contradictory instructions: the flag tells pip not to use the directory you just mounted. Use the mount for build speed and drop the flag; use the flag when you have no persistent builder and care about image size. Do not write both.

Why your CI has no cache at all

Everything above assumes the builder has seen a previous build. A fresh CI runner has an empty local cache, so a perfectly ordered Dockerfile rebuilds from nothing every time and the ordering work appears to have achieved nothing.

The fix is to export the cache somewhere shared. BuildKit supports cache exporters — --cache-to and --cache-from pointing at a registry, or the runner’s own cache backend — so a build on one machine can import what a build on another produced. The registry exporter is the portable option and it is worth knowing that mode=max exports intermediate layers as well as the final ones, which is what makes multi-stage builds cacheable across machines.

The important caveat: cache mounts are not exported by the registry cache exporter. They are local builder state. On ephemeral runners a cache mount does nothing at all, and the layer cache via --cache-from is the mechanism that helps. This is the single most common reason a Dockerfile that builds in forty seconds locally takes eleven minutes in CI.

Where the rule stops working

A file you did not mean to copy. If COPY . . pulls in .git, a log file or a __pycache__ directory, the layer misses on changes that have nothing to do with your code. A .dockerignore is a caching tool as much as a build context tool.
Metadata changes. COPY keys include file metadata, so a checkout that rewrites timestamps on every file can miss even with identical content. Some CI checkouts do this.
The base image moved. A mutable tag like python:3.12-slim resolves to a new digest when it is republished, and every layer beneath it misses. That is correct behaviour and it is also why reproducible builds pin by digest.
COPY --link changes the rules. Docker documents it as copying files onto an independent layer that is not invalidated when earlier commands change. It can preserve a cache hit across a base-image change, at the cost of the copy no longer seeing the filesystem beneath it — so it is not a drop-in for a COPY that overwrites existing paths.
You asked for a miss. --no-cache and --pull exist and are frequently left in a CI command from a debugging session months earlier. Check the build command before concluding the Dockerfile is at fault.