Docker BuildKit Cache Setup That Actually Speeds Up CI

#docker #devops

Originally published on kuryzhev.cloud

Last month a client asked me why their "cached" Docker builds still took nine minutes on every single pull request. They had --cache-from in their GitHub Actions workflow, a green checkmark, and a nagging suspicion something was off. Turned out their BuildKit cache had never actually hit once in three months — it was pulling a stale :latest tag as cache source and silently falling back to a full rebuild every time. This is the single most common failure mode I see with Docker BuildKit cache CI setups, and it's almost always invisible until someone actually times the build.

What BuildKit Cache Actually Does Under the Hood

The legacy Docker build cache (pre-BuildKit) was dead simple: each instruction in the Dockerfile produced a layer, and that layer was reused if the instruction and its parent layer hadn't changed. It's content-addressable, but tightly coupled to instruction order. Change line 3, and everything below it invalidates — no exceptions.

BuildKit changes the model. Cache is keyed by a digest computed from the actual inputs to a step: the base image digest, the build context checksum for anything copied in, and the resolved command. That's why two builds with identical Dockerfiles but different base image digests will miss cache even though the text is byte-identical — the input hash changed, not the instruction.

There are three cache backends that matter in CI. Inline cache (BUILDKIT_INLINE_CACHE=1) embeds cache metadata directly into the pushed image — this is deprecated since Buildx v0.10 in favor of registry cache. Registry cache (--cache-to type=registry) pushes cache blobs to a separate manifest in your registry, independent of the final image tag. And local/GHA cache (type=local, type=gha) stores cache on disk or in GitHub's Actions cache service, capped at 10GB per repo.

The key thing to internalize: a cache "hit" requires the build context checksum, base image digest, AND the instruction to all match what's in the cache manifest. Any COPY or ADD touching a changed file invalidates that layer and every layer after it. This is exactly why layer ordering matters so much, which is where most teams get it wrong. Docker's own BuildKit cache documentation covers the backend types in more depth if you want the full matrix.

How People Use It Wrong

The first mistake I ran into with that client: pulling --cache-from myimage:latest as the cache source. The problem is :latest drifts — it's whatever the most recent successful build pushed, which might be from a completely different branch with a different dependency tree. BuildKit computes the cache key against what's actually in that manifest, and if it doesn't match, you get a silent miss with zero error message. No warning, just a slow build that looks "normal."

The second mistake is structural: COPY . . before npm ci or pip install. I've seen this in maybe 70% of Dockerfiles I've audited. Every commit — even a one-line README change — invalidates the entire dependency install layer, because the build context checksum for that COPY includes every file in the repo. You end up reinstalling node_modules or your virtualenv on every single push, even when the lockfile hasn't moved.

The third mistake is relying purely on the CI runner's local disk. GitHub-hosted runners are ephemeral — each job gets a fresh VM. If you never export cache to a registry or the GHA backend, that "warm cache" from your last build simply doesn't exist anymore. I stopped trusting local-only caching on hosted runners after watching a team's "optimized" pipeline run cold for six months without anyone noticing, because nothing ever raised an error — the pipeline was just always slow. A related mistake: using --cache-from without --cache-to. You pull old cache but never write updates back, so the cache goes stale after the first real change and never refreshes.

The Correct Approach

The fix has two parts: Dockerfile structure and cache backend configuration. Structurally, order instructions from least volatile to most volatile — copy lockfiles first, install dependencies, then copy source last.

FROM node:20-slim@sha256:abcd1234...  # pinned digest avoids silent cache invalidation

WORKDIR /app

# 1. Copy only lockfiles first — this layer stays cached until deps actually change
COPY package.json package-lock.json ./

# 2. Use cache mount for npm store — persists package cache across builds
RUN --mount=type=cache,target=/root/.npm,sharing=locked \
    npm ci --prefer-offline

# 3. Source code copied last — only invalidates this layer + below on every commit
COPY . .

RUN npm run build

# Expected CI log on cache hit:
# => CACHED [2/5] COPY package.json package-lock.json ./
# => CACHED [3/5] RUN npm ci --prefer-offline
# => [4/5] COPY . .                                   0.4s
# => [5/5] RUN npm run build                          22.1s

Pinning the base image by digest matters more than people think — node:20-slim can get re-tagged upstream without any change on your end, which silently busts your cache even though nothing in your repo changed.

For the backend, registry cache is the most portable option since it works regardless of which runner picks up the job:

# .github/workflows/docker-build.yml
# CI pipeline demonstrating correct BuildKit cache setup with registry cache backend
name: docker-build

on:
  push:
    branches: [main]
  pull_request:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
        with:
          driver-opts: image=moby/buildkit:v0.13.2   # pin buildkit version explicitly

      - name: Log in to registry
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Build and push with registry cache
        uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: ghcr.io/org/app:${{ github.sha }}
          # cache-from pulls previous layers; cache-to writes updated ones
          cache-from: type=registry,ref=ghcr.io/org/app:buildcache
          cache-to: type=registry,ref=ghcr.io/org/app:buildcache,mode=max
          build-args: |
            BUILDKIT_INLINE_CACHE=0   # not needed, we use registry cache explicitly

      - name: Prune stale cache tags (weekly)
        if: github.event_name == 'schedule'
        run: |
          # crude example: delete cache manifests older than 14 days via crane
          crane ls ghcr.io/org/app | grep buildcache | while read tag; do
            echo "would prune $tag if older than 14d"
          done

Gotcha: if you see ERROR: failed to solve: failed to load cache key, don't panic — it usually just means the cache ref doesn't exist yet (first run ever) or your registry auth expired mid-pull. It's not a corruption issue, just resolve auth or let the first build populate the ref.

Advanced Patterns

Once you've got the basics working, there are a few patterns worth adopting for teams running multiple services or monorepos. First, mode=max vs mode=min: max exports every intermediate layer from every build stage, which is essential if you have multi-stage Dockerfiles with shared base stages across microservices — otherwise those intermediate stages never get cached for reuse elsewhere. mode=min only exports the final stage, which is cheaper but useless if other targets depend on the same intermediate layer.

Second, in monorepos, shard your cache keys by a hash of the Dockerfile plus the relevant lockfile, not just by repo name. Otherwise service A's cache pollutes service B's cache ref and you get cross-project false hits or unnecessary invalidation storms.

Third, if you're building several targets in one CI job, docker buildx bake with an HCL file lets you define cache-from/cache-to once and apply it across a matrix of targets, instead of copy-pasting the same flags into five separate build steps. It also avoids redundant registry pulls when multiple targets share a base stage. I wrote more about structuring shared build configs like this in our DevOps_DayS archive if you want to see it applied to a full CI matrix.

Watch out: multi-arch builds (--platform linux/amd64,linux/arm64) need separate cache refs per platform, or your arm64 build won't be able to reuse amd64 cache layers at all — I've seen teams assume one cache ref covers both architectures and wonder why arm64 builds are always cold.

Performance Notes

Numbers matter more than theory here. A cold CI runner with zero cache backend typically takes 8-12 minutes for a standard Node or Python app build. With registry cache configured correctly and dependency layers ordered properly, warm builds routinely drop to 30-90 seconds. That's the real payoff — but it's not free.

Registry cache push/pull adds latency per layer, usually 5-30 seconds depending on registry throughput and image size. It's a net win only if you're building frequently enough that the amortized savings outweigh that overhead — for a repo pushed a few times a day, it's an easy yes.

Storage cost is the tradeoff nobody budgets for. mode=max exports every intermediate layer, which can balloon registry storage to 2-3x your actual image size. On ECR, GCR, or ACR, that's billed per GB-month, and unbounded caches across dozens of feature branches will quietly grow your bill. Set a lifecycle rule to expire *:buildcache tags older than 14 days — it's a five-minute fix that prevents a very avoidable line item.

One more thing worth flagging as a security note, not just performance: cache blobs pushed to a shared or public registry can leak build secrets or leftover file contents if you're not using --mount=type=secret. Cache layers aren't scrubbed of env vars or files copied in before a later cleanup RUN — if a secret touched the filesystem at any point, it can persist in the cache manifest even after you delete it in a later layer. I've had to explain this one twice to teams who assumed a final RUN rm -rf was enough. Check the Docker build secrets docs before you bake anything sensitive into a cached stage.