Originally published on kuryzhev.cloud
The BuildKit cache that makes local rebuilds fast is usually missing on a CI runner. A developer changes one line of application code, pushes, and the pipeline reinstalls every dependency from scratch. Locally the same Dockerfile reuses nearly every layer.
The cause is mostly structural. Hosted runners and autoscaled build agents are ephemeral, so the layer store from the previous job is gone when the next one starts. Unless the pipeline exports cache somewhere durable and imports it again, every build is a cold build. The mistakes below are documented patterns rather than reports of any specific incident. They show up repeatedly in CI setups, and each has a known fix.
Context: where the BuildKit cache actually lives
BuildKit stores cache in the builder, not in the image. On a developer machine the builder is long-lived, so the cache persists across builds. In CI, a fresh builder starts empty unless you attach an external cache backend with --cache-from and --cache-to.
The documented backends include inline, registry, local, gha, s3 and azblob. They differ in what they can store and which builder drivers support them.
The default docker driver has limited export support unless the containerd image store is enabled. The docker-container driver supports the full set, and it is what docker buildx create and the setup actions use. See the official cache backend documentation for the current matrix, because support has changed across releases.
Two separate things must both be right for caching to pay off. The Dockerfile has to produce stable layer keys, and the exporter has to persist those layers and make them reachable on the next run. Most failures come from getting one right and assuming the other follows. For more pipeline-focused writing, the kuryzhev.cloud homepage lists other CI/CD topics.
Common failure 1: Dockerfile layers that invalidate themselves
A layer is reused only if its instruction and all its input files are unchanged, and every layer after a changed one is rebuilt. The classic mistake is copying the whole source tree before installing dependencies, so any edit invalidates the install step.
A second source of self-inflicted misses is build metadata. Declaring ARG GIT_SHA or a timestamp near the top changes the cache key for everything below it on every commit. A missing .dockerignore is the third: .git, local node_modules or CI artifacts get pulled into the build context, and COPY . . sees a different context each time.
This ordering keeps dependency layers stable and pushes volatile inputs to the end:
FROM node:24-slim AS build
WORKDIR /app
# Copy only manifests first: this layer changes only when dependencies change
COPY package.json package-lock.json ./
RUN npm ci
# Volatile source comes after the expensive install
COPY . .
RUN npm run build
# Declare build metadata late so it only affects layers below it
ARG GIT_SHA
LABEL org.opencontainers.image.revision=$GIT_SHA
Watch out for build arguments declared before expensive RUN commands. Every RUN after an ARG receives it implicitly as an environment variable. A changed value can therefore cause a cache miss even if the command never references it.
Also keep .git, node_modules, test output and local env files in .dockerignore. Verify the effect by running the same build twice and checking for CACHED markers in the plain progress output (--progress=plain).
Common failure 2: exporting cache that never gets reused
The second pattern is a pipeline that has --cache-to configured and still rebuilds everything. There are several documented reasons.
First, type=inline embeds cache metadata in the pushed image and only covers the final stage. In a multi-stage build, the expensive builder stage is not cached. The registry and gha backends accept mode=max, which exports intermediate stages too. The default is mode=min.
Second, the builder driver matters. On the plain docker driver without the containerd image store, exporting to anything other than inline is unsupported, so the pipeline should use a docker-container builder.
Third, some registries need extra exporter options. Amazon ECR, for example, is documented to require image-manifest=true and oci-mediatypes=true for registry cache. Check your registry's current guidance, because this is version- and vendor-dependent.
Fourth, the cache reference and scope can collide. Several branches writing to one tag overwrite each other. Parallel jobs writing the same gha scope can do the same.
Watch out for a silent import failure. A missing or unreadable cache reference typically produces a warning, not a failed job, so the build proceeds cold and looks normal. Search job logs for import warnings, or confirm a stage shows CACHED after a no-change rebuild.
Common failure 3: stale, oversized or unpersisted cache
Once caching works, the opposite problems appear. Cached layers are reused as long as their inputs match, and that is not the same as being current.
A RUN apt-get update && apt-get install -y ... layer can stay frozen for months because its instruction text never changes. A floating base tag like FROM node:24-slim can keep resolving to an old cached base unless the build pulls fresh (--pull). The result is an image that builds fast but misses OS security patches.
Size is the next issue. mode=max exports every intermediate layer, and CI-native storage has limits. GitHub Actions cache has a per-repository quota with eviction, as described in the Docker documentation for GitHub Actions cache. A large, frequently rewritten cache can evict itself and thrash. Check current quotas and the cache service version requirements in the GitHub documentation, since they have changed over time.
Finally, cache mounts, such as RUN --mount=type=cache,target=/root/.npm, are a different mechanism. They live in the builder's local storage and are not exported by --cache-to, so on an ephemeral runner they start empty every time. They speed up repeated builds on a persistent builder but do nothing for a fresh one unless you persist that directory by other means.
Watch out for secrets passed as build arguments. They can end up in image history or cache metadata. Use --mount=type=secret instead.
Safer operating pattern for the BuildKit cache
A dependable setup treats cache as a performance hint that must never affect correctness. It is also verified and refreshed on purpose. The example below uses a registry cache with per-branch fallback on GitHub Actions.
Branch names such as feature/login contain slashes, which are not valid in image tags, so the workflow sanitizes the ref first. Confirm current major versions of the actions before copying them.
name: build
on: [push]
jobs:
image:
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
steps:
- uses: actions/checkout@v5
- uses: docker/setup-buildx-action@v3 # creates a docker-container builder
- uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Compute a tag-safe cache ref
run: echo "CACHE_TAG=cache-${GITHUB_REF_NAME//\//-}" >> "$GITHUB_ENV"
- uses: docker/build-push-action@v6
with:
context: .
push: true
pull: true # refresh floating base tags
tags: ghcr.io/example/app:${{ github.sha }}
cache-from: |
type=registry,ref=ghcr.io/example/app:${{ env.CACHE_TAG }}
type=registry,ref=ghcr.io/example/app:cache-main
cache-to: type=registry,ref=ghcr.io/example/app:${{ env.CACHE_TAG }},mode=max,image-manifest=true,oci-mediatypes=true
Reading from the branch cache first and falling back to main means new branches start warm. Writing only to the branch's own ref avoids cross-branch overwrites.
Pinning a base image by digest, with an automated bump, is an alternative to pull: true on a floating tag. It gives you reproducibility without freezing security updates, and makes the pull flag redundant.
Use this checklist to review any pipeline:
[ ] Dependency manifests are copied and installed before source code
[ ] .dockerignore excludes .git, node_modules, build output, env files
[ ] ARGs for SHA/timestamp are declared after expensive layers
[ ] Builder uses the docker-container driver (not default docker)
[ ] cache-to uses mode=max if the Dockerfile is multi-stage
[ ] Registry-specific options (e.g., OCI media types) are verified
[ ] Cache refs are scoped per branch (tag-safe), with a main fallback
[ ] A scheduled job rebuilds with --no-cache --pull weekly
[ ] Second no-change build shows CACHED for install stages
[ ] Secrets use --mount=type=secret, never ARG
The scheduled cold build is easy to skip, but it matters. It catches layers that have gone stale and shows the true uncached build time. That number tells you whether the cache is helping or just hiding a slow Dockerfile. For flags and backend details that may have changed, the Docker build cache optimization guide is the reference to check.
Top comments (0)