Stop rebuilding from scratch: cache Docker layers on Cloud Build

Mazlum Tosun — Mon, 07 Sep 2026 22:31:01 +0000

How BuildKit + Artifact Registry turn your CI/CD pipelines around: if nothing changed, uv doesn't re-download a single package.

📦 All the code in this article is on GitHub: tosun-si/docker-cloud-build-ci-cd-cache

How I got here

Early in my career, when I built CI pipelines, I didn't have much of an
optimization or GreenOps sensitivity. Pipelines ran, images got built, and I
never really questioned the wasted work.

That changed after a side collaboration with my friend Guillaume Leroy a while back. It got me
paying attention to build efficiency — and once I started, the benefits were
obvious: my pipelines were faster, leaner, and I wasn't burning machines for
nothing. That last part matters: an optimized pipeline is also a greener one.

I became a convert. Today I apply these techniques everywhere — in my personal
projects, and as a platform engineer at my clients'. The developers love it
(CI/CD pipelines are dramatically faster), and the clients care too: carbon
footprint and FinOps are real topics for them. So I bring this discipline to
every CI/CD tool I work with.

And that's where Cloud Build surprised me. Almost nobody around me — including
in the GDE and Google community — persists the Docker cache on Cloud Build,
Google's serverless CI/CD tool. When I dug into why, the official guidance
explained it: Google's own docs point you to --cache-from <previous image>
(which, as we'll see, silently misses your multi-stage builder layers) or to
Kaniko — a tool Google itself archived on June 3, 2025 ("this project is
archived and no longer developed or maintained"). Neither path mentions
BuildKit's registry cache with mode=max, the one that actually persists the
expensive dependency layer. A handful of scattered blog posts cover it; the
official guidance still doesn't.

Everything ships as a container these days, so this is a lever almost every team
leaves on the table. It saves me real time every day, on my own projects and
with the teams I work with — and I wanted to write this up to share it with the
community.

One more reason it stays a blind spot: the default behavior genuinely makes it
feel like "Docker caching just doesn't work on Cloud Build." Every build starts
from scratch, re-downloads every dependency, rebuilds every layer. On a Python
app with a few dozen packages, that's 30–60 seconds wasted on every push,
even though nothing changed on the dependency side. The good news: it's not a
fatality. With BuildKit and a cache stored in Artifact Registry, you get a real
persistent cache shared across pipelines — and once you understand why the
default fails, the fix is three lines.

This is the first article in a series. Here we start from the classic
docker build (via docker buildx). In a second article, I'll show the same
mechanism with Docker Bake, which is just a declarative layer on top of the
same cache engine.

Why the cache "disappears" on Cloud Build

Locally, Docker's layer cache feels like magic: you rebuild, and Docker reuses unchanged layers straight from the local daemon. The cache storage is your machine.

Cloud Build runs on ephemeral workers. Every build starts on a fresh VM, with no state from previous builds. The VM's Docker daemon is empty. As a result:

There is no local cache to reuse, because there is no "local" that persists.

This isn't a bug, it's the model: isolation and reproducibility. But it means a bare docker build on Cloud Build will never cache anything between runs.

So the solution isn't to keep a local cache — that's impossible — it's to externalize the cache into a registry that every build shares. And you already have that registry: Artifact Registry.

Quick refresher: Docker's layer cache

Each instruction in a Dockerfile produces a layer. Docker reuses a layer as long as:

the instruction is identical, and
its input context (copied files, parent layer) is identical.

The moment a layer is invalidated, every layer below it is too. Hence the golden rule:

Copy what changes rarely first (the dependencies), what changes often next (the source code).

That's exactly what makes the Python + uv example so telling: resolving and installing dependencies is a heavy layer, but a stable one. As long as pyproject.toml and uv.lock don't move, we should never rebuild it.

The example app: FastAPI + uv

A minimal app, just enough to have real dependencies (FastAPI + uvicorn and their transitive tree — 21 packages).

pyproject.toml:

[project]
name = "cloud-build-cache-demo"
version = "0.1.0"
requires-python = ">=3.13"
dependencies = [
    "fastapi>=0.115.0",
    "uvicorn[standard]>=0.32.0",
]

# No [build-system]: this is an application, not a reusable package.

Notice there is no [build-system]. This is deliberate: it makes the app a virtual uv project — uv resolves and installs the dependencies, but never tries to build or install the app itself. The code just runs from source. That keeps the Docker build trivial and the dependency layer perfectly cacheable.

app/main.py:

from fastapi import FastAPI

app = FastAPI(title="Cloud Build cache demo")


@app.get("/")
def root() -> dict[str, str]:
    return {"message": "Hello from a cache-friendly Cloud Build pipeline"}


@app.get("/health")
def health() -> dict[str, str]:
    return {"status": "ok"}

The uv.lock is generated once with uv lock and committed — that's what makes the build reproducible and the cache deterministic.

Why a flat app/ folder at the repo root, not a src/ layout? The src layout earns its keep for a reusable package: it stops Python from importing your working tree instead of the installed wheel, and forces tests to run against what you actually ship. But this is a deployed application, not a library — we don't package it into a wheel at all. A root-level folder named after the app is simpler and reads better, and since uv treats it as a virtual project (no [build-system]), there's nothing to install and nothing to shadow.

The Dockerfile: multi-stage and cache-friendly

Two stages: a builder based on the uv image (which ships uv + the right CPython), and a runtime python:slim image without uv, running non-root.

# syntax=docker/dockerfile:1.7

ARG APP_DIR=/usr/local/src/app

# ---------- builder ----------
FROM ghcr.io/astral-sh/uv:python3.13-bookworm-slim AS builder
ARG APP_DIR
ENV APP_DIR=${APP_DIR}
WORKDIR ${APP_DIR}

ENV UV_COMPILE_BYTECODE=1 \
    UV_LINK_MODE=copy \
    UV_PYTHON_DOWNLOADS=0

# Dependencies only — a virtual uv project installs the deps but never the
# app itself. This layer is reused as long as pyproject.toml + uv.lock don't
# change: THIS is the layer we want to survive across Cloud Build runs.
COPY pyproject.toml uv.lock ./
RUN --mount=type=cache,target=/root/.cache/uv \
    uv sync --frozen --no-dev

# ---------- runtime (no uv) ----------
FROM python:3.13-slim-bookworm AS runtime
ARG APP_DIR=/usr/local/src/app
ENV APP_DIR=${APP_DIR} \
    PATH="${APP_DIR}/.venv/bin:${PATH}" \
    PYTHONPATH="${APP_DIR}" \
    PYTHONUNBUFFERED=1 \
    PYTHONDONTWRITEBYTECODE=1

WORKDIR ${APP_DIR}

RUN groupadd --system app \
 && useradd --system --gid app --home-dir ${APP_DIR} --shell /usr/sbin/nologin app \
 && chown app:app ${APP_DIR}

# The venv (deps) from the builder, then the app source. Copying the code last
# means a code change never invalidates the dependency layer above.
COPY --from=builder --chown=app:app ${APP_DIR}/.venv ${APP_DIR}/.venv
COPY --chown=app:app app ./app

USER app
EXPOSE 8000

ENTRYPOINT ["uvicorn"]
CMD ["app.main:app", "--host", "0.0.0.0", "--port", "8000"]

Two details do all the work:

The heavy uv sync layer is built from pyproject.toml + uv.lock alone, and the app source is copied after it (into the runtime stage). A code change never touches the dependency layer.
--mount=type=cache on /root/.cache/uv. uv's download cache, on top of the layer cache. We'll come back to it: on Cloud Build, that layer needs mode=max to survive.

Plus two hygiene points straight from the conventions: non-root runtime and a final image without uv (we only copy the produced .venv).

The tempting-but-wrong fix: `--cache-from` on the previous image

The trick most people reach for first — and, notably, the one Google's own
Best practices for speeding up builds
recommends:

# pull the previous image, use it as cache
docker pull "$IMAGE:latest" || true
docker build --cache-from "$IMAGE:latest" -t "$IMAGE:latest" .

It seems logical, but it's disappointing with multi-stage builds, for two reasons:

Only the final-stage layers live in the image. The builder stage layers (where uv sync runs!) aren't there. So uv sync re-runs on every build.
Without BuildKit and its "inline" cache, layer matching is brittle.

In other words: the expensive layer — installing the dependencies — is precisely the one this cache doesn't recover. That's where the "Docker caching is useless on Cloud Build" belief comes from. The docs' other suggestion, Kaniko cache, does handle intermediate layers — but Google archived Kaniko in June 2025, so building your pipeline on it today means adopting an unmaintained tool. Which leaves the approach the docs don't mention.

The real fix: BuildKit's registry cache

Let's put the two versions side by side. Same docker buildx build, two lines of difference.

Without cache — the baseline. Every run on a fresh worker rebuilds everything, uv sync included:

docker buildx build \
  --push -t "$IMAGE" .

With cache — BuildKit exports the entire build graph (including the intermediate layers of every stage) to a dedicated cache image, and imports it on the next run:

docker buildx build \
  --cache-from "type=registry,ref=$CACHE" \
  --cache-to   "type=registry,ref=$CACHE,mode=max" \
  --push -t "$IMAGE" .

That's it — two flags:

--cache-to type=registry,mode=max pushes all layers (final and intermediate) into $CACHE. It's mode=max that changes everything — mode=min (the default) would only export the final-stage layers, and we'd fall right back into the previous trap.
--cache-from type=registry: on the next build, BuildKit imports those layers from the registry before building. The uv sync layer is found by its hash → reused, nothing re-downloaded.

The cache is stored as an ordinary image, tagged :buildcache by convention in the same Artifact Registry repo as the app image. That registry is shared by every Cloud Build worker: that's the "inter-pipeline" cache.

mode=max is the keystone. It's the one setting that makes the dependency layer survive between builds on ephemeral workers. Without it, the whole exercise is pointless.

Here's the whole flow at a glance — build #1 exports the cache, build #2 (a fresh, unrelated worker) imports it and skips uv sync entirely:

The Cloud Build config: `build-python-app.cloudbuild.yaml`

We're focusing on the cache, so this config does one thing: build + push with
the registry cache. (Deploying the image — to Cloud Run or anywhere else — is a
separate concern; more on that at the end.)

steps:
  # Build + push with a container-driver Buildx builder.
  # The container driver is REQUIRED to export/import cache to a registry —
  # the default "docker" driver cannot do type=registry cache.
  - name: 'gcr.io/cloud-builders/docker'
    id: build-push
    env:
      - 'IMAGE_PATH=${_REGION}-docker.pkg.dev/$PROJECT_ID/${_REPO}/${_IMAGE}'
      - 'GIT_SHA=${SHORT_SHA}'
    script: |
      #!/usr/bin/env bash
      set -euo pipefail

      TAG="${GIT_SHA:-manual}"
      CACHE="${IMAGE_PATH}:buildcache"

      docker buildx create --name cloudbuilder --driver docker-container --use

      docker buildx build \
        --tag "${IMAGE_PATH}:${TAG}" \
        --tag "${IMAGE_PATH}:latest" \
        --cache-from "type=registry,ref=${CACHE}" \
        --cache-to   "type=registry,ref=${CACHE},mode=max" \
        --push \
        .

substitutions:
  _REGION: europe-west1
  _REPO: internal-images
  _IMAGE: cloud-build-cache-demo

options:
  logging: CLOUD_LOGGING_ONLY

Three things worth calling out:

docker buildx create --driver docker-container is the non-obvious bit. Cloud Build's default docker driver does not support type=registry cache. So we spin up a containerized BuildKit builder for the duration of the build. This one detail is what 90% of attempts are missing.
options.logging: CLOUD_LOGGING_ONLY isn't cosmetic. As soon as you run under a custom service account (impersonation / WIF), Cloud Build requires you to pick a log destination or the build fails with you must specify logging. CLOUD_LOGGING_ONLY sends logs to Cloud Logging only — no GCS bucket to manage. It's the recommended default today.
No machineType — the default (e2-standard-2) is plenty for a small Python build. Bump to E2_HIGHCPU_8 only for CPU-bound or multi-target builds; it's faster but billed at a higher per-minute rate. It has no effect on caching.

Note: on the very first build, --cache-from on a non-existent cache is a harmless warning — there's simply nothing to import yet.

GCP prerequisites

A Docker Artifact Registry repo — reuse an existing one (here it's internal-images) or create it:

gcloud artifacts repositories create internal-images \
  --repository-format=docker \
  --location=europe-west1

The app image and the :buildcache image both live in this repo, so a single repo is all you need.

IAM for the Cloud Build service account (the default <PROJECT_NUMBER>-compute@developer.gserviceaccount.com, or your dedicated SA) — it needs to push both the app image and the cache image, which live in the same repo, so a single binding covers it:

gcloud artifacts repositories add-iam-policy-binding internal-images \
  --location=europe-west1 \
  --member="serviceAccount:<CLOUD_BUILD_SA>" \
  --role="roles/artifactregistry.writer"

Wire the config to a push trigger on your main branch — $SHORT_SHA is then populated automatically on every merge, and the cache does its job silently from the second build onward. No manual invocation needed.

The demo: before / after

Build #1 (cold cache) — nothing in :buildcache. BuildKit resolves and installs the 21 packages, then pushes every layer to the cache. This is the slowest build.

Build #2 (change main.py only) — pyproject.toml/uv.lock unchanged:

BuildKit imports the cache from Artifact Registry;
the COPY pyproject.toml uv.lock layer → CACHED;
the uv sync layer → CACHED (zero packages downloaded);
only the layers from COPY app down are rebuilt.

In the Cloud Build logs you'll see it explicitly:

 => CACHED [builder 4/5] COPY pyproject.toml uv.lock ./
 => CACHED [builder 5/5] RUN uv sync --frozen --no-dev
 => [runtime 6/7] COPY app ./app

That CACHED on uv sync, on a brand-new VM that has never seen this project, is the whole point of this article. The layer comes from Artifact Registry, not from a local disk.

Build #3 (change a dependency) — now uv.lock changes, the layer is correctly invalidated and re-downloaded. That's the right behavior: the cache tracks exactly what actually changed.

Measured impact

Numbers make the case. Here's the same pipeline across the three scenarios,
one real Cloud Build run each (default machine type, region
europe-west1, build-only — no deploy step):

Scenario	Total build	`uv sync` layer	Packages installed
#1 — cold cache (first build)	51 s	rebuilt (2.6 s)	19
#2 — code change only (`main.py`)	36 s	CACHED	0
#3 — dependency bump (`+httpx`)	55 s	rebuilt (2.6 s)	22

The proof is right there in the Cloud Build logs — the exact same uv sync
step, on three fresh, unrelated workers:

# Build #1 (cold)        →  #16 [builder 4/4] RUN ... uv sync --frozen --no-dev
                            #16 DONE 2.6s

# Build #2 (code change) →  #14 [builder 4/4] RUN ... uv sync --frozen --no-dev
                            #14 CACHED            ← pulled from Artifact Registry

# Build #3 (dep bump)    →  #15 [builder 4/4] RUN ... uv sync --frozen --no-dev
                            #15 DONE 2.6s         ← uv.lock changed, correctly re-run

That #14 CACHED, on a worker that had never seen this project, is the whole
point: the dependency layer was rebuilt on build #1, exported to Artifact
Registry, and imported on build #2 — zero packages downloaded, uv sync
skipped entirely.

Read it honestly. On this deliberately tiny app, uv sync is only ~2.6 s
and 19 pure-Python wheels, so the total delta (51 s → 36 s) is dominated by
things the cache doesn't remove on an ephemeral worker: base-image pulls,
BuildKit startup, cache import/export I/O. The headline isn't "15 seconds
saved" — it's 0 packages installed and the uv sync layer fully skipped.
On a real service (dozens of deps, compiled wheels, a numpy/pyarrow in the
tree), that skipped layer is minutes, not seconds — and it's skipped on every
push where uv.lock hasn't moved.

Want to see it on video? The repo ships a heavy dependency group
(pandas + numpy, compiled wheels) that's off by default. Flip it on
(--build-arg INSTALL_HEAVY=1, or _INSTALL_HEAVY=1 on Cloud Build) and the
uv sync step jumps from ~3 s to minutes — so the cold-vs-CACHED contrast
is impossible to miss on screen. Same cache mechanism, just a louder signal.

Reproduce it yourself with three runs:

Run #1 — cold cache. Delete the :buildcache tag first (gcloud artifacts docker images delete .../<image> --delete-tags), then build. Baseline.
Run #2 — code only. Change a string in app/main.py, rebuild. uv.lock untouched → uv sync shows CACHED.
Run #3 — dependency bump. uv add <pkg>, rebuild. uv.lock changed → uv sync correctly re-runs.

Pull the durations and the per-step cache hits straight from the API:

gcloud builds list --limit=3 --format='table(id, duration, status)'
gcloud builds log <BUILD_ID> | grep -E 'CACHED|uv sync|DONE'

Gotchas worth knowing

docker-container driver is mandatory. Without it, --cache-to type=registry is ignored or fails silently.
mode=max or nothing. In mode=min, the builder stage's intermediate layers aren't exported → uv sync re-runs every time.
The :buildcache tag grows. Each build stacks layers onto it. Set an Artifact Registry cleanup policy, or rewrite the tag periodically. The cache is a convenience, not a source of truth.
Cache network cost. Pushing/pulling the cache has an I/O cost. On a small image it's a net win from build #2; on huge images, measure — sometimes mode=min on select stages is the better trade.

The angle nobody talks about: this is GreenOps

We frame CI caching as a speed win. It's also a sustainability one, and that's rarely said out loud.

Every uncached build re-runs uv sync: CPU cycles to resolve the graph, network to pull wheels, CPU again to unpack and byte-compile them. Multiply that by the number of builds a team ships per day — every push, every PR, every retry — across every developer. A 30-second dependency step that runs 50 times a day is 25 minutes of pure CPU burn, daily, producing an artifact bit-for-bit identical to the previous one.

Caching turns most of those runs into a near-instant registry pull. Concretely, that's:

Less compute → fewer CPU-seconds → less energy drawn in the datacenter → a smaller carbon footprint. This is textbook GreenOps: don't recompute what hasn't changed.
Less money → Cloud Build is billed by the build-minute, so the green win is also a FinOps win. The two point the same way.
Faster feedback → shorter pipelines, less waiting, less context-switching for the whole team.

The honest caveat: the cache isn't free. Pushing and pulling it has its own I/O, storage, and (small) compute footprint. It's net-green when the compute you avoid is larger than the cache transfer you add — which is exactly the case for dependency-heavy builds like this one, and the reason ordering the Dockerfile well matters so much. For a trivial image with no real dependency layer, the math can flip; measure before assuming.

The takeaway: CI optimization isn't just about developer experience (DX). Recomputing an identical artifact on every push is waste — of time, of money, and of energy. A well-placed cache removes all three at once.

What about deploying?

You may have noticed this config stops at "push to Artifact Registry" — no
gcloud run deploy, no rollout. That's on purpose. Deployment is its own topic
(runtime service accounts, IAM, IAP, traffic splitting…) and folding it in here
would only blur the one thing this article is about: the cache. Once the image
is in Artifact Registry, adding a deploy step — Cloud Run, GKE, wherever — is a
few extra lines, and I'll cover that shape in a dedicated article.

What's next: Docker Bake

This config works, but the --cache-from / --cache-to flags get verbose fast once you have several images (API, worker, front…). That's exactly the problem Docker Bake solves: you declare targets, tags and cache in a docker-bake.hcl, and a single docker buildx bake --push orchestrates all of it, registry cache included.

That's the topic of the second article — same BuildKit cache engine, same Artifact Registry, but a declarative, multi-target config. A YouTube video will follow to watch the whole thing run live.

Takeaways

Cloud Build runs on ephemeral workers → no local cache persists.
The fix: externalize the cache into Artifact Registry with BuildKit's registry cache.
The winning combo: docker buildx + docker-container driver + --cache-to type=registry,mode=max.
A well-ordered Dockerfile (dependencies before code) turns that cache into a concrete win: uv re-downloads nothing as long as uv.lock hasn't changed.
mode=max isn't a detail — it's the reason this works where the classic --cache-from fails.
It's not just speed: skipping the recompute of an identical artifact is GreenOps + FinOps — less CPU, less energy, less money, on every single push.

The full demo code is on GitHub. Clone it, point it at your GCP project, and watch the second build print CACHED on uv sync. 🎯

If you found this useful, follow me for more hands-on content on Google Cloud, Platform Engineering, Docker, DevOps, Data Engineering and AI agents — practical patterns from real projects and client work:

DEV Community: Mazlum Tosun