DEV Community: Mazlum Tosun

ELT with Dataform on Google Cloud

Mazlum Tosun — Wed, 16 Sep 2026 16:29:00 +0000

This article was originally published on Medium (Google Cloud Community).

1. Explanation of the use case presented in this article

In a previous article, I demonstrated how to build an ELT pipeline using dbt on Google Cloud. In this article, we'll explore the same use case — this time using Dataform, Google Cloud's native solution for managing SQL-based data workflows.

One of the key advantages of Dataform on Google Cloud is that it's a fully managed, serverless service, natively integrated with BigQuery.

As with the dbt-focused article, I'll use a football-related, real-world ELT pipeline that includes both staging and mart layers. I'll walk through how to create a Dataform repository manually, link it with a GitHub repository, and then automate the entire setup using Terraform. This approach enables a GitOps workflow, where the GitHub repository becomes the single source of truth for managing your Dataform resources.

Here is the diagram of this use case:

I also created a video on this topic on my GCP YouTube channel. Feel free to subscribe to the channel to support my work for the Google Cloud community.

English version:

French version:

🔧 CI/CD layer

The CI/CD pipeline is implemented using Cloud Build and Infrastructure as Code (IaC) with Terraform.
It automates the creation of the Dataform repository and establishes a link with a GitHub repository, enabling a GitOps-driven workflow.

📦 Applicative ELT pipeline

Data ingestion: raw files are loaded into BigQuery using bq load.
Staging layer: reads raw data and applies initial data cleaning and standardization.
Mart layer: transforms cleaned data into business-ready domain models, applying domain-specific logic.
Dynamic views generation: creates views for player statistics, dynamically generated from the domain data.
Dynamic tables generation: builds country-specific tables to expose player statistics by country.

2. Usual structure of a Dataform project

Dataform is fully managed and deeply integrated with BigQuery, making it seamless to use within the Google Cloud ecosystem. When getting started, the Dataform console makes it easy to generate boilerplate code and a project structure, helping you ramp up quickly.

On the Dataform main page, the first step is to click the "Create repository" button to start a new project.

In the repository creation menu, you can assign any name, select the europe-west1 region, and specify a service account with the necessary permissions — typically Dataform Editor and BigQuery Data Editor to get started. For production setups, use more restricted roles aligned with the principle of least privilege.

Then click on "Go to repositories":

Click on "Create development workspace":

Then open the created workspace:

Click on "Initialize workspace":

After this step, Dataform generates the example project structure. Next, open the workflow_settings.yaml file and click on "Install Packages" to set up the required dependencies.

The boilerplate project provides code that's ready to run out of the box. It includes:

a workflow_settings.yaml configuration file
a definitions/ folder containing sample models
an includes/ folder for reusable JavaScript functions

You can execute the pipeline directly by clicking "Start execution" → "Execute actions" in the UI.

In the "Selection of actions" panel, choose all actions for this example, then click on "Start execution" to run the pipeline.

Click on "Details":

The execution is successful:

By clicking on "View details", you can access the compiled SQL queries that Dataform generates from your models.

You can also view the compiled dependency graph, which visualizes the relationships between models and their execution order.

In the next section, we'll walk through our use case and explore a more realistic, real-world scenario.

3. Structure of the project presented in this use case

3.1 Dataform configuration

The workflow_settings.yaml file, located at the root of the GitHub repo and of the Dataform project, defines key configuration parameters:

the GCP project ID
the location (region) where the Dataform workflow runs
the BigQuery dataset used to store the Dataform-generated tables
the BigQuery dataset used to store assertion results
the Dataform core version used for execution

defaultProject: gb-poc-373711
defaultLocation: europe-west1
defaultDataset: qatar_fifa_world_cup_dataform
defaultAssertionDataset: qatar_fifa_world_cup_dataform_assertions
dataformCoreVersion: 3.0.42

3.2 Definitions and models

The definitions folder contains all the Dataform models, organized into staging and mart layers.

Dataform models are defined using either SQLX or JavaScript files.

3.3 Staging layer

In the staging layer, the SQLX file cleans the raw data and applies light transformations, typically producing BigQuery views as output.

The team_players_stat_raw_cleaned.sqlx file:

config {
    type: "view",
    columns: {
        goalsScored: "Goal scored for the player.",
        assistsProvided: "Assists provided by the player.",
        appearances: "Appearances for the player."
    }
}

SELECT
    nationality,
    SAFE_CAST(goalsScored AS INT64) AS goalsScored,
    SAFE_CAST(assistsProvided AS INT64) AS assistsProvided,
    SAFE_CAST(dribblesPerNinety AS FLOAT64) AS dribblesPerNinety,
    SAFE_CAST(appearances AS INT64) AS appearances,
    SAFE_CAST(totalDuelsWonPerNinety AS FLOAT64) AS totalDuelsWonPerNinety,
    SAFE_CAST(interceptionsPerNinety AS FLOAT64) AS interceptionsPerNinety,
    SAFE_CAST(tacklesPerNinety AS FLOAT64) AS tacklesPerNinety,
    brandSponsorAndUsed,
    club,
    savePercentage,
    IF(savePercentage <> "-", TRUE, FALSE) AS isGoalKeeperStatsExist,
    fifaRanking,
    position,
    playerName,
    cleanSheets,
    nationalTeamKitSponsor,
    nationalTeamJerseyNumber,
    playerDob
FROM `qatar_fifa_world_cup_dataform.team_players_stat_raw`

Similar to dbt, the config block in a Dataform model lets you define settings specific to that model. For example:

type: how the model is materialized (e.g. a view).
columns: documentation and descriptions for specific columns.

The rest of the model is a standard BigQuery SQL query that defines the transformation logic.

3.4 Mart layer

The team_players_stat.sqlx file:

config {
  type: "table",
  description: "Description of the table.",
  columns: team_player_stat_columns_descriptions.columns_descriptions,
  bigquery: {
    partitionBy: {
      field: "ingestionDate",
      dataType: "timestamp",
      granularity: "day"
    },
    clusterBy: ["teamName"]
  }
}

WITH

team_players_stat_raw AS (
    SELECT * FROM ${ref("team_players_stat_raw_cleaned")}
),

goalKeepersStats AS (
    SELECT
        nationality,
        STRUCT(
            playerName,
            appearances,
            savePercentage,
            cleanSheets
        ) AS goalKeeperStatsStruct
    FROM team_players_stat_raw
    WHERE isGoalKeeperStatsExist IS TRUE
),

goalKeeperStatsPerTeam AS (
    SELECT
        nationality,
        ARRAY_AGG(goalKeeperStatsStruct ORDER BY goalKeeperStatsStruct.savePercentage DESC LIMIT 1)[OFFSET(0)] AS stats
    FROM goalKeepersStats
    GROUP BY
        nationality
)

SELECT
    statRaw.nationality AS teamName,
    nationalTeamKitSponsor,
    fifaRanking,
    SUM(goalsScored) AS teamTotalGoals,
    CURRENT_TIMESTAMP() AS ingestionDate,
    goalKeeperStatsPerTeam.stats AS goalKeeper,
    ${team_players_stat_functions.build_player_stats(
      "goalsScored",
      "appearances",
      "brandSponsorAndUsed",
      "club",
      "position",
      "playerDob",
      "playerName"
    )}
    AS topScorers,
    ${team_players_stat_functions.build_player_stats(
      "assistsProvided",
      "appearances",
      "brandSponsorAndUsed",
      "club",
      "position",
      "playerDob",
      "playerName"
    )}
    AS bestPassers,
    ${team_players_stat_functions.build_player_stats(
      "dribblesPerNinety",
      "appearances",
      "brandSponsorAndUsed",
      "club",
      "position",
      "playerDob",
      "playerName"
    )}
    AS bestDribblers,
    ${team_players_stat_functions.build_player_stats(
      "appearances",
      "appearances",
      "brandSponsorAndUsed",
      "club",
      "position",
      "playerDob",
      "playerName"
    )}
    AS playersMostAppearances,
    ${team_players_stat_functions.build_player_stats(
      "totalDuelsWonPerNinety",
      "appearances",
      "brandSponsorAndUsed",
      "club",
      "position",
      "playerDob",
      "playerName"
    )}
    AS playersMostDuelsWon,
    ${team_players_stat_functions.build_player_stats(
      "interceptionsPerNinety",
      "appearances",
      "brandSponsorAndUsed",
      "club",
      "position",
      "playerDob",
      "playerName"
    )}
    AS playersMostInterception,
    ${team_players_stat_functions.build_player_stats(
      "tacklesPerNinety",
      "appearances",
      "brandSponsorAndUsed",
      "club",
      "position",
      "playerDob",
      "playerName"
    )}
    AS playersMostSuccessfulTackles
FROM team_players_stat_raw statRaw
JOIN goalKeeperStatsPerTeam ON statRaw.nationality = goalKeeperStatsPerTeam.nationality
GROUP BY
    statRaw.nationality,
    nationalTeamKitSponsor,
    fifaRanking,
    goalKeeperStatsPerTeam.stats

For the mart step, the config block declares a table as the model type:

config {
  type: "table",
  description: "Description of the table.",
  columns: team_player_stat_columns_descriptions.columns_descriptions,
  bigquery: {
    partitionBy: {
      field: "ingestionDate",
      dataType: "timestamp",
      granularity: "day"
    },
    clusterBy: ["teamName"]
  }
}

We also added clustering and partitioning options in the bigquery block to improve performance and reduce cost.

3.5 Mart step: column descriptions

Instead of embedding the column descriptions directly in the SQLX model, we extracted them into a separate JavaScript file. This is particularly useful when descriptions are long, as it keeps the model clean and readable.

You have two options for column descriptions: embed them in the SQLX model, or move them to a JS file. When descriptions are verbose, I recommend the JS file for clarity, maintainability and readability.

In the config block, the columns parameter directly references the JavaScript constant that holds the descriptions:

columns: team_player_stat_columns_descriptions.columns_descriptions,

This constant is declared in the team_player_stat_columns_descriptions.js file, in the includes folder. The nested objects (top_scorers_columns, best_passers_columns, …) are declared in the same file and describe the fields of each STRUCT:

const top_scorers_columns = {
    description: "An object containing the top scorers fields.",
    columns: {
        goals: "Total number of goals scored by the top scorers.",
        players: {
            description: "List of top-scoring players",
            columns: {
                playerName: "Name of the top-scoring player.",
                playerDob: "Date of birth of the player.",
                position: "Playing position of the player.",
                club: "Club that the player is affiliated with.",
                brandSponsorAndUsed: "Brand sponsor of the player's gear.",
                appearances: "Number of matches the player has played in."
            }
        }
    }
};

// ... same pattern for the other statistics

const columns_descriptions = {
    teamName: "Name of the national football team.",
    teamTotalGoals: "Total number of goals scored by the team in the tournament.",
    fifaRanking: "Current FIFA ranking of the national team.",
    nationalTeamKitSponsor: "Official sponsor providing kits for the national team.",
    topScorers: top_scorers_columns,
    bestPassers: best_passers_columns,
    bestDribblers: best_dribblers_columns,
    goalKeeper: goal_keeper_columns,
    playersMostAppearances: players_most_appearances_columns,
    playersMostDuelsWon: players_most_duels_won_columns,
    playersMostInterception: players_most_interceptions_columns,
    playersMostSuccessfulTackles: players_most_successful_tackles_columns
};

module.exports = {
    columns_descriptions
};

3.6 Mart step: reusable function to compute player statistics

Most player statistics — such as top scorers and best passers — are computed the same way. To avoid duplicating code, I created a reusable function that encapsulates this logic.

In Dataform, one way to define reusable logic is to create a JavaScript file in the includes folder. Here, the team_players_stat_functions.js file holds this function:

function build_player_stats(statIndicator, appearances, brandSponsorAndUsed, club, position, playerDob, playerName) {
    return `
    STRUCT(
      MAX(${statIndicator}) AS ${statIndicator},
      ARRAY_AGG(
        IF(
          ${statIndicator} = 0 OR ${statIndicator} = 0.00,
          NULL,
          STRUCT(
            ${appearances},
            ${brandSponsorAndUsed},
            ${club},
            ${position},
            ${playerDob},
            ${playerName}
          )
        )
        ORDER BY ${statIndicator} DESC LIMIT 1
      )[OFFSET(0)] AS players
    )
  `;
}

module.exports = {build_player_stats};

The build_player_stats function takes the column names as parameters and injects them into the generated SQL. To make it available in SQLX or JavaScript models, it must be exported with module.exports.

I gave more details on this use case and its data modeling in the article I previously wrote on dbt, which covers the same context.

3.7 Mart step: dynamic tables and views

One great feature of Dataform is the ability to write models in JavaScript instead of SQLX when needed. This is particularly useful when you need to generate logic or structure dynamically.

That's exactly what we demonstrate here: after computing the domain data, we dynamically generate one view per statistic and one table per country. JavaScript models are well suited for this kind of logic, letting us create models programmatically with more flexibility.

The stat_dynamic_tables_and_views.js file:

const statViews =
    [
        {
            columnToSelect: "goalKeeper",
            viewName: "goal_keeper"
        },
        {
            columnToSelect: "topScorers",
            viewName: "top_scorers"
        },
        {
            columnToSelect: "bestPassers",
            viewName: "best_passers"
        }
    ];

const statPerCountryTables = [
    {
        countryName: "France",
        tableName: "france_players"
    },
    {
        countryName: "Argentina",
        tableName: "argentina_players"
    }
]

statViews.forEach((view) => {
    publish(view.viewName + "_stat").query(
        ctx => `
            SELECT ${view.columnToSelect}
            FROM ${ctx.ref("team_players_stat")}
        `
    );
});

statPerCountryTables.forEach((table) => {
    publish(table.tableName + "_stat",
        {
            type: "table"
        })
        .query(
            ctx => `
                SELECT *
                FROM ${ctx.ref("team_players_stat")}
                WHERE teamName = "${table.countryName}"
            `
        );
});

We declare two const arrays — one for the views and one for the tables — and loop over each with forEach. The publish function then defines each view and table dynamically.

Below is the diagram of the ELT pipeline built with Dataform, with the data lineage showing how each component connects across the workflow.

4. Connect the GitHub repo to the Dataform repo from the console

In real-world projects, Dataform code is typically managed in GitHub repositories. Dataform can synchronize a repository over either HTTPS or SSH.

I prefer SSH, as it's both more secure and more convenient with GitHub. In this example, I had already added an SSH key to my GitHub account, which lets me reuse it across multiple repositories. You could also add a key at the repository level (deploy key), but managing it at the account level is more practical in my case.

To retrieve the GitHub SSH public host key in the format expected by a known_hosts file, run:

ssh-keyscan -t rsa github.com

It displays GitHub's public host key:

# github.com:22 SSH-2.0-4c545346
github.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj7ndNxQowgcQnjshcLrqPEiiphnt+VTTvDP6mHBL9j1aNUkY4Ue1gvwnGLVlOhGeYrnZaMgRK6+PKCUXaDbC7qtbW8gIkhL7aGCsOr/C56SJMy/BCZfxd1nWzAOxSDPgVsmerOBYfNqltV9/hWCqBywINIR+5dIg6JTJ72pcEpEjcYgXkE2YEFXV1JHnsKgbLWNlhScqb2UmyRkQyytRLtL+38TGxkxCflmO+5Z8CSSNY7GidjMIZ7Q4zMjA2n1nGrlTDkzwDCsw+wqFPGQA179cnfGWOWRVruj16z6XyvxvjJwbz0wQZ75XK5tKSb7FNyeIEs4TT4jk+S4dhPeAUC5y+bDYirYgM4GC7uEnztnZyaVWQ7B381AK4Qdrwt51ZqExKbQpTUNn+EjqoTwvqNj4kqx5QUCI0ThS/YkOxJCXmPUWZbhjpCg56i+2aB6CmK2JGhn57K5mj0MNdBXA4/WnwH6XoPWJzK5Nyu2zB3nAZp+S5hpQs+p1vN1/wsjk=

Make sure to copy the entire value, starting from ssh-rsa (or ssh-ed25519) all the way to the end of the line, without the github.com hostname.

Next, you need the private key of your SSH key pair, which Dataform uses to authenticate with your GitHub repository.

Use one of the following commands, depending on the type of key you generated:

cat ~/.ssh/id_rsa
# or
cat ~/.ssh/id_ed25519

⚠️ Be careful — never share this private key. It must be kept secure at all times.

Store the private key as a secret in Secret Manager. Dataform reads it from there to establish the SSH connection with your GitHub repository.

Open the Dataform repository you created earlier and go to the Settings tab. From there, click on "Connect with Git" to link your repository to a Git provider such as GitHub.

Then fill in the Git connection form:

the SSH URL of the remote GitHub repository
the default branch (e.g. main)
the secret version containing your private SSH key (from Secret Manager)
the SSH public host key, in known_hosts format (from ssh-keyscan)

To let Dataform read the private key from Secret Manager, grant the Secret Manager Secret Accessor role (roles/secretmanager.secretAccessor) to the Dataform service agent:

service-{PROJECT_NUMBER}@gcp-sa-dataform.iam.gserviceaccount.com

The connection is successful:

When you create a new Dataform workspace, it automatically pulls the files and project structure from the linked GitHub repository, using the configured default branch (typically main).

5. Connect the GitHub repo to the Dataform repo with Terraform

In this section, we automate the GitHub repository link with Terraform instead of configuring it manually. This aligns with GitOps best practices, where Git is the single source of truth for infrastructure and configuration.

The Terraform code structure:

The infra folder contains all the Terraform configuration files.

The backend.tf file defines the remote state backend, which uses Google Cloud Storage (GCS) to persist the Terraform state:

terraform {
  backend "gcs" {
  }
}

The versions.tf file pins the Terraform and Google Cloud provider versions:

terraform {
  required_version = ">= 1.9.8"

  required_providers {
    google      = "= 6.14.0"
    google-beta = "= 6.14.0"
  }
}

Input variables are declared in the variables.tf file to keep the configuration flexible:

variable "project_id" {
  type = string
}

variable "region" {
  description = "Location for load balancer and Cloud Run resources"
  default     = "europe-west1"
}

variable "dataform_repo_name" {
  description = "Dataform repo name."
  type        = string
}

variable "service_account_email" {
  description = "Service Account email for the creation of the Dataform repo."
  type        = string
}

The locals.tf file declares constants such as the SSH public host key and the Secret Manager secret version of the private SSH key. These locals centralize values used across the Terraform configuration.

locals {
  github_account_host_public_ssh_key_value      = "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj7ndNxQowgcQnjshcLrqPEiiphnt+VTTvDP6mHBL9j1aNUkY4Ue1gvwnGLVlOhGeYrnZaMgRK6+PKCUXaDbC7qtbW8gIkhL7aGCsOr/C56SJMy/BCZfxd1nWzAOxSDPgVsmerOBYfNqltV9/hWCqBywINIR+5dIg6JTJ72pcEpEjcYgXkE2YEFXV1JHnsKgbLWNlhScqb2UmyRkQyytRLtL+38TGxkxCflmO+5Z8CSSNY7GidjMIZ7Q4zMjA2n1nGrlTDkzwDCsw+wqFPGQA179cnfGWOWRVruj16z6XyvxvjJwbz0wQZ75XK5tKSb7FNyeIEs4TT4jk+S4dhPeAUC5y+bDYirYgM4GC7uEnztnZyaVWQ7B381AK4Qdrwt51ZqExKbQpTUNn+EjqoTwvqNj4kqx5QUCI0ThS/YkOxJCXmPUWZbhjpCg56i+2aB6CmK2JGhn57K5mj0MNdBXA4/WnwH6XoPWJzK5Nyu2zB3nAZp+S5hpQs+p1vN1/wsjk="
  github_account_private_ssh_key_secret_version = "projects/975119474255/secrets/github_account_mazlum_tosun_private_key/versions/latest"
}

The main.tf file creates the Dataform repository and configures the connection to the GitHub repository over SSH:

resource "google_dataform_repository" "world_cup_elt_dataform_repo" {
  provider = google-beta

  project = var.project_id
  name    = var.dataform_repo_name
  region  = var.region

  service_account = var.service_account_email

  workspace_compilation_overrides {
    default_database = var.project_id
  }
  git_remote_settings {
    url            = "ssh://git@github.com/tosun-si/${var.dataform_repo_name}.git"
    default_branch = "main"
    ssh_authentication_config {
      user_private_key_secret_version = local.github_account_private_ssh_key_secret_version
      host_public_key                 = local.github_account_host_public_ssh_key_value
    }
  }
}

Conclusion

This article presented a real-world ELT pipeline built with Dataform, covering key concepts such as staging and mart layers, concrete data transformations, column documentation, and JavaScript functions for dynamic logic.

It also included a DevOps and IaC part, so you can reproduce a complete, hands-on setup that follows modern best practices.

Dataform is a powerful choice for teams working with BigQuery, thanks to its fully managed experience and tight integration with GCP. Its support for a GitOps workflow — where GitHub repositories are the single source of truth — makes it even more appealing for collaborative, production-grade data projects.

Personally, I find using a programming language like JavaScript for dynamic logic far more effective than Jinja templating. JavaScript may not be the default choice for data engineers, but it offers better readability and more power when implementing complex or repetitive logic across your models.

All the code presented in this article is available in this GitHub repository:

tosun-si / world-cup-qatar-elt-dataform

Project showing a use case with an ELT using Dataform in Google Cloud

world-cup-qatar-elt-dataform

This repo shows a real world use case with Dataform, BigQuery and Google Cloud The raw and input data are represented by the Qatar Fifa World Cup Players stats some transformations are applied with the ELT pattern and Dataform to apply aggregation and business transformations.

The video in English:

https://youtu.be/c70ry7rrm6w

The video in French:

https://youtu.be/b-6naX68YRg

Airflow DAG - ELT pipeline orchestration

The pipeline is orchestrated by an Airflow DAG (Cloud Composer) with the following steps:

Load raw data to BigQuery — Loads NDJSON player stats from GCS into a BigQuery raw table using GCSToBigQueryOperator
Invoke Dataform workflow — Invokes the Dataform workflow config of the environment (released by CI/CD) using DataformCreateWorkflowInvocationOperator
Move processed files to cold storage — Moves the input file to a cold bucket using GCSToGCSOperator

The DAG configuration is managed via Airflow Variables, loaded from world_cup_qatar_elt_dataform_dags/config/variables/{env}/variables.json.

Deploy the Airflow DAG to Cloud Composer

The DAG…

View on GitHub

If you enjoyed this article, follow me for more content on Google Cloud, BigQuery, Dataform, DevOps and data engineering:

Running an AI Agent Locally: ADK, Gemma 4, and Docker Model Runner

Mazlum Tosun — Wed, 16 Sep 2026 15:39:30 +0000

This article was originally published on Medium (Google Cloud Community).

Cloud LLMs are great for production. But during development, every API call has latency, costs money, and requires credentials. What if the LLM ran on your machine, right next to your agent?

In my previous article, I built a football statistics agent using Google ADK, BigQuery MCP via Cloud API Registry, and Gemini 2.5 Flash as the LLM — deployed to Cloud Run and Vertex AI Agent Engine.

In this article, I take the same agent and replace Gemini with Gemma 4 running locally via Docker Model Runner. The agent code barely changes. The BigQuery MCP tools stay the same. But now the LLM runs on my laptop — no cloud inference, no API key, no cost.

The full source code is available on GitHub.

What is Docker Model Runner?

Docker Model Runner is a built-in feature of Docker Desktop that lets you pull and run LLMs locally — just like pulling container images. It exposes an OpenAI-compatible API on your machine, so any tool that speaks the OpenAI protocol can use it.

# Enable the Model Runner with TCP access
docker desktop enable model-runner --tcp 12434

# Pull the model
docker model pull ai/gemma4:E4B

# Verify
docker model ls

Note: On Docker Engine (Linux), TCP access is enabled by default on port 12434 — no extra flag needed. On Docker Desktop (macOS/Windows), TCP must be explicitly enabled with --tcp 12434 to expose the API on localhost. In this article, I'm using Docker Desktop on macOS, which is why this activation step is required.

Once pulled, the model is available at http://localhost:12434/engines/v1 — the same endpoint format as the OpenAI API. You can test it directly:

curl http://localhost:12434/engines/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ai/gemma4:E4B",
    "messages": [{"role": "user", "content": "Say hello"}],
    "max_tokens": 20
  }'

No API key needed. No cloud dependency. The model runs entirely on your machine.

What is Gemma 4?

Gemma 4 is Google's latest open-weight model family, available on Docker Hub in several sizes:

Variant	On-disk size*	Raw params	Use case
`ai/gemma4:E2B`	2.94 GiB	4.65B (2B effective)	Fastest, edge-friendly — great for rapid iteration
`ai/gemma4:E4B` (= `latest`)	4.74 GiB	7.52B (4B effective)	Best trade-off for local agents — current default
`ai/gemma4:4B`	~6 GB	4B dense (Q4_K_XL quant)	Traditional dense variant, quantized
`ai/gemma4:26B`	~18 GB	26B MoE (~4B active)	High quality, sparse mixture-of-experts
`ai/gemma4:31B`	~20 GB	31B dense	Maximum quality, heavy on resources

* Sizes reported by docker model ls after pull. The Docker Hub manifest shows slightly larger figures (e.g. E4B = 6.09 GB on Hub) because the GGUF is repackaged locally.

The E2B and E4B variants use the Per-Layer Embeddings (PLE) architecture — they have more actual parameters than their "effective" size, but run at the compute cost of the smaller effective size. It's the same Matryoshka-style approach introduced with Gemma 3n.

For an agent that makes 3-4 LLM roundtrips per question (reasoning, tool call, error handling, final answer), inference speed matters. The E4B variant gives the best trade-off between quality and speed for local development — it's also now the latest default.

Architecture: what changes, what stays

The beauty of this approach is how little changes. The agent framework (ADK), the tools (BigQuery MCP via Cloud API Registry), and the system instructions are identical to the cloud version. Only the model endpoint changes.

User Question
     │
     ▼
┌──────────────┐     ┌──────────────────────┐
│  Google ADK  │────▶│  Docker Model Runner │
│  LlmAgent    │◀────│  Gemma 4 (local)     │
└──────┬───────┘     └──────────────────────┘
       │
       ▼
┌──────────────────┐     ┌───────────┐
│ Cloud API Registry│────▶│ BigQuery  │
│ BigQuery MCP      │◀────│ (cloud)   │
└──────────────────┘     └───────────┘

	Cloud version (previous article)	Local version (this article)
LLM	Gemini 2.5 Flash (Vertex AI)	Gemma 4 E4B (Docker Model Runner)
Inference cost	Per-token pricing	Free (runs on your CPU/GPU)
Agent code	`LlmAgent` + API Registry MCP	Same
Tools	BigQuery MCP	Same
System instruction	Full schema + business rules	Same

For the details on the ADK agent setup, BigQuery MCP via Cloud API Registry, the dataset, and the system instruction, refer to my previous article. Here, I'll focus on what's new: the Docker Model Runner integration and the challenges of running Gemma locally with an agent framework.

The agent code: one model string change

Here's the original agent.py from the cloud version:

MODEL = "gemini-2.5-flash"

root_agent = LlmAgent(
    model=MODEL,
    name="football_stats_agent",
    instruction=SYSTEM_INSTRUCTION,
    tools=[toolset],
)

And the local version:

GEMMA_MODEL = os.environ.get("GEMMA_MODEL", "ai/gemma4:E4B")
MODEL = f"openai/{GEMMA_MODEL}"

root_agent = LlmAgent(
    model=MODEL,
    name="football_stats_agent",
    instruction=SYSTEM_INSTRUCTION,
    tools=[toolset],
)

The openai/ prefix tells ADK to route through LiteLLM (included in google-adk[extensions]) to the OpenAI-compatible endpoint — which is Docker Model Runner. The OPENAI_API_BASE environment variable points LiteLLM to the local endpoint:

export OPENAI_API_BASE=http://localhost:12434/engines/v1
export OPENAI_API_KEY=docker-model-runner  # Dummy key, required by LiteLLM
export GEMMA_MODEL=ai/gemma4:E4B

That's it for the model swap. But there's a catch.

The function calling problem

When I first ran the agent with Gemma, it crashed:

Conversation roles must alternate user/assistant/user/assistant/...

The issue: Gemma doesn't have native function calling support. When ADK sends tool-call messages (system → user → tool_result → assistant), Gemma's chat template rejects the non-alternating roles.

Gemini handles this natively — it understands function declarations and returns structured functionCall responses. Gemma needs a different approach: convert tool declarations into text prompts that the model can understand, then extract function calls from the model's text output.

ADK already has this built-in via GemmaFunctionCallingMixin. It also has a Gemma3Ollama class that combines this mixin with the Ollama provider. But Docker Model Runner uses the OpenAI API (not Ollama's /api/chat), so I needed a custom class.

GemmaModelRunner: bridging Gemma and Docker Model Runner

The solution is a custom LLM class that combines:

LiteLLM's OpenAI provider — routes requests to Docker Model Runner's /engines/v1/chat/completions
ADK's GemmaFunctionCallingMixin — converts tool declarations to text and extracts function calls from responses

from google.adk.models.gemma_llm import GemmaFunctionCallingMixin
from google.adk.models.lite_llm import LiteLlm
from google.adk.models.registry import LLMRegistry


class GemmaModelRunner(GemmaFunctionCallingMixin, LiteLlm):
    """Gemma model served by Docker Model Runner."""

    @classmethod
    def supported_models(cls) -> list[str]:
        return [r"openai/ai/gemma.*"]

    async def generate_content_async(self, llm_request, stream=False):
        self._move_function_calls_into_system_instruction(llm_request)

        async for response in super().generate_content_async(llm_request, stream):
            self._extract_function_calls_from_response(response)
            yield response


LLMRegistry.register(GemmaModelRunner)

This class:

Registers itself in ADK's LLM registry for model names matching openai/ai/gemma.*
Before each LLM call: moves function declarations from the tool config into the system instruction as text, and converts tool-result messages into user messages
After each LLM call: parses the model's text output to extract any function calls (e.g., when Gemma writes execute_sql(projectId="...", query="...") as text)

The import in agent.py triggers the registration:

import football_stats_agent.gemma_model_runner  # noqa: F401 — registers GemmaModelRunner

Dependencies

The cloud version only needs google-adk. The local version needs google-adk[extensions] to include LiteLLM:

[project]
name = "football-agent-adk-gemma"
version = "0.1.0"
requires-python = ">=3.13"
dependencies = [
    "google-adk[extensions]>=1.27.2",
    "google-cloud-aiplatform>=1.141.0",
]

Running the agent locally

With ADK web UI

# Pull the model
docker desktop enable model-runner --tcp 12434
docker model pull ai/gemma4:E4B

# Install dependencies
direnv allow
uv sync

# Authenticate with GCP (for BigQuery MCP)
gcloud auth application-default login

# Run
uv run adk web

Open http://localhost:8000, select football_stats_agent, and ask a question.

With Docker Compose

services:
  adk-agent:
    build:
      context: .
      dockerfile: Dockerfile
    ports:
      - "8080:8080"
    environment:
      - GCP_PROJECT_ID=gb-poc-373711
      - GEMMA_MODEL=ai/gemma4:E4B
      - OPENAI_API_BASE=http://host.docker.internal:12434/engines/v1
      - OPENAI_API_KEY=docker-model-runner
      - GOOGLE_APPLICATION_CREDENTIALS=/tmp/adc.json
    volumes:
      - ${HOME}/.config/gcloud/application_default_credentials.json:/tmp/adc.json:ro

Note host.docker.internal — from inside a container, this is how you reach Docker Model Runner running on the host.

docker compose up --build

The agent in action

I ran three test queries against the E4B agent to see how it handles different difficulty levels:

Simple query — top 3 scorers from France (~22s, one roundtrip)

When you ask "List the top 3 scorers from France", Gemma 4 E4B gets it right on the first try:

Reasons about the question and picks the execute_sql_readonly tool
Generates correct SQL directly using SAFE_CAST (the system instruction warns about string-typed numeric columns)
BigQuery returns results — Kylian Mbappe (8), Olivier Giroud (4), Aurelien Tchouaméni (1)
Formats the answer in natural language

No self-correction needed — E4B follows the system prompt's SAFE_CAST guidance upfront. A smaller model like E2B typically needs a failed attempt before switching to SAFE_CAST.

Complex query — performance rating comparison (~2.5 min, three roundtrips)

When you ask "Compare France and Argentina players by performance rating. Top 3 from each team", things get more interesting:

First attempt — generates a complex window-function SQL, BigQuery rejects it with a syntax error
Second attempt — fixes the syntax but fails on Bad int64 value: "-" (some numeric string columns contain "-")
Third attempt — wraps every cast in SAFE_CAST, query succeeds
Formats the answer with the performance rating formula applied correctly

The self-correction behavior is the same pattern the original Gemini cloud version produced — read the BigQuery error, adjust the SQL, retry. That Gemma 4 E4B handles this loop reliably on a local ~8B-raw-param model is a strong quality signal.

Choosing the right model variant

Hardware used for these benchmarks

All timings below were measured on my personal laptop:

MacBook Pro — Apple M1 Max (10 CPU cores: 8 performance + 2 efficiency)
32-core GPU, Metal 3 backend
32 GB unified memory
macOS 15.3.1

The M1 Max's unified memory architecture is a big part of why Gemma 4 runs comfortably here — the GPU and CPU share the same 32 GB pool, so the model weights don't need to be copied across a PCIe bus. With 32 GB total, E2B (2.94 GiB) and E4B (4.74 GiB) both leave plenty of headroom for Docker Desktop, the ADK runtime, the browser, and the rest of your dev environment. The larger 26B / 31B variants are technically loadable but you'll feel the squeeze.

Performance will vary significantly on other hardware (base Apple Silicon, Intel Macs, Linux with or without a discrete GPU). I haven't benchmarked those, so treat the timings above as one data point on one specific machine.

Timings

Full agent loop end-to-end including BigQuery roundtrips:

Query type	Roundtrips	E4B elapsed
Simple (e.g. "top 3 scorers")	1	~22s
Complex (e.g. "compare two teams by performance rating")	3 (with self-correction)	~2.5 min

Simple queries feel snappy. Complex queries requiring multiple SQL self-corrections take the time you'd expect from a local model making several inference passes.

The E4B variant (now the latest default) is the sweet spot: ~8B raw params running at the compute cost of a 4B model thanks to PLE. It generates correct SQL on the first try for straightforward questions and self-corrects cleanly on complex ones.

If you want maximum speed at the cost of some quality, drop down to E2B — smaller, roughly 2x faster per inference, but more likely to need an extra self-correction round before getting the SQL right.

If you need maximum quality, use 26B / 31B locally (expect multi-minute loops even on a beefy machine) — or switch back to Gemini for production, which is what I recommend:

Environment	Model	Why
Local dev (default)	Gemma 4 E4B (Docker Model Runner)	Free, no API key, good quality
Local dev (fastest)	Gemma 4 E2B (Docker Model Runner)	2x faster, lower quality
Production	Gemini 2.5 Flash (Vertex AI)	Higher quality, faster, scalable

The agent code is the same — only the GEMMA_MODEL environment variable (or the model ID for Gemini) changes.

Docker Model Runner version matters

One issue I hit: an older Docker Model Runner version bundled with my Docker Desktop install cannot load Gemma 4 models at all. Every attempt fails with:

unable to load runner: error waiting for runner to be ready:
inference backend took too long to initialize

This looks like a memory or timeout issue, but it's actually an architecture incompatibility. Gemma 4 uses a newer model architecture that older versions of the llama.cpp backend don't support.

The fix: update Docker Desktop to get a recent Model Runner release. You can check your version with:

docker model version

Key takeaways

Docker Model Runner makes local LLMs trivial — docker model pull and you have an OpenAI-compatible API on localhost. No Ollama, no manual setup.
Gemma doesn't do native function calling — You need ADK's GemmaFunctionCallingMixin to convert tool declarations into text prompts. The custom GemmaModelRunner class bridges this gap for Docker Model Runner's OpenAI API.
The agent code barely changes — Same ADK framework, same BigQuery MCP tools, same system instruction. Only the model string and one custom class change.
Pick the right Gemma 4 variant for local agents — 26B / 31B are too slow for multi-roundtrip loops. E4B (the latest default) is the sweet spot: ~22s for simple queries, ~2.5 min for complex ones requiring SQL self-correction. Drop to E2B only if you need maximum speed. Both use PLE so they punch above their effective size.
Keep Docker Desktop updated — Older Model Runner versions can't load newer model architectures. The error message is misleading — it's not a timeout, it's an incompatibility.
Local for dev, cloud for prod — Run Gemma 4 locally during development (free, private, fast feedback). Switch to Gemini on Vertex AI for production (better quality, scalable). The swap is one environment variable.

What's next

In the next article, I'll rebuild the same use case — football statistics agent on the Qatar 2022 World Cup dataset — but replace Google ADK with Docker Agent. Same Gemma 4 via Docker Model Runner, same BigQuery MCP tools, same system instructions — only the agent framework changes. A direct side-by-side comparison of the two approaches.

Try it yourself

The full source code is available on GitHub. Clone it, pull the model, and ask the agent about the 2022 World Cup:

"Who scored the most goals?"
"Show me the top 5 players by assists"
"Which goalkeepers had the highest save percentage?"
"Compare France and Argentina players by performance rating"

If you enjoyed this article, follow me for more content on AI agents, Google Cloud, Software, DevOps, Tech and data engineering:

Docker Bake in Practice — Part 1: From Bash Scripts to Declarative Builds

Mazlum Tosun — Fri, 11 Sep 2026 22:39:09 +0000

The first of a two-part series. Part 1 covers the fundamentals: what Bake is, why it exists, and the features that make it worth adopting. Part 2 will cover CI/CD integrations — Cloud Build, GitHub Actions, GitLab CI, and Dagger.

Why this article exists

For the past two years I've been giving a talk about Docker Bake at conferences across France and Morocco — DevLille, DevFest Toulouse, DevFest Lyon, Devoxx Morocco, and Cloud Native Days France. Every time I deliver it, the same thing happens: people come up afterwards and say "I had no idea Bake could do that" or "I've been writing 200-line bash scripts to do exactly this."

If you want the video version in French, you can watch the Cloud Native Days France recording. An English video walking through these articles is coming soon on my YouTube channel — subscribe if you'd like to be notified. If you prefer to read at your own pace, with copy-pasteable snippets, this article is for you.

All the code in this article comes from my companion repository — each section links to the relevant file so you can run the examples yourself.

tosun-si / docker-bake-playground

A curated collection of concrete, practical, and reusable examples for mastering Docker Bake.

docker-bake-playground

A curated collection of concrete, practical, and reusable examples for mastering Docker Bake.

Build the images with Docker Bake locally

docker buildx bake -f vars.hcl -f docker-bake-app-and-infra.hcl

Build and publish the images with Docker Bake locally

docker buildx bake -f vars.hcl -f docker-bake-app-and-infra.hcl --push

Printing the Bake file with the --print flag shows the interpolated value in the resolved build configuration.

docker buildx bake -f vars.hcl -f docker-bake-app-and-infra.hcl --print

Build the images for linter and tests

docker buildx bake -f vars.hcl -f docker-bake-lint-and-test.hcl validate

Run linter and tests with Compose

docker compose -f compose_lint_and_test.yaml up

Build the multiples Bake files locally

docker buildx bake -f vars.hcl -f docker-bake-app-and-infra.hcl -f docker-bake-lint-and-test.hcl default validate

Print multiple Bake files

docker buildx bake -f vars.hcl -f docker-bake-app-and-infra.hcl -f docker-bake-lint-and-test.hcl default validate --print

Build Bake file with inheritance

docker buildx bake -f vars.hcl -f docker-bake-inheritance.hcl

Build Bake file with matrix variants

docker buildx bake

…

View on GitHub

The code examples push to Google Cloud Artifact Registry, but Docker Bake is registry-agnostic — every snippet works against Docker Hub, ECR, ACR, or GHCR by changing the target tag.

The foundation: BuildKit, Buildx, and Bake

Before we touch a single HCL file, let's clear up the layering. These three names get thrown around as if they were interchangeable, but they sit on top of each other.

+----------------------------------------------------+
|  Bake                (declarative orchestration)   |
|  docker buildx bake -f file.hcl                    |
+----------------------------------------------------+
|  Buildx              (CLI plugin / frontend)       |
|  docker buildx build ...                           |
+----------------------------------------------------+
|  BuildKit            (build engine / backend)      |
|  parallel stages, cache mounts, multi-platform     |
+----------------------------------------------------+

BuildKit is the modern Docker build engine. It replaced the legacy builder a few years ago and is now the default in Docker Engine 23+. BuildKit is the piece doing the actual work: it parses your Dockerfile, builds stages in parallel where it can, manages the cache, handles multi-platform builds, mount caches, secrets, SSH forwarding, and so on.

Buildx is a Docker CLI plugin that exposes BuildKit's features through a friendlier interface. When you type docker buildx build ..., you're using Buildx as the frontend and BuildKit as the engine. Buildx also manages builders — named BuildKit instances you can swap between (local, remote, container-driven).

Bake is a subcommand of Buildx: docker buildx bake. It takes one or more declarative files (HCL, JSON, or a Compose file) and orchestrates many builds at once. Think of it as docker-compose but for building instead of running. You define your images once, in one place, and Bake builds them all — in parallel, with shared variables, inheritance, matrices, and groups.

That's the whole stack. You won't lose anything by treating BuildKit as "the engine," Buildx as "the CLI," and Bake as "the orchestrator."

A practical note on what you actually have installed: if you've installed Docker Desktop in the last few years, Buildx ships by default — and Buildx bundles BuildKit, so the engine comes with it. Bake is a subcommand of Buildx (docker buildx bake), which means the moment you have Buildx, you have Bake too. On Linux with Docker Engine 23+, Buildx is bundled as a CLI plugin out of the box as well. There's nothing extra to install — unless your Docker is unusually old, you can run every example in this article today.

The traditional approach: bash scripts

To appreciate why Bake exists, let's start with the pain. Imagine you have two images to build and push: an app and an infra. Multi-platform (amd64 + arm64), with provenance and SBOM attestations. Here's the straightforward bash version:

#!/usr/bin/env bash
set -euo pipefail

docker build \
  --platform linux/amd64,linux/arm64 \
  --file app/Dockerfile \
  --tag "${REPO_URL}/app_bake:${IMAGE_TAG_VERSION_APP}" \
  --provenance=true \
  --sbom=true \
  --push \
  .

docker build \
  --platform linux/amd64,linux/arm64 \
  --file infra/Dockerfile \
  --tag "${REPO_URL}/infra_bake:${IMAGE_TAG_VERSION_INFRA}" \
  --provenance=true \
  --sbom=true \
  --push \
  .

This works. But notice four things:

It's sequential. The infra build doesn't start until the app build finishes — even though they're completely independent. Parallelizing in bash means wrestling with &, wait, and trap to propagate exit codes correctly. Most teams don't bother, and ship a pipeline that's slower than it needs to be.
It duplicates everything. The platform list, the attestation flags, the --push — copy-pasted across every image.
It scales linearly. Five images? Five blocks. Ten images? Ten blocks.
It's not portable across operating systems. This is the silent killer in teams that mix Linux, macOS, and Windows laptops. The bash script above looks innocent, but the moment you refactor it into anything non-trivial you start relying on features that don't exist everywhere — and you end up maintaining a different version of the script per OS.

The natural reflex is to refactor into a loop:

declare -A paths=(
  [app]="app/Dockerfile"
  [infra]="infra/Dockerfile"
)

for service in "${!paths[@]}"; do
  upper_service="${service^^}"
  tag_var="IMAGE_TAG_VERSION_${upper_service}"
  tag_version="${!tag_var:-}"

  docker build \
    --platform linux/amd64,linux/arm64 \
    --file "${paths[$service]}" \
    --tag "${REPO_URL}/${service}_bake:${tag_version}" \
    --provenance=true \
    --sbom=true \
    --push \
    .
done

We've removed duplication, but at a cost: this script now uses associative arrays, indirect variable expansion (${!tag_var}), and bash string manipulation (${service^^}). Anyone who hasn't written bash in six months will need a few minutes and an LLM or Stack Overflow tab to read this. And it's still sequential.

The OS-portability problem gets worse here too. declare -A (associative arrays) and ${service^^} (uppercase expansion) require bash 4 or newer. macOS still ships bash 3.2 by default — Apple froze it years ago over the GPL v3 licensing change — so this exact script silently fails on a fresh Mac unless the developer has manually installed a newer bash via Homebrew. Windows developers need WSL or Git Bash. Linux users are fine. The end result is the dreaded "works on my machine" problem, and it lives in a script that's supposed to be the standard way to build images.

This is the moment Bake earns its place.

Enter Bake: same images, declaratively

Here's the same two builds expressed as a Bake file (docker-bake-app-and-infra.hcl):

group "default" {
  targets = ["app", "infra"]
}

target "app" {
  context    = "."
  dockerfile = "images/app/Dockerfile"
  platforms  = ["linux/amd64", "linux/arm64"]
  tags       = ["${REPO_URL}/app_bake:${IMAGE_TAG_VERSION_APP}"]
  attest = [
    "type=provenance,mode=max",
    "type=sbom"
  ]
}

target "infra" {
  context    = "."
  dockerfile = "images/infra/Dockerfile"
  platforms  = ["linux/amd64", "linux/arm64"]
  tags       = ["${REPO_URL}/infra_bake:${IMAGE_TAG_VERSION_INFRA}"]
  attest = [
    "type=provenance,mode=max",
    "type=sbom"
  ]
}

To build both images:

docker buildx bake -f vars.hcl -f docker-bake-app-and-infra.hcl

To build and push:

docker buildx bake -f vars.hcl -f docker-bake-app-and-infra.hcl --push

Four things to notice:

The intent is obvious. Anyone who reads HCL or Terraform feels at home immediately. No ${!var} tricks, no array juggling.
Parallel by default. Bake schedules the app and infra builds concurrently, and BuildKit shares cache layers between them. No &, no wait, no exit-code plumbing — parallelism is the default behavior, not a feature you opt into. Every target declared in an HCL file is fair game for parallel scheduling. Bash, by contrast, runs a for loop sequentially unless you go out of your way to fan it out.
One file, every OS. The same .hcl file produces the same builds on Linux, macOS, and Windows — Docker Desktop and Docker Engine normalize the invocation. No bash-version constraints, no per-OS scripts, no Homebrew gymnastics. New team members can clone the repo and docker buildx bake regardless of what laptop they're on. This is the standardization point that bash scripts will never give you, no matter how clean you write them.
The group "default" block tells Bake what to build when you don't pass a target. You can have multiple groups for different purposes (release builds, dev builds, validation, etc.).

You can also inspect the resolved configuration before building anything:

docker buildx bake -f vars.hcl -f docker-bake-app-and-infra.hcl --print

This prints the fully-interpolated JSON. Invaluable when something behaves unexpectedly.

And if you just want to validate the file — check the HCL syntax, the target definitions, and that referenced variables resolve — without building anything, there's --check:

docker buildx bake -f vars.hcl -f docker-bake-app-and-infra.hcl --check

I run this in pre-commit hooks and as the first step of every CI pipeline. It catches typos, missing variables, and broken inherits references in under a second, before any expensive build kicks off.

Variables and validators

You probably noticed ${REPO_URL} and ${IMAGE_TAG_VERSION_APP} in the file above. Those come from a separate file, vars.hcl, which I always pass first:

variable "PROJECT_ID" {
  validation {
    condition     = PROJECT_ID != ""
    error_message = "The variable 'PROJECT_ID' must not be empty."
  }
}

variable "LOCATION" {
  default = "europe-west1"
}

variable "REPO_NAME" {
  validation {
    condition     = REPO_NAME != ""
    error_message = "The variable 'REPO_NAME' must not be empty."
  }
}

variable "IMAGE_TAG_VERSION_APP" {
  validation {
    condition     = IMAGE_TAG_VERSION_APP != ""
    error_message = "The variable 'IMAGE_TAG_VERSION_APP' must not be empty."
  }
}

variable "REPO_URL" {
  default = "${LOCATION}-docker.pkg.dev/${PROJECT_ID}/${REPO_NAME}"
}

Two things make this worth highlighting:

Defaults and composition. LOCATION defaults to europe-west1. REPO_URL is composed from three other variables, so I never type the full Artifact Registry URL anywhere else.

Validators. Bake supports validation blocks on variables. If PROJECT_ID is missing or empty at build time, the build fails immediately with a clear message — before a single layer gets built. This is the kind of thing that, in bash, becomes a forest of if [[ -z "${X:-}" ]]; then echo "..." && exit 1 blocks at the top of every script. Here, it's declarative and lives next to the variable definition.

Variables are populated from the environment, so the usage pattern is just:

export PROJECT_ID="my-gcp-project"
export REPO_NAME="internal-images"
export IMAGE_TAG_VERSION_APP="0.1.0"
export IMAGE_TAG_VERSION_INFRA="0.1.0"

docker buildx bake -f vars.hcl -f docker-bake-app-and-infra.hcl

Inheritance: the `_common` pattern

Look back at the app and infra targets. They share context, platforms, and attest. Bake supports inheritance, so you can factor that out (docker-bake-inheritance.hcl):

group "default" {
  targets = ["app", "infra"]
}

target "_common" {
  context   = "."
  platforms = ["linux/amd64", "linux/arm64"]
  attest = [
    "type=provenance,mode=max",
    "type=sbom"
  ]
}

target "app" {
  inherits   = ["_common"]
  dockerfile = "images/app/Dockerfile"
  tags       = ["${REPO_URL}/app_bake:${IMAGE_TAG_VERSION_APP}"]
}

target "infra" {
  inherits   = ["_common"]
  dockerfile = "images/infra/Dockerfile"
  tags       = ["${REPO_URL}/infra_bake:${IMAGE_TAG_VERSION_INFRA}"]
}

By convention I prefix the inherited target with an underscore (_common) to signal "this is a base, don't build it directly." Bake doesn't enforce this — it's just a hint to readers.

A real-world example: Python lint and test images with `uv`

Enough abstract examples. Here's something concrete: a Python application I want to lint with Ruff and test with pytest, both run inside Docker so the CI environment matches local exactly.

I use uv as the Python package manager because it's an order of magnitude faster than pip or Poetry. The lint image looks like this (images/python_linter/Dockerfile):

FROM ghcr.io/astral-sh/uv:python3.11-alpine AS builder

ENV UV_COMPILE_BYTECODE=1 \
    UV_LINK_MODE=copy \
    WORKDIR=/usr/local/src/app

WORKDIR $WORKDIR

COPY pyproject.toml uv.lock ./

RUN --mount=type=cache,target=/root/.cache/uv \
    uv sync --locked

COPY python_app $WORKDIR/python_app

FROM python:3.11-alpine

ENV WORKDIR=/usr/local/src/app
WORKDIR $WORKDIR

COPY --from=builder $WORKDIR $WORKDIR

ENV PATH="$WORKDIR/.venv/bin:$PATH"

ENTRYPOINT ["ruff"]
CMD ["check", "python_app", "--exclude", "tests", "--output-format=concise", "--color=always"]

Two patterns worth highlighting:

Multi-stage build. The builder stage uses the ghcr.io/astral-sh/uv image, which already ships with uv installed. It resolves dependencies and creates a .venv. The final stage is plain python:3.11-alpine — no uv binary, no build tools, just the runtime and the venv. The result is a smaller, cleaner runtime image.

--mount=type=cache for uv. The line RUN --mount=type=cache,target=/root/.cache/uv uv sync --locked is a BuildKit feature: it gives the uv sync step a persistent cache directory that survives across builds. The first build downloads every wheel; every subsequent build reuses them. On a typical project this drops dependency-install time from ~30s to under 2s. It's the single biggest speedup you can add to a Python Dockerfile.

Layer order matters. Look closely at the order of operations in the builder stage: we COPY pyproject.toml uv.lock first, run uv sync second, and only then COPY python_app. This is not cosmetic — it's the single most important rule for keeping Docker's layer cache useful. Docker invalidates a layer (and every layer after it) the moment any of its inputs change. If we copied the whole source tree before uv sync, every change to a Python file — even a one-character typo fix — would invalidate the dependency-install layer and trigger a full reinstall of every package. By copying only the dependency manifest first, the expensive uv sync layer stays cached as long as pyproject.toml and uv.lock are untouched. Code changes only invalidate the cheap final COPY. This rule is surprisingly easy to get wrong, and getting it wrong is what turns a 30-second incremental build into a 4-minute one.

The Bake file that ties the lint and test images together is small (docker-bake-lint-and-test.hcl):

group "validate" {
  targets = ["lint", "test"]
}

target "lint" {
  context    = "."
  dockerfile = "images/python_linter/Dockerfile"
  tags       = ["${REPO_URL}/python-linter:latest"]
}

target "test" {
  context    = "."
  dockerfile = "images/python_tests/Dockerfile"
  tags       = ["${REPO_URL}/python-tests:latest"]
}

Note the group name is validate, not default. You build it explicitly:

docker buildx bake -f vars.hcl -f docker-bake-lint-and-test.hcl validate

Composing multiple Bake files in one command

In a real project you don't have one Bake file — you have several, each grouped by purpose: app images, infra images, lint/test, scanning, base images. Bake lets you pass any number of -f flags and any number of targets:

docker buildx bake \
  -f vars.hcl \
  -f docker-bake-app-and-infra.hcl \
  -f docker-bake-lint-and-test.hcl \
  default validate

This one command builds four images — app, infra, lint, test — all in parallel. BuildKit deduplicates shared layers across them, so the Python base layer (for example) is fetched once.

Note the trailing default validate: when composing multiple Bake files, you have to name the groups (or individual targets) to build — default from the app/infra file, validate from the lint/test file. Without them, Bake only runs the default group and the lint/test images stay unbuilt.

Compare that to the bash version, where parallelizing four docker build calls means wrestling with &, wait, and trap to propagate exit codes correctly. With Bake, parallelism is free and correct by default.

Bake from a remote GitHub repository

This one surprises a lot of people: Bake can pull configuration directly from a remote Git repository. No git clone needed.

docker buildx bake \
  -f bake.hcl \
  "https://github.com/crazy-max/buildx.git#remote-with-local" \
  --print

The URL acts as both the build context and the source of the Bake file. The #remote-with-local part is a Git ref (branch, tag, or commit).

Even better, you can combine a remote Bake file with a local one using the cwd:// prefix:

docker buildx bake \
  -f bake.hcl \
  -f cwd://local.hcl \
  "https://github.com/crazy-max/buildx.git#remote-with-local" \
  --print

The remote file provides the shared definitions; your local file overrides values for your environment. This is a clean pattern for sharing reusable Bake configurations across teams without making everyone vendor the same HCL into every repo.

Matrix: building variants of the same image

Here's a use case that comes up constantly in real projects — and the canonical example is the official PostgreSQL image on Docker Hub, which ships the same database across multiple base-image variants (Alpine, Bullseye, Bookworm) and multiple Postgres versions for each. That's exactly the pattern I want to demonstrate: you have one application and you need to ship it as many images — different OS bases, different versions per OS. Alpine for size-conscious users, Bullseye and Bookworm for compatibility with glibc-dependent libraries.

A quick guard rail before we go further: this pattern applies when you're shipping something downstream consumers will pick a base for — a database, a runtime, a developer tool, a library. Postgres, Python, Node, Ruby, MySQL — every major official image does this. If you're shipping a business application, you typically pick one base and stay there. Don't take the example below and fan your billing service out across five Alpine and Debian variants; you'll just multiply your image count, your scan surface, and your registry bill for no benefit. The point of matrix-variants is to serve multiple downstream audiences from one source, not to spray the same app across every base image you can think of.

In bash, that's a nested loop (scripts/build_matrix_variants.sh):

variants=("alpine" "bullseye" "bookworm")
declare -A versions
versions[alpine]="3.17 3.21"
versions[bullseye]="11.7 11.8"
versions[bookworm]="12.2 12.5"

for variant in "${variants[@]}"; do
  for version in ${versions[$variant]}; do
    dir="images/app-matrix/${variant}-${version//./-}"
    tag="myapp:${variant}-${version}"

    docker buildx build \
      --file "$dir/Dockerfile" \
      --tag "$tag" \
      --platform "linux/amd64" \
      --push \
      "$dir"
  done
done

It works. It's also six images built strictly one after another, and the syntax is hard to read and maintain. Every image you add — and every bit of conditional logic on top — makes it worse.

Here's the same thing in Bake (docker-bake-matrix-variants.hcl):

group "default" {
  targets = ["alpine_apps", "bullseye_apps", "bookworm_apps"]
}

target "_common" {
  context = "."
}

target "alpine_apps" {
  inherits = ["_common"]
  name     = "app-${variant}-${replace(version, ".", "-")}"
  matrix = {
    variant = ["alpine"]
    version = ["3.17", "3.21"]
  }
  dockerfile = "images/app-matrix/${variant}-${replace(version, ".", "-")}/Dockerfile"
  tags       = ["myapp:${variant}-${version}"]
}

target "bullseye_apps" {
  inherits = ["_common"]
  name     = "app-${variant}-${replace(version, ".", "-")}"
  matrix = {
    variant = ["bullseye"]
    version = ["11.7", "11.8"]
  }
  dockerfile = "images/app-matrix/${variant}-${replace(version, ".", "-")}/Dockerfile"
  tags       = ["myapp:${variant}-${version}"]
}

target "bookworm_apps" {
  inherits = ["_common"]
  name     = "app-${variant}-${replace(version, ".", "-")}"
  matrix = {
    variant = ["bookworm"]
    version = ["12.2", "12.5"]
  }
  dockerfile = "images/app-matrix/${variant}-${replace(version, ".", "-")}/Dockerfile"
  tags       = ["myapp:${variant}-${version}"]
}

Each matrix block expands into one target per combination of values. With variant = ["alpine"] and version = ["3.17", "3.21"], Bake produces two targets: app-alpine-3-17 and app-alpine-3-21. The name template controls the generated names, and ${variant} and ${version} are available everywhere in the target.

Run it:

docker buildx bake -f vars.hcl -f docker-bake-matrix-variants.hcl

All six images build in parallel. No nested loops, no exit-code propagation, no string substitution gymnastics.

Matrix with a list of items

The matrix above uses cartesian-product expansion (every variant × every version). Sometimes you want something different: a flat list of heterogeneous items, each with their own properties. Bake supports that too (docker-bake-matrix-items.hcl):

target "app" {
  name = "app-${item.tgt}-${replace(item.version, ".", "-")}"
  matrix = {
    item = [
      {
        tgt     = "lint"
        version = "1.0"
        ctx     = "."
        dockerf = "images/python_linter/Dockerfile"
        tag     = "${REPO_URL}/python-linter-matrix:latest"
      },
      {
        tgt     = "test"
        version = "2.0"
        ctx     = "."
        dockerf = "images/python_tests/Dockerfile"
        tag     = "${REPO_URL}/python-tests-matrix:latest"
      }
    ]
  }
  context    = item.ctx
  dockerfile = item.dockerf
  tags       = [item.tag]
}

Each entry in the item list is an object with its own fields. Inside the target, you reference them as item.tgt, item.dockerf, etc. This pattern is great when each image has its own Dockerfile path, its own tag, and its own context — but you want to define them all in one place rather than write a target per image.

Run it:

docker buildx bake -f vars.hcl -f docker-bake-matrix-items.hcl app

Bake from a Compose file: one source of truth

Most projects already have a compose.yaml that describes how services run together. Bake can read that file directly and use it as build input (compose_bake_example.yaml):

services:
  app:
    image: app-bake-example:latest
    build:
      context: .
      dockerfile: images/app/Dockerfile
      args:
        PROJECT_ID: gb-poc-373711
        LOCATION: europe-west1
        REPO_NAME: internal-images
        IMAGE_TAG_VERSION_APP: 0.1.0
      x-bake:
        tags:
          - ${LOCATION}-docker.pkg.dev/${PROJECT_ID}/${REPO_NAME}/app_bake:${IMAGE_TAG_VERSION_APP}
        platforms:
          - linux/amd64
          - linux/arm64

  infra:
    image: infra-bake-example:latest
    build:
      dockerfile: images/infra/Dockerfile
      x-bake:
        tags:
          - ${LOCATION}-docker.pkg.dev/${PROJECT_ID}/${REPO_NAME}/infra_bake:${IMAGE_TAG_VERSION_INFRA}
        platforms:
          - linux/amd64
          - linux/arm64

The build section is standard Compose. The x-bake extension is what Bake reads to enrich the build with multi-platform targets, tags, attestations, and other Bake-specific fields that Compose doesn't natively understand.

Build everything with:

docker buildx bake -f compose_bake_example.yaml

Run the same services with:

docker compose -f compose_bake_example.yaml up

Same file, both lifecycles. Build configuration and runtime configuration live in one place. For projects where the dev loop is "build, then run, then build again," this is a meaningful ergonomic win.

Wrapping up Part 1

We've covered a lot of ground:

The BuildKit / Buildx / Bake stack and what each layer does.
The pain of bash scripts for multi-image builds, and the two big things Bake gives you that bash never will: parallel builds by default (no &/wait gymnastics) and a single file that works identically on Linux, macOS, and Windows.
HCL targets, groups, variables with validators, and inheritance.
A real Python multi-stage build with uv and --mount=type=cache.
Composing multiple Bake files in a single command.
Pulling Bake files from a remote Git repository and combining them with local overrides.
Matrix builds — both cartesian-product (variants × versions) and item lists.
Using a Compose file as Bake input for a single source of truth.

If you've been writing bash scripts to coordinate docker build calls, you now have everything you need to throw them away.

Coming in Part 2

Part 2 will focus on Bake in CI/CD, which is where the parallelism, declarative configuration, and registry-cache features compound:

Google Cloud Build — running Bake with registry cache against Artifact Registry.
GitHub Actions — using the native Buildx action with GitHub Actions cache.
GitLab CI — pipeline stages with Bake on GitLab.com runners.
Dagger — a portable, programmable pipeline (with the Python SDK) that runs the same way locally and in the cloud. Build → Trivy scan → push, with the whole pipeline expressed as code.
The GreenOps angle: how registry-based Docker cache cuts CI compute time and energy, and how to wire it up correctly across all four tools.

Talks and resources

Talk recording (French) — Docker Bake at DevLille
Talk recording (French) — Docker Bake at DevFest Toulouse
Talk recording (French) — Docker Bake at Cloud Native Days France

English video version (coming soon) — my YouTube channel
Talk venues — DevLille, DevFest Toulouse, DevFest Lyon, Devoxx Morocco, Cloud Native Days France
Companion repository — docker-bake-playground
Official Bake documentation — docs.docker.com/build/bake

If you have feedback, questions, or a use case you'd like to see covered in Part 2, reach out on LinkedIn or open an issue on the repo.

If you enjoyed this article, follow me for more content on Docker, AI agents, Google Cloud, Software, Devops, Tech and data engineering:

Stop rebuilding from scratch: cache Docker layers on Cloud Build

Mazlum Tosun — Mon, 07 Sep 2026 22:31:01 +0000

How BuildKit + Artifact Registry turn your CI/CD pipelines around: if nothing changed, uv doesn't re-download a single package.

📦 All the code in this article is on GitHub: tosun-si/docker-cloud-build-ci-cd-cache

How I got here

Early in my career, when I built CI pipelines, I didn't have much of an
optimization or GreenOps sensitivity. Pipelines ran, images got built, and I
never really questioned the wasted work.

That changed after a side collaboration with my friend Guillaume Leroy a while back. It got me
paying attention to build efficiency — and once I started, the benefits were
obvious: my pipelines were faster, leaner, and I wasn't burning machines for
nothing. That last part matters: an optimized pipeline is also a greener one.

I became a convert. Today I apply these techniques everywhere — in my personal
projects, and as a platform engineer at my clients'. The developers love it
(CI/CD pipelines are dramatically faster), and the clients care too: carbon
footprint and FinOps are real topics for them. So I bring this discipline to
every CI/CD tool I work with.

And that's where Cloud Build surprised me. Almost nobody around me — including
in the GDE and Google community — persists the Docker cache on Cloud Build,
Google's serverless CI/CD tool. When I dug into why, the official guidance
explained it: Google's own docs point you to --cache-from <previous image>
(which, as we'll see, silently misses your multi-stage builder layers) or to
Kaniko — a tool Google itself archived on June 3, 2025 ("this project is
archived and no longer developed or maintained"). Neither path mentions
BuildKit's registry cache with mode=max, the one that actually persists the
expensive dependency layer. A handful of scattered blog posts cover it; the
official guidance still doesn't.

Everything ships as a container these days, so this is a lever almost every team
leaves on the table. It saves me real time every day, on my own projects and
with the teams I work with — and I wanted to write this up to share it with the
community.

One more reason it stays a blind spot: the default behavior genuinely makes it
feel like "Docker caching just doesn't work on Cloud Build." Every build starts
from scratch, re-downloads every dependency, rebuilds every layer. On a Python
app with a few dozen packages, that's 30–60 seconds wasted on every push,
even though nothing changed on the dependency side. The good news: it's not a
fatality. With BuildKit and a cache stored in Artifact Registry, you get a real
persistent cache shared across pipelines — and once you understand why the
default fails, the fix is three lines.

This is the first article in a series. Here we start from the classic
docker build (via docker buildx). In a second article, I'll show the same
mechanism with Docker Bake, which is just a declarative layer on top of the
same cache engine.

Why the cache "disappears" on Cloud Build

Locally, Docker's layer cache feels like magic: you rebuild, and Docker reuses unchanged layers straight from the local daemon. The cache storage is your machine.

Cloud Build runs on ephemeral workers. Every build starts on a fresh VM, with no state from previous builds. The VM's Docker daemon is empty. As a result:

There is no local cache to reuse, because there is no "local" that persists.

This isn't a bug, it's the model: isolation and reproducibility. But it means a bare docker build on Cloud Build will never cache anything between runs.

So the solution isn't to keep a local cache — that's impossible — it's to externalize the cache into a registry that every build shares. And you already have that registry: Artifact Registry.

Quick refresher: Docker's layer cache

Each instruction in a Dockerfile produces a layer. Docker reuses a layer as long as:

the instruction is identical, and
its input context (copied files, parent layer) is identical.

The moment a layer is invalidated, every layer below it is too. Hence the golden rule:

Copy what changes rarely first (the dependencies), what changes often next (the source code).

That's exactly what makes the Python + uv example so telling: resolving and installing dependencies is a heavy layer, but a stable one. As long as pyproject.toml and uv.lock don't move, we should never rebuild it.

The example app: FastAPI + uv

A minimal app, just enough to have real dependencies (FastAPI + uvicorn and their transitive tree — 21 packages).

pyproject.toml:

[project]
name = "cloud-build-cache-demo"
version = "0.1.0"
requires-python = ">=3.13"
dependencies = [
    "fastapi>=0.115.0",
    "uvicorn[standard]>=0.32.0",
]

# No [build-system]: this is an application, not a reusable package.

Notice there is no [build-system]. This is deliberate: it makes the app a virtual uv project — uv resolves and installs the dependencies, but never tries to build or install the app itself. The code just runs from source. That keeps the Docker build trivial and the dependency layer perfectly cacheable.

app/main.py:

from fastapi import FastAPI

app = FastAPI(title="Cloud Build cache demo")


@app.get("/")
def root() -> dict[str, str]:
    return {"message": "Hello from a cache-friendly Cloud Build pipeline"}


@app.get("/health")
def health() -> dict[str, str]:
    return {"status": "ok"}

The uv.lock is generated once with uv lock and committed — that's what makes the build reproducible and the cache deterministic.

Why a flat app/ folder at the repo root, not a src/ layout? The src layout earns its keep for a reusable package: it stops Python from importing your working tree instead of the installed wheel, and forces tests to run against what you actually ship. But this is a deployed application, not a library — we don't package it into a wheel at all. A root-level folder named after the app is simpler and reads better, and since uv treats it as a virtual project (no [build-system]), there's nothing to install and nothing to shadow.

The Dockerfile: multi-stage and cache-friendly

Two stages: a builder based on the uv image (which ships uv + the right CPython), and a runtime python:slim image without uv, running non-root.

# syntax=docker/dockerfile:1.7

ARG APP_DIR=/usr/local/src/app

# ---------- builder ----------
FROM ghcr.io/astral-sh/uv:python3.13-bookworm-slim AS builder
ARG APP_DIR
ENV APP_DIR=${APP_DIR}
WORKDIR ${APP_DIR}

ENV UV_COMPILE_BYTECODE=1 \
    UV_LINK_MODE=copy \
    UV_PYTHON_DOWNLOADS=0

# Dependencies only — a virtual uv project installs the deps but never the
# app itself. This layer is reused as long as pyproject.toml + uv.lock don't
# change: THIS is the layer we want to survive across Cloud Build runs.
COPY pyproject.toml uv.lock ./
RUN --mount=type=cache,target=/root/.cache/uv \
    uv sync --frozen --no-dev

# ---------- runtime (no uv) ----------
FROM python:3.13-slim-bookworm AS runtime
ARG APP_DIR=/usr/local/src/app
ENV APP_DIR=${APP_DIR} \
    PATH="${APP_DIR}/.venv/bin:${PATH}" \
    PYTHONPATH="${APP_DIR}" \
    PYTHONUNBUFFERED=1 \
    PYTHONDONTWRITEBYTECODE=1

WORKDIR ${APP_DIR}

RUN groupadd --system app \
 && useradd --system --gid app --home-dir ${APP_DIR} --shell /usr/sbin/nologin app \
 && chown app:app ${APP_DIR}

# The venv (deps) from the builder, then the app source. Copying the code last
# means a code change never invalidates the dependency layer above.
COPY --from=builder --chown=app:app ${APP_DIR}/.venv ${APP_DIR}/.venv
COPY --chown=app:app app ./app

USER app
EXPOSE 8000

ENTRYPOINT ["uvicorn"]
CMD ["app.main:app", "--host", "0.0.0.0", "--port", "8000"]

Two details do all the work:

The heavy uv sync layer is built from pyproject.toml + uv.lock alone, and the app source is copied after it (into the runtime stage). A code change never touches the dependency layer.
--mount=type=cache on /root/.cache/uv. uv's download cache, on top of the layer cache. We'll come back to it: on Cloud Build, that layer needs mode=max to survive.

Plus two hygiene points straight from the conventions: non-root runtime and a final image without uv (we only copy the produced .venv).

The tempting-but-wrong fix: `--cache-from` on the previous image

The trick most people reach for first — and, notably, the one Google's own
Best practices for speeding up builds
recommends:

# pull the previous image, use it as cache
docker pull "$IMAGE:latest" || true
docker build --cache-from "$IMAGE:latest" -t "$IMAGE:latest" .

It seems logical, but it's disappointing with multi-stage builds, for two reasons:

Only the final-stage layers live in the image. The builder stage layers (where uv sync runs!) aren't there. So uv sync re-runs on every build.
Without BuildKit and its "inline" cache, layer matching is brittle.

In other words: the expensive layer — installing the dependencies — is precisely the one this cache doesn't recover. That's where the "Docker caching is useless on Cloud Build" belief comes from. The docs' other suggestion, Kaniko cache, does handle intermediate layers — but Google archived Kaniko in June 2025, so building your pipeline on it today means adopting an unmaintained tool. Which leaves the approach the docs don't mention.

The real fix: BuildKit's registry cache

Let's put the two versions side by side. Same docker buildx build, two lines of difference.

Without cache — the baseline. Every run on a fresh worker rebuilds everything, uv sync included:

docker buildx build \
  --push -t "$IMAGE" .

With cache — BuildKit exports the entire build graph (including the intermediate layers of every stage) to a dedicated cache image, and imports it on the next run:

docker buildx build \
  --cache-from "type=registry,ref=$CACHE" \
  --cache-to   "type=registry,ref=$CACHE,mode=max" \
  --push -t "$IMAGE" .

That's it — two flags:

--cache-to type=registry,mode=max pushes all layers (final and intermediate) into $CACHE. It's mode=max that changes everything — mode=min (the default) would only export the final-stage layers, and we'd fall right back into the previous trap.
--cache-from type=registry: on the next build, BuildKit imports those layers from the registry before building. The uv sync layer is found by its hash → reused, nothing re-downloaded.

The cache is stored as an ordinary image, tagged :buildcache by convention in the same Artifact Registry repo as the app image. That registry is shared by every Cloud Build worker: that's the "inter-pipeline" cache.

mode=max is the keystone. It's the one setting that makes the dependency layer survive between builds on ephemeral workers. Without it, the whole exercise is pointless.

Here's the whole flow at a glance — build #1 exports the cache, build #2 (a fresh, unrelated worker) imports it and skips uv sync entirely:

The Cloud Build config: `build-python-app.cloudbuild.yaml`

We're focusing on the cache, so this config does one thing: build + push with
the registry cache. (Deploying the image — to Cloud Run or anywhere else — is a
separate concern; more on that at the end.)

steps:
  # Build + push with a container-driver Buildx builder.
  # The container driver is REQUIRED to export/import cache to a registry —
  # the default "docker" driver cannot do type=registry cache.
  - name: 'gcr.io/cloud-builders/docker'
    id: build-push
    env:
      - 'IMAGE_PATH=${_REGION}-docker.pkg.dev/$PROJECT_ID/${_REPO}/${_IMAGE}'
      - 'GIT_SHA=${SHORT_SHA}'
    script: |
      #!/usr/bin/env bash
      set -euo pipefail

      TAG="${GIT_SHA:-manual}"
      CACHE="${IMAGE_PATH}:buildcache"

      docker buildx create --name cloudbuilder --driver docker-container --use

      docker buildx build \
        --tag "${IMAGE_PATH}:${TAG}" \
        --tag "${IMAGE_PATH}:latest" \
        --cache-from "type=registry,ref=${CACHE}" \
        --cache-to   "type=registry,ref=${CACHE},mode=max" \
        --push \
        .

substitutions:
  _REGION: europe-west1
  _REPO: internal-images
  _IMAGE: cloud-build-cache-demo

options:
  logging: CLOUD_LOGGING_ONLY

Three things worth calling out:

docker buildx create --driver docker-container is the non-obvious bit. Cloud Build's default docker driver does not support type=registry cache. So we spin up a containerized BuildKit builder for the duration of the build. This one detail is what 90% of attempts are missing.
options.logging: CLOUD_LOGGING_ONLY isn't cosmetic. As soon as you run under a custom service account (impersonation / WIF), Cloud Build requires you to pick a log destination or the build fails with you must specify logging. CLOUD_LOGGING_ONLY sends logs to Cloud Logging only — no GCS bucket to manage. It's the recommended default today.
No machineType — the default (e2-standard-2) is plenty for a small Python build. Bump to E2_HIGHCPU_8 only for CPU-bound or multi-target builds; it's faster but billed at a higher per-minute rate. It has no effect on caching.

Note: on the very first build, --cache-from on a non-existent cache is a harmless warning — there's simply nothing to import yet.

GCP prerequisites

A Docker Artifact Registry repo — reuse an existing one (here it's internal-images) or create it:

gcloud artifacts repositories create internal-images \
  --repository-format=docker \
  --location=europe-west1

The app image and the :buildcache image both live in this repo, so a single repo is all you need.

IAM for the Cloud Build service account (the default <PROJECT_NUMBER>-compute@developer.gserviceaccount.com, or your dedicated SA) — it needs to push both the app image and the cache image, which live in the same repo, so a single binding covers it:

gcloud artifacts repositories add-iam-policy-binding internal-images \
  --location=europe-west1 \
  --member="serviceAccount:<CLOUD_BUILD_SA>" \
  --role="roles/artifactregistry.writer"

Wire the config to a push trigger on your main branch — $SHORT_SHA is then populated automatically on every merge, and the cache does its job silently from the second build onward. No manual invocation needed.

The demo: before / after

Build #1 (cold cache) — nothing in :buildcache. BuildKit resolves and installs the 21 packages, then pushes every layer to the cache. This is the slowest build.

Build #2 (change main.py only) — pyproject.toml/uv.lock unchanged:

BuildKit imports the cache from Artifact Registry;
the COPY pyproject.toml uv.lock layer → CACHED;
the uv sync layer → CACHED (zero packages downloaded);
only the layers from COPY app down are rebuilt.

In the Cloud Build logs you'll see it explicitly:

 => CACHED [builder 4/5] COPY pyproject.toml uv.lock ./
 => CACHED [builder 5/5] RUN uv sync --frozen --no-dev
 => [runtime 6/7] COPY app ./app

That CACHED on uv sync, on a brand-new VM that has never seen this project, is the whole point of this article. The layer comes from Artifact Registry, not from a local disk.

Build #3 (change a dependency) — now uv.lock changes, the layer is correctly invalidated and re-downloaded. That's the right behavior: the cache tracks exactly what actually changed.

Measured impact

Numbers make the case. Here's the same pipeline across the three scenarios,
one real Cloud Build run each (default machine type, region
europe-west1, build-only — no deploy step):

Scenario	Total build	`uv sync` layer	Packages installed
#1 — cold cache (first build)	51 s	rebuilt (2.6 s)	19
#2 — code change only (`main.py`)	36 s	CACHED	0
#3 — dependency bump (`+httpx`)	55 s	rebuilt (2.6 s)	22

The proof is right there in the Cloud Build logs — the exact same uv sync
step, on three fresh, unrelated workers:

# Build #1 (cold)        →  #16 [builder 4/4] RUN ... uv sync --frozen --no-dev
                            #16 DONE 2.6s

# Build #2 (code change) →  #14 [builder 4/4] RUN ... uv sync --frozen --no-dev
                            #14 CACHED            ← pulled from Artifact Registry

# Build #3 (dep bump)    →  #15 [builder 4/4] RUN ... uv sync --frozen --no-dev
                            #15 DONE 2.6s         ← uv.lock changed, correctly re-run

That #14 CACHED, on a worker that had never seen this project, is the whole
point: the dependency layer was rebuilt on build #1, exported to Artifact
Registry, and imported on build #2 — zero packages downloaded, uv sync
skipped entirely.

Read it honestly. On this deliberately tiny app, uv sync is only ~2.6 s
and 19 pure-Python wheels, so the total delta (51 s → 36 s) is dominated by
things the cache doesn't remove on an ephemeral worker: base-image pulls,
BuildKit startup, cache import/export I/O. The headline isn't "15 seconds
saved" — it's 0 packages installed and the uv sync layer fully skipped.
On a real service (dozens of deps, compiled wheels, a numpy/pyarrow in the
tree), that skipped layer is minutes, not seconds — and it's skipped on every
push where uv.lock hasn't moved.

Want to see it on video? The repo ships a heavy dependency group
(pandas + numpy, compiled wheels) that's off by default. Flip it on
(--build-arg INSTALL_HEAVY=1, or _INSTALL_HEAVY=1 on Cloud Build) and the
uv sync step jumps from ~3 s to minutes — so the cold-vs-CACHED contrast
is impossible to miss on screen. Same cache mechanism, just a louder signal.

Reproduce it yourself with three runs:

Run #1 — cold cache. Delete the :buildcache tag first (gcloud artifacts docker images delete .../<image> --delete-tags), then build. Baseline.
Run #2 — code only. Change a string in app/main.py, rebuild. uv.lock untouched → uv sync shows CACHED.
Run #3 — dependency bump. uv add <pkg>, rebuild. uv.lock changed → uv sync correctly re-runs.

Pull the durations and the per-step cache hits straight from the API:

gcloud builds list --limit=3 --format='table(id, duration, status)'
gcloud builds log <BUILD_ID> | grep -E 'CACHED|uv sync|DONE'

Gotchas worth knowing

docker-container driver is mandatory. Without it, --cache-to type=registry is ignored or fails silently.
mode=max or nothing. In mode=min, the builder stage's intermediate layers aren't exported → uv sync re-runs every time.
The :buildcache tag grows. Each build stacks layers onto it. Set an Artifact Registry cleanup policy, or rewrite the tag periodically. The cache is a convenience, not a source of truth.
Cache network cost. Pushing/pulling the cache has an I/O cost. On a small image it's a net win from build #2; on huge images, measure — sometimes mode=min on select stages is the better trade.

The angle nobody talks about: this is GreenOps

We frame CI caching as a speed win. It's also a sustainability one, and that's rarely said out loud.

Every uncached build re-runs uv sync: CPU cycles to resolve the graph, network to pull wheels, CPU again to unpack and byte-compile them. Multiply that by the number of builds a team ships per day — every push, every PR, every retry — across every developer. A 30-second dependency step that runs 50 times a day is 25 minutes of pure CPU burn, daily, producing an artifact bit-for-bit identical to the previous one.

Caching turns most of those runs into a near-instant registry pull. Concretely, that's:

Less compute → fewer CPU-seconds → less energy drawn in the datacenter → a smaller carbon footprint. This is textbook GreenOps: don't recompute what hasn't changed.
Less money → Cloud Build is billed by the build-minute, so the green win is also a FinOps win. The two point the same way.
Faster feedback → shorter pipelines, less waiting, less context-switching for the whole team.

The honest caveat: the cache isn't free. Pushing and pulling it has its own I/O, storage, and (small) compute footprint. It's net-green when the compute you avoid is larger than the cache transfer you add — which is exactly the case for dependency-heavy builds like this one, and the reason ordering the Dockerfile well matters so much. For a trivial image with no real dependency layer, the math can flip; measure before assuming.

The takeaway: CI optimization isn't just about developer experience (DX). Recomputing an identical artifact on every push is waste — of time, of money, and of energy. A well-placed cache removes all three at once.

What about deploying?

You may have noticed this config stops at "push to Artifact Registry" — no
gcloud run deploy, no rollout. That's on purpose. Deployment is its own topic
(runtime service accounts, IAM, IAP, traffic splitting…) and folding it in here
would only blur the one thing this article is about: the cache. Once the image
is in Artifact Registry, adding a deploy step — Cloud Run, GKE, wherever — is a
few extra lines, and I'll cover that shape in a dedicated article.

What's next: Docker Bake

This config works, but the --cache-from / --cache-to flags get verbose fast once you have several images (API, worker, front…). That's exactly the problem Docker Bake solves: you declare targets, tags and cache in a docker-bake.hcl, and a single docker buildx bake --push orchestrates all of it, registry cache included.

That's the topic of the second article — same BuildKit cache engine, same Artifact Registry, but a declarative, multi-target config. A YouTube video will follow to watch the whole thing run live.

Takeaways

Cloud Build runs on ephemeral workers → no local cache persists.
The fix: externalize the cache into Artifact Registry with BuildKit's registry cache.
The winning combo: docker buildx + docker-container driver + --cache-to type=registry,mode=max.
A well-ordered Dockerfile (dependencies before code) turns that cache into a concrete win: uv re-downloads nothing as long as uv.lock hasn't changed.
mode=max isn't a detail — it's the reason this works where the classic --cache-from fails.
It's not just speed: skipping the recompute of an identical artifact is GreenOps + FinOps — less CPU, less energy, less money, on every single push.

The full demo code is on GitHub. Clone it, point it at your GCP project, and watch the second build print CACHED on uv sync. 🎯

If you found this useful, follow me for more hands-on content on Google Cloud, Platform Engineering, Docker, DevOps, Data Engineering and AI agents — practical patterns from real projects and client work:

DEV Community: Mazlum Tosun

ELT with Dataform on Google Cloud

1. Explanation of the use case presented in this article

🔧 CI/CD layer

📦 Applicative ELT pipeline

2. Usual structure of a Dataform project

3. Structure of the project presented in this use case

3.1 Dataform configuration

3.2 Definitions and models

3.3 Staging layer

3.4 Mart layer

3.5 Mart step: column descriptions

3.6 Mart step: reusable function to compute player statistics

3.7 Mart step: dynamic tables and views

4. Connect the GitHub repo to the Dataform repo from the console

5. Connect the GitHub repo to the Dataform repo with Terraform

Conclusion

tosun-si / world-cup-qatar-elt-dataform

Project showing a use case with an ELT using Dataform in Google Cloud

world-cup-qatar-elt-dataform

Airflow DAG - ELT pipeline orchestration

Deploy the Airflow DAG to Cloud Composer

Running an AI Agent Locally: ADK, Gemma 4, and Docker Model Runner

What is Docker Model Runner?

What is Gemma 4?

Architecture: what changes, what stays

The agent code: one model string change

The function calling problem

GemmaModelRunner: bridging Gemma and Docker Model Runner

Dependencies

Running the agent locally

With ADK web UI

With Docker Compose

The agent in action

Simple query — top 3 scorers from France (~22s, one roundtrip)

Complex query — performance rating comparison (~2.5 min, three roundtrips)

Choosing the right model variant

Hardware used for these benchmarks

Timings

Docker Model Runner version matters

Key takeaways

What's next

Try it yourself

Docker Bake in Practice — Part 1: From Bash Scripts to Declarative Builds

Why this article exists

tosun-si / docker-bake-playground

A curated collection of concrete, practical, and reusable examples for mastering Docker Bake.

docker-bake-playground

Build the images with Docker Bake locally

Build and publish the images with Docker Bake locally

Printing the Bake file with the --print flag shows the interpolated value in the resolved build configuration.

Build the images for linter and tests

Run linter and tests with Compose

Build the multiples Bake files locally

Print multiple Bake files

Build Bake file with inheritance

Build Bake file with matrix variants

The foundation: BuildKit, Buildx, and Bake

The traditional approach: bash scripts

Enter Bake: same images, declaratively

Variables and validators

Inheritance: the _common pattern

A real-world example: Python lint and test images with uv

Composing multiple Bake files in one command

Bake from a remote GitHub repository

Matrix: building variants of the same image

Matrix with a list of items

Bake from a Compose file: one source of truth

Wrapping up Part 1

Coming in Part 2

Talks and resources

Stop rebuilding from scratch: cache Docker layers on Cloud Build

How I got here

Why the cache "disappears" on Cloud Build

Quick refresher: Docker's layer cache

The example app: FastAPI + uv

The Dockerfile: multi-stage and cache-friendly

The tempting-but-wrong fix: --cache-from on the previous image

The real fix: BuildKit's registry cache

The Cloud Build config: build-python-app.cloudbuild.yaml

Inheritance: the `_common` pattern

A real-world example: Python lint and test images with `uv`

The tempting-but-wrong fix: `--cache-from` on the previous image

The Cloud Build config: `build-python-app.cloudbuild.yaml`