DEV Community: Carlos Barbero

[Boost]

Carlos Barbero — Tue, 15 Sep 2026 22:10:49 +0000

Carlos Barbero

Sep 15

One Table, Many Engines: Running BigQuery, Spark, and Trino on a Single Iceberg Copy

#googlecloud #apacheiceberg #bigquery #dataengineering

14 min read

One Table, Many Engines: Running BigQuery, Spark, and Trino on a Single Iceberg Copy

Carlos Barbero — Tue, 15 Sep 2026 14:41:39 +0000

If you work with data — or you're just getting started in the cloud — you've probably heard terms like Data Warehouse, Data Lake, and Data Lakehouse. This article explains those concepts from scratch, details what Lakehouse for Apache Iceberg on Google Cloud (formerly BigLake) and Apache Iceberg actually are, and shows how to make three different tools read and modify the same data at the same time — no mess, no duplicated data.

Everything below was validated for real on GCP: Spark committed through the REST catalog, Trino read and wrote the same table, BigQuery ran DML on it, and we captured a genuine OCC rejection (HTTP 409 ConflictException). Full logs are in the repo's evidence/ folder.

The Problem with Data Copies

In most companies, different teams use different tools:

BI analysts use SQL inside BigQuery to build sales reports.
Data scientists run Apache Spark jobs to train forecasting models.
Infrastructure engineers use tools like Trino for fast ad-hoc queries.

In the traditional model, the BI team copied data into BigQuery. The data science team copied the same data to a Cloud Storage directory as Parquet.

That separation created three immediate problems:

Double or triple storage bills: the same terabytes were stored and billed multiple times.
Stale data: when an order was cancelled in the main system, one copy updated while the others kept the old value.
No unified security: you had to configure who could see what inside each tool separately.

The Data Lakehouse concept was born to end this duplication. The idea is simple: keep a single copy of the files in cheap cloud storage (like Google Cloud Storage) and let any query engine (SQL, Spark, Python, Trino) work directly on it, under clear transaction rules.

Core Concepts

Before the code, let's pin down what each piece does:

1. Google Cloud Storage (GCS)

The cloud hard drive. It stores raw files (CSV, JSON, Parquet) durably and cheaply. In our project, it's where the physical data and metadata files live.

2. The Parquet Format

A way to save tables to files. Instead of storing row by row like CSV, Parquet organizes data by column and applies heavy compression — queries read less disk and answer faster.

3. Apache Iceberg

Parquet alone is just a loose file. Try updating a single row inside a Parquet file and the whole file must be rewritten. Let two processes write to the same folder at once and one can silently overwrite the other.

Apache Iceberg is an open metadata layer. It organizes Parquet files into a real table, providing:

ACID transactions: either the whole write succeeds, or nothing changes. No half-written data.
Time travel: query exactly how the table looked 3 hours ago or last week — Iceberg keeps a history of versions called snapshots.
Schema evolution: adding or renaming columns doesn't break old queries.

4. What is Lakehouse for Apache Iceberg (formerly BigLake)?

Lakehouse for Apache Iceberg is the Google Cloud service — previously called BigLake — that breaks the barrier between file storage (Cloud Storage) and SQL analytics (BigQuery). If you find "BigLake" in older articles, APIs, or commands, it's the same product: Google renamed it to reflect that it is the platform's official open lakehouse foundation.

It offers two huge benefits:

Native Iceberg REST Catalog: Iceberg defines a web communication standard (the REST Catalog) to register tables and control which version of the data is current. Google Cloud implemented that official catalog inside Lakehouse — the Lakehouse Iceberg REST catalog, a central hub providing read/write interoperability between BigQuery, Google Cloud Managed Service for Apache Spark, and Iceberg-compatible OSS engines (Spark, Trino, Flink). No data is trapped in a proprietary format.
Unified security via Workload Identity: instead of downloading risky password files (like Service Account .json keys), Lakehouse uses the cloud's own execution identity to authorize queries, keeping access permissions centralized.

Renaming (BigLake → Lakehouse): although the product name is now Lakehouse, technical identifiers still use biglake — the API endpoint (biglake.googleapis.com), the CLI (gcloud biglake), table properties (gcp.biglake.*), the managed service account (blirc-*@gcp-sa-biglakerestcatalog...), the catalog type (CATALOG_TYPE_BIGLAKE), and the warehouse scheme (bl://). In this article we say "Lakehouse" in prose and keep biglake only inside literal commands and names.

How Three Engines Talk to the Same Table

In our project, we configured a single retail orders table (orders_iceberg) in Cloud Storage. Three applications access it in a coordinated way:

Figure 1: Lakehouse architecture with Apache Iceberg and the Lakehouse Iceberg REST Catalog — processing engines (BigQuery, Spark, Trino) accessing data through a centralized catalog on Google Cloud Storage.

What each tool does:

Google BigQuery: runs analytical queries and mutation commands like MERGE INTO, UPDATE, and DELETE directly on the table in GCS.
Serverless Spark (Dataproc): processes large data batches in the background without anyone provisioning servers.
Trino: a distributed SQL engine used by engineering teams for fast queries without locking production tables.

How Concurrency Control (OCC) Works

When Spark and BigQuery try to change the same table at the same time, how do we stop one from erasing the other's work?

Iceberg doesn't lock the table with pessimistic locks (which would make queries wait in line). It uses Optimistic Concurrency Control (OCC).

The process works like this:

Initial read: BigQuery and Spark both read the current table version from the Lakehouse catalog — say, snapshot number 10.
Isolated work: BigQuery computes its changes and writes new files to Cloud Storage. Spark does the same with its batch. Neither knows what the other is doing.
First to arrive wins: BigQuery finishes first and asks Lakehouse: "Commit my changes on top of snapshot 10." Since the current version is still 10, the write is accepted. The table's official snapshot becomes 11.
Conflict detection: seconds later, Spark finishes and asks Lakehouse: "Commit my changes on top of snapshot 10." Lakehouse refuses: "The table already moved to version 11 — your starting point is stale."
Automatic reconciliation: Spark doesn't break the system. It downloads the new version 11, checks whether the files BigQuery touched collide with its own, and — since they don't — applies its changes, creating snapshot 12.

The sequence diagram below shows this race step by step:

Figure 2: The optimistic commit cycle — BigQuery wins the race (snapshot 11), Spark's commit on the stale base (snapshot 10) is rejected by the catalog, and Spark retries on top of snapshot 11, producing snapshot 12.

All of this happens with no human intervention and without corrupting a single record.

What does this look like under the hood? The REST commit

Before the code, a concept that confuses many people: what is Iceberg's "main"?

Think of an Iceberg table as a Git repository:

Each snapshot is like a Git commit: an immutable version of the table (the list of Parquet files making up that state).
main is the table's default branch: a simple pointer saying "the current official snapshot is this one." When you run SELECT, your engine asks the catalog "where does main point?" and reads that snapshot.
A commit is the operation that moves that pointer — exactly like git push moves a repo's main. And, like Git, Iceberg lets you create other branches and tags pointing to specific snapshots (great for auditing and time travel), but main is always the "official" version everyone reads.

With that in mind, every engine talks to the catalog through the Iceberg REST API. Here's the full timeline of a write — notice the POST happens only once, at the end:

Figure 3: Timeline of a write — the engine fetches metadata with GET, moves Parquet files directly to/from GCS, and calls POST (the commit) only once at the end. If main moved in between, the catalog answers HTTP 409 and the engine retries on the new base.

Important details of that sequence:

Data never flows through the REST API. The biglake.googleapis.com endpoint only carries metadata (lightweight JSON). The heavy Parquet flows straight between the engine and GCS — that's why the architecture scales.
The POST is the commit: it carries two parts — requirements (safety conditions, like "only accept if main is still at snapshot X that I read in step 1") and updates (the changes: new snapshot, new pointer).
The catalog — not the engine — writes the final metadata.json. In the REST protocol, the client never moves the pointer by itself — it requests the change and the server validates it, writes the metadata, and updates main. That's why arbitration is central and identical for every engine.

That's exactly how we provoked a real conflict on purpose — we sent a commit claiming the base was an already-stale snapshot:

// POST https://biglake.googleapis.com/iceberg/v1/restcatalog
//      /v1/projects/{project}/catalogs/{catalog}/namespaces/{ns}/tables/{table}
{
  "requirements": [
    {
      // RACE CONDITION GUARD: "only commit if the main branch pointer
      // is still exactly at this snapshot" — the snapshot I read when
      // I started working. If another engine committed first, main
      // has moved and this number is stale → rejection.
      "type": "assert-ref-snapshot-id",
      "ref": "main",                        // the table's official branch
      "snapshot-id": 4808015153575422177    // the snapshot I read at the start
    }
  ],
  "updates": [
    {
      // What it would request if the requirement passed: move main's
      // pointer to the NEW snapshot described in this commit (with the
      // Parquet files I already wrote to GCS in step 3).
      // The catalog then materializes the corresponding metadata.json.
      "action": "set-current-snapshot",
      "snapshot-id": 9999999999
    }
  ]
}

Lakehouse's answer was exactly the protection we expected — HTTP 409 ConflictException:

Requirement failed: branch main has changed:
expected id 4808015153575422177 != 4196013214950993553

In plain English: "you claimed main was at snapshot 4808..., but it's already at 4196... — another engine committed before you. Commit rejected." That's step 4 of the flow happening for real, on the live API (full evidence in evidence/49_occ_conflict_409.txt).

Notice that nothing was locked: the losing engine simply re-reads main (now at the new snapshot), redoes its work on that base, and tries to commit again. It's the equivalent of git pull --rebase and pushing again — but automatic, at table level.

Step by Step: Running the Project

Prerequisites

Python 3 installed on your machine.
Google Cloud SDK (gcloud) installed and authenticated.
Docker installed (needed to run local Trino in Step 7).

Clone the repository: all the code, scripts, SQL, Mermaid diagrams, and evidence from this article are on GitHub:

git clone git@github.com:carlosrgomes/lakehouse-iceberg.git
cd lakehouse-iceberg

Step 1: Run the Local Unit Tests

You don't need to spend cloud money to understand the logic. We built a complete Iceberg REST catalog simulator that runs locally using only Python's standard library:

python3 -m unittest discover -s tests -p "test_*.py" -v

This runs 7 automated tests validating table creation, data insertion, MERGE INTO updates, and the correct rejection of conflicting writes.

Step 2: Run the Multithreaded Concurrency Simulation

To watch multiple engines racing for the same table in real time on your machine:

python3 -m src.concurrency_simulation

The script fires parallel processes simulating BigQuery, Spark, and Trino writing to the same simulated catalog, showing the safe progression of snapshots.

Step 3: Generate Synthetic Data

We built a data generator simulating a real e-commerce store with 5,000 orders (amounts, customers, categories, delivery statuses):

python3 scripts/generate_synthetic_data.py

It produces the local files scripts/synthetic_orders.csv and scripts/synthetic_orders.jsonl.

Step 4: Provision the Infrastructure on Google Cloud

When you're ready to run everything in the cloud, execute the provisioning script:

export PROJECT_ID="your-gcp-project"
export REGION="us-central1"
./scripts/deploy_gcp_resources.sh

The script safely performs the following steps:

Enables the required APIs (biglake.googleapis.com — the Lakehouse API — plus BigQuery, Storage, and Dataproc).
Creates the Cloud Storage bucket (lakehouse-iceberg-PROJECT_ID).
Creates the native Iceberg REST catalog in Lakehouse (lakehouse-rest-catalog).
Creates a catalog connection in BigQuery using Workload Identity, generating no private .json key files.
Grants the connection permission to read and write directly on the GCS bucket.
Configures environment variables for the Spark integration.

Creating the Iceberg table: after deployment, create the table manually via CLI (the table name comes from the JSON file):

gcloud biglake iceberg tables create \
    --catalog="lakehouse-rest-catalog" \   # Lakehouse catalog created by the deploy
    --namespace="retail_lakehouse" \      # logical "database" inside the catalog
    --project="${PROJECT_ID}" \
    --create-from-file="scripts/orders_iceberg_table.json"  # schema + properties

The scripts/orders_iceberg_table.json file holds the table's schema definition and settings. The essential parts, commented:

{
  "name": "orders_iceberg",
  // WHERE THE PHYSICAL FILES LIVE: the table's root inside the bucket.
  // This is where data/*.parquet and metadata/*.json|*.avro appear.
  "location": "gs://lakehouse-iceberg-PROJECT_ID/iceberg-warehouse/retail_lakehouse/orders_iceberg",
  "schema": {
    "type": "struct",
    "fields": [
      // Stable column IDs let you rename fields without breaking reads
      { "id": 1, "name": "order_id",    "type": "long",   "required": true },
      { "id": 2, "name": "customer_id", "type": "string", "required": false },
      { "id": 3, "name": "amount",      "type": "double", "required": false },
      { "id": 4, "name": "status",      "type": "string", "required": false }
    ]
  },
  "partition-spec": { "fields": [] },  // no partitioning (small demo table)
  "properties": {
    "write.format.default": "parquet"  // physical format of files in data/
  }
}

Enable BigQuery DML (needed once, for INSERT/MERGE/UPDATE/DELETE):

gcloud biglake iceberg tables update orders_iceberg \
    --catalog="lakehouse-rest-catalog" \
    --namespace="retail_lakehouse" \
    --project="${PROJECT_ID}" \
    # Enables BigQuery writes on the catalog table (reads work without it)
    --update-properties="gcp.biglake.bigquery-dml.enabled=true"

Step 5: Run a Batch Load on Serverless Spark (Dataproc)

To test Apache Spark writing to the same table managed by the Lakehouse Iceberg REST Catalog — with no manual servers — submit the job to Dataproc Serverless.

Required Iceberg dependencies: the job needs two jars on the classpath — iceberg-spark-runtime-3.5_2.12-1.10.2.jar (~45 MB) and iceberg-gcp-bundle-1.10.2.jar (~59 MB). They're kept in this repo's jars/ folder and staged to the lakehouse bucket before submitting:

# (optional) download from Maven Central if jars/ is empty
curl -L -o jars/iceberg-spark-runtime-3.5_2.12-1.10.2.jar \
    "https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-spark-runtime-3.5_2.12/1.10.2/iceberg-spark-runtime-3.5_2.12-1.10.2.jar"
curl -L -o jars/iceberg-gcp-bundle-1.10.2.jar \
    "https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-gcp-bundle/1.10.2/iceberg-gcp-bundle-1.10.2.jar"

# stage them into the lakehouse bucket
gcloud storage cp jars/iceberg-spark-runtime-3.5_2.12-1.10.2.jar jars/iceberg-gcp-bundle-1.10.2.jar \
    "gs://lakehouse-iceberg-${PROJECT_ID}/jars/"

gcloud dataproc batches submit pyspark scripts/spark_iceberg_batch_job.py \
    --region="us-central1" \
    --project="${PROJECT_ID}" \
    --version=2.2 \
    --deps-bucket="gs://lakehouse-iceberg-${PROJECT_ID}" \
    --jars="gs://lakehouse-iceberg-${PROJECT_ID}/jars/iceberg-spark-runtime-3.5_2.12-1.10.2.jar,gs://lakehouse-iceberg-${PROJECT_ID}/jars/iceberg-gcp-bundle-1.10.2.jar" \
    -- "${PROJECT_ID}" "lakehouse-iceberg-${PROJECT_ID}" "lakehouse-rest-catalog"

Note: the job uses iceberg-spark-runtime-3.5_2.12 and iceberg-gcp-bundle version 1.10.2 or newer — rest.auth.type authentication (GoogleAuthManager) only exists in Apache Iceberg 1.10+. Older versions (1.5.x, 1.9.x) fail with ClassNotFoundException or 401. The Spark commit really happens in the catalog: it's an atomic snapshot arbitrated by Lakehouse's OCC, not a loose Parquet write.

The config that connects Spark to the catalog (commented)

This is the most important snippet in the project — every property answers a real error we hit during testing. File: scripts/spark_iceberg_batch_job.py.

warehouse_path = f"bl://projects/{project_id}/catalogs/{catalog_id}"
#     ^-- NOT gs:// ! For CATALOG_TYPE_BIGLAKE catalogs, the "warehouse"
#         is the catalog's logical address (bl://...), which supports
#         tables spread across multiple buckets.

spark = SparkSession.builder \
    .appName("IcebergManagedSparkBatchWriter") \
    .config("spark.sql.defaultCatalog", "iceberg") \
    .config("spark.sql.catalog.iceberg", "org.apache.iceberg.spark.SparkCatalog") \
    .config("spark.sql.catalog.iceberg.type", "rest") \
    # The catalog is REST — any engine speaking this protocol sees the table
    .config("spark.sql.catalog.iceberg.uri",
            "https://biglake.googleapis.com/iceberg/v1/restcatalog") \
    #     ^-- Lakehouse's Iceberg REST endpoint (NOT the control-plane API
    #         v1/projects/.../catalogs — that one is only for gcloud/admin)
    .config("spark.sql.catalog.iceberg.warehouse", warehouse_path) \
    .config("spark.sql.catalog.iceberg.header.x-goog-user-project", project_id) \
    #     ^-- required header: which project Google bills/quota-counts
    #         catalog calls against
    .config("spark.sql.catalog.iceberg.rest.auth.type",
            "org.apache.iceberg.gcp.auth.GoogleAuthManager") \
    #     ^-- native Google auth: uses Dataproc's ambient credentials
    #         (Workload Identity), no JSON key. Requires Iceberg 1.10+
    .config("spark.sql.catalog.iceberg.oauth2-server-uri",
            "https://biglake.googleapis.com/iceberg/v1/restcatalog/v1/oauth/tokens") \
    #     ^-- endpoint for the catalog's OAuth2 token exchange/refresh
    .config("spark.sql.catalog.iceberg.io-impl",
            "org.apache.iceberg.gcp.gcs.GCSFileIO") \
    #     ^-- who reads/writes the physical Parquet files in GCS
    .config("spark.sql.catalog.iceberg.header.X-Iceberg-Access-Delegation",
            "vended-credentials") \
    #     ^-- CREDENTIAL VENDING: the catalog issues short-lived,
    #         scoped tokens so Spark can only reach the table's files —
    #         the engine doesn't need broad bucket access
    .config("spark.sql.catalog.iceberg.gcs.oauth2.refresh-credentials-endpoint",
            "https://oauth2.googleapis.com/token") \
    .config("spark.sql.extensions",
            "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") \
    .getOrCreate()

And the write itself is trivial — all the complexity lives in the configuration:

new_orders = spark.createDataFrame(
    [(901, "spark_batch_1", 450.0, "PROCESSED_BY_SPARK"),
     (902, "spark_batch_2", 275.5, "PROCESSED_BY_SPARK")],
    ["order_id", "customer_id", "amount", "status"])

# .writeTo().append() = writes new Parquet to GCS + POSTs the commit to
# the catalog with assert-ref-snapshot-id. Stale base → 409 → retry.
new_orders.writeTo("iceberg.retail_lakehouse.orders_iceberg").append()

The job runs the PySpark load, processes the data, and writes results back through the catalog.

Step 6: Query and Update Data via BigQuery SQL

With the table created, BigQuery reaches the Lakehouse catalog directly using the project.catalog.namespace.table syntax — no external table needed:

SELECT 
  customer_id, 
  status, 
  COUNT(*) as total_orders, 
  ROUND(SUM(amount), 2) as total_revenue
FROM `your-project.lakehouse-rest-catalog.retail_lakehouse.orders_iceberg`
GROUP BY customer_id, status
ORDER BY total_revenue DESC;

And to change an order without rewriting files by hand:

Note: to run DML (INSERT/MERGE/UPDATE/DELETE) on the catalog table, enable the gcp.biglake.bigquery-dml.enabled=true table property (via gcloud biglake iceberg tables update or a REST commit).

-- Atomic UPSERT: BigQuery writes the new Parquet files to GCS and the
-- catalog arbitrates the commit via OCC (rejects if another engine won).
MERGE INTO `your-project.lakehouse-rest-catalog.retail_lakehouse.orders_iceberg` T
--   ^-- project.catalog.namespace.table syntax: NOT an external table,
--       it's the Lakehouse catalog's Iceberg table directly
USING (
  SELECT 101 AS order_id, 'cust_1' AS customer_id, 200.0 AS amount, 'SHIPPED' AS status
) S
ON T.order_id = S.order_id
WHEN MATCHED THEN
  UPDATE SET amount = S.amount, status = S.status   -- order exists → update
WHEN NOT MATCHED THEN
  INSERT (order_id, customer_id, amount, status)    -- missing → insert
  VALUES (S.order_id, S.customer_id, S.amount, S.status);

Step 7: Run Trino (Local Docker and GCP Architecture)

Trino is a distributed (MPP) SQL engine that requires continuous, persistent communication between nodes for state discovery (/v1/info/state). For that reason, on production Google Cloud environments it runs on Google Kubernetes Engine (GKE) with StatefulSets, or locally via Docker:

cd docker
# Copy the environment variables example
cp .env.example .env
# Edit .env with your GCP values
# Start Trino
./start-trino.sh

The Trino connector pointing at the same catalog (docker/iceberg.properties), commented:

connector.name=iceberg                    # the "iceberg" catalog inside Trino
iceberg.catalog.type=rest                 # speaks the Iceberg REST protocol
iceberg.rest-catalog.uri=https://biglake.googleapis.com/iceberg/v1/restcatalog
iceberg.rest-catalog.warehouse=bl://projects/YOUR_PROJECT/catalogs/lakehouse-rest-catalog
iceberg.rest-catalog.security=GOOGLE      # native Google auth (needs Trino ~460+; we use 483)
iceberg.rest-catalog.google-project-id=YOUR_PROJECT
iceberg.file-format=PARQUET
fs.gcs.enabled=true                       # native gs:// support
gcs.auth-type=APPLICATION_DEFAULT         # uses ambient ADC — no JSON key

And in docker-compose.yml, the whole auth secret is just mounting your gcloud default credentials into the container:

services:
  trino-coordinator:
    image: trinodb/trino:483              # version with security=GOOGLE support
    volumes:
      - ./iceberg.properties:/etc/trino/catalog/iceberg.properties:ro
      - ${HOME}/.config/gcloud/application_default_credentials.json:/gcp/adc.json:ro
    environment:
      - GOOGLE_APPLICATION_CREDENTIALS=/gcp/adc.json   # ADC inside the container

After gcloud auth application-default login on the host, Trino reads and writes the same table:

-- inside the Trino CLI (docker exec -it trino-iceberg trino)
INSERT INTO iceberg.retail_lakehouse.orders_iceberg
VALUES (903, 'trino_adhoc_1', 99.99, 'ADDED_BY_TRINO');
-- atomic commit through the same REST catalog — shows up as
-- engine-name=trino in the snapshot history

Open the console in your browser at http://localhost:8088.

Tip: if the initial query list looks empty after logging in, clear the User filter field at the top of the screen to see all queries run on the cluster.

Step 8: Tearing Everything Down (Destroy Script)

To avoid unwanted charges after testing the lab, we built a script that safely and completely deletes every resource created:

./scripts/destroy_gcp_resources.sh

It cancels pending Spark executions, removes leftovers from previous Trino deployments (if any), deletes the Lakehouse namespaces and catalogs, removes the BigQuery datasets, and deletes the Cloud Storage bucket with all test files.

What We Actually Proved on GCP

This isn't just a local simulation. On the real barbero-gde project we ran the full cycle:

Spark committed 2 rows via the REST catalog (batch spark-iceberg-catalog-06, SUCCEEDED)
Trino read Spark's rows and appended its own (903, trino_adhoc_1)
BigQuery ran INSERT, MERGE, and UPDATE on the same table — including updating Trino's row to amount=500.0
Snapshot history shows engine-name = spark, trino, bigquery — three engines, one table, one copy of the data
A deliberately stale commit returned the real HTTP 409 ConflictException from Lakehouse

Final table state, visible identically from all three engines:

101  cust_1         200.0  SHIPPED
102  cust_2          85.5  COMPLETED
901  spark_batch_1  450.0  PROCESSED_BY_SPARK
902  spark_batch_2  275.5  PROCESSED_BY_SPARK
903  trino_adhoc_1  500.0  ADDED_BY_TRINO

Best Practices and Security

Zero passwords on disk: never download service-account JSON keys. Lakehouse uses native connections that automatically renew short-lived auth tokens.
Small-file control: continuous writes can generate hundreds of small Parquet files. In corporate environments, run periodic Spark compaction routines to merge small files into 128–512 MB blocks.
History and retention: configure policies to expire old snapshots after 7–14 days, keeping the metadata tree lean and queries fast.

This article was created for the Lakehouse/Iceberg Content Sprint 2026. Full source, scripts, and execution evidence: github.com/carlosrgomes/lakehouse-iceberg — if this helped you, a star is appreciated! #LakehouseIceberg

Agnostic Cluster Refactor Skill for Antigravity CLI: Building an AI Agent that Migrates Apps from AWS to GKE (Subagents, HITL Gate & Workload Identity)

Carlos Barbero — Tue, 30 Jun 2026 15:10:30 +0000

Have you ever inherited a codebase where import boto3 appears in 47 different files? Where AWS credentials live in hardcoded environment variables and file storage is a file.save("/tmp/...") that will blow up the moment it hits an ephemeral Kubernetes pod?

I did. And instead of refactoring everything by hand, I built an AI agent to do it for me — with mandatory human oversight before any production mutation.

This article documents what I built: a skill for the Antigravity CLI (agy) that scans cloud dependencies, spawns parallel subagents to refactor code and infrastructure, and validates everything on local Kubernetes before deploying to GKE with keyless Workload Identity.

The Problem: Invisible Vendor Lock-in

boto3 is the AWS SDK for Python. It seems harmless at first:

# Innocent on day 1
import boto3
s3 = boto3.client('s3', region_name='us-east-1')
s3.upload_fileobj(file, bucket_name, filename)

Six months later:

# examples/legacy-app/app.py — the real state after it grows
import os
import boto3
from flask import Flask, request, jsonify

app = Flask(__name__)

# "Temporary" hardcoded since 2022
DB_PASSWORD = os.getenv("DB_PASSWORD", "default-insecure-password")

S3_BUCKET = os.getenv("AWS_S3_BUCKET_NAME")
AWS_REGION = os.getenv("AWS_DEFAULT_REGION", "us-east-1")

s3_client = boto3.client(
    's3',
    aws_access_key_id=os.getenv("AWS_ACCESS_KEY_ID"),
    aws_secret_access_key=os.getenv("AWS_SECRET_ACCESS_KEY"),
    region_name=AWS_REGION
)

@app.route("/upload", methods=["POST"])
def upload_file():
    file = request.files['file']
    filename = file.filename
    if S3_BUCKET:
        s3_client.upload_fileobj(file, S3_BUCKET, filename)
        return jsonify({"message": f"Uploaded to AWS S3: {S3_BUCKET}"})
    else:
        # Fallback to local disk — will break in K8s ephemeral pods
        local_path = os.path.join("/tmp", filename)
        file.save(local_path)
        return jsonify({"message": f"Saved locally at {local_path}"})

Three coupling problems in a single file: proprietary SDK (boto3), AWS-specific credentials, and local disk storage that doesn't survive ephemeral Kubernetes pods.

Now multiply that by 10 services.

What We're Building

A skill for the Antigravity CLI that adds two commands to the agent chat:

/agnostic-cluster-refactor:scan-deps
/agnostic-cluster-refactor:spawn-refactor

The complete flow:

But before diving into the code, let me introduce the players.

The Players: agy, Agents, Skills, and Subagents

The Antigravity CLI (agy)

agy is not a script. It's an LLM-powered agent — you describe what you want in the chat and it decides how to do it, using a toolset: read_file, write_to_file, run_command, invoke_subagent.

The difference from a web chatbot: agy has access to your local filesystem, runs terminal commands, and operates in autonomous loops. It's an engineer working on your machine.

Script	Agent
`sed 's/boto3/gcs/g'` across all files	Analyzes the semantic context of each import and replaces it with the correct equivalent API
Fails if the environment changed	Adapts to the current state
Deterministic	Probabilistic + adaptive

Skills — Teaching Capabilities to an Agent

A skill is a SKILL.md file with YAML frontmatter that defines when and how the agent uses that capability. The agent reads the description field and decides whether the skill is relevant to the current task.

---
name: scan-deps
description: Scans the project for cloud-provider dependencies and generates
             dependency-map.json. Use when the user wants to map vendor lock-in
             before migrating to GKE.
---

## Steps

1. Ask which directory to scan
2. Run: python3 .agents/skills/.../scan_deps.py <PATH>
3. Present the DAG summary

💡 Key distinction: skills in .agents/skills/ are injected silently into context. To appear as a /command in autocomplete, you need a plugin installed at ~/.gemini/config/plugins/<plugin>/. More on that in Part 6.

Subagents — Real Parallelism

A subagent is a child agent with completely isolated context. It doesn't "see" the parent's history or the other subagent's — exactly what we want: the Backend agent can't get confused by the YAML the Infra agent is writing.

# Pseudocode — how agy orchestrates this
invoke_subagent(
    name="backend-engine",
    system_prompt="You are an expert in migrating boto3 to GCS...",
    toolset=["read_file", "write_to_file", "run_command"],
    workspace="/path/to/shadow-worktree-backend",
    message="Refactor the files from dependency-map.json"
)
# Subagent B is invoked in parallel — no blocking
invoke_subagent(
    name="infra-engine",
    toolset=["write_to_file"],  # write only — principle of least privilege
    workspace="/path/to/shadow-worktree-infra",
    message="Generate serviceaccount.yaml, deployment.yaml, ingress.yaml for GKE"
)

Each subagent operates in an isolated Git Worktree — a physical copy of the repository in a separate directory, on a different branch. If Subagent A introduces a bug, main stays untouched.

Part 1: The Dependency Scanner

The first step is mapping the problem. scan_deps.py walks the project with os.walk(), applies regex patterns by category, and generates a DAG (Directed Acyclic Graph) as JSON.

# scripts/scan_deps.py
patterns = {
    "storage": [
        r"google\.cloud\.storage",
        r"boto3.*s3",         # AWS-coupled
        r"aws-sdk.*s3"
    ],
    "messaging": [
        r"google\.cloud\.pubsub",
        r"boto3.*sqs",        # AWS-coupled
        r"kafka-python",
    ],
    "secrets": [
        r"boto3.*secretsmanager",
        r"python-dotenv",
    ],
    "databases": [
        r"psycopg2", r"pymongo"
    ]
}

for root, dirs, files in os.walk(path):
    dirs[:] = [d for d in dirs if not d.startswith('.')
               and d not in ['venv', 'node_modules', '__pycache__']]
    for file in files:
        if not file.endswith(('.py', '.js', '.yaml', '.tf')):
            continue
        with open(os.path.join(root, file)) as f:
            content = f.read()
            for dep_type, pattern_list in patterns.items():
                for pattern in pattern_list:
                    if re.search(pattern, content, re.IGNORECASE):
                        dependencies[dep_type].append({
                            "file": os.path.relpath(file_path, path),
                            "matched_pattern": pattern
                        })

The output is a dependency-map.json with the full dependency graph:

{
  "dependencies": {
    "storage": [
      { "file": "examples/legacy-app/app.py", "matched_pattern": "boto3.*s3" },
      { "file": "examples/legacy-app/api.py",  "matched_pattern": "boto3.*s3" }
    ],
    "messaging": [
      { "file": "examples/legacy-app/worker.py", "matched_pattern": "boto3.*sqs" }
    ]
  },
  "architectural_dag": {
    "nodes": [
      { "id": "application",   "type": "component" },
      { "id": "dep-storage",   "files": ["app.py", "api.py"] },
      { "id": "provider-aws",  "type": "cloud-provider" }
    ],
    "edges": [
      { "source": "application",  "target": "dep-storage",  "relation": "uses_storage"   },
      { "source": "dep-storage",  "target": "provider-aws", "relation": "coupled_to_aws" }
    ]
  },
  "recommended_action": "Execute '/spawn-refactor' targeting GCP GKE"
}

❓ Why a DAG and not a plain list? The graph reveals transitive relationships: app.py and worker.py both depend on AWS via boto3 — so they need to be refactored together. A list would only say "these files have boto3."

Part 2: The HITL Gate — No Agent Touches Production Without Confirmation

This was the most important design decision: how do I ensure the agent doesn't refactor the wrong file without me seeing what's happening first?

The answer lives in two places.

Hooks — The Automatic Security Guard

The .agents/hooks.json file registers a PreToolUse hook — a command that runs before any write_to_file the agent attempts:

{
  "hitl-production-gate": {
    "enabled": true,
    "PreToolUse": [
      {
        "matcher": "write_to_file|replace_file_content|multi_replace_file_content",
        "hooks": [
          {
            "type": "command",
            "command": "python3 .agents/skills/agnostic-cluster-refactor/scripts/scan_deps.py --check-only",
            "timeout": 5
          }
        ]
      }
    ]
  }
}

The hook receives a JSON payload via stdin and responds with a decision:

# scan_deps.py — --check-only mode
SAFE_WRITE_PREFIXES = ("examples/", "terraform/", ".agents/")

def check_only_hook():
    payload = json.load(sys.stdin)
    target = payload.get("toolCall", {}).get("args", {}).get("TargetFile", "")
    workspace_root = payload.get("workspacePaths", ["."])[0]
    rel_path = os.path.relpath(target, workspace_root)

    if not any(rel_path.startswith(p) for p in SAFE_WRITE_PREFIXES):
        print(json.dumps({
            "decision": "force_ask",
            "reason": f"[HITL Gate] '{rel_path}' is outside safe directories. Confirm before proceeding."
        }))
    else:
        print(json.dumps({"decision": "allow"}))

Three possible decisions the hook can return:

Decision	Effect
`"allow"`	Agent proceeds automatically
`"force_ask"`	agy pauses and asks the human
`"deny"`	Completely blocked, no prompt

Testing it from the command line:

# File OUTSIDE safe directories
echo '{"toolCall":{"name":"write_to_file","args":{"TargetFile":"/project/src/app.py"}},
      "workspacePaths":["/project"]}' | python3 scripts/scan_deps.py --check-only
# → {"decision": "force_ask", "reason": "[HITL Gate] 'src/app.py' is outside safe directories..."}

# File INSIDE safe directories
echo '{"toolCall":{"name":"write_to_file","args":{"TargetFile":"/project/examples/k8s/deployment.yaml"}},
      "workspacePaths":["/project"]}' | python3 scripts/scan_deps.py --check-only
# → {"decision": "allow"}

The Gate in SKILL.md — Explicit Confirmation Before Spawning

Beyond the automatic hook, the /spawn-refactor SKILL.md instructs the agent to always ask for explicit confirmation before spawning subagents:

## HITL Gate — mandatory before any mutation

Display the list of files that will be changed and ask:

  The following files will be modified:
    - examples/legacy-app/app.py    (replace boto3 → GCS)
    - examples/legacy-app/worker.py (replace SQS → Pub/Sub)

  Type YES to confirm or NO to abort.

Halt if the user does not confirm with YES.

🛡️ Two layers of protection: the hook catches any write automatically, and the SKILL.md forces you to see the full plan before anything moves.

Part 3: Before and After — Real Code

After Subagent A runs, app.py goes from the boto3 mess above to this:

# examples/refactored-app/app.py
import os
from flask import Flask, request, jsonify

app = Flask(__name__)

DB_PASSWORD = os.getenv("DB_PASSWORD")
if not DB_PASSWORD:
    raise RuntimeError("DB_PASSWORD environment variable is required!")

GCS_BUCKET_NAME = os.getenv("GCS_BUCKET_NAME", "local-mock")

# LOCAL_MOCK=true → bypasses GCS; useful for K8s plumbing tests without real credentials
LOCAL_MOCK = os.getenv("LOCAL_MOCK", "false").lower() == "true"

if LOCAL_MOCK:
    storage_client = None
    print("[LOCAL_MOCK] GCS disabled. Uploads will be simulated.")
else:
    from google.cloud import storage  # import only when we actually need GCS
    storage_client = storage.Client()  # zero credentials — ADC via Workload Identity

_mock_store: dict[str, bytes] = {}

@app.route("/health", methods=["GET"])
def health():
    return jsonify({
        "status": "healthy",
        "platform": "local-k8s" if LOCAL_MOCK else "gcp-gke",
        "gcs_bucket": GCS_BUCKET_NAME,
        "mock_mode": LOCAL_MOCK,
    })

@app.route("/upload", methods=["POST"])
def upload_file():
    file = request.files["file"]
    filename = file.filename

    if LOCAL_MOCK:
        data = file.read()
        _mock_store[filename] = data
        return jsonify({
            "message": f"[LOCAL_MOCK] {filename} stored in memory ({len(data)} bytes)",
            "gcs_uri": f"gs://local-mock/{filename}",
            "files_in_mock": list(_mock_store.keys()),
        })

    bucket = storage_client.bucket(GCS_BUCKET_NAME)
    blob = bucket.blob(filename)
    blob.upload_from_file(file)
    return jsonify({
        "message": f"Uploaded {filename} to {GCS_BUCKET_NAME}",
        "gcs_uri": f"gs://{GCS_BUCKET_NAME}/{filename}",
    })

@app.route("/files", methods=["GET"])
def list_files():
    if LOCAL_MOCK:
        return jsonify({"files": list(_mock_store.keys()), "source": "local-mock"})
    blobs = storage_client.list_blobs(GCS_BUCKET_NAME)
    return jsonify({"files": [b.name for b in blobs], "source": f"gs://{GCS_BUCKET_NAME}"})

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8080, debug=LOCAL_MOCK)

What changed:

Before	After
`import boto3`	`from google.cloud import storage` (conditional)
`boto3.client('s3', aws_access_key_id=...)`	`storage.Client()` — zero credentials
`file.save("/tmp/...")`	`blob.upload_from_file(file)`
`DB_PASSWORD` with insecure default	`RuntimeError` if missing

The Import That Cost Me an Hour

# ❌ Wrong — crashes at startup without GCP credentials
from google.cloud import storage
storage_client = storage.Client()   # RuntimeError before any request is handled

# ✅ Correct — import only happens when we actually need it
if LOCAL_MOCK:
    storage_client = None
else:
    from google.cloud import storage   # ← inside the else block
    storage_client = storage.Client()

from google.cloud import storage executes when Python loads the module — before serving any request. Without GCP credentials, the app crashes at startup. Moving the import inside else fixes it: with LOCAL_MOCK=true, the module is never imported.

Part 4: Testing on Local Kubernetes Without GCP Credentials

I wanted to validate the entire K8s stack (Deployment, ConfigMap, Secret, Service, health checks, routing) locally using Docker Desktop — without needing real GCP credentials.

The solution was LOCAL_MOCK=true combined with a Docker Desktop quirk that catches a lot of people off guard.

The Separate containerd Problem

Docker Desktop uses two completely separate runtimes that don't share images:

┌──────────────────────────────────────┐
│  Docker daemon                       │  ← docker build, docker images
│  (images here are NOT visible to K8s)│
└──────────────────────────────────────┘

┌──────────────────────────────────────┐
│  containerd                          │  ← used by the Kubernetes cluster
│  (separate namespace)                │
└──────────────────────────────────────┘

When you run docker build -t my-image ., the image exists in the Docker daemon but not in containerd. With imagePullPolicy: Never, K8s looks in containerd and fails:

Failed to pull image "my-image:local": ErrImageNeverPull

The fix: a local registry as the bridge between both runtimes.

# registry:2 on port 5001 (port 5000 is taken by macOS AirPlay)
docker run -d -p 5001:5000 --restart=always --name local-registry registry:2

Now the flow works end-to-end:

docker build → Docker daemon
      ↓
docker tag + push → localhost:5001 → registry:2
      ↓
containerd pulls from registry:2 ← K8s Pod starts successfully

The Makefile handles all of this in a single command:

REGISTRY       = localhost:5001
REGISTRY_IMAGE = $(REGISTRY)/agnostic-cluster-refactor:local

registry-start:
    @docker ps --filter name=local-registry --filter status=running | grep local-registry || \
        docker run -d -p 5001:5000 --restart=always --name local-registry registry:2

build: registry-start
    docker build -t agnostic-cluster-refactor:local .
    docker tag agnostic-cluster-refactor:local $(REGISTRY_IMAGE)
    docker push $(REGISTRY_IMAGE)
    @echo "Image available to K8s: $(REGISTRY_IMAGE)"

local-up:
    kubectl config use-context docker-desktop
    kubectl apply -f examples/k8s/local/secret-db.yaml
    kubectl apply -f examples/k8s/local/configmap.local.yaml
    kubectl apply -f examples/k8s/local/deployment.local.yaml
    kubectl apply -f examples/k8s/local/service.local.yaml
    kubectl rollout status deployment/agnostic-cluster-app --timeout=60s
    @echo "Access: http://localhost:8080/health"

The Local Deployment Manifest

# examples/k8s/local/deployment.local.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: agnostic-cluster-app
spec:
  replicas: 1
  template:
    spec:
      containers:
        - name: app
          image: localhost:5001/agnostic-cluster-refactor:local
          imagePullPolicy: Always   # always pull from local registry
          envFrom:
            - configMapRef:
                name: app-config-local   # injects LOCAL_MOCK=true
          env:
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: app-secrets
                  key: db-password
          readinessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 5

# examples/k8s/local/configmap.local.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config-local
data:
  GCS_BUCKET_NAME: "local-mock"
  GCP_PROJECT_ID: "local-dev"
  LOCAL_MOCK: "true"    # ← activates the in-memory store

Running it:

make build      # build + push to local registry
make local-up   # apply all manifests

curl http://localhost:8080/health
# {"status":"healthy","platform":"local-k8s","mock_mode":true,"gcs_bucket":"local-mock"}

curl -X POST http://localhost:8080/upload -F "file=@package.json"
# {"message":"[LOCAL_MOCK] package.json stored in memory (842 bytes)",
#  "gcs_uri":"gs://local-mock/package.json"}

curl http://localhost:8080/files
# {"files":["package.json"],"source":"local-mock"}

make local-down  # teardown

✅ Entire K8s stack validated — Deployment, ConfigMap, Secret, Service, health checks, routing — without a single GCP token.

Part 5: Workload Identity — Why JSON Keys Don't Belong in Production

On GKE, the story is completely different.

The naive approach:

os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "/app/sa-key.json"
storage_client = storage.Client()

This requires a JSON key file inside the container, which means:

The key goes into a Kubernetes Secret
The Secret gets mounted as a volume in the Pod
The key expires and needs manual rotation
If it leaks, anyone has full access to your GCS bucket

The Workload Identity approach: annotate a Kubernetes Service Account (KSA) with a Google Service Account (GSA) email:

# examples/k8s/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: refactored-app-ksa
  annotations:
    iam.gke.io/gcp-service-account: "gke-app-sa@MY_PROJECT.iam.gserviceaccount.com"

GKE's internal metadata server intercepts ADC calls from Pods, verifies the annotation, and returns a short-lived OAuth2 token:

The application code becomes:

# Zero credentials — works automatically on GKE
storage_client = storage.Client()

Terraform provisions the IAM binding automatically:

# terraform/iam.tf
resource "google_service_account_iam_member" "workload_identity" {
  service_account_id = google_service_account.app.name
  role               = "roles/iam.workloadIdentityUser"
  member = "serviceAccount:${var.project_id}.svc.id.goog[default/refactored-app-ksa]"
}

🔐 This binding is the handshake between the Kubernetes world and GCP IAM. Without it, no token is issued — storage.Client() returns a 403.

Part 6: The agy Plugin Structure

When I first tested, /scan-deps and /spawn-refactor did not appear in the agy autocomplete. I spent a good chunk of time debugging this.

The discovery: agy has three distinct skill-loading mechanisms:

Mechanism	Location	Shows in `/` autocomplete?
Project skill	`.agents/skills/<name>/SKILL.md`	❌ No
Global contextual skill	`~/.gemini/antigravity-cli/skills/`	❌ No
Plugin with namespace	`~/.gemini/config/plugins/<plugin>/`	✅ Yes

To make the commands appear, create the plugin structure:

mkdir -p ~/.gemini/config/plugins/agnostic-cluster-refactor/skills/scan-deps
mkdir -p ~/.gemini/config/plugins/agnostic-cluster-refactor/skills/spawn-refactor

cat > ~/.gemini/config/plugins/agnostic-cluster-refactor/plugin.json << 'EOF'
{
  "name": "agnostic-cluster-refactor",
  "version": "1.0.0",
  "description": "Migrates apps from AWS to GCP GKE with Workload Identity."
}
EOF

After restarting agy, the autocomplete shows:

/agnostic-cluster-refactor:scan-deps
/agnostic-cluster-refactor:spawn-refactor

The namespace prevents collisions — two different plugins can both have a skill named scan-deps and they'll appear as /plugin-a:scan-deps and /plugin-b:scan-deps.

What the Agent Actually Did

When I ran /agnostic-cluster-refactor:spawn-refactor and confirmed the HITL Gate, Gemini (the agy engine) orchestrated:

Subagent A (Backend) — in shadow-worktree-backend:

Read dependency-map.json to identify boto3 files
Replaced import boto3 → from google.cloud import storage, pubsub_v1 in each file
Replaced boto3.client('s3', ...) → storage.Client().bucket(...) with semantically equivalent calls
Replaced boto3.client('sqs', ...) → pubsub_v1.SubscriberClient()
Updated requirements.txt: removed boto3==1.28.0, added google-cloud-storage==2.10.0 and google-cloud-pubsub==2.18.0

Subagent B (Infra) — in shadow-worktree-infra:

Generated serviceaccount.yaml with the iam.gke.io/gcp-service-account annotation
Generated deployment.yaml with env vars via ConfigMap/Secret — no hardcoded credentials
Generated ingress.yaml with ingressClassName: gce (the current format, not the deprecated annotation)

All in isolated Git Worktrees, in parallel, without touching main.

Lessons Learned

1. The conditional import is intentional, not lazy.

When LOCAL_MOCK=true, from google.cloud import storage must not run at module level. Without GCP credentials, it throws at startup before any request is served. Import conditionally.

2. Docker Desktop K8s and the Docker daemon live in separate worlds.

imagePullPolicy: Never breaks with Docker Desktop because K8s uses containerd, not the daemon. Use a local registry on port 5001 (5000 is taken by macOS) and imagePullPolicy: Always.

3. .agents/workflows/ does not create slash commands in agy.

Skills in .agents/skills/ are context injections, not interactive commands. The / autocomplete requires a plugin installed in ~/.gemini/config/plugins/.

4. The HITL Gate needs two independent layers.

A hook catches unexpected writes automatically. But for /spawn-refactor — which modifies multiple files in parallel — explicit plan confirmation in the SKILL.md is non-negotiable. Without both layers, the agent can act before you understand the blast radius.

5. Workload Identity eliminates an entire security problem class.

No JSON keys in containers means no credential leaks in logs, no manual rotation, no hardcoded keys in Dockerfiles, and no Secret volumes mounted on Pod disk. The Metadata Server's short-lived tokens are genuinely safer.

Try It Yourself

# Clone
git clone https://github.com/carlosrgomes/agnostic-cluster-refactor
cd agnostic-cluster-refactor

# Test locally without GCP (Docker Desktop K8s)
make build      # build + push to local registry
make local-up   # apply manifests to docker-desktop context
curl http://localhost:8080/health

# Scan your own project's dependencies
python3 scripts/scan_deps.py /path/to/your/project
cat dependency-map.json | python3 -m json.tool

# Validate the HITL Gate hook
echo '{"toolCall":{"name":"write_to_file","args":{"TargetFile":"/project/src/main.py"}},
      "workspacePaths":["/project"]}' | python3 scripts/scan_deps.py --check-only
# → {"decision": "force_ask", ...}

# Teardown
make local-down

For the full GKE deployment with Workload Identity, the project README includes the Terraform that provisions all the infrastructure.

Conclusion

The project started from a real problem (boto3 everywhere) and ended up with a surprisingly complete solution: automatic dependency scanning, parallel subagent refactoring, mandatory human oversight, local K8s testing without cloud credentials, and keyless production auth.

What impressed me most wasn't the AI doing the refactoring — it was the supervision system design: hooks intercepting any write outside safe directories, SKILL.md with an explicit gate before destructive actions, and Git Worktrees ensuring main is never touched without human review.

An autonomous agent without oversight is a chaotic script. An agent with a well-designed HITL Gate is a trustworthy teammate.

carlosrgomes / agnostic-cluster-refactor

Agnostic Cluster Refactor — Skill para Antigravity CLI (agy)

Tutorial técnico completo: migração autônoma de aplicações acopladas à AWS para o Google Kubernetes Engine (GKE) usando o Antigravity CLI com Workload Identity, subagentes paralelos e HITL gate.

Índice

O Problema: Vendor Lock-in
Visão Geral da Solução
Guia Completo de Conceitos Técnicos — agy, Agents, Subagents, Skills, Plugins, Rules, Hooks, HITL, DAG, Worktrees, Workload Identity, K8s, Terraform
Arquitetura do Sistema
Estrutura do Repositório
Pré-requisitos
Passo 0 — Configurar o agy CLI
Passo 1 — Provisionar o GKE (Terraform)
Passo 2 — Escanear Dependências (/scan-deps)
Passo 3 — Orquestrar a Refatoração (/spawn-refactor)
Passo 4 — Aplicar os Manifestos Kubernetes
Passo 5 — Ativar o Workload Identity
Antes e Depois: Comparação de Código
Referência dos Manifestos Kubernetes
Referência Terraform
O Sistema de Hooks e HITL Gate
Verificação End-to-End
Troubleshooting

1. O Problema: Vendor Lock-in

Aplicações legadas acumulam acoplamentos…

View on GitHub

Local-First AI Done Right: How Gemma 4 E2B and 'Thinking Mode' Powered DiagramFlowAI

Carlos Barbero — Wed, 13 May 2026 00:08:54 +0000

Over the past few months, the tech world has shifted from "AI-capable" to "Agentic-driven". But as developers, we face a major challenge: How do we build autonomous, code-executing AI agents that respect privacy, run locally, and don't cost a fortune in API credits?

In this article, I’ll show you how I built DiagramFlowAI — a local-first architecture that uses Gemma 2 (via flutter_gemma) with a custom Thinking Mode to dynamically generate architectural diagrams, executing everything safely inside E2B sandboxes.

🚀 Live Repo: github.com/carlosrgomes/DiagramFlowAI

💡 If you find this architecture useful, don't forget to leave a star!

What I Built

DiagramFlowAI is a local-first desktop application (macOS, Windows, and Linux) that transforms natural language descriptions into production-ready architecture diagrams. It intelligently generates standard Mermaid syntax for general workflows, or outputs structured commands mapping icons for cloud architectures.

The application solves a very specific tension in modern software engineering: privacy versus productivity. When architects and engineers sketch out internal systems—such as authentication flows, proprietary data pipelines, or secure cloud perimeters—sending that data to a cloud-based LLM endpoint is often a compliance deal-breaker.

DiagramFlowAI is designed to be completely self-contained. Powered by flutter_gemma and LiteRT, it runs 100% locally. After the initial model download, it requires zero internet connection, uses no API keys, and has no telemetry. It’s an AI diagramming studio that respects your company’s security posture.

Demo

Code

github.com/carlosrgomes/DiagramFlowAI

How I Used Gemma 4

Most AI showcases default to the largest model available. I did the exact opposite. I deliberately built DiagramFlowAI around Gemma 4 E2B and E4B—the edge variants—and intentionally skipped the 31B Dense and 26B MoE models. Here is why the smallest variants were the secret to making this desktop app work, and how Gemma 4's "Thinking Mode" unlocked capabilities I didn't expect.

The Unfashionable Choice: Small over Large

If you're building a high-throughput backend, the 31B Dense or 26B MoE are obvious choices. However, my deployment constraints pointed in a completely different direction:

Democratic Hardware Requirements: A 31B dense model in 4-bit quantization demands around 16-20 GB of RAM. The E4B model comfortably fits within 4-6 GB and runs smoothly on integrated GPUs. That’s the difference between an app anyone can use and a toy restricted to high-end workstations.
Frictionless Onboarding: The moment a user has to paste an API key, onboarding conversion plummets. Because E2B and E4B are open weights, users can simply click "download" and start diagramming. No auth walls, no billing setups.
Snappy Cold Starts: In a desktop app, the first interaction needs to feel immediate. The E2B model loads and responds in seconds on modern M-series Macs and modern PCs, keeping the user in their flow state.

To give users flexibility, I built in a toggle between E2B (faster) and E4B (more accurate on complex syntax), rather than hardcoding a single option.

The Underrated Superpower: Thinking Mode

If there is one thing every developer building with Gemma 4 should internalize, it's the power of the reasoning trace. The flutter_gemma SDK exposes Gemma 4's internal reasoning as a distinct stream of ThinkingResponse chunks.

For diagram generation, this is a game-changer. Mermaid syntax is notoriously fragile—a stray colon, an unquoted string, or a missing end tag will break the entire render. Without Thinking Mode, a 4B parameter model will often confidently output syntactically broken DSLs in one shot.

With Thinking Mode enabled, the model spends a few hundred tokens planning its structure first ("OK, this is a sequence diagram, I need actor -> participant -> arrow -> response..."). Consequently, the final output is dramatically more reliable.

In the UI, I expose this trace as a collapsed accordion (e.g., "Thinking · 2.4s"). This subtle UX choice builds user trust and makes the generation wait feel highly productive, without overwhelming them with raw logs.

Pragmatic Patterns for 4B Models

Fighting with the model for a few weeks led me to a few hard-won architectural patterns:

Treat the System Prompt as a Grammar, Not a Personality: Small models pattern-match exceptionally well. My 500-line system prompt isn't about making the AI "helpful"; it's an output contract. I use explicit delimiters (<DIAGRAM>...</DIAGRAM>) and provide "syntax cards" showing the most common parser failures (e.g., NEVER write X). Teaching the model what not to do prevents entire classes of bugs.
Trust the Contract over Regex: Instead of fighting fragile markdown fences with complex Regex, I rely on the XML-style delimiters defined in the system prompt. Even when the model decides to write an explanatory paragraph, the actual code is safely wrapped and easily extracted.
Engineer the Recovery Loop: Even with Thinking Mode, complex diagrams might occasionally fail to parse. Instead of trying to prompt-engineer my way to a 100% success rate (which is near impossible at 4B), I built a small ReAct-style retry loop. If the Mermaid parser throws an error, the app feeds the exact error message back into a follow-up turn. The model almost always fixes its syntax on the second attempt.

Gemma 4 E2B and E4B prove that you don't need a massive, cloud-hosted LLM to ship a genuinely useful, structured AI application. If you map your deployment constraints, lean hard on the system prompt, enable Thinking Mode, and engineer a smart recovery loop, these edge models become a feature, not a compromise.

Do you think "Local-first AI" is the future for enterprise dev tools, or is the convenience of Cloud LLMs too hard to beat? Also, if you have ideas on how to improve the Mermaid recovery loop, let’s chat in the comments!