DEV Community: Google Developer Experts

Building File4Base: The Modern, Open-Source Alternative to old file bases (Powered by Antigravity)

Mario Ezquerro — Thu, 24 Sep 2026 22:07:55 +0000

For decades, platforms like Claris FileMaker, Microsoft Access, and 4D enabled businesses to build tailored database applications rapidly without needing dedicated enterprise engineering teams.

However, the legacy RAD (Rapid Application Development) ecosystem comes with well-known bottlenecks: proprietary runtimes, opaque file formats, steep licensing costs, and vendor lock-in.

Today, I’m excited to share File4Base: an open-source, modern alternative designed to bring back the speed and visual intuition of FileMaker, powered by clean architecture, open databases, and native cross-platform performance.

🎯 The Vision: What is File4Base?

File4Base aims to bridge the gap between relational databases and business users by providing:

A Visual Canvas Layout Engine: Design forms, lists, reports, and search queries dynamically.
First-Class Relational Schemas: Real PostgreSQL tables and columns under the hood—no black-box database engines.
Visual Scripting Workspace: Step-based automation (similar to FileMaker’s Script Workspace), serialized in structured JSON and executed deterministically.
Self-Hostable & Zero Lock-in: Fully dockerized backend with standard SQL data access.

🏗️ The Modern Stack

Instead of building a monolith, File4Base separates the presentation layer from the database engine using modern industry standards:

Client (Desktop & Tablet): Flutter (macOS, Windows, Linux). Native rendering, fast UI redraws, and cross-platform consistency.
Backend Core: Go (Golang). A lightweight, memory-efficient service managing metadata catalogs, generic dynamic CRUD operations, and permission enforcement.
Database: PostgreSQL 16. Rock-solid reliability with JSONB metadata support, native migrations, and real-time triggers via LISTEN/NOTIFY.
Deployment: Single command via docker compose up -d for the server infrastructure.

┌──────────────────────────────────────────────┐
│        Flutter Desktop Client (Native)       │
│  [Layout Designer]  [Browse]  [Scripts]      │
└──────────────────────┬───────────────────────┘
                       │ REST / WebSockets
┌──────────────────────▼───────────────────────┐
│              Go Backend Engine               │
│   (Schema Manager, CEL Evaluator, Auth)      │
└──────────────────────┬───────────────────────┘
                       │ pgx / SQL
┌──────────────────────▼───────────────────────┐
│                PostgreSQL 16                 │
│      (Dynamic Schemas + System Catalogs)     │
└──────────────────────────────────────────────┘

🤖 Built with Google Antigravity: Spec-Driven Development

A unique aspect of how File4Base is coming to life is the engineering workflow. We are orchestrating development through Google Antigravity.

Instead of writing repetitive boilerplate manually:

We write rigorous architectural blueprints, JSON schemas, and roadmap documents (.antigravity/rules.md, docs/specs/*).
Antigravity takes these specifications and iteratively develops tested modules in Go and Flutter.
The codebase stays decoupled, clean, and strictly aligned with documented architectural standards.

🚀 What's in Progress?

We've already mapped out the foundational layers:

[x] Monorepo architecture & Dockerized development environment.
[x] Complete FileMaker Menu & Feature functional reference guide.
[x] Script Workspace UI specification and mockups (3-panel workflow).
[ ] Backend metadata catalog (sys_tables, sys_columns, sys_layouts).
[ ] Dynamic JSON layout renderer in Flutter.

🤝 Join the Project!

File4Base is 100% open-source, and we are building it in public from day one.

If you have experience with:

Flutter / Dart (Desktop layouts, state management, drag-and-drop engines)
Go / Golang (Dynamic SQL builders, AST parsers, real-time WebSockets)
FileMaker / Low-Code architecture (Feature parity, ergonomic UX workflows)

We would love your feedback, stars, and contributions!

👉 GitHub Repository: https://github.com/file4base/file4base-app

Let us know in the comments: what is the single most critical feature you miss from classic desktop database builders?

Running a Jev-Style Decision Model on One TPU v6e: What Fits, What It Costs, and What Changes From a GPU

xbill — Thu, 24 Sep 2026 16:32:26 +0000

This article provides a step by step guide to running a Jev-style decision model on one Google Cloud TPU v6e chip with Gemma 4 and vLLM, and compares it with the same read on one NVIDIA L4 GPU. The measurement was pre-registered, and every per-item output is committed.

One v6e chip serves Gemma 4 E2B, E4B and 12B at bf16 and a 26B-A4B fp8 build; no 31B checkpoint loads. The same checkpoints give the same answers on the TPU and the L4, and the 26B decides in 27 to 33 ms against 61 ms on the L4. On demand, the TPU costs more per decision than the L4 or Jev; 12B is the size to pick.

https://github.com/xbill9/gemma4-dev/tree/main/jev-tpu

Why Measure This?

A Jev-style decision model answers a typed question with a probability for each allowed option: end the prompt where the answer starts, read the scores of the allowed label tokens, and apply a softmax over those. TypeSafe's Jev does this as a hosted service, and any open model can be read the same way.

A companion article measured this read on one NVIDIA L4 and set it beside Jev's published results. This one asks what changes on a TPU: which Gemma 4 sizes fit one v6e chip, whether the read works the same way, and what the chip buys in speed and cost.

At This Point You Should Have…

A Google Cloud project with v6e quota in a region that offers ct6e-standard-1t, and the gcloud CLI logged in
A Hugging Face token stored as the Secret Manager secret hf-token, readable by the Compute Engine default service account
A Cloud Storage bucket for results, set as BUCKET in tpu/run.sh and tpu/startup.sh
The repository cloned: git clone https://github.com/xbill9/gemma4-dev and cd gemma4-dev/jev-tpu

Step 1 — Pre-Register the Measurement

The models, image, serving flags, data, metrics and the order of the quantized attempts are written down and committed before any model call, in PREREGISTRATION.md; each departure from it is recorded there with a date before the affected results are scored.

git show -s --oneline e0cce69 4937fb8

e0cce69 jev-tpu: sibling of jev for one TPU v6e chip — copied read, data, suite and scoring code (proxy byte-identical); pre-registration for E2B/E4B/12B bf16 and exploratory quantized 26B-A4B/31B probes
4937fb8 jev-tpu: VM boot, serve and run drivers; pre-registration amended before any model call — suite built locally at 0e67403 (13/13 checksums) and re-checked on the VM

Step 2 — Pick Checkpoints That Fit One Chip

One v6e chip has 31.24 GiB of memory, of which vLLM uses up to 28.74 GiB:

Memory statistics | total_hbm_limit_gb=31.24GiB | total_hbm_limit_cap_gb=28.74GiB | total_hbm_used_gb=24.56GiB | total_hbm_avail_gb=4.19GiB

E2B, E4B and 12B fit at bf16. The 26B-A4B and 31B do not, so the pre-registration lists quantized builds to try in order:

Size	Checkpoint	Format	Size on disk
E2B	`google/gemma-4-E2B-it`	bf16
E4B	`google/gemma-4-E4B-it`	bf16
12B	`google/gemma-4-12B-it`	bf16
26B-A4B	`RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic`	fp8	26.67 GiB
31B	`google/gemma-4-31B-it-qat-w4a16-ct`	w4a16	21.67 GiB
31B	`cyankiwi/gemma-4-31B-it-AWQ-4bit`	4-bit	19.47 GiB

The 31B fp8 builds are 30.98 GiB, over the chip's usable memory, and were not tried.

Step 3 — Launch One v6e Chip

The VM runs the whole measurement from its startup script, fetching the code and the suite from Cloud Storage, and deletes itself when the run ends.

gcloud compute instances create jev-tpu-v6e1 --zone europe-west4-a \
  --machine-type ct6e-standard-1t \
  --image-family ubuntu-accel-2204-amd64-tpu-v5e-v5p-v6e --image-project ubuntu-os-accelerator-images \
  --boot-disk-size 200GB --scopes cloud-platform --maintenance-policy TERMINATE \
  --provisioning-model STANDARD --max-run-duration 6h --instance-termination-action DELETE \
  --metadata jev-code=<code-tarball>,jev-run=2026-09-24-v6e1 --metadata-from-file startup-script=tpu/startup.sh

NAME          ZONE            MACHINE_TYPE      PREEMPTIBLE  INTERNAL_IP    EXTERNAL_IP    STATUS
jev-tpu-v6e1  europe-west4-a  ct6e-standard-1t               10.164.15.207  34.32.155.166  RUNNING

--scopes cloud-platform lets the VM read the Hugging Face token from Secret Manager at boot, so the token never enters instance metadata.

Step 4 — Serve Each Model

tpu/serve.sh starts one model at a time with the same flags for every arm and waits for /v1/models. Abridged, with the token and volume options left out:

docker run -d --name vllm --privileged --net=host --shm-size 10gb vllm/vllm-tpu:nightly \
  vllm serve google/gemma-4-12B-it --tensor-parallel-size 1 --max-model-len 2048 --max-num-seqs 16 \
  --max-logprobs 32 --generation-config vllm --limit-mm-per-prompt '{"image":0,"audio":0}' --enable-prefix-caching

[jev-run 2026-09-24T14:42:48Z] READY google/gemma-4-E2B-it after 346s
[jev-run 2026-09-24T14:51:34Z] READY google/gemma-4-E4B-it after 421s
[jev-run 2026-09-24T15:02:01Z] READY google/gemma-4-12B-it after 512s
[jev-run 2026-09-24T15:14:59Z] READY RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic after 616s

The image was vllm/vllm-tpu@sha256:19a1a0526476f902eb83e1057f3d8938f35b457dd7716a30e9ab4f7bee90d507, vLLM 0.29.1rc1.dev468+g0b7f11a1e. On v6e it stores the KV cache in fp8 by default for every model:

INFO 09-24 14:53:37 [tpu_platform.py:232] Automatically using fp8_e5m2 for FP8 KV cache on TPU v6e.

So "bf16" below describes the weights. The L4 run kept the default 16-bit KV cache.

Step 5 — Read the Labels Out of the Top 32

On a GPU, the read asks vLLM for the log-probabilities of the label tokens by id. vLLM's TPU backend returns only the top-k log-probabilities, and a request for specific token ids fails. Four request shapes are sent before each model's reads, and the answers are kept:

== probe 3: {"prompt":[2,106,1645,108],"max_tokens":1,"temperature":0,"logprobs":32,"return_tokens_as_token_ids":true}
{"id":"cmpl-9ee6dc5f0461280b","object":"text_completion", ...
== probe 4: {"prompt":[2,106,1645,108],"max_tokens":1,"temperature":0,"logprobs":5,"logprob_token_ids":[236776,236799,236780],"return_tokens_as_token_ids":true}
{"error":{"message":"list index out of range","type":"InternalServerError","param":null,"code":500}}

So the read asks for the top 32 log-probabilities (JEV_TOPK=32) and takes the label tokens from them. A label inside the 32 gets its exact log-probability; a label outside gets the proxy's fallback value, the lowest returned log-probability minus 5, and every four-task record keeps how many labels came back.

Step 6 — Run and Score

Each model gets a five-example smoke read, the four tasks (300 examples each from sst2, AG News, DAIR Emotion and tweet_eval irony), the three choice tasks with their options reversed, a latency pass at one request at a time, and Bespoke Labs' 3,880-record public suite, rebuilt with Nimble's converters and checked against all 13 published checksums on the VM.

[jev-run 2026-09-24T14:36:51Z] suite: 13 of 13 subsets match
[jev-run 2026-09-24T15:02:42Z] 12b four tasks: 23s for 1200 decisions at concurrency 8
[jev-run 2026-09-24T15:04:25Z] 12b suite: 60s for 3880 records

python3 sweep_score.py --prefix 2026-09-24-v6e1 --arms e2b e4b 12b 26b-fp8
python3 nimble_suite/suite_sweep.py --prefix 2026-09-24-v6e1 --arms e2b e4b 12b 26b-fp8

Both write their tables into results/, and every figure below comes from those files and article_figures.py.

What Fits and Loads on One v6e Chip?

Size	Checkpoint	Serves	Time to serve
E2B	bf16	yes	346 s
E4B	bf16	yes	421 s
12B	bf16	yes	512 s
26B-A4B	fp8, 26.67 GiB	yes	616 s
31B	w4a16, 21.67 GiB	no	fails at 120 s
31B	4-bit, 19.47 GiB	no	fails at 120 s

Both 31B builds stop with the same error:

NotImplementedError: compressed-tensors scheme for layer 'model.language_model.layers.0.self_attn.q_proj' is not yet supported in the JAX path.

Gemma 4 runs only on the JAX path of vLLM's TPU backend, which loads fp8 but no 4-bit format for a dense model. The one 31B format it loads is 30.98 GiB, over one chip's usable memory.

Does the TPU Change the Answers?

No. E2B and E4B were also read on the L4, with the same checkpoints and prompts. The predicted label matched on 1,186 and 1,193 of 1,200 examples, and accuracy per task differed by at most 1.0 and 0.4 points. The 26B-A4B fp8 build here and the 4-bit AWQ build on the L4 are different quantizations of the same model and differ by at most 1.3 points per task and 0.7 points on the suite.

Task	Majority	E2B	E4B	12B	26B-A4B fp8
sst2	51.0%	88.7%	94.0%	95.0%	94.7%
AG News	25.0%	30.0%	83.3%	86.3%	86.0%
DAIR Emotion	35.0%	53.3%	54.3%	59.3%	58.3%
tweet_eval irony	60.3%	73.7%	84.0%	84.7%	91.0%

Per-task calibration, option-order changes and 90th-percentile times are in results/2026-09-24-v6e1-SWEEP.md.

How Do the Sizes Compare With Jev?

The Jev comparison is the L4 article's subject, with its caveats in full. On the same 3,880 public records:

Questions	Jev 1.13.0, published	26B-A4B fp8	12B	E4B	E2B
All, 3,880	77.3%	76.0%	76.2%	72.8%	68.5%
Yes/no, 1,399	84.6%	84.2%	85.1%	81.5%	76.2%
Multiple choice, 1,848	82.8%	79.7%	78.1%	74.5%	70.8%
Five-level rating, 633	45.2%	47.4%	51.0%	48.5%	44.9%
Median ECE after 50 labels (Jev 0.071 as shipped)		0.070	0.070	0.077	0.092

Jev leads 12B by 1.1 points overall (95% range −0.8 to +3.0) and 26B-A4B by 1.3 (−0.6 to +3.2), and on multiple choice by 4.7 and 3.2. The L4's 4-bit 26B scored 75.3%, 2.1 points behind with a range from 0.2 to 4.0, so read the 26B as one to two points behind Jev on either platform. Jev's per-record answers are unpublished, so these ranges compare two independent proportions, and records that share a passage widen them. Gemma was read in the PR proxy's prompt format with no tuning; a different prompt could move the multiple-choice gap in either direction.

Did the Top-32 Read Lose Anything?

Not on accuracy. At least one label came back on every four-task read, so the predicted label is exact; only the probabilities of labels outside the 32 are approximate.

On the four tasks, moving every missing label from the fallback value to the top-32 bound, the most probability it could have had, changes raw ECE by less than 0.001 and ECE after 50 labels by at most 0.005. That check was added after the run. The suite records do not keep how many labels came back, so the suite calibration figures have no such check; by a lower-bound count, at least 10.2% of 12B's suite records and 17.8% of 26B-A4B's used the fallback value.

The share of probability on the allowed labels varies too. On irony, 12B's most likely token was a label on 5 of 300 reads, with a median of 3.1% of its probability on the two labels; 26B-A4B's was a label on 161 of 300 AG News reads, with a median of 55.2%. The read rescales the labels to sum to one, so the answer reads as confident either way, and label_mass in every record shows it.

TPU v6e Against the L4

	One TPU v6e chip	One NVIDIA L4 (`g6.xlarge`)
Memory for the model	🥇 28.74 GiB	23034 MiB
Largest Gemma 4 read here	12B at bf16, 26B-A4B at fp8	26B-A4B at 4-bit
Accuracy, same checkpoint	same answers on 1,186 to 1,193 of 1,200	same
26B time per decision, one at a time	🥇 27 to 33 ms	61 ms
Price per hour, on demand	$2.97	🥇 $0.8048
26B cost per million decisions, on demand	$10.37	🥇 at most $5.43
Label read	top 32 log-probabilities	🥇 exact label token ids
KV cache	fp8 by default	🥇 16-bit

In this table 🥇 marks the better value. The 26B decides 1.8 to 2.2 times faster on the TPU. The chip costs 3.7 times as much per hour on demand, or 1.7 times at the flex-start price of $1.35, a mode that had no capacity during this run. The L4's cost comes from its first run's throughput with the client on a home connection, so it is an upper bound; the TPU's comes from the request times at eight in flight.

🔎 Tip: Serve the Same Checkpoint on Both Platforms First

Before comparing hardware, serve one checkpoint on both with the same prompts and compare predicted labels record by record. It checks the whole read, from chat template to label tokens; here the TPU's top-32 read and the L4's exact read agreed on 1,186 and 1,193 of 1,200 answers.

So, Which One?

For a Jev-style decision service on a TPU, Gemma 4 12B at bf16 plus one temperature fitted on about 50 labels. It matches the 26B on the suite, 76.2% against 76.0%, though the 26B scores 6.3 points higher on irony; it reaches a median ECE of 0.070 after fitting, decides in 26 to 27 ms, and needs no third-party quantized checkpoint. Check label_mass on your own task first: on irony, 12B put a median of 3.1% of its probability on the labels.

Choose the TPU when time per decision matters or the rest of the stack is already on Google Cloud: it halves the 26B's time per decision against the L4. Choose the L4, or Jev itself, when cost per decision matters: on demand, 12B on one v6e chip costs $8.19 per million decisions, against Jev's $5.54 and at most $5.43 for the 26B on an L4.

What Does It Cost?

Measured from the request times at eight in flight, one v6e chip answers about 274 decisions a second with E2B, 206 with E4B, 101 with 12B and 80 with 26B-A4B. At the europe-west4 on-demand price of $2.97 an hour that is $3.01, $4.00, $8.19 and $10.37 per million decisions; at the flex-start price of $1.35, $1.37, $1.82, $3.72 and $4.71. TypeSafe prices Jev at $5.54 per million decisions at the L4 run's median prompt of 132 tokens. The chip is charged by the hour whether busy or idle, and eight requests in flight is below the 16 the server allows, so these figures hold only while the chip is kept at least this busy.

The measured run took 48 minutes of on-demand time, $2.39, and all instance time for the measurement came to at most $3.25.

Teardown

The VM deletes itself when tpu/run.sh finishes, and --max-run-duration 6h with --instance-termination-action DELETE is the backstop. Confirm it is gone:

gcloud compute instances describe jev-tpu-v6e1 --zone europe-west4-a

ERROR: (gcloud.compute.instances.describe) Could not fetch resource:
 - The resource 'projects/aisprint-491218/zones/europe-west4-a/instances/jev-tpu-v6e1' was not found

Summary

The goal of this article was to run a Jev-style decision model on one TPU v6e chip with Gemma 4 and find what changes from a GPU. The key to the solution was the same prompts, labels and scoring as the L4 run, a pre-registered design, and a label read taken from the top 32 log-probabilities. The results were:

🟢 One v6e chip serves Gemma 4 E2B, E4B and 12B at bf16 and a 26B-A4B fp8 build
❌ No 31B checkpoint loads: the w4a16 and 4-bit builds fail on the JAX path, and the fp8 build is 30.98 GiB
🟢 The same checkpoints give the same answers on the TPU and the L4: 1,186 and 1,193 of 1,200
🟢 The 26B decides in 27 to 33 ms on the TPU against 61 ms on the L4
⚠️ On demand, 26B costs $10.37 per million decisions on the TPU against at most $5.43 on the L4 and $5.54 for Jev
⚠️ vLLM on TPU returns only the top-k log-probabilities; reading labels from the top 32 changes no answer, and changes four-task calibration by at most 0.005
🟢 12B matches the 26B: 76.2% and 76.0% on the 3,880-record suite, against Jev's 77.3%
🟢 After 50 labels, 12B and 26B-A4B reach a median ECE of 0.070, against Jev's 0.071 as shipped
🟢 The whole measurement cost at most $3.25 of instance time

Scope: one TPU v6e chip (ct6e-standard-1t, on demand) in europe-west4-a, vLLM 0.29.1rc1.dev468+g0b7f11a1e at the image digest above, E2B, E4B and 12B with bf16 weights and 26B-A4B through RedHatAI's fp8 build, every model with vLLM's default fp8 KV cache on v6e, --max-model-len 2048, one run per model, 300 examples per task plus the 3,880-record public suite, client on the VM. The top-32 read and on-demand capacity are dated deviations from the pre-registration, and the top-32 bound check, added after the run, covers the four tasks only. L4 figures come from the companion run on a g6.xlarge and g6.4xlarge in us-east-1, and the two platforms were timed with different clients. Every source dataset was published before Gemma 4 and may be in its training data. No Jev call was made: the Jev figures are Bespoke Labs' published results on the same records, from one run of Jev 1.13.0 by a company that sells a competing model, counting an invalid Jev response as wrong, with Jev's probabilities rounded to two decimals by its API. Parts of the analysis and writing were done with AI assistance (Claude); every figure comes from the committed output files.

The strategy for running a Jev-style decision model on TPU with Gemma 4 was validated with an incremental step by step approach.

References

Code, pre-registration and per-item results: https://github.com/xbill9/gemma4-dev/tree/main/jev-tpu
Companion L4 measurement: https://dev.to/gde/plain-gemma-4-26b-vs-jev-on-one-ec2-l4-21-points-behind-overall-level-on-yesno-45-behind-on-15k6
Companion review of the independent evidence on Jev: https://dev.to/gde/jev-after-eight-days-of-independent-tests-level-with-mid-price-llms-behind-the-frontier-1kln
vLLM TPU documentation: https://docs.vllm.ai/projects/tpu/en/latest/
tpu-inference: https://github.com/vllm-project/tpu-inference
26B-A4B fp8 checkpoint: https://huggingface.co/RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic
Gemma 4 12B: https://huggingface.co/google/gemma-4-12B-it
Bespoke Labs, public-suite results for Jev 1.13.0: https://github.com/bespokelabsai/nimble/blob/0e67403/docs/PUBLIC_BENCHMARKS.md
Cloud TPU v6e: https://cloud.google.com/tpu/docs/v6e
Guo et al., On Calibration of Modern Neural Networks: https://arxiv.org/abs/1706.04599

Jev After Eight Days of Independent Tests: Level With Mid-Price LLMs, Behind the Frontier

xbill — Thu, 24 Sep 2026 02:00:52 +0000

This article provides a review of the independent evidence on TypeSafe's Jev, the open models built to replace it, and the prior art behind both, as of September 23, 2026. Every figure below is traced to a primary source, and re-scored from committed per-item outputs wherever the author published them. This is a snapshot eight days after launch, and the arXiv preprints it cites are days old and unrefereed.

On accuracy, Jev sits level with mid-price LLMs and 6.5 to 11.5 points behind the frontier in the cleanest comparison. Out of the box its probabilities are the best calibrated of the models measured on familiar English tasks and are wrong in both directions off them, and one temperature fitted on 50 to a few hundred labels fixes most of the error.

https://github.com/xbill9/gemma4-dev/tree/main/jev

What Is Jev?

Jev is a hosted model from TypeSafe AI, launched on September 15, 2026. It answers typed questions about a piece of text: choice picks one option from a list, score places the text on an ordered scale, and noul returns the probability that a statement is true. It writes no text.

The API is POST /v1/systemone at api.typesafe.ai. The price is $0.042 per million input tokens, and output is free. Access is through a waitlist, OpenRouter and Vercel's AI Gateway.

TypeSafe calls this category a System One model, after Kahneman's fast and intuitive system. The company raised a $40M seed round led by DCVC, and its co-founders are Diogo Almeida (CEO), Sasha Sheng (COO) and Erik Gafni (CTO).

The read itself is an old technique: end the prompt where the answer would start, take the model's scores for only the allowed label tokens, and apply a softmax over them. Any open model can be read this way; vLLM exposes it through logprob_token_ids, SGLang through /v1/score, and Featherless AI's SimpleJev wraps it as a Jev-compatible server.

What Counts as Evidence Here?

The review covers 14 arXiv preprints read in full, 104 GitHub evaluation repositories, 33 dev.to and Medium posts, TypeSafe's own pages and documentation, and press and forum coverage. Each figure carries one of three levels of support.

Re-scored: recomputed from committed per-item outputs, and it matched
Checked: the source's arithmetic recomputed from its published tables, with no per-item data available
Reported: the author's statement, which cannot be checked from outside

Studies are graded A (human or public labels, at least 200 items, stated method, committed outputs), B (real but small, synthetic, LLM-labelled or unreproducible) or C (a demo or a restatement).

By that rubric, the posts split sharply. Of 33 dev.to and Medium posts read in full, 5 are careful measurements on human or public labels, 11 are small, synthetic or scored by agreement between models, and 17 restate vendor claims. Of the 104 GitHub repositories, 33 grade A, 57 grade B, 5 grade C, and 9 are tools. Of the 14 arXiv preprints, 10 call TypeSafe's hosted Jev.

Who ran a study matters too. The studies with committed per-item outputs and no product in the race are Ibrahim and Zaki, Janardhan, sanand0, SamuelSacco and OmarMujahid, the last the most favourable to Jev; Rafe and Das also has none, and withholds its per-item outputs. Every's review was written by a launch partner. LangChain, which ships the langchain-typesafe integration, reported 100% on 500 judgments that are 5 test cases repeated 100 times (not re-checked). And the "15.9% faster pipeline" headline came from a TypeSafe employee's three-case demo. Bespoke Labs sells Nimble, and the replica authors test their own models. Matt Mastracci wrote the DiffusionGemma PR he compares with Jev, and Google's Gemma account shared his thread. heiko-hotz, whose SNIPS comparison includes Gemma 4 31B, works at Google. synthorai, which found Jev behind flash LLMs, runs a gateway that sells access to the models it compared.

Where Does Jev Land on Accuracy?

Level with mid-price LLMs and behind the frontier, at a fraction of the frontier's price. That trade is the product's design, and the numbers below show its size.

The largest pre-registered study, Ibrahim and Zaki's replication of a social-science annotation suite with 7,977 human-labelled items (arXiv 2609.24574), has Jev behind the best of 19 LLMs on 14 of 15 tasks, by a median 11.6 macro-F1 points. That comparator is the best model per task, picked after the results; within Jev's price band, Gemma 4 31B and Qwen3 235B score 3 and 2 F1 points above it. Its one per-item file checked, the 266-item dialect task, re-scores exactly to the paper's table.

Manjunath Janardhan's 200-item, six-model comparison on BANKING77, BoolQ, Yelp and ChaosNLI re-scores exactly from its committed logs.

Model	Accuracy	Calibration error (ECE)
🥇 Claude Fable 5.1	84.0%	0.064
🥈 GPT-6 Astra	79.0%	0.119
🥉 DeepSeek V4.1 Flash	76.0%	0.138
MiniMax M3	75.5%	0.112
Kimi K3	74.5%	0.119
Jev	72.5%	0.161

The two frontier leads (+11.5 and +6.5 points) exclude zero in their 95% intervals. Kimi K3, MiniMax M3 and DeepSeek V4.1 Flash sit within noise of Jev.

Banking77 is the one task many authors ran independently. Eight runs, one of them JevBench's own first-party run, put Jev at 0.753 to 0.840, with a median of 0.809. In every study that ran LLMs beside it, the best of them was ahead.

What Does Jev Do Well?

Binary and few-class decisions, fast, with answers that always fit the schema.

Spam: 98.33% on 18,514 emails, level with TF-IDF logistic regression at 98.39%, and it held up on newer mail where the trained baseline fell to 72.5%; the question wording was tuned on labelled errors from the same sets
Reranking: level with a dedicated reranker on eight retrieval sets, 0.692 against 0.691
Monitoring code for backdoors: AUROC of 0.976 and 0.970, catching about 90% of backdoors at 2% false positives; a best-of-5 attack cut detection to 60%
Schema compliance: zero invalid answers across 23,703 decision-model calls, Jev's included, in the largest study
Repeatability: 1.33% of answers changed between identical passes in one study, 2.2% in the phishing benchmark, and 0 of 96 repeats in another
Language: German cost 0.5 points on MASSIVE
Speed and price: about 0.1 s of server time, at $0.042 per million input tokens

Are Its Probabilities Calibrated?

They rank answers usefully; out of the box they beat most alternatives on familiar English tasks and miss badly off them.

TypeSafe publishes no calibration error, reliability plot, Brier score or log loss for Jev on any dataset. Its only confidence-to-accuracy figure is one cookbook of 60 SEC filings: 90% right at confidence 0.9 or above, 40% below.

The independent picture depends on the comparison.

On Bespoke Labs' 13-subset public suite, Jev's median ECE is 0.071, lower than Nimble-9B's on 11 of 13 subsets, and lower than every Gemma 4 read in the companion article (0.114 to 0.180) before any fitting.
Against LLMs that write their confidence as a number, Jev's own probabilities usually win. Its median ECE of 0.157 on the social-science tasks beats 16 of 19 LLMs.
Once every model gets one fitted temperature, 15 of those LLMs beat it.
Against LLMs that return full probability distributions, Jev's calibration error is the highest in Janardhan's run (0.161) and in sanand0's nine-model Banking77 pilot (0.138).

The direction of the error changes with the data. On public multi-class sets Jev is overconfident: on GoEmotions, labels it scored between 0.80 and 0.95 matched the human label 15% of the time. On crash narratives and synthetic items it is under-confident. Within one judge study, refit temperatures ran from 0.65 to 4.45, and its authors conclude that no single temperature fits.

The API also rounds every probability to 0.01. In one committed sample, 70.4% of choice probabilities came back as exactly 0, and noul is clamped between 0.01 and 0.98.

🔎 Tip: Fit a Temperature on 50 to a Few Hundred of Your Own Labels

A small refit fixes most of the calibration error, for Jev and for any open model read the same way.

A two-parameter refit on civil_comments took Jev's ECE from 0.16–0.21 to under 0.025 on a held-out half
Transferring a fitted slope and refitting only the intercept on 50 labels cut mean ECE by 74%
Out-of-fold Platt scaling cut Jev's ECE on crash narratives by 3.35x
In the companion article, one temperature fitted on 50 labels brought plain Gemma 4 26B's median ECE from 0.180 to 0.080 on the public suite, against Jev's 0.071 as shipped

The open replicas report the same order of correction: Nimble fitted 2.179, Luce 2.5 to 3.3 on unseen tasks. Luce's rule is to ship with 100 to 300 real labelled items and fit the temperature on them.

How Much Faster and Cheaper Is It?

Speed runs from 0.5x, slower than a local Gemma, to 12.1x faster; cost runs from 0.6x, dearer, to 478x cheaper, depending on the comparison model, the network path and where the clock runs.

Comparison	Source	Faster by	Cheaper by
GPT-5.6 Luna, reasoning off, client clock	OmarMujahid	1.04x	4.3x
DeepSeek, Banking77	sanand0	—	1.18x
Haiku 4.5, phishing verdict	jev-phishing-bench	2.9x	12x
Mid-size open model, 47 account reviews	devopsdaily	12.1x	7.4x
Claude Fable 5.1, via OpenRouter	Janardhan	9.9x	478x
19 LLMs, social-science tasks (median)	arXiv 2609.24574	—	18.6x
DeepSeek, 6G edge orchestration	arXiv 2609.23136	1.3x	0.6x (Jev dearer)
DeepSeek, edge orchestration, fees per correct completion	arXiv 2609.22753, same lab	—	3.2–3.4x
Local Gemma 4 26B-A4B Q4 on a mini-PC	ikkun1222	0.5–0.65x (Jev slower)	—

The server itself is fast: about 105 ms in one run, and about 76 ms once the network round trip is subtracted in another. Most of the cost gap comes from output being free. In devopsdaily's workload 83% of the old bill was output, and per input token the two prices differed by 1.31x. In the companion article, plain Gemma 4 26B on one EC2 L4 at full load cost at most $5.43 per million decisions, against Jev's $5.54 at the same 132-token prompts.

Where Does It Break?

On wording, language, and questions whose answer is absent from the input.

Option names move answers. With the question, input, rubric and option set held fixed, swapping which rubric sits behind "no" and "yes" changed 32.5% of Jev's answers, against about 2% with neutral names, and AUROC fell from 0.81 to 0.58 (arXiv 2609.26758, n = 1,200; reported). The paper is graded C as evidence about Jev: it names no dataset or Jev version, its gold labels likely come from a teacher LLM, its code is unreleased, and no generative LLM was given the same swap.

Bare labels mislead it. A router given option names with no descriptions sent all 40 hard tasks to the cheap model, at a median confidence of 0.96. One-line option descriptions fixed 37 of 40.

Language shift costs accuracy. Russian XNLI dropped from 88.3% to 77.3% with ECE tripling, and Spanish cost 3 to 6 points.

Jev is most confidently wrong where the answer does not follow from the input. With no "unknown" option, every answer on KoBBQ bias questions is wrong by design, and Jev gave them 0.79 confidence. On a fair die its choice probabilities averaged 83% against a true 17%, while noul gave 19%. On heart-risk data its probabilities ran about 3x too high, and a chat LLM on the same 5,000 people ranked risk slightly better (AUC 0.79 against 0.77).

Questions batched in one request cannot see each other's answers. One reviewer's request returned "suspend" for an account the same response classified as a developer sending tests.

TypeSafe's own list of known weaknesses names literal reading, arithmetic and counting, dates, long irrelevant input, adversarial content and phrasing: "refund" scored 0.72 and "not a refund" 0.47 on the same ticket.

What Does TypeSafe Claim, Checked?

193.6x faster and 444.6x cheaper. The source is TypeSafe's four in-house workflow evals, which the launch post calls "on the higher end of real world gains". TypeSafe discloses that the workflows "were made by individuals on our model capabilities team, so some bias could exist", and that its LLM comparators ran through its own probability adapter, which it says is "slower and more expensive". The comparator is never named. Checked against the published summary points, 444.6x fits only Jev against Opus 5 in the workflow setup. Averaged over all eight workflow setups, the figures are 97.8x faster and 149.2x cheaper. Timings came from laptops on the US West Coast, with no case counts, run counts or variance.

Eval accuracy. Reference labels are "an average of the responses of GPT-6 Astra and Claude Fable 5.1, both at high thinking", so accuracy there means agreement with two LLMs. Jev scores 67.8%, equal to Sonnet 5, with GPT-5.6 Sol leading at 74.1%. On invoice processing Jev scores 61.8% against Sol's 79.1%. TypeSafe states that it chose to publish no public-benchmark results.

"0% hallucination". TypeSafe's launch post: "Our number is not empirical. Schema matching is guaranteed." Every answer is a valid option; the wrong-but-valid rate is the accuracy figures above.

RLCD. Reinforcement learning for calibrated decisions has no paper, patent or method description. The CEO said on Hacker News that the architecture is "close to the chest for now, but we have talked about writing a paper." On Latent Space he agreed that all of Jev's training data is synthetic. Model size and base are undisclosed.

Pricing. The launch post says "We can't prove it isn't subsidized", and the home-page FAQ says "We can serve Jev profitably at our current prices." How long the launch price holds is an open question for anyone building on it.

The frontier comparison. On Hacker News a commenter described Jev as "basically a zero-shot classifier" that classifies "as accurately (they claim) as a frontier-level LLM", and the CEO replied "exactly right!" The independent accuracy results above are the test of the second half.

The founder. TypeSafe's pages say Diogo Almeida "co-invented RLHF". He is not an author of Christiano et al. (2017), which introduced it; he is a primary author of InstructGPT (arXiv 2203.02155), which applied it to instruction following, and on Latent Space he separates the two.

How the claims travelled:

Claim as circulated	Primary source
194x faster and 445x cheaper against GPT-6 Astra (Tom's Hardware)	Astra is a reference labeller with no scored setup; 444.6x fits Opus 5
Coverage of arXiv 2609.26758: Jev flips 70.4 of 100 answers	The paper gives 70.4 for an open ModernBERT head; Jev's is 32.5%
"Its own employee measured 15.9% on a real pipeline" (Cherry Creek News)	3 cases of a demo support ticket; costs modelled from an estimated price

Several self-published evaluations also state more than their own data supports, in both directions. The full report lists the corrections for Luce, Laya and NanoJev and uses the corrected figures from every study.

What Predates Jev?

Every piece of the mechanism.

Piece	Prior art
Score options by label likelihood	GPT-3 §2.4 (2020); MMLU §4.1 (2021); lm-evaluation-harness `multiple_choice`
Map each label to one token	Verbalizers, Schick & Schütze (arXiv 2001.07676)
Label probabilities carry prior bias	Calibrate Before Use, Zhao et al. (2021)
One fitted temperature fixes overconfidence	Guo et al. (2017); Kadavath et al. (2022), a temperature of 2.5 for an RLHF policy, beside Jev's 2.66, Nimble's 2.179 and Luce's 2.5 to 3.3
Post-training worsens calibration	GPT-4 report, Fig. 8: MMLU ECE 0.007 to 0.074 after PPO
Cross-entropy rewards reporting true probabilities	Gneiting & Raftery (2007): the log score is strictly proper
Cheap model first, defer when unsure	Selective classification (2017); FrugalGPT

The visible contribution is the packaging: a typed API, one hosted endpoint, output priced at zero, and many questions evaluated against one input in a single request. Whether RLCD adds more can be judged once it is published; Jev's native probabilities beating most LLMs' self-reported confidence suggests the training does something.

What About the Open Alternatives?

Each trained replica is built on Qwen or an encoder, and each publishes its own evaluation.

Project	Base	Own headline	Independent evidence
Nimble-9B (Bespoke Labs)	Qwen3.5-9B + LoRA	75.9% vs Jev 77.3% on 13 public sets	none; run by the seller, with raw probabilities (fitted temperature 2.179); its own model trails Jev
Kev (Jared Palmer)	Qwen3.5 + LoRA + pointer head	Kev-9B 0.852 test vs Jev 0.857 dev	none
Luce (scienthoon)	Qwen3-4B + LoRA	86.9% vs Jev 84.7% on 500 kubernetes issues	none
Laya (Convai)	ModernBERT-large 421M	0.766 vs 0.727 after training on the benchmark	ikkun1222: 29.2 / 60.0 / 40.0 on three Japanese tasks; the card's own zero-shot 0.362 sits below its 0.461 majority baseline
SimpleJev (Featherless)	any HF model; demo serves Gemma 4 26B	no accuracy published	none
DiffusionGemma via vLLM PR #57250	one denoise step of `diffusiongemma-26B-A4B-it`	198/201 vs Jev 191/201, "roughly tied"	ywchiu: 32.2% vs Jev 61.4% on synthetic routing; companion article: 75.9% vs Jev 77.3% on the public suite

SimpleJev's README states that its probabilities "are not calibrated probabilities of correctness". The DiffusionGemma headline comes from 201 items built by the PR's author; publishing the items would let others reproduce it.

Labels beat zero-shot wherever they were tried. A 310M Japanese encoder trained on 200 rows beat Jev on news topics by 12 points, 88.8% against 76.8%, and tied it on two sentiment tasks, and fine-tuned models led by 2 to 15 points on five public splits. On phishing, Luce trained on 1,000 labels reached 97.4% against Jev's 62.6% on the same benchmark, though on different items. On that benchmark, a two-line regex scored 91.6%, five narrow Jev questions combined by logistic regression reached 95.0% on a held-out half, and Haiku 4.5 asked the same five questions reached 93.2%, a difference too small to be significant (p = 0.063).

thejevai.com, registered on September 20 and promoted on Hugging Face as the endpoint for API keys, has no stated connection to TypeSafe. TypeSafe's API is api.typesafe.ai.

Compare and Contrast

	Jev	Trained open replica	Plain open model, label scores read
Labels needed	none to run; about 50 to fit a temperature	hundreds to thousands, plus a training run	none to run; about 50 to fit a temperature
Accuracy on human labels	level with Kimi K3, MiniMax M3 and DeepSeek V4.1 Flash; 6.5 to 11.5 behind the frontier	trails or matches Jev generically; beats it when trained on the task's own labels	Gemma 4 26B, 4-bit: 75.3% vs Jev 77.3% on the 3,880-record suite; level on yes/no, 4.5 behind on multiple choice
Calibration	🥇 median ECE 0.071 as shipped on the public suite; none published by the vendor	own evaluations	median ECE 0.180 as shipped, 0.080 after 50 labels
Where it runs	TypeSafe, OpenRouter, Vercel	your hardware	your hardware
Cost	$0.042 per million input tokens; $5.54 per million decisions at 132 tokens	your hardware	at most $5.43 per million decisions on an L4 at full load

So, Which One?

For binary and few-class decisions in English, at high volume, with no infrastructure to run and no labels yet, Jev is fast and cheap, its accuracy sits with mid-price LLMs, and its probabilities are the best calibrated out of the box on familiar English tasks. Plan on labelling 50 to a few hundred decisions to set thresholds and fit its probabilities.

For multi-class work with many options, decisions where the rule lives outside the input, or non-English text, where Jev loses accuracy, a model trained on your own labels won wherever one was tried; for non-English that evidence is one Japanese study.

For data that cannot leave your network, any open model read by label scores gives the same typed interface. In the companion article plain Gemma 4 26B trails Jev by 2.1 points overall and 4.5 on multiple choice, and needs about 50 labels to come within 0.01 of Jev's calibration.

What Is Still Unmeasured?

The published Gemma data points so far:

Gemma 4 31B beats Jev's median F1 in the pre-registered social-science study, 0.611 against 0.581
Gemma 4 31B scores 98.86% on SNIPS against Jev's 97.14% after its output's code fences were stripped; 80.57% before
Gemma 4 26B-A4B trails Jev by 11.6 points on Japanese news topics and ties it on two sentiment tasks
Gemma 4 E4B, stating its confidence as text, scores 81.0% against Jev's 95.5% on synthetic emails
Plain Gemma 4 26B read by its label probabilities, in the companion article: on four public tasks (sst2, AG News, DAIR Emotion, tweet_eval irony) and on Bespoke Labs' 3,880-record, 13-subset suite, 2.1 points behind Jev overall

Still unpublished: Jev's behaviour across model versions, calibration across difficulty on real data, and whether a generative LLM shows the same option-name sensitivity.

Summary

The goal of this article was to establish what independent evidence shows about TypeSafe's Jev and its open alternatives. The key to the solution was tracing every figure to its primary source and re-scoring from published per-item outputs wherever they existed. The results were:

🟢 Jev returns a valid option on every call, and its server time is about 0.1 s
⚠️ Accuracy sits with mid-price LLMs: 72.5% against 74.5–76.0% for Kimi K3, MiniMax M3 and DeepSeek V4.1 Flash in one six-model study, and a median 11.6 F1 behind the best LLM per task in the largest pre-registered one
🟢 Strong on binary and few-class work: 98.33% on 18,514 emails, and level with a dedicated reranker
🟢 Best calibrated out of the box on familiar English tasks: median ECE 0.071 on the 13-subset public suite
🟢 50 to a few hundred of your own labels and one fitted temperature fix most of its calibration error, and most of an open model's
⚠️ Measured speed gains run from 0.5x to 12.1x and cost gains from 0.6x to 478x; TypeSafe's 193.6x and 444.6x have no named comparator
⚠️ No calibration metric is published by TypeSafe, and the direction of its calibration error changes by domain
⚠️ Swapping option names changed about a third of its answers in one unreplicated study
⚠️ Plain Gemma 4 26B read by label probabilities (companion article) trails Jev by 2.1 points on the 3,880-record suite, is level on yes/no, 4.5 behind on multiple choice, and needs 50 labels to come within 0.01 of Jev's calibration
❌ RLCD has no published method, and the model's size, base and training data are undisclosed

Scope: sources read on September 23, 2026, eight days after launch: 14 arXiv preprints, 104 GitHub repositories, 33 dev.to and Medium posts, and TypeSafe's own pages; the companion article's public-suite run finished on September 24 UTC. Every accuracy, calibration and timing figure here is a third party's measurement, marked in the full report as re-scored, checked or reported, except the Gemma figures in Compare and Contrast and the Summary, which are the author's own measurement in the companion article; that article's Jev figures come from Bespoke Labs, which sells a competing model. Every run that states a version used jev-1.13.0. Sources were gathered and audited with AI assistance (Claude), and every figure was checked against its source, cited in the full report, except LangChain's, marked not re-checked there. The author is a Google Developer Expert and an AWS Community Builder; neither Google nor AWS funded, reviewed or saw this work. The author has no relationship with TypeSafe, Bespoke Labs or any replica reviewed.

The strategy for using primary sources to evaluate Jev was validated with an incremental step by step approach.

References

Full evidence review with every source: https://github.com/xbill9/gemma4-dev/blob/main/jev/reports/Jev%20independent%20evidence%20review.md
Companion measurement article: https://dev.to/gde/plain-gemma-4-26b-vs-jev-on-one-ec2-l4-21-points-behind-overall-level-on-yesno-45-behind-on-15k6
TypeSafe launch post: https://typesafe.ai/blog/introducing-system-one-models-and-jev
TypeSafe models and pricing: https://docs.typesafe.ai/models
TypeSafe evals: https://evals.typesafe.ai/
Jev 1.13 known weaknesses: https://docs.typesafe.ai/model-jaggedness/jev-1.13
Ibrahim & Zaki, arXiv 2609.24574: https://arxiv.org/abs/2609.24574
Rafe & Das, arXiv 2609.24052: https://arxiv.org/abs/2609.24052
Li et al., arXiv 2609.26550: https://arxiv.org/abs/2609.26550
Option naming, arXiv 2609.26758: https://arxiv.org/abs/2609.26758
Janardhan, jev-frontier-bench: https://github.com/manjunathshiva/jev-frontier-bench
Nimble public benchmarks: https://github.com/bespokelabsai/nimble/blob/main/docs/PUBLIC_BENCHMARKS.md
jev-phishing-bench: https://github.com/anisselbd/jev-phishing-bench
Luce: https://github.com/scienthoon/luce
OmarMujahid, jev-decision-bench: https://github.com/OmarMujahid/jev-decision-bench
bitnovus, jev-spam-eval: https://github.com/bitnovus/jev-spam-eval
SamuelSacco, jev-exploration: https://github.com/SamuelSacco/jev-exploration
sanand0, BANKING77 pilot: https://sanand0.github.io/llmevals/jev/
heiko-hotz, jev-evaluation: https://github.com/heiko-hotz/jev-evaluation
ywchiu, jev_benchmark: https://github.com/ywchiu/jev_benchmark
ikkun1222, Jev against a 310M encoder: https://dev.to/ikkun1222/jev-vs-a-310m-encoder-i-trained-myself-750-rows-three-tasks-two-different-winners-242e
SimpleJev: https://github.com/featherless-ai/simple-jev
vLLM PR #57250: https://github.com/vllm-project/vllm/pull/57250
Matt Mastracci, Jev against DiffusionGemma: https://x.com/mmastrac/status/2100626193943052784
Guo et al., On Calibration of Modern Neural Networks: https://arxiv.org/abs/1706.04599
Kadavath et al., Language Models (Mostly) Know What They Know: https://arxiv.org/abs/2207.05221
Brown et al., GPT-3: https://arxiv.org/abs/2005.14165
Hendrycks et al., MMLU: https://arxiv.org/abs/2009.03300
Schick & Schütze, verbalizers: https://arxiv.org/abs/2001.07676
Zhao et al., Calibrate Before Use: https://arxiv.org/abs/2102.09690
Gneiting & Raftery, Strictly Proper Scoring Rules, Prediction, and Estimation, Journal of the American Statistical Association 102 (2007)
Geifman & El-Yaniv, Selective Classification: https://arxiv.org/abs/1705.08500
Chen et al., FrugalGPT: https://arxiv.org/abs/2305.05176
GPT-4 technical report: https://arxiv.org/abs/2303.08774
InstructGPT: https://arxiv.org/abs/2203.02155
Christiano et al. 2017: https://arxiv.org/abs/1706.03741

Plain Gemma 4 26B vs Jev on One EC2 L4: 2.1 Points Behind Overall, Level on Yes/No, 4.5 Behind on Multiple Choice

xbill — Thu, 24 Sep 2026 02:00:37 +0000

This article provides a step by step guide to measuring Gemma 4 26B as a Jev-style decision model on an AWS EC2 L4 GPU, and compares a plain read of its label probabilities against DiffusionGemma's one-step read and against Jev's published results. The measurement was pre-registered, and every per-item output is committed.

On Bespoke Labs' 3,880-record public suite, plain Gemma 4 26B trails Jev 1.13.0 by 2.1 points overall, shows no measurable difference on yes/no questions, and trails by 4.5 points on multiple choice. Jev is better calibrated out of the box, and one temperature fitted on 50 labels brings Gemma's median calibration error within 0.01 of Jev's. Against DiffusionGemma, the plain read is level on accuracy, worse calibrated before fitting and similar after, and 1.9 to 5.1 times faster per decision.

https://github.com/xbill9/gemma4-dev/tree/main/jev

Why Measure This?

A Jev-style decision model answers a typed question with a probability for each allowed option, in one forward pass, with no generated text. TypeSafe's Jev does this as a hosted service. Any open model can do it: end the prompt where the answer starts, read the scores of the allowed label tokens, and apply a softmax over those.

A companion review of the independent evidence on Jev found two open questions for Gemma. Plain Gemma read this way had no published accuracy or calibration result. DiffusionGemma, read through vLLM PR #57250, had been described by Google's Gemma account on September 18 as "yielding well-calibrated decision distributions", with no published measurement behind it.

This run answers both on the same GPU, the same prompts, the same label tokens and the same scoring code, using community 4-bit (AWQ) builds of both models. A third run puts both, and Gemma 4 E4B, on the public suite where Bespoke Labs has published results for Jev, so the Gemma numbers sit beside Jev's on identical records.

At This Point You Should Have…

An AWS account with quota for one G-family instance, and the AWS CLI logged in
A security group, subnet and instance profile with AmazonSSMManagedInstanceCore for remote commands
Python 3 with transformers and pybase64 for the client side
The repository cloned: git clone https://github.com/xbill9/gemma4-dev and cd gemma4-dev/jev

Step 1 — Pre-Register the Measurement

The models, image, serving flags, data, metrics and comparisons are written down and committed before any model call. The file is PREREGISTRATION.md.

git show -s --oneline 3c67f62 d94471d a5cda09 894323e adf0c16

3c67f62 jev: pre-registration for the L4 AWQ run; option-order variant, label-count calibration curve, EC2 user-data
d94471d jev: run 2026-09-23-l4-awq — Gemma 4 26B label logits vs DiffusionGemma one-step reads, both AWQ 4-bit on one EC2 L4; per-item outputs, summary, host evidence
a5cda09 jev: pre-registration addendum (latency pass on-instance, off-label tokens, E2B/E4B bf16 arms); 20-split calibration analysis; archived run log and throughput arithmetic
894323e jev: plain arm builds its prompt from the served model's own chat template (26B unchanged, checked); E4B/E2B deviation recorded
adf0c16 jev: Bespoke Labs public-suite run — rebuild with checksum checks, runner via the proxy's Jev parser, scorer matched to Nimble's definitions, instance driver, generic launcher; pre-registration addendum

The latency and small-model runs were added in one pre-registration addendum after the first run, and the public-suite run in another; each was committed before its own first call, and each deviation from the plan is recorded there. The pre-registration commits to publishing every result, including any where either arm does worse.

Step 2 — Pick a Matched Pair of Checkpoints

An NVIDIA L4 has 24 GB of memory, so both 26B models run at 4 bits. Both checkpoints come from the same uploader and apply the same quantization settings to both models, so what differs is the model and how it is read.

	Plain arm	Diffusion arm
Checkpoint	`cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit`	`cyankiwi/diffusiongemma-26B-A4B-it-AWQ-INT4`
Format	compressed-tensors, 4-bit, group 32, symmetric	same
Size	17.19 GB	17.22 GB

Both arms share one set of serving flags.

Step 3 — Launch One L4 on EC2

The security group opens port 8000 to one address only.

aws ec2 create-security-group --region us-east-1 --group-name jev-eval-sg \
  --description "jev eval: vLLM 8000 from one IP" --vpc-id <vpc-id> --query GroupId --output text
aws ec2 authorize-security-group-ingress --region us-east-1 --group-id <sg-id> \
  --protocol tcp --port 8000 --cidr <your-ip>/32

The instance boots the regional GPU Deep Learning AMI and runs aws/user-data.sh, which pulls a vLLM nightly that contains PR #57250 and serves one model at a time.

aws ec2 run-instances --region us-east-1 --image-id <gpu-dlami> --instance-type g6.xlarge \
  --subnet-id <subnet-id> --security-group-ids <sg-id> \
  --iam-instance-profile Name=<profile> --user-data file://aws/user-data.sh \
  --block-device-mappings 'DeviceName=/dev/sda1,Ebs={VolumeSize=100,VolumeType=gp3,DeleteOnTermination=true}' \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=jev-eval},{Key=ManagedBy,Value=jev}]'

i-0c5b14e913b4c1019 g6.xlarge   us-east-1a

A g6.xlarge has the same L4 as the larger sizes with 16 GB of host memory, so user-data.sh adds a 16 GB swapfile on hosts under 30 GB; both 17 GB checkpoints loaded with it in place.

Step 4 — Serve the Plain Arm

One flag set serves both arms: Triton attention, eager mode, prefix caching, 2048-token context, 16 sequences and 32 logprobs. The diffusion arm adds a 64-token canvas.

/opt/jev/serve.sh ar

The client waits on /v1/models until the model answers.

READY after ~480s
['cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit']

That covers the image pull, the 17 GB download and the load. The host record from the instance:

NVIDIA L4, 595.91.07, 23034 MiB
image: vllm/vllm-openai:nightly-e9757321527ca1ecd514c07c1418dd2c53da3d19 digest: sha256:f75fec992c293dd41042745f8aa71bfdf12b188b6a01e0baf075e4bca5d297e5
vllm 0.29.1rc1.dev573+ge97573215 torch 2.13.0+cu130

Step 5 — Check the Read Before the Full Run

Both arms use the prompt, answer template, label tokens and probability code from the PR's own structured_server.py. run_eval.py confirms both chat templates render the identical prefix before it sends anything. A five-example run checks the path end to end.

python3 run_eval.py --arm autoregressive --upstream http://<host>:8000 \
  --model cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit --run smoke --limit 5 --concurrency 2

autoregressive sst2: 5/5 (5s)
autoregressive ag_news: 5/5 (1s)
autoregressive emotion: 5/5 (0s)
autoregressive irony: 5/5 (0s)

Each record stores the label probabilities, labels_returned, which confirms every label came back with a real probability, and label_mass, the share of the model's probability that landed on the allowed labels.

Step 6 — Run Both Arms

The data is 300 labelled examples each from sst2, AG News, DAIR Emotion and tweet_eval irony, stratified with a fixed seed. sst2 and irony carry annotators' labels, AG News its source's news categories, and DAIR Emotion labels taken from the hashtags on each tweet. Each arm runs once as written, and the three choice tasks run once more with their options listed in reverse order.

python3 run_eval.py --arm autoregressive --upstream $U --model $PLAIN --run 2026-09-23-l4-awq --concurrency 8
python3 run_eval.py --arm autoregressive --upstream $U --model $PLAIN --run 2026-09-23-l4-awq --concurrency 8 --variant reversed
/opt/jev/serve.sh diffusion
python3 run_eval.py --arm diffusion --upstream $U --model $DIFF --run 2026-09-23-l4-awq --concurrency 4
python3 run_eval.py --arm diffusion --upstream $U --model $DIFF --run 2026-09-23-l4-awq --concurrency 4 --variant reversed
wc -l results/2026-09-23-l4-awq/diffusion-*.jsonl

    300 results/2026-09-23-l4-awq/diffusion-ag_news.jsonl
    300 results/2026-09-23-l4-awq/diffusion-ag_news--reversed.jsonl
    300 results/2026-09-23-l4-awq/diffusion-emotion.jsonl
    300 results/2026-09-23-l4-awq/diffusion-emotion--reversed.jsonl
    300 results/2026-09-23-l4-awq/diffusion-irony.jsonl
    300 results/2026-09-23-l4-awq/diffusion-sst2.jsonl
    300 results/2026-09-23-l4-awq/diffusion-sst2--reversed.jsonl
   2100 total

The diffusion arm reads each example four times with different noise in the answer slots, as the PR's proxy does. The DiffusionGemma server came up in about 300 seconds with the image already pulled.

Step 7 — Score It

python3 score.py --run 2026-09-23-l4-awq

score.py writes SUMMARY.md and summary.json beside the records. Every figure below comes from those two files. Readouts: plain is one read of plain Gemma. diffusion is the proxy's automatic readout, the mean of four DiffusionGemma reads on the 96% to 100% of examples where its rule re-reads and one read on the rest; the label-count tables use the four-read mean throughout.

How Accurate Is Each One?

Level on three tasks, plain ahead on AG News.

Task	Always-majority	Plain Gemma	DiffusionGemma	Difference (95% range)
sst2	51.0%	95.0%	93.3%	−1.7 (−4.0 to +0.7)
AG News	25.0%	🥇 86.7%	83.0%	−3.7 (−6.7 to −0.7)
DAIR Emotion	35.0%	58.3%	60.3%	+2.0 (−1.0 to +5.0)
tweet_eval irony	60.3%	89.7%	87.3%	−2.3 (−5.7 to +1.0)

Throughout, level means the 95% range of the difference includes zero, and 🥇 in an accuracy table marks a lead whose range excludes it; in Compare and Contrast 🥇 marks the better value. Only the AG News difference excludes zero.

Are the Raw Probabilities Calibrated?

DiffusionGemma's are closer. Plain Gemma's are strongly overconfident.

Task	Plain Gemma ECE	DiffusionGemma ECE	Difference (95% range)
sst2	0.051	0.045	−0.005 (−0.021 to +0.020)
AG News	0.129	0.119	−0.010 (−0.030 to +0.020)
DAIR Emotion	0.386	0.261	−0.125 (−0.154 to −0.089)
tweet_eval irony	0.098	0.043	−0.055 (−0.077 to −0.014)

ECE is expected calibration error over 15 bins: the gap between how confident a model says it is and how often it is right. DiffusionGemma is better calibrated on emotion and irony, and level on the other two.

The fitted temperatures show the size of the gap. On the pre-registered single split, one temperature that best fits the labels was 3.48 to 7.24 for plain Gemma once 25 or more labels were used, and 0.73 to 2.36 for DiffusionGemma. A temperature above 1 means the model's probabilities are too extreme.

What Do 50 Labels Buy?

Most of the correction for plain Gemma; a smaller gain for DiffusionGemma, and none on irony.

Each task's examples split into a fitting half and a held-out half. One temperature is fitted on the first N labels of the fitting half and scored on the held-out half. On the pre-registered single split, 50 labels took plain Gemma from 0.040, 0.139, 0.440 and 0.088 to 0.029, 0.064, 0.102 and 0.042 on sst2, AG News, DAIR Emotion and irony, and DiffusionGemma from 0.047, 0.132, 0.311 and 0.055 to 0.025, 0.061, 0.115 and 0.054.

One split of 150 held-out examples moves ECE by several hundredths, so the same fit was repeated over 20 random splits, an analysis added after the run. Means, with the lowest and highest split in brackets:

Task	Plain, 0 labels	Plain, 50 labels	Diffusion, 0 labels	Diffusion, 50 labels
sst2	0.048	0.039 (0.021–0.061)	0.050	0.042 (0.025–0.081)
AG News	0.130	0.066 (0.037–0.132)	0.121	0.085 (0.060–0.138)
DAIR Emotion	0.392	0.089 (0.047–0.161)	0.266	0.111 (0.062–0.155)
tweet_eval irony	0.099	0.060 (0.035–0.130)	0.054	0.066 (0.031–0.116)

With no labels, DiffusionGemma is better calibrated on emotion and irony. With 50, plain Gemma's mean is at or below DiffusionGemma's on all four tasks, and the split-to-split ranges overlap throughout.

What Does DiffusionGemma's Read Do?

About half of its probability lands outside the allowed labels. The median share on the allowed labels per task:

Task	Plain Gemma	DiffusionGemma
sst2	1.000	0.503
AG News	0.999	0.370
DAIR Emotion	1.000	0.392
tweet_eval irony	1.000	0.593

At the answer slot the server returns the allowed labels plus the single most likely token. In the latency run that token was <eos>, the end-of-sequence token, in 40% of reads on sst2, 53% on AG News and 64% on DAIR Emotion, and the in 24% of reads on irony. The proxy rescales the label probabilities to sum to one, so the answer reads as confident either way; label_mass is the field that shows it, and it is worth logging in any deployment.

The proxy re-reads when the answer slot's entropy is above 0.1. It computes that entropy from the returned tokens' full-vocabulary probabilities without rescaling them, so a label holding half the probability on its own already scores 0.35. It re-read 100% of sst2 and AG News examples, 99.0% of DAIR Emotion and 96.3% of irony.

The re-reads barely change the answer: one read and the mean of four are within 0.6 points of accuracy on every task (93.3% and 93.3% on sst2, 83.3% and 83.0% on AG News, 60.7% and 60.3% on DAIR Emotion, 86.7% and 87.3% on irony). How much the four reads disagree separates right from wrong well on sst2 and irony, AUROC 0.891 and 0.873, and weakly on AG News and DAIR Emotion, 0.685 and 0.635.

Does Option Order Change the Answer?

For both models, by similar amounts. Listing the choice options in reverse order changed:

Task	Plain Gemma	DiffusionGemma
sst2	6 of 300 (2.0%)	7 of 300 (2.3%)
AG News	14 of 300 (4.7%)	21 of 300 (7.0%)
DAIR Emotion	27 of 300 (9.0%)	26 of 300 (8.7%)

How Fast Is Each One?

A second run timed both arms with the client on the instance itself, against localhost, one request at a time, over the first 100 examples of each task. Same image, flags and checkpoints, on a g6.4xlarge: the same L4 with a larger host.

python3 run_eval.py --arm autoregressive --upstream http://localhost:8000 --model $PLAIN --run 2026-09-23-l4-latency --limit 100 --concurrency 1
python3 run_eval.py --arm diffusion --upstream http://localhost:8000 --model $DIFF --run 2026-09-23-l4-latency --limit 100 --concurrency 1 --reads 4 --keep-top 5

| Task | Plain, median | Plain, 90th pct | Diffusion one read, median | Diffusion one read, 90th pct | Diffusion 4 reads (first + 3 parallel), median |
|---|---|---|---|---|---|
| sst2 | 61 | 62 | 118 | 119 | 306 |
| ag_news | 61 | 62 | 119 | 120 | 307 |
| emotion | 61 | 62 | 119 | 120 | 308 |
| irony | 61 | 61 | 121 | 122 | 312 |

Times are in milliseconds. A plain Gemma 26B decision takes 61 ms on one L4 for these prompts, which run 105 to 299 tokens with a median of 132; longer inputs such as full tickets or logs take longer. One DiffusionGemma read takes 1.9 to 2.0 times as long, and the automatic rule's four reads 5.0 to 5.1 times, which is the path it takes on nearly every call. At 61 ms one request at a time, one L4 answers about 16 plain decisions a second before any batching.

🔎 Tip: Fit One Temperature Before You Trust a Threshold

Both models give usable probabilities after one fitted temperature. score.py does it with a grid search on log loss:

def fit_temperature(probs, gold_idx):
    return min(T_GRID, key=lambda t: nll([temper(p, t) for p in probs], gold_idx))

Label 50 real decisions from the task, fit on them, and set any act-or-escalate threshold on the rescaled probabilities. Plain Gemma needs the larger correction, and both reach similar calibration once it is applied.

What About the Smaller Gemma 4 Models?

The label read works with any Gemma 4. DiffusionGemma ships only at 26B-A4B, so the smaller models get the plain arm alone, run in bf16 on the same L4 with the same flags. Their chat templates end at the model turn without the empty thought block the 26B template adds, so their prompt follows their own template. These two arms were added in a pre-registration addendum after the first run, and are exploratory. The 26B column is 4-bit and the others bf16, so the gaps mix model size with precision. All three read letter labels (A to F) with the meanings listed in the prompt, the format of the PR's proxy; smaller models may do better with the label words as the answers themselves.

Task	26B, 4-bit	E4B, bf16	E2B, bf16
sst2	95.0%	94.3%	88.7%
AG News	86.7%	83.7%	30.3%
DAIR Emotion	58.3%	54.0%	53.3%
tweet_eval irony	89.7%	84.3%	72.7%

E4B is 0.7 to 5.4 points behind the 26B, needs a smaller correction (fitted temperature 1.89 to 3.77 at 50 labels, against 4.65 to 7.05, both means over 20 splits), and reaches similar calibration once corrected: 0.042, 0.071, 0.089 and 0.081 against the 26B's 0.039, 0.066, 0.089 and 0.060. It changed more answers when the options were reversed on AG News, 35 of 300 against 14.

E2B answered "world" on 277 of 300 AG News examples, and on 263 when the options were listed in reverse order, so the same option wins whatever its position, and its topic accuracy sits near the 25% majority rate. Its fitted temperatures reach 7.99, the top of the pre-registered search range, so its corrected calibration figures understate what a wider search would reach.

How Does It Compare With Jev?

Bespoke Labs, which makes the open Nimble-9B decision model, ran Jev 1.13.0 through its API on a 13-subset public suite and published the results per subset, with the converters and record ids. The suite holds 3,880 human-labelled records: yes/no questions from BoolQ, PAWS, SQuAD 2.0, Civil Comments and Aegis 2.0; multiple choice from MultiNLI, PubMedQA, VitaminC and MASSIVE intents in English and German; and five-level ratings from HelpSteer2 and SummEval.

A third run, on a g6.xlarge with the same image and flags, rebuilt the suite from the public sources and read it with all three Gemma arms. All 13 rebuilt subsets matched the published checksums, so the Gemma figures and Bespoke Labs' Jev figures come from the same records.

Each record is a Jev request, which the PR's proxy converts with its own Jev parser, so the Gemma arms see the same prompt format as the first run: yes/no questions answered with the word yes or no, multiple-choice options lettered A, B, C with the answer read as a letter, and rating levels numbered from 1. Gemma was read in that format as published, with no prompt tuning, while Jev takes the request in its own format; a different prompt could move the multiple-choice gap in either direction. Scoring uses Bespoke Labs' definitions, and tests/test_suite_scoring.py checks it against Bespoke Labs' own code.

bash nimble_suite/build.sh <workdir>
python3 nimble_suite/run_suite.py --arm autoregressive --upstream http://localhost:8000 \
  --model $PLAIN --records <workdir>/public --run 2026-09-24-l4-suite
python3 nimble_suite/suite_stats.py --run 2026-09-24-l4-suite --run 2026-09-24-l4-suite-e4b

Accuracy pooled over records, with the 95% range in brackets:

Questions	Records	Jev 1.13.0, published	Nimble-9B, published	Plain Gemma 26B	DiffusionGemma 26B	Gemma 4 E4B
All	3,880	77.3% (76.0–78.6)	75.9%	75.3% (73.9–76.6)	75.9% (74.5–77.2)	73.3% (71.9–74.6)
Yes/no	1,399	84.6% (82.6–86.4)	80.1%	84.8% (82.8–86.6)	84.5% (82.5–86.3)	81.9% (79.8–83.8)
Multiple choice	1,848	82.8% (81.1–84.5)	81.1%	78.3% (76.4–80.1)	77.3% (75.4–79.2)	74.8% (72.8–76.8)
Five-level rating	633	45.2% (41.3–49.1)	51.2%	45.5% (41.7–49.4)	52.8% (48.9–56.6)	49.6% (45.7–53.5)

Jev leads plain Gemma by 2.1 points over all records (95% range 0.2 to 4.0) and by 4.5 on multiple choice (2.0 to 7.1). Jev's per-record answers are unpublished, so these ranges compare two independent proportions: pairing would narrow them, and records that share a passage or article widen them, most of all for ratings. The overall gap's range starts at 0.2 points, so it is the least secure of the three; the multiple-choice gap is the firm one. PubMedQA and VitaminC each account for 33 of the 84 records behind it, and PubMedQA has the largest single-subset gap, 77.2% against 64.0%.

On yes/no questions pooled, plain Gemma shows no measurable difference from Jev, 84.8% against 84.6%, a difference anywhere from 2.8 points ahead to 2.5 behind. Within them Jev is ahead on BoolQ by 5.7 points and plain Gemma on Civil Comments and SQuAD 2.0 by 6.0 and 6.4; PAWS, 5.2 points to Jev, is within its range.

On ratings DiffusionGemma leads both Jev, by 7.6 points (2.1 to 13.1), and plain Gemma, on exact-level accuracy; Bespoke Labs advises reading that beside the error of the probability-weighted level, which this article does not report. Plain Gemma and DiffusionGemma are level over all 3,880 records: 187 right only for plain, 211 right only for DiffusionGemma, exact McNemar p = 0.25. On ratings DiffusionGemma was right alone on 97 records against 51, p = 0.0002, and the whole gap comes from SummEval, whose 384 records come from 24 news articles. Records from one article move together, so that p-value overstates the evidence.

Bespoke Labs' open Nimble-9B scores 75.9% over all records, level with the two 26B reads. Gemma 4 E4B trails the 26B by 2.0 points over the whole suite, with single subsets ranging from 9.6 points ahead to 8.0 behind, and trails Jev by 4.1 (2.2 to 6.0).

Median calibration error over the 13 subsets, with Bespoke Labs' 10 bins:

	ECE as shipped	ECE after 50 labels	Brier as shipped
Jev 1.13.0, published	0.071		0.267
Nimble-9B, published	0.109		0.314
Plain Gemma 26B	0.180	0.080	0.359
DiffusionGemma 26B	0.114	0.074	0.290
Gemma 4 E4B	0.173	0.077	0.359

As shipped, Jev has a lower calibration error than plain Gemma on all 13 subsets, than DiffusionGemma on 11 and than E4B on 12. Brier score, which rewards accuracy and calibration together, favours Jev too: plain Gemma and E4B beat it on 2 of 13 subsets, DiffusionGemma on 4. DiffusionGemma's raw calibration error is lower than plain Gemma's on all 13, and its median share on the allowed labels was 66.5% per read on this suite.

After one temperature per subset, fitted on 50 labels from that subset, the Gemma medians sit 0.003 to 0.009 above Jev's as-shipped median. Per subset, fitted plain Gemma is still above Jev on 8 of 13, by up to 0.049, and the Gemma arms are at or below Jev on 4 to 6. The fitted figures are scored on the held-out half of each subset, 72 to 300 records, and ECE reads higher on fewer records, which works against the Gemma columns. Jev's figures are as shipped, and a temperature fitted on Jev's own output could lower them as well.

Matt Mastracci, who wrote vLLM PR #57250, compared DiffusionGemma with Jev on 201 hand-built items on September 17, and Google's Gemma account shared the thread the next day. DiffusionGemma answered 198 correctly and Jev 191; he called the two "roughly tied" and DiffusionGemma "the winner, I think." Seven of Jev's ten errors fell in one set, 89 words drawn from five sentences, and the items, code and per-item outputs are unpublished. On the public records DiffusionGemma is 1.4 points behind Jev, anywhere from 0.4 ahead to 3.3 behind, so both measurements find the two close overall; by question type they part, level on yes/no, 5.5 points behind on multiple choice (3.0 to 8.1) and 7.6 ahead on ratings. Plain Gemma 26B, read the same way without the diffusion step, lands within 0.6 points of DiffusionGemma over the suite (p = 0.25), so this suite shows no accuracy gain from the diffusion step.

His timings show the same cost for the re-reads: 1.9 to 4.6 times one read on his eight sets, against 2.6 times here. One read beat Jev's API on seven of his eight sets and the automatic re-reads lost on all eight; his single reads were timed warm and the re-reads cold, which exaggerates that second gap.

Compare and Contrast

	Jev 1.13.0, published	Plain Gemma 4 26B, label read	DiffusionGemma 26B, one-step read
Accuracy, public suite	🥇 77.3%	75.3%, 2.1 behind Jev; level with DiffusionGemma	75.9%, 1.4 behind Jev, a range that includes a tie
Yes/no, public suite	84.6%	84.8%	84.5%
Multiple choice, public suite	🥇 82.8%	78.3%	77.3%
Five-level rating, public suite	45.2%	45.5%	🥇 52.8%
Median ECE, public suite, as shipped	🥇 0.071	0.180	0.114, lower than plain on all 13 subsets
Median ECE, public suite, after 50 labels		0.080	0.074
Median Brier, public suite, as shipped	🥇 0.267	0.359	0.290
Accuracy, four tasks		🥇 level on three, ahead on AG News	level on three, 3.7 points behind on AG News
Raw calibration, four tasks		overconfident, fitted temperature 3.5 to 7.2	🥇 closer, fitted temperature 0.7 to 2.4
Probability on the allowed labels		🥇 about 100%	37% to 59%, 66.5% on the public suite
Time per decision		🥇 61 ms on the instance	118 to 121 ms for one read, 306 to 312 ms with the automatic rule
Option-order changes, four tasks		2.0% to 9.0%	2.3% to 8.7%
Price per million decisions	$5.54 at 132 input tokens	at most $5.43 on a `g6.xlarge` at full load	at most $31.72
Serving	TypeSafe's hosted API	any vLLM	vLLM with PR #57250

So, Which One?

Between the two Gemma 4 26B reads, for a Jev-style decision service on one L4: plain Gemma read by its label probabilities, plus one temperature fitted on about 50 labels. It is at least as accurate on the four tasks and level with DiffusionGemma over the public suite, faster, and places all of its probability on the answers you allowed.

DiffusionGemma's lower raw calibration error has a range excluding zero on two of four tasks and holds on all 13 public-suite subsets, and it matters when no labels exist at all. With 50 labels the difference is no longer detectable on the four tasks, and on the suite the two split 8 to 5, a count within chance (sign test p = 0.58). On five-level ratings DiffusionGemma scored higher than both Jev and plain Gemma, on few source articles; a rating task is the one place to try both.

Against Jev, on the same public records: plain Gemma 4 26B trails Jev by 2.1 points overall, shows no measurable difference on yes/no questions pooled, and trails it by 4.5 points on multiple choice. Its median calibration error comes within 0.01 of Jev's as-shipped median once one temperature per subset is fitted on 50 labels from that subset; per subset it stays higher on 8 of 13. Jev is the better calibrated with no labels at all.

Where a smaller model has to do, Gemma 4 E4B in bf16 gives up 0.7 to 5.4 points against the 26B on the four tasks and 2.0 points over the public suite. E2B falls apart on four-way topic classification with this prompt.

What Does a Decision Cost?

The run logs give a throughput. Plain Gemma answered 2,100 decisions in 51 seconds, 41.2 a second, at client concurrency 8. DiffusionGemma, reading each example four times, answered 2,100 in 298 seconds, 7.0 a second, at concurrency 4. The client was on a home connection, so these are lower bounds on what the L4 serves.

At the g6.xlarge on-demand price of $0.8048 an hour, that is at most $5.43 per million decisions for plain Gemma and $31.72 for DiffusionGemma with four reads, which ran at half the concurrency, so its figure is the looser bound. TypeSafe prices Jev at $0.042 per million input tokens, which is $5.54 per million decisions at this run's median of 132 input tokens and $12.56 at the longest, 299: about the same as the L4 at full load. The L4 is charged by the hour whether busy or idle, so its per-decision cost holds only at full load. The public suite's longer prompts would raise both the L4's cost and Jev's.

The first run, from launch to termination in under 1.2 hours, cost at most $0.97 of instance time. The second, on a g6.4xlarge at $1.3232 an hour for 1.19 hours, cost $1.57. The public-suite run, on a g6.xlarge for 0.82 hours, cost $0.66, including rebuilding the suite and reading 11,640 records across three models. All three, $3.20 together, exclude the prorated 100 GB volume. All ran on demand; G-family spot capacity in us-east-1 was unavailable at launch time.

Teardown

Every command on the instance went through AWS Systems Manager, so no SSH port was ever open, and port 8000 was open to one address.

aws ec2 terminate-instances --region us-east-1 --instance-ids i-0c5b14e913b4c1019

shutting-down

Then delete the security group, and confirm that nothing tagged ManagedBy=jev remains.

Summary

The goal of this article was to measure Gemma 4 26B as a Jev-style decision model, read two ways, for accuracy and calibration on labelled data, and to set it beside Jev's published results on a public suite. The key to the solution was a matched pair of 4-bit checkpoints on one EC2 L4, identical prompts and label tokens, and a measurement pre-registered before any call. The results were:

⚠️ On Bespoke Labs' 3,880-record public suite, plain Gemma 4 26B trails Jev 1.13.0 by 2.1 points overall (range 0.2 to 4.0) and 4.5 on multiple choice (2.0 to 7.1)
🟢 No measurable difference from Jev on yes/no questions pooled, 84.8% against 84.6%
⚠️ Jev is the best calibrated with no labels: median ECE 0.071 against 0.114 to 0.180 for the Gemma arms
🟢 One temperature fitted on 50 labels brings the Gemma medians to 0.074 to 0.080; per subset plain Gemma stays above Jev on 8 of 13
⚠️ DiffusionGemma 1.4 points behind Jev over the suite (0.4 ahead to 3.3 behind): 5.5 behind on multiple choice, 7.6 ahead on ratings; Mastracci's hand-built items had it 198 against 191, both close overall
🟢 Plain Gemma and DiffusionGemma level on accuracy: plain ahead only on AG News of the four tasks, 75.3% against 75.9% over the suite, p = 0.25
🟢 DiffusionGemma better calibrated as shipped, on emotion and irony and on all 13 suite subsets; after 50 labels the difference is no longer detectable
⚠️ DiffusionGemma places only 37% to 59% of its probability on the allowed labels, and the proxy's rescaling hides it
⚠️ The automatic re-read rule fires on 96% to 100% of examples, costs 2.6 times one read, and moves accuracy by at most 0.6 points
🟢 61 ms per plain decision on the instance, and at most $5.43 per million at full load, level with Jev's $5.54 at this run's prompt length
🟢 Gemma 4 E4B in bf16 0.7 to 5.4 points behind the 26B on the four tasks and 2.0 over the suite
❌ Gemma 4 E2B answered "world" on 277 of 300 AG News examples

Scope: one NVIDIA L4 in us-east-1 on three instances, a g6.xlarge for the accuracy run, a g6.4xlarge for the latency and small-model run and a g6.xlarge for the public suite, vLLM 0.29.1rc1.dev573+ge97573215, the 26B models 4-bit from the same uploader with the same quantization settings and E4B and E2B in bf16, 300 examples per task plus the 3,880-record public suite, one run per arm. Google's reference checkpoints are bf16, and 4-bit quantization may affect a diffusion model differently from an autoregressive one, so results at bf16 may differ for either arm. Latency comes from 100 examples per task with the client on the instance; throughput comes from the first run's client on a home connection. The latency, E4B and E2B arms were added in a pre-registration addendum after the first run, and E4B and E2B are exploratory; the 20-split calibration and the public suite's ranges, paired test, medians and per-subset counts were added after their runs, and the per-subset and pooled suite figures are the pre-registered ones. The suite's E4B arm ran under its own run name, 2026-09-24-l4-suite-e4b, with the same instance, image, flags and records. All four test sets and all 13 suite datasets were published before Gemma 4 and may be in its training data, and the high irony scores may indicate it there; whether they are in Jev's is unknown. No Jev call was made: the Jev and Nimble-9B figures are Bespoke Labs' published results on the same records, from one run of Jev 1.13.0 by a company that publishes a competing model, counting an invalid Jev response as wrong, with Jev's probabilities rounded to two decimals by its API. Code, pre-registration and every per-item output are in the repository. Parts of the analysis and writing were done with AI assistance (Claude); every figure comes from the committed output files.

The strategy for using label probabilities to run Gemma 4 as a decision model was validated with an incremental step by step approach.

References

Code, pre-registration and per-item results: https://github.com/xbill9/gemma4-dev/tree/main/jev
Companion review of the independent evidence on Jev: https://dev.to/gde/jev-after-eight-days-of-independent-tests-level-with-mid-price-llms-behind-the-frontier-1kln
vLLM PR #57250, DiffusionGemma structured reads: https://github.com/vllm-project/vllm/pull/57250
Plain checkpoint: https://huggingface.co/cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit
Diffusion checkpoint: https://huggingface.co/cyankiwi/diffusiongemma-26B-A4B-it-AWQ-INT4
DiffusionGemma model card: https://huggingface.co/google/diffusiongemma-26B-A4B-it
Google's Gemma account on DiffusionGemma's calibration: https://x.com/googlegemma/status/2101069861598482817
Matt Mastracci, Jev against DiffusionGemma on hand-built items: https://x.com/mmastrac/status/2100626193943052784
Guo et al., On Calibration of Modern Neural Networks: https://arxiv.org/abs/1706.04599
Amazon EC2 G6 instances: https://aws.amazon.com/ec2/instance-types/g6/
TypeSafe Jev: https://docs.typesafe.ai/concepts/system-one
Bespoke Labs, Nimble public-suite results for Jev 1.13.0 and Nimble-9B: https://github.com/bespokelabsai/nimble/blob/0e67403/docs/PUBLIC_BENCHMARKS.md

A 4 GB Laptop GPU vs a 6-Core CPU on Gemma 4, Re-Measured in ABBA Order: 4.1x

xbill — Wed, 23 Sep 2026 16:48:49 +0000

This article provides a step by step guide to measuring a laptop GPU against the CPU in the same chassis for serving Gemma 4 E2B through llama.cpp, with both builds on a fresh Debian sid toolchain and the passes run in ABBA order so heat cannot pick the winner. A suite of Python MCP tools is built to simplify management of the llama.cpp hosted deployment.

https://github.com/xbill9/gemma4-dev

The GTX 1650 Ti decodes 4.14x faster than the i7-10750H it shares a chassis with, prefills 3.42x faster and finishes requests 3.62x faster end to end. Running the two devices in one order and then the other moves that decode ratio from 4.09x to 4.17x, so a single-order benchmark on this laptop is off by about 2%.

What Is Being Measured?

The same 3.35 GB quantization-aware GGUF, google/gemma-4-E2B-it-qat-q4_0-gguf, served by llama-server twice on one laptop: once on the CPU, once on the GPU. Both builds come from one llama.cpp commit and the two command lines differ by one flag.

An earlier run of this comparison is published as A 4 GB Laptop GPU Beats a 12-Core CPU by 4.3x on Gemma 4. This one re-measures it after the laptop moved to Debian sid, with three changes: a newer llama.cpp commit built on gcc 16.2 and CUDA 13.4, thread flags set for this CPU's real core count, and a run order that cancels thermal drift.

At This Point You Should Have…

An NVIDIA GPU with a CUDA driver — here a GTX 1650 Ti with Max-Q Design, driver 615.71.09
The CUDA toolkit — nvcc 13.4 here
cmake and a host compiler CUDA accepts — gcc 16.2.0 here
Python 3.10 or newer, the system python3, no virtualenv
The repository cloned, with local-llamacpp-1650ti-2b-q4_0/ and local-llamacpp-cpu-2b-q4_0/ side by side
Claude Code, or any MCP client that speaks stdio

The Laptop


Machine	Lenovo Yoga 9 15IMH5
CPU	Intel Core i7-10750H, 6 cores / 12 threads, AVX2
GPU	GeForce GTX 1650 Ti with Max-Q Design, 4096 MiB, compute capability 7.5
OS	Debian GNU/Linux forky/sid, kernel 7.2.6
Compiler	gcc 16.2.0
CUDA	13.4 (V13.4.92)
llama.cpp	`f95b0d9`, build 318

The CPU and the Max-Q card sit under one cooling system, which is why the run order below matters.

Step 1 — Check the Toolchain Still Supports the Card

A distribution upgrade can move the compiler past what CUDA accepts, and a CUDA upgrade can drop old GPUs. Both were close here.

$ nvidia-smi --query-gpu=name,compute_cap,memory.total,driver_version --format=csv
name, compute_cap, memory.total [MiB], driver_version
NVIDIA GeForce GTX 1650 Ti with Max-Q Design, 7.5, 4096 MiB, 615.71.09

$ nvcc --list-gpu-arch
compute_75
compute_80
compute_86
...

The card is compute capability 7.5, and compute_75 is the first entry CUDA 13.4 lists. This laptop's GPU is the oldest architecture the current toolkit still builds for.

$ gcc --version
gcc (Debian 16.2.0-3) 16.2.0

$ grep -n 'later than 16' /usr/local/cuda/include/crt/host_config.h
137:#error -- unsupported GNU version! gcc versions later than 16 are not supported! ...

CUDA 13.4 accepts gcc up to 16, and sid ships 16.2.

Step 2 — Build Both Devices From One Commit

Each rig's Makefile carries its own build target. The GPU build targets only this card's architecture, sm_75; the CPU build turns every GPU backend off.

# local-llamacpp-1650ti-2b-q4_0
cmake -S ~/llama.cpp -B ~/llama.cpp/build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=75 -DGGML_NATIVE=ON -DCMAKE_BUILD_TYPE=Release
cmake --build ~/llama.cpp/build --config Release -j6 --target llama-server llama-bench

# local-llamacpp-cpu-2b-q4_0
cmake -S ~/llama.cpp -B ~/llama.cpp/build-cpu -DGGML_CUDA=OFF -DGGML_VULKAN=OFF -DGGML_NATIVE=ON -DCMAKE_BUILD_TYPE=Release
cmake --build ~/llama.cpp/build-cpu --config Release -j6 --target llama-server llama-bench

Both binaries report the same commit and compiler:

$ ~/llama.cpp/build/bin/llama-server --version
version: 0.4.1-dev (build 318, commit f95b0d9)
built with GNU 16.2.0 for Linux x86_64

$ ~/llama.cpp/build-cpu/bin/llama-server --version
version: 0.4.1-dev (build 318, commit f95b0d9)
built with GNU 16.2.0 for Linux x86_64

And only one of them can see the card:

$ ~/llama.cpp/build/bin/llama-server --list-devices
Available devices:
  CUDA0: NVIDIA GeForce GTX 1650 Ti with Max-Q Design (3732 MiB, 3671 MiB free)

$ CUDA_VISIBLE_DEVICES= ~/llama.cpp/build-cpu/bin/llama-server --list-devices
Available devices:
  (none)

🔎 Tip: Always Give the CUDA Build a Job Count

cmake --build -j with no number starts every CUDA source file at once. llama.cpp has about 188 of them, each nvcc forks cicc and cudafe++, and a 15 GiB laptop runs out of memory. At -j6 the full CUDA build peaked at 4.0 GiB and never touched swap, so the limit costs nothing. Both Makefiles pass -j$(BUILD_JOBS) with a default of 6.

After a compiler or CUDA upgrade, delete the build directory before configuring. A stale CMakeCache.txt keeps the old compiler, and /usr/local/cuda is a symlink, so the cached path looks current.

Step 3 — Serve One Device at a Time on the Same Port

Both devices serve on 127.0.0.1:8080, so the endpoint, the prompts and the benchmark script stay fixed while the device changes underneath. The flags match except for -ngl, the number of layers offloaded to the GPU:

llama-server -m gemma-4-E2B_q4_0-it.gguf --host 127.0.0.1 --port 8080 \
  -ngl {0|99} -c 8192 -ctk f16 -ctv f16 -fa 1 -t 6 -tb 12 --parallel 1 --metrics

-t 6 -tb 12 matches the CPU's six physical cores and twelve threads, and both devices get it. GPU decode barely responds to thread count, so matching them costs the GPU nothing and removes one difference between the two runs.

For the GPU side there is a small wrapper, llamacpp-1650ti, that reads these values out of the rig's tpu.env, starts the server in the background and checks which device answered:

$ llamacpp-1650ti start
llamacpp-1650ti: starting /home/xbill/models/gemma-4-E2B-it-qat-q4_0/gemma-4-E2B_q4_0-it.gguf on 127.0.0.1:8080 (-ngl 99)
llamacpp-1650ti: pid 88166, log /home/xbill/gemma4-dev/local-llamacpp-1650ti-2b-q4_0/run/llama-server.log
llamacpp-1650ti: waiting up to 180s for http://127.0.0.1:8080/health
     0s  VRAM 3 MiB, 0 %
llamacpp-1650ti: healthy after 2s -- http://127.0.0.1:8080
llamacpp-1650ti: device=gpu · pid=88166 · -ngl 99 · mapped: ggml-cuda, libcublas, libcuda, libcudart · /home/xbill/llama.cpp/build/bin/llama-server

$ llamacpp-1650ti stop
llamacpp-1650ti: sent SIGTERM to pid 88166; VRAM is released on exit

The wrapper refuses to start if anything already holds the port, and refuses to stop the CPU build if that is what holds it.

Step 4 — Read the Device Off the Running Process

Both devices answer the same URL with the same JSON, so the HTTP response cannot say which one produced it. The rigs' attest.py reads it from /proc instead: /proc/<pid>/exe for the binary, /proc/<pid>/maps for the GPU libraries it has loaded, /proc/<pid>/cmdline for the real -ngl. The benchmark script refuses to start if the answer is the wrong device, and writes it at the top of every log:

arm: device=cpu · pid=72478 · -ngl 0 · /home/xbill/llama.cpp/build-cpu/bin/llama-server
arm: device=gpu · pid=74308 · -ngl 99 · mapped: ggml-cuda, libcublas, libcuda, libcudart · /home/xbill/llama.cpp/build/bin/llama-server

maps is used because llama.cpp loads its CUDA backend at runtime, where ldd does not see it. Each report also records the SHA-256 of the executable that ran: 9c88c7821fcc11a6… for the CPU build and 9f2a8b0b6ed5365d… for the GPU build, the same on both passes of each.

The MCP servers expose the same check as a tool, so an agent that starts a server can confirm what it started before it measures anything.

Step 5 — Run in ABBA Order With a Temperature Gate

Each device gets two passes, in the order CPU, GPU, GPU, CPU. Each device then has one early pass and one late pass, so a slow drift in the laptop's temperature over the session affects both equally. Before every pass the script waits at least 120 seconds and until the CPU package is at or below 50 °C and the GPU at or below 45 °C.

13:20:12 pkg=38C gpu=36C throttle=35105 START cpu pass1
13:27:57 pkg=85C gpu=52C throttle=44709 END cpu pass1 rc=0
13:30:00 pkg=45C gpu=40C throttle=44730 START gpu pass1
13:32:12 pkg=79C gpu=56C throttle=45150 END gpu pass1 rc=0
13:34:15 pkg=42C gpu=39C throttle=45150 START gpu pass2
13:36:26 pkg=78C gpu=55C throttle=48034 END gpu pass2 rc=0
13:38:29 pkg=41C gpu=39C throttle=48060 START cpu pass2
13:46:31 pkg=82C gpu=53C throttle=67346 END cpu pass2 rc=0

throttle is the CPU package's cumulative thermal-throttle counter. CPU pass 1 added 9,604 events, the two GPU passes 420 and 2,879, and CPU pass 2 added 19,279. A CPU pass takes about eight minutes and a GPU pass about two.

Each pass is the same sweep: four prompt lengths by two output lengths, three repeats per cell, one request at a time.

python3 sweep.py --base http://127.0.0.1:8080/v1 --out <run>/passN --rig <rig> --expect-device <cpu|gpu>

The Result

Each figure is the mean of the two passes' per-cell medians. Decode is the token rate measured off the response stream; TTFT is time to first token.

in tok	out tok	CPU decode	GPU decode	GPU lead	CPU TTFT ms	GPU TTFT ms	GPU lead
94	32	18.31	71.89	3.93x	1084	381	2.84x
94	128	17.96	71.57	3.98x	1171	385	3.04x
516	32	17.06	70.61	4.14x	5528	1608	3.44x
516	128	17.19	70.35	4.09x	5527	1610	3.43x
998	32	16.10	69.25	4.30x	11071	3242	3.41x
998	128	16.77	69.38	4.14x	11057	3246	3.41x
1959	32	16.14	68.52	4.25x	23118	6497	3.56x
1959	128	16.37	68.22	4.17x	22718	6496	3.50x

Medians over the eight cells: decode 4.14x, prefill 3.42x, end to end 3.62x.

Decode stays nearly flat on both devices across a 21x range of prompt length (94 to 1959 tokens), from 18.31 down to 16.14 tok/s on the CPU and from 71.89 down to 68.22 on the GPU, while TTFT grows in proportion to the prompt on both. Decode reads the model once per token and is limited by memory bandwidth; prefill does arithmetic over the whole prompt. The GPU wins both, by different margins, and the prefill margin grows with the prompt from 2.84x to 3.56x.

Is the Difference Bigger Than the Noise?

By an order of magnitude. The decode ratio stays between 3.93x and 4.30x in every cell. The noise is the drift between a device's two passes:

	decode drift, pass 1 to pass 2	worst cell
GPU	median +0.6%, range 0.0% to +1.3%	—
CPU	median -1.1%, range -9.3% to +0.1%	998 in / 32 out: 16.89 → 15.32

Within a pass, the three repeats of a cell spread by at most 1.14% on the GPU. On the CPU they spread by at most 2.69% in pass 1 and 14.05% in pass 2, which had twice the throttle events. CPU time to first token also rose in pass 2, by up to 7.4% at the longest prompt, while the GPU's moved by at most 0.9%.

All of the measurable noise in this comparison comes from the CPU, and it tracks heat.

What Is Run Order Worth?

Each ordering can be read on its own. CPU pass 1 against GPU pass 1 is the CPU-first design; GPU pass 2 against CPU pass 2 is GPU-first.

order	decode ratio	prefill ratio
CPU first, then GPU	4.09x	3.38x
GPU first, then CPU	4.17x	3.47x
ABBA, both	4.14x	3.42x

Running the CPU first understates the GPU's lead, because the CPU pass runs while the laptop is still cool and the GPU pass inherits its heat. On this laptop the effect is about 2% of the ratio. ABBA order removes it for the cost of one extra pass per device.

🔎 Tip: Count Tokens, Not Stream Chunks

llama.cpp can put more than one token in a streamed chunk, so a decode rate computed from chunk timing undercounts. Recomputed as completion tokens over the measured decode time, the rates read 16.77-20.27 tok/s on the CPU and 69.87-79.58 on the GPU, and the median ratio stays at 4.14x. Both devices run the same server code, so the undercount cancels in the ratio. It does not cancel when comparing either absolute figure against a different server.

🔎 Tip: Confirm the Prompt Cache Stayed Cold

llama-server reuses the key/value cache for a prompt prefix it has seen, which makes time to first token look faster than prefill really is. Each prompt starts with a unique prefix, and --metrics exposes a counter that confirms it:

cpu pass1: llamacpp:prompt_tokens_total 28553 llamacpp:prompt_tokens_cached_total 0
cpu pass2: llamacpp:prompt_tokens_total 28553 llamacpp:prompt_tokens_cached_total 0
gpu pass1: llamacpp:prompt_tokens_total 28553 llamacpp:prompt_tokens_cached_total 0
gpu pass2: llamacpp:prompt_tokens_total 28553 llamacpp:prompt_tokens_cached_total 0

None of the 28,553 prompt tokens in any pass came from the cache.

Compare and Contrast

	🥇 GTX 1650 Ti	🥈 i7-10750H
Decode, single stream	68.22-71.89 tok/s	16.10-18.31 tok/s
Time to first token, 1959-token prompt	6.5 s	22.7-23.1 s
Pass-to-pass drift	≤1.3%	up to 9.3%
Throttle events per pass	420 and 2,879	9,604 and 19,279
Pass length	about 2 minutes	about 8 minutes

So, Which One?

The GPU, for any interactive use of this model on this laptop. It is about four times faster on every prompt length measured, it is steadier, and it heats the shared package far less than the CPU does for the same work.

The CPU build is still worth keeping: it serves the same model at 16-18 tok/s when the card is busy or absent. It is also the device to benchmark with the most care, because its numbers move with the laptop's temperature.

How Does This Compare With the Earlier Run?

The earlier run measured 4.27x decode and 3.63x prefill. Three things changed at once — the llama.cpp commit (c6824a9 to f95b0d9), the thread flags (-t 4 -tb 8 to -t 6 -tb 12) and the run order — so the difference between the two results cannot be assigned to any one of them. The more-threads explanation fits the direction, since CPU time to first token fell and the prefill ratio narrowed with it, but this run does not isolate it.

Summary

The goal of this article was to re-measure a 4 GB laptop GPU against the 6-core CPU in the same chassis for serving Gemma 4 E2B, after a toolchain upgrade and with thermal drift controlled. The key to the solution was ABBA ordering with a temperature gate, identical flags on both devices apart from -ngl, and reading the serving device from /proc on every pass. The results were:

🟢 GPU decode is 4.14x the CPU, 3.93x to 4.30x across every cell
🟢 GPU prefill is 3.42x, growing with prompt length; end to end is 3.62x
🟢 The GTX 1650 Ti still builds and runs on CUDA 13.4 with gcc 16.2, as the oldest architecture that toolkit supports
⚠️ Run order is worth about 2% of the ratio here: 4.09x CPU-first against 4.17x GPU-first
⚠️ The CPU carries all of the measurable noise, up to 9.3% between passes and 14% within one, and it follows the throttle count
❌ The earlier 4.27x and this 4.14x cannot be compared directly, because commit, threads and order all changed together

Scope: one laptop, one GGUF, llama.cpp f95b0d9 built twice with gcc 16.2 and CUDA 13.4, eight paired cells, two passes per device in ABBA order, three repeats per cell per pass, one request at a time, prompt cache verified cold, page cache warm throughout.

The strategy for using MCP for local accelerator comparison was validated with an incremental step by step approach.

References

Repository: https://github.com/xbill9/gemma4-dev
GPU rig: https://github.com/xbill9/gemma4-dev/tree/main/local-llamacpp-1650ti-2b-q4_0
CPU rig: https://github.com/xbill9/gemma4-dev/tree/main/local-llamacpp-cpu-2b-q4_0
Run report: https://github.com/xbill9/gemma4-dev/blob/main/local-llamacpp-1650ti-2b-q4_0/benchmarks/runs/2026-09-22-paired-sweep-1650ti/REPORT.md
Earlier run: A 4 GB Laptop GPU Beats a 12-Core CPU by 4.3x on Gemma 4
Model: https://huggingface.co/google/gemma-4-E2B-it-qat-q4_0-gguf
llama.cpp: https://github.com/ggml-org/llama.cpp

Production RAG on the Lakehouse with BigQuery Vector Search and Apache Iceberg

Vo Tu Duc — Wed, 23 Sep 2026 04:24:08 +0000

As teams rush to build with Generative AI, they're creating a dangerous chasm between their data and AI stacks. This common architectural flaw introduces massive technical debt and business risk.

The Problem: The Great Divide Between Data and AI Stacks

The explosion of Generative AI and Building a RAG Context Manager with Apps Script and Gemini Pro (RAG) has unlocked incredible potential, but it has also exposed a fundamental architectural flaw in how many organizations are building these systems. As teams rush to production, they often inadvertently create a deep chasm between their core data infrastructure and their new AI stack. This divide isn't just an inconvenience; it's a source of significant technical debt, operational complexity, and business risk.
At the heart of the issue is the separation of concerns gone awry. The data lives in one universe—the data lakehouse, governed by decades of best practices in security, governance, and reliability. The AI, particularly the vector search component, lives in another—a specialized, often external, database. Bridging this gap requires brittle pipelines, data duplication, and fragmented security models, ultimately undermining the very reliability and trustworthiness we seek to build into our AI applications.

Why Traditional Vector Databases Create Data Silos

The conventional approach to building a RAG system follows a familiar, yet problematic, pattern. You begin with your curated, high-quality data residing in a centralized platform like a data lakehouse. This is your source of truth. To make this data accessible to a Large Language Model (LLM), you must:

Extract: Pull the data out of your source-of-truth system (e.g., an Apache Iceberg table in your lakehouse).
Transform: Chunk the text into manageable pieces and generate vector embeddings for each chunk using a model.
Load: Push both the vector embeddings and the associated text/metadata into a separate, standalone vector database (e.g., Pinecone, Milvus, Weaviate). This ETL-like process effectively creates a read replica of your original data, but one that is optimized for vector similarity search. While this works for a proof-of-concept, it's a recipe for disaster in production. You've just created a new data silo.

This new silo is completely disconnected from the original data's lifecycle. It has its own infrastructure to manage, its own APIs to learn, and its own failure modes to handle. More importantly, it requires a complex and often fragile synchronization process to keep it from becoming stale. Every time data is updated, deleted, or added in the source system, a corresponding change must be perfectly orchestrated and propagated to the vector database. This adds immense operational overhead and introduces a new, critical point of failure in your AI stack.

The Challenge of Metadata Drift and Security Fragmentation

The consequences of this data silo extend far beyond mere operational complexity. Two critical challenges emerge that directly impact the quality and security of your AI application: metadata drift and security fragmentation.
Metadata Drift is the silent killer of RAG system accuracy. It occurs when the data in your source-of-truth lakehouse changes, but those changes aren't immediately and atomically reflected in the vector database.
Consider a product catalog table in your lakehouse. A product's price is updated, or its status changes to "recalled." If your synchronization pipeline fails or runs on a delay, your RAG-powered chatbot could retrieve the old, stale context from the vector database and confidently provide a customer with an incorrect price or, worse, recommend a recalled product. This isn't just a technical glitch; it's a direct erosion of user trust and a potential business liability. The vector index has "drifted" from the ground truth, and your RAG system is now hallucinating based on outdated facts.
Security Fragmentation presents an equally severe governance and compliance risk. Your enterprise data lakehouse is built upon a robust, unified security model. You have fine-grained controls—IAM roles, row-level access policies, and column-level security—that dictate precisely who can see what data.
When you copy that data into a separate vector database, you are forced to reimplement that entire security model from scratch in a new environment. This is not only a duplication of effort but also a massive security risk. It's incredibly difficult to keep two disparate security models perfectly in sync. An employee who leaves the company might have their access revoked in the lakehouse, but their access to the sensitive data copied in the vector store might persist. This fragmentation creates security gaps, doubles the administrative burden, and makes compliance audits a nightmare.

Introducing the Lakehouse as the Single Source of Truth for Enterprise AI

What if we could eliminate the divide? What if, instead of moving the data to a separate AI system, we brought the AI capabilities directly to the data? This is the foundational principle of building production-grade AI on the Lakehouse.
The modern data lakehouse, combining the scalability of a data lake with the performance and transactional integrity of a data warehouse, is already the established single source of truth for enterprise analytics. It houses your most valuable, curated, and governed data assets. By integrating vector search as a native feature within this platform—as BigQuery has done—we can fundamentally change the architectural paradigm.
In this model, vector embeddings are not shipped to an external system; they become just another data type, a new column (ARRAY<FLOAT64>) in your existing Apache Iceberg or BigQuery native tables. The vector index is built directly on top of this column, co-located with the source data it represents.
This elegant simplification solves our critical challenges:

No More Silos: Data never leaves the lakehouse. There is no duplication, no ETL to a separate vector store, and no synchronization pipelines to maintain.
Zero Data Drift: When you update a row in your Iceberg table, the change is atomic. The text, the metadata, and the vector embedding are all updated together in a single transaction. The vector index is always perfectly consistent with the source of truth because it is the source of truth.
Unified Security and Governance: The same robust security model that protects your entire data estate automatically applies to your vector embeddings and search queries. The row-level permissions, column-level security, and IAM policies you've already defined are inherited seamlessly. There is one security model to manage, not two. By treating vector search as a first-class workload within the data lakehouse, we move from a brittle, fragmented architecture to a unified, robust, and secure platform for enterprise AI.

Architectural Blueprint: A Unified RAG Pipeline on Google Cloud

To build a robust, production-grade RAG system on the lakehouse, we need more than just a collection of services; we need a cohesive architecture where each component plays a specific, complementary role. Our blueprint unifies data management, machine learning, and analytics within a single, governable ecosystem on Google Cloud. This approach moves beyond siloed vector databases, bringing AI capabilities directly to your data's center of gravity—the data lakehouse. The result is a streamlined, scalable, and cost-effective pipeline that transforms raw information into intelligent, contextual responses.

Core Components: BigQuery, Building Self-Correcting Agentic Workflows with Vertex AI, Apache Iceberg, and Cloud Storage.

The power of this architecture lies in the synergy between four key Google Cloud and open-source technologies. Let's break down the role of each player.

Google Cloud Storage (GCS): This is the foundational layer of our lakehouse. GCS acts as the scalable, durable, and cost-effective landing zone for all our raw, unstructured source data—PDFs, Word documents, Markdown files, transcripts, and more. It is the "lake" where our data assets reside in their native format before being processed.
Apache Iceberg: This is the star of our data management strategy. Iceberg is not just a file format; it's an open table format that brings the reliability and structure of a traditional database directly to the vast data lake on GCS. In our RAG pipeline, Iceberg is critical for several reasons:
Transactional Integrity: It provides ACID-like transactional guarantees for our embeddings and metadata, preventing data corruption during concurrent writes.
Schema Evolution: It allows us to evolve our data schema (e.g., adding new metadata fields) without rewriting the entire dataset.
Performance: Features like partition evolution and file pruning optimize query performance, which is crucial when dealing with billions of vectors.
Openness: As an open standard, it prevents vendor lock-in and ensures our core data assets are portable and accessible by various engines like Spark, Flink, and, most importantly for us, BigQuery.
Vertex AI: This is our intelligence engine, providing the state-of-the-art models needed to understand and represent our text data. Specifically, we leverage the Vertex AI Embedding APIs (e.g., text-embedding-004). These managed, scalable endpoints take our processed text chunks as input and convert them into high-dimensional numerical vectors (embeddings). This process is the heart of the "retrieval" mechanism, as it encodes the semantic meaning of our text into a format that machines can compare for similarity.
BigQuery: BigQuery is the central nervous system that unifies the entire architecture. It has evolved far beyond a traditional data warehouse and serves two primary functions in our pipeline:
The Lakehouse Query Engine: Through its BigLake capabilities, BigQuery can directly read from and query the Apache Iceberg tables stored on GCS. This allows us to use familiar SQL to manage, inspect, and analyze our text chunks and their corresponding embeddings without moving the data.
The Vector Search Engine: This is the game-changer. BigQuery has native Vector Search functionality. It can build and manage a highly efficient Approximate Nearest Neighbor (ANN) index directly on the embedding column within our Iceberg table. This eliminates the need for a separate, dedicated vector database, consolidating our entire RAG backend into a single, powerful platform.

Data Flow: From Unstructured Data to Indexed Embeddings in Iceberg

The process of converting raw documents into a searchable vector index follows a clear, automated data pipeline. This is the "indexing" half of the RAG workflow.

Ingestion: The pipeline begins when new unstructured documents (e.g., annual-report-2023.pdf) are uploaded to a designated GCS bucket.
Parsing and Chunking: An event-driven process, such as a Cloud Function or a more robust Dataflow job, is triggered by the new file. This process is responsible for:
Parsing: Extracting the raw text content from the source file.
Chunking: Strategically splitting the extracted text into smaller, semantically coherent chunks. The chunking strategy (e.g., fixed size, recursive character splitting, etc.) is a critical factor in the quality of the retrieval results.
Embedding Generation: For each text chunk, the processing job makes an API call to a Vertex AI Embedding model endpoint. The API responds with a high-dimensional vector (e.g., a 768-dimension array of floating-point numbers) that captures the semantic essence of that chunk.
Writing to Iceberg: The original text chunk, its newly generated vector embedding, and any relevant metadata (e.g., source document name, page number, chunk ID) are packaged together. This structured record is then appended to our primary Apache Iceberg table residing on GCS. Thanks to Iceberg's transactional nature, this write operation is atomic and safe.
Vector Indexing: After the data is written to the Iceberg table, we use a simple BigQuery DDL statement to create or update a VECTOR_INDEX. BigQuery automatically handles the complex process of building the ANN index in the background. This index is what enables lightning-fast similarity searches across potentially billions of vectors during the retrieval step.

The Role of BigQuery Vector Search as the Unifying Engine

The most transformative aspect of this architecture is how BigQuery Vector Search acts as the unifying force, collapsing what were once disparate systems into a single, cohesive plane.
Traditionally, a RAG pipeline required managing at least three separate systems: an object store for raw files (GCS), a dedicated vector database for ANN search (e.g., Pinecone, Weaviate), and a data warehouse for structured metadata and analytics (BigQuery). This separation introduces complexity in data movement (ETL), security, governance, and operational overhead.
Our blueprint eliminates this fragmentation.

Data Stays Put: By querying Iceberg tables on GCS via BigLake and building the vector index in place, BigQuery brings the compute to the data. There is no need to duplicate and move terabytes of embedding data into a separate database, significantly simplifying the data pipeline and reducing costs.
Unified Governance and Security: Your embeddings and metadata are governed by the same robust security model you already use for BigQuery. You can manage access control at the project, dataset, table, and even column level using familiar IAM policies. This is a massive win for enterprise security and compliance.
Powerful Hybrid Search with SQL: This is the killer feature. Because the vector embeddings live in the same table as structured metadata, you can perform sophisticated filtered vector searches in a single, elegant SQL query. Consider a query like: "Find the top 5 document chunks most similar to {user_query}, but only from documents published after '2023-01-01' and tagged with 'finance'." In a siloed system, this is a complex multi-step process: query the vector DB, get IDs, then query the data warehouse with those IDs to filter. With BigQuery, it's a single, optimized query, unlocking powerful, context-aware retrieval that is simply not feasible otherwise. By positioning BigQuery as the central engine for both data management and vector search, we create a truly unified, scalable, and operationally efficient foundation for production-grade RAG on the lakehouse.

Step 1 Preparing and Embedding Your Enterprise Data

The foundation of any high-performing RAG system isn't the LLM—it's the data. The quality, structure, and semantic representation of your knowledge base directly dictate the relevance and accuracy of the generated responses. In a Lakehouse architecture, this first step is about establishing a robust, scalable, and open foundation for your data and then transforming it into a format that machine learning models can understand: high-dimensional vectors.
We'll tackle this by first defining our data's home using Apache Iceberg tables in BigQuery, and then processing our raw documents into vectorized "chunks" using Vertex AI's powerful embedding models.

Setting Up Apache Iceberg Tables in BigQuery

Before we can ingest anything, we need a destination. Why Apache Iceberg? In the context of a Lakehouse, Iceberg provides critical features that traditional data warehousing tables lack. It's an open table format that decouples the table structure from the physical storage (in our case, Google Cloud Storage), offering schema evolution, time travel, and efficient file-level operations. This makes it perfect for managing large, evolving datasets of document chunks and their corresponding embeddings.
We'll create a BigQuery "BigLake" table backed by Iceberg. This table will serve as our "vector store" source of truth, holding the original text chunks, their vector embeddings, and any relevant metadata.

Here’s the DDL to create our core table, doc_embeddings_iceberg:

CREATE OR REPLACE TABLE your_dataset.doc_embeddings_iceberg (
chunk_id STRING NOT NULL OPTIONS(description="Unique identifier for the text chunk"),
doc_source STRING OPTIONS(description="Identifier for the original source document, e.g., GCS path"),
chunk_text STRING OPTIONS(description="The actual text content of the chunk"),
embedding ARRAY<FLOAT64> OPTIONS(description="The 768-dimension vector embedding from Vertex AI"),
created_at TIMESTAMP
)
OPTIONS (
format = 'ICEBERG',
table_version = 2,
uris = ['gs://your-gcs-bucket/iceberg-warehouse/doc_embeddings'],
connector = 'biglake-connector-v1' -- Ensure your BigLake connection is set up
);

Let's break down the key components of this schema:

chunk_id: A unique primary key for each piece of text. This is crucial for referencing and updating specific chunks. A UUID or a hash of the content works well.
doc_source: Links the chunk back to its parent document. This is vital for providing citations and context in the final RAG output.
chunk_text: The raw text that was vectorized. We store this so we can retrieve the actual content to feed into the LLM's context window.
embedding: The star of the show. This ARRAY<FLOAT64> column will hold the numerical vector generated by our embedding model.
OPTIONS: We explicitly define the format as ICEBERG and specify the GCS path where the underlying Parquet and metadata files will be stored. This is the core of the Lakehouse pattern—SQL on open files in your data lake.

Using Vertex AI Embedding Models for High-Dimensional Vectors

With our table ready, we need a way to convert text into meaningful vectors. An embedding is a dense vector representation of a piece of data (in our case, text) where semantically similar items are located closer together in the vector space.
Google's Vertex AI offers state-of-the-art embedding models that are managed, scalable, and optimized for various tasks. For our RAG use case, we'll use the text-embedding-004 model, which generates a 768-dimensional vector. Its task_type parameter is specifically designed to optimize embeddings for retrieval, making it ideal for creating a searchable knowledge base.
Here’s a JSON-to-Video Automated Rendering Engine snippet demonstrating how to generate embeddings for a batch of text chunks using the Vertex AI SDK:

import vertexai
from vertexai.language_models import TextEmbeddingModel
def generate_embeddings(
project_id: str,
location: str,
text_chunks: list[str]
) -> list[list[float]]:
"""Generates embeddings for a list of text chunks."""
vertexai.init(project=project_id, location=location)
# We use the latest text embedding model, optimized for retrieval
model = TextEmbeddingModel.from_pretrained("text-embedding-004")
# The 'task_type' is critical for optimizing vectors for RAG
# 'RETRIEVAL_DOCUMENT' is used for the text being indexed.
# 'RETRIEVAL_QUERY' would be used for the user's input query.
embeddings = model.get_embeddings(
text_chunks,
task_type="RETRIEVAL_DOCUMENT"
)
# Extract the numerical vector from the response object
return [embedding.values for embedding in embeddings]
# --- Example Usage ---
my_project_id = "gcp-project-id"
my_location = "us-central1"
my_chunks = [
"Apache Iceberg is an open table format for huge analytic datasets.",
"BigQuery vector search enables efficient similarity search on embeddings.",
"A Lakehouse architecture combines the benefits of data lakes and data warehouses."
]
vector_embeddings = generate_embeddings(my_project_id, my_location, my_chunks)
# The output 'vector_embeddings' is a list of lists,
# where each inner list is a 768-dimension vector.
print(f"Generated {len(vector_embeddings)} embeddings.")
print(f"Dimension of first embedding: {len(vector_embeddings[0])}")

This function is the core of our text-to-vector transformation. It takes a list of strings and returns a corresponding list of 768-dimension floating-point vectors, ready to be inserted into our Iceberg table.

Batch Ingestion and Structuring Data for Vectorization

Now we connect the pieces. The final step is to create a scalable batch pipeline that reads raw documents, processes them into chunks, generates embeddings, and loads the results into our BigQuery Iceberg table.
1. Data Sourcing and Chunking
Your enterprise data likely lives in various formats (PDFs, DOCX, HTML) and locations (GCS, Confluence, etc.). The first task is to extract the raw text. Once you have the text, you must break it down into smaller, semantically meaningful chunks. This is perhaps the most important tuning parameter in a RAG system.

Why Chunk? LLMs have a limited context window, and embedding models work best on focused, concise pieces of text. Sending an entire 100-page document to an embedding model is ineffective.
Chunking Strategy: A simple fixed-size chunk (e.g., 500 characters) is a start, but it can awkwardly split sentences or ideas. A better approach is to use a recursive character text splitter, which tries to split on natural boundaries like paragraphs (\n\n), then sentences (.), then spaces (). Adding a small overlap between chunks (e.g., 50 characters) helps preserve context across boundaries. 2. The Batch Processing Pipeline A production-grade ingestion pipeline can be orchestrated with tools like Cloud Run, Cloud Functions, or Apache Beam on Dataflow. The logic remains the same:

# This is a conceptual pipeline structure, not a complete, runnable script.
# You would use libraries like 'google-cloud-bigquery' and 'pypdf'
from google.cloud import bigquery
import uuid
def process_and_ingest_documents(documents_to_process: list[str]):
"""
Conceptual pipeline to chunk, embed, and ingest documents.
"""
all_rows_to_insert = []
for doc_path in documents_to_process:
# Step 1: Extract text from the source document (e.g., a PDF in GCS)
raw_text = extract_text_from_pdf(doc_path) # Your custom text extraction logic
# Step 2: Chunk the text using a chosen strategy
text_chunks = chunk_text_recursively(raw_text, chunk_size=512, chunk_overlap=50)
# Step 3: Generate embeddings for the chunks in batches
# (API has a limit on items per call)
chunk_embeddings = generate_embeddings(
project_id="gcp-project-id",
location="us-central1",
text_chunks=text_chunks
)
# Step 4: Structure the data for insertion
for i, chunk in enumerate(text_chunks):
row = {
"chunk_id": str(uuid.uuid4()),
"doc_source": doc_path,
"chunk_text": chunk,
"embedding": chunk_embeddings[i],
"created_at": "CURRENT_TIMESTAMP()" # Let BigQuery handle this
}
all_rows_to_insert.append(row)
# Step 5: Batch load the data into the BigQuery Iceberg table
# The BigQuery Python client can handle streaming inserts or batch loads from GCS
if all_rows_to_insert:
client = bigquery.Client()
table_id = "your_dataset.doc_embeddings_iceberg"
# For large volumes, loading from a file (JSON, Parquet) in GCS is more robust
# For simplicity, this example uses streaming inserts
errors = client.insert_rows_json(table_id, all_rows_to_insert)
if not errors:
print(f"Successfully inserted {len(all_rows_to_insert)} rows.")
else:
print(f"Encountered errors while inserting rows: {errors}")
# --- Example Invocation ---
# In a real pipeline, this list would come from scanning a GCS bucket
source_docs = ["gs://my-knowledge-base/doc1.pdf", "gs://my-knowledge-base/doc2.pdf"]
process_and_ingest_documents(source_docs)

By executing this pipeline, you systematically convert your unstructured enterprise documents into a structured, vectorized dataset within your Lakehouse. This Iceberg table is now the single source of truth for your knowledge base, ready to be indexed for lightning-fast similarity search in the next step.

Step 2: Indexing and Searching Directly on Iceberg Tables

With our embeddings now residing in an Apache Iceberg table managed by BigQuery, we can unlock the power of high-performance retrieval without any data movement. This is where the tight integration between the Lakehouse storage format and BigQuery's analytical engine truly shines. We will create a vector index directly on the Iceberg table, enabling low-latency similarity searches that are essential for a responsive RAG application.

Creating a Vector Index on Your BigQuery Iceberg Table

A vector index is a specialized data structure that reorganizes your high-dimensional embedding data to enable Approximate Nearest Neighbor (ANN) search. Instead of exhaustively comparing a query vector to every single vector in your table (a brute-force approach), the index allows the system to quickly narrow down the search to a small, promising subset of candidates. This is the key to achieving millisecond-level latency on datasets with millions or even billions of vectors.
In BigQuery, creating a vector index is a straightforward DDL operation. Let's assume our Iceberg table is named rag_documents and has the following simplified schema:

doc_id (STRING, PRIMARY KEY)
chunk_text (STRING)
embedding (ARRAY<FLOAT64>) You would create an index on the embedding column using the CREATE VECTOR INDEX statement.

CREATE VECTOR INDEX my_doc_index
ON my_dataset.rag_documents(embedding)
OPTIONS(
index_type = 'IVF',
distance_type = 'COSINE',
ivf_options = '{"num_lists": 500}'
);

Let's break down the OPTIONS:

index_type = 'IVF': This specifies the Inverted File Index, a highly efficient and widely used ANN indexing algorithm. IVF works by clustering the vectors into partitions (or lists). During a search, it only inspects the partitions closest to the query vector, dramatically reducing the search space.
distance_type = 'COSINE': This defines the metric used to measure similarity. For embeddings generated by modern transformer models (like those from Vertex AI or OpenAI), COSINE similarity is almost always the correct choice. It measures the angle between two vectors, making it robust to differences in vector magnitude. Other options include EUCLIDEAN (L2 distance) and DOT_PRODUCT.
ivf_options = '{"num_lists": 500}': This JSON string configures the IVF index. The most critical parameter is num_lists, which sets the number of partitions to create.
Choosing num_lists: The optimal value depends on your dataset size. A good starting point is the square root of the number of rows in your table. For a table with 1 million vectors, a value between 100 and 1,000 is reasonable. A higher num_lists can lead to faster queries but may require tuning the query-time probe_count (which we'll cover next) to maintain high recall. Index creation is an asynchronous background job. You can monitor its progress by querying the information schema:

SELECT
index_name,
table_name,
coverage_percentage,
last_refresh_time
FROM
my_dataset.INFORMATION_SCHEMA.VECTOR_INDEXES
WHERE
table_name = 'rag_documents';

A coverage_percentage of 100 indicates that the index is fully built and ready for use. BigQuery automatically keeps the index updated as new data is inserted into your Iceberg table.

Executing Low-Latency Similarity Searches with the VECTOR_SEARCH Function

Once the index is active, you can perform searches using the VECTOR_SEARCH function. This function is the core of the retrieval step in your RAG pipeline. It takes a query vector and efficiently finds the top_k most similar vectors from your indexed table.
The basic syntax is:
VECTOR_SEARCH(TABLE table_name, column_to_search, query_vector, top_k => k, options => '...')
Here is a practical example. Imagine your application has generated an embedding for the user's question, "What are the latest query optimization techniques?". You would use that embedding to find the most relevant document chunks.

-- Assume @query_embedding is a query parameter passed from your application
-- For this example, we'll use a placeholder array.
DECLARE query_embedding ARRAY<FLOAT64>;
SET query_embedding = [0.1, 0.2, 0.3, ...]; -- Your 768 or 1536-dimension query vector
SELECT
base.doc_id,
base.chunk_text,
search_results.distance
FROM
VECTOR_SEARCH(
TABLE my_dataset.rag_documents,          -- The table with the index
'embedding',                             -- The indexed column
query_embedding,                         -- The vector to search for
top_k => 10,                             -- Number of results to return
options => '{"probe_count": 20}'
) AS search_results
-- Join back to the base table to retrieve the actual text content
JOIN
my_dataset.rag_documents AS base
ON
search_results.doc_id = base.doc_id
ORDER BY
search_results.distance; -- COSINE distance is 0 for identical, 2 for opposite

Key Points:

The options Parameter: The probe_count option is the most important performance tuning knob at query time. It tells the IVF index how many partitions (or lists) to inspect during the search. A higher probe_count increases the chance of finding the true nearest neighbors (higher recall) at the cost of slightly higher latency. A good starting value is sqrt(num_lists).
The JOIN Pattern: VECTOR_SEARCH returns the primary key columns of your table (doc_id in this case) and the distance for each match. You must JOIN these results back to your base table to retrieve other columns like chunk_text, which you'll need to pass to the Large Language Model.

Query Optimization for Production Workloads

For a production RAG system, performance and accuracy are paramount. Simply running a basic vector search is often not enough. You need to consider filtering and tuning to ensure your application is both fast and relevant.

Pre-filtering vs. Post-filtering

A common requirement in RAG is to search only within a subset of documents. For example, you might want to find information relevant only to a specific user, product, or date range. There are two ways to apply these filters:

Post-filtering (Inefficient): You run the vector search on the entire table and then apply a WHERE clause to the final result set. This is highly inefficient because the vector search wastes resources finding top matches that are immediately discarded by the filter.
Pre-filtering (Efficient): You apply the filter before the vector search. BigQuery's engine is smart enough to push these predicates down, meaning the vector search only operates on the subset of data that matches your filter. This dramatically reduces the search space, lowers latency, and reduces cost. To implement pre-filtering, apply the WHERE clause to the base table within the VECTOR_SEARCH function itself. Let's add a source_year column to our table and compare the two approaches. Inefficient Post-filtering:

-- AVOID THIS PATTERN
SELECT
base.doc_id,
base.chunk_text
FROM
VECTOR_SEARCH(
TABLE my_dataset.rag_documents, 'embedding', @query_embedding, top_k => 10
) AS s
JOIN
my_dataset.rag_documents AS base ON s.doc_id = base.doc_id
WHERE
base.source_year > 2022; -- Filter is applied AFTER the expensive search

Efficient Pre-filtering:

-- USE THIS PATTERN
SELECT
base.doc_id,
base.chunk_text
FROM
VECTOR_SEARCH(
-- The filter is applied to a subquery on the base table
TABLE (SELECT * FROM my_dataset.rag_documents WHERE source_year > 2022),
'embedding',
@query_embedding,
top_k => 10
) AS s
JOIN
my_dataset.rag_documents AS base ON s.doc_id = base.doc_id;

By filtering the table before it's passed to VECTOR_SEARCH, you ensure the ANN search is performed only on the relevant slice of your data, leading to significant performance gains in production workloads. This is a critical optimization for building scalable, multi-tenant RAG applications on the Lakehouse.

Step 3 Grounding the Conversational Agent

With a robust retrieval mechanism in place, the next critical step is to use the retrieved information to generate a coherent, accurate, and contextually relevant answer. This is the "Generation" part of Retrieval-Augmented Generation (RAG). It involves skillfully weaving the search results from BigQuery into a prompt that instructs a Large Language Model (LLM) on how to synthesize a final response. This process transforms raw, retrieved data into a conversational and helpful answer, ensuring the model's output is grounded in the facts contained within our Iceberg table.

Integrating Vector Search Results into a Language Model Prompt

The core of grounding lies in Prompt Engineering for Reliable Autonomous Workspace Agents for Reliable Autonomous Workspace Agents. We are not simply asking the LLM a question; we are providing it with a specific set of instructions and the exact context it must use to formulate its answer. A well-structured RAG prompt is the key to minimizing hallucinations and ensuring factual consistency.

A typical RAG prompt consists of three main components:

System Instructions: This is the preamble that defines the LLM's persona, its task, and its constraints. It's where you enforce the rule that the model must base its answer only on the provided context. This is your primary defense against the model reverting to its parametric knowledge and making things up.
Retrieved Context: This is the payload from our BigQuery VECTOR_SEARCH query. We take the text from the top-k retrieved document chunks and concatenate them into a single block of text. It's good practice to clearly delineate each document chunk, for instance, by numbering them or separating them with a distinct marker.
The User's Question: The final part of the prompt is the original query from the user. By combining these elements, we create a single, comprehensive prompt that gives the LLM everything it needs to generate a grounded response. Here is a template illustrating this structure. Notice how we use placeholders like {context} and {question} which our application logic will replace with the actual data at runtime.

You are an expert Q&A system that is a world-class expert on internal company documentation.
Your instructions are:
1. Answer the user's QUESTION based ONLY on the provided CONTEXT.
2. Do not use any prior knowledge or information outside of the CONTEXT.
3. If the CONTEXT does not contain the answer, you MUST state that you cannot answer the question with the information provided.
4. Synthesize the information from the CONTEXT into a clear and concise answer. Do not simply copy and paste sections.
5. If the CONTEXT includes source URIs, cite the relevant sources in your answer.
---
CONTEXT:
&#123;context&#125;
---
QUESTION:
&#123;question&#125;
Final Answer:

Building the RAG Logic to Synthesize Answers from Retrieved Context

The orchestration logic is the glue that connects our BigQuery vector index to the LLM. This logic, typically implemented in an application backend (e.g., a Python service running on Cloud Run or a Cloud Function), executes a precise sequence of operations for every incoming user query.
The end-to-end flow is as follows:

Receive Query: The application receives a question from the end-user.
Embed Query: The raw question string is passed to the same text embedding model (e.g., textembedding-gecko@003) that was used to embed the documents in our Iceberg table. This generates a query vector.
Execute Vector Search: The application constructs and executes a VECTOR_SEARCH query against BigQuery, passing the query vector as a parameter. It retrieves the base_document (the original text chunk) and any other relevant metadata for the top-k most similar documents.
Format Context: The retrieved base_document texts are collected and formatted into a single string. For example, they can be joined together with a separator like \n---\n. This string will replace the {context} placeholder in our prompt template.
Construct Final Prompt: The formatted context string and the original user question are injected into the predefined prompt template.
Invoke LLM: The complete, final prompt is sent to a generative model API, such as Vertex AI's Gemini 1.0 Pro (gemini-1.0-pro).
Return Response: The LLM processes the prompt and generates a response based on the provided context. This response is then returned to the user, completing the RAG cycle. This sequence ensures that every answer is freshly generated based on the most relevant documents available in the Lakehouse at that moment.

Example Implementation of a Question-Answering Pipeline

Let's translate the logic above into a practical Python implementation. This example uses the google-cloud-bigquery and vertexai client libraries to orchestrate the entire pipeline. This function encapsulates the full RAG process: embedding the query, searching BigQuery, and generating the final answer with Gemini.

import vertexai
from vertexai.language_models import TextEmbeddingModel, TextGenerationModel
from google.cloud import bigquery
# --- Configuration ---
PROJECT_ID = "your-gcp-project-id"
LOCATION = "US"
BQ_DATASET = "rag_dataset"
BQ_TABLE = "iceberg_docs_embedded"
EMBEDDING_MODEL_NAME = "textembedding-gecko@003"
GENERATION_MODEL_NAME = "gemini-1.0-pro" # Or your preferred Gemini model
# --- Initialize clients ---
vertexai.init(project=PROJECT_ID, location=LOCATION)
bq_client = bigquery.Client(project=PROJECT_ID)
embedding_model = TextEmbeddingModel.from_pretrained(EMBEDDING_MODEL_NAME)
# It's best practice to initialize the model once
generation_model = TextGenerationModel.from_pretrained(GENERATION_MODEL_NAME)
PROMPT_TEMPLATE = """
You are an expert Q&A system that is a world-class expert on internal company documentation.
Your instructions are:
1. Answer the user's QUESTION based ONLY on the provided CONTEXT.
2. Do not use any prior knowledge or information outside of the CONTEXT.
3. If the CONTEXT does not contain the answer, you MUST state that you cannot answer the question with the information provided.
4. Synthesize the information from the CONTEXT into a clear and concise answer.
---
CONTEXT:
{context}
---
QUESTION:
{question}
Final Answer:
"""
def get_rag_response(question: str, top_k: int = 5) -> str:
"""
Orchestrates the RAG pipeline:
1. Embeds the user question.
2. Searches BigQuery for relevant documents.
3. Generates a response using an LLM.
"""
# 1. Embed the user's question
question_embedding = embedding_model.get_embeddings([question])[0].values
# 2. Execute VECTOR_SEARCH in BigQuery
sql_query = f"""
SELECT
base_document,
distance
FROM
VECTOR_SEARCH(
TABLE `{PROJECT_ID}.{BQ_DATASET}.{BQ_TABLE}`,
'embedding',
(SELECT {question_embedding} AS embedding),
top_k => {top_k},
distance_type => 'COSINE'
)
"""
query_job = bq_client.query(sql_query)
results = query_job.result()
# 3. Format the retrieved context
context_chunks = [row.base_document for row in results]
if not context_chunks:
return "I could not find any relevant information to answer your question."
context_string = "\n\n---\n\n".join(context_chunks)
# 4. Construct the final prompt
final_prompt = PROMPT_TEMPLATE.format(context=context_string, question=question)
# 5. Invoke the LLM to generate the final answer
response = generation_model.predict(
prompt=final_prompt,
temperature=0.2,
max_output_tokens=1024,
top_k=40,
top_p=0.95,
)
return response.text
# --- Example Usage ---
if __name__ == '__main__':
user_question = "What are the key performance metrics for the Q3 marketing campaign?"
answer = get_rag_response(user_question)
print("--- Question ---")
print(user_question)
print("\n--- Answer ---")
print(answer)

Production Considerations: Security, Performance, and Governance

Moving a Retrieval-Augmented Generation (RAG) system from a proof-of-concept to a production environment introduces a host of non-functional requirements that are critical for success. It's no longer just about getting the right answer; it's about delivering that answer securely, performantly, and in a way that aligns with your organization's governance and cost management principles. Building your RAG system on a lakehouse architecture with BigQuery and Iceberg provides a powerful foundation to address these challenges head-on, allowing you to leverage existing enterprise-grade features rather than building new solutions from scratch.

Leveraging Existing Lakehouse Security for AI Workloads

One of the most significant advantages of this architecture is the ability to extend your existing data security and governance framework to your AI workloads. Your vector embeddings and source documents are not siloed in a separate, specialized database; they are first-class citizens within your BigQuery lakehouse, inheriting its robust security posture.
Unified Access Control with IAM:
Access to both the source Apache Iceberg tables and the BigQuery vector indexes is managed through Google Cloud's Identity and Access Management (IAM). This means you can use the same roles and permissions you've already defined for your analytical workloads. There's no need to manage a separate set of credentials or access policies for your RAG application's data layer. A service account for your RAG application can be granted a fine-grained role, like roles/bigquery.dataViewer, on only the specific datasets it needs to access.
Fine-Grained Data Segmentation:
For sensitive data, you can enforce granular control using BigQuery's built-in security features:

Column-Level Security (CLS): Restrict access to specific columns containing sensitive information. For instance, you could prevent the embedding model pipeline from accessing columns with Personally Identifiable Information (PII) in your source Iceberg table, even if other parts of the table are needed.
Row-Level Security (RLS): This is a game-changer for multi-tenant or department-specific RAG applications. You can create policies that filter which rows (i.e., which documents or text chunks) are visible to a user or service account based on their identity. An HR-specific RAG bot, for example, could be restricted to only query documents where department = 'HR'.

-- Example of a Row-Level Access Policy
-- This policy ensures that users can only query vectors
-- related to their own department.
CREATE ROW ACCESS POLICY department_filter
ON my_project.my_dataset.document_embeddings
GRANT TO ("group:sales-team@example.com")
FILTER USING (department = 'Sales');

Auditing and Lineage:
Every query, including vector searches, is logged in Cloud Audit Logs. This provides an immutable record of what data was accessed, by whom, and when. This is invaluable for compliance, security audits, and debugging. You can trace a specific generated response back to the exact VECTOR_SEARCH query that was run, providing full data lineage for your AI application's knowledge retrieval step.
Network Security with VPC Service Controls:
For organizations with stringent data exfiltration requirements, you can place your BigQuery datasets and the underlying Cloud Storage buckets for your Iceberg tables within a VPC Service Controls perimeter. This creates a virtual network boundary, ensuring that your sensitive data and embeddings can only be accessed by authorized services and networks, effectively preventing data from leaving your trusted environment.

Benchmarking Indexing and Query Performance

Performance in a RAG system is a multi-faceted concern, primarily revolving around the trade-off between search quality (recall) and speed (latency). A systematic benchmarking approach is essential to find the right balance for your application's Service Level Objectives (SLOs).
Indexing Performance:
The creation of a vector index in BigQuery is an asynchronous, back-end process. The time it takes is influenced by the number of vectors, their dimensionality, and the index configuration. While you don't need to manage the underlying compute, you should monitor the build process.
You can track the progress of index creation using the INFORMATION_SCHEMA:

SELECT
table_name,
index_name,
coverage_percentage,
last_refresh_time
FROM
`my_project.my_dataset.INFORMATION_SCHEMA.VECTOR_INDEXES`
WHERE
table_name = 'document_embeddings';

An index is queryable before it reaches 100% coverage, but performance and recall will improve as it approaches full coverage. For production systems, your data ingestion pipeline should have a step to verify that the coverage_percentage is 100 before routing live traffic to a newly refreshed index.
Query Performance: The Latency vs. Recall Trade-off:
Approximate Nearest Neighbor (ANN) search, which powers VECTOR_SEARCH, is designed to be fast by trading perfect accuracy for speed.

Latency: The time it takes for the VECTOR_SEARCH function to return results. This is a critical metric for user-facing applications.
Recall: The percentage of the true nearest neighbors that are returned by the query. 100% recall would be equivalent to a brute-force (exact) search, which is computationally expensive. In BigQuery, the primary tuning knob for this trade-off is the num_lists_to_search option within ivf_options. A higher value instructs the query engine to scan more of the index's "inverted file" lists, increasing the probability of finding the true nearest neighbors (higher recall) at the cost of increased processing and higher latency. A Practical Benchmarking Strategy:
Establish Ground Truth: On a representative sample of your data (e.g., 10k-100k vectors), run a brute-force distance calculation to find the true top-K nearest neighbors for a set of test queries. This is your "ground truth."
Run Experiments: Execute the VECTOR_SEARCH function against the full, indexed dataset using the same test queries. Vary the num_lists_to_search parameter for each run (e.g., 10, 20, 50, 100).
Measure and Plot: For each parameter setting, calculate the average query latency and the recall (i.e., (number of true neighbors found) / K).
Find the Sweet Spot: Plot your results on a latency vs. recall curve. This visualization will help you and your product stakeholders make an informed decision, choosing the lowest num_lists_to_search value that meets your application's minimum recall requirement, thereby optimizing for the lowest possible latency and cost.

Cost Management Strategies for Embedding and Search Operations

Generative AI workloads can become expensive if not managed carefully. A proactive approach to cost optimization is crucial for building a sustainable, production-grade RAG system.
1. Embedding Costs:
The initial and ongoing cost of generating embeddings via an external model API is often the largest component.

Incremental Embedding: This is the most effective cost-control strategy. Instead of re-embedding your entire corpus on every update, leverage Apache Iceberg's time-travel capabilities. By querying a snapshot of the table from the last time the pipeline ran, you can easily identify only the new or modified rows that require embedding. This transforms a potentially massive, expensive batch job into a small, efficient, and low-cost incremental update.
Model Selection: Carefully choose your embedding model. Higher-dimensional, state-of-the-art models are more expensive per token. Evaluate whether a smaller, more cost-effective model provides sufficient performance for your specific use case.

2. Indexing and Storage Costs:

Storage: BigQuery charges for the storage of the vector index itself. While typically much smaller than the raw data, this should be monitored via your Google Cloud billing reports.
Compute: The CREATE VECTOR INDEX job consumes BigQuery compute resources. Since this is often an infrequent operation (e.g., daily or weekly), its cost is usually predictable and manageable. Plan these jobs during off-peak hours if you are using a shared slot pool. 3. Query Costs: Vector search queries contribute to your BigQuery analysis costs.
Query Optimization: As determined during benchmarking, use the lowest possible num_lists_to_search that meets your recall SLOs. This directly reduces the amount of data processed per query.
Caching: For frequently asked questions or popular search terms, implement a caching layer (e.g., Redis, Memorystore) in your application to store the retrieved context. This avoids re-running the same VECTOR_SEARCH query repeatedly, saving significant cost and reducing latency.
Edition and Capacity Planning: Align your BigQuery edition with your workload. For applications with predictable, high query volumes, purchasing reserved slots with the Enterprise or Enterprise Plus editions can be more cost-effective than the on-demand model. BigQuery's autoscaling is ideal for handling spiky, unpredictable traffic without overprovisioning.
Monitoring and Alerting: Use the INFORMATION_SCHEMA.JOBS view to monitor the bytes processed by your vector search queries. Set up Cloud Billing budgets and alerts to get notified if costs exceed your forecasts, allowing you to take corrective action before you get a surprise at the end of the month.

Conclusion: Unifying AI and Data on the Lakehouse

We've journeyed through a paradigm shift in building AI applications—moving from complex, fragmented architectures to a streamlined, powerful model centered on the data lakehouse. By integrating vector search capabilities directly into BigQuery and leveraging the open standard of Apache Iceberg, we've demonstrated that production-grade RAG is not just feasible but fundamentally more efficient and secure when AI is brought to the data. This approach dissolves the traditional boundaries between analytical and AI workloads, paving the way for a new generation of data-driven, intelligent applications built directly on your source of truth.

Recap of Key Benefits: Reduced Complexity and Enhanced Security

The advantages of this unified architecture are immediate and substantial, primarily revolving around simplification and fortification.

Reduced Architectural Complexity: The most significant benefit is the radical simplification of the tech stack. By eliminating the need for a separate, specialized vector database, you collapse multiple systems into one. This means:
No More Data Duplication: Your embeddings and source data coexist, managed under a single storage layer with Apache Iceberg. This eradicates complex and brittle ETL pipelines designed solely to sync data between your analytical store and a vector database.
Streamlined MLOps: The entire lifecycle—from data ingestion and transformation in BigLake, to embedding generation with remote models, to indexing and querying with VECTOR_SEARCH—occurs within a cohesive GCP environment. This simplifies orchestration, monitoring, and maintenance.
Unified Interface: Developers and analysts can leverage familiar SQL to perform sophisticated similarity searches, lowering the barrier to entry and accelerating development cycles.
Enhanced Data Security and Governance: Centralizing your vector data within the lakehouse means it inherits the robust security posture of your core data platform.
Single Governance Pane: Data security is no longer a federated problem. All your existing BigQuery security controls—IAM permissions, column-level security, row-level access policies, and data masking—apply directly to your vector embeddings.
Minimized Data Egress: Sensitive data doesn't need to be exfiltrated to another system for processing. It remains within the secure perimeter of your Google Cloud project, drastically reducing the attack surface and simplifying compliance audits.

The Future of In-Database Vector Processing

What we've built here is not an endpoint but a glimpse into the future of data platforms. The trend is clear: databases are evolving from passive repositories into active, intelligent engines capable of handling diverse workloads, including AI. We can anticipate several exciting developments on this front:

Deeper Model Integration: Expect tighter integration of embedding models directly within the database engine. Imagine SQL functions that can generate embeddings on the fly from raw text or image data without calling an external endpoint, further simplifying the pipeline.
Advanced, Automated Indexing: While the IVFFlat index is powerful, the future will likely bring more advanced, native indexing strategies like HNSW (Hierarchical Navigable Small World) to platforms like BigQuery. This will offer even greater performance and flexibility, with the database optimizer intelligently choosing the best strategy for your query.
End-to-End In-Database RAG: The ultimate evolution is a fully managed, in-database RAG function. A single SQL query or stored procedure could potentially take a user's question, perform the vector search, retrieve context, pass it to a generative model like Gemini, and return a fully synthesized answer—all as a single, atomic database operation.
The Power of Open Formats: Apache Iceberg's role here is critical. It ensures that as these powerful in-database AI capabilities emerge across different engines (like Spark, Flink, or Trino), your underlying data remains open, accessible, and free from vendor lock-in. Your vector-enabled data lakehouse becomes a truly interoperable asset.

Next Steps and Further Resources

You now have the architectural blueprint for building a scalable, secure, and efficient RAG system on the lakehouse. The next step is to put it into practice.

Experiment and Adapt: Take the principles and code from this article and apply them to your own datasets. Start with a small-scale proof-of-concept to understand the nuances of index tuning and query performance for your specific use case.
Explore Advanced Features: Dive deeper into BigQuery's capabilities, such as using remote models for embedding generation or integrating the vector search results into complex analytical queries to uncover new insights.
Contribute to the Community: Engage with the open-source communities around Apache Iceberg and related projects. Sharing your findings and contributing to the ecosystem benefits everyone building on the modern data stack. To continue your journey, here are some valuable resources:
Google Cloud Documentation: BigQuery Vector Search Overview
Apache Iceberg Project: Official Iceberg Documentation
Google Cloud Blog: Building AI Applications on BigQuery
Vertex AI Model Garden: Explore Text Embedding Models

Gemma 4 on a Tesla T4, Part 2: The Minimum GCE VM and a Script to Drive It

xbill — Tue, 22 Sep 2026 17:24:01 +0000

This article provides a step by step deployment guide for **Gemma 4 E2B* to a Tesla T4 hosted GPU enabled system. A suite of Python MCP tools is built to simplify management of the vLLM hosted deployment. Part 1 measured which checkpoint the card runs fastest; this part builds the machine underneath it, installs the stack after first boot, and walks through vllm-t4, the shell script that owns the host state the MCP tools do not.*

github.com/xbill9/gemma4-dev/tree/main/gpu-vllm-t4-2b


Host	Compute Engine `n1-standard-2`, `us-west2-b` — 2 vCPU, 7.8 GB RAM
GPU	1x Tesla T4, Turing, compute capability 7.5, 15360 MiB
Image	`debian-13-trixie-v20260609`, no driver preinstalled
Disks	32 GB pd-ssd boot, 250 GB pd-balanced at `/opt1`
Software	vLLM 0.29.0, torch 2.13.0+cu130, driver 615.71.09
Result	362 s from `vllm-t4 start` to a healthy endpoint, 13371 MiB of 15360 claimed

Where Do I Start?

Part 1 began with the GPU already working. This part begins with a Google Cloud project and nothing in it.

The work splits in two. Everything before first boot is a single gcloud command whose choices are GPU choices: which zone sells a T4, what a GPU does to the maintenance policy, and how much disk to attach given where the checkpoints land. Everything after first boot is software on a Debian image that ships no NVIDIA driver at all.

At This Point You Should Have…

A Google Cloud project with billing enabled and the Compute Engine API turned on
GPU quota in the region you intend to use — T4 GPU quota in that region starts at zero on a new project
gcloud authenticated locally
A Hugging Face account, for the Gemma checkpoints

What the Machine Is

Read from the instance metadata server and the guest OS on the running VM:

machine-type: n1-standard-2
zone:         us-west2-b
image:        projects/debian-cloud/global/images/debian-13-trixie-v20260609
scheduling:   {"automaticRestart":"TRUE","onHostMaintenance":"TERMINATE","preemptible":"FALSE"}
disks:        [{"deviceName":"debian13","type":"PERSISTENT-SSD"},
               {"deviceName":"persistent-disk-1","type":"PERSISTENT-BALANCED"}]

Tesla T4, 7.5, 15360 MiB, 615.71.09
nproc: 2
MemTotal: 7436 MiB

On Compute Engine the T4 attaches to the N1 machine family, so the host shape is an N1 choice. This one is the smallest N1 that has been used for this work. Two vCPU costs startup time, and 7.8 GB of RAM is the number that decides the swapfile section below.

Creating the VM

The command below is reconstructed from the running VM's own metadata — machine type, zone, image, scheduling policy and both disk types are read back from the instance, and the accelerator from nvidia-smi. Re-running the create call was out of scope for this article.

gcloud compute instances create gemma4-t4 \
  --zone=us-west2-b \
  --machine-type=n1-standard-2 \
  --accelerator=type=nvidia-tesla-t4,count=1 \
  --maintenance-policy=TERMINATE \
  --restart-on-failure \
  --image-project=debian-cloud \
  --image-family=debian-13 \
  --boot-disk-size=32GB \
  --boot-disk-type=pd-ssd \
  --create-disk=name=gemma4-t4-data,size=250GB,type=pd-balanced,auto-delete=no

Three flags in there are about the GPU.

The Maintenance Policy Is Forced

--maintenance-policy=TERMINATE is required on any instance with an attached GPU. Compute Engine live-migrates ordinary VMs during host maintenance and cannot migrate one with a GPU, so the API rejects the default MIGRATE policy. The metadata confirms what the instance ended up with:

{"automaticRestart":"TRUE","onHostMaintenance":"TERMINATE","preemptible":"FALSE"}

Pairing it with --restart-on-failure means maintenance stops the VM and brings it back. The server process does not come back with it, and neither does the swapfile — there is a reboot checklist at the end of this article for that reason.

Two Disks, Because the Checkpoints Do Not Fit on One

The boot disk is 32 GB and the model cache goes on a second 250 GB disk mounted at /opt1:

NAME     SIZE TYPE MOUNTPOINT
sda       32G disk
├─sda1  31.9G part /
├─sda14    3M part
└─sda15  124M part /boot/efi
sdb      250G disk /opt1

A bf16 E2B checkpoint is 10.2 GB and the QAT build is 8.3 GB, so both together are most of a 32 GB root disk before pip has unpacked a CUDA torch wheel. Splitting them keeps the root disk for the OS and puts every multi-gigabyte write on the larger, cheaper volume. ~/.cache is a symlink to /opt1/cache, so Hugging Face downloads land there without any environment variable.

The data disk is mounted from /etc/fstab and survives a reboot:

UUID=8bd96fe8-e301-41d9-8ccc-8123ce89c4a8 /opt1 ext4 discard,defaults,nofail 0 2

Measure Disk Per Path, Never Once

Mounted on    1B-blocks        Avail
/           33570021376   8450138112
/tmp         3898789888   3897544704
/opt1      263086084096 200857059328

Three filesystems, and the two multi-gigabyte writes an install makes land on different ones. /tmp is its own 3.9 GB filesystem, which is where pip unpacks wheels, and a CUDA torch plus its NVIDIA runtime dependencies do not fit in it. A single df / reports the wrong answer for all three writes.

After Boot: the Driver

The Debian 13 image carries no NVIDIA driver. nvidia-smi does not exist on a fresh boot. The driver comes from NVIDIA's own CUDA repository for Debian 13:

/etc/apt/sources.list.d/cuda-debian13-x86_64.list:
deb [signed-by=/usr/share/keyrings/cuda-archive-keyring.gpg] https://developer.download.nvidia.com/compute/cuda/repos/debian13/x86_64/ /

sudo apt-get install -y linux-headers-$(uname -r)
wget https://developer.download.nvidia.com/compute/cuda/repos/debian13/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt-get update
sudo apt-get install -y nvidia-driver

What that pulls in:

linux-headers-7.1.8+deb13-cloud-amd64   7.1.8-1~bpo13+1
nvidia-driver                           615.71.09-2
nvidia-driver-cuda                      615.71.09-2
nvidia-kernel-open-dkms                 615.71.09-2
nvidia-kernel-support                   615.71.09-2

The kernel module is the open variant and DKMS compiles it against the running kernel, so the matching linux-headers package has to be installed first. The running kernel here is 7.1.8+deb13-cloud-amd64 and its headers come from backports, so the headers install gets its own line ahead of the driver.

Reboot, then check:

$ nvidia-smi --query-gpu=name,compute_cap,memory.total,driver_version --format=csv
name, compute_cap, memory.total [MiB], driver_version
Tesla T4, 7.5, 15360 MiB, 615.71.09

Compute capability 7.5 is Turing, and it is the number that governs everything in Part 1: no bfloat16 datapath, no fp8, and a 64 KiB shared-memory ceiling per block that the Triton attention kernel has to be clamped to fit.

The Swapfile Is Part of the Deploy

7.8 GB of RAM, and the VM ships with no swap. vLLM is killed by the kernel while staging E2B weights without some, which reads as a crash with no traceback.

sudo fallocate -l 16G /opt1/swapfile
sudo chmod 600 /opt1/swapfile
sudo mkswap /opt1/swapfile
sudo swapon /opt1/swapfile

The swapfile lives on the data disk, so the file itself persists. /etc/fstab has no swap line, so a reboot leaves the file in place and disabled. While serving the QAT build, 3592 MiB of the 16 GB is in use.

Python, and Where Packages Land

There are two interpreters on this host and the default is the one with no room:

python3             -> pyenv 3.12.13, site-packages on /        8.45 GB free
/usr/bin/python3.13 -> user site at /opt1/pyuser            200.86 GB free

The MCP server runs under python3. vLLM runs under /usr/bin/python3.13 with PYTHONUSERBASE=/opt1/pyuser, which puts its packages on the large disk. A relocated user site is the same interpreter's own site directory on a different filesystem, so site-packages stays system-wide and no virtualenv is involved.

Installing vLLM

Both redirections below are load-bearing on this host — one for where packages land, one for where pip unpacks them:

PYTHONUSERBASE=/opt1/pyuser PIP_CACHE_DIR=/opt1/pipcache TMPDIR=/opt1/tmp \
  /usr/bin/python3.13 -m pip install --user --break-system-packages -U \
  --upgrade-strategy eager \
  --extra-index-url https://download.pytorch.org/whl/cu130 vllm

vLLM pins torch to an exact version, so this moves torch with it. Installing torch with --no-deps skips cuDNN and the NVIDIA runtime wheels, and import torch then fails on libcudnn.so.9.

torch        2.13.0+cu130
transformers 5.17.0
triton       3.7.1
vllm         0.29.0

The published cu130 wheels carry sm_75, so Turing needs no source build. make arch asks the interpreter for the list instead of reading the directory.

The Turing Clamp

Gemma 4 mixes two attention widths — 256 in its sliding-window layers, 512 in its global layers — and vLLM forces its Triton backend for that mix. At width 512 the kernel asks for more shared memory than a Turing block allows. make patch clamps the tile sizes in the installed vLLM:

✅ **Patched**, in the site-packages this host's `python3` imports.

__FILE__/opt1/pyuser/lib/python3.13/site-packages/vllm/v1/attention/ops/triton_unified_attention.py
CLAMP PRESENT
OCCURRENCES 1

Reinstalling or upgrading vLLM reverts it, because the patch edits a file in site-packages and pip replaces that file. Re-run make verify-patch after any vLLM change.

Enter `vllm-t4`

At this point the host is ready and the remaining job is running the server. The rig's tpu.env holds every serving value, and the MCP tools cover capacity, architecture and patching. Three things are neither rig config nor MCP concerns: the swapfile, launching a process that outlives the shell, and waiting for the endpoint to answer.

~/bin/vllm-t4 owns those three. Every serving value it passes to vLLM is read out of the rig's tpu.env at call time and none is spelled in the script:

rigval() {
  sed -n "s/^$1=//p" "$rig/tpu.env" | tail -1
}

tpu.env is a dotenv file whose comments are prose, so sourcing it would both fail as shell and drag TMPDIR and PYTHONUSERBASE into the calling shell. Reading one key at a time keeps the file authoritative without importing it.

Why the Script Launches with `nohup`

make serve runs python3 -c "asyncio.run(server.start_vllm_server())". asyncio terminates the child subprocess when that short-lived interpreter exits, so the engine is gone about a second after launch, having written nothing to the log. The Part 1 sweep was started under the long-lived MCP server process, where the parent stays alive.

vllm-t4 start launches with nohup and disown from bash, and writes the same run/vllm.pid and run/vllm.log that the rig's own status and stop tools read, so the script and the MCP tools agree about what is running.

The Options

Command	What it does
`vllm-t4 start`	Enable swap, confirm the clamp, launch detached, wait for `/health`, print status
`vllm-t4 start nowait`	The same, returning as soon as the process is up
`vllm-t4 stop`	`make stop` — SIGTERM to the pid, VRAM released on exit
`vllm-t4 status`	`make status` — serving or not, plus claimed VRAM
`vllm-t4 query`	`make query` — one chat completion against the endpoint
`vllm-t4 log`	`tail -f` on `run/vllm.log`
`vllm-t4 swap`	Create and enable the swapfile, without starting anything

Four environment variables override the defaults:

Variable	Default	Use
`VLLM_T4_RIG`	`$HOME/gemma4-dev/gpu-vllm-t4-2b`	Point at a different rig directory
`VLLM_T4_SWAPFILE`	`/opt1/swapfile`	Put swap on another volume
`VLLM_T4_SWAPSIZE`	`16G`	Size it differently
`VLLM_T4_WAIT`	`1800`	Seconds to wait for `/health`

`start` Runs Three Guards Before It Launches

Each guard costs less than the failure it prevents.

Swap, because the engine is killed during weight loading without it and the kernel log is the only place that says so. ensure_swap creates the file if it is absent, enables it if it exists, and reports when it is already on.

The clamp, because an unpatched engine spends minutes loading and then dies with an out-of-resources error that gets attributed to configuration. The check matches the verifier's positive string and refuses on anything else:

case $out in
  *'✅ **Patched**'*) echo "$prog: Turing clamp confirmed" ;;
  *) echo "$out" >&2
     die "the Turing clamp is not confirmed -- run 'make -C $rig patch'" ;;
esac

A verifier that cannot import vLLM at all answers neither way, so a whitelist of known-bad strings would admit every unknown-bad one. Matching the single good answer fails closed.

The config, because an empty value from tpu.env becomes an empty CLI argument. Every key is checked for a value and PYTHON_BIN for executability before anything launches.

A Full Start

$ vllm-t4 start
vllm-t4: swap already on: /opt1/swapfile (16777212 KB)
vllm-t4: Turing clamp confirmed
vllm-t4: starting google/gemma-4-E2B-it-qat-w4a16-ct on 127.0.0.1:8000
vllm-t4: pid 17215, log /home/xbill_glitnir_com/gemma4-dev/gpu-vllm-t4-2b/run/vllm.log
vllm-t4: waiting up to 1800s for http://127.0.0.1:8000/health
     0s  VRAM 0 MiB, 0 %
    75s  VRAM 0 MiB, 0 %
    90s  VRAM 1093 MiB, 10 %
   105s  VRAM 9005 MiB, 10 %
   166s  VRAM 9301 MiB, 0 %
   181s  VRAM 8403 MiB, 0 %
   241s  VRAM 13371 MiB, 0 %
   347s  VRAM 13371 MiB, 0 %
vllm-t4: healthy after 362s -- http://127.0.0.1:8000
✅ Serving at http://127.0.0.1:8000 (pid 17215).

VRAM 13371 MiB, 15360 MiB, 0 %

VRAM is printed beside the clock because for the first minutes a compiling engine and a dead one look identical from outside, and claimed device memory is what tells them apart. The shape of that column is the startup: nothing for 75 seconds while Python imports and the weights are read off disk, 9005 MiB once the weights are resident, a dip to 8403 while the engine profiles, then 13371 when the KV cache is allocated.

What the Engine Allocated

The same start, in run/vllm.log:

non-default args: {'host': '127.0.0.1', 'model': 'google/gemma-4-E2B-it-qat-w4a16-ct', 'dtype': 'float16', 'max_model_len': 16384, 'gpu_memory_utilization': 0.9, 'max_num_seqs': 8}
Casting torch.bfloat16 to torch.float16.
Using MarlinLinearKernel for CompressedTensorsWNA16
Model loading took 8.02 GiB memory and 88.981559 seconds
Available KV cache memory: 4.66 GiB
GPU KV cache size: 519,681 tokens, Maximum concurrency for 16,384 tokens per request: 31.72x
init engine (profile, create kv cache, warmup model) took 61.77 s (compilation: 2.59 s)

Casting torch.bfloat16 to torch.float16 is the checkpoint's stored dtype meeting --dtype float16 from tpu.env. Turing has no bfloat16 datapath, so PyTorch would upconvert regardless; setting the flag makes the conversion a decision with a record.

Compilation took 2.59 s here against 112.77 s on the first start of this stack, because torch.compile caches its artifacts under ~/.cache and that cache is warm. A first start on a fresh VM pays the full compile, and on 2 vCPU it is the largest single item in the wall clock.

`status` and `query`

$ vllm-t4 status
✅ Serving at http://127.0.0.1:8000 (pid 17215).

VRAM 13371 MiB, 15360 MiB, 0 %

$ vllm-t4 query
{
    "model": "google/gemma-4-E2B-it-qat-w4a16-ct",
    "choices": [{"message": {"role": "assistant",
      "content": "A TPU, or Tensor Processing Unit, is a specialized type of integrated circuit designed to significantly accelerate the mathematical operations central to training and running machine learning models, particularly those involving matrix multiplications."},
      "finish_reason": "stop"}],
    "system_fingerprint": "vllm-0.29.0-9a66a08c",
    "usage": {"prompt_tokens": 18, "total_tokens": 56, "completion_tokens": 38}
}

That target posts to /v1/chat/completions. Raw /v1/completions returns an empty string on an instruction-tuned checkpoint, so an empty result there means the wrong endpoint was called.

Querying It Directly

The server is the standard OpenAI-compatible one, on 127.0.0.1:8000:

$ curl -fsS http://127.0.0.1:8000/v1/models
{"object":"list","data":[{"id":"google/gemma-4-E2B-it-qat-w4a16-ct","object":"model",
 "owned_by":"vllm","root":"google/gemma-4-E2B-it-qat-w4a16-ct","max_model_len":16384}]}

curl -fsS http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"google/gemma-4-E2B-it-qat-w4a16-ct",
       "messages":[{"role":"user","content":"Name the four inner planets, comma separated."}],
       "temperature":0,"max_tokens":64}'

Mercury, Venus, Earth, Mars
usage: {'prompt_tokens': 18, 'total_tokens': 26, 'completion_tokens': 8}

Streaming works the same way, with "stream": true:

chunks: 16
data: {"object":"chat.completion.chunk","choices":[{"delta":{"role":"assistant","content":""}}]}
data: {"object":"chat.completion.chunk","choices":[{"delta":{"content":"1"}}]}
...
data: [DONE]

assembled: 1, 2, 3, 4, 5

Prometheus metrics are on /metrics, which is the route to per-request counters without a benchmark tool:

vllm:num_requests_running{engine="0",model_name="google/gemma-4-E2B-it-qat-w4a16-ct"} 0.0
vllm:prompt_tokens_total{...} 53.0
vllm:generation_tokens_total{...} 49.0

The model id in every request body has to match what /v1/models reports, which is MODEL_NAME from tpu.env. Serving the bf16 build for an A/B means overriding MODEL_NAME and MODEL_SAFETENSORS_BYTES in the environment, and the id in the request body changes with it.

What Each Subcommand Refuses

$ vllm-t4 start                                  # one is already running
vllm-t4: already running (pid 14977) -- 'vllm-t4 status', or 'vllm-t4 stop' first   [exit 1]

$ vllm-t4 bogus
usage: vllm-t4 [start|stop|status|query|log|swap] [nowait]                          [exit 2]

$ vllm-t4 swap                                   # already enabled
vllm-t4: swap already on: /opt1/swapfile (16777212 KB)                              [exit 0]

$ VLLM_T4_RIG=/nonexistent vllm-t4 status
vllm-t4: rig not found: /nonexistent (set VLLM_T4_RIG)                              [exit 1]

If the engine dies during the wait, the script prints the last 20 log lines and points at the kernel log, because a memory kill leaves no traceback in vLLM's own output:

if [ -z "`running_pid`" ] ; then
  echo "$prog: the engine died during startup; last 20 log lines:" >&2
  tail -20 "$rig/run/vllm.log" >&2
  echo "$prog: if there is no traceback, check 'dmesg -T | grep -i oom'" >&2

Stopping

$ vllm-t4 stop
✅ Sent SIGTERM to vLLM (pid 14977). VRAM is released on exit.

The T4 stays attached and the VM stays billed — stopping the server is not releasing capacity here.

VRAM returns to 0 MiB, 15360 MiB within seconds. Swap is left enabled, since it costs nothing idle and the next start needs it.

The VM and its attached T4 bill by the hour whether vLLM runs or not. gcloud compute instances stop gemma4-t4 --zone us-west2-b stops the instance charge and keeps both disks, which keep billing at the much lower storage rate. Deleting the instance with auto-delete=no on the data disk leaves the 250 GB of checkpoints for the next VM.

After a Reboot

Three things come back on their own and one does not.

	Survives a reboot
`/opt1` data disk	🟢 in `/etc/fstab`
NVIDIA driver	🟢 DKMS module, loads at boot
Turing clamp	🟢 a file in site-packages
`/opt1/swapfile`	❌ the file persists, the `swapon` does not

vllm-t4 start re-enables swap every time, so the checklist after a reboot is one command. Adding a swap line to /etc/fstab makes it survive on its own.

Summary

The goal of this article was to build the smallest Compute Engine VM that serves Gemma 4 E2B on one Tesla T4, deploy the stack after first boot, and document the script that runs it. The key to the solution was keeping the rig's tpu.env authoritative for every serving value and giving the script only the three things that are host state: the swapfile, a detached launch, and waiting for the endpoint. The measured results were:

🟢 362 s from vllm-t4 start to a healthy endpoint on a warm compile cache, 13371 MiB of 15360 claimed
🟢 8.02 GiB of weights and 4.66 GiB of KV cache, giving 519,681 tokens and 31.72x concurrency at 16,384 tokens per request
🟢 One gcloud command, one apt repository for the driver, one pip install, one patch
⚠️ --maintenance-policy=TERMINATE is required for an attached GPU, and stops the VM during host maintenance
⚠️ 7.8 GB of host RAM needs a 16 GB swapfile, and the swapon does not survive a reboot
❌ A single df reports the wrong free space for all three writes the install makes

One Tesla T4 on one n1-standard-2 VM in us-west2-b, Debian 13, vLLM 0.29.0 on torch 2.13.0+cu130, driver 615.71.09. The start timing is a single run on a warm torch.compile cache; a first start on a fresh VM pays the full compilation, which was 112.77 s on this host. The gcloud compute instances create command is reconstructed from the running instance's metadata and was not re-executed. Throughput figures for the two checkpoints are in Part 1.

The strategy for using MCP for Tesla T4 deployment and benchmarking was validated with an incremental step by step approach.

What Nobody Is Using in Your Google Cloud Projects, and What It Costs

xbill — Tue, 22 Sep 2026 15:14:36 +0000

This article provides a step by step guide to building a Google Cloud waste scanner from source, running it across every project your credentials can see, pricing each finding from the Cloud Billing Catalog API, and drafting the cleanup. A suite of Python checks is built to cover Compute Engine, Cloud SQL, Storage, DNS, KMS, Secret Manager, Artifact Registry, Filestore, Cloud Logging, Cloud Monitoring and GKE.

https://github.com/xbill9/zombiescan-gcp

What Gets Left Behind

A Persistent Disk survives the instance it was attached to. A static IP outlives the migration that freed it. A GKE cluster keeps charging its management fee after the last node pool scaled to zero. Each one bills every hour and reports nothing.

The billing console shows the total. The Recommender surfaces candidates. A figure that drives a decision names one resource, in one project, and its monthly cost.

zombiescan produces that figure for 20 classes of resource across two packs.

At This Point You Should Have…

A Google Cloud account and Application Default Credentials, from gcloud auth application-default login
uv on the path, and Python 3.11 or newer
Read access to the projects you mean to scan — the predefined roles/viewer covers every call, and zombiescan-scanner-role.yaml, under policy/ in the repository, is a 16-permission custom role that covers exactly as much

Step 1 — Build It From Source

git clone https://github.com/xbill9/zombiescan-gcp
cd zombiescan-gcp
uv sync

Resolved 39 packages in 0.58ms
Checked 38 packages in 0.19ms

uv sync reads the committed uv.lock, so the resolved set is the one the tests ran against.

uv run zombiescan --version

zombiescan, version 0.1.0

Step 2 — Install It as a Tool

Working from the clone keeps uv run in front of every command. Installing it puts zombiescan on the path instead.

uv tool install git+https://github.com/xbill9/zombiescan-gcp

Both routes run the same engine. The rest of this article uses the installed form.

Step 3 — Authenticate

gcloud auth application-default login

google.auth.default() picks that up. No key file is downloaded, and no service account is created.

Every worker thread in a scan shares one credentials object, and the first token is fetched before the threads start. Refreshing credentials signs through OpenSSL, and one credentials object refreshed from several threads at once ends the process instead of raising, so the wrapper serialises the refreshes behind a lock.

Step 4 — List the Checks

zombiescan checks

  disabled-kms-key  Disabled KMS key versions still billing  (core)
  empty-vpc-network  VPC networks with nothing running in them  (core)
  gke-idle-cluster  GKE clusters running no nodes  (gke)
  idle-cloud-nat  Cloud NAT gateways serving no instances  (core)
  idle-filestore  Filestore instances nothing can mount  (core)
  idle-forwarding-rule  Load balancers with no backends  (core)
  orphaned-snapshot  Snapshots of deleted disks  (core)
  stale-artifact-repository  Artifact Registry repositories with no recent pushes  (core)
  stale-secret  Secrets nothing has updated in 90 days  (core)
  stopped-instance  Stopped instances still paying for disks  (core)
  stopped-sql-instance  Stopped Cloud SQL instances still paying for storage  (core)
  unattached-disk  Unattached Persistent Disks  (core)
  unbounded-log-bucket  Log buckets retaining logs indefinitely  (core)
  unmanaged-gcs-bucket  Versioned buckets with no lifecycle rule  (core)
  unused-dns-zone  Cloud DNS zones publishing nothing  (core)
  unused-firewall-rule  Firewall rules matching nothing  (core)
  unused-image  Custom images nothing boots from  (core)
  unused-static-ip  Unused static IP addresses  (core)
  unused-subnet  Subnets reserving ranges nothing uses  (core)
  unused-uptime-check  Uptime checks monitoring deleted resources  (core)

Nineteen ship in the core pack and one in gke. GKE sits in its own pack because it carries its own API, its own rate section and its own price fetcher, which is what a third-party pack has to supply.

Step 5 — Which APIs the Checks Need

zombiescan apis

  artifactregistry.googleapis.com  1 check(s)
  cloudkms.googleapis.com  1 check(s)
  compute.googleapis.com  12 check(s)
  container.googleapis.com  1 check(s)
  dns.googleapis.com  1 check(s)
  file.googleapis.com  1 check(s)
  logging.googleapis.com  1 check(s)
  monitoring.googleapis.com  1 check(s)
  secretmanager.googleapis.com  1 check(s)
  sqladmin.googleapis.com  1 check(s)
  storage.googleapis.com  1 check(s)

An API that is not enabled on a project is skipped, not reported as an error: a
project that never used a service has no waste in it.

Each check declares the APIs it calls. That declaration generates this list and the read-only custom role, and the test suite fails when a check names an API the role does not cover.

Step 6 — Scan One Project

zombiescan scan --project glitnir-dev

Scanning as application default credentials
1 project(s), 20 check(s) — read-only

zombiescan — 2 findings across 1 project

Check                      Found  Monthly
stale-artifact-repository      1    $0.01
empty-vpc-network              1    $0.00

Every call is a list or a get. The scan has no code path that deletes, modifies or releases anything.

Step 7 — Scan Every Project

zombiescan scan --all-projects

Scanning as application default credentials
75 project(s), 20 check(s) — read-only

zombiescan — 713 findings across 75 projects

Check                      Found  Monthly
unattached-disk               36  $153.53
unused-static-ip              12   $89.06
unused-image                  64   $75.25
stopped-instance              12   $35.76
orphaned-snapshot             39   $28.66
stale-artifact-repository     13    $4.75
stale-secret                   9    $0.54
unused-dns-zone                1    $0.20
empty-vpc-network             13    $0.00
unmanaged-gcs-bucket         118    $0.00
unused-firewall-rule          43    $0.00
unused-subnet                349    $0.00
unused-uptime-check            4    $0.00

Estimated waste: $387.74/month ($4,652.87/year)

--all-projects asks Resource Manager's projects.search which projects the caller can see, then fans out over them. 75 projects times 20 checks is 1,500 project/check pairs, and 411 of them were skipped because the API was not enabled on that project. The sweep took 79 seconds.

Thirteen of the 20 checks fired. Five of those thirteen classes carry no charge and account for 527 of the 713 findings: empty VPC networks, versioned buckets with no lifecycle rule, firewall rules matching nothing, unused subnets and uptime checks monitoring deleted resources. They are reported because they accumulate without limit and block deletions.

Step 8 — Read the Per-Resource Table

The detail table sorts the costliest findings first and guarantees a row to every check that fired.

┏━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ Project        ┃ Location                ┃ Resource               ┃  Monthly ┃
┡━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
│ comglitn       │ global                  │ w5-opt                 │   $53.19 │
│ glitnir-ba1    │ europe-west1-b          │ disk-20260326-160533   │   $50.00 │
│ glitnir-ba1    │ europe-west1-c          │ disk-20260326-152217   │   $50.00 │
│ glitnir-mb1    │ us-east1-c              │ mb1                    │   $21.76 │
│ comglitn       │ us-central1-c           │ disk-opt-w1            │   $20.00 │
│ xbill-b1       │ us                      │ snapshot-b1-east       │   $16.57 │
│ comglitn       │ us-central1-c           │ w2                     │   $10.88 │
│ glitnir-sx1    │ us-east4                │ sx2                    │    $8.03 │
│ swoonbox1      │ us-east4                │ swoonbox-landing-4     │    $8.03 │
│ comglitn       │ us-central1             │ w1                     │    $7.30 │
└────────────────┴─────────────────────────┴────────────────────────┴──────────┘

Project is the first column because a resource id is unique inside a project and nowhere else. Location holds a zone, a region or global, read off each resource.

A ~ marks an estimate or an upper bound, and the per-finding note in JSON output names the reason.

🔎 Tip: --min-cost 5 hides findings under five dollars a month, --limit 0 prints every row instead of the top 25, and --location us-central1 keeps a region's zones along with the region itself.

Step 9 — Where a Check Runs

A check runs once per project. Two properties of Google's APIs make that a complete sweep.

Compute Engine's aggregatedList returns disks, addresses, instances, routers, subnetworks and forwarding rules across every zone and region in one call. Most other APIs accept locations/-, a wildcard meaning every location.

Cloud KMS and Artifact Registry reject locations/-. Those two enumerate the API's locations and walk them in parallel, and each worker gets its own client, because the HTTP connection underneath a discovery client cannot be shared between threads.

Step 10 — Where the Prices Come From

The bundled price table is generated from the Cloud Billing Catalog API.

python -m zombiescan.pricing.refresh

A scan reads table.json from the installed package, so it runs offline and adds no latency. Rates are looked up by key and region: ctx.pricing.rate("disk.gb_month", region=...).

Verified rates in us-central1, from the bundled table:

Rate	us-central1
pd-standard	$0.04/GB/month
pd-balanced	$0.10/GB/month
pd-ssd	$0.17/GB/month
Snapshots and custom images	$0.05/GB/month
Idle static IP	$0.010/hour
Forwarding rule minimum	$0.025/hour
GKE cluster	$0.10/hour
Secret version	$0.06/month
KMS software key version	$0.06/month
KMS HSM key version	$1.00/month
Artifact Registry storage	$0.10/GB/month

At 730 hours to the month, arithmetic over those hourly rates gives $7.30 a month for an idle static IP, $18.25 for a forwarding rule, and $73.00 for a GKE cluster with no nodes on it.

Step 11 — Read the First Tier That Charges

Google fronts many SKUs with a free allowance priced at zero: the first 30 GB of standard Persistent Disk, the first 0.5 GB of Artifact Registry, the first six secret versions. A fetcher that reads tier 0 records the rate as free, which prices every finding in that section at nothing and reports a clean project.

unit_price() takes the first tier that charges. Reading a named tier deliberately — the way the Cloud DNS fetcher walks the zone tiers at $0.20 for the first 25 and $0.10 beyond them — is a separate call.

A SKU published against the region global has no per-region entry at all. Cloud NAT addresses, Artifact Registry storage and log retention are all published that way, so they live in a global section with a scope="global" rate spec.

The refresher refuses to write a table that loses or empties a section the previous one had, because an empty section means a matcher stopped matching.

Step 12 — JSON Output

zombiescan scan --project glitnir-dev --json findings.json

{
  "schema_version": 3,
  "scan": {
    "generated": "2026-09-21T19:55:55Z",
    "duration_seconds": 5.41,
    "principal": "application default credentials",
    "projects": ["glitnir-dev"],
    "pairs_attempted": 20,
    "pairs_unavailable": 3,
    "complete": true
  },
  "pricing": {
    "generated": "2026-09-21T16:35:39Z",
    "basis": "Google Cloud Billing Catalog API, on-demand USD list prices",
    "excludes": ["committed use discounts", "sustained use discounts", "private pricing", "credits"]
  },
  "totals": {
    "monthly_cost": 0.01,
    "annual_cost": 0.12,
    "finding_count": 2
  }
}

Each finding carries its own reason, cost, remediation command and a details block:

{
  "check": "stale-artifact-repository",
  "resource_id": "gcr.io",
  "project": "glitnir-dev",
  "location": "us",
  "reason": "DOCKER repository holding 0.1 GB with nothing pushed to it in 282 days",
  "monthly_cost": 0.01,
  "approximate_cost": true,
  "remediation": "gcloud artifacts repositories delete gcr.io --location=us --project=glitnir-dev --quiet",
  "details": {
    "stored_gb": 0.1,
    "idle_days": 282,
    "usd_per_gb_month": 0.1,
    "note": "upper bound: Artifact Registry bills each unique layer once, and images sharing a base layer are counted once per image here"
  }
}

The pricing block dates the table and lists what list prices exclude, so a report read six months later says which rates produced it.

Step 13 — The HTML Report

zombiescan scan --project glitnir-dev --html report.html

One self-contained file with the styles inline, which prints to PDF without fetching anything.

Step 14 — The Cleanup Plan

zombiescan scan --project glitnir-dev --script cleanup.sh

#!/usr/bin/env bash
# zombiescan cleanup plan — generated 2026-09-21T19:55:55Z
#
# READ EVERY LINE BEFORE RUNNING THIS.
# zombiescan generated this file and did not run it. Deleting Google
# Cloud resources is not reversible. Where a backup is possible the
# command takes one first, but a backup is not a substitute for
# knowing what you are deleting.
#
# 2 resource(s), about $0.01/month.

set -euo pipefail

# glitnir-dev us gcr.io — DOCKER repository holding 0.1 GB with nothing pushed to it in 282 days
# saves about $0.01/month
gcloud artifacts repositories delete gcr.io --location=us --project=glitnir-dev --quiet

# glitnir-dev global default — VPC network runs no instances; it still holds none. This is the auto-created default network
# saves about $0.00/month
gcloud compute networks delete default --project=glitnir-dev --quiet

Every generated command carries --project and --quiet, and interpolated resource ids are shell-quoted. The test suite checks all three at the source, because a remediation command is printed for an operator to run.

Step 15 — Cleaning Up

clean is a separate command, and a dry run is what it does with no flags.

zombiescan clean --project glitnir-dev

Scanning as application default credentials — 1 project(s)

dry run — 1 of 2 finding(s) can be cleaned, 1 cannot
  skip empty-vpc-network default: a network cannot be deleted until everything
inside it is gone -- subnets, routes, firewall rules, Cloud Routers, peerings
and any Private Service Connect attachment -- and the order depends on what else
references them. Clean the findings inside the network first; this one goes away
with them

stale-artifact-repository  gcr.io  glitnir-dev · us · $0.01/mo
  DOCKER repository holding 0.1 GB with nothing pushed to it in 282 days
    → IRREVERSIBLE delete Artifact Registry repository gcr.io and its images
      artifactregistry.projects.locations.repositories.delete({'name':
'projects/glitnir-dev/locations/us/repositories/gcr.io'})

Would free about $0.01/month across 1 resource(s); 1 include irreversible steps.
Nothing was changed. Re-run with --apply to perform these.

--apply gates one thing: whether a planned step is sent to Google Cloud. It never changes which steps get planned, so the preview above is what runs.

Planning is read-only. A cleaner may read — fetching an instance's disks to clear autoDelete, reading a router's NAT list — and it yields the mutations as objects for the runner to send.

Where the API allows a backup, the backup step is ordered before the destruction, and a failed step stops the rest of that finding. A failed snapshot can never be followed by the delete that assumed it.

IRREVERSIBLE marks a step with no recovery window. A released static IP is gone, so it carries the mark; a destroyed KMS key version is held for 24 hours, so it does not.

A check with no cleaner reports the finding as unsupported with a reason, which is the skip line above. The test suite fails a check that has neither a cleaner nor a reason.

🔎 Tip: clean --from findings.json acts on a report already reviewed, and refuses a report produced by different credentials.

Step 16 — The Read-Only Role

gcloud iam roles create zombiescanScanner --project=PROJECT_ID \
  --file=policy/zombiescan-scanner-role.yaml --quiet

Sixteen permissions, every one a list or a get. The file is generated from the apis= each check declares, and the suite keeps it in step.

clean needs more than this, deliberately: the identity that reports waste should be unable to delete what it reports.

Step 17 — The Claude Code Plugin

/plugin marketplace add xbill9/zombiescan-gcp

The plugin ships two slash commands, a skill and an MCP server over the same engine.

  scan_project  Scan Google Cloud projects for unused resources and price them
  estimate_savings  Total, count and break down the findings in a report
  explain_finding  Explain why a check treats a resource as waste
  list_checks  List the installed checks
  plan_cleanup  Show exactly what `zombiescan clean` would do

Every tool is read-only. plan_cleanup builds the step objects and stops there; the function that sends them to Google Cloud is unreachable from the server, and a test asserts the module names no other clean.* attribute.

The tools return computed figures — totals, counts, breakdowns, cheapest and costliest — and echo the filter they applied. A tool that returned rows for the model to add up would move the arithmetic to the place least able to do it.

The protocol is JSON-RPC over stdio in the standard library, with no MCP SDK dependency for about a hundred lines of framing.

Three Findings That Differ From the AWS Instinct

Resource	Google Cloud behaviour
Idle Cloud NAT	Gateway uptime bills per VM using it, so a gateway serving nothing costs the addresses it holds — $3.65 a month each at $0.005/hour. An AWS NAT gateway bills a flat hourly charge whatever uses it.
GKE cluster	$0.10/hour whatever runs on it. Scaling every node pool to zero removes the node cost and leaves the $73.00 a month management fee.
Reserved static IP	Google bills an idle static IP at a higher hourly rate than one attached to a running instance, so the waste costs more than the work.

What the Checks Leave Alone

A false positive here costs an outage, so each check states the condition it declines to act on.

Condition	Treatment
Load balancer with backends that are unhealthy	An outage; left alone
Snapshot whose source disk still exists	Load-bearing; left alone
Disk attached to a stopped instance	Reported under `stopped-instance`, counted once
Google-managed KMS key, disabled	Free; never reported
Key or secret already scheduled for destruction	Already on a timer; left alone
Default VPC network with nothing in it	Reported, and flagged as the auto-created default
An API not enabled on a project	Skipped; a service never used holds no waste

empty-vpc-network reports without a cleanup step. A network deletes only after its subnets, routes, firewall rules, Cloud Routers and peerings are gone, in an order that depends on what references them, so the finding says so where a command would otherwise appear.

Four of the 75 projects returned permission denied: no access on every check, producing 42 skipped project/check pairs. Those appear in the report as a list, and the monthly total covers the projects that were readable.

Compare and Contrast

	zombiescan	Recommender / Active Assist	Billing reports	Cross-project SaaS
Per-resource dollar figure	🥇 yes	partial	aggregate only	yes
Credentials leave the machine	🥇 never	n/a, Google-side	n/a, Google-side	❌ service account granted
Deletes on request	🥈 opt-in, dry run first	❌ no	❌ no	🥇 yes
Price source	Billing Catalog API, dated	Google-side	your bill	vendor
Runs offline after install	🥇 yes	❌ no	❌ no	❌ no
Breadth	20 checks	broader	n/a	broader

So, Which One?

Billing reports answer what the projects spent. Recommender covers more ground and reaches areas outside these 20 checks, including the metric-driven judgements that need a lookback window.

zombiescan fits the case where the answer has to be per-resource, priced, and produced without granting anything access to the projects. A laptop, existing credentials, and about 80 seconds for 75 projects.

Cost

A scan costs nothing. List and get calls carry no charge, and the price table ships with the package, so a scan makes no Billing Catalog call.

Regenerating the table calls the Cloud Billing Catalog API, which is also free.

Tests

uv run pytest -q

324 passed, 4 deselected in 0.37s

The suite runs offline against recorded API responses, with no credentials. The four deselected tests reach a real project and run under ZOMBIESCAN_LIVE=1.

Recorded responses are keyed the way the discovery client returns them, and every aggregated one includes a scope holding only a warning, which is how Compute Engine reports an empty zone. A check that walks past that shape without skipping it fails here instead of in production.

Teardown

uv tool uninstall zombiescan-gcp

Nothing is left in any project. The tool creates no service account, no role, no bucket and no stored state. Revoke the local credentials with gcloud auth application-default revoke.

Summary

The goal of this article was to audit every Google Cloud project a set of credentials can reach and attach a monthly cost to each unused resource. The key to the solution was fanning out per project instead of per region, and fetching every rate from the Cloud Billing Catalog API while keeping the scan read-only. The results were:

🟢 20 checks in two packs across Compute Engine, Cloud SQL, Storage, DNS, KMS, Secret Manager, Artifact Registry, Filestore, Cloud Logging, Cloud Monitoring and GKE
🟢 75 projects swept in 79 seconds, 713 findings, $387.74 a month and $4,652.87 a year
🟢 One call per project: aggregatedList covers every zone and region, and locations/- covers every location for the APIs that accept it
🟢 Prices dated in every report, with the first charging tier read instead of a free tier 0
🟢 JSON against a published schema, a self-contained HTML report, and a shell cleanup plan that is printed and never run
🟢 clean previews by default, orders a backup before the destruction it protects, and marks steps with no recovery window
⚠️ 411 of 1,500 project/check pairs were skipped because the API was not enabled, and 42 more for lack of access to 4 projects, so $387.74 covers the projects that were readable
⚠️ Figures are on-demand list prices, excluding committed use discounts, sustained use discounts, private pricing and credits
⚠️ Artifact Registry is an upper bound: it bills a shared layer once, and a finding counts it once per image
❌ empty-vpc-network reports without a cleanup step, because a network needs its dependencies removed in an order that depends on what references them
❌ Idle judgements that need a lookback window — Cloud SQL by connection count, KMS keys by audit log — need Cloud Monitoring and are outside these 20 checks

Scope: one set of Application Default Credentials, 75 projects reachable through projects.search, 20 checks, a single sweep per figure quoted, run from one laptop. The read-only guarantee holds for the packs in this repository, which are reviewed; a pack installed from PyPI runs with the same credentials and carries no such guarantee.

The strategy for using per-resource pricing for Google Cloud waste detection was validated with an incremental step by step approach.

References

Repository, MIT: https://github.com/xbill9/zombiescan-gcp
Findings JSON Schema: https://github.com/xbill9/zombiescan-gcp/blob/main/docs/findings.schema.json
Pack author guide: https://github.com/xbill9/zombiescan-gcp/blob/main/docs/PACKS.md
Cloud Billing Catalog API, services.skus.list: https://cloud.google.com/billing/docs/reference/rest/v1/services.skus/list
Compute Engine aggregatedList: https://cloud.google.com/compute/docs/reference/rest/v1/disks/aggregatedList
VPC pricing, external IP addresses: https://cloud.google.com/vpc/network-pricing
GKE cluster management fee: https://cloud.google.com/kubernetes-engine/pricing
Application Default Credentials: https://cloud.google.com/docs/authentication/application-default-credentials

Share State Across Dart Isolates Without Losing Your Mind: Enter shared_map

Randal L. Schwartz — Sun, 20 Sep 2026 20:43:19 +0000

This is Part 3 of the Dart and Flutter series—practical guides, architectural deep dives, and hard-earned engineering lessons from the field. Each article is completely standalone.

Dart’s concurrency model is built on Isolates. Unlike threads in Java, C++, or Go, Dart isolates share no memory. Each isolate has its own private heap and its own single-threaded event loop.

This "share-nothing" model is a brilliant design decision. It completely eliminates data races, deadlocks, mutex contention, and tricky thread-synchronization bugs.

Until, of course, you actually need to share data across isolates.

Imagine this common production scenario:

You’re building a Flutter app that crunches heavy data in the background—perhaps resizing multiple images, decoding massive JSON payloads, computing cryptographic hashes, or running complex ML calculations. To keep your UI silky smooth at 120 FPS, you offload the work to background isolates using Isolate.run.

Now suppose all these concurrent background workers need access to a shared, in-memory cache (like parsed metadata, authentication tokens, or shared computation results) to avoid duplicate work.

How do you do that in Dart?

Traditionally, you only had two bad choices:

Serialize and copy the whole data structure back and forth across isolate boundaries every time. For large maps or high-frequency operations, this burns CPU and produces massive GC pressure.
Hand-roll a message-passing server using ReceivePort and SendPort. You have to invent custom request/response DTOs, generate unique request correlation IDs, wire up response completers, and write 150 lines of brittle plumbing just to perform a simple key-value lookup.

There is a third, vastly superior option that almost nobody talks about: package:shared_map.

What is `shared_map`?

Created by veteran Dart engineer Graciliano M. Passos, shared_map provides a versatile, synchronized Map data structure designed specifically to be shared across Dart isolates and asynchronous workflows.

Here is what makes it an architectural gem:

Zero Dependencies: It is pure, clean Dart with zero third-party dependencies.
160 / 160 Pub Points: Flawless quality score on pub.dev and fully Dart 3 compatible.
Universal Platform Support: Runs anywhere Dart runs—iOS, Android, macOS, Windows, Linux, Web, and backend CLI/servers.
Familiar Map Semantics: You interact with it using standard async key-value methods like get(), put(), putIfAbsent(), and update().

Instead of you manually orchestrating ports, shared_map manages the cross-isolate communication protocol transparently under the hood.

How It Works: The Reference Pattern

The core mental model of shared_map is dead simple:

Main Instance: You create a SharedMap on your primary isolate (like your Flutter UI thread or main server loop). This instance acts as the authoritative source of truth.
Shared Reference: You call .sharedReference() to generate a lightweight, serializable token.
Auxiliary Instance: You pass that lightweight token across an isolate boundary (e.g. into Isolate.run). Inside the isolate, you reconstruct a proxy instance using SharedMap.fromSharedReference(ref).

Any reads, writes, or mutations performed by the worker isolate are automatically dispatched back to the main instance and synchronized across all isolates!

See It In Action

Let’s write a complete, self-contained example. We'll simulate multiple concurrent worker isolates crunching data, reading from a shared cache, and populating cache entries on the fly:

import 'dart:isolate';
import 'package:shared_map/shared_map.dart';

void main() async {
  // 1. Create a SharedStore and a SharedMap on the main isolate
  final store = SharedStore('app_cache');
  final userCache = await store.getSharedMap<String, String>('users');

  // Seed an initial value
  await userCache!.put('user_101', 'Randal (Admin)');

  // 2. Extract the lightweight, serializable reference
  final cacheReference = userCache.sharedReference();

  print('--- Spawning Background Worker 1 ---');

  // 3. Pass the reference into a background isolate
  final worker1Result = await Isolate.run(() async {
    // Reconstitute the synchronized map proxy
    final workerMap = SharedMap<String, String>.fromSharedReference(cacheReference);

    // Read the value previously stored by the main isolate:
    final user = await workerMap.get('user_101');
    print('[Worker 1] Read from shared cache: $user');

    // Put a new value into the shared cache from this background worker:
    await workerMap.put('user_102', 'Wilhelm (Engineer)');
    return 'Worker 1 finished';
  });

  print(worker1Result);

  print('--- Spawning Background Worker 2 ---');

  // 4. Spawn a second isolate to prove cross-isolate synchronization
  final worker2Result = await Isolate.run(() async {
    final workerMap = SharedMap<String, String>.fromSharedReference(cacheReference);

    // Worker 2 can immediately read what Worker 1 just wrote!
    final user102 = await workerMap.get('user_102');
    print('[Worker 2] Read value written by Worker 1: $user102');

    // Use putIfAbsent atomically
    final user103 = await workerMap.putIfAbsent('user_103', 'Guest User');
    return '[Worker 2] Added: $user103';
  });

  print(worker2Result);

  // 5. Verify the main isolate reflects all updates
  print('--- Back on Main Isolate ---');
  print('Total entries in cache: ${await userCache.length()}');
  print('user_102 on main: ${await userCache.get('user_102')}');
  print('user_103 on main: ${await userCache.get('user_103')}');
}

The Console Output

--- Spawning Background Worker 1 ---
[Worker 1] Read from shared cache: Randal (Admin)
Worker 1 finished
--- Spawning Background Worker 2 ---
[Worker 2] Read value written by Worker 1: Wilhelm (Engineer)
[Worker 2] Added: Guest User
--- Back on Main Isolate ---
Total entries in cache: 3
user_102 on main: Wilhelm (Engineer)
user_103 on main: Guest User

Notice what just happened:

Two independent background isolates communicated and shared data back to the main thread.
Not a single SendPort, ReceivePort, Completer, or serialization boilerplate line was written.

Superpower: Local Read Caching with `SharedMapCached`

If your worker isolates perform thousands of rapid reads, you might not want every single get() call to perform a cross-isolate message dispatch.

shared_map includes a built-in subclass called SharedMapCached:

final cachedWorkerMap = SharedMapCached<String, String>.fromSharedReference(
  cacheReference,
  // Cache items locally in this isolate for high-throughput reads
  timeout: const Duration(seconds: 30),
);

When you query an existing key with SharedMapCached, it caches the value locally in the worker's isolate heap. If subsequent reads occur within the timeout window, they resolve instantly without cross-isolate latency.

Grouping Maps with `SharedStore`

In complex applications, you rarely have just one cache. You might have:

An image cache (SharedMap<String, Uint8List>)
A user profile cache (SharedMap<String, UserProfile>)
A rate limiter map (SharedMap<String, int>)

Instead of passing dozens of individual references around, you pass a single SharedStoreReference:

// Main thread:
final store = SharedStore('global_store');
await store.getSharedMap<String, String>('tokens');
await store.getSharedMap<String, int>('rate_limits');

final storeRef = store.sharedReference();

// Inside any background isolate:
await Isolate.run(() async {
  final workerStore = SharedStore.fromSharedReference(storeRef);

  // Dynamically resolve any map registered under this store:
  final tokens = await workerStore.getSharedMap<String, String>('tokens');
  final rateLimits = await workerStore.getSharedMap<String, int>('rate_limits');

  // ...
});

Senior Engineering Wisdom: When to Use (and Not Use) `shared_map`

As with any tool, understanding the architectural sweet spot is key.

✅ When to Reach for `shared_map`:

CPU-Intensive Worker Coordination: When running pools of background isolates (Isolate.run or persistent worker isolates) that need shared lookups.
In-Memory Deduping & Caching: Preventing concurrent workers from calculating or downloading the exact same asset twice.
Cross-Isolate Metrics & Counters: Collecting stats, telemetry, or rate-limit tokens across multiple threads.

⚠️ When NOT to Use `shared_map`:

Persistent On-Disk Storage: shared_map is an in-memory data structure. If your data must survive app restarts, use SQLite, Drift, or a persistent key-value store.
Single-Isolate Applications: If all your code runs on the root UI isolate, a standard Dart Map<K, V> or reactive Signal is all you need—there's no reason to pay the asynchronous abstraction cost.

Summary

Dart's isolate architecture keeps our code safe from concurrency bugs, but you shouldn't have to write hundreds of lines of port plumbing just to share an in-memory cache across worker tasks.

By bringing in shared_map:

You keep isolates isolated and UI threads responsive.
You eliminate manual SendPort and ReceivePort spaghetti code.
You get atomic, synchronized key-value storage with zero external dependencies.

Add shared_map: ^1.1.9 to your pubspec.yaml and stop reinventing isolate messaging from scratch.

What's your take?

How do you currently coordinate data between background isolates in your Flutter apps? Have you been writing custom ports, or relying on Isolate.run return values? Let me know in the comments below!

Randal L. Schwartz is a Google Developer Expert (GDE) for Dart & Flutter and veteran software architect.

📺 Watch deep dives and walkthroughs on YouTube: @RandalOnDartAndFlutter
💻 Connect on GitHub: @RandalSchwartz

One Iceberg MCP Server, Seven Catalogs: What It Takes to Reach Each One

xbill — Sun, 20 Sep 2026 20:30:58 +0000

This article provides a step by step guide to one MCP server for Apache Iceberg tables, pointed at seven Iceberg REST catalogs in turn. The server offers four read-only tools, and one environment variable decides which catalog it reads.

https://github.com/xbill9/lakehouse-iceberg-2026

What is this project trying to Do?

An MCP server gives any MCP client, such as Claude Code or another coding agent, a fixed set of tools. This one has four, all reads:

iceberg_list_tables — every namespace.table in the catalog
iceberg_describe_table — columns, partitioning, snapshots and the metadata location
iceberg_count_rows — the exact row count from the snapshot summary
iceberg_scan_table — a few rows, plus an exact count, min and max

Every Iceberg REST catalog speaks the same protocol, so in principle the same server should work against all of them. This project checks that against seven catalogs:

Apache Polaris 1.7.0, in Docker on the same machine
Google BigLake, Microsoft OneLake, AWS Glue, AWS S3 Tables and Snowflake Horizon, over the internet
Databricks Unity, not run: its trial account has ended

The server is called directly over MCP, with no AI model involved, so each result depends only on the catalog and the server's settings. Measured on 2026-09-18 and 2026-09-19 (UTC), one run per catalog, reads only.

Where do I start?

The strategy is an incremental step by step approach.

First, the local Polaris catalog, which needs only Docker. Then the MCP server, called directly to check all four tools. Then one catalog at a time: its login, its storage package, and a run of the same four calls.

At This Point You Should Have…

Docker, for the local Polaris catalog
Python 3.10+ with pyiceberg 0.12.0 — this run used Python 3.14.7
For each managed catalog you want to reach, an account with a table in it and a working command-line login: gcloud, az, aws or a Snowflake key pair

Step 1 — Start Polaris

$ git clone https://github.com/xbill9/lakehouse-iceberg-2026
$ cd lakehouse-iceberg-2026/iceberg-conformance
$ ./polaris-up.sh
$ export POLARIS_CLIENT_ID=root POLARIS_CLIENT_SECRET=s3cr3t
$ python3 seed_table.py --catalog apache-polaris

That creates probe_ns.probe_table: 11 rows, partitioned by day, with four snapshots.

Step 2 — Look at the Server

The server is iceberg-mcp-hosts/servers/iceberg_mcp.py. It speaks MCP over stdio as newline-delimited JSON-RPC and answers initialize, tools/list, tools/call and ping, with no MCP SDK.

It picks its catalog from two environment variables:

$ ICEBERG_CATALOG=aws-glue ICEBERG_CATALOGS_FILE=catalogs.yaml python3 servers/iceberg_mcp.py

catalogs.yaml holds one entry per catalog: the URL, the warehouse, and how to log in. The server code is the same for every catalog.

Step 3 — Call It Without a Model

sweep_catalogs.py starts the server once per catalog and makes the same MCP calls each time: initialize, tools/list, then the four tools on the first table listed.

$ cd ../iceberg-mcp-hosts
$ python3 sweep_catalogs.py --only apache-polaris
apache-polaris       probe_ns.probe_table                 list_tables=ok(1.0s)  describe_table=ok(0.4s)  count_rows=ok(0.0s)  scan_table=ok(0.1s)

The scan's reply shows what a client receives:

id | ts | payload | region
0 | 2026-09-01 00:00:00+00:00 | row-0-0 | None
2 | 2026-09-01 02:00:00+00:00 | row-0-2 | None
3 | 2026-09-01 03:00:00+00:00 | row-0-3 | None

3 of 11 row(s) shown, read from snapshot-id 1196292829914853564
COUNT: exactly 11 row(s) are in the table in snapshot-id 1196292829914853564. Exact, over the whole table.
MIN and MAX of id over those 11 row(s): 0 and 23. Exact.

A failed catalog call comes back as text starting CATALOG ERROR, so an agent can still say what it could not read. The sweep counts that text as a failure.

Step 4 — Add One Entry per Catalog

Each managed catalog needs its own login in catalogs.yaml. The server builds a PyIceberg RestCatalog from it:

catalog	login	how files are read
Polaris	OAuth2 client ID and secret	PyArrow, local `file:`
BigLake	`gcloud` token, `x-goog-user-project` header	PyArrow, `gs://`
OneLake	`az` token	fsspec with `adlfs`, `abfss://`
Glue	SigV4, service `glue`	PyArrow, `s3://`, local AWS login
S3 Tables	SigV4, service `s3tables`	fsspec with `s3fs`, catalog-issued credential
Horizon	Snowflake key-pair JWT	PyArrow, `s3://`, catalog-issued credential

S3 Tables issues a storage credential when asked with the X-Iceberg-Access-Delegation: vended-credentials header. Horizon returns one without being asked.

Step 5 — Install the Storage Packages

Listing, describing and counting read only catalog metadata, so they work with pyiceberg alone. Scanning reads data files, and three catalogs need packages that pyiceberg does not install by default. Without them the server answers three of the four tools and the scan fails:

CATALOG ERROR while scanning dbo.probe_table: ModuleNotFoundError: No module named 'adlfs'.

CATALOG ERROR while scanning probe_ns.probe_table: ModuleNotFoundError: No module named 's3fs'.

AWS logins made with the newer aws login command need one more package before any call works:

CATALOG ERROR while listing tables: MissingDependencyException: Missing Dependency: Using the login credential provider requires an additional dependency. You will need to pip install "botocore[crt]" before proceeding.

$ pip install adlfs s3fs "botocore[crt]"

package	needed for
`adlfs`	OneLake scans
`s3fs`	S3 Tables scans
`botocore[crt]`	Glue and S3 Tables with an `aws login` session

🔎 Tip: Use the `az` Credential Directly for OneLake

OneLake's data files need an Azure storage credential as well as the catalog token. DefaultAzureCredential is the usual choice, and it tries the Azure VM metadata service before the az login. Off Azure, that attempt waits out its retries:

     917ms No environment configuration found.
     921ms ManagedIdentityCredential will use IMDS
  553989ms DefaultAzureCredential acquired a token from AzureCliCredential
AzureCliCredential      0.6s
DefaultAzureCredential  553.1s

Through the MCP server, a three-row OneLake scan took 858.5 seconds with DefaultAzureCredential and 3.4 seconds with AzureCliCredential. Every other tool call was unaffected, because only the scan reads files.

Step 6 — Run All Six

$ python3 sweep_catalogs.py --only apache-polaris,google-lakehouse,microsoft-onelake,aws-glue,aws-s3tables,snowflake-horizon
$ python3 sweep_catalogs.py --report-only

catalog             measured (UTC)        table                   rows  partitioned     seconds: list describe count scan
apache-polaris      2026-09-18T23:58:37Z  probe_ns.probe_table      11  ts_day          1.0 0.5 0.0 0.1
aws-glue            2026-09-18T23:58:51Z  probe_ns.probe_table      11  ts_day          2.1 0.7 0.2 1.5
aws-s3tables        2026-09-18T23:58:57Z  probe_ns.probe_table      11  ts_day          2.2 0.3 0.2 2.0
google-lakehouse    2026-09-19T00:11:14Z  probe_ns.probe_table      11  ts_day          10.1 1.6 0.5 3.6
microsoft-onelake   2026-09-18T23:58:45Z  dbo.probe_table            6  (unpartitioned) 2.5 0.2 0.2 3.2
snowflake-horizon   2026-09-18T23:59:02Z  PROBE_NS.PROBE_TABLE      12  ts_day          7.3 2.2 1.4 1.9

All four tools work on all six catalogs. Each one reported the same four tools from tools/list, and no call returned an error.

The first call on each catalog also builds the client and fetches a login token, which is why list is the slowest column. Each figure is a single call, so the times show scale only.

Step 7 — Read What Came Back

The tables were created separately for an earlier article, and the server reports what each catalog holds:

OneLake names its namespace dbo, stores id as an optional int, and has 6 rows in an unpartitioned table
Horizon returns upper-case names, PROBE_NS.PROBE_TABLE, and has 12 rows
The other four hold the same 11-row table, partitioned by day

A client only has to use the names iceberg_list_tables returns. The server passes them back unchanged, so dbo and upper-case names need no special handling.

🔎 Tip: Horizon Hands Out Storage Credentials Unasked

Loading a Horizon table leaves S3 credentials on the client's file reader, although the catalog entry asks for none:

snowflake-horizon
  client.region
  py-io-impl
  s3.access-key-id
  s3.secret-access-key
  s3.session-token
  s3.session-token-expires-at-ms

Horizon's catalog entry supplies no storage login, so its scan reads files with that credential. It also means a Horizon table load carries a live storage credential, so logs of that response need redacting.

Step 8 — Use It From an MCP Client

Any MCP client that starts stdio servers can run it. For Claude Code, a project .mcp.json:

{
  "mcpServers": {
    "iceberg": {
      "command": "python3",
      "args": ["iceberg-mcp-hosts/servers/iceberg_mcp.py"],
      "env": {
        "ICEBERG_CATALOG": "aws-glue",
        "ICEBERG_CATALOGS_FILE": "iceberg-conformance/catalogs.yaml"
      }
    }
  }
}

To reach a different catalog, change ICEBERG_CATALOG. The measurements above call the server directly and do not use this file.

Compare and Contrast

catalog	four tools	extra package	login on the machine
Apache Polaris	🟢	—	OAuth2 client secret
Google BigLake	🟢	—	`gcloud`
Microsoft OneLake	🟢	`adlfs`	`az`
AWS Glue	🟢	`botocore[crt]` with `aws login`	`aws`
AWS S3 Tables	🟢	`s3fs`, `botocore[crt]` with `aws login`	`aws`
Snowflake Horizon	🟢	—	key pair
Databricks Unity	not run

Summary

The goal of this article was to point one Iceberg MCP server at seven catalogs and record what each one takes. The key to the solution was keeping the server code fixed, changing only the catalog entry, and calling the tools directly so that no model sits between the catalog and the result. The results were:

🟢 All four tools work on all six catalogs run: Polaris, BigLake, OneLake, Glue, S3 Tables and Horizon
🟢 The only change between catalogs is one environment variable and one entry in catalogs.yaml
⚠️ Three packages outside pyiceberg's defaults are needed: adlfs for OneLake, s3fs for S3 Tables, and botocore[crt] for an aws login session; without them the metadata tools work and the scan fails
⚠️ DefaultAzureCredential took 553.1 seconds off Azure; AzureCliCredential took 0.6
⚠️ Horizon returns S3 credentials with a table load, unasked
❌ Databricks Unity was not run, because its trial account has ended

Scope: iceberg_mcp.py 1.0.0 on pyiceberg 0.12.0, pyarrow 25.0.1, s3fs 2026.9.0, adlfs 2026.8.0, botocore 1.43.75 and azure-identity 1.25.3, on Python 3.14.7. Apache Polaris 1.7.0 in Docker with local file storage; the managed catalogs over the internet from one machine, AWS in us-east-1. One run per catalog on 2026-09-18 and 2026-09-19 (UTC); Unity not run. Reads only. The tools are checked for answering without error; the tables differ between catalogs, so values are reported and not compared. Managed catalogs do not report a version.

The strategy for pointing one Iceberg MCP server at seven catalogs was validated with an incremental step by step approach.

References

Dart Enhanced Enums Are Secretly Factories: Unlocking Constructor Tearoffs

Randal L. Schwartz — Sun, 20 Sep 2026 20:03:10 +0000

This is Part 2 of the Dart and Flutter series—practical guides, architectural deep dives, and hard-earned engineering lessons from the field. Each article is completely standalone.

How many times have you written (or reviewed) a piece of code that looks like this?

NotificationWidget buildNotification(NotificationType type, NotificationData data) {
  switch (type) {
    case NotificationType.email:
      return EmailNotificationWidget(data);
    case NotificationType.sms:
      return SmsNotificationWidget(data);
    case NotificationType.push:
      return PushNotificationWidget(data);
  }
}

Or worse, a dedicated NotificationWidgetFactory class containing a 40-line switch statement or a mutable Map<NotificationType, Function> registry.

It feels routine. It’s what we were taught in classic OOP textbooks. But it introduces subtle friction:

The enum NotificationType knows nothing about the widgets it represents.
The factory switch ladder must be updated every time a new case is added.
The creation logic is split across multiple files and layers.

What if your enum wasn't just a list of identifiers, but was itself the polymorphic factory?

By marrying two features of modern Dart—Enhanced Enums and Constructor Tearoffs—you can delete the switch ladders and turn your enum values into self-instantiating factories in under 15 lines of code.

The Two Ingredients: A Brief History

To understand how clean this pattern is, we have to appreciate two language features that quietly revolutionized Dart over the last couple of years:

1. Constructor Tearoffs (Dart 2.15+)

Before Dart 2.15, if you wanted to pass a constructor as a first-class function, you had to wrap it in an awkward lambda:

// The old, clunky way:
final builders = [(data) => EmailNotification(data)];

Dart 2.15 introduced Constructor Tearoffs. Constructors became first-class closures. You can reference default constructors using .new, or named constructors directly by name:

// The modern Dart way:
final builders = [EmailNotification.new];
final parsers = [User.fromJson];

2. Enhanced Enums (Dart 2.17+)

Before Dart 2.17, Dart enums were glorified integers. They had an index and a name, and virtually nothing else.

With Enhanced Enums, enums gained full class powers:

They can declare final fields.
They can have const constructors.
They can implement interfaces and mixins.
They can define methods, getters, and operator overloads.

When you put constructor tearoffs inside enhanced enums, something magical happens.

The Fusion: Enums as Self-Instantiating Factories

Let’s model a common domain scenario: a document rendering engine. We have different document types that share a common interface:

abstract class Document {
  String get title;
  void render();
}

class PdfDocument implements Document {
  @override
  final String title;
  PdfDocument(this.title);

  @override
  void render() => print('Rendering PDF: $title');
}

class MarkdownDocument implements Document {
  @override
  final String title;
  MarkdownDocument(this.title);

  @override
  void render() => print('Rendering Markdown: $title');
}

class HtmlDocument implements Document {
  @override
  final String title;
  HtmlDocument(this.title);

  @override
  void render() => print('Rendering HTML: $title');
}

Now, instead of writing an external DocumentFactory or a switch statement, we declare an Enhanced Enum where each enum member holds a tearoff reference to its class constructor:

enum DocumentType {
  pdf(PdfDocument.new),
  markdown(MarkdownDocument.new),
  html(HtmlDocument.new);

  // A field holding a function that creates a Document given a String title
  final Document Function(String title) create;

  const DocumentType(this.create);
}

Look closely at DocumentType:

Document Function(String title) create: A strongly typed function signature stored as a final field.
pdf(PdfDocument.new): We pass the constructor tearoff directly to the enum value.
const DocumentType(this.create): The constructor is const, so the entire enum remains compile-time constant!

How You Use It

Instantiating a polymorphic object is now as simple as calling the field on the enum instance:

void main() {
  const selectedType = DocumentType.markdown;

  // Polymorphic instantiation with ZERO switch statements:
  final doc = selectedType.create('Architecture_Notes.md');

  doc.render(); // Output: Rendering Markdown: Architecture_Notes.md
}

No switch statement. No map lookups. No reflection. If you add a new enum value (e.g. epub), the compiler forces you to supply a matching constructor tearoff right there. You cannot accidentally forget to handle it.

Real-World Example: Polymorphic API Payload Parsers

This pattern shines when deserializing polymorphic JSON payloads (like webhooks, analytics events, or push notifications).

Imagine an incoming stream of server events:

{
  "type": "login",
  "payload": {"userId": "usr_42", "timestamp": 1711000000}
}

We have distinct payload models:

abstract class EventPayload {}

class LoginPayload implements EventPayload {
  final String userId;
  LoginPayload.fromJson(Map<String, dynamic> json) : userId = json['userId'] as String;
}

class PurchasePayload implements EventPayload {
  final double amount;
  PurchasePayload.fromJson(Map<String, dynamic> json) : amount = (json['amount'] as num).toDouble();
}

Instead of a bulky JSON parsing switch, our enum maps incoming strings directly to the named constructor tearoff (.fromJson):

enum EventType {
  login(LoginPayload.fromJson),
  purchase(PurchasePayload.fromJson);

  final EventPayload Function(Map<String, dynamic>) fromJson;
  const EventType(this.fromJson);

  static EventType? fromString(String name) =>
      EventType.values.where((e) => e.name == name).firstOrNull;
}

Now, your dispatcher parses any incoming event in two clean lines:

EventPayload parseEvent(String typeName, Map<String, dynamic> rawPayload) {
  final eventType = EventType.fromString(typeName) ?? 
      (throw UnsupportedError('Unknown event: $typeName'));

  return eventType.fromJson(rawPayload);
}

Real-World Example: Flutter Widget Builders

In Flutter applications, you frequently have a selection control (tabs, filters, or segmented buttons) that drives which widget to render:

enum DashboardView {
  analytics(AnalyticsView.new),
  activity(ActivityView.new),
  settings(SettingsView.new);

  final Widget Function({Key? key}) builder;
  const DashboardView(this.builder);
}

In your widget tree:

class DashboardScreen extends StatelessWidget {
  final DashboardView currentView;
  const DashboardScreen({super.key, required this.currentView});

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      body: currentView.builder(),
    );
  }
}

When you add a 4th tab tomorrow, you don't hunt through widget tree switch statements—you simply declare it on the enum.

A Senior Perspective: When NOT to Use This Pattern

Every pattern has architectural boundaries. While enum constructor tearoffs are powerful, here are two caveats to keep in mind:

1. Beware of Layer Inversion (Separation of Concerns)

If your enum lives in your pure Dart Domain Layer (core business logic), do not attach Flutter widget constructor tearoffs to it. Doing so couples your domain models to package:flutter.

Good: An enum in the presentation layer mapping UI modes to widget constructors.
Good: An enum in the data layer mapping API event types to DTO constructors.
Bad: A domain entity enum importing flutter/material.dart.

2. When to Prefer Dart 3 Sealed Classes & Pattern Matching

Dart 3 introduced sealed class hierarchies and exhaustive switch expressions:

// Alternative: Dart 3 pattern matching
Widget buildView(DashboardView view) => switch (view) {
  DashboardView.analytics => const AnalyticsView(),
  DashboardView.activity => const ActivityView(),
  DashboardView.settings => const SettingsView(),
};

Which should you choose?

Use Enum Constructor Tearoffs when the creation parameters are identical, the association between the enum and the class is 1:1, and you want self-contained encapsulation with zero boilerplate.
Use Sealed Classes & Pattern Matching when each subclass takes radically different parameters, when cases have unique construction logic, or when you want to avoid coupling the enum to the concrete implementations.

Conclusion

Enhanced Enums and Constructor Tearoffs are two of modern Dart's finest language ergonomics. When combined, they eliminate entire classes of boilerplate:

Self-documenting: The enum member explicitly declares the constructor that builds it.
Compile-time safe: Missing a constructor is impossible; the compiler will not let you compile an enum member without satisfying the signature.
Zero switch statements: Replaces sprawling factory classes with a clean, single-line invocation.

Next time you catch yourself writing a 30-line switch statement just to instantiate a class from an enum value, pause. Let the enum do the work.

What's your take?

Have you started using constructor tearoffs in your enums, or do you prefer Dart 3 switch expressions? Let me know in the comments below!

Randal L. Schwartz is a Google Developer Expert (GDE) for Dart & Flutter and veteran software architect.

📺 Watch deep dives and walkthroughs on YouTube: @RandalOnDartAndFlutter
💻 Connect on GitHub: @RandalSchwartz

Stop Paying the build_runner Tax: Why I Refuse to Use Mockito in Modern Dart

Randal L. Schwartz — Sun, 20 Sep 2026 19:48:39 +0000

This is Part 1 of the Dart and Flutter series—practical guides, architectural deep dives, and hard-earned engineering lessons from the field. Each article is completely standalone.

Picture this familiar scene:

You’re deep in the zone. You’re practicing disciplined Test-Driven Development (TDD). You write a failing test, write the minimal code to satisfy it, and refactor. Red, green, refactor. Your fingers are flying.

Then, you add a single new method to your UserRepository interface:

Future<void> updatePreferences(UserPreferences prefs);

You switch back to your test file to stub it, and everything grinds to a dead stop. Your IDE lights up with red squiggly lines because the generated mock doesn't know about updatePreferences yet.

You sigh, drop into the terminal, and type the dreaded incantation:

dart run build_runner build --delete-conflicting-outputs

Now you wait. 15 seconds. 30 seconds. On a large enterprise Flutter codebase, maybe two full minutes. The fan spins up. Your train of thought derails. By the time the builder finishes generating boilerplate Dart files, you've lost your flow state.

Why on earth are we running heavyweight compile-time code generation just to create a mock object for a unit test?

It’s time to stop using mockito in modern Dart. There is a much better way.

How Did We End Up Here?

To be fair to the mockito maintainers, mockito didn’t set out to be slow.

In the pre-null-safety era of Dart, mockito was the undisputed king. You created a mock class with class MockRepo extends Mock implements Repo {}, and Dart’s runtime dynamic invocation (noSuchMethod) handled everything under the hood with zero code generation. It was snappy and painless.

Then came Sound Null Safety in Dart 2.12.

Null safety is one of the best things that ever happened to Dart, but it broke mockito's runtime magic. In a soundly typed null-safe language, if an interface method declares that it returns a non-nullable User, noSuchMethod cannot simply return null while waiting for your when(...) stub to register. Returning null violates the type system and throws an immediate runtime error before your stubbing logic even runs.

Because Dart has deliberately avoided heavy runtime reflection (dart:mirrors is disabled in Flutter for performance and tree-shaking reasons), mockito chose the only route available to it at the time: compile-time code generation via build_runner.

It solved the type-safety problem, but it introduced the dreaded build_runner tax.

The Three Sins of Modern Mockito

Relying on mockito today saddles your codebase with three distinct engineering bottlenecks:

1. The Iteration Tax (Destroying the TDD Flow)

TDD relies on tight feedback loops measured in milliseconds. When every interface change or mock signature update requires running code generation, the feedback loop stretches into tens of seconds. Developers stop running tests frequently. They batch their changes, commit without verifying, and run tests as an afterthought. Velocity plummets.

2. Git Diff Pollution & Merge Conflict Hell

For every test file with mocks, mockito generates a .mocks.dart sibling file. These files are thousands of lines of machine-generated boilerplate.

They pollute pull request reviews.
They clutter repository search results.
When two developers touch overlapping service interfaces on separate branches, merging their generated .mocks.dart files triggers brutal, unreadable merge conflicts.

3. Brittle Annotation Coupling

With mockito, your tests are coupled to global library annotations:

// test/user_service_test.dart
import 'user_service_test.mocks.dart';

@GenerateMocks([HttpClient, UserRepository])
void main() {
  // ...
}

Want to mock a third dependency? You have to edit the annotation list, rerun build_runner, wait for the file system to update, and import the generated symbols. It's clunky and mechanical.

Enter Mocktail: Zero-Config Runtime Mocking

Enter mocktail, created by Felix Angelov (of Bloc and Very Good Ventures fame).

mocktail delivers the exact same intuitive API you know from mockito, but with zero code generation. No build_runner. No .mocks.dart files. No annotations.

How? By leveraging two native features of Dart:

Dart’s implicit interfaces: In Dart, every class implicitly defines an interface. You can implement any class without inheriting its implementation.
Closures for stubbing: Instead of evaluating the mock invocation directly (which triggers the null-safety problem), mocktail wraps the invocation inside a closure: () => mock.getUser(). This defers execution until mocktail can safely intercept and return the registered stub or a fallback value.

Here is how you declare a mock in mocktail:

import 'package:mocktail/mocktail.dart';

// No annotations. No build_runner. Just pure, clean Dart.
class MockUserRepository extends Mock implements UserRepository {}
class MockHttpClient extends Mock implements HttpClient {}

That’s it. You define the class right in your test file (or in a shared test_helpers.dart), hit save, and run your tests instantly.

Side-by-Side: Mockito vs. Mocktail

The migration from mockito to mocktail requires practically zero mental re-learning:

Feature	`mockito`	`mocktail` 🍹
Code Generation	Requires `build_runner`	None (Pure runtime)
Declaration	`@GenerateMocks([UserRepo])`	`class MockUserRepo extends Mock implements UserRepo {}`
Synchronous Stub	`when(repo.name).thenReturn('Randal')`	`when(() => repo.name).thenReturn('Randal')`
Asynchronous Stub	`when(repo.fetch()).thenAnswer((_) async => user)`	`when(() => repo.fetch()).thenAnswer((_) async => user)`
Argument Matchers	`anyNamed('id')`, `anyString`	`any(named: 'id')`, `any()`
Verification	`verify(repo.login()).called(1)`	`verify(() => repo.login()).called(1)`

Notice the only primary syntax difference: mocktail passes the invocation as an anonymous function () => mock.method().

That simple closure is the secret sauce that bypasses the need for compile-time code generation.

The One "Gotcha": Custom Type Fallbacks

To be an objective engineer, you must evaluate the trade-offs. mocktail has exactly one gotcha, and once you understand it, it takes 5 seconds to address.

The Problem

When you use flexible argument matchers like any() on a method that accepts a non-nullable custom type:

when(() => mockRepo.saveUser(any())).thenAnswer((_) async => true);

Dart’s sound type system insists that something matching the type User must be passed into saveUser during internal matcher registration. Under the hood, mocktail cannot synthesize an instance of your custom User class out of thin air. If it passed null, Dart would throw a type error.

If you don't tell mocktail how to satisfy that type, it will throw an informative exception:

Bad state: A test tried to use any() or captureAny() on a User which was not registered.
Register a fallback value using `registerFallbackValue`.

The Solution

You define a dummy Fake and register it in setUpAll:

import 'package:flutter_test/flutter_test.dart';
import 'package:mocktail/mocktail.dart';

// 1. Create a Fake implementation that satisfies the type checker
class FakeUser extends Fake implements User {}

void main() {
  setUpAll(() {
    // 2. Register it once before any tests run
    registerFallbackValue(FakeUser());
  });

  test('saves user profile', () async {
    final repo = MockUserRepository();
    when(() => repo.saveUser(any())).thenAnswer((_) async => true);

    final service = UserService(repo);
    final result = await service.updateProfile(User(id: '123', name: 'Randal'));

    expect(result, isTrue);
    verify(() => repo.saveUser(any())).called(1);
  });
}

That’s all there is to it. A two-line Fake registration in setUpAll completely eliminates the need to run build_runner on your entire project.

How to Migrate Without Boiling the Ocean

If you’re staring at an existing codebase with dozens of build_runner-powered .mocks.dart files, you don't need to rewrite everything in a single frantic weekend PR.

Take an incremental approach:

Add mocktail to dev_dependencies:

   dev_dependencies:
     mocktail: ^1.0.4

Migrate one test file at a time:
- Whenever you open an existing test file to add or modify a test, migrate just that file.
- Delete the .mocks.dart import.
- Replace @GenerateMocks with simple class MockX extends Mock implements X {} declarations.
- Wrap your when and verify invocations in () => ....
- Delete the obsolete .mocks.dart file from disk.
Commit the diff: Notice how clean and readable your git diff is without hundreds of lines of autogenerated noise.

The Verdict

The official Flutter documentation and cookbooks still mention mockito largely due to historical inertia. But we no longer live in 2020.

In modern Dart, testing should be fast, expressive, and friction-free. Waiting on build_runner just to verify that a service method was called with the right argument is a waste of your machine's CPU cycles and your mental bandwidth.

Switch to mocktail. Your TDD cycle will thank you.

What's your take?

Are you still running build_runner for your mocks, or have you already made the switch to mocktail (or hand-rolled fakes)? Let me know in the comments below!

Randal L. Schwartz is a Google Developer Expert (GDE) for Dart & Flutter and veteran software architect.

📺 Watch deep dives and walkthroughs on YouTube: @RandalOnDartAndFlutter
💻 Connect on GitHub: @RandalSchwartz