DEV Community: xbill

Count It or Compute It: When a Tool Returns Rows, the Models That Count Them Right Spend the Tokens

xbill — Mon, 28 Sep 2026 19:04:50 +0000

This is a submission for the Kaggle Benchmarking Challenge

This article reports a Kaggle benchmark for a step every agent performs and few people test, counting what a tool returns, and ends with a step by step guide to reproducing it.

An agent's search, database or API tool usually hands back a list of records. When the user asks how many, the model does the counting. That step passes every quick test with a handful of rows, and nothing flags it when it goes wrong: the model sends the right query and quotes a confident number.

This benchmark asks ten models the same 68 counting questions with two versions of one tool: count_ids returns the exact count, list_ids returns the matching ids.

With the count, every model but one answered every question about 330 ids correctly. With the ids, the models split in two. The ones that spent 2,700 to 8,200 output tokens per question counted 15 to 21 of 21 lists correctly. The ones that answered in under 600 tokens counted 0 to 10.

The surprise was who landed where: Claude Opus 5, the most expensive model in the lineup, answered in 579 tokens and counted 9 of 21. Gemma 4 26B, a 26B open-weight model, spent 8,153 and counted 20.

https://github.com/xbill9/devto-kaggle

https://www.kaggle.com/benchmarks/xbillwork/count-it-or-compute-it

What I Benchmarked

The itch is eleven ids:

0, 2, 3, 20, 21, 22, 23, 10, 11, 12, 13

How many are 10 or more? The answer is 8, and every model here gets it. Eleven ids say nothing about three hundred, and eleven is about the size of the lists most quick tests use.

Language models' trouble with counting is familiar from examples like counting the letters in a word. This benchmark measures it where agents meet it: inside tool use, with every miss traced to the query or the count, and every answer priced in tokens.

The benchmark holds everything fixed except who does the arithmetic:

Task	What the model gets	Who counts
`count-engine`	`count_ids(where)`, which returns the exact count, minimum and maximum	The tool
`count-rows-tool`	`list_ids(where)`, which returns the matching ids	The model, from the returned list
`count-python-tool`	Every id in the prompt, plus `run_python` with `ids` already defined	The model's code, if it writes any

In the first two tasks the ids never appear in the prompt, so the only difference is whether the tool returns a number or a list. Both tasks check every filter the model sends by the ids it selects, so id > 9 counts as right for "10 or more", and every wrong answer is traced to either a wrong query or a wrong count.

Each task asks 68 questions: lists of 11, 110 and 330 ids, seven phrasings of the threshold ("10 or more", "no less than", "under", "between 5 and 9 inclusive" and so on), three seeds each, and the original eleven ids five times. Every threshold is an id in the list, so > and >= always give different answers.

Models Tested

Vendor	Models
Google	Gemini 2.5 Flash, Gemini 3.7 Flash, Gemini 3.8 Flash, Gemma 4 26B A4B
Anthropic	Claude Haiku 4.5, Claude Sonnet 5, Claude Opus 5
OpenAI	GPT-5.4 nano, GPT-5.4 mini, gpt-oss-20b

The lineup takes a small, a mid-sized and a large model from each vendor, plus the open-weight models from Google and OpenAI, so the comparison covers price, size and what anyone can run. Each model runs with the settings Kaggle's model proxy serves it with. Some of those reason before answering and some answer straight away, and that setting tracks the result more closely than size or price.

GPT-6 Astra is refused function tools by the proxy (Function tools with reasoning_effort are not supported for gpt-6-astra in /v1/chat/completions). Gemini 3.5 Flash-Lite and both Qwen 3 Next 80B models returned 429 or 503 on most calls.

Findings

At 330 ids, the size where the models separate:

Model	Rows tool correct	Output tokens per question, rows tool	Engine correct	Rows-tool cost vs engine
Gemma 4 26B A4B	20/21	8,153	21/21	18.6x
Gemini 3.7 Flash	20/21	2,785	21/21	13.5x
Gemini 3.8 Flash	21/21	2,742	21/21	14.3x
gpt-oss-20b	15/21	2,684	19/21	5.2x
Gemini 2.5 Flash	0/21	580	21/21	2.6x
Claude Opus 5	9/21	579	21/21	2.4x
Claude Sonnet 5	10/21	401	21/21	1.9x
Claude Haiku 4.5	5/21	143	21/21	1.5x
GPT-5.4 mini	5/21	39	21/21	1.9x
GPT-5.4 nano	0/21	35	21/21	1.7x

The Claude rows-tool figures come from their 2026-09-25 run, which asked the same 68 questions with the same tool; their 2026-09-28 rows-tool run stopped on the daily quota. Every other figure comes from the 2026-09-28 runs.

1. With the Count in the Tool, Every Model Is Right at a Flat Cost

Every model but one answered all 21 engine questions about 330 ids correctly, at 35 to 451 output tokens per question, and each model's tokens stayed about the same from 11 ids to 330. Turning "no less than 244" into id >= 244 and quoting the number back is something every model here does reliably. The exception, gpt-oss-20b, answered 5 of 68 engine questions with a number other than the count it was given, all of them 0, 1 or 2, and answered once without calling the tool.

2. With the Rows, Quick Tests Pass and Larger Lists Fail

Eight of the ten models counted all 26 questions about 11 ids correctly. At 330 ids, five of those eight counted 10 or fewer of 21 correctly. None of them returned an error or a hedge: each answer was a single confident number.

3. The Models That Count Right Spend the Tokens

The models split into two groups by how many tokens they spend, and model size and price do not predict the split. That was the surprise. The models that count correctly spend more tokens as the list grows: Gemma 4 26B went from 394 output tokens per question at 11 ids to 8,153 at 330. The models that miscount spend about the same at every size: GPT-5.4 nano spent 35 tokens per question at 11, 110 and 330 ids, which leaves no room to count anything. Claude Opus 5, the most expensive model here, spent 579 output tokens per question at 330 ids and counted 9 of 21 correctly; Gemma 4 26B, an open-weight model, spent 8,153 and counted 20.

The same models landed in the same group in every rows-tool run, three or four runs per model between 2026-09-25 and 2026-09-28. The order inside a group moves by a few questions between runs.

Counting right by reasoning costs 5 to 19 times what the engine costs for the same answer. An earlier run with 1,100 ids in the prompt shows the same link from the other side: with output capped at 8,192 tokens, Gemini 3.7 Flash counted 1 of 21 lists correctly, against 19 and 15 of 21 without the cap, and still gave a number every time.

4. The Query Was Right Every Time

No answer on either task rested on a filter that selected the wrong ids. Every miss on the rows tool was the model counting the correct list wrong, apart from 3 answers given without calling the tool and 1 that could not be read as a number. The failure is in the arithmetic.

5. Python Works When the Model Uses It

With every id in the prompt and a Python tool available, 7 of 10 models scored 68 of 68; Gemini 2.5 Flash called the tool on 1 of 42 questions about 110 and 330 ids and scored 37.

What It Changed About How I Think About These Models

In these runs, counting a list was work the model did in tokens, and the models that answered straight away had not done it. A tool that returns rows moves that work onto the model and makes its accuracy depend on a setting the caller may never have looked at. The count belongs in the tool: return the count, the minimum and the maximum, and every model in this lineup answers correctly at a fraction of the tokens.

Compare and Contrast

	Engine	Rows tool	Python tool
Who counts	The tool	The model, from the returned list	The model's code, if it writes any
Output tokens per question at 330 ids	35 to 451	35 to 8,153	—
What went wrong	Quoting a number other than the count	Miscounting the right list	Skipping the tool, no `print()`
Answers built on a wrong filter	0	0	—

So, Which One?

🟢 Engine — put the count, minimum and maximum in the tool. Every model is right at a flat cost.
⚠️ Python tool — reliable when the model uses it, prints the result and quotes it.
❌ Rows tool — correct only with a model that reasons through the list, at 5 to 19 times the cost per question.

What I'd Measure Next

The same model with reasoning on and off. Claude Opus 5 with extended thinking against its default. The runs here show that tokens and accuracy move together across models; this would show whether switching reasoning on is enough to fix one model.
Other arithmetic. Sums, averages and the largest value over returned rows, which the same tool-shape question applies to.
Harder filters. "At least 10 but under 20" and "outside 5 to 9", where a wrong filter with an exact count would show up.

How to Reproduce It

These steps rebuild the benchmark from the repository; the tasks, the runs and every table above come from these commands.

At This Point You Should Have…

A Kaggle account, with the Kaggle CLI installed (pip install kaggle, version 2.2.4 here) and logged in with kaggle auth login
Python 3 for the local checks and the report; the kaggle_benchmarks library the tasks import is already installed on Kaggle
The repository cloned: git clone https://github.com/xbill9/devto-kaggle and cd devto-kaggle

Step 1 — Check the Questions Locally

Every question is generated in code, and its expected count comes from running a filter over the ids. check.py builds all 68 questions with no model calls and no Kaggle library:

python3 tasks/check.py

ok: 68 rows per task, by size {11: 26, 110: 21, 330: 21}

Step 2 — Push Each Task

Each file in tasks/ is one Kaggle task. The first @kbench.task in the file returns the share of questions answered correctly, because Kaggle names the task and reads its score from the first one it finds. A second task answers one question and runs once per question:

@kbench.task(name="count-engine")
def count_engine(llm) -> float:
    runs = count_engine_row.evaluate(
        llm=[llm], evaluation_data=pd.DataFrame(ROWS), n_jobs=4, on_failure="continue")
    return summarize(runs, len(ROWS), "engine")

Each question's result keeps the answer, every filter the model sent, and the tokens and cost Kaggle's model proxy reported for it. The push slug must match the task name:

kaggle b t push count-engine -f tasks/count_engine.py --wait
kaggle b t status count-engine

Task:     count-engine
Version:  13
Status:   Completed
Created:  2026-09-28 16:11:45

Step 3 — Run the Models

kaggle b auth -y fetches the short-lived key for Kaggle's model proxy, and kaggle b t models lists the model slugs. A run takes one -m per model:

kaggle b t run count-rows-tool -m gemini-2.5-flash -m gemini-3.8-flash -m gemma-4-26b-a4b-it -m gpt-oss-20b -m gpt-5.4-nano-2026-03-17 -m gpt-5.4-mini-2026-03-17 --wait

All runs completed:
  gpt-5.4-mini-2026-03-17: COMPLETED
  gpt-5.4-nano-2026-03-17: COMPLETED
  gpt-oss-20b: COMPLETED
  gemma-4-26b-a4b-it: COMPLETED
  gemini-3.8-flash: COMPLETED
  gemini-2.5-flash: COMPLETED

🔎 Tip: Cap the Output to Stay Inside the Quota

Kaggle's model proxy reserves the worst-case cost of a call, based on the output-token limit, and refuses the call when that exceeds what is left of the day's quota. With no limit set, GPT-6 Astra reserved $6.40 per call and Claude Opus 5 $3.20. Passing a limit keeps the reservation to cents:

llm.prompt(prompt, schema=int, extra_api_params={"max_completion_tokens": 8192})

Step 4 — Download the Runs and Build the Tables

report.py builds every table in this article from the downloaded runs, including the tokens and cost per question:

for t in count-engine count-rows-tool count-python-tool; do kaggle b t download $t -o results; done
python3 tasks/report.py results

| Model | Size | Engine correct | Engine out tokens | Engine $ | Rows correct | Rows out tokens | Rows $ |
|---|---|---|---|---|---|---|---|
| Gemini 3.8 Flash | 330 | 21/21 | 140 | 0.0010 | 21/21 | 2,742 | 0.0146 |
| GPT-5.4 nano | 330 | 21/21 | 35 | 0.0002 | 0/21 | 35 | 0.0003 |

Step 5 — Group the Tasks Into a Benchmark

A benchmark is created in the Kaggle web UI. On the benchmark page, Add Tasks adds the tasks and Add Models the models. Under Settings, set the overall score to Average of task scores: the default, Percentage of tasks passed, ignores tasks that return a number. Setting the visibility to Public is permanent.

🔎 Tip: The Leaderboard Shows Each Model's Latest Run

Each leaderboard cell shows the model's most recent run of that task, so a run that fails on the quota replaces a complete one before it. Check the quota before a run, and rerun the pair if one fails. pending.py lists pairs whose latest run is incomplete.

My Benchmark

https://www.kaggle.com/benchmarks/xbillwork/count-it-or-compute-it (the benchmark: the three tasks and their leaderboard)
https://www.kaggle.com/benchmarks/tasks/xbillwork/count-engine
https://www.kaggle.com/benchmarks/tasks/xbillwork/count-rows-tool
https://www.kaggle.com/benchmarks/tasks/xbillwork/count-python-tool

Summary

The goal of this article was to measure whether models count correctly what their tools return. The key to the solution was changing only what the tool returns, a count or a list, and recording every filter and every token.

The results were:

🟢 With the count in the tool, every model but gpt-oss-20b answered all 330-id questions correctly.
⚠️ With the rows, the models that spent 2,700 to 8,200 output tokens per question counted 15 to 21 of 21 lists of 330 correctly.
❌ The models that answered in under 600 tokens counted 0 to 10 of 21, Claude Opus 5 among them, with no error to show it.

Each model ran each task on Kaggle's model proxy with its default settings and output capped at 8,192 tokens; the engine and rows-tool figures come from runs on 2026-09-28, apart from the Claude rows-tool figures from 2026-09-25, the Python-tool figures from 2026-09-28, and the token-cap result from earlier in-context runs with lists of 1,100 ids.

The strategy for benchmarking counting across 10 models was validated with an incremental step by step approach.

References

This benchmark's code: https://github.com/xbill9/devto-kaggle
Kaggle Benchmarking Challenge: https://dev.to/challenges/kaggle-2026-09-23
Kaggle Benchmarks: https://www.kaggle.com/benchmarks
The tasks are built on Kaggle's kaggle-benchmarks library: https://github.com/Kaggle/kaggle-benchmarks
Kaggle's benchmark-writing skill, used as the reference for the CLI workflow: https://github.com/Kaggle/kaggle-skills/blob/main/write-kaggle-benchmarks/SKILL.md

Google's QAT Gemma 4 26B-A4B on One TPU v6e: 15.6x the KV Cache and 1.9x the Throughput of FP8

xbill — Sat, 26 Sep 2026 22:35:11 +0000

This article provides a step by step guide to serving Google's quantization-aware-trained (QAT) Gemma 4 26B-A4B on one Google Cloud TPU v6e chip with vLLM, and compares it with the FP8 build that is the only 26B serving on one chip today. Every per-record output, log and script is committed.

The QAT 26B serves on one v6e chip at 17.43 GiB of HBM, with 53,888 tokens of KV cache and 1,283 output tokens per second. RedHat's FP8 build uses 27.99 GiB, holds 3,456 tokens and serves 668. Against the full-precision bf16 model, served across four chips as a reference, the QAT checkpoint reads a 3,880-record classification suite 0.4 points lower on an NVIDIA GPU and 1.1 points lower on the TPU; FP8 reads 0.4 points lower. The route is a lossless repack of Google's "unquantized" QAT export and one new method in vLLM's TPU backend; the same repacked checkpoint loads unpatched on vLLM 0.30.0 on an NVIDIA L4.

https://github.com/xbill9/gemma4-dev/tree/main/jev-tpu-31b

Why Measure This?

Gemma 4 26B-A4B is a mixture-of-experts model: 25.8B parameters, 128 experts per layer, 8 active per token. At bf16 it needs 48.07 GiB, and one v6e chip has 28.74 GiB of usable HBM.

Google trained 4-bit versions of every Gemma 4 size with quantization-aware training, and its model card lists three formats. For 26B-A4B two of them apply:

GGUF (Q4_0), for llama.cpp. vLLM has no GGUF loader.
"Unquantized" QAT checkpoints, bf16 at 48.07 GiB.
Compressed Tensors (w4a16), "for native, optimized inference with vLLM", listed for "Gemma 4 E2B, E4B, 12B, and 31B".

So for vLLM on one chip the only 26B choice is a third-party build. RedHat's FP8 checkpoint fits with 0.75 GiB to spare, enough KV cache for 3,456 tokens, which is one and a half 2,048-token requests.

This article builds the missing W4A16 checkpoint from Google's own QAT weights and serves it. The result is on Hugging Face as xbill9/gemma-4-26B-A4B-it-qat-q4_0-w4a16-ct.

At This Point You Should Have…

A Google Cloud project with v6e quota in a region that offers ct6e-standard-1t, and the gcloud CLI logged in
A Cloud Storage bucket for checkpoints and results
About 70 GB of local disk and Python 3 with numpy, for the repack
Two clones: git clone https://github.com/xbill9/gemma4-dev and git clone -b gemma4-w4a16-moe https://github.com/xbill9/tpu-inference, the branch of #3660

Step 1 — Look Inside the "Unquantized" Export

google/gemma-4-26B-A4B-it-qat-q4_0-unquantized stores bf16 tensors, but the values came out of QAT for Q4_0: every group of 32 weights along the input dimension already sits on a 16-level grid, step × level with level from −8 to 7. Group size 32 is measured: sampled groups of 64 fail the same test.

That makes a 4-bit checkpoint a change of container. The one trap is the step. The textbook Q4_0 rule, step = max|w| / 8, assumes the largest weight in a group sits at level ±8. When a group's peak sits at level 5, that rule derives 5/8 of the true step and re-rounds every weight onto a grid that does not contain it: about 5% median error per group, and every shape check still passes.

So the repack recovers the step instead: for m from 1 to 8 it tries max|w| / m and keeps the first that reproduces the whole group, then refines the step by least squares over the 32 values.

Step 2 — Repack to Compressed-Tensors W4A16

huggingface-cli download google/gemma-4-26B-A4B-it-qat-q4_0-unquantized \
  --local-dir ~/models/gemma-4-26B-A4B-it-qat-q4_0-unquantized
python3 repack_q4_0.py repack ~/models/gemma-4-26B-A4B-it-qat-q4_0-unquantized \
  ~/models/gemma-4-26B-A4B-it-qat-q4_0-w4a16-ct --workers 6

model-layer-000.safetensors: 1186 tensors, 0 kept bf16
model-layer-001.safetensors: 1186 tensors, 0 kept bf16
...
total 15.29 GiB in 31 shards; 0 tensors kept bf16 for off-grid groups; 222 ignored modules

The output is published as xbill9/gemma-4-26B-A4B-it-qat-q4_0-w4a16-ct, so Steps 2 and 3 can be skipped by downloading it. It writes compressed-tensors pack-quantized symmetric int4 with group size 32, the format of Google's other QAT releases. Attention, the dense MLP and all 3,840 experts are quantized; the router, embeddings, norms and vision tower are copied unchanged. The experts ship fused in the source, experts.gate_up_proj as [128, 1408, 2816], and are written one module per expert, experts.{i}.{gate,up,down}_proj, which is the layout vLLM already reads for int4 mixture-of-experts checkpoints. A tensor with any group off the grid would stay bf16; none did. It ran in 388 seconds on a 16-core machine with 15 GB of RAM, streaming one layer at a time.

Step 3 — Verify Every Group

python3 repack_q4_0.py verify ~/models/gemma-4-26B-A4B-it-qat-q4_0-unquantized \
  ~/models/gemma-4-26B-A4B-it-qat-q4_0-w4a16-ct

verify rereads both checkpoints from disk and counts, for every group, whether the stored level is the source value's rank on the stored grid:

Tensors	Groups of 32	Values bit-identical
Experts, gate and up	475,791,360	92.6%
Experts, down	237,895,680	92.6%
Dense MLP	16,727,040	92.0–92.5%
Attention	34,693,120	89.7–90.4%

All 748 copied tensors are byte-identical. The 7–10% of values that differ in their last bits do so through the scale: Q4_0 carries a 16-bit float step, and the scale is stored at bf16 because that is the precision vLLM's TPU W4A16 layers load it at. The worst relative difference is 1.1e-2. The levels, which carry the QAT training, all match.

Step 4 — Add W4A16 Experts to vLLM's TPU Backend

Gemma 4 runs only on the JAX path of tpu-inference, vLLM's TPU backend. Two pieces serve this checkpoint there:

W4A16 for linear layers: tpu-inference #3653, approved, which serves Google's other QAT sizes.
W4A16 for the experts: a WNA16FusedMoEMethod, tpu-inference #3660, stacked on #3653.

The experts method loads each expert's packed int4 weights and 32-wide scales, fuses gate and up, and hands them to the GMM kernel unchanged. Three properties of the existing mixture-of-experts path decide how:

The default processing re-quantizes expert weights with one scale per output channel. For QAT weights that replaces the trained grid, so the method lays out the weights itself.
The fused mixture-of-experts kernel needs quantization blocks that are multiples of 256, so group 32 goes to the GMM backend, whose gmm_v2 kernel takes a scale per 32-wide group.
gmm_v2 keeps activations in bf16 when the group is narrower than the matrix unit, 256 columns on v6e, and dequantizes each weight tile in fast on-chip memory before the multiply. A group of 32 stays weight-only 4-bit.

Its unit tests include a forward pass through a real expert layer at the 26B's shape, 2816 hidden and 704 intermediate, against a NumPy reference.

The same two changes serve the checkpoint across four chips at tensor parallelism 4. There, two layers split their input rows across chips mid-group, 176 of the experts' 704 rows and 528 of the dense MLP's 2112 per chip, so each 32-row scale is stored as two 16-row groups; every weight keeps the same scale. On a v6e-4 the model uses 21.75 GiB across the four chips, holds 407,168 KV tokens and serves 2,073 output tokens per second, and it reads the suite exactly as it does on one chip.

Step 5 — Launch One v6e Chip

The VM applies the patches to the pinned vLLM TPU image, runs the unit tests on the chip, copies the checkpoint from Cloud Storage, serves it, runs the read, and deletes itself.

gcloud compute instances create jev-tpu-31b-moe2 --zone europe-west4-a \
  --machine-type ct6e-standard-1t \
  --image-family ubuntu-accel-2204-amd64-tpu-v5e-v5p-v6e --image-project ubuntu-os-accelerator-images \
  --boot-disk-size 200GB --scopes cloud-platform --maintenance-policy TERMINATE \
  --provisioning-model FLEX_START --reservation-affinity none --request-valid-for-duration 2h \
  --max-run-duration 4h --instance-termination-action DELETE \
  --metadata jev-code=<code-tarball>,jev-run=2026-09-26-moe2,jev-patches="kvshare.diff wna16.diff moe.diff",jev-gcs-models=gs://<bucket>/jev-tpu-31b/models/gemma-4-26B-A4B-it-qat-q4_0-w4a16-ct \
  --metadata-from-file startup-script=tpu/startup_quant.sh,jev-arms=<arms-file>

The server runs with the flags of every other run in the repository, so results pair record for record:

vllm serve /work/models/gemma-4-26B-A4B-it-qat-q4_0-w4a16-ct --tensor-parallel-size 1 \
  --max-model-len 2048 --max-num-seqs 16 --max-logprobs 32 --generation-config vllm \
  --limit-mm-per-prompt '{"image":0,"audio":0,"video":0}' --enable-prefix-caching

Step 6 — Serve and Read

[jev-quant 2026-09-26T13:39:47Z] pytest: 36 passed, 17 warnings in 40.64s
[jev-quant 2026-09-26T13:53:13Z] READY /work/models/gemma-4-26B-A4B-it-qat-q4_0-w4a16-ct after 796s
[jev-quant 2026-09-26T13:53:31Z] 26b-q4w4 smoke: ok 20 records, labels returned [6, 6, 5, 5, 6, 2, 2, 2, 2, 2, 1, 2, 3, 3, 4, 2, 2, 2, 2, 2]
[jev-quant 2026-09-26T13:53:55Z] 26b-q4w4 four tasks: 24s for 1200 decisions at concurrency 8
[jev-quant 2026-09-26T13:55:36Z] 26b-q4w4 suite: 58s for 3880 records
[jev-quant 2026-09-26T13:55:56Z] 26b-q4w4 load: 1283.3 tok/s, range 1260.1 to 1288.4

The read is the one used throughout this repository: four 300-example classification tasks, the same three-choice tasks with the options reversed, and Bespoke Labs' 3,880-record public suite, each answer taken as the highest-probability label. Throughput is 16 concurrent requests of exactly 256 output tokens, median of three passes. The first boot compiled for 334 seconds; the saved compile cache removes that on later boots.

What Fits on One v6e Chip?

Memory statistics | total_hbm_limit_gb=31.24GiB | total_hbm_limit_cap_gb=28.74GiB | total_hbm_used_gb=17.43GiB | total_hbm_avail_gb=11.32GiB

26B-A4B build	HBM used	Free	KV cache
RedHat FP8	27.99 GiB	0.75 GiB	3,456 tokens
QAT W4A16	17.43 GiB	11.32 GiB	53,888 tokens

The 17.43 GiB breaks down, by arithmetic from the checkpoint's tensor headers, into 14.10 GiB of experts, 1.38 GiB of embeddings, 1.07 GiB of vision encoder and 0.86 GiB of attention and dense MLP. The vision encoder is resident because the 26B loads through the multimodal model class even with images switched off. vLLM reports the 53,888 tokens as room for 26 concurrent 2,048-token requests.

Does the Accuracy Hold?

The reference is the bf16 model, google/gemma-4-26B-A4B-it, served with the same flags at tensor parallelism 4 on a v6e-4, where its 48 GiB fits: 61.16 GiB of HBM across four chips and 1,982 output tokens per second. Each build is paired with it on the same records:

Build	Suite	Difference from bf16 (95% range)
bf16, four chips	76.4%
RedHat FP8, TPU	76.0%	−0.4 points (−0.9 to +0.2)
QAT W4A16, NVIDIA L4	76.0%	−0.4 points (−1.2 to +0.3)
QAT W4A16, TPU	75.3%	−1.1 points (−1.8 to −0.3)

On the four classification tasks every build is within noise of bf16: every difference's 95% range spans zero. On the suite, the QAT checkpoint on the GPU lands as close to bf16 as FP8 does; the same weights on the TPU lose 0.7 points more. The checkpoint is the same file on both, so that gap belongs to the TPU's int4 path. Paired directly, W4A16 on the TPU reads 0.7 points below FP8 (−1.5 to +0.1).

The Same Checkpoint on an NVIDIA GPU

The repacked checkpoint loads on stock vLLM with no patches. On one NVIDIA L4 (g2-standard-8) with vllm/vllm-openai at vLLM 0.30.0 and the same flags:

[jev-gpu 2026-09-26T14:11:08Z] READY after 255s
[jev-gpu 2026-09-26T14:11:08Z] memory: Model loading took 14.8 GiB memory and 55.495697 seconds GPU KV cache size: 17,990 tokens, Maximum concurrency for 2,048 tokens per request: 8.78x
[jev-gpu 2026-09-26T14:16:23Z] load: 393.7 tok/s, range 389.5 to 396.3

vLLM picks its existing Marlin int4 kernels for both the experts and the linear layers. Paired with the TPU run, the four tasks agree within one example each, and the suite reads 0.7 points higher on the L4 (76.0% against 75.3%, 95% range +0.3 to +1.1). The KV cache dtype is ruled out as the cause: a TPU run with --kv-cache-dtype bfloat16 set explicitly allocates the same 53,888 tokens and reads the four tasks record for record, with the suite 0.2 points from the default run. The two platforms run different int4 kernels, gmm_v2 on the TPU and Marlin on the GPU.

🔎 Tip: 12B Needs One Flag

Gemma 4 12B ships as Gemma4UnifiedForConditionalGeneration, which the TPU backend does not register, so vLLM falls back to its PyTorch path. Its decoder is the one Gemma4ForCausalLM already serves:

vllm serve google/gemma-4-12B-it --hf_overrides '{"architectures": ["Gemma4ForCausalLM"]}'

That loads bf16 12B on the JAX path at 22.18 GiB against 24.56 GiB on the PyTorch path, and the same flag serves the QAT W4A16 12B at 9.46 GiB with #3653. With the override, vLLM treats the model as text-only and needs no --limit-mm-per-prompt.

Compare and Contrast

One v6e chip	RedHat FP8	QAT W4A16
Source weights	bf16 model, quantized by RedHat	Google's QAT weights
HBM used	27.99 GiB	🥇 17.43 GiB
KV cache	3,456 tokens	🥇 53,888 tokens
Output tokens/s	668	🥇 1,283
Suite accuracy	🥇 76.0%	75.3%
Flex-start cost per million output tokens	$0.56	🥇 $0.29
Runs on stock vLLM TPU today	🥇 yes	needs #3653 and #3660

So, Which One?

For serving on one v6e chip, the QAT W4A16 build: it holds 15.6 times the KV cache, serves 1.92 times the tokens per second, and reads the suite 0.7 points below FP8. On a GPU the same checkpoint reads as close to bf16 as FP8 does. FP8 remains the build that runs on a stock vLLM TPU image today. On NVIDIA GPUs the same QAT checkpoint needs nothing beyond stock vLLM.

What Does It Cost?

At europe-west4's v6e rates of $1.35 per chip-hour on flex-start and $2.97 on demand, the measured throughputs work out to, by arithmetic:

Build	Flex-start	On demand
QAT W4A16, 1,283 tok/s	$0.29 per million output tokens	$0.64
FP8, 668 tok/s	$0.56	$1.23

These assume the chip runs at the measured load around the clock, 16 concurrent requests of 256 tokens.

Teardown

Each VM deletes itself when its run ends, and --max-run-duration deletes it regardless. Confirm nothing is left:

gcloud compute instances list --format='value(name,zone.basename(),status)'

The checkpoint stays in Cloud Storage for the next boot; delete it with gcloud storage rm -r gs://<bucket>/jev-tpu-31b/models/ when done.

Summary

The goal of this article was to serve Google's QAT Gemma 4 26B-A4B on one TPU v6e chip with vLLM. The key to the solution was a lossless repack of Google's bf16 QAT export into compressed-tensors W4A16, and a W4A16 experts method for vLLM's TPU backend that keeps the checkpoint's 32-wide scales. The results were:

🟢 The QAT 26B serves on one v6e chip at 17.43 GiB, against 27.99 GiB for RedHat's FP8 build
🟢 53,888 tokens of KV cache against 3,456, and 1,283 output tokens per second against 668
🟢 Every one of 765 million weight groups repacks onto its 4-bit grid, and all 748 unquantized tensors copy byte for byte
⚠️ Against bf16 the suite reads 1.1 points lower on the TPU (−1.8 to −0.3) and 0.4 lower on an NVIDIA L4 (−1.2 to +0.3); FP8 reads 0.4 lower
🟢 The repacked checkpoint loads unpatched on vLLM 0.30.0 on an NVIDIA L4
⚠️ On TPU it needs #3653 and #3660, neither merged yet
🟢 Across four chips (tensor parallelism 4) it serves at 21.75 GiB with 407,168 KV tokens and reads the suite as it does on one chip
🟢 Gemma 4 12B serves on the TPU backend's JAX path with one --hf_overrides flag

Scope: the bf16 reference and the four-chip QAT run used a v6e-4 (ct6e-standard-4t) at tensor parallelism 4, every other TPU arm on one TPU v6e chip (ct6e-standard-1t, flex-start) in europe-west4-a, vLLM 0.29.1rc1.dev468+g0b7f11a1e at vllm/vllm-tpu@sha256:19a1a052… with #3299, #3653 and #3660 applied, one run per build, --max-model-len 2048, vLLM's default KV cache dtype. The FP8 read comes from a run two days earlier on the same image without the patches, and its throughput from a second VM the same day; the four-task reads repeat record for record across VMs here, the suite within 0.1 points, and throughput moved about 2% between VMs. The GPU run used one NVIDIA L4 in us-central1-a with vLLM 0.30.0. Costs are arithmetic from list prices and the measured throughput. The repacked checkpoint is unofficial and derived from Google's release under Apache 2.0. Parts of the analysis and writing were done with AI assistance (Claude); every figure comes from the committed output files.

The strategy for serving Google's QAT Gemma 4 26B on one TPU v6e chip was validated with an incremental step by step approach.

References

The repacked checkpoint: https://huggingface.co/xbill9/gemma-4-26B-A4B-it-qat-q4_0-w4a16-ct
Code, repack, per-record results and logs: https://github.com/xbill9/gemma4-dev/tree/main/jev-tpu-31b
tpu-inference #3660, W4A16 experts method: https://github.com/vllm-project/tpu-inference/pull/3660
tpu-inference #3653, W4A16 linear method: https://github.com/vllm-project/tpu-inference/pull/3653
Google's QAT source checkpoint: https://huggingface.co/google/gemma-4-26B-A4B-it-qat-q4_0-unquantized
RedHat FP8 build: https://huggingface.co/RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic
Google's QAT announcement: https://blog.google/innovation-and-ai/technology/developers-tools/quantization-aware-training-gemma-4/
Bespoke Labs public suite: https://github.com/bespokelabsai/nimble/blob/0e67403/docs/PUBLIC_BENCHMARKS.md
vLLM TPU documentation: https://docs.vllm.ai/projects/tpu/en/latest/
Cloud TPU v6e: https://cloud.google.com/tpu/docs/v6e

Gemma 4 on Amazon SageMaker: QAT Weights Decode 2.05x Faster Than bf16 on One L4

xbill — Sat, 26 Sep 2026 11:10:27 +0000

This article gives a short background on Amazon SageMaker real-time endpoints, then measures Gemma 4 E2B's quantization-aware trained (QAT) checkpoint against the full-size bf16 release on the same NVIDIA L4 endpoint. A suite of Python MCP tools is built to simplify management of the vLLM hosted deployment.

https://github.com/xbill9/sagemaker-gemma


Models	`google/gemma-4-E2B-it` (bf16) and `google/gemma-4-E2B-it-qat-w4a16-ct` (QAT, 4-bit weights)
Hardware	SageMaker `ml.g6.xlarge`, 1x NVIDIA L4, 24 GB
Region	`us-east-2`
Software	AWS vLLM SageMaker container, vLLM 0.30.0
Result	QAT decodes at 105.1 tok/s against 51.3, and serves 1077.25 tok/s at 16 parallel requests against 619.1, with the same score on 40 checked questions

SageMaker Hosting in Five Minutes

SageMaker real-time inference is three objects, created in order:

Object	What it holds
Model	A container image, its environment variables, and an IAM role
Endpoint config	Which model runs on which instance types, and how many instances
Endpoint	The running HTTPS service, billed per instance-hour while it exists

Requests go through aws sagemaker-runtime invoke-endpoint, signed with your AWS credentials. SageMaker health-checks the container, routes traffic to it and writes its log to CloudWatch under /aws/sagemaker/Endpoints/<name>.

The container here is the vLLM build AWS publishes for SageMaker. It reads vLLM settings from SM_VLLM_ environment variables, downloads the model from Hugging Face at start-up, and accepts OpenAI-style chat bodies. Switching checkpoints is one variable: SM_VLLM_MODEL.

SageMaker JumpStart also lists Gemma 4 as ready-made packages:

aws sagemaker list-hub-contents --hub-name SageMakerPublicHub --hub-content-type Model \
  --query "HubContentSummaries[].HubContentName" --output text | tr '\t' '\n' | grep gemma-4

huggingface-llm-gemma-4-31b-it-nvfp4
huggingface-vlm-gemma-4-12b-it
huggingface-vlm-gemma-4-26b-a4b-it
huggingface-vlm-gemma-4-31b-it
huggingface-vlm-gemma-4-31b-it-fp8-block
huggingface-vlm-gemma-4-e2b-instruct
huggingface-vlm-gemma-4-e4b-it

None of the seven is a Google QAT checkpoint, so this comparison uses the vLLM container with a Hugging Face model ID.

Three account limits shape every deployment:

Quota is per instance type, per region, counted in instances. This account holds 1 for each single-L4 type in us-east-1, us-east-2 and us-west-2.
Capacity is separate. us-east-1 and us-west-2 each left an L4 request in Creating for about 30 minutes and then returned InsufficientInstanceCapacity. us-east-2 placed one within minutes every time it was asked.
A fallback list (InstancePools in the endpoint config) holds quota for every type in it, so two endpoints cannot share one region's single-L4 quota.

With one L4 per region to work with, the two checkpoints ran one after the other on the same instance type in the same region.

Where Do I Start?

The deployment itself, from quota check to teardown with the aws CLI and the MCP server, is the first article in this pair: https://dev.to/aws-builders/gemma-4-on-an-amazon-sagemaker-endpoint-aws-cli-nvidia-l4-and-an-mcp-server-2c9d

This article starts from a working endpoint and changes one thing: the checkpoint.

At This Point You Should Have…

The repository above, with aws login done and mcp 2.x installed
A SageMaker quota of at least 1 for ml.g6.xlarge in a region with L4 capacity
The full-size endpoint from the first article deployed as gemma-4-e2b

What QAT Changes

Google trains the QAT checkpoint with 4-bit weights in the loop, then exports it in the compressed-tensors format vLLM reads natively. The -w4a16-ct suffix means 4-bit weights, 16-bit activations. Google publishes the same model in four QAT forms; only this one loads in vLLM:

Checkpoint	vLLM
`-qat-w4a16-ct`	🟢 loads, 4-bit weights
`-qat-q4_0-unquantized`	⚠️ stored at 16-bit, no memory saving
`-qat-q4_0-gguf`	❌ GGUF, for llama.cpp and Ollama
`-qat-mobile-*`	❌ on-device formats

Deploying it is the first article's Steps 4 to 6 with two variables changed:

export NAME=gemma-4-e2b-qat
export MODEL_ID=google/gemma-4-E2B-it-qat-w4a16-ct

vLLM names the format when it starts:

quantization=compressed-tensors

What the Engine Allocates

The container log records the weights, the load time and the KV cache vLLM builds from what is left of the L4's memory:

Model loading took 9.75 GiB memory and 82.751806 seconds
GPU KV cache size: 723,484 tokens, Maximum concurrency for 8,192 tokens per request: 88.32x

Model loading took 8.01 GiB memory and 66.190265 seconds
GPU KV cache size: 867,999 tokens, Maximum concurrency for 8,192 tokens per request: 105.96x

	bf16	QAT	QAT / bf16
Weights (GiB)	9.75	8.01	0.82
KV cache (tokens)	723,484	867,999	1.2
Weight load (s)	82.75	66.19
Create to `InService` (min)	9.9	10.1

The 4-bit weights save 18% of GPU memory. The checkpoint's own header shows why: only the transformer body is 4-bit.

Part of the QAT file	GB	Share	Stored as
Per-layer embedding	4.698	56.5%	BF16
Vocabulary embedding	1.611	19.4%	BF16
Transformer body	1.056	12.7%	packed 4-bit
Audio tower	0.614	7.4%	BF16
Vision tower	0.337	4.1%	BF16

The memory the smaller weights free goes to the KV cache.

How the Measurement Works

compare.py runs the same three measurements against each endpoint, at temperature 0, through the same aws sagemaker-runtime invoke-endpoint call:

Decode. Fixed-length replies of 16 and 512 tokens (ignore_eos), five of each. The decode rate is (512 − 16) / (median time at 512 − median time at 16), which cancels the aws CLI start-up and the network round trip.
Parallel requests. 1, 4 and 16 requests at once, 256 tokens each, two batches per level. Throughput is total output tokens divided by the batch's wall time.
Answers. 40 fixed questions with exact answers: 15 two-digit multiplications, 15 three-number sums, 10 capitals. Scored by regular expression.

python3 compare.py measure docs/runs/2026-09-25-qat-vs-bf16 gemma-4-e2b-qat@us-east-2

wrote docs/runs/2026-09-25-qat-vs-bf16/measure-gemma-4-e2b-qat.json
{
  "decode_tokens_per_second": 105.1,
  "per_call_fixed_cost_seconds": 0.562,
  "load_tokens_per_second": {
    "1": 85.35,
    "4": 328.7,
    "16": 1077.25
  },
  "quality": "37/40"
}

combine computes every ratio from the two result files:

python3 compare.py combine compare.json measure-gemma-4-e2b.json measure-gemma-4-e2b-qat.json

                               gemma-4-e2b     gemma-4-e2b-qat  ratio
weights_gib                           9.75                8.01  0.82
kv_cache_tokens                     723484              867999  1.2
load_seconds                     82.751806           66.190265  
decode_tokens_per_second              51.3               105.1  2.05
load_c1_tokens_per_second             45.6               85.35  1.87
load_c4_tokens_per_second            171.6               328.7  1.92
load_c16_tokens_per_second           619.1             1077.25  1.74
quality_correct                         37                  37  
identical answers: 35/40

Decode Speed

	bf16	QAT
Decode (tokens/s)	51.3	🥇 105.1
512-token reply, fastest (s)	10.491	🥇 5.392
512-token reply, slowest (s)	10.661	🥇 5.526
Per-call client cost (s)	0.633	0.562

QAT decodes 2.05x faster. The slowest QAT reply finished in about half the time of the fastest bf16 one, so the gap is far larger than the spread between repeats.

Each decode step reads every transformer layer's weights from GPU memory, so decode speed follows how many bytes those layers take. Those are the layers the QAT export stores at 4 bits. The embeddings are looked up one row per token, so their 16-bit size costs memory and little time.

Parallel Requests

Requests at once	bf16 tok/s	QAT tok/s	QAT / bf16
1	45.6	🥇 85.35	1.87
4	171.6	🥇 328.7	1.92
16	619.1	🥇 1077.25	1.74

QAT leads at every level. The ratio narrows at 16, where a batch shares each weight read across more requests and the per-token saving counts for less. The single-request figures sit below the decode rate because each call also pays the aws CLI start-up.

Does It Still Answer Correctly?

Questions	bf16	QAT
Multiplication (15)	15	15
a + b − c (15)	12	12
Capitals (10)	10	10
Total (40)	37	37

35 of the 40 answers are identical character for character. The other five are all sums, and each model misses three of them:

Question	Expected	bf16	QAT
876 + 608 − 558	926	❌ 1026	✅ 926
257 + 388 − 290	355	✅ 355	❌ 655
782 + 571 − 534	819	❌ 829	✅ 819
470 + 238 − 983	−275	✅ −275	❌ −285
150 + 937 − 139	948	❌ 918	❌ 1010

The two models make the same number of mistakes on different questions. Forty questions are enough to show a large loss and too few to measure a small one.

Re-Measured in A-B-A Order

Each endpoint runs on its own physical instance, and the two ran 15 minutes apart. To check that the gap belongs to the checkpoint, the bf16 endpoint was deployed a second time after QAT and measured again:

Run	Start (UTC)	Decode tok/s	16 at once tok/s	Correct
bf16	18:36	51.3	619.1	37
QAT	18:50	105.1	1077.25	37
bf16 again	19:10	51.5	625.15	37

The two bf16 runs agree within 1.21% on every speed figure and give identical answers to all 40 questions. The second bf16 instance loaded its weights in 82.09 s against 82.75 s the first time.

Compare to Other Deployments

The same bf16 model on the same GPU has been measured on two other platforms in this series, with vllm bench serve at 128 output tokens:

Platform	1 request tok/s
Cloud Run, NVIDIA L4	49.63
EC2 `g6.2xlarge`, NVIDIA L4	46.09
SageMaker `ml.g6.xlarge`, NVIDIA L4	45.6

The three sit within 9% of each other. SageMaker's figure includes the aws CLI start-up in every call; its decode rate with that removed is 51.3.

On a Tesla T4, QAT decoded 1.79x faster than bf16 for the same model. On the L4 the ratio is 2.05x.

The methods differ: the other runs used vllm bench serve with random prompts, other vLLM versions and their own hosts. Read the rows as a shape.

And Price/Performance?

ml.g6.xlarge in us-east-2 is $1.1267 an hour on demand:

aws pricing get-products --region us-east-1 --service-code AmazonSageMaker \
  --filters Type=TERM_MATCH,Field=instanceName,Value=ml.g6.xlarge Type=TERM_MATCH,Field=regionCode,Value=us-east-2

ml.g6.xlarge USE2-Host:ml.g6.xlarge 1.1267000000 Hrs | $1.1267 per Hosting ml.g6.xlarge hour in US East (Ohio)

The same hourly price buys twice the tokens. Per million output tokens (arithmetic):

Requests at once	bf16 $/M	QAT $/M
1	6.86	🥇 3.67
4	1.82	🥇 0.95
16	0.51	🥇 0.29

The endpoint bills while it exists, busy or idle, so these figures hold only while it is kept busy.

So, Which One?

	bf16	QAT
Decode speed	51.3 tok/s	🥇 105.1 tok/s
16 requests at once	619.1 tok/s	🥇 1077.25 tok/s
GPU memory for weights	9.75 GiB	🥇 8.01 GiB
Checked answers	37 / 40	37 / 40
Cost per million tokens at 16	$0.51	🥇 $0.29

On an L4 SageMaker endpoint, the QAT checkpoint is the default choice for Gemma 4 E2B: the same instance, one changed environment variable, about twice the tokens per dollar, and no measured change in answers. Keep bf16 as the reference when a task's accuracy needs a larger evaluation than 40 questions.

What Stops the Meter

delete_endpoint removes the endpoint, its config and its model:

🗑️ `gemma-4-e2b-qat` in `us-east-2`
- endpoint: deleted
- endpoint-config: deleted
- model: deleted

A Creating endpoint refuses deletion, so a deploy runs until it reaches InService or Failed before it can be removed.

Summary

The goal of this article was to measure what Gemma 4 E2B's QAT checkpoint changes on a SageMaker L4 endpoint. The key to the solution was changing only SM_VLLM_MODEL between two deployments on the same instance type, and measuring decode speed with the per-call client cost removed. The measured results were:

🟢 QAT decodes at 105.1 tok/s against bf16's 51.3, 2.05x
🟢 At 16 requests at once QAT serves 1077.25 tok/s against 619.1, 1.74x
🟢 Both score 37 of 40 on checked questions, with 35 identical answers
🟢 A second bf16 deployment after QAT reproduced the first within 1.21%
⚠️ The 4-bit export saves 18% of weight memory, because the embeddings stay at 16-bit
⚠️ Getting an L4 took three regions: us-east-1 and us-west-2 returned InsufficientInstanceCapacity after about 30 minutes each

Scope: one account, SageMaker ml.g6.xlarge with one NVIDIA L4 in us-east-2, vLLM 0.30.0 from the AWS container, three deployments on 2026-09-25 each on its own instance, measured in bf16, QAT, bf16 order. Prompts were short; long-prompt behaviour was not measured. Decode used five replies per length, parallel throughput two batches per level, and quality 40 questions at temperature 0. Every request went through the aws CLI from one client machine.

The strategy for using MCP for SageMaker deployment and benchmarking was validated with an incremental step by step approach.

References

Repository: https://github.com/xbill9/sagemaker-gemma
Part one, deploying Gemma 4 to SageMaker: https://dev.to/aws-builders/gemma-4-on-an-amazon-sagemaker-endpoint-aws-cli-nvidia-l4-and-an-mcp-server-2c9d
Gemma 4 E2B QAT w4a16: https://huggingface.co/google/gemma-4-E2B-it-qat-w4a16-ct
Gemma 4 E2B: https://huggingface.co/google/gemma-4-E2B-it
Gemma 4 on a Tesla T4, QAT vs bf16: https://dev.to/gde/gemma-4-on-a-tesla-t4-qat-weights-decode-179x-faster-than-bf16-2fi4
2B Gemma 4 on Cloud Run with an NVIDIA L4: https://dev.to/gde/2b-gemma-4-deployment-with-cloud-run-nvidia-l4-mcp-sdk-2x-and-claude-code-4ml3
SageMaker real-time inference: https://docs.aws.amazon.com/sagemaker/latest/dg/realtime-endpoints.html
AWS Deep Learning Containers: https://github.com/aws/deep-learning-containers
compressed-tensors: https://github.com/neuralmagic/compressed-tensors

Gemma 4 on an Amazon SageMaker Endpoint: AWS CLI, NVIDIA L4, and an MCP Server

xbill — Sat, 26 Sep 2026 11:10:24 +0000

This article provides a step by step deployment guide for Gemma 4 E2B to an Amazon SageMaker hosted GPU enabled system. A suite of Python MCP tools is built to simplify management of the vLLM hosted deployment with Claude Code.

https://github.com/xbill9/sagemaker-gemma

What is this project trying to Do?

This project serves Gemma 4 E2B from a SageMaker real-time endpoint on one NVIDIA L4 GPU, using the vLLM container AWS publishes for SageMaker. Every AWS call is a plain aws CLI command, so each step can be run by hand or by the MCP server.

A SageMaker real-time endpoint is a managed HTTPS inference server. SageMaker places the container on a GPU instance, health-checks it, routes requests to it and writes its logs to CloudWatch. There is no instance to patch, no security group to open and no load balancer to build.

Where do I start?

The strategy for starting MCP development for model management is a incremental step by step approach.

First, the basic development environment is setup with the required system variables and a working Claude Code configuration.

Then, the Python MCP server is brought up over stdio and validated with Claude Code in the local environment. The deployment follows as eight steps, each shown as the raw aws command and the MCP tool that runs it.

At This Point You Should Have…

An AWS account and the AWS CLI v2, signed in with aws login
A SageMaker endpoint quota of at least 1 for a single-L4 instance type (ml.g6.xlarge, ml.g6.2xlarge or ml.g6.4xlarge)
Python 3.11 or newer with mcp 2.x
Claude Code or Gemini CLI installed and working
jq for reading JSON replies

Setup the Basic Environment

Clone the repository and install the one requirement into the system Python:

git clone https://github.com/xbill9/sagemaker-gemma
cd sagemaker-gemma
python3 -m pip install -r requirements.txt
cp .env.example .env

.env is gitignored. It holds the settings every tool reads:

cat .env

# Copy to .env (gitignored) and edit. sm.py reads it, so the MCP server, CLI and Makefile all see it.
AWS_REGION=us-east-2
MODEL_ID=google/gemma-4-E2B-it
INSTANCE_TYPE=ml.g6.xlarge
ENDPOINT_NAME=gemma-4-e2b
ROLE_NAME=sagemaker-gemma-execution-role
MAX_MODEL_LEN=8192
# Leave empty to use the newest SageMaker vLLM image in the region.
IMAGE_URI=
# Fallback instance types, highest priority first (same GPU keeps runs comparable).
INSTANCE_POOLS=ml.g6.xlarge,ml.g6.2xlarge,ml.g6.4xlarge

Gemma 4 is Apache-2.0 and ungated on Hugging Face, so no Hugging Face token is needed.

Model Management Tool with MCP Stdio Transport

The simplest MCP transport is stdio: the client launches the server as a local process and talks to it over stdin and stdout. In this project Claude Code is the MCP client. The server is one file, server.py, on the MCP Python SDK 2.x:

mcp = MCPServer(RIG_NAME)
READ_ONLY = ToolAnnotations(readOnlyHint=True, idempotentHint=True)
WRITE = ToolAnnotations(destructiveHint=False)
DESTRUCTIVE = ToolAnnotations(destructiveHint=True)

Every tool carries one of the three annotations, so a client can tell a status check from a deploy from a delete.

The tools do no AWS work themselves. They call sm.py, which runs each request as an aws CLI subprocess:

def aws(*args: str, region: str | None = REGION, parse: bool = True) -> Any:
    """Run `aws <args> --output json` and return the parsed result."""
    cmd = ["aws", *args, "--output", "json"]

The CLI renews an aws login session on its own, so a server that stays up for hours keeps working credentials. sm.py also drops any AWS_SESSION_TOKEN the server inherits from its parent process, because a static token expires inside a long-running server and outranks the login session.

Running the Python Code

The project can be linted:

make lint

All checks passed!
16 files already formatted

and tested:

make test

----------------------------------------------------------------------
Ran 16 tests in 0.010s

OK

The tests replace the aws subprocess with a fake, so they run offline with no credentials. One of them compares the registered tool set and annotations against a fixed list: a tool that failed to register, or a delete that lost its destructive flag, fails the suite. ✅

Test the Protocol by Hand

A client speaks JSON-RPC over stdio. Hold stdin open with sleep, or the server sees end-of-input and exits before it answers:

{ printf '%s\n' \
  '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"probe","version":"0"}}}' \
  '{"jsonrpc":"2.0","method":"notifications/initialized"}' \
  '{"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}}'; sleep 3; } \
  | python3 server.py 2>/dev/null

Summarised:

1 sagemaker-gemma
2 ['check_quotas', 'delete_endpoint', 'deploy_endpoint', 'find_vllm_image', 'get_deployment_config', 'get_endpoint_logs', 'get_endpoint_status', 'get_help', 'list_endpoints', 'query_model', 'verify_model_health']

🟢 The server answers the handshake and lists 11 tools.

Claude Code .mcp.json

Claude Code reads .mcp.json in the project directory and launches the server with the system python3:

{
  "mcpServers": {
    "sagemaker-gemma": {
      "command": "python3",
      "args": ["/home/xbill/sagemaker-gemma/server.py"]
    }
  }
}

Gemini CLI reads the same entry from .gemini/settings.json.

Validation with Claude Code

claude mcp get sagemaker-gemma

sagemaker-gemma:
  Scope: Project config (shared via .mcp.json)
  Status: ✔ Connected

From inside Claude Code, get_help returns the resolved settings and the order of work:

### sagemaker-gemma

Serving `google/gemma-4-E2B-it` with vLLM on a **SageMaker real-time endpoint**, managed
entirely through the aws CLI.

| Setting | Value |
| --- | --- |
| Region | `us-east-2` |
| Endpoint | `gemma-4-e2b` |
| Instance types (priority order) | `ml.g6.xlarge`, `ml.g6.2xlarge`, `ml.g6.4xlarge` |
| Max model length | `8192` |
| Image | `newest SageMaker vLLM image (find_vllm_image)` |
| Execution role | `sagemaker-gemma-execution-role` |

**Order of work:** check_quotas → deploy_endpoint → get_endpoint_status until
`InService` (about 10 minutes once an instance is placed) → verify_model_health
→ query_model → delete_endpoint.

The steps below follow that order. Each one shows the raw CLI command; deploy_endpoint runs Steps 2 to 6 in one call.

export AWS_REGION=us-east-2 AWS_PAGER=
export NAME=gemma-4-e2b
export MODEL_ID=google/gemma-4-E2B-it

Step 1 — Check the Endpoint Quota

SageMaker endpoint quotas are per instance type, per region, and count instances.

aws service-quotas list-service-quotas --service-code sagemaker \
  --query "Quotas[?QuotaName=='ml.g6.xlarge for endpoint usage'].Value"

[
    1.0
]

The check_quotas tool reads the same quota for each fallback type across the US regions:

| Instance | us-east-1 | us-east-2 | us-west-1 | us-west-2 |
| --- | ---: | ---: | ---: | ---: |
| `ml.g6.xlarge` | 1 | 1 | - | 1 |
| `ml.g6.2xlarge` | 1 | 1 | - | 1 |
| `ml.g6.4xlarge` | 1 | 1 | - | 1 |

Regions with quota for at least one of these types: us-east-1, us-east-2, us-west-2.

A - means the type is not offered in that region. A 0 means a quota increase request first.

🔎 Tip: One Quota Call per Region

list-service-quotas pages through every SageMaker quota in the region, and the Service Quotas API is rate limited per account. Twelve calls at once, one per type per region, came back as:

An error occurred (TooManyRequestsException) when calling the ListServiceQuotas operation (reached max retries: 2)

check_quotas makes one call per region, one region at a time, and filters the types from that one result. sm.py also sets AWS_RETRY_MODE=adaptive so the CLI backs off and retries.

Step 2 — Find the vLLM Container

The AWS vLLM repository holds pinned release tags such as 0.30.0-gpu-py312-cu130-ubuntu24.04-sagemaker-v1.1, floating aliases such as 0.30-gpu-py312, and -soci index tags. Older release lines receive patch rebuilds, so the most recent push can carry an older vLLM. Sort the pinned tags by version:

TAG=$(aws ecr describe-images --registry-id 763104351884 --repository-name vllm \
  --query "imageDetails[].imageTags[]" --output text | tr '\t' '\n' \
  | grep -E -- '^[0-9]+\.[0-9]+\.[0-9]+-.*-sagemaker-v[0-9]+\.[0-9]+$' | sort -V | tail -1)
export IMAGE=763104351884.dkr.ecr.$AWS_REGION.amazonaws.com/vllm:$TAG
echo $IMAGE

763104351884.dkr.ecr.us-east-2.amazonaws.com/vllm:0.30.0-gpu-py312-cu130-ubuntu24.04-sagemaker-v1.1

The find_vllm_image tool applies the same rule. The image runs vLLM 0.30.0.

Step 3 — Create the Execution Role

SageMaker assumes this role to pull the image and write logs. It is created once per account.

aws iam create-role --role-name sagemaker-gemma-execution-role \
  --assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"sagemaker.amazonaws.com"},"Action":"sts:AssumeRole"}]}'
aws iam attach-role-policy --role-name sagemaker-gemma-execution-role \
  --policy-arn arn:aws:iam::aws:policy/AmazonSageMakerFullAccess
export ROLE=$(aws iam get-role --role-name sagemaker-gemma-execution-role --query Role.Arn --output text)

On a second run create-role reports EntityAlreadyExists; get-role still sets ROLE. deploy_endpoint creates the role only when it is missing.

Step 4 — Create the Model

A SageMaker model pairs an image with its settings. The container turns each SM_VLLM_ variable into the matching vLLM flag: SM_VLLM_MAX_MODEL_LEN becomes --max-model-len.

aws sagemaker create-model --model-name $NAME --execution-role-arn $ROLE \
  --primary-container "{\"Image\":\"$IMAGE\",\"Environment\":{
    \"SM_VLLM_MODEL\":\"$MODEL_ID\",
    \"SM_VLLM_MAX_MODEL_LEN\":\"8192\",
    \"SM_VLLM_GPU_MEMORY_UTILIZATION\":\"0.9\"}}"

{
    "ModelArn": "arn:aws:sagemaker:us-east-2:<account-id>:model/gemma-4-e2b"
}

Step 5 — Create the Endpoint Config With Fallback Instance Types

The endpoint config says where the model runs. InstancePools lists up to five instance types in priority order, and SageMaker places the first one with a free instance. All three types below carry one L4, so the model sees the same GPU whichever is placed.

aws sagemaker create-endpoint-config --endpoint-config-name $NAME \
  --production-variants "[{\"VariantName\":\"AllTraffic\",\"ModelName\":\"$NAME\",
    \"InitialInstanceCount\":1,
    \"InstancePools\":[{\"InstanceType\":\"ml.g6.xlarge\",\"Priority\":1},
                       {\"InstanceType\":\"ml.g6.2xlarge\",\"Priority\":2},
                       {\"InstanceType\":\"ml.g6.4xlarge\",\"Priority\":3}],
    \"ContainerStartupHealthCheckTimeoutInSeconds\":1800,
    \"ModelDataDownloadTimeoutInSeconds\":1800}]"

{
    "EndpointConfigArn": "arn:aws:sagemaker:us-east-2:<account-id>:endpoint-config/gemma-4-e2b"
}

The two 1800-second timeouts give the container time to download the weights and compile before SageMaker's health check gives up.

🔎 Tip: A Fallback List Holds Quota for Every Type in It

With one Gemma endpoint running on ml.g6.xlarge from the three-type list above, a second endpoint asking for ml.g6.2xlarge in the same region was refused:

ResourceLimitExceeded: The account-level service limit 'ml.g6.2xlarge for endpoint usage' is 1 Instances, with current utilization of 1 Instances and a request delta of 1 Instances.

With a quota of 1 per type, a second endpoint goes in another region, or uses types outside the first endpoint's list.

Step 6 — Create the Endpoint

aws sagemaker create-endpoint --endpoint-name $NAME --endpoint-config-name $NAME

{
    "EndpointArn": "arn:aws:sagemaker:us-east-2:<account-id>:endpoint/gemma-4-e2b"
}

Billing starts when an instance is placed. aws sagemaker wait endpoint-in-service --endpoint-name $NAME blocks until it is ready; get_endpoint_status reports the same state and the instance type that was placed:

✅ `gemma-4-e2b` in `us-east-2`: **InService**
- Instance: `ml.g6.xlarge`

The container log is in CloudWatch, and get_endpoint_logs tails it:

aws logs tail /aws/sagemaker/Endpoints/$NAME --follow

Start-up on the L4, in minutes after create-endpoint:

Phase	Minutes
Weights loaded, 9.75 GiB in 82.75 s	7.4
KV cache sized, 723,484 tokens	9.2
`InService`	9.9

🔎 Tip: A Creating Endpoint Cannot Be Deleted

create-endpoint cannot be undone until the endpoint settles. A delete-endpoint sent while it is Creating is refused:

aws: [ERROR]: An error occurred (ValidationException) when calling the DeleteEndpoint operation: Cannot update in-progress endpoint "arn:aws:sagemaker:us-east-2:<account-id>:endpoint/gemma-4-e2b".

The endpoint finishes starting, bills from the moment its instance is placed, and can be deleted once it reaches InService or Failed. Check the model ID and instance types before Step 6.

🔎 Tip: Capacity Is Separate From Quota

A quota of 1 lets you request one instance; the region still has to have one free. In us-east-1, two requests for L4 instances each stayed in Creating for about 30 minutes and then failed:

Unable to provision requested ML compute capacity due to InsufficientInstanceCapacity error. Please retry using a different ML instance type or after some time.

The same request in us-east-2 placed an instance at once. While SageMaker waits for capacity, no container starts and the CloudWatch log group never appears, which is how get_endpoint_logs tells a capacity wait from a slow model load. No charge accrues during the wait. When it fails, repeat Steps 4 to 6 in another region where Step 1 shows the same quota.

Step 7 — Cross Check the Deployed Model

The vLLM container accepts an OpenAI chat body on invoke-endpoint:

echo '{"messages":[{"role":"user","content":"Why is the sky blue?"}],"max_tokens":256}' > req.json
aws sagemaker-runtime invoke-endpoint --endpoint-name $NAME \
  --content-type application/json --body fileb://req.json out.json
jq -r '.choices[0].message.content' out.json | head -1
jq -c '{model,usage}' out.json

{
    "ContentType": "application/json",
    "InvokedProductionVariant": "AllTraffic"
}
The sky is blue due to a phenomenon called **Rayleigh scattering**. This process is caused by how sunlight interacts with the Earth's atmosphere.
{"model":"google/gemma-4-E2B-it","usage":{"prompt_tokens":15,"total_tokens":271,"completion_tokens":256,"prompt_tokens_details":null,"completion_tokens_details":null}}

verify_model_health sends one short request and checks for a reply:

✅ model=`google/gemma-4-E2B-it` tokens=2 wall=0.749s reply='ok'

Step 8 — Teardown

The endpoint bills by the hour until it is deleted. Deleting the config and the model as well leaves nothing behind. delete_endpoint runs all three:

aws sagemaker delete-endpoint --endpoint-name $NAME
aws sagemaker delete-endpoint-config --endpoint-config-name $NAME
aws sagemaker delete-model --model-name $NAME

Each command prints nothing and exits 0. list_endpoints confirms the account is clear:

### 0 endpoint(s) matching `gemma`

Summary

The goal of this article was to deploy Gemma 4 E2B to an Amazon SageMaker real-time endpoint with the AWS CLI and manage it from an MCP server. The key to the solution was the AWS vLLM SageMaker container, which turns the deployment into three create- calls and makes the endpoint answer OpenAI-style chat requests. The deployment results were:

🟢 Eight CLI steps take an account from quota check to a serving endpoint and back to nothing
🟢 The endpoint reached InService 9.9 minutes after create-endpoint, with the weights using 9.75 GiB of the L4's 24 GB
🟢 The MCP server runs every step from Claude Code, marks each tool read-only, write or destructive, and passes its tests offline
🟢 InstancePools gives one endpoint config several fallback instance types
⚠️ L4 capacity varied by region: us-east-1 refused twice, about 30 minutes each, while us-east-2 placed an instance at once
⚠️ A fallback list holds quota for every type in it

Scope: one account, ml.g6.xlarge with one NVIDIA L4, vLLM 0.30.0 from the AWS container, google/gemma-4-E2B-it at full precision, deployed in us-east-2 on 2026-09-25. Start-up times are from a single deployment.

The strategy for using MCP for SageMaker deployment was validated with an incremental step by step approach.

References

Repository: https://github.com/xbill9/sagemaker-gemma
Gemma 4 E2B on Hugging Face: https://huggingface.co/google/gemma-4-E2B-it
AWS Deep Learning Containers: https://github.com/aws/deep-learning-containers
SageMaker real-time inference: https://docs.aws.amazon.com/sagemaker/latest/dg/realtime-endpoints.html
CreateEndpointConfig API: https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateEndpointConfig.html
MCP Python SDK: https://github.com/modelcontextprotocol/python-sdk
vLLM: https://docs.vllm.ai/

Gemma 4 on Amazon SageMaker: QAT Weights Decode 2.05x Faster Than bf16 on One L4

xbill — Fri, 25 Sep 2026 21:55:07 +0000

https://github.com/xbill9/sagemaker-gemma


Models	`google/gemma-4-E2B-it` (bf16) and `google/gemma-4-E2B-it-qat-w4a16-ct` (QAT, 4-bit weights)
Hardware	SageMaker `ml.g6.xlarge`, 1x NVIDIA L4, 24 GB
Region	`us-east-2`
Software	AWS vLLM SageMaker container, vLLM 0.30.0
Result	QAT decodes at 105.1 tok/s against 51.3, and serves 1077.25 tok/s at 16 parallel requests against 619.1, with the same score on 40 checked questions

SageMaker Hosting in Five Minutes

SageMaker real-time inference is three objects, created in order:

Object	What it holds
Model	A container image, its environment variables, and an IAM role
Endpoint config	Which model runs on which instance types, and how many instances
Endpoint	The running HTTPS service, billed per instance-hour while it exists

SageMaker JumpStart also lists Gemma 4 as ready-made packages:

aws sagemaker list-hub-contents --hub-name SageMakerPublicHub --hub-content-type Model \
  --query "HubContentSummaries[].HubContentName" --output text | tr '\t' '\n' | grep gemma-4

huggingface-llm-gemma-4-31b-it-nvfp4
huggingface-vlm-gemma-4-12b-it
huggingface-vlm-gemma-4-26b-a4b-it
huggingface-vlm-gemma-4-31b-it
huggingface-vlm-gemma-4-31b-it-fp8-block
huggingface-vlm-gemma-4-e2b-instruct
huggingface-vlm-gemma-4-e4b-it

None of the seven is a Google QAT checkpoint, so this comparison uses the vLLM container with a Hugging Face model ID.

Three account limits shape every deployment:

Quota is per instance type, per region, counted in instances. This account holds 1 for each single-L4 type in us-east-1, us-east-2 and us-west-2.
Capacity is separate. us-east-1 and us-west-2 each left an L4 request in Creating for about 30 minutes and then returned InsufficientInstanceCapacity. us-east-2 placed one within minutes every time it was asked.
A fallback list (InstancePools in the endpoint config) holds quota for every type in it, so two endpoints cannot share one region's single-L4 quota.

With one L4 per region to work with, the two checkpoints ran one after the other on the same instance type in the same region.

Where Do I Start?

This article starts from a working endpoint and changes one thing: the checkpoint.

At This Point You Should Have…

The repository above, with aws login done and mcp 2.x installed
A SageMaker quota of at least 1 for ml.g6.xlarge in a region with L4 capacity
The full-size endpoint from the first article deployed as gemma-4-e2b

What QAT Changes

Checkpoint	vLLM
`-qat-w4a16-ct`	🟢 loads, 4-bit weights
`-qat-q4_0-unquantized`	⚠️ stored at 16-bit, no memory saving
`-qat-q4_0-gguf`	❌ GGUF, for llama.cpp and Ollama
`-qat-mobile-*`	❌ on-device formats

Deploying it is the first article's Steps 4 to 6 with two variables changed:

export NAME=gemma-4-e2b-qat
export MODEL_ID=google/gemma-4-E2B-it-qat-w4a16-ct

vLLM names the format when it starts:

quantization=compressed-tensors

What the Engine Allocates

The container log records the weights, the load time and the KV cache vLLM builds from what is left of the L4's memory:

Model loading took 9.75 GiB memory and 82.751806 seconds
GPU KV cache size: 723,484 tokens, Maximum concurrency for 8,192 tokens per request: 88.32x

Model loading took 8.01 GiB memory and 66.190265 seconds
GPU KV cache size: 867,999 tokens, Maximum concurrency for 8,192 tokens per request: 105.96x

	bf16	QAT	QAT / bf16
Weights (GiB)	9.75	8.01	0.82
KV cache (tokens)	723,484	867,999	1.2
Weight load (s)	82.75	66.19
Create to `InService` (min)	9.9	10.1

The 4-bit weights save 18% of GPU memory. The checkpoint's own header shows why: only the transformer body is 4-bit.

Part of the QAT file	GB	Share	Stored as
Per-layer embedding	4.698	56.5%	BF16
Vocabulary embedding	1.611	19.4%	BF16
Transformer body	1.056	12.7%	packed 4-bit
Audio tower	0.614	7.4%	BF16
Vision tower	0.337	4.1%	BF16

The memory the smaller weights free goes to the KV cache.

How the Measurement Works

compare.py runs the same three measurements against each endpoint, at temperature 0, through the same aws sagemaker-runtime invoke-endpoint call:

Decode. Fixed-length replies of 16 and 512 tokens (ignore_eos), five of each. The decode rate is (512 − 16) / (median time at 512 − median time at 16), which cancels the aws CLI start-up and the network round trip.
Parallel requests. 1, 4 and 16 requests at once, 256 tokens each, two batches per level. Throughput is total output tokens divided by the batch's wall time.
Answers. 40 fixed questions with exact answers: 15 two-digit multiplications, 15 three-number sums, 10 capitals. Scored by regular expression.

python3 compare.py measure docs/runs/2026-09-25-qat-vs-bf16 gemma-4-e2b-qat@us-east-2

wrote docs/runs/2026-09-25-qat-vs-bf16/measure-gemma-4-e2b-qat.json
{
  "decode_tokens_per_second": 105.1,
  "per_call_fixed_cost_seconds": 0.562,
  "load_tokens_per_second": {
    "1": 85.35,
    "4": 328.7,
    "16": 1077.25
  },
  "quality": "37/40"
}

combine computes every ratio from the two result files:

python3 compare.py combine compare.json measure-gemma-4-e2b.json measure-gemma-4-e2b-qat.json

                               gemma-4-e2b     gemma-4-e2b-qat  ratio
weights_gib                           9.75                8.01  0.82
kv_cache_tokens                     723484              867999  1.2
load_seconds                     82.751806           66.190265  
decode_tokens_per_second              51.3               105.1  2.05
load_c1_tokens_per_second             45.6               85.35  1.87
load_c4_tokens_per_second            171.6               328.7  1.92
load_c16_tokens_per_second           619.1             1077.25  1.74
quality_correct                         37                  37  
identical answers: 35/40

Decode Speed

	bf16	QAT
Decode (tokens/s)	51.3	🥇 105.1
512-token reply, fastest (s)	10.491	🥇 5.392
512-token reply, slowest (s)	10.661	🥇 5.526
Per-call client cost (s)	0.633	0.562

QAT decodes 2.05x faster. The slowest QAT reply finished in about half the time of the fastest bf16 one, so the gap is far larger than the spread between repeats.

Parallel Requests

Requests at once	bf16 tok/s	QAT tok/s	QAT / bf16
1	45.6	🥇 85.35	1.87
4	171.6	🥇 328.7	1.92
16	619.1	🥇 1077.25	1.74

Does It Still Answer Correctly?

Questions	bf16	QAT
Multiplication (15)	15	15
a + b − c (15)	12	12
Capitals (10)	10	10
Total (40)	37	37

35 of the 40 answers are identical character for character. The other five are all sums, and each model misses three of them:

Question	Expected	bf16	QAT
876 + 608 − 558	926	❌ 1026	✅ 926
257 + 388 − 290	355	✅ 355	❌ 655
782 + 571 − 534	819	❌ 829	✅ 819
470 + 238 − 983	−275	✅ −275	❌ −285
150 + 937 − 139	948	❌ 918	❌ 1010

The two models make the same number of mistakes on different questions. Forty questions are enough to show a large loss and too few to measure a small one.

Re-Measured in A-B-A Order

Run	Start (UTC)	Decode tok/s	16 at once tok/s	Correct
bf16	18:36	51.3	619.1	37
QAT	18:50	105.1	1077.25	37
bf16 again	19:10	51.5	625.15	37

The two bf16 runs agree within 1.21% on every speed figure and give identical answers to all 40 questions. The second bf16 instance loaded its weights in 82.09 s against 82.75 s the first time.

Compare to Other Deployments

The same bf16 model on the same GPU has been measured on two other platforms in this series, with vllm bench serve at 128 output tokens:

Platform	1 request tok/s
Cloud Run, NVIDIA L4	49.63
EC2 `g6.2xlarge`, NVIDIA L4	46.09
SageMaker `ml.g6.xlarge`, NVIDIA L4	45.6

The three sit within 9% of each other. SageMaker's figure includes the aws CLI start-up in every call; its decode rate with that removed is 51.3.

On a Tesla T4, QAT decoded 1.79x faster than bf16 for the same model. On the L4 the ratio is 2.05x.

The methods differ: the other runs used vllm bench serve with random prompts, other vLLM versions and their own hosts. Read the rows as a shape.

And Price/Performance?

ml.g6.xlarge in us-east-2 is $1.1267 an hour on demand:

aws pricing get-products --region us-east-1 --service-code AmazonSageMaker \
  --filters Type=TERM_MATCH,Field=instanceName,Value=ml.g6.xlarge Type=TERM_MATCH,Field=regionCode,Value=us-east-2

ml.g6.xlarge USE2-Host:ml.g6.xlarge 1.1267000000 Hrs | $1.1267 per Hosting ml.g6.xlarge hour in US East (Ohio)

The same hourly price buys twice the tokens. Per million output tokens (arithmetic):

Requests at once	bf16 $/M	QAT $/M
1	6.86	🥇 3.67
4	1.82	🥇 0.95
16	0.51	🥇 0.29

The endpoint bills while it exists, busy or idle, so these figures hold only while it is kept busy.

So, Which One?

	bf16	QAT
Decode speed	51.3 tok/s	🥇 105.1 tok/s
16 requests at once	619.1 tok/s	🥇 1077.25 tok/s
GPU memory for weights	9.75 GiB	🥇 8.01 GiB
Checked answers	37 / 40	37 / 40
Cost per million tokens at 16	$0.51	🥇 $0.29

What Stops the Meter

delete_endpoint removes the endpoint, its config and its model:

🗑️ `gemma-4-e2b-qat` in `us-east-2`
- endpoint: deleted
- endpoint-config: deleted
- model: deleted

A Creating endpoint refuses deletion, so a deploy runs until it reaches InService or Failed before it can be removed.

Summary

🟢 QAT decodes at 105.1 tok/s against bf16's 51.3, 2.05x
🟢 At 16 requests at once QAT serves 1077.25 tok/s against 619.1, 1.74x
🟢 Both score 37 of 40 on checked questions, with 35 identical answers
🟢 A second bf16 deployment after QAT reproduced the first within 1.21%
⚠️ The 4-bit export saves 18% of weight memory, because the embeddings stay at 16-bit
⚠️ Getting an L4 took three regions: us-east-1 and us-west-2 returned InsufficientInstanceCapacity after about 30 minutes each

The strategy for using MCP for SageMaker deployment and benchmarking was validated with an incremental step by step approach.

References

Repository: https://github.com/xbill9/sagemaker-gemma
Part one, deploying Gemma 4 to SageMaker: https://dev.to/aws-builders/gemma-4-on-an-amazon-sagemaker-endpoint-aws-cli-nvidia-l4-and-an-mcp-server-2c9d
Gemma 4 E2B QAT w4a16: https://huggingface.co/google/gemma-4-E2B-it-qat-w4a16-ct
Gemma 4 E2B: https://huggingface.co/google/gemma-4-E2B-it
Gemma 4 on a Tesla T4, QAT vs bf16: https://dev.to/gde/gemma-4-on-a-tesla-t4-qat-weights-decode-179x-faster-than-bf16-2fi4
2B Gemma 4 on Cloud Run with an NVIDIA L4: https://dev.to/gde/2b-gemma-4-deployment-with-cloud-run-nvidia-l4-mcp-sdk-2x-and-claude-code-4ml3
SageMaker real-time inference: https://docs.aws.amazon.com/sagemaker/latest/dg/realtime-endpoints.html
AWS Deep Learning Containers: https://github.com/aws/deep-learning-containers
compressed-tensors: https://github.com/neuralmagic/compressed-tensors

Gemma 4 on an Amazon SageMaker Endpoint: AWS CLI, NVIDIA L4, and an MCP Server

xbill — Fri, 25 Sep 2026 21:50:14 +0000

https://github.com/xbill9/sagemaker-gemma

What is this project trying to Do?

Where do I start?

The strategy for starting MCP development for model management is a incremental step by step approach.

First, the basic development environment is setup with the required system variables and a working Claude Code configuration.

At This Point You Should Have…

An AWS account and the AWS CLI v2, signed in with aws login
A SageMaker endpoint quota of at least 1 for a single-L4 instance type (ml.g6.xlarge, ml.g6.2xlarge or ml.g6.4xlarge)
Python 3.11 or newer with mcp 2.x
Claude Code or Gemini CLI installed and working
jq for reading JSON replies

Setup the Basic Environment

Clone the repository and install the one requirement into the system Python:

git clone https://github.com/xbill9/sagemaker-gemma
cd sagemaker-gemma
python3 -m pip install -r requirements.txt
cp .env.example .env

.env is gitignored. It holds the settings every tool reads:

cat .env

# Copy to .env (gitignored) and edit. sm.py reads it, so the MCP server, CLI and Makefile all see it.
AWS_REGION=us-east-2
MODEL_ID=google/gemma-4-E2B-it
INSTANCE_TYPE=ml.g6.xlarge
ENDPOINT_NAME=gemma-4-e2b
ROLE_NAME=sagemaker-gemma-execution-role
MAX_MODEL_LEN=8192
# Leave empty to use the newest SageMaker vLLM image in the region.
IMAGE_URI=
# Fallback instance types, highest priority first (same GPU keeps runs comparable).
INSTANCE_POOLS=ml.g6.xlarge,ml.g6.2xlarge,ml.g6.4xlarge

Gemma 4 is Apache-2.0 and ungated on Hugging Face, so no Hugging Face token is needed.

Model Management Tool with MCP Stdio Transport

mcp = MCPServer(RIG_NAME)
READ_ONLY = ToolAnnotations(readOnlyHint=True, idempotentHint=True)
WRITE = ToolAnnotations(destructiveHint=False)
DESTRUCTIVE = ToolAnnotations(destructiveHint=True)

Every tool carries one of the three annotations, so a client can tell a status check from a deploy from a delete.

The tools do no AWS work themselves. They call sm.py, which runs each request as an aws CLI subprocess:

def aws(*args: str, region: str | None = REGION, parse: bool = True) -> Any:
    """Run `aws <args> --output json` and return the parsed result."""
    cmd = ["aws", *args, "--output", "json"]

Running the Python Code

The project can be linted:

make lint

All checks passed!
16 files already formatted

and tested:

make test

----------------------------------------------------------------------
Ran 16 tests in 0.010s

OK

Test the Protocol by Hand

A client speaks JSON-RPC over stdio. Hold stdin open with sleep, or the server sees end-of-input and exits before it answers:

{ printf '%s\n' \
  '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"probe","version":"0"}}}' \
  '{"jsonrpc":"2.0","method":"notifications/initialized"}' \
  '{"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}}'; sleep 3; } \
  | python3 server.py 2>/dev/null

Summarised:

1 sagemaker-gemma
2 ['check_quotas', 'delete_endpoint', 'deploy_endpoint', 'find_vllm_image', 'get_deployment_config', 'get_endpoint_logs', 'get_endpoint_status', 'get_help', 'list_endpoints', 'query_model', 'verify_model_health']

🟢 The server answers the handshake and lists 11 tools.

Claude Code .mcp.json

Claude Code reads .mcp.json in the project directory and launches the server with the system python3:

{
  "mcpServers": {
    "sagemaker-gemma": {
      "command": "python3",
      "args": ["/home/xbill/sagemaker-gemma/server.py"]
    }
  }
}

Gemini CLI reads the same entry from .gemini/settings.json.

Validation with Claude Code

claude mcp get sagemaker-gemma

sagemaker-gemma:
  Scope: Project config (shared via .mcp.json)
  Status: ✔ Connected

From inside Claude Code, get_help returns the resolved settings and the order of work:

### sagemaker-gemma

Serving `google/gemma-4-E2B-it` with vLLM on a **SageMaker real-time endpoint**, managed
entirely through the aws CLI.

| Setting | Value |
| --- | --- |
| Region | `us-east-2` |
| Endpoint | `gemma-4-e2b` |
| Instance types (priority order) | `ml.g6.xlarge`, `ml.g6.2xlarge`, `ml.g6.4xlarge` |
| Max model length | `8192` |
| Image | `newest SageMaker vLLM image (find_vllm_image)` |
| Execution role | `sagemaker-gemma-execution-role` |

**Order of work:** check_quotas → deploy_endpoint → get_endpoint_status until
`InService` (about 10 minutes once an instance is placed) → verify_model_health
→ query_model → delete_endpoint.

The steps below follow that order. Each one shows the raw CLI command; deploy_endpoint runs Steps 2 to 6 in one call.

export AWS_REGION=us-east-2 AWS_PAGER=
export NAME=gemma-4-e2b
export MODEL_ID=google/gemma-4-E2B-it

Step 1 — Check the Endpoint Quota

SageMaker endpoint quotas are per instance type, per region, and count instances.

aws service-quotas list-service-quotas --service-code sagemaker \
  --query "Quotas[?QuotaName=='ml.g6.xlarge for endpoint usage'].Value"

[
    1.0
]

The check_quotas tool reads the same quota for each fallback type across the US regions:

| Instance | us-east-1 | us-east-2 | us-west-1 | us-west-2 |
| --- | ---: | ---: | ---: | ---: |
| `ml.g6.xlarge` | 1 | 1 | - | 1 |
| `ml.g6.2xlarge` | 1 | 1 | - | 1 |
| `ml.g6.4xlarge` | 1 | 1 | - | 1 |

Regions with quota for at least one of these types: us-east-1, us-east-2, us-west-2.

A - means the type is not offered in that region. A 0 means a quota increase request first.

🔎 Tip: One Quota Call per Region

list-service-quotas pages through every SageMaker quota in the region, and the Service Quotas API is rate limited per account. Twelve calls at once, one per type per region, came back as:

An error occurred (TooManyRequestsException) when calling the ListServiceQuotas operation (reached max retries: 2)

check_quotas makes one call per region, one region at a time, and filters the types from that one result. sm.py also sets AWS_RETRY_MODE=adaptive so the CLI backs off and retries.

Step 2 — Find the vLLM Container

TAG=$(aws ecr describe-images --registry-id 763104351884 --repository-name vllm \
  --query "imageDetails[].imageTags[]" --output text | tr '\t' '\n' \
  | grep -E -- '^[0-9]+\.[0-9]+\.[0-9]+-.*-sagemaker-v[0-9]+\.[0-9]+$' | sort -V | tail -1)
export IMAGE=763104351884.dkr.ecr.$AWS_REGION.amazonaws.com/vllm:$TAG
echo $IMAGE

763104351884.dkr.ecr.us-east-2.amazonaws.com/vllm:0.30.0-gpu-py312-cu130-ubuntu24.04-sagemaker-v1.1

The find_vllm_image tool applies the same rule. The image runs vLLM 0.30.0.

Step 3 — Create the Execution Role

SageMaker assumes this role to pull the image and write logs. It is created once per account.

aws iam create-role --role-name sagemaker-gemma-execution-role \
  --assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"sagemaker.amazonaws.com"},"Action":"sts:AssumeRole"}]}'
aws iam attach-role-policy --role-name sagemaker-gemma-execution-role \
  --policy-arn arn:aws:iam::aws:policy/AmazonSageMakerFullAccess
export ROLE=$(aws iam get-role --role-name sagemaker-gemma-execution-role --query Role.Arn --output text)

On a second run create-role reports EntityAlreadyExists; get-role still sets ROLE. deploy_endpoint creates the role only when it is missing.

Step 4 — Create the Model

A SageMaker model pairs an image with its settings. The container turns each SM_VLLM_ variable into the matching vLLM flag: SM_VLLM_MAX_MODEL_LEN becomes --max-model-len.

aws sagemaker create-model --model-name $NAME --execution-role-arn $ROLE \
  --primary-container "{\"Image\":\"$IMAGE\",\"Environment\":{
    \"SM_VLLM_MODEL\":\"$MODEL_ID\",
    \"SM_VLLM_MAX_MODEL_LEN\":\"8192\",
    \"SM_VLLM_GPU_MEMORY_UTILIZATION\":\"0.9\"}}"

{
    "ModelArn": "arn:aws:sagemaker:us-east-2:<account-id>:model/gemma-4-e2b"
}

Step 5 — Create the Endpoint Config With Fallback Instance Types

aws sagemaker create-endpoint-config --endpoint-config-name $NAME \
  --production-variants "[{\"VariantName\":\"AllTraffic\",\"ModelName\":\"$NAME\",
    \"InitialInstanceCount\":1,
    \"InstancePools\":[{\"InstanceType\":\"ml.g6.xlarge\",\"Priority\":1},
                       {\"InstanceType\":\"ml.g6.2xlarge\",\"Priority\":2},
                       {\"InstanceType\":\"ml.g6.4xlarge\",\"Priority\":3}],
    \"ContainerStartupHealthCheckTimeoutInSeconds\":1800,
    \"ModelDataDownloadTimeoutInSeconds\":1800}]"

{
    "EndpointConfigArn": "arn:aws:sagemaker:us-east-2:<account-id>:endpoint-config/gemma-4-e2b"
}

The two 1800-second timeouts give the container time to download the weights and compile before SageMaker's health check gives up.

🔎 Tip: A Fallback List Holds Quota for Every Type in It

With one Gemma endpoint running on ml.g6.xlarge from the three-type list above, a second endpoint asking for ml.g6.2xlarge in the same region was refused:

ResourceLimitExceeded: The account-level service limit 'ml.g6.2xlarge for endpoint usage' is 1 Instances, with current utilization of 1 Instances and a request delta of 1 Instances.

With a quota of 1 per type, a second endpoint goes in another region, or uses types outside the first endpoint's list.

Step 6 — Create the Endpoint

aws sagemaker create-endpoint --endpoint-name $NAME --endpoint-config-name $NAME

{
    "EndpointArn": "arn:aws:sagemaker:us-east-2:<account-id>:endpoint/gemma-4-e2b"
}

✅ `gemma-4-e2b` in `us-east-2`: **InService**
- Instance: `ml.g6.xlarge`

The container log is in CloudWatch, and get_endpoint_logs tails it:

aws logs tail /aws/sagemaker/Endpoints/$NAME --follow

Start-up on the L4, in minutes after create-endpoint:

Phase	Minutes
Weights loaded, 9.75 GiB in 82.75 s	7.4
KV cache sized, 723,484 tokens	9.2
`InService`	9.9

🔎 Tip: A Creating Endpoint Cannot Be Deleted

create-endpoint cannot be undone until the endpoint settles. A delete-endpoint sent while it is Creating is refused:

aws: [ERROR]: An error occurred (ValidationException) when calling the DeleteEndpoint operation: Cannot update in-progress endpoint "arn:aws:sagemaker:us-east-2:<account-id>:endpoint/gemma-4-e2b".

The endpoint finishes starting, bills from the moment its instance is placed, and can be deleted once it reaches InService or Failed. Check the model ID and instance types before Step 6.

🔎 Tip: Capacity Is Separate From Quota

A quota of 1 lets you request one instance; the region still has to have one free. In us-east-1, two requests for L4 instances each stayed in Creating for about 30 minutes and then failed:

Unable to provision requested ML compute capacity due to InsufficientInstanceCapacity error. Please retry using a different ML instance type or after some time.

Step 7 — Cross Check the Deployed Model

The vLLM container accepts an OpenAI chat body on invoke-endpoint:

echo '{"messages":[{"role":"user","content":"Why is the sky blue?"}],"max_tokens":256}' > req.json
aws sagemaker-runtime invoke-endpoint --endpoint-name $NAME \
  --content-type application/json --body fileb://req.json out.json
jq -r '.choices[0].message.content' out.json | head -1
jq -c '{model,usage}' out.json

{
    "ContentType": "application/json",
    "InvokedProductionVariant": "AllTraffic"
}
The sky is blue due to a phenomenon called **Rayleigh scattering**. This process is caused by how sunlight interacts with the Earth's atmosphere.
{"model":"google/gemma-4-E2B-it","usage":{"prompt_tokens":15,"total_tokens":271,"completion_tokens":256,"prompt_tokens_details":null,"completion_tokens_details":null}}

verify_model_health sends one short request and checks for a reply:

✅ model=`google/gemma-4-E2B-it` tokens=2 wall=0.749s reply='ok'

Step 8 — Teardown

The endpoint bills by the hour until it is deleted. Deleting the config and the model as well leaves nothing behind. delete_endpoint runs all three:

aws sagemaker delete-endpoint --endpoint-name $NAME
aws sagemaker delete-endpoint-config --endpoint-config-name $NAME
aws sagemaker delete-model --model-name $NAME

Each command prints nothing and exits 0. list_endpoints confirms the account is clear:

### 0 endpoint(s) matching `gemma`

Summary

🟢 Eight CLI steps take an account from quota check to a serving endpoint and back to nothing
🟢 The endpoint reached InService 9.9 minutes after create-endpoint, with the weights using 9.75 GiB of the L4's 24 GB
🟢 The MCP server runs every step from Claude Code, marks each tool read-only, write or destructive, and passes its tests offline
🟢 InstancePools gives one endpoint config several fallback instance types
⚠️ L4 capacity varied by region: us-east-1 refused twice, about 30 minutes each, while us-east-2 placed an instance at once
⚠️ A fallback list holds quota for every type in it

The strategy for using MCP for SageMaker deployment was validated with an incremental step by step approach.

References

Repository: https://github.com/xbill9/sagemaker-gemma
Gemma 4 E2B on Hugging Face: https://huggingface.co/google/gemma-4-E2B-it
AWS Deep Learning Containers: https://github.com/aws/deep-learning-containers
SageMaker real-time inference: https://docs.aws.amazon.com/sagemaker/latest/dg/realtime-endpoints.html
CreateEndpointConfig API: https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateEndpointConfig.html
MCP Python SDK: https://github.com/modelcontextprotocol/python-sdk
vLLM: https://docs.vllm.ai/

Two Iceberg Clients, One Protocol: Where the Time Goes

xbill — Fri, 25 Sep 2026 16:40:44 +0000

This article provides a step by step comparison of the Rust and Python clients for Apache Iceberg REST catalogs. It times both clients on the same operations against the same tables, then breaks one request down to see where the time goes.

https://github.com/xbill9/lakehouse-iceberg-2026

What is this project trying to Do?

Two Apache clients talk to the same catalogs:

iceberg-catalog-rest 0.10.1, the Rust client, built in release mode
pyiceberg 0.12.0, the Python client, on Python 3.14.7

They run against three catalogs:

Apache Polaris 1.7.0 in Docker on the same machine, so there is no network time
Google BigLake and Microsoft OneLake over the internet

Three things are measured: how long each call takes, how long each client takes to start, and where the time in one call is spent.

The benchmark measures speed only. It does not check that answers are correct, and a client that lacks an operation you need is the wrong choice however fast it is.

Why Not Just Count Features?

The Rust client supports 13 of the 25 catalog endpoints tested, and pyiceberg supports 21. Anyone can count that from the two repositories, and it only holds for these two versions.

How long a call takes is in neither repository, so that is what this article measures.

Where do I start?

The strategy for comparing the two clients is an incremental step by step approach.

First, both clients are timed against a local catalog, where only the client's own time is measured. Then one request is broken down to find where the time goes. The same benchmark then runs against two managed catalogs, the two clients' requests are compared, and startup time is measured last.

At This Point You Should Have…

Rust 1.94 or newer, and a release build — this run used rustc 1.98.1
Python 3.10+ with pyiceberg 0.12.0
Docker, for the local Polaris catalog
A quiet machine — this one is a 16-core Linux host
Optional: a managed Iceberg REST catalog, for the internet runs

Step 1 — Build Both Clients and Start Polaris

$ git clone https://github.com/xbill9/lakehouse-iceberg-2026
$ cd lakehouse-iceberg-2026/iceberg-conformance && ./polaris-up.sh
$ export POLARIS_CLIENT_ID=root POLARIS_CLIENT_SECRET=s3cr3t
$ cd ../iceberg-rust-client && cargo build --release

The benchmark will not save results from a debug build.

Step 2 — Time Both Clients Locally

To keep the comparison fair:

Each client runs in its own new process. The benchmark script is Python, so running pyiceberg inside it would give Python a head start.
Startup is timed separately. Each process connects and warms up before any call is timed.
The clients take turns. They alternate going first, so a machine that slows down slows both.
Every timing is saved, and the medians are computed from the saved files.

$ python3 bench.py --rounds 4 --iters 15
apache-polaris     6 ops, 2 clients, 4 rounds x 15 iters = 720 samples
$ python3 bench_report.py
wrote evidence/bench-comparison.txt   8 run(s) across 3 catalog(s)

Six read operations both clients support, median time in microseconds, lowest to highest across four runs on two days:

op	rust	pyiceberg	python ÷ rust
list_namespaces	301.3–768.1	883.2–1479.4	1.90x–4.29x
load_namespace	312.2–699.3	784.7–1451.2	2.08x–3.52x
head_namespace	328.4–584.8	1048.6–1362.7	2.33x–3.19x
list_tables	327.0–552.1	1003.2–1551.3	2.56x–3.53x
load_table	4634.1–5764.4	6405.6–7344.8	1.23x–1.38x
head_table	4266.1–5298.0	5167.9–6155.4	1.14x–1.21x

Rust is 1.90x to 4.29x faster on the four small calls, and 1.14x to 1.38x on the two table calls, where the server itself does more work.

The numbers vary a lot between runs. The Rust time for list_namespaces ranged from 301.3 to 768.1 microseconds, a spread of 154.9%. That is why the table shows ranges.

With no network involved, these ratios are as large as the difference can get. Step 4 adds the internet.

Step 3 — Break One Request Down

The same list_namespaces request is timed in stages on one pyiceberg connection: the raw HTTP request, then parsing the JSON, then building Python objects, then the full client call. The same raw HTTP request is then timed from Rust.

$ python3 bench_breakdown.py --iters 120

Across seven runs on the local catalog, over two days:

The raw HTTP request is 90% to 94% of pyiceberg's call time.
The same HTTP request takes 269.8 to 479.7 microseconds from Rust's HTTP library, reqwest, and 864.3 to 1286.4 microseconds from Python's, requests.
Everything pyiceberg does after the HTTP request adds 57.3 to 134.7 microseconds.

So 85% to 90% of the difference between the two clients is the HTTP library. That percentage is calculated from the two ranges above, run by run.

🔎 Tip: Small Differences Are Noise Here

The breakdown tool prints, next to each stage, whether the time that stage added is bigger than the normal run-to-run noise. Only JSON parsing was. How the rest of pyiceberg's 57.3 to 134.7 microseconds splits up cannot be measured at this sample size.

One setting is worth knowing about. requests re-reads proxy settings from the environment on every call unless trust_env=False is set. Turning it off saved between 32.9 and 89.9 microseconds in each of five runs, but that saving was bigger than the noise in only one run.

Step 4 — Time Them Over the Internet

The same benchmark against BigLake and OneLake. Both clients use the same login token, created once before the run:

$ python3 bench.py --only google-lakehouse --rounds 4 --iters 15 --storage opendal
google-lakehouse   6 ops, 2 clients, 4 rounds x 15 iters = 720 samples
$ python3 bench.py --only microsoft-onelake --rounds 4 --iters 15 --storage opendal
microsoft-onelake  6 ops, 2 clients, 4 rounds x 15 iters = 720 samples

Two runs each, median time in milliseconds:

op	BigLake rust	BigLake py	ratio	OneLake rust	OneLake py	ratio
list_namespaces	102.4–103.1	92.2–106.7	0.90x–1.04x	36.0–36.4	35.9–36.0	0.99x–1.00x
load_namespace	199.4–201.6	199.1–200.9	0.99x–1.01x	44.6–45.1	45.3–46.3	1.02x–1.03x
head_namespace	198.4–200.4	199.2–200.0	0.99x–1.01x	44.8–45.9	45.3–46.6	0.99x–1.04x
list_tables	202.7–205.5	203.2–206.1	1.00x	45.0–46.3	45.4–45.6	0.98x–1.01x
load_table	296.4–297.1	281.9–284.3	0.95x–0.96x	36.3–39.0	160.7–167.7	4.12x–4.62x
head_table	201.7–205.0	199.9–201.4	0.98x–0.99x	53.2–54.4	53.4–53.8	0.98x–1.01x

Apart from load_table, every ratio is between 0.90x and 1.04x. The local difference is under a millisecond per call, and a round trip of 36 to 200 milliseconds hides it.

On BigLake, pyiceberg was 4% to 5% faster on load_table in both runs, for a reason this test did not identify.

🔎 Tip: BigLake Allows 75 Catalog Requests a Minute

BigLake limits catalog requests to 75 a minute per project, and answers faster bursts with HTTP 429. The benchmark discards any timing that did not return success, and bench_breakdown.py --pace-ms waits between requests. The limit is visible with:

$ gcloud alpha services quota list --service=biglake.googleapis.com \
    --consumer=projects/<project> \
    --format="table(metric,consumerQuotaLimits[0].unit,consumerQuotaLimits[0].quotaBuckets[0].effectiveLimit)"
biglake.googleapis.com/irc_catalog_requests          1/min/{project}       75
biglake.googleapis.com/irc_read_requests             1/min/{project}       600

Step 5 — Compare the Two Clients' Requests

OneLake's load_table took 4.12x to 4.62x longer through pyiceberg, while every other OneLake call was even. The two clients send different requests.

pyiceberg sends the header X-Iceberg-Access-Delegation: vended-credentials by default (catalog/rest/__init__.py:881). It asks the catalog to include a temporary storage credential with the table. The Rust client sends no such header.

bench_delegation.py times the same pyiceberg request with and without the header:

$ python3 bench_delegation.py --only microsoft-onelake --iters 40

row                                  p50       p90
GET, header sent                   146.6     235.5
GET, header removed                 38.1      41.3
whole load_table                   148.5     182.4
whole load_table, no header         36.6      42.4

With the header, OneLake adds a storage credential to the response and takes 108.5 ms longer. Without it, pyiceberg's load_table takes 36.6 ms, inside the Rust client's 36.3–39.0 ms. On Polaris and BigLake the header made no measurable difference.

So pyiceberg pays 108.5 to 111.9 ms and gets a credential for reading the table's files. The Rust client pays nothing and gets no credential. The companion article on the Rust client covers what happens when it then tries to read the files.

Step 6 — Measure Startup Time

Startup here means starting a new process, connecting, and answering each of the six operations once. Median of five starts per client per run:

catalog	rust	pyiceberg	python takes longer by
Polaris, local, 4 runs	16.4–26.0 ms	525.8–557.9 ms	507.2–535.1 ms
BigLake, 2 runs	1376.3–1410.3 ms	1911.8–2016.7 ms	535.5–606.4 ms
OneLake, 2 runs	955.9–996.2 ms	1723.5–1772.4 ms	767.6–776.2 ms

Over the internet both clients wait on the same round trips, so the ratio drops from 21.4x–33.6x locally to 1.4x on BigLake and 1.8x on OneLake. The extra half second stays. It matters for a CLI, a Lambda function or a short-lived agent, and hardly at all for a service that starts once.

Where the half second goes, median of five, across seven local runs:

step	time
start Python, `python3 -c pass`	9.0–11.2 ms
`import pyiceberg.catalog.rest`	a further 337.9–366.3 ms
connect: fetch the catalog config and log in	6.8–12.4 ms

Most of it is importing the library. Connecting to the catalog is the smallest part.

On OneLake the import took 338.9 and 341.4 ms, the same as locally, and fetching the config took 447.3 and 493.8 ms. Both clients fetch the config; only pyiceberg pays for the import.

How This Compares to Published Lambda Numbers

Cold Starts Are Dead measured AWS Lambda start times of 88.3 ms for Python 3.13 on arm64 (106.2 ms on x86_64) and 14.1 ms for Rust (17.0 ms), at 512 MB, with hello-world functions and no libraries.

The local Python startup here is about six times that: 525.8 to 557.9 ms against 88.3 ms. The difference is mostly the 337.9 to 366.3 ms spent importing pyiceberg.

Compare and Contrast

	🦀 iceberg-catalog-rest	🐍 pyiceberg
`list_namespaces`, local, µs	🥇 301.3–768.1	883.2–1479.4
Small calls over the internet	tie, 0.90x–1.04x	tie
HTTP library	🥇 `reqwest`	`requests`, 85%–90% of the difference
Extra startup time	🥇 —	507.2–776.2 ms
Of which, importing the library	—	337.9–366.3 ms
`load_table` on OneLake, ms	🥇 36.3–39.0, no credential	160.7–167.7, with a storage credential
Requests storage credentials	no	🥇 yes, by default
Endpoints supported	13 of 25	🥇 21 of 25

So, Which One?

For a long-running service, pick by features. Over the internet the per-call difference disappears, and startup happens once. Choose the client that supports the operations and login method you need.

For a CLI, a Lambda function or a short-lived agent, startup decides it. Python adds half a second or more per process on every catalog tested.

Summary

The goal of this article was to measure how fast the Rust and Python Iceberg REST clients are. The key to the solution was running both as separate processes against the same catalogs, timing startup separately, and breaking one request down to see where the time goes. The results were:

🟢 Locally, Rust is 1.90x to 4.29x faster on small calls, and 85% to 90% of that difference is the HTTP library
🟢 Over the internet the difference disappears: 0.90x to 1.04x on BigLake and OneLake
⚠️ Python starts 507.2 to 776.2 ms slower on every catalog, mostly from importing the library (337.9 to 366.3 ms locally)
⚠️ OneLake's 4x on load_table comes from pyiceberg requesting a storage credential by default, which the Rust client never does
❌ pyiceberg was 4% to 5% faster on BigLake's load_table, cause unknown

Scope: iceberg-catalog-rest 0.10.1 in a release build with rustc 1.98.1 and rustls 0.23.45, and pyiceberg 0.12.0 on Python 3.14.7 with requests 2.34.2 over OpenSSL 3.5.7, on one 16-core Linux host. Apache Polaris 1.7.0 in Docker on the same machine with local file storage: four benchmark runs on 2026-09-17 and 2026-09-18, and seven breakdown runs of 80 to 120 requests each. Google BigLake and Microsoft OneLake over the internet on 2026-09-18: two benchmark runs each. Each benchmark run is 720 timed calls across six read operations; startup is the median of five starts per client per run. No writes were timed. Managed catalogs do not report a version, and each internet run is one region at one point in time.

The strategy for comparing two Iceberg REST clients by speed was validated with an incremental step by step approach.

References

Two Iceberg Clients, One Protocol: Where the Time Goes

xbill — Fri, 25 Sep 2026 16:40:41 +0000

https://github.com/xbill9/lakehouse-iceberg-2026

What is this project trying to Do?

Two Apache clients talk to the same catalogs:

iceberg-catalog-rest 0.10.1, the Rust client, built in release mode
pyiceberg 0.12.0, the Python client, on Python 3.14.7

They run against three catalogs:

Apache Polaris 1.7.0 in Docker on the same machine, so there is no network time
Google BigLake and Microsoft OneLake over the internet

Three things are measured: how long each call takes, how long each client takes to start, and where the time in one call is spent.

The benchmark measures speed only. It does not check that answers are correct, and a client that lacks an operation you need is the wrong choice however fast it is.

Why Not Just Count Features?

The Rust client supports 13 of the 25 catalog endpoints tested, and pyiceberg supports 21. Anyone can count that from the two repositories, and it only holds for these two versions.

How long a call takes is in neither repository, so that is what this article measures.

Where do I start?

The strategy for comparing the two clients is an incremental step by step approach.

At This Point You Should Have…

Rust 1.94 or newer, and a release build — this run used rustc 1.98.1
Python 3.10+ with pyiceberg 0.12.0
Docker, for the local Polaris catalog
A quiet machine — this one is a 16-core Linux host
Optional: a managed Iceberg REST catalog, for the internet runs

Step 1 — Build Both Clients and Start Polaris

$ git clone https://github.com/xbill9/lakehouse-iceberg-2026
$ cd lakehouse-iceberg-2026/iceberg-conformance && ./polaris-up.sh
$ export POLARIS_CLIENT_ID=root POLARIS_CLIENT_SECRET=s3cr3t
$ cd ../iceberg-rust-client && cargo build --release

The benchmark will not save results from a debug build.

Step 2 — Time Both Clients Locally

To keep the comparison fair:

Each client runs in its own new process. The benchmark script is Python, so running pyiceberg inside it would give Python a head start.
Startup is timed separately. Each process connects and warms up before any call is timed.
The clients take turns. They alternate going first, so a machine that slows down slows both.
Every timing is saved, and the medians are computed from the saved files.

$ python3 bench.py --rounds 4 --iters 15
apache-polaris     6 ops, 2 clients, 4 rounds x 15 iters = 720 samples
$ python3 bench_report.py
wrote evidence/bench-comparison.txt   8 run(s) across 3 catalog(s)

Six read operations both clients support, median time in microseconds, lowest to highest across four runs on two days:

op	rust	pyiceberg	python ÷ rust
list_namespaces	301.3–768.1	883.2–1479.4	1.90x–4.29x
load_namespace	312.2–699.3	784.7–1451.2	2.08x–3.52x
head_namespace	328.4–584.8	1048.6–1362.7	2.33x–3.19x
list_tables	327.0–552.1	1003.2–1551.3	2.56x–3.53x
load_table	4634.1–5764.4	6405.6–7344.8	1.23x–1.38x
head_table	4266.1–5298.0	5167.9–6155.4	1.14x–1.21x

Rust is 1.90x to 4.29x faster on the four small calls, and 1.14x to 1.38x on the two table calls, where the server itself does more work.

The numbers vary a lot between runs. The Rust time for list_namespaces ranged from 301.3 to 768.1 microseconds, a spread of 154.9%. That is why the table shows ranges.

With no network involved, these ratios are as large as the difference can get. Step 4 adds the internet.

Step 3 — Break One Request Down

$ python3 bench_breakdown.py --iters 120

Across seven runs on the local catalog, over two days:

The raw HTTP request is 90% to 94% of pyiceberg's call time.
The same HTTP request takes 269.8 to 479.7 microseconds from Rust's HTTP library, reqwest, and 864.3 to 1286.4 microseconds from Python's, requests.
Everything pyiceberg does after the HTTP request adds 57.3 to 134.7 microseconds.

So 85% to 90% of the difference between the two clients is the HTTP library. That percentage is calculated from the two ranges above, run by run.

🔎 Tip: Small Differences Are Noise Here

Step 4 — Time Them Over the Internet

The same benchmark against BigLake and OneLake. Both clients use the same login token, created once before the run:

$ python3 bench.py --only google-lakehouse --rounds 4 --iters 15 --storage opendal
google-lakehouse   6 ops, 2 clients, 4 rounds x 15 iters = 720 samples
$ python3 bench.py --only microsoft-onelake --rounds 4 --iters 15 --storage opendal
microsoft-onelake  6 ops, 2 clients, 4 rounds x 15 iters = 720 samples

Two runs each, median time in milliseconds:

op	BigLake rust	BigLake py	ratio	OneLake rust	OneLake py	ratio
list_namespaces	102.4–103.1	92.2–106.7	0.90x–1.04x	36.0–36.4	35.9–36.0	0.99x–1.00x
load_namespace	199.4–201.6	199.1–200.9	0.99x–1.01x	44.6–45.1	45.3–46.3	1.02x–1.03x
head_namespace	198.4–200.4	199.2–200.0	0.99x–1.01x	44.8–45.9	45.3–46.6	0.99x–1.04x
list_tables	202.7–205.5	203.2–206.1	1.00x	45.0–46.3	45.4–45.6	0.98x–1.01x
load_table	296.4–297.1	281.9–284.3	0.95x–0.96x	36.3–39.0	160.7–167.7	4.12x–4.62x
head_table	201.7–205.0	199.9–201.4	0.98x–0.99x	53.2–54.4	53.4–53.8	0.98x–1.01x

Apart from load_table, every ratio is between 0.90x and 1.04x. The local difference is under a millisecond per call, and a round trip of 36 to 200 milliseconds hides it.

On BigLake, pyiceberg was 4% to 5% faster on load_table in both runs, for a reason this test did not identify.

🔎 Tip: BigLake Allows 75 Catalog Requests a Minute

$ gcloud alpha services quota list --service=biglake.googleapis.com \
    --consumer=projects/<project> \
    --format="table(metric,consumerQuotaLimits[0].unit,consumerQuotaLimits[0].quotaBuckets[0].effectiveLimit)"
biglake.googleapis.com/irc_catalog_requests          1/min/{project}       75
biglake.googleapis.com/irc_read_requests             1/min/{project}       600

Step 5 — Compare the Two Clients' Requests

OneLake's load_table took 4.12x to 4.62x longer through pyiceberg, while every other OneLake call was even. The two clients send different requests.

bench_delegation.py times the same pyiceberg request with and without the header:

$ python3 bench_delegation.py --only microsoft-onelake --iters 40

row                                  p50       p90
GET, header sent                   146.6     235.5
GET, header removed                 38.1      41.3
whole load_table                   148.5     182.4
whole load_table, no header         36.6      42.4

Step 6 — Measure Startup Time

Startup here means starting a new process, connecting, and answering each of the six operations once. Median of five starts per client per run:

catalog	rust	pyiceberg	python takes longer by
Polaris, local, 4 runs	16.4–26.0 ms	525.8–557.9 ms	507.2–535.1 ms
BigLake, 2 runs	1376.3–1410.3 ms	1911.8–2016.7 ms	535.5–606.4 ms
OneLake, 2 runs	955.9–996.2 ms	1723.5–1772.4 ms	767.6–776.2 ms

Where the half second goes, median of five, across seven local runs:

step	time
start Python, `python3 -c pass`	9.0–11.2 ms
`import pyiceberg.catalog.rest`	a further 337.9–366.3 ms
connect: fetch the catalog config and log in	6.8–12.4 ms

Most of it is importing the library. Connecting to the catalog is the smallest part.

On OneLake the import took 338.9 and 341.4 ms, the same as locally, and fetching the config took 447.3 and 493.8 ms. Both clients fetch the config; only pyiceberg pays for the import.

How This Compares to Published Lambda Numbers

The local Python startup here is about six times that: 525.8 to 557.9 ms against 88.3 ms. The difference is mostly the 337.9 to 366.3 ms spent importing pyiceberg.

Compare and Contrast

	🦀 iceberg-catalog-rest	🐍 pyiceberg
`list_namespaces`, local, µs	🥇 301.3–768.1	883.2–1479.4
Small calls over the internet	tie, 0.90x–1.04x	tie
HTTP library	🥇 `reqwest`	`requests`, 85%–90% of the difference
Extra startup time	🥇 —	507.2–776.2 ms
Of which, importing the library	—	337.9–366.3 ms
`load_table` on OneLake, ms	🥇 36.3–39.0, no credential	160.7–167.7, with a storage credential
Requests storage credentials	no	🥇 yes, by default
Endpoints supported	13 of 25	🥇 21 of 25

So, Which One?

For a CLI, a Lambda function or a short-lived agent, startup decides it. Python adds half a second or more per process on every catalog tested.

Summary

🟢 Locally, Rust is 1.90x to 4.29x faster on small calls, and 85% to 90% of that difference is the HTTP library
🟢 Over the internet the difference disappears: 0.90x to 1.04x on BigLake and OneLake
⚠️ Python starts 507.2 to 776.2 ms slower on every catalog, mostly from importing the library (337.9 to 366.3 ms locally)
⚠️ OneLake's 4x on load_table comes from pyiceberg requesting a storage credential by default, which the Rust client never does
❌ pyiceberg was 4% to 5% faster on BigLake's load_table, cause unknown

The strategy for comparing two Iceberg REST clients by speed was validated with an incremental step by step approach.

References

What One Rust Client Can Reach Across Seven Iceberg Catalogs

xbill — Fri, 25 Sep 2026 16:20:56 +0000

This article provides a step by step guide to pointing the Apache Rust client for Iceberg REST catalogs at seven catalogs and recording what works. A Python script runs a small Rust program against each catalog and saves every result.

https://github.com/xbill9/lakehouse-iceberg-2026

What is this project trying to Do?

An earlier article tested what seven Iceberg REST catalogs support. This one asks from the other side: if you write a lakehouse tool in Rust today, will it work with the catalog you already pay for, and what do you need to set up? The client is iceberg-catalog-rest 0.10.1 from the Apache Iceberg project, and the catalogs, tests and test table are the same as in that article:

Apache Polaris 1.7.0, running locally as the reference
Google BigLake and Microsoft OneLake, over the internet
AWS Glue and AWS S3 Tables, over the internet
Databricks Unity and Snowflake Horizon, not run here

The result is a short one. This client implements 13 of the 25 operations the earlier article tested, it logs in to five of the seven catalogs, and on every catalog it logs in to, all 13 answer. Two lines of Cargo.toml stand between it and those five. The other two are AWS, they require a signature this client cannot produce, and the Rust answer to that is two more catalog crates that do not implement quite the same operations.

What Does Each Result Mean?

The test suite is 33 checks covering 25 of the 35 operations in the Iceberg REST specification. Each check gets one of five results:

result	meaning
`OK`	the client sent the request and got an answer
`FAILED`	the client sent the request and got an error
`IMPLICIT`	the client sends this request on its own, and there is no way to call it directly
`NOT-EXPRESSIBLE`	the client has no method for this operation
`NOT-ISSUED`	the client supports it, but it is a write, and that run only reads

Where do I start?

The strategy for testing the client is an incremental step by step approach.

First, a local Polaris catalog is started and the client's source is read to list which operations it supports. The test program is then built and run against Polaris, reads first and writes after, then against the managed catalogs one login method at a time, and finally it tries to read a table's files.

At This Point You Should Have…

Rust 1.94 or newer and cargo — this run used rustc 1.98.1
Docker, for the local Polaris catalog
Python 3.10+ with pyyaml, for the test scripts
Optional: logins for any managed catalog — gcloud for BigLake, az for OneLake, the AWS CLI for Glue and S3 Tables

Step 1 — Start Polaris

Polaris runs locally with permissive settings. Anything that fails here is a problem in the test scripts, so it is fixed before any cloud catalog is tested:

$ git clone https://github.com/xbill9/lakehouse-iceberg-2026
$ cd lakehouse-iceberg-2026/iceberg-conformance
$ ./polaris-up.sh
$ export POLARIS_CLIENT_ID=root POLARIS_CLIENT_SECRET=s3cr3t

Step 2 — Build the Test Program

The versions are pinned exactly, so every result applies to one release:

iceberg = "=0.10.1"
iceberg-catalog-rest = "=0.10.1"
iceberg-storage-opendal = { version = "=0.10.1", features = ["opendal-gcs", "opendal-azdls"] }
reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] }

The last two lines add a TLS backend and the cloud storage backends. What You Add Beyond the Client, near the end, says what happens without each of them.

$ cd ../iceberg-rust-client
$ cargo build --release
    Finished `release` profile [optimized] target(s) in 1m 41s

Step 3 — List What the Client Supports

Each of the 33 checks is matched to a method in the client's published source code, with the file and line number, and check_refs.py confirms all 25 of those lines still sit inside the function they name. Counting each endpoint once, since five checks use the same update_table endpoint:

13 of 25 distinct endpoints are expressible through this client.
Counted per probe the figure is 19 of 33, which is the same fact
weighted by how many probes paper 1 happened to point at one endpoint.

The 12 endpoints it cannot reach split two ways:

Missing — 11 endpoints. All seven view operations, scan planning, metrics reporting, the separate credentials endpoint, and commitTransaction. The client builds eight URLs in total (catalog.rs:177-215), and none of them is a view, a scan plan, a metrics report or a transaction.
Stubbed — 1 endpoint. Catalog::update_namespace exists, but returns the error "Updating namespace not supported yet!" (catalog.rs:659) and sends nothing.

Two endpoints it does reach, it reaches only in part, and both are counted among the 13: load_table cannot ask for ?snapshots=all, and list_namespaces handles paging internally and never sends pageSize. It also reaches registerTable, which is one of the ten operations the earlier article did not cover, so it sits outside this count of 25.

Line numbers quoted here were read by hand and are archived, with the lines themselves, in evidence/rust-source-citations.txt.

Step 4 — Run the Reads Against Polaris

$ python3 run_rust.py --storage opendal
apache-polaris     1 implicit, 14 not-expressible, 11 not-issued, 7 ok
wrote evidence/rust-run-apache-polaris.json

All 7 supported read operations work. The one IMPLICIT result is the config request the client sends when it first connects, and the 11 NOT-ISSUED are writes this run does not send.

Step 5 — Run the Writes Against Polaris

The client has a method for each of those 11, and the read-only run sends none of them. A method that compiles can still fail on the wire, so this step sends them. Polaris is local, permissive and disposable, so the writes run there and on no other catalog, and every request goes through a small logging proxy that records what the client sent beside what came back:

$ python3 run_writes.py
apache-polaris     11 ok, 1 unsupported, 0 failed
wrote evidence/rust-write-surface.txt   22 request(s)

  probe                          verdict      ms     endpoint
  create_namespace               OK           60     reachable
  update_namespace_props         UNSUPPORTED  0      unsupported
  create_table                   OK           64     reachable
  commit_table                   OK           112    reachable
  commit_remove_properties       OK           119    reachable
  commit_add_schema              OK           116    reachable
  commit_set_current_schema      OK           116    reachable
  commit_upgrade_format_version  OK           99     reachable
  rename_table                   OK           3      reachable
  drop_table_purge               OK           16     reachable
  drop_table                     OK           3      reachable
  drop_namespace                 OK           1      reachable

All eleven work, and the scratch namespace is dropped at the end. The one refusal is the stub from Step 3, and the proxy log shows why it takes 0 ms: no request to the properties endpoint appears anywhere in the run.

🔎 Tip: The Proxy Log Shows Three Things

A properties commit always sends both update kinds. Setting a property and removing one produce the same pair, set-properties and remove-properties, because the crate's properties action builds both every time (update_properties.rs:95). The two differ in their contents.

Adding a column is one request carrying two updates, with a requirement attached:

  POST    200  /v1/quickstart_catalog/namespaces/irc_probe_rust_1790009108/tables/t1
          updates: add-schema, set-current-schema
          requirements: assert-current-schema-id

Every commit re-reads the table first, so one property change is a GET and then a POST. That reload is also why a transaction commits against the table's current state, which is open upstream as apache/iceberg-rust #3134.

The log also confirms one thing: a two-level namespace goes out as ...%1Fchild, the unit separator the specification asks for.

Step 6 — Match the Login to Each Catalog

The client can log in with a token, with an OAuth2 client ID and secret, or with fixed extra headers. It has no AWS request signing.

login	catalogs	with this client
OAuth2	Polaris	built in
token from an environment variable	Unity	built in
key-pair JWT	Horizon	built in: a `credential` with no colon is sent as `client_secret` with no `client_id` (`catalog.rs:238`), which is what Horizon expects
`gcloud` or `az` login	BigLake, OneLake	a token created outside the client, which the client cannot renew
AWS SigV4 signing	Glue, S3 Tables	not supported

Glue and S3 Tables require SigV4 on the catalog requests themselves. The specification does not describe that — its security schemes are OAuth2 and bearer tokens, and the signing in it covers storage access — but pyiceberg signs anyway, with rest.sigv4-enabled, rest.signing-name and rest.signing-region, so the Python REST client reaches all seven catalogs and the Rust one reaches five. The request to add it, apache/iceberg-rust #1236, has been open since April 2025.

Fixed headers look like a way round it, and they are worth one measurement: a signature minted for GET /v1/config and passed as a static header got a 200 on that request and a 403 on the next, on Glue and S3 Tables both, with all 7 checks refused through the client. A signature covers the request it signs.

The token the client does hold is refreshed by regenerate_token(), which only knows how to repeat an OAuth2 login, so a long-running program on BigLake or OneLake mints its own.

Step 7 — Reach AWS Through the Other Two Crates

Rust does reach both AWS catalogs. The same Apache project publishes iceberg-catalog-glue and iceberg-catalog-s3tables, both 0.10.1, which call the AWS APIs directly and sign as the AWS SDK does. All three crates implement the same Catalog trait, so a program written against dyn Catalog swaps between them by changing a dependency.

What changes with the dependency is the set of operations that answers. Reading each crate's impl Catalog for block:

  Catalog trait method     rest       glue       s3tables
  update_namespace         refused    sent       refused
  drop_table               sent       sent       refused
  register_table           sent       sent       refused

The other 12 trait methods are sent by all three. Each refusal says why:

  rest       update_namespace         'Updating namespace not supported yet!'
  s3tables   drop_table               'drop_table is not supported for S3Tables; use purge_table instead'
  s3tables   register_table           'Registering a table is not supported yet'
  s3tables   update_namespace         'Update namespace is not supported for s3tables catalog'

One of those four is the service speaking: S3 Tables requires a purge, which the earlier article measured from the wire, so that refusal is the catalog's own rule carried faithfully by the crate. The other three are crate-level gaps, and they land at runtime, since all three crates satisfy the same trait and compile the same way.

So a Rust tool covering all seven catalogs carries three catalog implementations and an operation set that varies by which one is loaded. In Python, one client covers all seven.

Step 8 — Point It at a Managed Catalog

$ python3 run_rust.py --only google-lakehouse --only microsoft-onelake --storage opendal
google-lakehouse   1 implicit, 14 not-expressible, 11 not-issued, 7 ok
wrote evidence/rust-run-google-lakehouse.json
microsoft-onelake  1 implicit, 14 not-expressible, 11 not-issued, 7 ok
wrote evidence/rust-run-microsoft-onelake.json

The same result as Polaris: all 7 supported read operations work on both, over the internet, with a token minted by gcloud and by az.

Step 9 — Read the Table's Files

load_table sets up file access but does not read anything, so each run also reads the table's metadata file through the client. The output lists the names of the settings the storage library received, never their values:

apache-polaris
  fileio config keys: ...
  storage credential keys among them: none
  read: ok, scheme file, 5614 bytes, 21 ms
google-lakehouse
  fileio config keys: ...
  storage credential keys among them: none
  read: ok, scheme gs, 5497 bytes, 451 ms
microsoft-onelake
  fileio config keys: ...
  storage credential keys among them: none
  read: FAILED after 47066 ms
  error: Unexpected => Failure in doing io operation, source: Unexpected
  (persistent) at read, context: { timeout: 10 } => io timeout reached

Local files and Google Cloud Storage read fine, the second one on the Google login already on the machine. No catalog passed along a storage credential, because the client never asks for one: it sends no X-Iceberg-Access-Delegation header, so the storage library finds a login on its own.

Azure is where that runs out. The Azure backend, opendal-service-azdls 0.57.0, has code to use your az login and is never given a way to run the az command (backend.rs:297), which leaves an account key, a SAS token or a service principal secret — and OneLake has no account key. Asking for a credential does not change it: OneLake hands one out, and with the delegation header passed as a static header the read still timed out, after 47181 ms. The client reads the credential from the response (types.rs:226) and copies only the response's config settings into file access (catalog.rs:455). Both halves are open upstream, as #2931 and #1442.

What You Add Beyond the Client

	needed for	what happens without it
a `reqwest` TLS feature	any `https://` catalog	every request fails before any response: 7 failed, 0 ok, no HTTP status. Upstream #2888
`iceberg-storage-opendal`	loading any cloud-stored table	`load_table` refuses: "StorageFactory must be provided for RestCatalog"
a storage login in your environment	reading any table file	the read fails or hangs; catalog-issued credentials are read and unused
a SigV4 signer	Glue and S3 Tables over REST	no login at all, and Step 7's two crates are the way round it

The first two are one line each. iceberg-catalog-rest 0.10.1 declares reqwest with TLS switched off and offers no feature to switch it on, so a build that never names a TLS backend fails on the first https:// request; Cargo merges features across dependencies, so your own line fixes it, and a project already using reqwest with TLS will never see it. The core iceberg 0.10.1 crate ships two storage factories, local files and memory (io/storage/local_fs.rs:330, io/storage/memory.rs:250), with its README pointing at the other crate on line 70.

Compare and Contrast

catalog	login	ok	failed	table files
Apache Polaris	🟢 OAuth2	7 reads, 11 writes	0	🟢 read
Google BigLake	🟢 `gcloud` token	7 reads	0	🟢 read, on the machine's Google login
Microsoft OneLake	🟢 `az` token	7 reads	0	❌ timed out
AWS Glue	❌ SigV4	0	7	—
AWS S3 Tables	❌ SigV4	0	7	—
Databricks Unity	token, not run	—	—	—
Snowflake Horizon	key-pair JWT, not run	—	—	—

Behind those numbers every catalog that answered showed the same shape: 14 endpoints the client has no method for, and every endpoint it does implement returning an answer.

Summary

The goal of this article was to point the Apache Rust Iceberg REST client at seven catalogs and record what works and what it takes. The key to the solution was matching every test to a line in the client's source code, running the same tests against each catalog, and logging every request the client sent. The results were:

🟢 Every endpoint the client implements answered on every catalog it could log in to: 7 reads on Polaris, BigLake and OneLake, and 11 writes on Polaris, with 0 failures
🟢 13 of 25 endpoints are implemented; of the other 12, 11 are missing and 1 is a stub that sends no request
⚠️ Two lines of Cargo.toml stand in front of that: a TLS backend (upstream #2888) and iceberg-storage-opendal for cloud storage
❌ Glue and S3 Tables require SigV4, which this client cannot send, so a Rust tool covering all seven catalogs loads three catalog crates where Python loads one — and those three disagree on three of the 15 Catalog methods, at runtime (upstream #1236, open since April 2025)
❌ OneLake's files could not be read: the Azure backend cannot use an az login, and a credential from the catalog is read and then ignored (upstream #2931, #1442)

Scope: iceberg-catalog-rest 0.10.1 with iceberg 0.10.1, iceberg-storage-opendal 0.10.1 and reqwest 0.12.28 with rustls-tls, built with rustc 1.98.1. Source read 2026-09-04, versions captured 2026-09-17, catalog runs 2026-09-18, write run 2026-09-21, one run each from one machine in one region. Polaris 1.7.0 ran in Docker with permissive settings and local file storage, and the writes ran there alone, so those results describe one permissive server. Glue and S3 Tables were tested only with the signature experiment, and Unity and Horizon were not run at all, their rows coming from the source. The tests check that each operation answers; the answers themselves are not checked. Three of the seven catalogs were on trial accounts in the earlier article, and managed catalogs do not report a version.

The strategy for testing what one Rust Iceberg client can reach was validated with an incremental step by step approach.

References

What One Rust Client Can Reach Across Seven Iceberg Catalogs

xbill — Fri, 25 Sep 2026 16:20:54 +0000

https://github.com/xbill9/lakehouse-iceberg-2026

What is this project trying to Do?

Apache Polaris 1.7.0, running locally as the reference
Google BigLake and Microsoft OneLake, over the internet
AWS Glue and AWS S3 Tables, over the internet
Databricks Unity and Snowflake Horizon, not run here

What Does Each Result Mean?

The test suite is 33 checks covering 25 of the 35 operations in the Iceberg REST specification. Each check gets one of five results:

result	meaning
`OK`	the client sent the request and got an answer
`FAILED`	the client sent the request and got an error
`IMPLICIT`	the client sends this request on its own, and there is no way to call it directly
`NOT-EXPRESSIBLE`	the client has no method for this operation
`NOT-ISSUED`	the client supports it, but it is a write, and that run only reads

Where do I start?

The strategy for testing the client is an incremental step by step approach.

At This Point You Should Have…

Rust 1.94 or newer and cargo — this run used rustc 1.98.1
Docker, for the local Polaris catalog
Python 3.10+ with pyyaml, for the test scripts
Optional: logins for any managed catalog — gcloud for BigLake, az for OneLake, the AWS CLI for Glue and S3 Tables

Step 1 — Start Polaris

Polaris runs locally with permissive settings. Anything that fails here is a problem in the test scripts, so it is fixed before any cloud catalog is tested:

$ git clone https://github.com/xbill9/lakehouse-iceberg-2026
$ cd lakehouse-iceberg-2026/iceberg-conformance
$ ./polaris-up.sh
$ export POLARIS_CLIENT_ID=root POLARIS_CLIENT_SECRET=s3cr3t

Step 2 — Build the Test Program

The versions are pinned exactly, so every result applies to one release:

iceberg = "=0.10.1"
iceberg-catalog-rest = "=0.10.1"
iceberg-storage-opendal = { version = "=0.10.1", features = ["opendal-gcs", "opendal-azdls"] }
reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] }

The last two lines add a TLS backend and the cloud storage backends. What You Add Beyond the Client, near the end, says what happens without each of them.

$ cd ../iceberg-rust-client
$ cargo build --release
    Finished `release` profile [optimized] target(s) in 1m 41s

Step 3 — List What the Client Supports

13 of 25 distinct endpoints are expressible through this client.
Counted per probe the figure is 19 of 33, which is the same fact
weighted by how many probes paper 1 happened to point at one endpoint.

The 12 endpoints it cannot reach split two ways:

Missing — 11 endpoints. All seven view operations, scan planning, metrics reporting, the separate credentials endpoint, and commitTransaction. The client builds eight URLs in total (catalog.rs:177-215), and none of them is a view, a scan plan, a metrics report or a transaction.
Stubbed — 1 endpoint. Catalog::update_namespace exists, but returns the error "Updating namespace not supported yet!" (catalog.rs:659) and sends nothing.

Line numbers quoted here were read by hand and are archived, with the lines themselves, in evidence/rust-source-citations.txt.

Step 4 — Run the Reads Against Polaris

$ python3 run_rust.py --storage opendal
apache-polaris     1 implicit, 14 not-expressible, 11 not-issued, 7 ok
wrote evidence/rust-run-apache-polaris.json

All 7 supported read operations work. The one IMPLICIT result is the config request the client sends when it first connects, and the 11 NOT-ISSUED are writes this run does not send.

Step 5 — Run the Writes Against Polaris

$ python3 run_writes.py
apache-polaris     11 ok, 1 unsupported, 0 failed
wrote evidence/rust-write-surface.txt   22 request(s)

  probe                          verdict      ms     endpoint
  create_namespace               OK           60     reachable
  update_namespace_props         UNSUPPORTED  0      unsupported
  create_table                   OK           64     reachable
  commit_table                   OK           112    reachable
  commit_remove_properties       OK           119    reachable
  commit_add_schema              OK           116    reachable
  commit_set_current_schema      OK           116    reachable
  commit_upgrade_format_version  OK           99     reachable
  rename_table                   OK           3      reachable
  drop_table_purge               OK           16     reachable
  drop_table                     OK           3      reachable
  drop_namespace                 OK           1      reachable

🔎 Tip: The Proxy Log Shows Three Things

Adding a column is one request carrying two updates, with a requirement attached:

  POST    200  /v1/quickstart_catalog/namespaces/irc_probe_rust_1790009108/tables/t1
          updates: add-schema, set-current-schema
          requirements: assert-current-schema-id

The log also confirms one thing: a two-level namespace goes out as ...%1Fchild, the unit separator the specification asks for.

Step 6 — Match the Login to Each Catalog

The client can log in with a token, with an OAuth2 client ID and secret, or with fixed extra headers. It has no AWS request signing.

login	catalogs	with this client
OAuth2	Polaris	built in
token from an environment variable	Unity	built in
key-pair JWT	Horizon	built in: a `credential` with no colon is sent as `client_secret` with no `client_id` (`catalog.rs:238`), which is what Horizon expects
`gcloud` or `az` login	BigLake, OneLake	a token created outside the client, which the client cannot renew
AWS SigV4 signing	Glue, S3 Tables	not supported

The token the client does hold is refreshed by regenerate_token(), which only knows how to repeat an OAuth2 login, so a long-running program on BigLake or OneLake mints its own.

Step 7 — Reach AWS Through the Other Two Crates

What changes with the dependency is the set of operations that answers. Reading each crate's impl Catalog for block:

  Catalog trait method     rest       glue       s3tables
  update_namespace         refused    sent       refused
  drop_table               sent       sent       refused
  register_table           sent       sent       refused

The other 12 trait methods are sent by all three. Each refusal says why:

  rest       update_namespace         'Updating namespace not supported yet!'
  s3tables   drop_table               'drop_table is not supported for S3Tables; use purge_table instead'
  s3tables   register_table           'Registering a table is not supported yet'
  s3tables   update_namespace         'Update namespace is not supported for s3tables catalog'

So a Rust tool covering all seven catalogs carries three catalog implementations and an operation set that varies by which one is loaded. In Python, one client covers all seven.

Step 8 — Point It at a Managed Catalog

$ python3 run_rust.py --only google-lakehouse --only microsoft-onelake --storage opendal
google-lakehouse   1 implicit, 14 not-expressible, 11 not-issued, 7 ok
wrote evidence/rust-run-google-lakehouse.json
microsoft-onelake  1 implicit, 14 not-expressible, 11 not-issued, 7 ok
wrote evidence/rust-run-microsoft-onelake.json

The same result as Polaris: all 7 supported read operations work on both, over the internet, with a token minted by gcloud and by az.

Step 9 — Read the Table's Files

apache-polaris
  fileio config keys: ...
  storage credential keys among them: none
  read: ok, scheme file, 5614 bytes, 21 ms
google-lakehouse
  fileio config keys: ...
  storage credential keys among them: none
  read: ok, scheme gs, 5497 bytes, 451 ms
microsoft-onelake
  fileio config keys: ...
  storage credential keys among them: none
  read: FAILED after 47066 ms
  error: Unexpected => Failure in doing io operation, source: Unexpected
  (persistent) at read, context: { timeout: 10 } => io timeout reached

What You Add Beyond the Client

	needed for	what happens without it
a `reqwest` TLS feature	any `https://` catalog	every request fails before any response: 7 failed, 0 ok, no HTTP status. Upstream #2888
`iceberg-storage-opendal`	loading any cloud-stored table	`load_table` refuses: "StorageFactory must be provided for RestCatalog"
a storage login in your environment	reading any table file	the read fails or hangs; catalog-issued credentials are read and unused
a SigV4 signer	Glue and S3 Tables over REST	no login at all, and Step 7's two crates are the way round it

Compare and Contrast

catalog	login	ok	failed	table files
Apache Polaris	🟢 OAuth2	7 reads, 11 writes	0	🟢 read
Google BigLake	🟢 `gcloud` token	7 reads	0	🟢 read, on the machine's Google login
Microsoft OneLake	🟢 `az` token	7 reads	0	❌ timed out
AWS Glue	❌ SigV4	0	7	—
AWS S3 Tables	❌ SigV4	0	7	—
Databricks Unity	token, not run	—	—	—
Snowflake Horizon	key-pair JWT, not run	—	—	—

Behind those numbers every catalog that answered showed the same shape: 14 endpoints the client has no method for, and every endpoint it does implement returning an answer.

Summary

🟢 Every endpoint the client implements answered on every catalog it could log in to: 7 reads on Polaris, BigLake and OneLake, and 11 writes on Polaris, with 0 failures
🟢 13 of 25 endpoints are implemented; of the other 12, 11 are missing and 1 is a stub that sends no request
⚠️ Two lines of Cargo.toml stand in front of that: a TLS backend (upstream #2888) and iceberg-storage-opendal for cloud storage
❌ Glue and S3 Tables require SigV4, which this client cannot send, so a Rust tool covering all seven catalogs loads three catalog crates where Python loads one — and those three disagree on three of the 15 Catalog methods, at runtime (upstream #1236, open since April 2025)
❌ OneLake's files could not be read: the Azure backend cannot use an az login, and a credential from the catalog is read and then ignored (upstream #2931, #1442)

The strategy for testing what one Rust Iceberg client can reach was validated with an incremental step by step approach.

References

Running a Jev-Style Decision Model on One TPU v6e: What Fits, What It Costs, and What Changes From a GPU

xbill — Thu, 24 Sep 2026 16:32:26 +0000

This article provides a step by step guide to running a Jev-style decision model on one Google Cloud TPU v6e chip with Gemma 4 and vLLM, and compares it with the same read on one NVIDIA L4 GPU. The measurement was pre-registered, and every per-item output is committed.

One v6e chip serves Gemma 4 E2B, E4B and 12B at bf16 and a 26B-A4B fp8 build; no 31B checkpoint loads. The same checkpoints give the same answers on the TPU and the L4, and the 26B decides in 27 to 33 ms against 61 ms on the L4. On demand, the TPU costs more per decision than the L4 or Jev; 12B is the size to pick.

https://github.com/xbill9/gemma4-dev/tree/main/jev-tpu

Why Measure This?

A Jev-style decision model answers a typed question with a probability for each allowed option: end the prompt where the answer starts, read the scores of the allowed label tokens, and apply a softmax over those. TypeSafe's Jev does this as a hosted service, and any open model can be read the same way.

A companion article measured this read on one NVIDIA L4 and set it beside Jev's published results. This one asks what changes on a TPU: which Gemma 4 sizes fit one v6e chip, whether the read works the same way, and what the chip buys in speed and cost.

At This Point You Should Have…

A Google Cloud project with v6e quota in a region that offers ct6e-standard-1t, and the gcloud CLI logged in
A Hugging Face token stored as the Secret Manager secret hf-token, readable by the Compute Engine default service account
A Cloud Storage bucket for results, set as BUCKET in tpu/run.sh and tpu/startup.sh
The repository cloned: git clone https://github.com/xbill9/gemma4-dev and cd gemma4-dev/jev-tpu

Step 1 — Pre-Register the Measurement

The models, image, serving flags, data, metrics and the order of the quantized attempts are written down and committed before any model call, in PREREGISTRATION.md; each departure from it is recorded there with a date before the affected results are scored.

git show -s --oneline e0cce69 4937fb8

e0cce69 jev-tpu: sibling of jev for one TPU v6e chip — copied read, data, suite and scoring code (proxy byte-identical); pre-registration for E2B/E4B/12B bf16 and exploratory quantized 26B-A4B/31B probes
4937fb8 jev-tpu: VM boot, serve and run drivers; pre-registration amended before any model call — suite built locally at 0e67403 (13/13 checksums) and re-checked on the VM

Step 2 — Pick Checkpoints That Fit One Chip

One v6e chip has 31.24 GiB of memory, of which vLLM uses up to 28.74 GiB:

Memory statistics | total_hbm_limit_gb=31.24GiB | total_hbm_limit_cap_gb=28.74GiB | total_hbm_used_gb=24.56GiB | total_hbm_avail_gb=4.19GiB

E2B, E4B and 12B fit at bf16. The 26B-A4B and 31B do not, so the pre-registration lists quantized builds to try in order:

Size	Checkpoint	Format	Size on disk
E2B	`google/gemma-4-E2B-it`	bf16
E4B	`google/gemma-4-E4B-it`	bf16
12B	`google/gemma-4-12B-it`	bf16
26B-A4B	`RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic`	fp8	26.67 GiB
31B	`google/gemma-4-31B-it-qat-w4a16-ct`	w4a16	21.67 GiB
31B	`cyankiwi/gemma-4-31B-it-AWQ-4bit`	4-bit	19.47 GiB

The 31B fp8 builds are 30.98 GiB, over the chip's usable memory, and were not tried.

Step 3 — Launch One v6e Chip

The VM runs the whole measurement from its startup script, fetching the code and the suite from Cloud Storage, and deletes itself when the run ends.

gcloud compute instances create jev-tpu-v6e1 --zone europe-west4-a \
  --machine-type ct6e-standard-1t \
  --image-family ubuntu-accel-2204-amd64-tpu-v5e-v5p-v6e --image-project ubuntu-os-accelerator-images \
  --boot-disk-size 200GB --scopes cloud-platform --maintenance-policy TERMINATE \
  --provisioning-model STANDARD --max-run-duration 6h --instance-termination-action DELETE \
  --metadata jev-code=<code-tarball>,jev-run=2026-09-24-v6e1 --metadata-from-file startup-script=tpu/startup.sh

NAME          ZONE            MACHINE_TYPE      PREEMPTIBLE  INTERNAL_IP    EXTERNAL_IP    STATUS
jev-tpu-v6e1  europe-west4-a  ct6e-standard-1t               10.164.15.207  34.32.155.166  RUNNING

--scopes cloud-platform lets the VM read the Hugging Face token from Secret Manager at boot, so the token never enters instance metadata.

Step 4 — Serve Each Model

tpu/serve.sh starts one model at a time with the same flags for every arm and waits for /v1/models. Abridged, with the token and volume options left out:

docker run -d --name vllm --privileged --net=host --shm-size 10gb vllm/vllm-tpu:nightly \
  vllm serve google/gemma-4-12B-it --tensor-parallel-size 1 --max-model-len 2048 --max-num-seqs 16 \
  --max-logprobs 32 --generation-config vllm --limit-mm-per-prompt '{"image":0,"audio":0}' --enable-prefix-caching

[jev-run 2026-09-24T14:42:48Z] READY google/gemma-4-E2B-it after 346s
[jev-run 2026-09-24T14:51:34Z] READY google/gemma-4-E4B-it after 421s
[jev-run 2026-09-24T15:02:01Z] READY google/gemma-4-12B-it after 512s
[jev-run 2026-09-24T15:14:59Z] READY RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic after 616s

The image was vllm/vllm-tpu@sha256:19a1a0526476f902eb83e1057f3d8938f35b457dd7716a30e9ab4f7bee90d507, vLLM 0.29.1rc1.dev468+g0b7f11a1e. On v6e it stores the KV cache in fp8 by default for every model:

INFO 09-24 14:53:37 [tpu_platform.py:232] Automatically using fp8_e5m2 for FP8 KV cache on TPU v6e.

So "bf16" below describes the weights. The L4 run kept the default 16-bit KV cache.

Step 5 — Read the Labels Out of the Top 32

On a GPU, the read asks vLLM for the log-probabilities of the label tokens by id. vLLM's TPU backend returns only the top-k log-probabilities, and a request for specific token ids fails. Four request shapes are sent before each model's reads, and the answers are kept:

== probe 3: {"prompt":[2,106,1645,108],"max_tokens":1,"temperature":0,"logprobs":32,"return_tokens_as_token_ids":true}
{"id":"cmpl-9ee6dc5f0461280b","object":"text_completion", ...
== probe 4: {"prompt":[2,106,1645,108],"max_tokens":1,"temperature":0,"logprobs":5,"logprob_token_ids":[236776,236799,236780],"return_tokens_as_token_ids":true}
{"error":{"message":"list index out of range","type":"InternalServerError","param":null,"code":500}}

So the read asks for the top 32 log-probabilities (JEV_TOPK=32) and takes the label tokens from them. A label inside the 32 gets its exact log-probability; a label outside gets the proxy's fallback value, the lowest returned log-probability minus 5, and every four-task record keeps how many labels came back.

Step 6 — Run and Score

Each model gets a five-example smoke read, the four tasks (300 examples each from sst2, AG News, DAIR Emotion and tweet_eval irony), the three choice tasks with their options reversed, a latency pass at one request at a time, and Bespoke Labs' 3,880-record public suite, rebuilt with Nimble's converters and checked against all 13 published checksums on the VM.

[jev-run 2026-09-24T14:36:51Z] suite: 13 of 13 subsets match
[jev-run 2026-09-24T15:02:42Z] 12b four tasks: 23s for 1200 decisions at concurrency 8
[jev-run 2026-09-24T15:04:25Z] 12b suite: 60s for 3880 records

python3 sweep_score.py --prefix 2026-09-24-v6e1 --arms e2b e4b 12b 26b-fp8
python3 nimble_suite/suite_sweep.py --prefix 2026-09-24-v6e1 --arms e2b e4b 12b 26b-fp8

Both write their tables into results/, and every figure below comes from those files and article_figures.py.

What Fits and Loads on One v6e Chip?

Size	Checkpoint	Serves	Time to serve
E2B	bf16	yes	346 s
E4B	bf16	yes	421 s
12B	bf16	yes	512 s
26B-A4B	fp8, 26.67 GiB	yes	616 s
31B	w4a16, 21.67 GiB	no	fails at 120 s
31B	4-bit, 19.47 GiB	no	fails at 120 s

Both 31B builds stop with the same error:

NotImplementedError: compressed-tensors scheme for layer 'model.language_model.layers.0.self_attn.q_proj' is not yet supported in the JAX path.

Gemma 4 runs only on the JAX path of vLLM's TPU backend, which loads fp8 but no 4-bit format for a dense model. The one 31B format it loads is 30.98 GiB, over one chip's usable memory.

Does the TPU Change the Answers?

No. E2B and E4B were also read on the L4, with the same checkpoints and prompts. The predicted label matched on 1,186 and 1,193 of 1,200 examples, and accuracy per task differed by at most 1.0 and 0.4 points. The 26B-A4B fp8 build here and the 4-bit AWQ build on the L4 are different quantizations of the same model and differ by at most 1.3 points per task and 0.7 points on the suite.

Task	Majority	E2B	E4B	12B	26B-A4B fp8
sst2	51.0%	88.7%	94.0%	95.0%	94.7%
AG News	25.0%	30.0%	83.3%	86.3%	86.0%
DAIR Emotion	35.0%	53.3%	54.3%	59.3%	58.3%
tweet_eval irony	60.3%	73.7%	84.0%	84.7%	91.0%

Per-task calibration, option-order changes and 90th-percentile times are in results/2026-09-24-v6e1-SWEEP.md.

How Do the Sizes Compare With Jev?

The Jev comparison is the L4 article's subject, with its caveats in full. On the same 3,880 public records:

Questions	Jev 1.13.0, published	26B-A4B fp8	12B	E4B	E2B
All, 3,880	77.3%	76.0%	76.2%	72.8%	68.5%
Yes/no, 1,399	84.6%	84.2%	85.1%	81.5%	76.2%
Multiple choice, 1,848	82.8%	79.7%	78.1%	74.5%	70.8%
Five-level rating, 633	45.2%	47.4%	51.0%	48.5%	44.9%
Median ECE after 50 labels (Jev 0.071 as shipped)		0.070	0.070	0.077	0.092

Jev leads 12B by 1.1 points overall (95% range −0.8 to +3.0) and 26B-A4B by 1.3 (−0.6 to +3.2), and on multiple choice by 4.7 and 3.2. The L4's 4-bit 26B scored 75.3%, 2.1 points behind with a range from 0.2 to 4.0, so read the 26B as one to two points behind Jev on either platform. Jev's per-record answers are unpublished, so these ranges compare two independent proportions, and records that share a passage widen them. Gemma was read in the PR proxy's prompt format with no tuning; a different prompt could move the multiple-choice gap in either direction.

Did the Top-32 Read Lose Anything?

Not on accuracy. At least one label came back on every four-task read, so the predicted label is exact; only the probabilities of labels outside the 32 are approximate.

On the four tasks, moving every missing label from the fallback value to the top-32 bound, the most probability it could have had, changes raw ECE by less than 0.001 and ECE after 50 labels by at most 0.005. That check was added after the run. The suite records do not keep how many labels came back, so the suite calibration figures have no such check; by a lower-bound count, at least 10.2% of 12B's suite records and 17.8% of 26B-A4B's used the fallback value.

The share of probability on the allowed labels varies too. On irony, 12B's most likely token was a label on 5 of 300 reads, with a median of 3.1% of its probability on the two labels; 26B-A4B's was a label on 161 of 300 AG News reads, with a median of 55.2%. The read rescales the labels to sum to one, so the answer reads as confident either way, and label_mass in every record shows it.

TPU v6e Against the L4

	One TPU v6e chip	One NVIDIA L4 (`g6.xlarge`)
Memory for the model	🥇 28.74 GiB	23034 MiB
Largest Gemma 4 read here	12B at bf16, 26B-A4B at fp8	26B-A4B at 4-bit
Accuracy, same checkpoint	same answers on 1,186 to 1,193 of 1,200	same
26B time per decision, one at a time	🥇 27 to 33 ms	61 ms
Price per hour, on demand	$2.97	🥇 $0.8048
26B cost per million decisions, on demand	$10.37	🥇 at most $5.43
Label read	top 32 log-probabilities	🥇 exact label token ids
KV cache	fp8 by default	🥇 16-bit

In this table 🥇 marks the better value. The 26B decides 1.8 to 2.2 times faster on the TPU. The chip costs 3.7 times as much per hour on demand, or 1.7 times at the flex-start price of $1.35, a mode that had no capacity during this run. The L4's cost comes from its first run's throughput with the client on a home connection, so it is an upper bound; the TPU's comes from the request times at eight in flight.

🔎 Tip: Serve the Same Checkpoint on Both Platforms First

Before comparing hardware, serve one checkpoint on both with the same prompts and compare predicted labels record by record. It checks the whole read, from chat template to label tokens; here the TPU's top-32 read and the L4's exact read agreed on 1,186 and 1,193 of 1,200 answers.

So, Which One?

For a Jev-style decision service on a TPU, Gemma 4 12B at bf16 plus one temperature fitted on about 50 labels. It matches the 26B on the suite, 76.2% against 76.0%, though the 26B scores 6.3 points higher on irony; it reaches a median ECE of 0.070 after fitting, decides in 26 to 27 ms, and needs no third-party quantized checkpoint. Check label_mass on your own task first: on irony, 12B put a median of 3.1% of its probability on the labels.

Choose the TPU when time per decision matters or the rest of the stack is already on Google Cloud: it halves the 26B's time per decision against the L4. Choose the L4, or Jev itself, when cost per decision matters: on demand, 12B on one v6e chip costs $8.19 per million decisions, against Jev's $5.54 and at most $5.43 for the 26B on an L4.

What Does It Cost?

Measured from the request times at eight in flight, one v6e chip answers about 274 decisions a second with E2B, 206 with E4B, 101 with 12B and 80 with 26B-A4B. At the europe-west4 on-demand price of $2.97 an hour that is $3.01, $4.00, $8.19 and $10.37 per million decisions; at the flex-start price of $1.35, $1.37, $1.82, $3.72 and $4.71. TypeSafe prices Jev at $5.54 per million decisions at the L4 run's median prompt of 132 tokens. The chip is charged by the hour whether busy or idle, and eight requests in flight is below the 16 the server allows, so these figures hold only while the chip is kept at least this busy.

The measured run took 48 minutes of on-demand time, $2.39, and all instance time for the measurement came to at most $3.25.

Teardown

The VM deletes itself when tpu/run.sh finishes, and --max-run-duration 6h with --instance-termination-action DELETE is the backstop. Confirm it is gone:

gcloud compute instances describe jev-tpu-v6e1 --zone europe-west4-a

ERROR: (gcloud.compute.instances.describe) Could not fetch resource:
 - The resource 'projects/aisprint-491218/zones/europe-west4-a/instances/jev-tpu-v6e1' was not found

Summary

The goal of this article was to run a Jev-style decision model on one TPU v6e chip with Gemma 4 and find what changes from a GPU. The key to the solution was the same prompts, labels and scoring as the L4 run, a pre-registered design, and a label read taken from the top 32 log-probabilities. The results were:

🟢 One v6e chip serves Gemma 4 E2B, E4B and 12B at bf16 and a 26B-A4B fp8 build
❌ No 31B checkpoint loads: the w4a16 and 4-bit builds fail on the JAX path, and the fp8 build is 30.98 GiB
🟢 The same checkpoints give the same answers on the TPU and the L4: 1,186 and 1,193 of 1,200
🟢 The 26B decides in 27 to 33 ms on the TPU against 61 ms on the L4
⚠️ On demand, 26B costs $10.37 per million decisions on the TPU against at most $5.43 on the L4 and $5.54 for Jev
⚠️ vLLM on TPU returns only the top-k log-probabilities; reading labels from the top 32 changes no answer, and changes four-task calibration by at most 0.005
🟢 12B matches the 26B: 76.2% and 76.0% on the 3,880-record suite, against Jev's 77.3%
🟢 After 50 labels, 12B and 26B-A4B reach a median ECE of 0.070, against Jev's 0.071 as shipped
🟢 The whole measurement cost at most $3.25 of instance time

Scope: one TPU v6e chip (ct6e-standard-1t, on demand) in europe-west4-a, vLLM 0.29.1rc1.dev468+g0b7f11a1e at the image digest above, E2B, E4B and 12B with bf16 weights and 26B-A4B through RedHatAI's fp8 build, every model with vLLM's default fp8 KV cache on v6e, --max-model-len 2048, one run per model, 300 examples per task plus the 3,880-record public suite, client on the VM. The top-32 read and on-demand capacity are dated deviations from the pre-registration, and the top-32 bound check, added after the run, covers the four tasks only. L4 figures come from the companion run on a g6.xlarge and g6.4xlarge in us-east-1, and the two platforms were timed with different clients. Every source dataset was published before Gemma 4 and may be in its training data. No Jev call was made: the Jev figures are Bespoke Labs' published results on the same records, from one run of Jev 1.13.0 by a company that sells a competing model, counting an invalid Jev response as wrong, with Jev's probabilities rounded to two decimals by its API. Parts of the analysis and writing were done with AI assistance (Claude); every figure comes from the committed output files.

The strategy for running a Jev-style decision model on TPU with Gemma 4 was validated with an incremental step by step approach.

References

Code, pre-registration and per-item results: https://github.com/xbill9/gemma4-dev/tree/main/jev-tpu
Companion L4 measurement: https://dev.to/gde/plain-gemma-4-26b-vs-jev-on-one-ec2-l4-21-points-behind-overall-level-on-yesno-45-behind-on-15k6
Companion review of the independent evidence on Jev: https://dev.to/gde/jev-after-eight-days-of-independent-tests-level-with-mid-price-llms-behind-the-frontier-1kln
vLLM TPU documentation: https://docs.vllm.ai/projects/tpu/en/latest/
tpu-inference: https://github.com/vllm-project/tpu-inference
26B-A4B fp8 checkpoint: https://huggingface.co/RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic
Gemma 4 12B: https://huggingface.co/google/gemma-4-12B-it
Bespoke Labs, public-suite results for Jev 1.13.0: https://github.com/bespokelabsai/nimble/blob/0e67403/docs/PUBLIC_BENCHMARKS.md
Cloud TPU v6e: https://cloud.google.com/tpu/docs/v6e
Guo et al., On Calibration of Modern Neural Networks: https://arxiv.org/abs/1706.04599

What Nobody Is Using in Your Azure Subscriptions, and What It Costs

xbill — Thu, 24 Sep 2026 15:32:30 +0000

This article provides a step by step guide to building an Azure waste scanner from source, running it across every subscription your az login can see, pricing each finding from the Azure Retail Prices API, and drafting the cleanup. A suite of Python checks is built to cover Compute, Networking, Storage, SQL, Key Vault, Container Registry, App Service, Container Apps, AI Services, Machine Learning, Log Analytics and AKS.

https://github.com/xbill9/zombiescan-azure

The Third Cloud

This is the third scanner in a series. The first covered an AWS account and the second covered Google Cloud projects:

The shape carries over: a CLI, a bundled price table, a cleanup plan that is printed and never run, and a Claude Code plugin with an MCP server over the same engine. What changes is everything underneath — how Azure authenticates, how it answers a list call, how it prices a disk, and which resources keep billing once nobody uses them.

What Gets Left Behind

A managed disk survives the VM it was attached to. A public IP outlives the load balancer that held it. A NAT gateway keeps its hourly fee after the last subnet moved away, and a provisioned model deployment bills every PTU every hour whether a request arrives or not. Each one reports nothing.

Cost Management shows the total. Azure Advisor surfaces candidates. A figure that drives a decision names one resource, in one subscription and resource group, and its monthly cost.

zombiescan produces that figure for 30 classes of resource across two packs.

At This Point You Should Have…

An Azure account and the Azure CLI, signed in with az login
uv on the path, and Python 3.11 or newer
Read access to the subscriptions you mean to scan — the built-in Reader role covers every call, and zombiescan-scanner-role.json, under policy/ in the repository, is a custom role holding exactly the read actions the checks use

Step 1 — Build It From Source

git clone https://github.com/xbill9/zombiescan-azure
cd zombiescan-azure
uv sync

Resolved 18 packages in 0.48ms
Checked 17 packages in 0.13ms

uv sync reads the committed uv.lock, so the resolved set is the one the tests ran against. The dependency list is click and rich; every Azure call goes over HTTPS with urllib from the standard library.

uv run zombiescan --version

zombiescan, version 0.1.0

Step 2 — Install It as a Tool

Working from the clone keeps uv run in front of every command. Installing it puts zombiescan on the path instead.

uv tool install git+https://github.com/xbill9/zombiescan-azure

Both routes run the same engine. The rest of this article uses the installed form.

Step 3 — Authenticate, and See Which Account You Are

az login

zombiescan shells out to az account get-access-token and reuses the result. No key file is downloaded, no client secret is stored, and there is no azure-identity dependency.

An ARM token is issued for exactly one tenant. Microsoft lets one email address be both a work or school account and a personal Microsoft account, and those are two directories with two sets of subscriptions. So the scanner reads the subscription list from az account list --all, which spans every identity the CLI has signed into, and holds one token per tenant.

Every scan opens by naming each tenant, the kind of account signed into it, and each subscription:

Scanning as xbill@glitnir.com
Tenant Default Directory — personal Microsoft account
  domain        xbillglitnircom.onmicrosoft.com
  tenant id     40482c55-d00d-4c6d-8903-643d76a74b9c
  signed in as  xbill@glitnir.com
  subscription  Azure subscription 1 (default)
                3db3ce66-50b6-4d11-91ef-5950cf4039ed

az account list reports a personal account and a work account as the same user with the same email. The access token tells them apart: a personal Microsoft account signs in through live.com and its token carries idp: live.com, a work or school account in its own directory carries no idp at all, and a guest from another directory names its home issuer there.

🔎 Tip: every token is fetched on the main thread before any worker starts. Concurrent az account get-access-token processes contend on the MSAL token cache in ~/.azure, and a corrupted cache costs an az login.

Step 4 — List the Checks

zombiescan checks

  aks-idle-cluster  AKS clusters running no nodes  (aks)
  deallocated-vm  Stopped VMs still paying for disks  (core)
  disabled-key-vault-key  Disabled Key Vault keys still billed  (core)
  empty-ai-services-account  AI Services accounts with no model deployed  (core)
  empty-container-apps-environment  Container Apps environments with no app  (core)
  empty-container-registry  Container registries holding no images  (core)
  empty-resource-group  Resource groups holding nothing  (core)
  empty-vnet  Virtual networks with nothing running in them  (core)
  idle-app-service-plan  App Service plans hosting no apps  (core)
  idle-container-app  Always-on container apps serving nothing  (core)
  idle-dedicated-host  Dedicated hosts running no VMs  (core)
  idle-load-balancer  Load balancers with no backends  (core)
  idle-ml-compute  ML compute running or held idle  (core)
  idle-nat-gateway  NAT gateways with no subnets  (core)
  idle-provisioned-deployment  Provisioned model deployments serving nothing  (core)
  idle-workload-profile  Dedicated workload profiles running no app  (core)
  orphaned-nic  Network interfaces with no VM  (core)
  orphaned-snapshot  Snapshots of deleted disks  (core)
  paused-sql-database  Paused SQL databases still paying for storage  (core)
  stale-key-vault-secret  Secrets with no new version in 90 days  (core)
  unattached-disk  Unattached managed disks  (core)
  unbounded-log-workspace  Log workspaces with no ingestion cap  (core)
  unmanaged-storage-account  Versioned storage accounts with no lifecycle policy  (core)
  unused-availability-test  Availability tests watching a deleted resource  (core)
  unused-capacity-reservation  Capacity reservations holding unused slots  (core)
  unused-dns-zone  DNS zones with no records  (core)
  unused-image  Managed images nothing boots from  (core)
  unused-nsg  Security groups protecting nothing  (core)
  unused-public-ip  Public IP addresses attached to nothing  (core)
  unused-subnet  Subnets reserving a range against nothing  (core)

Twenty-nine ship in the core pack and one in aks. AKS sits in its own pack because it carries its own resource provider, its own rate section and its own price fetcher, which is what a third-party pack has to supply.

Step 5 — Which Resource Providers the Checks Read

zombiescan providers

  Microsoft.App  3 check(s)
  Microsoft.CognitiveServices  2 check(s)
  Microsoft.Compute  6 check(s)
  Microsoft.ContainerRegistry  1 check(s)
  Microsoft.ContainerService  1 check(s)
  Microsoft.Insights  3 check(s)
  Microsoft.KeyVault  2 check(s)
  Microsoft.MachineLearningServices  1 check(s)
  Microsoft.Network  9 check(s)
  Microsoft.OperationalInsights  1 check(s)
  Microsoft.Resources  1 check(s)
  Microsoft.Sql  1 check(s)
  Microsoft.Storage  1 check(s)
  Microsoft.Web  1 check(s)

This list decides more on Azure than on the other two clouds. ARM answers a list call against an unregistered resource provider with HTTP 200 and an empty page. A subscription that has never used App Service returns {"value": []} from /providers/Microsoft.Web/serverfarms, with no error. A check that simply ran would find nothing and report a clean subscription.

So every check declares the providers it reads, the engine reads each subscription's registrations once, and a check whose provider is missing is counted as skipped. The same declaration generates the read-only custom role, and the test suite fails when a check names a provider the role does not cover.

Step 6 — Scan One Subscription

zombiescan scan

1 subscription(s), 30 check(s) — read-only

No waste found across 1 subscription. Nothing to clean up.

4 of 30 subscription/check pair(s) were skipped: their resource provider is not registered, so there
is nothing of that kind here. 'zombiescan providers' lists what each check reads.

With no flag it scans the subscription az is set to. The four skipped pairs are Key Vault's two checks, App Service and SQL: this subscription has never registered those providers, so it has no resources of those kinds.

Every call is a list, a get or a Resource Graph query. The scan has no code path that deletes, modifies or releases anything.

Step 7 — Scan Every Subscription, in Every Tenant

zombiescan scan --all-subscriptions

--all-subscriptions takes every enabled subscription in az account list --all, across every tenant the CLI has signed into, and runs az account list --refresh first to pick up subscriptions created since the last login. On this account that is one subscription in one tenant, and the sweep of 30 checks took 8.65 seconds.

A second identity is one az login away. After it, the same command reaches both directories, requests for each subscription carry that tenant's token, and the header lists both tenants with their account kinds.

Step 8 — What a Finding Looks Like

This subscription is clean, so the rows below come from the test suite's recorded ARM responses — the same JSON each check is tested against — priced from the real bundled price table.

uv run python articles/zombiescan-azure/show-fixture-findings.py

49 findings from 30 checks, recorded responses, prices generated 2026-09-23T17:19:30Z
total $32,345.26/month

Check                              Resource                           Monthly
idle-provisioned-deployment        acct-ptu/ptu-idle              $21,900.00
idle-ml-compute                    ws-research/gpu-warm            $4,467.60
idle-dedicated-host                host-idle                       $3,084.25
idle-workload-profile              env-single/d4-solo                $522.62
idle-app-service-plan              prod-plan                         $459.90
unused-capacity-reservation        cr-partial                        $420.48
empty-container-apps-environment   env-empty-dedicated               $297.81
idle-workload-profile              env-mixed/d4-idle                 $224.81
idle-ml-compute                    ws-research/ci-forever            $213.89
unused-capacity-reservation        cr-idle                           $140.16
aks-idle-cluster                   scaled-to-zero                     $73.00
idle-nat-gateway                   egress-gw-old                      $32.85
deallocated-vm                     batch-runner                       $24.90
idle-load-balancer                 api-lb                             $21.90
unattached-disk                    orphan-data                        $19.71
idle-container-app                 app-idle                           $11.83
orphaned-nic                       web-01-nic-old                      $3.65
unattached-disk                    tiny-scratch                        $0.60

The top of that list is where Azure's waste concentrates now: a 15-PTU regional model deployment serving nothing is $21,900 a month, a GPU cluster whose minimum keeps two idle nodes up is $4,467.60, and an empty DSv3-Type3 dedicated host is $3,084.25. A forgotten disk is $19.71.

Every finding carries its subscription, resource group, location and full ARM id. No az command works without the resource group, and the ARM id is the only identifier unique across a tenant.

Step 9 — Where a Check Runs

A check runs once per subscription. Two properties of ARM make that a complete sweep.

A list call at /subscriptions/<id>/providers/Microsoft.Compute/disks returns every disk in every resource group and every region, in one call. And Azure Resource Graph answers a cross-type join in one KQL query, so the check for stopped VMs reads power state for every VM at once without an instance-view call per machine.

So @check has no region scope. Each finding reads its own location off the resource, and --location filters findings afterwards.

🔎 Tip: follow nextLink even when the first page is empty. Listing AI Services accounts across a subscription returned an empty first page with a nextLink, and the account on the second page. A reader that stopped at page one would report the subscription clean.

Step 10 — Pinned API Versions

Azure has no "latest". Every request carries a required api-version, and azure.API_VERSIONS pins one per resource type, resolved by longest prefix so a sub-type inherits its parent's version.

A retired version produces InvalidResourceType, which is 404-shaped, and a 404 reads as "nothing here". The check would stop reporting and the subscription would look that much cleaner. The live test, gated behind ZOMBIESCAN_LIVE=1, sends every pin to ARM and fails on the first one ARM rejects.

Step 11 — Where the Prices Come From

The bundled price table is generated from the Azure Retail Prices API, which is public: no credentials and no subscription.

python -m zombiescan.pricing.refresh

A scan reads table.json from the installed package, so it runs offline and adds no latency. Rates are looked up by key and region: ctx.pricing.rate("disk.tier_month", region=..., variant="P10 LRS").

Verified rates in eastus, from the bundled table:

Rate	eastus
Standard static public IP	$0.005/hour
NAT Gateway	$0.045/hour
Standard Load Balancer rules	$0.025/hour
AKS Standard control plane	$0.10/hour
Dedicated host `DSv3-Type3`	$4.225/hour
`Standard_D2s_v3`, Linux	$0.096/hour
Provisioned throughput, regional	$2.00 per PTU-hour
Provisioned throughput, global	$1.00 per PTU-hour
Container Apps Dedicated management fee	$0.10/hour

At 730 hours to the month, arithmetic over those hourly rates gives $3.65 for an idle public IP, $32.85 for a NAT gateway, $18.25 for load balancer rules, $73.00 for an AKS Standard control plane and $3,084.25 for an empty dedicated host.

Step 12 — Four Ways a Price Goes Wrong

Each of these produces a plausible table with the wrong numbers in it.

Filter on priceType eq 'Consumption'. The same meter is published as Reservation and DevTestConsumption too, at a fraction of the price.

Take the first tier that charges. Azure returns a tiered meter's rows in no guaranteed order, and tier 0 is often a free allowance. unit_price() sorts by tierMinimumUnits and reads the first nonzero row.

Match the meter name as well as the SKU. P80 LRS Disk, P80 LRS Disk Mount and P80 LRS Disk Operations all share the SKU "P80 LRS"; only the first is the capacity charge. The same care covers Windows rows: most VM products spell it "Windows", and a few GPU series abbreviate it to "Win".

Some meters have no ARM region. NAT Gateway and Load Balancer are published against "Global", and Azure DNS against a billing geography spelled "Zone 1". Those live in a global section.

The refresher refuses to write a table that loses or empties a section the previous one had, because an empty section means a matcher stopped matching.

Step 13 — Managed Disks Are Priced by Tier

A managed disk bills at the tier its provisioned size falls into, flat per month:

Disk	Tier	eastus per month
1 GiB Premium SSD	P1	$0.60
65 GiB Premium SSD	P10	$19.71
128 GiB Premium SSD	P10	$19.71
4 GiB Standard HDD	S4	$1.54
32 GiB Standard HDD	S4	$1.54

A 65 GiB Premium disk and a 128 GiB one cost the same. Standard HDD has no rung below S4, so a 4 GiB Standard disk bills as a 32 GiB one. The price table is keyed by tier and redundancy — "P10 LRS", "P10 ZRS" — because zone redundancy costs about half as much again: $29.57 for a P10 ZRS.

Premium SSD v2 and Ultra bill per provisioned GiB, and both bill provisioned IOPS and throughput on top, which the finding says.

Step 14 — Idle Needs a Week of Metrics

Most checks answer from one list call. Two need usage:

idle-provisioned-deployment reads ModelRequests on each account over seven days, split by ModelDeploymentName, so one call covers every deployment on the account
idle-container-app reads Requests on each app with minReplicas of one or more

Both go through helpers.metric_totals, which asks Azure Monitor for the Total over P7D and returns the sum. A deployment that served nothing has no series at all, and a missing series reads as zero. Both checks declare Microsoft.Insights, the provider behind the metrics endpoint, so the registration check covers it too.

On this subscription the container app recorded 0 requests in the seven days, and the gpt-5-mini deployment returned no series. Both stay out of the report: the app has minReplicas: 0 and the deployment is pay-per-token, so each costs nothing idle.

Step 15 — JSON Output

zombiescan scan --all-subscriptions --json findings.json

{
  "schema_version": 4,
  "scan": {
    "generated": "2026-09-23T17:47:01Z",
    "duration_seconds": 8.65,
    "principal": "xbill@glitnir.com",
    "subscriptions": ["3db3ce66-50b6-4d11-91ef-5950cf4039ed"],
    "pairs_attempted": 30,
    "pairs_unavailable": 4,
    "complete": true
  },
  "pricing": {
    "generated": "2026-09-23T17:19:30Z",
    "basis": "Azure Retail Prices API, pay-as-you-go USD list prices",
    "excludes": ["reservations", "savings plans", "Azure Hybrid Benefit", "dev/test rates", "enterprise agreement pricing", "credits"]
  },
  "totals": {
    "monthly_cost": 0,
    "annual_cost": 0,
    "finding_count": 0,
    "by_check": {}
  }
}

The pricing block dates the table and lists what list prices exclude, so a report read six months later says which rates produced it. pairs_unavailable carries the skipped count, and complete is false when every pair failed or was skipped, so a CI job reading the file can tell an all-clear from a scan that reached nothing.

Step 16 — The HTML Report

zombiescan scan --all-subscriptions --html report.html

One self-contained file with the styles inline, which prints to PDF without fetching anything. With no findings its headline reads "No waste found" and names how many subscription/check pairs ran, how many were skipped for an unregistered provider, and how many failed.

Step 17 — The Cleanup Plan

zombiescan scan --all-subscriptions --script cleanup.sh

#!/usr/bin/env bash
# zombiescan cleanup plan — generated 2026-09-23T17:47:01Z
#
# READ EVERY LINE BEFORE RUNNING THIS.
# zombiescan generated this file and did not run it. Some of these
# deletions can be undone and some cannot: Key Vault and SQL keep a
# recovery window, a released public IP address is gone for good.

Every generated command is built by one helper, which adds --resource-group and --subscription and wraps interpolated names in shell quoting. The test suite checks both at the source.

🔎 Tip: --yes exists only on the az commands that would otherwise prompt, and passing it to one that does not is an error. az disk delete takes it; az snapshot delete and az network nic delete reject it. The helper holds the list of commands that take it, and the live test re-checks that list against the installed CLI.

Step 18 — Cleaning Up

clean is a separate command, and a dry run is what it does with no flags.

zombiescan clean --all-subscriptions

Nothing to clean.

--apply gates one thing: whether a planned step is sent to Azure. It never changes which steps get planned, so the preview is what runs.

Planning is read-only. A cleaner may read — re-listing a resource group to confirm it is still empty, re-reading a model deployment's request count, re-reading a capacity reservation's allocation — and it yields the mutations as objects for the runner to send. A plan refuses when the resource has changed since the scan: a host with a VM placed on it, an account that gained a deployment, a reservation that filled up.

Where the API allows a backup, it comes first. The disk cleaner takes an incremental snapshot before the delete, and a failed step stops the rest of that finding.

IRREVERSIBLE marks a step with no recovery window at all. A released public IP is gone, and so is a deleted Container Apps environment's static IP, so both carry the mark. A Key Vault key is held by mandatory soft-delete, a SQL database restores from point-in-time backups, and a deleted AI Services account is recoverable for 48 hours, so none of those does.

A check with no cleaner reports its findings as unsupported with a reason. idle-container-app is one: lowering minReplicas trades the idle charge for a cold start on the next request, and that trade belongs to whoever owns the app.

Step 19 — The Read-Only Role

az role definition create --role-definition policy/zombiescan-scanner-role.json

Forty-four actions: forty-three reads and Container Registry's listUsages, the one management-plane call that reports how much a registry stores. The file is generated from the providers= each check declares, and the suite keeps it in step.

clean needs more than this, deliberately: the identity that reports waste should be unable to delete what it reports.

Step 20 — The Claude Code Plugin

/plugin marketplace add xbill9/zombiescan-azure

The plugin ships two slash commands, a skill and an MCP server over the same engine.

  scan_subscription  Scan Azure subscriptions for unused resources and price them
  estimate_savings  Total, count and break down the findings in a report
  explain_finding  Explain why a check treats a resource as waste
  list_checks  List the installed checks
  plan_cleanup  Show exactly what `zombiescan clean` would do

Every tool is read-only. plan_cleanup builds the step objects and stops there; the function that sends them to Azure is unreachable from the server, and a test asserts the module names no other clean.* attribute.

The tools return computed figures — totals, counts, breakdowns, cheapest and costliest — and echo the filter they applied. A tool that returned rows for the model to add up would move the arithmetic to the place least able to do it.

Step 21 — What a Scan Cannot See

A clean scan answers one question: is anything billing that nothing uses? It says nothing about a project that has gone quiet as a whole. This subscription holds one resource group, research-mesh-rg, with six resources: a container app and its environment, a Foundry account with one project and a gpt-5-mini deployment, a container registry and a Log Analytics workspace.

Cost Management reports what the group was charged, and the totals below are Azure's own:

Resource	Last 30 days	Since Aug 13
`researchmeshacr`, Container Registry Basic	$5.08	$6.87
`research-mesh-foundry`, gpt-5-mini tokens	$0.008	$0.25
Log Analytics workspace	$0.00	$0.00
Container app and environment	no charges	no charges

$5.09 over the last 30 days, and the registry is 99.8% of it. Its Basic tier bills a flat daily fee whether anyone pulls an image or not. Everything else bills on use, and use is close to zero: the app served 0 requests in seven days and the deployment under a cent of tokens in thirty.

None of the six trips a check, because each is in use by another: the registry holds the image the app runs, the account has a deployment, and the environment has an app. Finding a group like this takes Cost Management data, which is a different source from anything a scan reads.

Three Clouds, Same Resource

The same idle resource costs differently on each cloud. Figures are from each article's bundled price table.

Resource	AWS (us-east-1)	Google Cloud (us-central1)	Azure (eastus)
Idle NAT gateway	🥈 $32.85/month flat	🥇 the addresses it holds, $3.65 each	🥈 $32.85/month flat
128 GB disk, attached to nothing	🥇 $10.24, gp3 per GB	🥈 $12.80, pd-balanced per GB	🥉 $19.71, P10 by tier
Managed Kubernetes, no nodes	—	🥈 $73.00 management fee	🥇 $0.00 on Free, $73.00 on Standard
Unit a check fans out over	region	project	subscription
Price source	AWS Price List API	Cloud Billing Catalog API	Azure Retail Prices API, no credentials

Two checks exist only on Azure. Orphaned NICs, because a network interface is a resource of its own that outlives its VM and blocks deleting the public IP, subnet and virtual network it references. And empty resource groups, because a resource group is a container inside a subscription.

🔎 Tip: an Azure public IP bills the same rate attached or idle, so nothing in the price signals that it is unused. On Google Cloud an idle static IP costs more than one in use.

What the Checks Leave Alone

A false positive here costs an outage, so each check states the condition it declines to act on.

Condition	Treatment
Public IP held by an orphaned NIC, a deallocated VM or an idle load balancer	Priced on the holder's finding, counted once
Disk reserved by a stopped VM	Reported under `deallocated-vm`, counted once
Disk in the `ActiveSAS` state	Something is reading it; left alone
Snapshot whose source disk still exists	Load-bearing; left alone
AI Services account with a project but no deployment	A project can use models hosted elsewhere; left alone
Container app with no ingress	A worker processes queues, so a zero request count proves nothing; left alone
Resource provider not registered	Skipped and counted; a service never used holds no waste

Compare and Contrast

	zombiescan	Azure Advisor	Cost Management	Cross-subscription SaaS
Per-resource dollar figure	🥇 yes	partial	per resource, after the fact	yes
Credentials leave the machine	🥇 never	n/a, Azure-side	n/a, Azure-side	❌ service principal granted
Deletes on request	🥈 opt-in, dry run first	❌ no	❌ no	🥇 yes
Runs offline after install	🥇 yes	❌ no	❌ no	❌ no
Breadth	30 checks	broader	everything billed	broader

So, Which One?

Cost Management answers what a subscription spent, down to the resource, and it is the only source that sees a quiet project like research-mesh-rg. Advisor covers more ground, including right-sizing judgements this tool leaves alone.

zombiescan fits the case where the answer has to be per-resource, priced, and produced without granting anything access to the subscriptions. A laptop, an existing az login, and under ten seconds for this subscription.

Cost

A scan costs nothing. ARM list and get calls, Resource Graph queries and Azure Monitor metric reads carry no charge, and the price table ships with the package.

Regenerating the table calls the Azure Retail Prices API, which is public and free.

Tests

uv run pytest -q

380 passed, 10 deselected in 0.34s

The suite runs offline against recorded ARM responses, with no credentials. The ten deselected tests reach a real subscription and run under ZOMBIESCAN_LIVE=1: they verify every pinned api-version against ARM and every --yes command against the installed CLI, two facts about the outside world that a comment cannot keep true.

Recorded responses are keyed by the tail of the request path, and they include resources that are in use, because half of what these checks do is decline to report.

Teardown

uv tool uninstall zombiescan-azure

Nothing is left in any subscription. The tool creates no service principal, no role assignment, no storage account and no stored state. Sign the CLI out with az logout.

Summary

The goal of this article was to audit every Azure subscription a set of az credentials can reach and attach a monthly cost to each unused resource. The key to the solution was holding one token per tenant, checking provider registration before trusting an empty page, and pricing every finding from the Azure Retail Prices API while keeping the scan read-only. The results were:

🟢 30 checks in two packs across Compute, Networking, Storage, SQL, Key Vault, Container Registry, App Service, Container Apps, AI Services, Machine Learning, Log Analytics and AKS
🟢 Every tenant the CLI has signed into, from one az login each, with the account kind — personal or work — named in the header
🟢 One call per subscription: ARM lists are subscription-wide and Resource Graph joins resource types in one query
🟢 Unregistered providers skipped and counted, so an empty page never reads as a clean subscription
🟢 Disks priced by tier and redundancy, and the priciest classes — provisioned model throughput, GPU clusters, dedicated hosts — priced from their own meters
🟢 JSON against a published schema, a self-contained HTML report, and a shell cleanup plan that is printed and never run
🟢 clean previews by default, re-reads each resource before planning, and marks steps with no recovery window
⚠️ The subscription scanned here is clean: 0 findings, with 4 of 30 pairs skipped for unregistered providers, so the per-resource figures come from the recorded test responses
⚠️ Figures are pay-as-you-go list prices, excluding reservations, savings plans, Azure Hybrid Benefit and credits
⚠️ Two checks judge idleness from seven days of Azure Monitor metrics, which is one window among several reasonable ones
❌ A resource group that has gone quiet as a whole trips no check; seeing it takes Cost Management data

Scope: one personal Microsoft account with one tenant and one subscription, 30 checks, a single sweep per figure quoted, run from one laptop. Rates are eastus from a table generated 2026-09-23; the AWS and Google Cloud figures come from those articles' own price tables. The read-only guarantee holds for the packs in this repository, which are reviewed; a pack installed from PyPI runs with the same credentials and carries no such guarantee.

The strategy for using per-resource pricing for Azure waste detection was validated with an incremental step by step approach.

References

Repository, MIT: https://github.com/xbill9/zombiescan-azure
Findings JSON Schema: https://github.com/xbill9/zombiescan-azure/blob/main/docs/findings.schema.json
Pack author guide: https://github.com/xbill9/zombiescan-azure/blob/main/docs/PACKS.md
Part one, AWS: https://dev.to/aws-builders/find-the-aws-resources-nobody-is-using-and-what-they-cost-you-35bj
Part two, Google Cloud: https://dev.to/gde/what-nobody-is-using-in-your-google-cloud-projects-and-what-it-costs-1k0
Azure Retail Prices API: https://learn.microsoft.com/en-us/rest/api/cost-management/retail-prices/azure-retail-prices
Azure Resource Graph: https://learn.microsoft.com/en-us/azure/governance/resource-graph/overview
Resource providers and registration: https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/resource-providers-and-types
Managed disk pricing: https://azure.microsoft.com/en-us/pricing/details/managed-disks/
Capacity reservation billing: https://learn.microsoft.com/en-us/azure/virtual-machines/capacity-reservation-overview
Dedicated host pricing: https://learn.microsoft.com/en-us/azure/virtual-machines/dedicated-hosts