DEV Community: xbill

Six Serving Engines on One EC2 G6: Gemma 4 E2B on an NVIDIA L4

xbill — Fri, 06 Nov 2026 18:41:59 +0000

This article provides a step by step comparison of six inference serving engines on one Amazon EC2 G6 instance: llama.cpp, vLLM, SGLang, mistral.rs, a plain PyTorch engine and Google's LiteRT-LM, each serving Gemma 4 E2B on Google's QAT 4-bit weights from one NVIDIA L4.

https://github.com/xbill9/gemma4-dev

The question is which engine to put on a G6 when one small model has the whole GPU. Each engine gets the same weights, the same prompts, the same 1-to-64-request load sweep and the same 1,319 GSM8K and 400 BFCL questions, and each is measured with and without Google's E2B drafter wherever the engine supports one. Every engine ran on one instance, one at a time, with the card to itself.

The Results in Brief

Answer quality barely moves between engines. GSM8K lands between 0.8954 and 0.9014 on the five CUDA engines with the drafter on or off, and every paired difference against llama.cpp is within the run-to-run range. LiteRT-LM scores level on GSM8K and lower on BFCL.

For one request, five engines are within 8% of each other without a drafter (134.7 to 144.6 tokens per second end to end) and LiteRT-LM runs at 0.62 to 0.73x llama.cpp. The drafter decides the order: it gives vLLM 1.96x, llama.cpp 1.53x, SGLang 1.39x and LiteRT-LM 1.15x, and it slows mistral.rs to 0.95x.

Under load the batching engines separate. At 64 requests of 128 tokens in and 512 out, vLLM reaches 3,716 tokens per second without a drafter and 4,516 with it, SGLang 2,967, mistral.rs 1,825 and llama.cpp 1,056. PyTorch and LiteRT-LM serve one request at a time.

With each engine at its best setting, vLLM is the fastest at every level from 1 to 64 requests, on both shapes, and its drafter never loses. At the g6.4xlarge on-demand price that is $0.08 per million output tokens at full load.

What Changes and What Stays

Engine	Version	Weights	Installed with
llama.cpp	`7182ef71e`	exact Q4_0 GGUF	CMake, CUDA 13.1, sm_89
vLLM	0.33.0	W4A16, int4 tables	pip, its own `torch 2.14.0+cu130`
SGLang	0.5.23	W4A16, int4 tables	pip, cp313 wheel on Python 3.14, two patches
mistral.rs	v0.9.5	exact Q4_0 GGUF	`cargo install`, CUDA 13.1, sm_89
PyTorch	this repo's int4 engine	W4A16	the DLAMI's torch 2.13
LiteRT-LM	0.19.0	exact Q4_0, and Google's int2/4/8 file	pip, GPU through WebGPU on Vulkan

The same for all six: one g6.4xlarge on-demand in us-east-1d (16 vCPUs, 64 GiB, one NVIDIA L4 with 23,034 MiB, driver 595.112.07), the PyTorch 2.13 / Ubuntu 26.04 Deep Learning AMI, Gemma 4 E2B on Google's QAT Q4_0 grid in every container, greedy decoding, and the system Python 3.14 with no virtual environment. The exact Q4_0 GGUF and the W4A16 checkpoint carry the same 4-bit values: xbill9/gemma-4-E2B-it-qat-q4_0-exact-gguf for llama.cpp and mistral.rs, xbill9/gemma-4-E2B-it-qat-q4_0-w4a16-ct-text-emb4 for vLLM and SGLang.

At This Point You Should Have…

An AWS account with G-family vCPU quota in us-east-1 and an instance profile with SSM access
An S3 bucket for moving files to and from the instance
The system python3 with boto3, and the MCP server in gpu-pytorch-g6-2b-q4w4a16/
A clone: git clone https://github.com/xbill9/gemma4-dev

Step 1 — Find a G6 With Capacity

The rigs name a g6.2xlarge (8 vCPUs, one L4). On 2026-11-05 it had no spot or on-demand capacity in any us-east-1 zone, and neither did g6.4xlarge spot. The MCP tool create_g6_instance walks every subnet, spot first and then on-demand:

subnet-0c2872fe4182b9ec1 spot=True: ❌ AWS InsufficientInstanceCapacity: There is no Spot capacity available that matches your request.
subnet-09e0f13c0a5b43092 spot=True: ❌ AWS InsufficientInstanceCapacity: There is no Spot capacity available that matches your request.
subnet-0c2872fe4182b9ec1 spot=False: ❌ AWS InsufficientInstanceCapacity: We currently do not have sufficient g6.4xlarge capacity in the Availability Zone you requested (us-east-
subnet-0d2b2038965e8d0c0 spot=False: ✅ Launching `i-040ed957b67c2d514` (g6.4xlarge, on-demand, 1x L4) in `us-east-1`.
AMI: `ami-0f5139ed9ed52b805`

Both sizes carry the same L4. The g6.4xlarge doubles the vCPUs and memory, which only llama.cpp's prompt threads and the build steps use.

🔎 Tip: all access is SSM and S3. The instance has no SSH key and its role has no S3 access. Commands go through SSM Run Command, which returns at most 24,000 characters, so files larger than that move through presigned S3 URLs generated on the workstation.

Step 2 — One Rig Directory per Engine

Each engine has its own directory with a tpu.env that decides every serving flag, a Makefile run on the instance, and its measurements under benchmarks/runs/:

gpu-llamacpp-g6-2b-q4_0/
gpu-vllm-g6-2b-q4w4a16emb4/
gpu-sglang-g6-2b-q4w4a16emb4/
gpu-mistralrs-g6-2b-q4_0/
gpu-pytorch-g6-2b-q4w4a16/
gpu-litertlm-g6-2b-int248/
gpu-litertlm-g6-2b-q4w4a16emb4/

gpu-vllm-g6-2b-q4w4a16emb4/flight/setup.sh installs every engine in order and logs each step, so a failed step reruns alone:

GEMMA4_DEV=/opt/gemma4-dev MODEL_S3_URI=s3://... bash setup.sh llamacpp vllm sglang mistralrs litert
18:48:44 START llamacpp
18:56:56 END llamacpp
18:56:56 START vllm
18:59:18 END vllm
18:59:18 START sglang
18:59:57 END sglang

Step 3 — What the Deep Learning AMI Needs

The AMI ships the NVIDIA driver and a PyTorch 2.13 environment, and no CUDA toolkit. Four things differ from a desktop install:

vLLM brings its own torch. The system Python has no torch, so pip install vllm==0.33.0 pulls its pinned wheel from PyPI, built for CUDA 13.0, and it runs on driver 595.112.07:

torch 2.14.0+cu130 cuda 13.0 NVIDIA L4 (8, 9)

llama.cpp and mistral.rs build against CUDA 13.1 from NVIDIA's apt repository, for sm_89. mistral.rs also links cuRAND, which is a separate package: libcurand-dev-13-1.
LiteRT-LM finds the L4 through Vulkan. The AMI lists the L4 as a Vulkan device, and LiteRT-LM's WebGPU backend serves on it.
pip puts scripts in ~/.local/bin. SSM runs commands as root with HOME=/root, and that directory is off its PATH, so vllm, litert-lm and SGLang's ninja are found only once it is added.

🔎 Tip: SGLang 0.5.23 needs two patches for this checkpoint. Its Gemma 4 attention shim runs only for multimodal configs, and it leaves the int4 embed_tokens and per-layer table as uninitialized bf16. The rig's patches/ keep both tables packed on the card.

Step 4 — Size Each Engine for the Whole Card

The card serves one engine at a time and nothing else, so each engine gets the most the L4 will hold:

vLLM reserves 85% of the card (--gpu-memory-utilization 0.85), 64 running requests, 16,384 tokens each.
SGLang reserves 80% (--mem-fraction-static 0.80), 64 running requests, decode CUDA graphs up to 64.
llama.cpp runs 64 slots over a 65,536-token cache for the load sweep, and one slot of 131,072 for accuracy.
mistral.rs runs 32 requests over a fixed 16,384-token cache.

🔎 Tip: mistral.rs plans activation memory per running request. It sets aside about 395 MiB for each, so 64 requests do not load on the L4's 23 GB:

Error: Primary device cuda[0] needs 28292 MiB (2976 MiB of fixed model components, 25316 MiB of activations, 0 MiB of post-load reservations), but only 21192 MiB is usable under the memory goal.

At 32 requests a 65,536-token cache leaves too little room; --max-seqs 32 --kv-cache 16384tokens --max-num-batched-tokens 4096 serves in 5,286 MiB.

Step 5 — One Request at a Time

flight/engines.py sends six chat prompts, 256 tokens each, three times per engine, and takes the geometric mean of each prompt's median, end to end. Then 4 requests at once:

arm              e2e tok/s vs llama  decode TTFT ms 4 at once  x one    memory ready s repeat
llama                135.7    1.000   136.9      29     437.6   3.23  2930 MiB     6.1   6/6
llama-d2             207.7    1.531   211.4      30     373.2   1.80  3004 MiB     4.1   6/6
mistralrs            144.6    1.066   144.4      11     500.0   3.46  4664 MiB    24.1   6/6
pytorch              134.7    0.993       -       -     134.5   1.00  3248 MiB   236.2   6/6
litert-int248         98.4    0.725   125.2     512     100.5   1.02  1489 MiB     9.4   6/6
vllm                 140.4    1.035   140.7      16     537.2   3.83 18228 MiB   167.3   6/6
vllm-d4              275.3    2.030   281.2      19     711.3   2.58 18918 MiB   132.5   6/6
sglang               136.5    1.006   139.7      50     475.4   3.48 18345 MiB   307.2   6/6
sglang-d4            189.6    1.398   198.1      55     508.3   2.68 15191 MiB    61.1   6/6

Engine	No drafter	Best drafter	Drafter gain
vLLM	140.4	🥇 275.3 (4 tokens)	1.96x
llama.cpp	135.7	207.7 (2 tokens)	1.53x
SGLang	136.5	189.6 (4 tokens)	1.39x
mistral.rs	144.6	137.6 (1 token)	0.95x
PyTorch	134.7	—	—
LiteRT-LM	98.4	112.8 (its own)	1.15x

LiteRT-LM's exact Q4_0 build runs at 84.4 tokens per second, and LiteRT-LM takes about half a second to its first token against 11 to 55 ms for the others. Every engine repeats its own greedy text on all six prompts.

Step 6 — 1 to 64 Requests

sweep.py runs two shapes, 512 tokens in / 128 out and 128 in / 512 out, at 1 to 64 requests at once, five repeats, a unique prompt per request so no prefix cache can hit, output timed from the stream. flight/suite.py runs every sweep and every accuracy job one server at a time, and marks each finished job so a rerun skips it:

python3 suite.py /opt/flight-out/2026-11-06-suite-g6 sweep:vllm,sweep:vllm-d1,sweep:vllm-d4,sweep:sglang,...
2026-11-06 04:06:20 START sweep:vllm ready_s=39.1 gpu=52, 2040 MHz, 19296 MiB
2026-11-06 04:07:33   sweep:vllm in512-out128 rc=0
2026-11-06 04:10:07   sweep:vllm in128-out512 rc=0
2026-11-06 04:10:08 END sweep:vllm ok=True

Aggregate output tokens per second without a drafter, 128 in / 512 out:

Requests	vLLM	SGLang	mistral.rs	llama.cpp
1	138.8	134.6	133.9	133.8
8	991.0	967.4	626.5	597.5
64	🥇 3,715.7	2,967.1	1,825.2	1,056.3

512 in / 128 out:

Requests	vLLM	SGLang	mistral.rs	llama.cpp
1	132.8	129.6	133.1	125.4
8	738.3	708.3	591.2	428.1
64	🥇 1,593.7	1,373.0	1,132.0	630.8

All 126 levels completed. vLLM's output at 64 requests is 26.8x its one-request rate on long replies. SGLang reaches 0.80 to 0.86x vLLM at 64, mistral.rs 0.49 to 0.71x and llama.cpp 0.28 to 0.40x.

Step 7 — Each Engine at Its Best

best_per_level.py picks each engine's fastest measured setting at each level, 128 in / 512 out:

Requests	vLLM	SGLang	llama.cpp	mistral.rs
1	🥇 453 (4 drafted)	283 (4 drafted)	271 (2 drafted)	134 (none)
2	🥇 856 (4 drafted)	552 (4 drafted)	430 (2 drafted)	254 (none)
4	🥇 1,483 (4 drafted)	875 (4 drafted)	596 (2 drafted)	468 (none)
8	🥇 2,399 (4 drafted)	1,223 (4 drafted)	791 (2 drafted)	627 (none)
16	🥇 2,823 (4 drafted)	1,673 (none)	1,021 (2 drafted)	1,339 (none)
32	🥇 3,665 (1 drafted)	2,257 (none)	1,132 (2 drafted)	1,843 (none)
64	🥇 4,516 (4 drafted)	2,967 (none)	1,170 (2 drafted)	1,825 (none)

512 in / 128 out:

Requests	vLLM	SGLang	llama.cpp	mistral.rs
1	🥇 345 (4 drafted)	214 (4 drafted)	224 (2 drafted)	133 (none)
8	🥇 1,217 (4 drafted)	711 (4 drafted)	494 (2 drafted)	591 (none)
16	🥇 1,389 (1 drafted)	1,026 (none)	572 (2 drafted)	1,121 (none)
64	🥇 1,710 (1 drafted)	1,373 (none)	631 (none)	1,132 (none)

vLLM's drafter pays at every level: with 4 drafted tokens 2.60x / 3.26x no drafter for one request, still 1.02x / 1.22x at 64. SGLang's drafter pays up to 4 to 8 requests and loses from 16 (0.73x / 0.69x at 64). llama.cpp's 2-token drafter pays at every level on long replies (1.11x at 64) and up to 16 requests on long prompts.

Step 8 — Answer Quality

gen_eval.py asks each engine all 1,319 GSM8K test questions (2,048-token limit) and the 400 BFCL simple function-calling questions (512), one request at a time, greedy, thinking off:

Engine	GSM8K	BFCL	GSM8K with drafter	BFCL with drafter
llama.cpp	0.8984	0.9250	0.8976	0.9200
vLLM	0.8969	0.9225	0.8992	0.9225
SGLang	0.8954	0.9150	0.8992	0.9150
mistral.rs	0.9014	0.9200	—	—
PyTorch	0.8961	0.9225	—	—
LiteRT-LM, exact Q4_0	0.8999	0.8600	—	—
LiteRT-LM, Google's file	0.8908	0.8250	—	—

Paired question by question, every engine is level with llama.cpp on GSM8K, and the drafter changes no score. On BFCL, SGLang is 4 questions behind llama.cpp, a gap bigger than the run-to-run range, and LiteRT-LM is 26 and 40 behind.

The drafter cuts the time to finish GSM8K:

Engine	No drafter	With drafter	Speedup
vLLM	3,918 s	🥇 1,270 s	3.08x
SGLang	4,035 s	1,956 s	2.06x
llama.cpp	3,971 s	2,007 s	1.98x
mistral.rs	3,918 s	—	—
PyTorch	3,991 s	—	—
LiteRT-LM	5,018 s	—	—

🔎 Tip: send thinking off explicitly to SGLang and mistral.rs. "chat_template_kwargs": {"enable_thinking": false} keeps replies short on every engine; mistral.rs thinks by default on Gemma 4.

Step 9 — Cost

At the g6.4xlarge on-demand price, $1.3232 an hour in us-east-1 from the AWS Price List API, the measured throughput works out to (arithmetic, from cost.py):

Setting	128 in / 512 out	512 in / 128 out
vLLM + drafter, 1 request	$0.81 per M tokens	$1.06 per M tokens
vLLM + drafter, 64 requests	🥇 $0.08	🥇 $0.21 (1 drafted)
SGLang, 64 requests	$0.12	$0.27
mistral.rs, 64 requests	$0.20	$0.32
llama.cpp + drafter, 64 requests	$0.31	$0.59

The whole sweep and accuracy suite ran 12.2 hours, $16.14 on demand. A g6.2xlarge carries the same L4 at $0.9776 an hour; it was not measured here.

Step 10 — Tear Down

Results come off the instance as one tarball through a presigned S3 PUT, and then the instance is terminated:

python3 -c "import boto3; print(boto3.client('ec2', region_name='us-east-1').terminate_instances(InstanceIds=['i-040ed957b67c2d514'])['TerminatingInstances'][0]['CurrentState']['Name'])"
shutting-down

Compare and Contrast

	vLLM	SGLang	llama.cpp	mistral.rs
1 request, best	🥇 275.3	189.6	207.7	144.6
64 requests, 128/512, best	🥇 4,516	2,967	1,170	1,825
Drafter under load	🟢 pays at every level	❌ 0.69x at 64	🟢 to 16–64	❌ slower
GSM8K level	🟢 yes	🟢 yes	🟢 yes	🟢 yes
BFCL level	🟢 yes	⚠️ 4 behind	🟢 yes	🟢 yes
Runs as installed	🟢 yes	❌ two patches	🟢 yes	⚠️ needs cuRAND, fixed cache

PyTorch and LiteRT-LM serve one request at a time:

	PyTorch	LiteRT-LM
1 request, best	134.7	112.8
GSM8K level	🟢 yes	🟢 yes
BFCL level	🟢 yes	❌ 26–40 behind
Runs as installed	🟢 yes	⚠️ Vulkan

So, Which One?

On a G6 serving one small model, run vLLM with Google's W4A16 drafter at 4 tokens: it is the fastest engine for one request (275.3 tokens per second) and at every load to 64 requests, it finishes GSM8K 3.08x sooner than without the drafter at the same score, and at full load it costs $0.08 per million output tokens on demand. SGLang without a drafter reaches 0.66x of that at 64 requests and needs two patches. llama.cpp with its 2-token drafter is the simplest to build and serves one user at 207.7 tokens per second, but tops out near 1,170 tokens per second. mistral.rs, PyTorch and LiteRT-LM suit one user at a time.

Summary

The goal of this article was to find the inference serving engine to run for Gemma 4 E2B on an Amazon EC2 G6. The key was holding everything else fixed: one instance, the same 4-bit values in every container, the same prompts, load sweep and question sets, and each engine sized for the whole L4. The six-engine results were:

🟢 All six engines give the same GSM8K answers (0.8908 to 0.9014) with the drafter on or off.
🟢 vLLM with its drafter is the fastest engine at every level from 1 to 64 requests: 275.3 tokens per second for one request and 4,516 at 64 (128 in / 512 out).
🟢 vLLM's drafter pays at every load on the L4, and finishes GSM8K 3.08x sooner.
🟢 At full load vLLM serves a million output tokens for $0.08 on demand.
⚠️ The drafter's gain depends on the engine: 1.96x vLLM, 1.53x llama.cpp, 1.39x SGLang, 1.15x LiteRT-LM, 0.95x mistral.rs.
⚠️ g6.2xlarge had no capacity in us-east-1 on the day; g6.4xlarge on-demand carries the same L4.
❌ mistral.rs loads at most 32 requests on the L4's 23 GB.
❌ SGLang's drafter cuts output to 0.69 to 0.73x of running without it at 64 requests.

Scope: one g6.4xlarge on-demand in us-east-1d, one NVIDIA L4, the Deep Learning AMI with driver 595.112.07, Gemma 4 E2B on Google's QAT Q4_0 grid, one engine at a time, greedy decoding. One-request speed is six prompts of 256 tokens, three repeats; load is two synthetic shapes at 1 to 64 requests, five repeats each; accuracy is one pass of GSM8K (1,319) and BFCL simple (400). The one-request runs were on 2026-11-05 and the sweeps and accuracy on 2026-11-06, on the same instance; costs are arithmetic from the on-demand price.

The strategy for comparing inference serving engines on Amazon EC2 G6 was validated with an incremental step by step approach.

References

Retro Techno Generator: Your Pulse Plays 90s Acid Techno, Your Dancing Plays the Filters

xbill — Fri, 06 Nov 2026 18:40:02 +0000

This is a submission for the Presage SmartSpectra SDK Challenge

This article provides a step by step guide to Retro Techno Generator, a music toy built on the Presage SmartSpectra SDK for Node.js. A webcam reads your pulse, breathing and facial expressions, and a browser page turns them into 90s acid techno and hardcore with Web Audio. Every sound is synthesised; there are no samples.

https://github.com/xbill9/retro-techno-generator

Your breathing and pulse steer the music, your face plays the lead, and your dancing opens the filters. Pulse and breathing need you to hold still, and the music asks you to move, so the app makes the two take turns: dance and the movement drives the sound, freeze for a few seconds and it re-reads your pulse. A hard dance raises your pulse, and the next still moment pushes the tempo up.

What I Built

A Node server runs a SmartSpectra session on the local camera and streams the signals over a WebSocket to a page on localhost. The page is the instrument: a 16-step sequencer, a 303-style acid line, drum synthesis, pads, piano stabs and a hoover, all built from Web Audio oscillators and filters.

Signal	What it plays
Pulse	Tempo is pulse × 2, held between 100 and 190 BPM. Below about 70: ambient pads. 70 to 90: acid techno. Above about 90: early-90s hardcore breaks and piano stabs
Movement	Opens the master filter, brightens the acid line and doubles the hi-hats
Breathing waveform	Sways the master filter while you are still
Smile	Opens the acid filter and switches to a major key
Surprise	Fires a hoover synth
Blink	Stutters the drums and tears the visuals
Talking	Pitched-up formant "vocal" chops

The style switch has 3 BPM of hysteresis at each edge, so a pulse sitting at 70 does not flip between ambient and acid every bar. A dropdown overrides the pulse-picked style for anyone who wants to stay in hardcore.

Demo

The clip is 34 seconds of a live camera session, recorded with the page's own Record button, with the camera preview hidden. The status line reads "in position", the breathing rate follows the SDK from 14 down to 6 and back to 12 breaths per minute, VITALS switches to HELD while I move and back to LIVE when I freeze, and the mode dropdown steps through acid, hardcore and ambient. The pulse readout holds its starting value of 72 for the whole clip: no pulse reading arrived during that session, so the tempo stays on its default.

Code

https://github.com/xbill9/retro-techno-generator

MIT licence. Setup is in the README and below.

At this point you should have…

Node 20 or later on Linux x64/arm64, macOS Apple Silicon or Windows x64
A webcam and a desk lamp you can point at your face
A free SmartSpectra API key from https://physiology.presagetech.com/auth/register

Step 1 — Install and Run

git clone https://github.com/xbill9/retro-techno-generator
cd retro-techno-generator
npm install
echo "SMARTSPECTRA_API_KEY=your-key" > .env
npm start

Retro Techno Generator on http://localhost:8080 (live camera)

Open http://localhost:8080 and press Start. npm run fake runs the whole synth with sliders in place of the camera and needs no key, which is the quickest way to hear it.

Step 2 — Pick a Camera

node server.js --list-cameras

Set CAMERA=<id> in .env to use a camera other than the default.

How It Responds to Human Signals

Step 3 — Open the SmartSpectra Session

The server creates the session when the page sends start, and requests breathing, cardio and face metrics in one session:

sdk = new S.SmartSpectraSDK({
  apiKey: KEY,
  requestedMetrics: [...S.breathingMetrics, ...S.cardioMetrics, ...S.faceMetrics],
  enableTelemetry: false,
});
sdk.on('metrics', (buf) => onMetrics(S.decodeMetrics(buf)));
sdk.on('validationStatus', (code, _ts, hint) => broadcast({ type: 'status', code, hint }));
sdk.on('videoOutput', onVideo);
sdk.useCamera(S.CameraSelection.default);
sdk.start();

onMetrics takes the latest pulse rate, breathing rate and upper-chest breathing trace, the blink and talking flags, and the expression scores, and sends them to the page as one small JSON message. On Stop the server calls stopAsync() and then destroy(), which turns the camera off.

Step 4 — Show the Person Where to Stand

The SDK measures only when it can see your face and upper chest, lit from the front, and it reports what is wrong through validationStatus codes. With a face in view and the chest below the frame it repeats this every frame:

VALID 16 Move up, or tilt the camera down.
VALID 7 Place more of the chest in view.

A status line is easy to miss when you cannot see yourself, so the page has a mirror view. The SDK's videoOutput event delivers each processed frame, 1280×720 RGB at 30 frames per second on this laptop's camera:

VO 2764800 1280 720 3840 0
video frames 149

The server shrinks every third frame or so to 320×180 and sends about ten a second as a binary WebSocket message. The page draws it mirrored, with a dashed head-and-shoulders outline, a border that is red until you are in position, yellow while the first readings arrive and green once they flow, and one large instruction mapped from the SDK code:

Code	SDK name	On screen
5	`kTooDark`	Too dark. Put a light in front of you
7	`kChestNotVisible`	⬇ Show your shoulders: lean back or tilt the screen forward
12	`kExcessiveMotion`	Moving. Freeze for a few seconds to re-read pulse
16	`kFaceTooLow`	⬆ Move up, or tilt the screen forward

After the first good reading the preview shrinks to a corner thumbnail and stays there, so dancing does not bring the full panel back. A Pixelate button draws the preview as coarse blocks, enough to line up against the outline without showing a recognisable face, for streaming or recording. Camera view hides the preview entirely, which is how the demo was recorded.

Step 5 — Turn Movement Into an Input

kExcessiveMotion is the SDK saying what every camera vitals method needs: stillness. A dance app cannot ask for that all the time, so the same 320×180 frames give a movement level, the mean brightness change per pixel between consecutive previews:

for (let p = 0, q = 4; p < luma.length; p++, q += 3) {
  luma[p] = (out[q] * 77 + out[q + 1] * 150 + out[q + 2] * 29) >> 8;
  if (prevLuma) diff += Math.abs(luma[p] - prevLuma[p]);
}
broadcast({ type: 'motion', value: diff / luma.length });

Camera noise sets a baseline that depends on the room light. Sitting still in a dim room, 100 samples measured:

{"n":100,"min":3.03,"p50":3.56,"p90":8.58,"max":17.58}

The page tracks that baseline from below and subtracts it, so sitting still reads 0.00 and moving reads up to about 0.37 on a 0 to 1 scale. Movement then drives the master filter, the acid filter and the hi-hat density.

Step 6 — Read Vitals Only When Still

Pulse and breathing updates are accepted only while the movement level stays under 0.15 and has stayed there for 1.5 seconds. Otherwise the last good values hold. The HUD shows the state as VITALS LIVE or VITALS HELD. Sampled every 500 ms in the page:

["0.00","LIVE",72],["0.06","LIVE",72],["0.26","HELD",72],["0.14","HELD",72],
["0.37","HELD",72],["0.17","HELD",72],["0.04","HELD",72],["0.02","LIVE",72]

The tempo closes 5% of the gap to the held pulse on every sixteenth-note step, so a fresh reading after a dance ramps the track up over a few bars.

🔎 Tip: light from in front matters more than anything else. With a window behind you the camera exposes for the window, and the SDK reports Too dark or finds no face at all. A desk lamp aimed at your face fixes both.

Prize Categories

Most Creative or Delightful Use. The idea came from the rave flyers of the early 90s promising music that moves with you. Here the music listens to the body directly: your pulse picks between ambient, acid and hardcore, a smile flips the key to major, a surprised face fires a hoover, and a blink stutters the drums. The part that makes it play like an instrument is the turn-taking between dancing and stillness. Dance hard, freeze, and you hear your own heart rate push the track into hardcore.

Consent and Data Handling

Consent first. The page opens on a consent screen that says what the camera is for. The camera turns on only when you press Start and turns off when you press Stop.
Video stays on the machine. Frames go from the camera to the SDK in the local Node process. A 320×180 copy goes to the page for the mirror view. No frame is recorded or written to disk. Pixelate reduces that view to coarse blocks for anyone streaming or recording their screen. The app writes no camera frame to disk. The one exception is the ● Record button, used for the demo: only while it is on, it saves a .webm of the page as shown, with the synth audio, to the browser's Downloads folder.
Local only. The server listens on 127.0.0.1, so nothing outside the machine can open the page or the WebSocket.
Nothing is stored. No measurements, frames or logs of signals are kept. Closing the page or pressing Stop ends everything.
SDK telemetry is off (enableTelemetry: false). Per Presage's Telemetry & Privacy page, the SDK then starts no telemetry session and transmits no session summary, and even when on, that summary never contains video frames or measured values.
Network. The SDK needs a connection: measurements run inside an authenticated session with Presage's servers, opened with your API key.
No AI services are used by the app.
A music toy. It makes no health or medical claims and takes no decisions from the numbers. It turns them into music.

Summary

The goal of this article was to build a music toy that responds to the body in real time with the Presage SmartSpectra SDK. The key to the solution was letting dancing and stillness take turns: movement drives the sound, and pulse and breathing are read only in the still moments between. The results were:

🟢 Breathing, pulse, smile, surprise, blink and talking each drive a part of the music
🟢 In the demo the breathing rate updates live from the SDK, from 14 to 6 and back to 12 breaths per minute
❌ No pulse reading arrived during the recorded session, so the demo's tempo stays on its default
🟢 The SDK's videoOutput frames feed a mirror view with plain instructions mapped from the SDK's validation codes
🟢 The same frames give a movement level that reads 0.00 sitting still and drives the filters while you dance
🟢 Pulse and breathing update only while still, and the HUD shows LIVE or HELD
⚠️ The SDK needs your face and upper chest in frame and light in front of you before it sends any metrics
⚠️ The movement baseline was tuned in one dim room; a much noisier camera may need a different scale

Scope: one Linux laptop with its integrated 1280×720 camera, SmartSpectra Node SDK 3.4.0, Chrome, measured on 2026-11-06. The movement figures come from one 100-sample still run and one 17-second mixed run in a dim room, and the demo is one 34-second session. macOS and Windows were not tested.

The strategy for turning real-time human signals into music with the SmartSpectra SDK was validated with an incremental step by step approach.

This app and article were created with the help of AI: the code, the write-up and the demo recording tool were built with Claude Code, and every figure was checked against the runs and files linked in the repo.

References

The code: https://github.com/xbill9/retro-techno-generator
Presage SmartSpectra SDK documentation: https://smartspectra.presagetech.com/docs/
SDK Telemetry & Privacy: https://smartspectra.presagetech.com/docs/telemetry-and-privacy
SmartSpectra Node SDK on npm: https://www.npmjs.com/package/@smartspectra/node-sdk
The challenge: https://dev.to/challenges/presage-2026-11-04

The A2A CLI on Azure: One Client for Container Apps, Agent Framework and Localhost

xbill — Fri, 06 Nov 2026 18:21:08 +0000

This article provides a step by step guide to running A2A agents on Azure and driving them with the A2A CLI. A Rust agent on the official A2A Rust SDK runs in Azure Container Apps. A Microsoft Agent Framework agent on an Azure OpenAI model runs in Container Apps behind Entra ID and hands its work to the Rust agent. Both also run on a laptop, and one set of a2a commands checks all four.

https://github.com/xbill9/rust-dev/tree/main/a2a-agentframework-azure

The A2A CLI drove all four endpoints with the same commands, and 19 of 19 checks passed. Every hop speaks A2A 1.0: the Rust SDK serves it, Agent Framework serves it, and the Agent Framework agent calls the Rust agent with the A2A 1.0 Python client. On Azure the CLI needs one extra flag, an Entra ID bearer token, which az login already provides. Model access, registry pulls and the token all run on Entra ID, with no API key anywhere.

What Is Being Built?

Agent	Where it runs	A2A	Model
`mersenne-agent` (Rust)	laptop, port 8304	1.0, JSON-RPC and REST	none
`mersenne-agent` (same image)	Container Apps, 0.25 vCPU, 0.5 GiB	1.0, JSON-RPC and REST	none
Agent Framework coordinator	laptop, port 8309	1.0, JSON-RPC and REST	`gpt-5.4-mini` on Azure OpenAI
Agent Framework coordinator (same code)	Container Apps, Entra ID auth	1.0, JSON-RPC and REST	`gpt-5.4-mini` on Azure OpenAI

a2a CLI ──► Container Apps auth (Entra ID token) ──► Agent Framework Agent ──Entra ID──► Azure OpenAI gpt-5.4-mini
                                                           └── find_mersenne_primes tool
                                                                 └── a2a-sdk 1.x client ──► Container Apps (A2A 1.0) ──► mersenne-agent
a2a CLI ────────────────────────────────────────────────────────────────────────────────► Container Apps (A2A 1.0)

The workload is a Lucas-Lehmer test over the first N known Mersenne prime exponents, timed by the agent itself. Every reply carries the line Found first N Mersenne primes in X ms. and a data part with requested, found and elapsed_ms.

Versions, read from the machine on 2026-11-06:

Component	Version
A2A CLI (`a2a`)	v0.3.1
`a2a-lf` / `a2a-server-lf` (Rust SDK, A2A 1.0)	0.4.2 / 0.5.3
`agent-framework-core` / `agent-framework-openai` / `agent-framework-a2a`	1.24.0 / 1.18.0 / 1.0.0b261030
`a2a-sdk` (Python, A2A 1.0) / `azure-identity`	1.3.0 / 1.27.0
Python / Rust	3.14.7 / 1.99.1
Azure CLI	2.90.0
Model	`gpt-5.4-mini` 2026-03-17, GlobalStandard
Region	eastus2

At This Point You Should Have…

The A2A CLI on your PATH as a2a: go install github.com/a2aproject/a2a-cli@latest, then rename a2a-cli to a2a
Rust, Docker and Python 3
The Azure CLI with the containerapp extension, signed in with az login to a subscription where you can create resource groups and Entra ID app registrations

a2a version

a2a v0.3.1

Step 1 — The Model and the Shared Resources

git clone https://github.com/xbill9/rust-dev && cd rust-dev/a2a-agentframework-azure
scripts/deploy-model.py

The script creates one resource group in eastus2 holding a container registry, a user-assigned managed identity, a Container Apps environment and an AIServices account with a gpt-5.4-mini deployment. It grants "Cognitive Services OpenAI User" on the account twice: to the signed-in user, for the laptop, and to the managed identity, for the containers.

$ az group create -n a2a-agentframework-azure-rg -l eastus2
$ az containerapp env create -n a2a-af-env -g a2a-agentframework-azure-rg -l eastus2 --logs-destination none
$ az cognitiveservices account create -n a2a-af-openai-defaba -g a2a-agentframework-azure-rg -l eastus2 --kind AIServices --sku S0 --custom-domain a2a-af-openai-defaba --yes
$ az cognitiveservices account deployment create ... --model-name gpt-5.4-mini --model-version 2026-03-17 --model-format OpenAI --sku-name GlobalStandard --sku-capacity 50
✅ https://a2a-af-openai-defaba.cognitiveservices.azure.com/ deployment gpt-5.4-mini

The account gets a custom subdomain, and every call to the model authenticates with Entra ID. The coordinator holds no key.

Step 2 — Run Both Agents on the Laptop

scripts/start-local.sh

up: 8304 rust agent, 8309 agent framework coordinator -> http://localhost:8304

The coordinator is an Agent Framework Agent with one tool, served over A2A 1.0 by agent_framework_a2a on the a2a-sdk server routes. The same file runs on the laptop and in Container Apps:

client = OpenAIChatClient(
    model=MODEL,
    azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],
    credential=DefaultAzureCredential(),
)
agent = Agent(client=client, name=NAME, description=DESCRIPTION,
              instructions=INSTRUCTIONS, tools=[find_mersenne_primes])
handler = DefaultRequestHandler(
    agent_executor=WholeTextA2AExecutor(agent, stream=True),
    task_store=InMemoryTaskStore(),
    agent_card=card,
)
return Starlette(routes=[
    *create_agent_card_routes(card),
    *create_rest_routes(handler, path_prefix="/rest"),
    *create_jsonrpc_routes(handler, "/"),
])

DefaultAzureCredential picks up az login on the laptop and the managed identity in Azure. The CLI reads both cards and shows each one's protocol version:

a2a card get -a http://localhost:8304
a2a card get -a http://localhost:8309

Name:         Mersenne Prime Agent Rust
Version:      0.1.0
Interfaces:
  JSONRPC (A2A v1.0)     http://localhost:8304/jsonrpc
  HTTP+JSON (A2A v1.0)   http://localhost:8304/rest
Name:         Agent Framework Mersenne Coordinator
Version:      0.1.0
Interfaces:
  JSONRPC (A2A v1.0)     http://localhost:8309/
  HTTP+JSON (A2A v1.0)   http://localhost:8309/rest

Step 3 — The Tool Calls the Rust Agent With the A2A 1.0 Python Client

Agent Framework's A2A package is built on a2a-sdk 1.x, the same protocol version as the Rust agent. The tool sends a data part and reads the Rust agent's structured report straight from the reply:

value = Value()
value.struct_value.update({"count": count})
request = SendMessageRequest(message=Message(
    role=Role.ROLE_USER, message_id=os.urandom(16).hex(), parts=[Part(data=value)],
))
async with httpx.AsyncClient(timeout=60) as http:
    client = await create_client(
        RUST_AGENT_URL, ClientConfig(streaming=False, httpx_client=http))
    async for event in client.send_message(request):
        task = event.task if event.HasField("task") else None
        for artifact in (task.artifacts if task else []):
            for part in artifact.parts:
                if part.HasField("data"):
                    report = MessageToDict(part.data)
                elif part.HasField("text"):
                    summary = part.text

The tool refuses a reply whose requested or found differs from the count the model asked for, and the instructions tell the model to quote the agent's own reply line. The model picks N; the Rust agent does the counting and the timing. Data parts cross a protobuf Struct, so integers arrive as floats and the tool reads them with int().

🔎 Tip: Two A2A SDK Versions on One Python

agent-framework-a2a requires a2a-sdk 1.x, and the same machine runs a Strands rig whose a2a-sdk is pinned to 0.3:

# agent-framework-a2a wheel metadata (Requires-Dist)
Requires-Dist: agent-framework-core>=1.22.0,<2
Requires-Dist: a2a-sdk>=1.0.0,<2

# shared pyenv Python (used by the Strands rig)
a2a-sdk 0.3.26

scripts/start-local.sh installs the coordinator's pinned requirements into build/local-site with pip install --target, and starts the coordinator with PYTHONPATH=build/local-site. The two SDK versions never meet, and the container image installs the same pins.

Step 4 — The Rust Agent on Container Apps

scripts/deploy-rust.py builds docker/agent.Dockerfile, pushes it to the registry and creates a container app with external ingress. The app pulls with the managed identity, which holds AcrPull, so there is no registry password. The script computes the app's address from the environment's default domain before creating it and sets ADVERTISE_URL to it, so the card points at Container Apps from the first revision.

scripts/deploy-rust.py

$ docker build --platform linux/amd64 -f docker/agent.Dockerfile -t a2aafazuredefaba.azurecr.io/mersenne-agent:20261106T153734Z /home/xbill/rust-dev/a2a-agentframework-azure
$ docker push a2aafazuredefaba.azurecr.io/mersenne-agent:20261106T153734Z
$ az containerapp create -n a2a-rust-mersenne ... --target-port 8080 --ingress external --registry-identity <identity> --cpu 0.25 --memory 0.5Gi --min-replicas 1 --max-replicas 1 --env-vars PORT=8080 HOST=0.0.0.0 ADVERTISE_URL=https://a2a-rust-mersenne.braveground-531dbad2.eastus2.azurecontainerapps.io
✅ https://a2a-rust-mersenne.braveground-531dbad2.eastus2.azurecontainerapps.io/.well-known/agent-card.json

The commands are the laptop's, with the Container Apps address:

scripts/test-cli.sh aca-rust

== aca-rust rust agent (https://a2a-rust-mersenne.braveground-531dbad2.eastus2.azurecontainerapps.io)
Interfaces:
  JSONRPC (A2A v1.0)     https://a2a-rust-mersenne.braveground-531dbad2.eastus2.azurecontainerapps.io/jsonrpc
  HTTP+JSON (A2A v1.0)   https://a2a-rust-mersenne.braveground-531dbad2.eastus2.azurecontainerapps.io/rest
✅ aca-rust card (A2A v1.0)
✅ aca-rust send
✅ aca-rust stream
✅ aca-rust rest
✅ aca-rust data-part

Step 5 — The Coordinator on Container Apps Behind Entra ID

scripts/deploy-coordinator.py does four things:

creates an Entra ID app registration with the identifier URI api://<app id>, one delegated scope access_as_user, version 2 access tokens, and the Azure CLI's client id pre-authorized on that scope
builds docker/coordinator.Dockerfile and pushes it
creates the container app with RUST_AGENT_URL set to the Container Apps Rust agent and AZURE_CLIENT_ID set to the managed identity
turns on Container Apps built-in authentication with the Microsoft provider, answering 401 to any request without a valid token

scripts/deploy-coordinator.py

$ az containerapp auth microsoft update -n a2a-af-coordinator -g a2a-agentframework-azure-rg --client-id 67ce79d7-7432-4261-b394-d32c2fc36890 --issuer https://login.microsoftonline.com/<tenant>/v2.0 --allowed-audiences api://67ce79d7-7432-4261-b394-d32c2fc36890,67ce79d7-7432-4261-b394-d32c2fc36890 --yes
$ az containerapp auth update -n a2a-af-coordinator -g a2a-agentframework-azure-rg --enabled true --unauthenticated-client-action Return401
✅ https://a2a-af-coordinator.braveground-531dbad2.eastus2.azurecontainerapps.io/.well-known/agent-card.json (401 without a token)

From the CLI, the coordinator takes one flag beyond the laptop's: a bearer token for the app registration.

URL=$(scripts/azure_state.py get coordinator_url)
TOKEN=$(scripts/entra-token.py)    # az account get-access-token --resource api://<app id>
a2a --auth "Bearer $TOKEN" card get -a "$URL"
a2a --auth "Bearer $TOKEN" send -a "$URL" --transport rest "Find the first 3 Mersenne primes"

Name:         Agent Framework Mersenne Coordinator
Interfaces:
  JSONRPC (A2A v1.0)     https://a2a-af-coordinator.braveground-531dbad2.eastus2.azurecontainerapps.io/
  HTTP+JSON (A2A v1.0)   https://a2a-af-coordinator.braveground-531dbad2.eastus2.azurecontainerapps.io/rest
Status:   completed (2026-11-06T16:20:22Z)
Artifacts:
  [msg_0a2ef6a5893a0e6d006aedfe460ca48195931f337d1ca8db4e] Found first 3 Mersenne primes in 0.0030ms.
2, 3, 5

The CLI takes the base URL, the card sits at the standard /.well-known/agent-card.json path, and each call gets its own A2A context with no session header. Without a token, and with a malformed one, the card itself answers 401:

$ curl -s -o /dev/null -w '%{http_code}' $URL/.well-known/agent-card.json
401
$ curl -s -o /dev/null -w '%{http_code}' -H 'Authorization: Bearer not-a-jwt' $URL/.well-known/agent-card.json
401

🔎 Tip: Pre-Authorize the Azure CLI on the App Registration

With the Azure CLI's client id listed under preAuthorizedApplications for the access_as_user scope, az account get-access-token --resource api://<app id> returns a token for the coordinator with no consent prompt and no client secret. The token's claims, payload only:

{
  "aud": "67ce79d7-7432-4261-b394-d32c2fc36890",
  "iss": "https://login.microsoftonline.com/40482c55-d00d-4c6d-8903-643d76a74b9c/v2.0",
  "ver": "2.0",
  "scp": "access_as_user",
  "azp": "04b07795-8ddb-461a-bbee-02f9e1bf7b46"
}

azp is the Azure CLI. With requestedAccessTokenVersion: 2 the audience is the bare app id and the issuer ends in /v2.0, so the built-in authentication lists both api://<app id> and <app id> as allowed audiences and uses the v2 issuer.

🔎 Tip: Pin aiohttp for the Async Azure Credential

azure.identity.aio needs aiohttp for managed identity, and nothing in the Agent Framework or a2a-sdk dependency tree installs it. The same requirements without the pin, in a clean python:3.14-slim-trixie container:

aiohttp dist-info dirs installed: 0
ImportError: aiohttp package is not installed

coordinator/requirements.txt pins aiohttp==3.14.5. A shared Python on a laptop that already has aiohttp from another package hides the gap, so the clean container is the place to check it.

Step 6 — One Reply, One Artifact

Agent Framework's OpenAI client streams a reply as text deltas, then sends one more update holding the whole text under the same message id. One streamed reply, every text update:

1 msg_02d7537ef107 'Found'
2 msg_02d7537ef107 ' first'
...
21 msg_02d7537ef107 ' 7'
22 msg_02d7537ef107 'Found first 4 Mersenne primes in 0.0026ms.\n2, 3, 5, 7'
# last update repeats everything streamed before it under the same message_id: True

A2AExecutor appends every update to one artifact, which leaves the reply in the task twice. WholeTextA2AExecutor lets the closing update replace the artifact, so streaming clients see the deltas and the finished task holds the reply once, as a single text part:

class WholeTextA2AExecutor(A2AExecutor):
    async def handle_events(self, item, updater, streamed_artifact_ids=None, default_artifact_id=None):
        if isinstance(item, AgentResponseUpdate):
            key = item.message_id or default_artifact_id or ""
            text = "".join(c.text for c in item.contents if c.type == "text" and c.text)
            so_far = self._streamed.get(key, "")
            if text and so_far and text == so_far:
                del self._streamed[key]
                if streamed_artifact_ids is not None:
                    streamed_artifact_ids.discard(key)  # next add_artifact replaces
            elif text:
                self._streamed[key] = so_far + text
        await super().handle_events(item, updater, streamed_artifact_ids, default_artifact_id)

a2a --auth "Bearer $TOKEN" send -a "$URL" -o json "Find the first 10 Mersenne primes"

"artifacts": [
  {
    "artifactId": "msg_0df9fa92840724f8006aedfe47b9208195b7c9c1698118b7c1",
    "parts": [
      {
        "text": "Found first 10 Mersenne primes in 0.0549ms.\n2, 3, 5, 7, 13, 17, 19, 31, 61, 89"
      }
    ]
  }
]

The coordinator checks in the next step require the reply line exactly once, and the timing script counts a call only when it appears once.

Step 7 — The Same Checks on All Four Endpoints

scripts/test-cli.sh all starts the laptop agents and runs one set of checks against each endpoint. Each Rust agent gets card (A2A v1.0), send, stream, REST and a data part. Each coordinator gets card (A2A v1.0), send, stream and REST, and the Container Apps coordinator also gets a request with no token, which must answer 401.

scripts/test-cli.sh all

== local rust agent (http://localhost:8304)
✅ local card (A2A v1.0)
✅ local send
✅ local stream
✅ local rest
✅ local data-part
== local agent framework coordinator (http://localhost:8309)
✅ local card (A2A v1.0)
✅ local send
✅ local stream
✅ local rest
== aca-rust rust agent (https://a2a-rust-mersenne.braveground-531dbad2.eastus2.azurecontainerapps.io)
✅ aca-rust card (A2A v1.0)
✅ aca-rust send
✅ aca-rust stream
✅ aca-rust rest
✅ aca-rust data-part
-- without a token (expect 401)
✅ aca-coordinator rejects no token (401)
== aca-coordinator agent framework coordinator (https://a2a-af-coordinator.braveground-531dbad2.eastus2.azurecontainerapps.io)
✅ aca-coordinator card (A2A v1.0)
✅ aca-coordinator send
✅ aca-coordinator stream
✅ aca-coordinator rest

The Container Apps coordinator checks ask for 7, 11 and 3 primes. All three reached the Container Apps Rust agent, and the coordinator's log shows each one:

INFO:coordinator:rust agent https://a2a-rust-mersenne.braveground-531dbad2.eastus2.azurecontainerapps.io: Found first 7 Mersenne primes in 0.0059ms.
INFO:coordinator:rust agent https://a2a-rust-mersenne.braveground-531dbad2.eastus2.azurecontainerapps.io: Found first 11 Mersenne primes in 0.0339ms.
INFO:coordinator:rust agent https://a2a-rust-mersenne.braveground-531dbad2.eastus2.azurecontainerapps.io: Found first 3 Mersenne primes in 0.0039ms.

Step 8 — Timing the Same Command on Each Platform

scripts/cli-bench.py starts one a2a process per call and times it, so wall time includes starting the CLI, fetching the card, TLS on Azure and the reply. Targets take turns, one call each per round, so drift during the run lands on every target alike. Every reply is checked, and the statistics come from the script (results/summary-20261106.txt). N=10 and 20 calls per target throughout.

scripts/cli-bench.py --targets local-rust,aca-rust --reps 20 --n 10

local-rust               ok 20/20  wall median 10.5 ms (min 7.9, max 12.3)  compute median 0.02268 ms
aca-rust                 ok 20/20  wall median 164.2 ms (min 143.4, max 199.9)  compute median 0.02985 ms

scripts/cli-bench.py --targets local-coordinator,aca-coordinator --reps 20 --n 10

local-coordinator        ok 20/20  wall median 1836.0 ms (min 1374.5, max 4833.5)
aca-coordinator          ok 20/20  wall median 1891.2 ms (min 1581.2, max 4372.4)

The Rust agent's own compute time is a fraction of a millisecond on both machines, so the 153.7 ms between the laptop and Container Apps medians is the network and TLS. A coordinator call is an Azure OpenAI call plus a nested A2A call to a Rust agent. The Container Apps coordinator's median sits 55.2 ms above the laptop's, and both ranges span more than a second: the model call sets the time, and where the coordinator runs moves it by less than the spread between calls.

Step 9 — Teardown

make teardown

$ az ad app delete --id 67ce79d7-7432-4261-b394-d32c2fc36890
$ az group delete -n a2a-agentframework-azure-rg --yes
$ az cognitiveservices account purge -n a2a-af-openai-defaba -g a2a-agentframework-azure-rg -l eastus2
deleted a2a-agentframework-azure-rg

Everything except the app registration lives in one resource group, so one delete removes it. The app registration lives in Entra ID and is deleted first. The script then purges the AIServices account, so no soft-deleted copy is left behind. Afterwards:

$ az group exists -n a2a-agentframework-azure-rg
false
$ az cognitiveservices account list-deleted --query "[].name"
[]

The Container Apps and model addresses shown in this article belong to that deployment and no longer resolve. make deploy recreates all three steps under new addresses, and make test-cli T=all checks them again.

Compare and Contrast

Endpoint	CLI flags beyond the laptop's	A2A	Median wall time, N=10
Rust, laptop 🥇	none	1.0	10.5 ms
Rust, Container Apps 🥈	none, HTTPS address	1.0	164.2 ms
Agent Framework, laptop	none	1.0	1836.0 ms
Agent Framework, Container Apps	`--auth` with an Entra ID token	1.0	1891.2 ms

So, Which One?

For an agent with no model in it, Container Apps is the short path: one container app with external ingress, a public HTTPS address and no client changes. Every call costs the network trip, 153.7 ms here, on top of a computation that takes a fraction of a millisecond.

For an agent built around a model, the same Container Apps environment hosts it behind Entra ID with built-in authentication and no code in the agent. The CLI adds one flag, and az login supplies the token. The model call sets the response time, wherever the agent runs.

For talking to any of them, the A2A CLI is the one client. With Agent Framework on A2A 1.0, the agents talk to each other with the Python SDK and the laptop talks to all of them with the CLI, over one protocol version end to end.

Summary

The goal of this article was to show that the A2A CLI drives the same agents on a laptop and on Azure Container Apps with the same commands. The key to the solution was A2A 1.0 on every hop, Entra ID for every credential, and one set of checks run against every endpoint. The results were:

🟢 19 of 19 CLI checks passed across four endpoints: card, send, stream, REST and data parts on the Rust agent; card, send, stream, REST and a 401 without a token on the coordinator
🟢 Agent Framework serves A2A 1.0 with JSON-RPC and REST, and its tool calls the Rust 1.0 agent with the a2a-sdk 1.x client
🟢 The Container Apps coordinator's calls reached the Container Apps Rust agent, confirmed in its log
🟢 No API keys and no registry password: Entra ID for the model, the image pull and the CLI's token
⚠️ The coordinator needs WholeTextA2AExecutor, or each reply lands in the task twice
⚠️ The async Azure credential needs aiohttp pinned; nothing else installs it
⚠️ Container Apps added 153.7 ms per Rust agent call over the laptop; the coordinators' medians differ by 55.2 ms, inside a spread of more than a second

Scope: one laptop (13th Gen Intel Core i7-1360P, Linux), one Azure region (eastus2), one replica of each app at 0.25 vCPU and 0.5 GiB, one model (gpt-5.4-mini), measured on 2026-11-06. Timing used N=10 with 20 calls per target, one run per comparison. Every Azure figure includes the trip from the laptop. Both apps kept one replica running, so cold starts were not measured; cost, gRPC, task commands, push notifications and load were not tested.

The strategy for running A2A agents on Azure and driving them with the A2A CLI was validated with an incremental step by step approach.

References

The rig: https://github.com/xbill9/rust-dev/tree/main/a2a-agentframework-azure
CLI check transcript: https://github.com/xbill9/rust-dev/blob/main/a2a-agentframework-azure/results/test-cli-all-20261106T162111Z.txt
Timing summary: https://github.com/xbill9/rust-dev/blob/main/a2a-agentframework-azure/results/summary-20261106.txt
A2A CLI: https://github.com/a2aproject/a2a-cli
A2A Rust SDK: https://github.com/a2aproject/a2a-rs
A2A Python SDK: https://github.com/a2aproject/a2a-python
Microsoft Agent Framework: https://github.com/microsoft/agent-framework
Authentication and authorization in Azure Container Apps: https://learn.microsoft.com/azure/container-apps/authentication
Managed identities in Azure Container Apps: https://learn.microsoft.com/azure/container-apps/managed-identity
Configure Azure OpenAI with Microsoft Entra ID authentication: https://learn.microsoft.com/azure/ai-foundry/openai/how-to/managed-identity

One Word, Two GPUs: Why Gemma 4 Answers the Same Question Differently on a Laptop and a Tesla T4

xbill — Fri, 06 Nov 2026 16:31:06 +0000

This article provides a step by step investigation of why Gemma 4 E2B, run from the same GGUF file with the same llama.cpp build and greedy decoding, gives different answers on a GTX 1650 Ti laptop GPU and a Tesla T4, followed through one GSM8K question from the first differing word down to the order in which each card adds its numbers.

https://github.com/xbill9/gemma-graveyard

Same File, Same Build, Two Different Answers

Give a laptop GPU and a cloud GPU the same Gemma 4 file, the same llama.cpp build and temperature zero, and ask them the same grade-school math question. The laptop answers $20, and the Tesla T4 answers 2000.

Each card repeats itself exactly: the same answer, token for token, every time it runs. What differs is the number of processors on each chip. llama.cpp splits every big matrix product into one share per processor, 16 shares on the laptop and 40 on the T4, and float32 addition rounds the last digits differently when the same numbers are grouped differently. Those last-digit differences build up through the model's layers. After 124 identical tokens the model reaches a near-tie between " spent" and " saved", the two cards fall on opposite sides of it, and every word after that follows from the one that was picked. One stock llama.cpp setting, -ub 1, makes the two cards agree on every token.

A Week Down One Comment

On 2026-10-29 the L4 article in this series picked up one line from a reader:

Same files, same build, and still 2 BF16 questions graded differently across the two cards is the part I keep coming back to.

The same question had been nagging here too. Byte-identical model files, one llama.cpp tag, temperature zero: two cards ought to print the same words. At BF16 they disagreed on the grade of 2 questions. At Q8_0 they disagreed on the wording of most answers.

A week of runs on three NVIDIA cards later, every one of those differences is accounted for. This article follows one of them, question test-358, from the answer down to the arithmetic.

The Results in Brief

The GTX 1650 Ti answers test-358 with $20, which is right. The Tesla T4 answers 2000, which is graded wrong.
The two cards print the same 124 tokens, then part at one word: the 1650 picks " spent" at 51.4% against " saved" at 47.8%; the T4 picks " saved" at 59.1% against " spent" at 40.4%.
Each card repeats itself exactly. Run twice on different days, the 1650 gives the same score at all 97,116 tokens compared.
Two things make the cards differ. llama.cpp splits each big matrix product into one share per streaming multiprocessor (SM), and the 1650 has 16 where the T4 has 40. One matrix stored in BF16 goes to NVIDIA's cuBLAS library, which also picks its method by the card.
Adding the same 8,960 float32 numbers in 16 shares gives 47.0389595; in 40 shares it gives 47.0389099.
With one stock setting, -ub 1, the two cards match on all 250 answers and all 97,450 token scores at Q8_0, and on all 107,195 at Q4_0. It costs 14 to 15% of total run time.
The model's own scores cannot say which answers will change grade. Catching all 18 that change across seven setups means flagging 164 of 250 questions.

At This Point You Should Have…

The repository, xbill9/gemma-graveyard, with its 250 held-out GSM8K questions in data/heldout.jsonl.
llama.cpp b11699 (7182ef71e) built with CUDA for compute capability 7.5 on every card.
The Gemma 4 E2B Q8_0 GGUF, sha256 6b0301104c89a43c3bec5bae4b14d744f6000bc21e00e46e887fbbfe024de38a, on each machine.
Two cards of the same generation: a GTX 1650 Ti with Max-Q Design (laptop, 4 GB) and a Tesla T4 (GCE n1-standard-2). Both are Turing, compute capability 7.5.

Step 1 — Ask Both Cards the Same Question

graveyard.py run starts its own llama-server with -ngl 99 --seed 0, sends each question with greedy decoding and thinking off, grades the last number in the answer, and with --top-logprobs 2 records the two highest-scoring candidates at every generated token.

./graveyard.py run Q8_0 ~/graveyard-gguf/e2b-Q8_0.gguf data/heldout.jsonl out/1650-default \
  --llama ~/graveyard-llama --top-logprobs 2

The question:

question: Marisa gets $5 as pocket money every day from her parents. She buys 4 lollipops worth 25 cents each every day and saves the change in her piggy bank. How much money does she put in her piggy bank if she saves for 5 days?
gold answer: 20

The two cards' last lines, from docs/flip_derived.py:

GTX 1650 Ti tokens generated: 412; final line: 'Answer: 20.00'; extracted 20.0; right True
T4 tokens generated: 345; final line: 'Answer: 2000'; extracted 2000.0; right False

Both cards do the sum correctly. The T4 works in cents and reports 2000 cents. The grader wants dollars.

Step 2 — Find the Word Where the Answers Part

The model writes one token at a time, and each token is the highest-scoring candidate given everything written so far. So the two answers are identical up to the first token where the cards rank the candidates differently, and everything after that follows from it.

python3 docs/flip_derived.py

tokens identical on both cards before the split: 124
text before the split (last 120 chars): 'tep 3: Calculate the amount of money Marisa spends each day.**\nMarisa receives \\$5 (which is 500 cents) every day.\nMoney'
GTX 1650 Ti at the split: ' spent' 51.4%, ' saved' 47.8%
T4 at the split: ' saved' 59.1%, ' spent' 40.4%
GTX 1650 Ti next 12 tokens: ' spent per day = Total pocket money - Cost of lollip'
T4 next 12 tokens: ' saved per day = Money received - Money spent\nMoney saved'

At "Money ___" the model is close to a coin flip between " spent" and " saved" on both cards. The 1650 lands on one side and the T4 on the other. From there the 1650 writes "Money spent per day", the T4 writes "Money saved per day", and 221 tokens later they reach different final lines.

🔎 Tip: the split is almost always a near-tie. Across all 165 Q8_0 answers where the two cards' text differs, the median lead of the top candidate at the split is 0.12 logits on the 1650 and 0.13 on the T4, and in 157 of the 165 the two cards hold the same two candidates in opposite order.

Edward Lorenz met the same shape when he repeated a weather simulation with one value rounded from .506127 to .506, and the whole pattern his program produced was transformed. A fully deterministic calculation, a difference in the sixth decimal place, and a result that changes completely.

Step 3 — Check That Each Card Repeats Itself

A difference between cards means something only if each card agrees with itself. The 1650 ran Q8_0 at default settings on 2026-11-02, and again on 2026-11-03 with a patched build whose new switches were all off:

same card twice: 1650 default 2026-11-02 vs 1650 patched build, variables unset, 2026-11-03: text differs 0, grade differs 0, tokens compared 97116, same score 97116, median lead at the split None / None, top two swapped 0 of 0

Every answer matches word for word, and every one of 97,116 token scores is the same number. The T4 gives the same result against its own earlier run (98,637 of 98,637). Each card is deterministic. The answers depend on which card runs them.

Step 4 — Count Each Card's Processors

Both cards report compute capability 7.5. They differ in how many streaming multiprocessors (SMs) they have, read from the driver with tools/sm_count.py:

NVIDIA GeForce GTX 1650 Ti with Max-Q Design sm=16 cc=7.5 smem_per_block=49152
Tesla T4 sm=40 cc=7.5 smem_per_block=49152

llama.cpp reads the SM count once at start-up (ggml/src/ggml-cuda/ggml-cuda.cu:353). Its matrix kernel for quantized weights splits a prompt's work into one share per SM so that every SM stays busy (mmq.cuh, the "stream-k" split), then adds the shares' partial sums together. Sixteen SMs means sixteen partial sums. Forty SMs means forty.

Float32 addition rounds after every step, so the grouping changes the last digits. docs/flip_derived.py adds the same 8,960 random float32 numbers both ways:

8,960 float32 numbers, numpy default_rng(0).standard_normal, each share added in order, then the shares
16 shares of 560: 47.0389595
40 shares of 224: 47.0389099
exact sum (math.fsum in float64): 47.0389371
16-share and 40-share sums equal: False

Both are valid float32 results, either side of the exact sum. The model makes sums like this in every layer for every token, and at a near-tie like "Money ___" a change in the sixth digit is enough to swap the two candidates.

The test: a small patch (tools/graveyard-probe.patch) adds GGML_CUDA_NSM_OVERRIDE, which replaces the SM count llama.cpp reads. Told it has 40 SMs, the 1650 splits its work the way the T4 does:

1650 default vs T4 default: text differs 165, grade differs 4, tokens compared 50572, same score 29969, ...
1650 told 40 SMs vs T4 default: text differs 32, grade differs 2, tokens compared 87580, same score 85282, ...

The SM count accounts for 133 of the 165 differing answers.

Step 5 — Find the Matrix That Goes to cuBLAS

The other 32 come from one matrix. The Q8_0 file stores per_layer_model_proj (1536 × 8960) in BF16, because llama-quantize leaves it unquantized (src/llama-quant.cpp:324). llama.cpp has no batch kernel for BF16 on compute capability 7.5, so for any prompt longer than 8 tokens it hands that product to cuBLAS. With the patch's GGML_CUDA_LOG_CUBLAS=1, 200 of 200 logged cuBLAS calls are this one matrix.

cuBLAS chooses its own method by the card. NVIDIA's documentation, section 2.1.4, states the limit:

By design, all cuBLAS API routines from a given toolkit version, generate the same bit-wise results at every run when executed on GPUs with the same architecture and the same number of SMs.

The 1650 and the T4 share the architecture and differ in SM count, so cuBLAS is free to give them different bits. The SM override cannot reach it, because cuBLAS reads the real card.

The patch's GGML_CUDA_SPLIT_TO_VEC=1 sends that one product through llama.cpp's single-token kernel instead. With both changes:

1650 both fixes vs T4 matrix fix: text differs 0, grade differs 0, tokens compared 97380, same score 97380, ...

All 250 answers match, and all 97,380 token scores are the same number. The same two changes make the cards match at Q4_0 and at every K-quant rung from Q6_K to Q2_K.

Step 6 — Make Them Agree With One Stock Setting

-ub 1 sets llama.cpp's micro-batch to one token. The prompt then goes through the same single-token kernels a generated token uses, which split no work by SM count and never call cuBLAS. It needs no patch.

tools/ub1-ladder.sh 1650 ~/graveyard-gguf ~/graveyard-llama

1650 -ub 1 vs T4 -ub 1: text differs 0, grade differs 0, tokens compared 97450, same score 97450, ...
Q4_0: 1650 -ub 1 vs T4 -ub 1: text differs 0, grade differs 0, tokens compared 107195, same score 107195, ...

With -ub 1 on both cards, both cards answer test-358 with $20 and get 226 of 250 right at Q8_0.

Step 7 — What Agreement Costs

Generation speed stays within 1 token per second of default on both cards (42.5 to 42.9 on the 1650, 57.0 to 58.0 on the T4). Prompt processing slows down, more on the T4 than on the 1650. From the server logs and the summed per-answer times:

card | route | prompt tokens/s | prompt slowdown | median generation tokens/s | whole run, summed seconds | whole run vs default
GTX 1650 Ti | default | 289 | 1.0x | 42.8 | 2358 | +0%
GTX 1650 Ti | both fixes | 239 | 1.2x | 42.5 | 2403 | +2%
GTX 1650 Ti | -ub 1 | 58 | 5.0x | 42.9 | 2696 | +14%
T4 | default | 1270 | 1.0x | 57.9 | 1722 | +0%
T4 | matrix fix | 1060 | 1.2x | 57.0 | 1734 | +1%
T4 | -ub 1 | 81 | 15.7x | 58.0 | 1986 | +15%

GSM8K prompts are short, so a 15.7x slower prompt adds 15% to the whole run. Long prompts were not measured; they spend a larger share of each run on the prompt.

Step 8 — Ask Whether the Model Can Warn You

If the model knew when it was at a coin flip, it could flag the answers at risk. Seven Q8_0 setups give seven sets of answers: the T4, the 1650 and the L4 at default settings, the L4 with -ub 1, the 1650 and T4 with -ub 1 (identical), the 1650 told 40 SMs, and the 1650 and T4 with both fixes (identical). tools/route_spread.py counts them:

"runs": 7, "questions": 250, "text_varies": 219, "grade_varies": 18

219 of 250 answers are worded differently in at least one setup, and 18 change grade. test-358 is right in 3 of the 7.

Flag a question when any token in the T4's default answer had a lead below a threshold:

flag when some token's lead is below	questions flagged	of the 18 that change grade	of the 232 that do not
0.05	112	12	100
0.1	164	18	146
0.25	218	18	200

Catching all 18 means flagging 164 questions. Nearly every answer has a near-tie somewhere, and most of them change only the wording. Looking only at the last 12 tokens, where the final number is written, catches 1 of the 18.

Step 9 — Ask Which Setup Is Right

On the 1650 alone, each rung under -ub 1, under default settings and under both fixes, from docs/evidence-flips/ladder-routes.txt:

rung	right, `-ub 1`	right, default	right, both fixes	grade changes, `-ub 1` vs default
Q8_0	226	229	229	9
Q6_K	231 🥇	226	228	7
Q5_K_M	228	226	230 🥇	10
Q4_K_M	223	222	226 🥇	13
Q4_0	226 🥇	216	224	18
Q3_K_M	190	183	195 🥇	37
Q2_K	3 🥇	5	2	6

No setup wins every rung. The totals move by 3 to 12 per rung while 6 to 37 individual questions change grade. Each setup is its own sample of the model's near-ties, and the score is an average over which way they happened to fall.

Compare and Contrast

	default settings	SM count + matrix fix	`-ub 1`
Needs a patch	no	yes	no 🥇
1650 and T4 answers match, Q8_0	85 of 250	250 of 250 🥇	250 of 250 🥇
Whole run, T4	baseline 🥇	+1%	+15%
Prompt speed, T4	1,270 tokens/s 🥇	1,060 tokens/s	81 tokens/s
Right of 250, Q8_0, T4	227	229	226

So, Which One?

For a benchmark that two people on two cards need to reproduce, run with -ub 1 and say so: it makes these two cards agree to the last digit with stock llama.cpp, for 15% more run time on short prompts. For serving, the default settings are faster and no less accurate; the cost is that an answer from one card is a sample, and another card can word it differently or grade it differently.

Report a score with the card, the build and the settings that produced it. Two cards that agree on a total can still disagree on which questions they got right.

Summary

The goal of this article was to explain why the same Gemma 4 E2B file gives different answers on two GPUs of the same generation. The key to the solution was following one question, test-358, to the single word where the two cards part, and then removing each cause of the difference until the two cards matched at every token. The results were:

🟢 Each card is deterministic: the same card run twice matches on all 97,116 token scores.
🟢 Two causes explain every difference: llama.cpp splits matrix work by SM count (16 against 40), and cuBLAS handles one BF16 matrix by the card. Removing both gives 250 of 250 matching answers.
🟢 Stock -ub 1 makes the GTX 1650 Ti and the T4 match on every token score at Q8_0 and Q4_0, for 14 to 15% more run time.
⚠️ At default settings the two cards word 165 of 250 Q8_0 answers differently and grade 4 differently. test-358 is right on the 1650 and wrong on the T4.
⚠️ Seven setups on Q8_0 change the grade of 18 questions; scores stay between 225 and 230.
❌ The model's own scores cannot pick out the answers at risk: catching all 18 means flagging 164 of 250.

Scope: Gemma 4 E2B, 250 held-out GSM8K questions, greedy, seed 0, thinking off, 1,024 max tokens, llama.cpp b11699 built for compute capability 7.5 with CUDA 13.4, one run per setup. Cards: a GTX 1650 Ti with Max-Q Design (laptop), a Tesla T4 on GCE, and an L4 on AWS for the seven-setup count. The T4 rungs under -ub 1 stop at Q4_0; the K-quant rungs under -ub 1 are on the 1650 only. Speed is compared within each card only. Every figure here is recomputed from the committed runs by docs/flip_derived.py and make verify.

The strategy for tracing a GPU-to-GPU answer flip in Gemma 4 down to the order of its additions was validated with an incremental step by step approach.

References

The comment that started it, on the L4 article
Performance vs Quality: Gemma 4 E2B Quantization on an NVIDIA L4 and a Tesla T4, the L4 article
Four GPUs, One Quantization Ladder: Gemma 4 E2B From a 4 GB Laptop to an NVIDIA L4
gemma-graveyard on GitHub
SM count and cuBLAS report
-ub 1 report
cuBLAS documentation, Results Reproducibility
When the Butterfly Effect Took Flight, MIT Technology Review
llama.cpp
google/gemma-4-E2B-it
GSM8K

The A2A CLI on AWS: One Client for Lightsail, Bedrock AgentCore and Localhost

xbill — Thu, 05 Nov 2026 18:13:26 +0000

This article provides a step by step guide to running A2A agents on AWS and driving them with the A2A CLI. A Rust agent on the official A2A Rust SDK runs in an Amazon Lightsail container. A Strands agent on an Amazon Bedrock model runs in Amazon Bedrock AgentCore Runtime and hands its work to the Rust agent. Both also run on a laptop, and one set of a2a commands checks all four.

https://github.com/xbill9/rust-dev/tree/main/a2a-strands-aws

The A2A CLI drove all four endpoints with the same commands, 16 of 16 checks passed, and the CLI also carried the call from the Strands agent to the Rust agent. The two agents speak different protocol versions: the Rust SDK serves A2A 1.0 and Strands serves A2A 0.3. The CLI reads both. Strands' own Python client reads only 0.3, so the Strands agent calls the Rust agent by running the same a2a binary the laptop uses. On AWS the CLI needs a login token, a session header and the full agent card URL, and nothing else changes.

What Is Being Built?

Agent	Where it runs	A2A	Model
`mersenne-agent` (Rust)	laptop, port 8204	1.0, JSON-RPC and REST	none
`mersenne-agent` (same image)	Lightsail container service, nano	1.0, JSON-RPC and REST	none
Strands coordinator	laptop, port 8209	0.3, JSON-RPC	Claude Haiku 5.5 on Bedrock
Strands coordinator (same code)	Bedrock AgentCore Runtime	0.3, JSON-RPC	Claude Haiku 5.5 on Bedrock

a2a CLI ──► AgentCore Runtime (A2A 0.3, Cognito token) ──► Strands Agent ──Bedrock──► Claude Haiku 5.5
                                                                └── find_mersenne_primes tool
                                                                      └── a2a CLI ──► Lightsail (A2A 1.0) ──► mersenne-agent
a2a CLI ──────────────────────────────────────────────────────────────────────────► Lightsail (A2A 1.0)

The workload is the one from the sibling rig: a Lucas-Lehmer test over the first N known Mersenne prime exponents, timed by the agent itself. Every reply carries the line Found first N Mersenne primes in X ms. and a data part with requested, found and elapsed_ms.

Versions, read from the machine on 2026-11-05:

Component	Version
A2A CLI (`a2a`)	v0.3.1
`a2a-lf` / `a2a-server-lf` (Rust SDK, A2A 1.0)	0.4.2 / 0.5.3
`strands-agents` / `a2a-sdk` / `bedrock-agentcore`	1.64.0 / 0.3.26 / 1.25.3
Rust	1.99.1, edition 2024
Model	`us.anthropic.claude-haiku-5-5` on Bedrock
Region	us-east-1

At This Point You Should Have…

The A2A CLI on your PATH as a2a: go install github.com/a2aproject/a2a-cli@latest, then rename a2a-cli to a2a
Rust, Go, Docker and Python 3 with pip install "strands-agents[a2a]" "bedrock-agentcore[a2a]"
AWS credentials with Bedrock model access in us-east-1, and the lightsailctl plugin for the AWS CLI

a2a version

a2a v0.3.1
commit:  none
built:   unknown
go:      go1.26.3
os/arch: linux/amd64

Step 1 — Run Both Agents on the Laptop

git clone https://github.com/xbill9/rust-dev && cd rust-dev/a2a-strands-aws
scripts/start-local.sh

up: 8204 rust agent, 8209 strands coordinator -> http://localhost:8204

The coordinator is a Strands Agent with one tool, served over A2A by the AgentCore SDK's serve_a2a. The same file runs on the laptop and in AgentCore:

def create_agent(_context_id: str) -> Agent:
    return Agent(
        name="Strands Mersenne Coordinator",
        model=BedrockModel(model_id=MODEL_ID),
        system_prompt=SYSTEM_PROMPT,
        tools=[find_mersenne_primes],
        callback_handler=None,
    )

serve_a2a(StrandsA2AExecutor(agent_factory=create_agent), agent_card(), port=port)

The CLI reads both cards and shows each one's protocol version:

a2a card get -a http://localhost:8204
a2a card get -a http://localhost:8209

Name:         Mersenne Prime Agent Rust
Version:      0.1.0
Interfaces:
  JSONRPC (A2A v1.0)     http://localhost:8204/jsonrpc
  HTTP+JSON (A2A v1.0)   http://localhost:8204/rest
Name:         Strands Mersenne Coordinator
Version:      0.1.0
Interfaces:
  JSONRPC (A2A v0.3.0)   http://localhost:8209/
Streaming:    true

Step 2 — Why the Strands Tool Calls the A2A CLI

Strands pins a2a-sdk below 0.4, so its A2A client and server speak protocol 0.3. An A2A 1.0 card lists its endpoints under supportedInterfaces and drops the top-level url field that a 0.3 card requires. scripts/check-sdk03-card.py fetches each card with the 0.3 client:

scripts/check-sdk03-card.py http://localhost:8204
scripts/check-sdk03-card.py http://localhost:8209

a2a-sdk 0.3.26, card from http://localhost:8204
A2AClientJSONError: JSON Error: Failed to validate agent card structure from http://localhost:8204/.well-known/agent-card.json: [{"type":"missing","loc":["url"],"msg":"Field required", ...
a2a-sdk 0.3.26, card from http://localhost:8209
ok: Strands Mersenne Coordinator, protocolVersion 0.3.0, url http://localhost:8209/

The A2A CLI reads both, so the tool runs it with JSON output and a data part, then reads the Rust agent's structured report:

cmd = [
    A2A_BIN, "send", "-a", RUST_AGENT_URL, "-o", "json", "--timeout", "60s",
    "--data-part", json.dumps({"count": count}),
]
proc = subprocess.run(cmd, capture_output=True, text=True, timeout=90)

The tool refuses a reply whose requested or found differs from the count the model asked for, and the system prompt tells the model to quote the agent's own reply line. The model picks N; the Rust agent does the counting and the timing.

a2a send -a http://localhost:8209 "Find the first 9 Mersenne primes"

Artifacts:
  [agent_response] Found first 9 Mersenne primes in 0.0298ms.

Exponents: 2, 3, 5, 7, 13, 17, 19, 31, 61

Step 3 — The Rust Agent on Lightsail

scripts/deploy-lightsail.py creates a nano container service, builds docker/agent.Dockerfile, pushes it with aws lightsail push-container-image and deploys it with ADVERTISE_URL set to the service's own HTTPS address, so the card's interfaces point at Lightsail. The health check is the agent card path.

scripts/deploy-lightsail.py

creating container service a2a-rust-mersenne (nano, scale 1)
$ docker build --platform linux/amd64 -f docker/agent.Dockerfile -t a2a-rust-mersenne:local /home/xbill/rust-dev/a2a-strands-aws
$ aws lightsail push-container-image --region us-east-1 --service-name a2a-rust-mersenne --label agent --image a2a-rust-mersenne:local
pushed :a2a-rust-mersenne.agent.222
✅ https://a2a-rust-mersenne.6wpv8vensby5c.us-east-1.cs.amazonlightsail.com/.well-known/agent-card.json

The commands are the laptop's, with the Lightsail address:

scripts/test-cli.sh lightsail

== lightsail rust agent (https://a2a-rust-mersenne.6wpv8vensby5c.us-east-1.cs.amazonlightsail.com)
Interfaces:
  JSONRPC (A2A v1.0)     https://a2a-rust-mersenne.6wpv8vensby5c.us-east-1.cs.amazonlightsail.com/jsonrpc
  HTTP+JSON (A2A v1.0)   https://a2a-rust-mersenne.6wpv8vensby5c.us-east-1.cs.amazonlightsail.com/rest
✅ lightsail card (A2A v1.0)
✅ lightsail send
✅ lightsail stream
✅ lightsail rest
✅ lightsail data-part

Step 4 — An arm64 Image Built on an x86 Laptop

AgentCore Runtime runs arm64 containers, serves A2A on port 9000, and expects the card at /.well-known/agent-card.json and a health answer at /ping. serve_a2a provides all three. The image has to be arm64, and this laptop has no arm64 emulator, so scripts/build-coordinator.sh assembles it from parts that are already built for arm64:

pip downloads aarch64 wheels: --platform manylinux_2_28_aarch64 --python-version 3.13 --only-binary=:all:
Go cross-compiles the A2A CLI: GOOS=linux GOARCH=arm64 CGO_ENABLED=0 go build
the Dockerfile only copies, so Docker never runs an arm64 binary

FROM python:3.13-slim-trixie
COPY build/coordinator/site /opt/site
COPY build/coordinator/a2a /usr/local/bin/a2a
COPY coordinator/main.py /app/main.py
ENV PYTHONPATH=/opt/site PYTHONUNBUFFERED=1 DOCKER_CONTAINER=1 A2A_PORT=9000
WORKDIR /app
EXPOSE 9000
CMD ["python", "main.py"]

file build/coordinator/a2a
docker image inspect a2a-strands-coordinator:local --format '{{.Os}}/{{.Architecture}}'

build/coordinator/a2a: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, Go BuildID
linux/arm64

ARCH=amd64 scripts/build-coordinator.sh builds the same image for x86, which runs on the laptop and checks the image layout before AgentCore sees it.

Step 5 — The Strands Agent on Bedrock AgentCore Runtime

scripts/deploy-agentcore.py pushes the image to ECR and creates an IAM role with ECR pull, CloudWatch Logs and bedrock:InvokeModel*. It creates a Cognito user pool with one app client and one user, and an AgentCore runtime with serverProtocol: A2A, a customJWTAuthorizer on that pool, public networking, and RUST_AGENT_URL set to the Lightsail agent. A second update sets AGENTCORE_RUNTIME_URL once the runtime's ARN exists, so the card advertises the AgentCore address.

scripts/deploy-agentcore.py

  runtime CREATING (version 1)
  runtime READY (version 1)
  runtime UPDATING (version 2)
  runtime READY (version 2)
✅ https://bedrock-agentcore.us-east-1.amazonaws.com/runtimes/arn%3Aaws%3Abedrock-agentcore%3Aus-east-1%3A106059658660%3Aruntime%2Fa2a_strands_coordinator-Rtn6xkF0PS/invocations/

From the CLI, AgentCore takes a bearer token from Cognito and a session id header of at least 33 characters:

CARD=$(scripts/aws_state.py get agentcore_card)
TOKEN=$(scripts/cognito-token.py)
SID="a2a-cli-$(cat /proc/sys/kernel/random/uuid)"
a2a --auth "Bearer $TOKEN" --svc-param "X-Amzn-Bedrock-AgentCore-Runtime-Session-Id=$SID" card get -a "$CARD"

Name:         Strands Mersenne Coordinator
Description:  Strands agent on Amazon Bedrock (A2A 0.3) that delegates Mersenne prime runs to a Rust A2A 1.0 agent through the a2a CLI.
Version:      0.1.0
Interfaces:
  JSONRPC (A2A v0.3.0)   https://bedrock-agentcore.us-east-1.amazonaws.com/runtimes/arn%3Aaws%3Abedrock-agentcore%3Aus-east-1%3A106059658660%3Aruntime%2Fa2a_strands_coordinator-Rtn6xkF0PS/invocations
Streaming:    true
Skills:
  mersenne-via-rust    Mersenne primes via the Rust agent

🔎 Tip: Give the CLI the Full Card URL for AgentCore

The CLI reads a URL with no card path as a host, and asks that host's origin for /.well-known/agent-card.json. For AgentCore the origin is https://bedrock-agentcore.us-east-1.amazonaws.com, which answers 404:

$ a2a --auth "Bearer $TOKEN" --svc-param "X-Amzn-Bedrock-AgentCore-Runtime-Session-Id=$SID" card get -a "https://bedrock-agentcore.us-east-1.amazonaws.com/runtimes/arn%3A...%2Fa2a_strands_coordinator-Rtn6xkF0PS/invocations/"
Error: failed to resolve agent card: card request failed, status: 404 Not Found
hint: check the --agent-card reference (host, full card URL, or local file path)

Append .well-known/agent-card.json to the invocations URL and the same command returns the card. The deploy script stores that URL as agentcore_card.

Step 6 — The Same Checks on All Four Endpoints

scripts/test-cli.sh all starts the laptop agents and runs one set of checks against each endpoint. Each Rust agent gets card (A2A v1.0), send, stream, REST and a data part. Each coordinator gets card (A2A v0.3), send and stream. Every check looks for the agent's reply line with the right N.

scripts/test-cli.sh all

== local rust agent (http://localhost:8204)
✅ local card (A2A v1.0)
✅ local send
✅ local stream
✅ local rest
✅ local data-part
== local strands coordinator (http://localhost:8209)
✅ local card (A2A v0.3)
✅ local send
✅ local stream
== lightsail rust agent (https://a2a-rust-mersenne.6wpv8vensby5c.us-east-1.cs.amazonlightsail.com)
✅ lightsail card (A2A v1.0)
✅ lightsail send
✅ lightsail stream
✅ lightsail rest
✅ lightsail data-part
== agentcore strands coordinator (https://bedrock-agentcore.us-east-1.amazonaws.com/runtimes/arn%3A...%2Fa2a_strands_coordinator-Rtn6xkF0PS/invocations/.well-known/agent-card.json)
✅ agentcore card (A2A v0.3)
✅ agentcore send
✅ agentcore stream

The AgentCore checks ask for 7 and 11 primes. Both requests reached Lightsail, through the CLI inside the AgentCore container. The Lightsail container log shows them:

INFO mersenne_agent: computed requested=7 found=7 elapsed_ms=0.005935
INFO mersenne_agent: computed requested=11 found=11 elapsed_ms=0.056663000000000005

Step 7 — Timing the Same Command on Each Platform

scripts/cli-bench.py starts one a2a process per call and times it, so wall time includes starting the CLI, fetching the card, TLS on AWS and the reply. Targets take turns, one call each per round, so drift during the run lands on every target alike. Every reply is checked, and the statistics come from the script (results/summary-20261105.txt). N=10 throughout.

scripts/cli-bench.py --targets local-rust,lightsail-rust --reps 20 --n 10

local-rust               ok 20/20  wall median 27.7 ms (min 24.1, max 35.1)  compute median 0.05682 ms
lightsail-rust           ok 20/20  wall median 154.4 ms (min 140.4, max 174.3)  compute median 0.03928 ms

scripts/cli-bench.py --targets local-coordinator,agentcore-coordinator --reps 5 --n 10
scripts/cli-bench.py --targets local-coordinator,agentcore-coordinator --reps 5 --n 10 --reuse-session

local-coordinator        ok 5/5  wall median 2596.2 ms (min 2288.0, max 2662.2)
agentcore-coordinator    ok 5/5  wall median 3416.5 ms (min 3364.7, max 3482.8)
local-coordinator        ok 5/5  wall median 2459.9 ms (min 2184.5, max 2683.8)
agentcore-coordinator    ok 5/5  wall median 2997.1 ms (min 2849.7, max 12177.7)

The Rust agent's own compute time is a fraction of a millisecond on both machines, so the 126.7 ms between the laptop and Lightsail medians is the network and TLS. A coordinator call is a Bedrock model call plus a nested CLI call to the Rust agent. AgentCore added 820.3 ms over the laptop with a new session id per call and 537.2 ms with one session for the run. In the one-session run, the first AgentCore call took 12177.7 ms and the other four stayed near the median.

Step 8 — Teardown

make teardown

deleted runtime a2a_strands_coordinator
deleted user pool a2a-strands-aws
deleted role a2a-strands-coordinator-runtime
deleted ECR repository a2a-strands-coordinator
deleted a2a-rust-mersenne

make deploy recreates both, and make test-cli T=all checks them again.

Compare and Contrast

Endpoint	CLI flags beyond the laptop's	A2A	Median wall time, N=10
Rust, laptop 🥇	none	1.0	27.7 ms
Rust, Lightsail 🥈	none, HTTPS address	1.0	154.4 ms
Strands, laptop	none	0.3	2596.2 ms
Strands, AgentCore	`--auth`, session header, full card URL	0.3	3416.5 ms new session per call, 2997.1 ms one session

So, Which One?

For an agent with no model in it, Lightsail is the short path: one container, a public HTTPS address, no client changes. Every call costs the network trip, 126.7 ms here, on top of a sub-millisecond computation.

For an agent built around a model, AgentCore Runtime gives it a managed home with inbound authentication, and costs the CLI two flags and the full card URL. Reuse a session id across calls when a client sends many.

For talking to any of them, the A2A CLI is the one client. It reads A2A 1.0 and 0.3 cards, and it fits inside an agent as well as on a laptop: the Strands agent reaches a Rust 1.0 agent with the same binary, where its own 0.3 client cannot read the card.

Summary

The goal of this article was to show that the A2A CLI drives the same agents on a laptop, on Amazon Lightsail and on Amazon Bedrock AgentCore Runtime with the same commands. The key to the solution was one set of checks run against every endpoint, and a Strands tool that calls the Rust agent through the CLI. The results were:

🟢 16 of 16 CLI checks passed across four endpoints: card, send, stream, REST and data parts on the Rust agent, card, send and stream on the Strands agent
🟢 The CLI read A2A 1.0 cards (Rust SDK) and A2A 0.3 cards (Strands) with the same commands
🟢 Inside AgentCore the CLI carried the agent-to-agent call to Lightsail, confirmed in the Lightsail log
🟢 An arm64 AgentCore image built on an x86 laptop with no emulator
⚠️ AgentCore needs --auth, a session id header and the full card URL; the invocations URL alone returns 404
⚠️ Lightsail added 126.7 ms per call over the laptop, AgentCore 820.3 ms with a new session per call and 537.2 ms with one session
❌ The a2a-sdk 0.3 Python client that Strands uses cannot parse an A2A 1.0 card

Scope: one laptop (13th Gen Intel Core i7-1360P, Linux), one AWS region (us-east-1), one Lightsail power (nano), one model (us.anthropic.claude-haiku-5-5), measured on 2026-11-05. Timing used N=10 with 20 calls per Rust target and 5 per coordinator target, and each coordinator comparison is a single run. Every AWS figure includes the trip from the laptop. gRPC, task commands, push notifications and load were not tested.

The strategy for running A2A agents on AWS and driving them with the A2A CLI was validated with an incremental step by step approach.

References

The rig: https://github.com/xbill9/rust-dev/tree/main/a2a-strands-aws
The sibling rig, the same Rust agent on a laptop: https://github.com/xbill9/rust-dev/tree/main/a2a-benchmark-rust
CLI check transcript: https://github.com/xbill9/rust-dev/blob/main/a2a-strands-aws/results/test-cli-all-20261105T163653Z.txt
Timing summary: https://github.com/xbill9/rust-dev/blob/main/a2a-strands-aws/results/summary-20261105.txt
A2A CLI: https://github.com/a2aproject/a2a-cli
A2A Rust SDK: https://github.com/a2aproject/a2a-rs
Strands Agents: https://strandsagents.com/
Deploy A2A servers in AgentCore Runtime: https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/runtime-a2a.html
A2A protocol contract for AgentCore Runtime: https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/runtime-a2a-protocol-contract.html
Amazon Lightsail container services: https://docs.aws.amazon.com/lightsail/latest/userguide/amazon-lightsail-container-services.html

Android 17 on a 4.9 Kernel: LineageOS 24.0 on the Pixel 3a

xbill — Thu, 05 Nov 2026 16:37:16 +0000

This article provides a step by step guide to running LineageOS 24.0, which is Android 17, on a Google Pixel 3a and its original 4.9 Linux kernel. It continues the LineageOS 23.2 port on the same phone. Every patch, script and build is published.

https://github.com/xbill9/android_kernel_google_msm-4.9/releases/tag/lineage-24.0-sargo-20261027

The Pixel 3a boots LineageOS 24.0 on kernel 4.9.337 with SELinux enforcing in 83 s, with Wi-Fi, Bluetooth, the sensors, Chrome and Google Play Services working, and updates that keep user data. Android 17 assumes a kernel of at least 5.10 in more places than Android 16 did. The port needed one kernel fix, for a cgroup BPF reference count the 4.9 backport had missed, five forks of platform code (fscrypt keys, the BPF loader, netd and Connectivity, and the native app zygote), and device-tree changes for ueventd, gralloc and the ION memory allocator. The /init logger from the 23.2 port found each one.

What is this project trying to Do?

The Pixel 3a (codename sargo) has a Snapdragon 670 and a 4.9 kernel. Its last Google build is Android 12. LineageOS carries it officially to 22.2, and the previous article in this series brought it to 23.2, Android 16, by backporting three kernel features.

The goal here: keep the phone current and out of a landfill by running the newest LineageOS release, 24.0, on the kernel it already has.

The kernel carries a BPF backport that reports itself to Android as 5.4. The NetBpfLoad lines from the first 24.0 boot show the version Android reads, 0x504012e, against the 5.10 it asks for:

NetBpfLoad: skipping map sk_storage which requires kernel version 0x504012e >= 0x50a0000
NetBpfLoad: skipping map packet_trace_ringbuf which requires kernel version 0x504012e >= 0x50a0000

At This Point You Should Have…

A Google Pixel 3a (sargo), bootloader unlocked
The LineageOS 24.0 source with the local manifest from the release
A build machine with room for the tree; the 24.0 tree here is 266 GB, and the home machine is an i7-3770 with 8 threads and 31 GB of RAM
adb and fastboot
A clone: git clone https://github.com/xbill9/android_kernel_google_msm-4.9

The first full build ran on a 192-core EC2 instance and the tree then moved to the home machine. That half of the work has its own article: https://dev.to/aws-builders/rent-the-cores-keep-the-phone-building-lineageos-24-on-ec2-and-iterating-at-home-4pmc

Step 1 — Carry the 23.2 Tree to 24.0

The device tree starts from the 23.2 port, on a new lineage-24.0 branch of the bonito fork. Android 17 dropped several HIDL HALs and moved a few Pixel components, so the tree changed in the same places the Pixel 4a (sunfish) did:

vibrator HAL (drv2624) carried in the tree; thermal from hardware/google/pixel/thermal
AIDL lights on /sys/class/backlight/panel0-backlight, since Android 17 has no HIDL lights
power stats AIDL only, HIDL dumpstate dropped
the fingerprint blob patched to lineage.frameworks.stats@1.0, as sunfish does
KERNEL_CROSS_COMPILE with the GCC 4.9 binutils: LineageOS 24 builds kernels with LLVM only, and 4.9 predates LLVM=1 and links LTO with GNU gold

The full build then stopped on six more device-tree errors, among them unknown type logger_app, two genfscon rules for /sys/class/typec, and a framework compatibility matrix that starts at level 7. A small shim keeps the SecureUI blob linking against Android 17's SurfaceComposerClient::createSurface. The list, with commits, is in the 24.0 README.

The lunch form changed too: lunch lineage_sargo cp2a userdebug, with spaces. The first complete package:

lineage-24.0-20261023-UNOFFICIAL-sargo.zip  1,208,903,032 bytes

Step 2 — Read the Boot With the /init Logger

The 23.2 port built a 4 KB static /init, kmsglog.c, that execs the real init as process 1 and copies the kernel log to the metadata partition. For 24.0 it gained three things:

it also appends logcat to /metadata/logcat.txt, by entering init's root through /proc/1/root
kmsglog.reboot=<s> on the kernel command line dumps every blocked task and reboots to the bootloader after that many seconds, so a hung boot ends by itself
the log file is truncated each boot and wraps at 2 MB

add-logger.sh repacks any boot image with it:

EXTRA_CMDLINE="androidboot.selinux=permissive kmsglog.reboot=150" \
  ./add-logger.sh boot.img boot-log.img

test-boot-b.sh RAM-boots the image on slot b, waits, and pulls /metadata/boot-kmsg.txt and /metadata/logcat.txt from the fallback system. The flashed slot keeps a working build, so a failed test falls back to it with no hands on the phone.

🔎 Tip: /metadata is only 10 MB. It also holds vold's keys and the feature flags in /metadata/aconfig, so the logger caps the kernel log at 2 MB, rotates logcat at 1 MB times two, and the test script deletes the logs after pulling them.

Step 3 — ueventd

The first permissive boot shows keymaster, gatekeeper, citadeld and surfaceflinger aborting. Their device nodes, /dev/qseecom, /dev/citadel0 and /dev/kgsl-3d0, still have root-only permissions.

Android 17's ueventd reads only /system/etc/ueventd.rc, which imports /vendor/etc/ueventd.rc. The device installed its rules to the old /vendor/ueventd.rc, which ueventd no longer parses. Bonito 31b5a72 installs them to vendor/etc.

Step 4 — Gralloc

Next, the composer HAL aborts:

F GraphicBufferMapper: gralloc-mapper is missing
F DEBUG   : Abort message: 'gralloc-mapper is missing'

Android 17's libui loads only IMapper 4.0 and newer on API 36 and later. The sdm845 display stack provides IMapper 2.1. libui still has the older path behind LEGACY_GRALLOC, and bonito 433912b builds it with that flag through Soong config.

Step 5 — The fscrypt Session Keyring

With a full build flashed, the phone goes to recovery 4.9 s into the boot:

E vold    : Failed to install fscrypt key to /data: Inappropriate ioctl for device
E init: Exec service failed, status 25: Rebooting into recovery, reason: enablefilecrypto_failed

The 4.9 kernel has no FS_IOC_ADD_ENCRYPTION_KEY, so file encryption keys go into a keyring that every process shares through a session keyring. Android 17 removed that path from both sides: init no longer joins a session keyring, and vold no longer installs keys into it.

Two forks bring it back from 23.2:

system/core 5742820: init joins a session keyring at second stage, and mount_all creates the fscrypt keyring on encrypted devices
system/vold fce7dad: isFsKeyringSupported(), installKeyLegacy(), evictKeyLegacy() and the cache drop after eviction, with inline keyctl wrappers since Android 17 removed libkeyutils

🔎 Tip: do not accept the factory reset. Recovery offers one after enablefilecrypto_failed. The data on /data is intact, and the other slot still boots it.

Step 6 — BPF on a Kernel That Reports 5.4

Android 17's network stack loads its BPF programs per kernel version, and several now need 5.8 or 5.10. Each missing one stopped the boot in a different place:

init: Reboot start, reason: reboot,bpfloader-failed, reboot_target: bpfloader-failed
E NetdUpdatable: libnetd_updatable_init: Failed: Failed to get program from /sys/fs/bpf/netd_shared/prog_netd_egress_stats: No such file or directory

Two forks handle them:

system/bpf 72b37ef: bpfRingbufProg is a test program that needs BPF ring buffers, from 5.8. It is marked critical, so its load failure rebooted userdebug builds. It is now non-critical.
packages/modules/Connectivity 5c298de: NetBpfLoad warns below 5.10, as it did in 23.2; it hands off to uprobestatsbpfload only on 5.8 and later; the 5.4 ingress and egress stats programs in netd.c extend to the newest API level, since Android 17 has none below 5.10; and ConnectivityService skips the two ring-buffer listeners when their maps are absent, which aborted system_server.

Step 7 — netd Stuck in rcu_barrier

With the forks in, about one permissive boot in five reaches the boot animation and stops. The logger's task dump shows netd blocked in the kernel:

[<00000000edff5dad>] _rcu_barrier+0x210/0x2b0
[<000000007d48fdfd>] netdev_run_todo+0xc4/0x9b4
[<00000000610d42d3>] rtnetlink_rcv+0x58/0x78
[<000000004b18ba16>] netlink_unicast+0x2b0/0x468
[<000000004bbdc794>] netlink_sendmsg+0x4bc/0x604
netd            S    0  1595      1 0x00000000

system_server waits for netd until its watchdog kills it, and the cycle repeats. The kernel log of these boots carries one warning that names the cause:

percpu ref (cgroup_bpf_release_fn) <= 0 (0) after switching to atomic

cgroup_sk_clone() reached this tree from 4.9-stable before the tree had a cgroup_bpf reference count. The BPF backport added cgroup_bpf_get() to cgroup_sk_alloc() and cgroup_bpf_put() to cgroup_sk_free(), and left the clone path out. Every socket made by accept() dropped a reference it never took. When a cgroup was removed, its BPF programs were freed while sockets in it could still run them.

Kernel e7a12e10fc29 is one line: take the reference, as upstream does since ad0f75e5f57c. Before the fix 2 of 9 boots hung; after it 8 of 8 booted.

Step 8 — ION Under SELinux Enforcing

Permissive boots reached the home screen. Enforcing boots stopped at a black screen with no USB. The permissive logs show the reason, from 21 domains:

avc:  denied  { read } for  comm="qseecomd" name="ion" dev="tmpfs" ino=12110 scontext=u:r:tee:s0 tcontext=u:object_r:device:s0 tclass=chr_file permissive=1

Android 17 removed /dev/ion from the platform file_contexts and every ion_device rule from the platform policy. This kernel has ION and no DMA-BUF heaps. Three bonito commits put it back in vendor policy:

2cbfb4b: label /dev/ion as ion_device
e2ca4b1: the Android 16 ion_device rules for surfaceflinger, keymaster, gatekeeper, gralloc and qseecomd
a6e99a3: four more accesses an enforcing boot denies, for the sensors, Wi-Fi, USB gadget and power HALs

Three boots from flash with all three took 81, 48 and 80 s, with Wi-Fi up each time.

Step 9 — The Native App Zygote

The phone booted, but Chrome exited with Timed out waiting for GPU channel and then every app launch stalled for minutes. Chrome 156 runs its renderers in a native app zygote, and the zygote died on start:

Failed to check capability bound: GetCapBoundFailure(EINVAL)

Android 17's zygote_next drops the capability bounding set before it forks apps, and it reads every capability up to 40. Linux 5.9 added CAP_PERFMON, CAP_BPF and CAP_CHECKPOINT_RESTORE (38 to 40); 4.9 returns EINVAL for them. A capability the kernel does not know cannot be in the bounding set, so the system/zygote fork, d6dadb5, skips it, as the C++ zygote already does. Chrome opens pages and its sandboxed renderers start.

Step 10 — CHRE and Active Edge

One service still restarted about every 5 s, 146 times in the first 14 minutes:

dynamic loading failed for chre_slpi on domain 0 (dlerror undefined symbol #25 __sensors_island_start in ./libchre_slpi_skel.so)

The sensor DSP firmware on the phone rejects the CHRE program from the vendor blobs. Its only user is Active Edge, the squeeze gesture. Bonito 7f1369f drops ElmyraService, and 3f29377 stops vendor.chre once boot completes. The accelerometer and light sensor keep reporting without it.

Step 11 — Google Apps

MindTheGapps 17.0.0 ships Play Services as an APEX whose payload is EROFS, a filesystem the 4.9 kernel cannot mount:

apexd: Mounting failed ... No such device

repack-gapps.sh rebuilds the zip with the same APK installed as /product/priv-app/PrebuiltGmsCoreVic, with the APEX's permission files. On sargo it needs 682 MiB of the 900 MiB /product reserve.

Flashed onto a phone that is already set up, Google apps never get their default runtime permissions, and Play Services refuses location requests. post-install-24.sh grants them, then runs the background compile job. After an update every app sits at the verify filter until the phone is idle and charging. Gmail, one run each:

Gmail	Start	Janky frames scrolling
Uncompiled (`verify`)	1,372 ms	3.37%
Compiled (`speed-profile`)	🥇 1,010 ms	🥇 0.09%

The compile job took 2 min 53 s.

Step 12 — Install and Update Without Wiping

update_engine installs a full build into the inactive slot from the running system and keeps /data:

10:59:56 update_engine install to the inactive slot
[INFO:update_engine_client_android.cc(102)] onPayloadApplicationComplete(ErrorCode::kSuccess (0))
[INFO:update_engine_client_android.cc(94)] onStatusUpdate(UPDATE_STATUS_UPDATED_NEED_REBOOT (6), 0)
[INFO:update_engine_client_android.cc(367)] Command took 300501 ms
11:04:57 installed; not rebooting

On a build machine, phone-update-24.sh syncs, builds, installs, sideloads the repacked Google apps and compiles the apps. On a laptop with only adb, phone-install-24.sh does the same from the release zip; INSTALL.md has the steps. The one manual step is in recovery: accept the test-key signature warning, then reboot.

The phone after the update:

932AY05KVV             device usb:5-2 product:sargo model:Pixel_3a device:sargo transport_id:34
24.0-20261027-UNOFFICIAL-sargo
4.9.337-gf002f96f2646
Enforcing
1
_b
17
2026-10-01
Wifi is enabled

🔎 Tip: list every fork in the local manifest. repo sync resets any project the local manifest does not replace to LineageOS upstream, and a build from that tree loses the fork commits. On this phone that build went straight back to enablefilecrypto_failed. update-forks.sh reports how far each fork is behind upstream and rebases it.

Compare and Contrast

	LineageOS 23.2 on this kernel	LineageOS 24.0 on this kernel
Android	16	🥇 17
Kernel changes	three backports from 4.14 and 4.17	23.2's plus one cgroup BPF fix
Platform forks	none	five: core, vold, bpf, Connectivity, zygote
SELinux	🟢 enforcing	🟢 enforcing, with ION rules in vendor policy
Google apps	🟢 MindTheGapps 16	⚠️ MindTheGapps 17, repacked
Active Edge	⚠️ not tested	❌ off
Signing	⚠️ test keys	⚠️ test keys

So, Which One?

For a spare Pixel 3a, 24.0 is the current one: Android 17, the October 2026 security patch level, and the same update path that keeps your data. Back up before moving from 23.2: the update tools here install over 24.0 builds only, and a first install wipes the phone.

Summary

The goal of this project was to run Android 17 on the Pixel 3a's original 4.9 kernel. The key to the solution was the same /init logger as the 23.2 port, now with logcat and a task dump on a timed reboot, which turned each stopped boot into a log line. The results were:

🟢 LineageOS 24.0 boots on kernel 4.9.337 with SELinux enforcing in 83 s and installs over the previous build, keeping user data.
🟢 One kernel fix: cgroup_sk_clone() takes its cgroup_bpf reference. 2 of 9 boots hung before it; 8 of 8 booted after.
🟢 Five platform forks cover a kernel below 5.10: the fscrypt session keyring, the BPF loader, netd and Connectivity, and the native app zygote.
🟢 Wi-Fi connects, Bluetooth turns on, the sensors report, Chrome renders pages.
🟢 Play Services runs as a system app from repacked MindTheGapps 17, and network location gives a fix.
🟢 Compiling apps after an update brings Gmail from 1,372 ms to 1,010 ms to start.
⚠️ Signed with test keys; recovery shows a signature warning on every install.
🟢 Mobile data works from a Google Fi data-only SIM in the 20261104-1751 build, over IPv4 and IPv6; the four fixes are in the mobile data porting notes.
⚠️ Calls, SMS, audio by ear, taking photos, fingerprint enrolment, NFC payments and satellite GPS are untested.
❌ Active Edge is off: the sensor DSP firmware rejects the CHRE program.

Scope: one Pixel 3a (sargo, 932AY05KVV) with no SIM fitted, LineageOS 24.0 built 2026-10-27 from bonito lineage-24.0 at 3f29377 and the five lineage-24.0 forks, kernel 4.9.337 from this repository. The hang counts come from 9 and 8 boots of one phone; the Gmail figures are one run each. Parts of the analysis and writing were done with AI assistance (Claude); every log line comes from the phone or the build machine.

The strategy for running Android 17 on a 4.9 kernel was validated with an incremental step by step approach.

References

Release, build and kernel source: https://github.com/xbill9/android_kernel_google_msm-4.9/releases/tag/lineage-24.0-sargo-20261027
Build, install and boot-test notes: https://github.com/xbill9/android_kernel_google_msm-4.9/tree/main/backport-tools/lineage-24.0
Mobile data with Google Fi, porting notes: https://github.com/xbill9/android_kernel_google_msm-4.9/blob/main/backport-tools/lineage-24.0/PORTING-NOTES-mobile-data.md
Mobile data release: https://github.com/xbill9/android_kernel_google_msm-4.9/releases/tag/lineage-24.0-sargo-20261104-1751
Install from a laptop: https://github.com/xbill9/android_kernel_google_msm-4.9/blob/main/INSTALL.md
Device tree: https://github.com/xbill9/android_device_google_bonito/tree/lineage-24.0
Fork, system/core: https://github.com/xbill9/android_system_core
Fork, system/vold: https://github.com/xbill9/android_system_vold
Fork, system/bpf: https://github.com/xbill9/android_system_bpf
Fork, Connectivity: https://github.com/xbill9/android_packages_modules_Connectivity
Fork, system/zygote: https://github.com/xbill9/android_system_zygote
The 23.2 port: https://dev.to/gde/android-16-on-a-49-kernel-lineageos-232-on-the-pixel-3a-5e79
The EC2 build: https://dev.to/aws-builders/rent-the-cores-keep-the-phone-building-lineageos-24-on-ec2-and-iterating-at-home-4pmc
MindTheGapps 17.0.0 arm64: https://github.com/MindTheGapps/17.0.0-arm64

Gemma 4 E2B in JAX on One TPU v5e: bf16 or 4-Bit, Python or Rust, and How It Stands Against vLLM

xbill — Wed, 04 Nov 2026 22:49:08 +0000

This article provides a step by step guide to serving Gemma 4 E2B on a single Cloud TPU v5e chip with JAX. A pure-JAX engine serves the model from Python, a second engine builds the same graph in Rust and hands it to the libtpu compiler that JAX itself uses, and vLLM serves the same checkpoint as the reference. Every number below was measured on one flex-start v5e-1 VM on 2026-11-03, and every run is committed with its logs.

https://github.com/xbill9/jax-dev

Why One v5e Chip?

A v5e-1 is the smallest Cloud TPU you can rent: one chip, 16.909 GB of HBM, in a ct5lp-hightpu-1t VM. Gemma 4 E2B in bf16 puts 9.257 GB of weights on it, which leaves room for the KV cache with no quantization at all. That makes it the place to ask three questions with nothing else in the way:

Does 4-bit pay off for a model that already fits in bf16?
How much of the JAX engine's time is Python?
Where does a hand-written JAX engine stand against vLLM on the same chip?

At This Point You Should Have…

A Google Cloud project with the TPU API enabled and flex-start quota for v5e (TPUV5sLitepodPerProjectPerZoneForTPUAPI) in us-west4-a
An authenticated gcloud with the alpha components
The repository cloned, and its tpu-jax-v5e1-2b directory as the working directory

No Hugging Face token is needed: google/gemma-4-E2B-it and both 4-bit builds are public.

Step 1 — Provision a Flex-Start v5e-1

Each rig ships an MCP server, server.py, whose tools wrap the gcloud calls. The rig's tpu.env sets ACCELERATOR_TYPE=v5litepod-1, and the server derives the machine type from it. workload="jax" installs Python 3.13 and jax[tpu] on a bare VM with no Docker:

$ python3 -c 'import asyncio, server as s; print(asyncio.run(s.create_tpu_vm_instance(instance_name="jax-v5e1-2b", zone="us-west4-a", accelerator="v5litepod-1", workload="jax", max_run_duration="4h", request_valid_for="30m")))'
Executing gcloud command: gcloud compute instances create jax-v5e1-2b --project=<project-id> --zone=us-west4-a --machine-type=ct5lp-hightpu-1t --provisioning-model=FLEX_START --request-valid-for-duration=30m --max-run-duration=4h --instance-termination-action=DELETE ... --boot-disk-size=200GB --scopes=cloud-platform
🚀 Flex-start TPU VM `jax-v5e1-2b` (ct5lp-hightpu-1t, 1 chip(s)) created in us-west4-a; installing Python 3.13 + `jax[tpu]`.
NAME         ZONE        MACHINE_TYPE      PREEMPTIBLE  STATUS
jax-v5e1-2b  us-west4-a  ct5lp-hightpu-1t               RUNNING

wait_for_jax_ready watches the serial console until the startup script has seen a TPU device from JAX:

🟢 JAX environment ready on `jax-v5e1-2b`. Verify with `verify_jax_tpu`; run workloads over SSH with `python3.13`.

$ python3.13 -c "import jax; print(jax.__version__, jax.devices())"
0.12.0.1 [TpuDevice(id=0, process_index=0, coords=(0,0,0), core_on_chip=0)]

The flex-start VM deletes itself after --max-run-duration, so a forgotten VM stops billing on its own.

Step 2 — Load E2B Into the Pure-JAX Engine

jax_engine.py is a Gemma 4 E2B decoder written directly in JAX: jitted prefill, a static KV cache, and one jitted decode step per token. It reads the Hugging Face safetensors itself, so no PyTorch is installed. --quant-mode fp16 is its dense path, which keeps the bf16 weights as they are:

$ python3.13 jax_openai_server.py --model google/gemma-4-E2B-it --quant-mode fp16 --max-model-len 2048
JAX devices: [TpuDevice(id=0, process_index=0, coords=(0,0,0), core_on_chip=0)]
Loading tokenizer: google/gemma-4-E2B-it
Loaded 9.26 GB of parameters on TPU_0(process=0,(0,0,0,0)) in 5.4s (KV cache: bf16)
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

The server speaks the OpenAI chat API, streamed or not, so any OpenAI client can talk to it.

Step 3 — bf16 or 4-Bit?

Two 4-bit builds of E2B load into the same engine with --quant-mode w4a16: Google's google/gemma-4-E2B-it-qat-w4a16-ct, and xbill9/gemma-4-E2B-it-qat-q4_0-w4a16-ct-text, a repack that stores the QAT grid exactly. tools/ab_bench.py runs each in its own process on 8 prompts, 128 greedy tokens each, one untimed run and three timed:

loaded google/gemma-4-E2B-it in 64.7s, weights 9.257 GB, HBM 9.260 / 16.909 GB
loaded xbill9/gemma-4-E2B-it-qat-q4_0-w4a16-ct-text in 39.2s, weights 6.559 GB, HBM 6.561 / 16.909 GB
loaded google/gemma-4-E2B-it-qat-w4a16-ct in 51.9s, weights 6.559 GB, HBM 6.562 / 16.909 GB

arm                                                tok/s  x ref weights GB  HBM GB
google/gemma-4-E2B-it (fp16)                       132.6  1.000      9.257   9.260
xbill9/gemma-4-E2B-it-qat-q4_0-w4a16-ct-text (w4a16)   103.0  0.777      6.559   6.561
google/gemma-4-E2B-it-qat-w4a16-ct (w4a16)         103.1  0.777      6.559   6.562

bf16 decodes 1.29x faster than either 4-bit build and prefills in 7.4 ms against 23.2 ms. The 4-bit builds save 2.70 GB of HBM. This engine unpacks the int4 weights inside every matmul, and on a chip where bf16 already fits, that unpacking costs more time than the smaller weights save.

The repack and Google's export tie on speed and size. Their outputs differ: none of the 8 completions match token for token, and they agree for 26.6 of 128 tokens on average. Which of the two is closer to the trained QAT weights is measured in the earlier article listed under References.

Step 4 — The Same Graph, Built From Rust

JAX traces Python into StableHLO and hands it to the TPU's PJRT plugin, libtpu.so, to compile and run. The tpu-jaxrust-v5e1-2b rig writes that StableHLO directly. rust/src/gemma.rs builds the forward pass op by op, following jax_e_model.py down to where JAX promotes bf16 to f32:

fn rms_norm(b: &mut Builder, x: &V, w: Option<&V>, eps: f64) -> V {
    let dt = x.dt;
    let last = x.rank() as i64 - 1;
    let d = x.dim(-1) as f64;
    let xf = b.convert(x, DType::F32);
    let sq = b.mul(&xf, &xf);
    let sum = b.reduce_sum(&sq, &[last]);
    let var = b.scale(&sum, 1.0 / d);
    let var = b.add_scalar(&var, eps);
    let r = b.rsqrt(&var);
    ...

rust/src/hlo.rs prints the module as MLIR text, and rust/src/pjrt.c, a small C shim over the vendored PJRT C API header, opens the same libtpu.so that JAX loaded and compiles it. A PyO3 module, jaxrust_rs, hands the result to Python for the tokenizer:

platform {'plugin': '/usr/local/lib/python3.13/dist-packages/libtpu/libtpu.so', 'header_api': (0, 117), 'plugin_api': (0, 116), 'devices': 1}
build {'rustc': 'rustc 1.101.0-nightly (1c3a65069 2026-11-02)', 'pjrt_header': '0.117', 'crate': '0.1.0'}
engine Engine(context=256, weight_bytes=9257336832, cache_bytes=4718592) in 8.4s, {'load_s': 2.318838774, 'compile_s': 5.73853671}

The weights upload in 2.3 s and the decode program compiles in 5.7 s. There is no JAX in this path, and the Rust crate has no ML dependencies: pyo3, safetensors, memmap2 and serde_json.

Step 5 — Does Rust Produce the Same Tokens as JAX?

tools/parity.py runs each engine in its own process on the same 8 prompts, with the same chat template and the same leading <bos>, and compares the last-position logits after the first prompt and every greedy token:

logits after prompt 0: max |diff| 0.2306, mean |diff| 0.07346, argmax 7216 vs 7216, top-10 overlap 10/10, KL(ref||got) 1.31e-06
prompts tokenized the same: 8 of 8; completions identical: 7 of 8; matching prefix per prompt: [128, 128, 128, 128, 128, 128, 42, 128] (mean 117.2 of 128)
decode tok/s (median over prompts): jax 131.6, rust 150.7, ratio 1.145
weights on device: jax 9.257 GB, rust 9.257 GB

Both engines pick the same next token with the same top 10, 1.31e-6 apart in KL divergence. Seven of eight 128-token completions match token for token; the eighth parts at token 42, where "Time until 4:00 pm" and "Next full hour: 4:00 pm" were nearly tied. Timed inside each engine, Rust decodes 1.145x faster on identical weights.

Step 6 — Serve All Three Through One Client

The Rust engine gets an OpenAI-style server of its own, rust_openai_server.py, and vLLM 0.33 runs from the vllm/vllm-tpu image on the same VM. Each server takes port 8000 alone, and tools/serve_bench.py times all three the same way: streamed chat, one request at a time, temperature 0, exactly 128 tokens (ignore_eos), context 2048, 8 prompts with one untimed and three timed requests each:

$ python3 tools/serve_bench.py compare vllm.json jax.json rust.json
engine      decode tok/s           range  first token ms  tokens  same text as vllm
vllm               148.7   148.7-148.7              13.0     128              8 of 8
jax                126.7   126.0-128.4              12.5     128              6 of 8
rust               148.4   147.8-148.6               8.9     128              6 of 8
prompt tokens per engine: {'vllm': [20, 23, 23, 21, 20, 24, 35, 17], 'jax': [20, 23, 23, 21, 20, 24, 35, 17], 'rust': [20, 23, 23, 21, 20, 24, 35, 17]}
repeats identical: {'vllm': 8, 'jax': 8, 'rust': 8}
ratio jax / vllm: 0.852
ratio rust / vllm: 0.998

The Rust engine decodes at vLLM's rate, 148.4 against 148.7 tokens per second, and reaches the first token in 8.9 ms against 13.0. The Python JAX engine decodes at 0.852x vLLM with a first token level with it. All three tokenize every prompt to the same length, and each repeats its own text exactly. The JAX and Rust engines part from vLLM at the same two near-ties, in prompts 2 and 6.

🔎 Tip: Time Every Engine by the Same Clock

Engines report their own speed in their own way: inside the decode loop, per HTTP response, with or without the first token. One client gives every engine the same definition. In serve_bench.py, decode speed is completion tokens minus one, over the time from the first text chunk to the finish chunk, with the token count read from the server's usage chunk. The clock stops at the finish chunk because, past the end of the answer, ignore_eos tokens can decode to empty text. The same Rust engine reads 150.7 tokens per second timed inside the engine at context 256, and 148.4 through this client at context 2048.

🔎 Tip: libtpu Wants One Replica and One Partition

PJRT's compile call takes a serialized xla.CompileOptionsProto. With empty options, libtpu asks for zero cores and refuses to compile ("Requested total cores: 0"). Six bytes set one replica and one partition:

/// `xla.CompileOptionsProto { executable_build_options { num_replicas: 1, num_partitions: 1 } }`
/// serialized by hand: field 3 (length 4) holding fields 4 and 5, varint 1 each.
const COMPILE_OPTIONS: [u8; 6] = [0x1a, 0x04, 0x20, 0x01, 0x28, 0x01];

Two rules of the MLIR text format surface at the same compile call: a float literal needs a decimal point (2.0e0), and an empty integer array is written array<i64>.

Compare and Contrast

One v5e chip, E2B	JAX engine, bf16	JAX engine, 4-bit	Rust engine, bf16	vLLM 0.33, bf16
Decode tok/s, timed in the engine	🥈 131.6	🥉 103.0	🥇 150.7	—
Decode tok/s, one client	🥉 126.7	not run	🥈 148.4	🥇 148.7
First token, one client	12.5 ms	not run	🥇 8.9 ms	13.0 ms
Weights on the chip	9.257 GB	🥇 6.559 GB	9.257 GB	—
Same text as vLLM	6 of 8	not run	6 of 8	—
Concurrent requests	no	no	no	yes

So, Which One?

For one user at a time, the Rust engine: it decodes at vLLM's rate, answers first, and puts the same 9.257 GB on the chip. For JAX work in Python, the bf16 JAX engine: 0.852x vLLM, with the whole model written in JAX you can read and change. For more than one user, vLLM, which batches requests and is the only one of the three that does. On this chip keep E2B in bf16: the 4-bit builds save 2.70 GB and decode at 0.777x its speed in this engine.

Summary

The goal of this article was to measure Gemma 4 E2B in JAX on one TPU v5e chip: bf16 against 4-bit, Python against Rust, and both against vLLM. The key to the solution was holding everything else fixed: one VM, one checkpoint revision, the same 8 prompts through each server's own chat template, and one client timing every engine. The results were:

🟢 The Rust engine, the same graph built from Rust and compiled by JAX's own libtpu, decodes at vLLM's rate (0.998x) with the fastest first token, 8.9 ms
🟢 It matches the JAX engine: same argmax and top 10, KL 1.31e-6, 7 of 8 completions token for token
🟢 bf16 E2B fits a v5e chip (HBM 9.260 / 16.909 GB in use) and decodes 1.29x faster than 4-bit in the JAX engine
⚠️ The Python JAX engine decodes at 0.852x vLLM over HTTP; where its time goes was not measured
⚠️ The repack and Google's 4-bit export run at the same speed and size, with different outputs
❌ Neither JAX engine takes concurrent requests, so vLLM's batching is not compared here

Scope: one flex-start ct5lp-hightpu-1t VM in us-west4-a (one TPU v5 lite chip, 16.909 GB HBM) on 2026-11-03; JAX 0.12.0.1 and libtpu 0.0.51 for both JAX-side engines, vLLM 0.33.0rc1.dev430 from vllm/vllm-tpu@sha256:f1f925f5…; google/gemma-4-E2B-it at revision 4adf1574. Every figure is the median over 8 prompts of the median of 3 timed runs, greedy, 128 tokens, one request at a time. The in-engine runs used context 4096 (bf16 against 4-bit: 132.6 tok/s for bf16) and 256 (JAX against Rust: 131.6 for bf16); the one-client run used 2048 for all three. Parts of the analysis and writing were done with AI assistance (Claude); every figure comes from the committed run files.

The strategy for serving Gemma 4 E2B with JAX on a single TPU v5e chip was validated with an incremental step by step approach.

References

The monorepo: https://github.com/xbill9/jax-dev
The pure-JAX rig: https://github.com/xbill9/jax-dev/tree/main/tpu-jax-v5e1-2b
The Rust rig: https://github.com/xbill9/jax-dev/tree/main/tpu-jaxrust-v5e1-2b
bf16 against 4-bit run: https://github.com/xbill9/jax-dev/tree/main/tpu-jax-v5e1-2b/benchmarks/runs/2026-11-03-ab-v5e1
JAX against Rust run: https://github.com/xbill9/jax-dev/tree/main/tpu-jaxrust-v5e1-2b/benchmarks/runs/2026-11-03-parity-v5e1
One-client run: https://github.com/xbill9/jax-dev/tree/main/tpu-jaxrust-v5e1-2b/benchmarks/runs/2026-11-03-serve-v5e1
The repack against Google's 4-bit export, on a Colab v5e: https://dev.to/gde/gemma-4-e2b-in-pure-jax-on-a-colab-tpu-googles-4-bit-export-against-an-exact-repack-4dle
The model: https://huggingface.co/google/gemma-4-E2B-it
The repack: https://huggingface.co/xbill9/gemma-4-E2B-it-qat-q4_0-w4a16-ct-text
Google's 4-bit export: https://huggingface.co/google/gemma-4-E2B-it-qat-w4a16-ct
StableHLO specification: https://openxla.org/stablehlo/spec
PJRT C API header: https://github.com/openxla/xla/blob/261d4d4979f65a89b614a6a5baae1790ce51f048/xla/pjrt/c/pjrt_c_api.h

The A2A CLI and Rust: The Same Code, Three Ways to Serve It

xbill — Wed, 04 Nov 2026 01:04:48 +0000

This article provides a step by step guide to the A2A CLI, driven against Rust agents that speak A2A Protocol v1.0. The same Rust code, a Lucas-Lehmer test over the first N Mersenne prime exponents, is served three ways: on the official A2A Rust SDK, as a plain stdin/stdout program wrapped by a2a server --exec, and behind an ADK coordinator on Gemini started with the ADK CLI. Each one is timed from the A2A CLI and from a Rust client.

https://github.com/xbill9/rust-dev/tree/main/a2a-benchmark-rust

The A2A CLI talks to all three with the same commands, and a2a server --exec turns a 30-line program with no A2A code into an agent that ADK can call. One a2a send from a shell takes a median 20.74 ms against the SDK agent, of which 7.62 ms is starting the CLI. From a Rust client the SDK agent adds 0.55 to 1.40 ms per call, the wrapped program 3.33 to 5.05 ms, and the ADK coordinator a median 1328 ms on gemini-3.5-flash and 2283 ms on gemini-3.8-flash. Every call through N=26 returned the right count of primes.

What Is Being Built?

Agent	Port	How it speaks A2A
`mersenne-agent`	8104	Official Rust SDK, JSON-RPC at `/jsonrpc` and HTTP+JSON at `/rest`
`mersenne-exec`	8105	A stdin/stdout Rust program served by `a2a server --exec`
`rust_bench`	8106	ADK `LlmAgent` on Gemini, served by `adk api_server --a2a`
echo	8108	`a2a server --echo`, for the first command

a2a CLI ──► adk api_server --a2a (:8106) ──Gemini──► RemoteA2aAgent ──► mersenne-agent (:8104)
                    rust_bench                                     └──► a2a server --exec (:8105) ──► mersenne-exec
                        └── run_benchmark tool ──► mersenne-bench ──► both Rust agents

Versions, read from the machine on 2026-11-03:

Component	Version
A2A protocol	1.0
A2A CLI (`a2a`)	v0.3.1
`a2a-lf` / `a2a-server-lf` / `a2a-client-lf`	0.4.2 / 0.5.3 / 0.2.9
Rust	1.99.1, edition 2024
`google-adk` / `a2a-sdk` / `google-genai`	2.13.0 / 1.3.0 / 2.34.0
Model	`gemini-3.8-flash` on Vertex AI

The a2a-*-lf crates are the official SDK from a2aproject/a2a-rs. Agent cards carry the protocol as Major.Minor, so they read 1.0.

Step 1 — Install the A2A CLI

The CLI is one Go binary. Homebrew, WinGet and release binaries work; from source:

go install github.com/a2aproject/a2a-cli@latest
mv "$(command -v a2a-cli)" "$(dirname "$(command -v a2a-cli)")/a2a"
a2a version

a2a v0.3.1
commit:  none
built:   unknown
go:      go1.26.3
os/arch: linux/amd64

go install names the binary a2a-cli; the docs and the CLI's own skill call it a2a.

Step 2 — Talk to an Echo Server

a2a server --echo starts an agent that sends back what it receives, so the client can be checked before any agent code exists:

a2a server --echo --port 8108 &
a2a send -a http://localhost:8108 "hello world from A2A"

Task:     01a1a35f-2b2b-7f3a-92a2-e3dc6c69e359
Context:  01a1a35f-2b2c-7120-ac45-a74adab9d1d2
Status:   completed (2026-11-03T16:44:45-05:00)
Artifacts:
  [01a1a35f-2b2c-71ba-8e6c-22c4b7c210df] hello world from A2A
History:
  [user] hello world from A2A

Every send creates a task with an id, a context, a final status and its artifacts. The Go servers started by the CLI stamp local time; the Rust SDK agent stamps UTC.

Step 3 — Build and Start the Rust Agents

The rig is one Cargo project with four binaries: the SDK agent, the stdin/stdout program, the benchmark client, and a shared library holding the Lucas-Lehmer code.

git clone https://github.com/xbill9/rust-dev
cd rust-dev/a2a-benchmark-rust
cargo build --release
scripts/start-all.sh

agents up: 8104 native, 8105 exec

start-all.sh rebuilds, stops any running copy, starts the SDK agent on 8104 and a2a server --exec on 8105 with logs in logs/, and waits until both cards answer. scripts/test-cli.sh runs the CLI commands below against them and checks each reply.

Step 4 — Read an Agent Card

card get fetches /.well-known/agent-card.json and prints what the agent offers:

a2a card get -a http://localhost:8104

Name:         Mersenne Prime Agent Rust
Description:  A Rust A2A v1.0 agent (official a2a-rs SDK) that confirms the first n Mersenne primes with the Lucas-Lehmer test and reports the elapsed time.
Version:      0.1.0
Interfaces:
  JSONRPC (A2A v1.0)     http://localhost:8104/jsonrpc
  HTTP+JSON (A2A v1.0)   http://localhost:8104/rest
Streaming:    true
Skills:
  find-mersenne-rust   Find Mersenne Primes in Rust

The card lists two interfaces. The CLI picks one, and --transport chooses.

Step 5 — Send, Stream, Switch Transport, Send Data

A plain message:

a2a send -a http://localhost:8104 "Calculate the first 12 Mersenne primes"

Status:   completed (2026-11-03T21:44:45Z)
  Found first 12 Mersenne primes in 0.0902ms.
Artifacts:
  [mersenne-report] Found first 12 Mersenne primes in 0.0902ms. {"elapsed_ms":0.090239,"exponents":[2,3,5,7,13,17,19,31,61,89,107,127],"found":12,"requested":12}

The mersenne-report artifact carries two parts: the summary as text, and the same result as a data part, so a client reads the timing from a field.

--stream prints each event as it arrives, here the working status before the result:

a2a send -a http://localhost:8104 --stream "first 8"

[status] working
Task:     01a1a35f-2b64-77b5-9417-18b1f425b17d
Context:  01a1a35f-2b64-77b5-9417-18b24253b719
Status:   completed (2026-11-03T21:44:45Z)
  Found first 8 Mersenne primes in 0.0164ms.

--transport rest sends the same request over HTTP+JSON, and --data-part sends structured input:

a2a send -a http://localhost:8104 --transport rest "first 4"
a2a send -a http://localhost:8104 --data-part '{"count": 6}'

  Found first 4 Mersenne primes in 0.0080ms.
  Found first 6 Mersenne primes in 0.0108ms.

🔎 Tip: Data Parts Arrive as Floats

A2A v1.0 carries data parts as a protobuf Struct, whose only number type is a double. {"count": 6} sent by the CLI reaches the agent as 6.0, so a strict integer field fails to deserialise. Read numbers with as_f64 and check for a whole value:

if let PartContent::Data(v) = &part.content
    && let Some(n) = v.get("count").and_then(|n| n.as_f64())
    && n >= 1.0
    && n.fract() == 0.0
{
    return n as usize;
}

Step 6 — Script It With JSON Output

-o json prints the whole task, which pipes into anything:

a2a send -a http://localhost:8104 -o json 'first 3' | python3 -c 'import sys,json; t=json.load(sys.stdin)["task"]; print(t["status"]["state"], [p for a in t["artifacts"] for p in a["parts"] if "data" in p][0]["data"])'

TASK_STATE_COMPLETED {'elapsed_ms': 0.007166, 'exponents': [2, 3, 5], 'found': 3, 'requested': 3}

Step 7 — Tasks: List, Start in the Background, Wait

The SDK agent keeps its tasks in memory, and the CLI reads them back:

a2a task list -a http://localhost:8104

ID                                    STATUS     CONTEXT
01a1a35f-2b4f-7190-b6df-81bb7b76c2cc  completed  01a1a35f-2b4f-7190-b6df-81bca35e8b6d
01a1a35f-2b64-77b5-9417-18b1f425b17d  completed  01a1a35f-2b64-77b5-9417-18b24253b719
01a1a35f-2bab-7012-b094-1022dce75f36  completed  01a1a35f-2bab-7012-b094-1023824b35eb

--async returns as soon as the task exists. N=25 takes over a second, long enough to catch it running:

a2a send -a http://localhost:8104 --async "first 25"

Task:     01a1a3f5-9da1-746b-abae-84307e4e1735
Context:  01a1a3f5-9da1-746b-abae-8431b422e72e
Status:   working (2026-11-04T00:29:05Z)
History:
  [user] first 25

task get --wait polls until the task finishes:

a2a task get -a http://localhost:8104 01a1a3f5-9da1-746b-abae-84307e4e1735 --wait --poll-interval 500ms

Task:     01a1a3f5-9da1-746b-abae-84307e4e1735
Context:  01a1a3f5-9da1-746b-abae-8431b422e72e
Status:   completed (2026-11-04T00:29:06Z)
  Found first 25 Mersenne primes in 1509.7683ms.

Step 8 — The Native Rust Agent

The SDK agent is one AgentExecutor. It emits a working status, runs the big-integer work on a blocking thread, then returns the finished task:

let done = async move {
    // Big-integer work is CPU-bound; keep it off the async workers.
    let report = tokio::task::spawn_blocking(move || find_mersenne_primes(count))
        .await
        .map_err(|e| A2AError::internal(format!("worker panicked: {e}")))?;
    let summary = report.summary();
    let data = serde_json::to_value(&report)
        .map_err(|e| A2AError::internal(format!("serialise report: {e}")))?;
    // ... Task { status: Completed, artifacts: [Part::text(summary), Part::data(data)] }
};
Box::pin(stream::once(async { Ok(working) }).chain(stream::once(done)))

The server wires the SDK's routers into axum, one per transport plus the card:

let handler = Arc::new(DefaultRequestHandler::new(MersenneExecutor, InMemoryTaskStore::new()));
let card = Arc::new(StaticAgentCard::new(agent_card(&base)));

let app = axum::Router::new()
    .nest("/jsonrpc", a2a_server::jsonrpc::jsonrpc_router(handler.clone()))
    .nest("/rest", a2a_server::rest::rest_router(handler))
    .merge(a2a_server::agent_card::agent_card_router(card));

The Lucas-Lehmer test runs on num-bigint with the reduction mod 2^p − 1 done by folding the high bits onto the low ones. The timed region holds only the tests, on a monotonic clock, and no prime is ever turned into a decimal string. Two slow tests check the arithmetic: one runs the test on every prime exponent up to 9941 and asserts it accepts exactly the first 22 known ones, the other confirms all 26:

cargo test --release --lib -- --ignored

test tests::all_26_exponents_prime ... ok
test tests::exhaustive_through_n22 ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 5 filtered out; finished in 25.70s

Step 9 — Serve a Plain Program With `a2a server --exec`

mersenne-exec knows nothing about A2A. It reads the message on stdin and prints two lines:

echo 'first 5' | target/release/mersenne-exec

{"requested":5,"found":5,"elapsed_ms":0.007023000000000001,"exponents":[2,3,5,7,13]}
Found first 5 Mersenne primes in 0.0070ms.

The A2A CLI makes it an agent:

a2a server --exec "$PWD/target/release/mersenne-exec" --chunk=$'\n' \
  --name "Mersenne Prime Exec Rust" \
  --advertise-address "http://localhost:8105" --port 8105

The same client commands work against it:

a2a send -a http://localhost:8105 --stream "first 8"

Status:   submitted
History:
  [user] first 8
[status] working
[artifact] {"requested":8,"found":8,"elapsed_ms":0.01543,"exponents":[2,3,5,7,13,17,19,31]}Found first 8 Mersenne primes in 0.0154ms.
[status] completed

The exec server starts the program once per message, hands it the text, and returns stdout as the artifact.

🔎 Tip: Two `--exec` Flags ADK Needs

--advertise-address http://localhost:8105. The exec server's card names 127.0.0.1 by default, and ADK's RemoteA2aAgent refuses a card whose RPC URL has a different origin from the one the card came from.
--chunk=$'\n'. With it, each output line streams as its own chunk. Without it, stream requests get UNSUPPORTED_OPERATION: the Go CLI falls back to polling and ADK's Python client fails. ADK keeps only the last chunk, so mersenne-exec prints the JSON line first and the summary last.

Step 10 — Put ADK in Front

ADK needs its Python package and Gemini on Vertex AI, signed in with application-default credentials:

pip install -U "google-adk[a2a]"
gcloud auth application-default login
source scripts/adk-env.sh

adk-env.sh sets GOOGLE_GENAI_USE_VERTEXAI=TRUE, GOOGLE_CLOUD_LOCATION=global, the project, and MODEL_NAME, which defaults to gemini-3.8-flash.

adk/rust_bench/agent.py is an ADK LlmAgent with both Rust agents as RemoteA2aAgent sub-agents and one tool:

native_agent = RemoteA2aAgent(
    name="native_rust_agent",
    description="Rust A2A v1.0 agent on the official a2a-rs SDK. ...",
    agent_card=f"{NATIVE_URL}{AGENT_CARD_WELL_KNOWN_PATH}",
)

root_agent = LlmAgent(
    name="rust_bench",
    model=os.getenv("MODEL_NAME", "gemini-3.8-flash"),
    tools=[run_benchmark],
    sub_agents=[native_agent, exec_agent],
)

The ADK CLI runs it three ways:

adk run rust_bench                                # terminal
adk web --port 8107                               # dev UI
adk api_server --a2a --port 8106 adk              # A2A server, from the rig root

adk api_server --a2a serves each agent folder that holds an agent.json card, at /a2a/<name>; scripts/start-adk.sh starts it in the background. The A2A CLI then treats the coordinator like any other agent:

a2a send -a http://localhost:8106/a2a/rust_bench/.well-known/agent-card.json \
  --timeout 120s "Find the first 9 Mersenne primes with the exec agent"

Status:   completed (2026-11-03T21:44:51Z)
Artifacts:
  [f2d2f27d-24ce-4120-8368-d966c99617a9] Found first 9 Mersenne primes in 0.0236ms.
History:
  [user] Find the first 9 Mersenne primes with the exec agent
  [agent] {"args":{"agent_name":"exec_rust_agent"},"id":"call_1332004","name":"transfer_to_agent"}
  [agent] {"id":"call_1332004","name":"transfer_to_agent","response":{"result":null}}
  [agent] {"requested":9,"found":9,"elapsed_ms":0.023635,"exponents":[2,3,5,7,13,17,19,31,61]}
  [agent] Found first 9 Mersenne primes in 0.0236ms.

The history shows the hand-off: Gemini calls transfer_to_agent, the Rust agent answers, and its JSON report comes back through two A2A hops.

Step 11 — Keep the Model Away From the Numbers

A benchmark through an LLM can time the model writing a number. Two rules keep the timing in code:

The benchmark client checks every reply. mersenne-bench reads the timing from the agent's data part or JSON line, rejects a report where requested differs from N or found differs from min(N, 26), and rejects an LLM-path reply with no transfer_to_agent to a Rust agent. Each row records where its report came from and who it was routed to.
ADK benchmarks go through a tool. run_benchmark runs mersenne-bench and returns medians already computed and rounded. The instruction tells Gemini to quote them, and scripts/check-adk-reply.py verifies a reply: one run_benchmark call, with the arguments asked for, and every number in the answer a value the tool returned.

A benchmark request sent from the CLI, saved with -o json. The tool call and the head of the answer:

a2a send -a $CARD --timeout 300s -o json "Benchmark both agents for N=3 to 6 with 2 reps" > reply.json

[tool call] {"max_n": 6, "min_n": 3, "reps": 2}
### Arguments Used
- `min_n`: 3
- `max_n`: 6
- `reps`: 2

The answer goes on with a summary per agent and a row per N. The check:

scripts/check-adk-reply.py reply.json 3 6 2

state: TASK_STATE_COMPLETED
run_benchmark calls: 1
args sent: {'max_n': 6, 'min_n': 3, 'reps': 2}  requested: {'min_n': 3, 'max_n': 6, 'reps': 2}
numbers in reply: 57  not from the tool: 0
PASS

scripts/adk-reliability.sh sends ten benchmark requests with varied ranges and ten requests that name no agent. On gemini-3.8-flash, 10 of 10 benchmark replies passed the checker and 10 of 10 unnamed requests went to native_rust_agent.

The sweep covers the whole exponent table, N=1 to 26. Exponent 23209 is a 6,987-digit number, and the arithmetic stays in BigUint throughout.

Step 12 — Benchmark From a Rust Client

REPS=20 scripts/bench.sh          # native + exec, N=1..26
ADK=1 REPS=3 scripts/bench.sh     # adds the ADK series

mersenne-bench is built on the SDK's client crate, a2a-client-lf. Each call records the agent's own compute time and the client's round trip; overhead is the round trip minus compute, per call. Direct calls, medians of 20 reps:

N	native compute ms	native overhead ms	exec compute ms	exec overhead ms
1	0.001039	0.731	0.0004555	4.046
10	0.04249	0.793	0.04200	4.391
20	26.33	0.970	60.21	4.374
22	181.1	1.255	264.6	3.558
24	781.1	1.073	868.9	3.335
26	2484	1.396	2619	3.588

Through ADK, medians of 3 reps, one run per model. All 26 rows in each run went to native_rust_agent and read its data part:

N	3.5-flash compute ms	3.5-flash round trip ms	3.8-flash compute ms	3.8-flash round trip ms
1	0.002051	1665.053	0.002615	2073.666
10	0.04513	1394.245	0.04390	2458.155
20	72.95	1332.994	68.52	2336.569
22	279.8	1868.178	287.3	2679.324
26	2226	3455.492	2241	4123.353

The slowest single call on gemini-3.8-flash took 29.0 s, at N=9.

Step 13 — Time the A2A CLI Itself

scripts/cli-bench.py runs one a2a send -o json process per call, the way a shell script would, and applies the same checks to each reply. Wall time covers starting the CLI, fetching the card, sending, and exiting. Two reference commands, 20 runs each:

Command	Median ms	Min ms	Max ms
`a2a version`	7.17	5.57	9.62
`a2a card get` (native)	9.18	8.30	11.17

The full sweep, N=1 to 26, 5 reps direct and 3 through ADK:

Agent	Calls	Overhead per call, per-N medians	Median of the per-N medians
native	130	10.92 to 21.82 ms	20.28 ms
exec	130	9.43 to 20.81 ms	17.51 ms
ADK, 3.8-flash	78	1825.22 to 3017.92 ms	2217.55 ms

Every ADK call went to native_rust_agent; the slowest took 22693.6 ms. The sweep times each agent in its own block, so native and exec ran minutes apart. Run in alternating order, N=1, 30 runs each (scripts/cli-transport.py):

native, default (JSON-RPC): median 20.74 ms, min 18.60, max 26.51
native, --transport rest: median 21.49 ms, min 19.63, max 26.15
exec (HTTP+JSON only): median 20.81 ms, min 18.67, max 28.07
a2a version: median 7.62 ms, min 6.07, max 10.71

From a shell, the transport and the serving path make no difference to the CLI's cost: about 21 ms per call, about a third of it starting the process. Against the Rust client's 0.55 to 1.40 ms, the CLI fits scripts and checks, and the Rust client fits a benchmark.

The CLI's overhead in the sweep drops to about 11 ms for both agents once compute passes 100 ms (N=21 and up). Why is untested.

🔎 Tip: Compare Compute Only Between Runs Paced Alike

The same native process at N=20 takes 26.07 ms when calls arrive back to back and 68.09 ms when each call follows a 1.3 s idle gap (mersenne-bench --gap-ms 1300). ADK's model round trip spaces calls out too, and its mid-range compute times sit close to the gap series: 68.52 ms at N=20 on gemini-3.8-flash. The CPU-level cause, clock scaling or core placement on a hybrid laptop CPU, is untested.

Compare and Contrast

Serving path	Code the agent needs	Overhead from a Rust client	`a2a send` from a shell, N=1
Official Rust SDK 🥇	an `AgentExecutor` and three routers	0.55 to 1.40 ms	20.74 ms
`a2a server --exec` 🥈	none: stdin in, stdout out	3.33 to 5.05 ms	20.81 ms
ADK coordinator 🥉	an `agent.py`, an `agent.json` card	1328 ms on 3.5-flash, 2283 ms on 3.8-flash	2664.00 ms on 3.8-flash

Rust-client ranges are the smallest and largest per-N medians; the ADK figures are the medians of its per-N medians. The exec figure includes starting a process for every message; how it splits between process start and the HTTP hop was not measured. At N=26 the work itself dominates every path: 2484 ms native and 2619 ms exec, direct.

So, Which One?

For an agent built to be one, use the SDK: about a millisecond of protocol cost, both transports, tasks the CLI can list and wait on.

For a program that already exists, a2a server --exec makes it an agent with no code change. A few milliseconds per call buys that, and ADK can call it once the two flags are set.

For routing by natural language, put ADK in front and keep the timing out of the model: a tool for anything computed, and a check that the model sent the arguments asked for.

For talking to any of them, the A2A CLI is the one client: card get, send, --stream, task get --wait, and -o json for scripts. Each call costs about 21 ms, so time agents from a long-lived client.

Summary

The goal of this article was to drive the A2A CLI against Rust agents on A2A v1.0, and to measure what each way of serving the same Rust code costs per call. The key to the solution was one benchmark client that reads timing from a structured report and checks that every reply answers the question asked. The results on one laptop were:

🟢 The A2A CLI drove all three agents with the same commands: card, send, stream, REST, data parts, JSON output, task list, --async and task get --wait
🟢 Every call through N=26, from the Rust client and from the CLI, reported found = min(N, 26)
🟢 The SDK agent added 0.55 to 1.40 ms per call from a Rust client
🟢 a2a server --exec served an A2A-unaware program to the CLI and to ADK
🟢 On gemini-3.8-flash, 10 of 10 benchmark replies quoted only tool values, and 10 of 10 unnamed requests went to the native agent
⚠️ One a2a send from a shell costs about 21 ms whatever the agent, 7.62 ms of it starting the CLI
⚠️ The wrapped program added 3.33 to 5.05 ms per call from a Rust client
⚠️ ADK added a median 1328 ms per call on gemini-3.5-flash and 2283 ms on gemini-3.8-flash, with one call at 29.0 s
❌ Compute times from calls spaced by idle gaps run slower and do not compare with back-to-back runs

Scope: one 13th Gen Intel Core i7-1360P laptop on Linux, release build, every agent and client on the same machine, 2026-11-03 and 2026-11-04. Rust-client series have 20 reps per N direct and 3 through ADK, with one run per model on Vertex AI's shared global endpoint, so the difference between the two Gemini models is a single sample. The CLI series has 5 reps per N direct and 3 through ADK, on gemini-3.8-flash only, and reliability was checked on that model only, ten requests of each kind. The gRPC transport, task cancel, task subscribe, push notifications and authentication were not exercised.

The strategy for serving and benchmarking Rust A2A agents with the A2A CLI was validated with an incremental step by step approach.

References

The rig: https://github.com/xbill9/rust-dev/tree/main/a2a-benchmark-rust
CLI tour transcripts: https://github.com/xbill9/rust-dev/blob/main/a2a-benchmark-rust/results/cli-tour-20261103T214445Z.md and https://github.com/xbill9/rust-dev/blob/main/a2a-benchmark-rust/results/cli-tour-20261104T002904Z.md
CLI timing: https://github.com/xbill9/rust-dev/blob/main/a2a-benchmark-rust/results/cli-bench-20261104T002328Z.md
Introducing the A2A CLI: https://a2a-protocol.org/latest/blog/2026/10/01/introducing-a2a-cli/
A2A CLI: https://github.com/a2aproject/a2a-cli
A2A Rust SDK: https://github.com/a2aproject/a2a-rs
Agent Development Kit: https://google.github.io/adk-docs/
Why Did My Benchmark Stop at N=22?: https://dev.to/gde/why-did-my-benchmark-stop-at-n22-a-debugging-story-in-nine-bugs-3m2l

Gemma 4 in Rust on TorchTPU: A Native TPU Rig With PyO3 and MCP

xbill — Mon, 02 Nov 2026 21:32:30 +0000

This article provides a step by step guide to serving Gemma 4 E2B from a Cloud TPU v5e-1 VM with the model written in Rust and every operation run by TorchTPU. A Claude Code MCP server provisions the TPU, the VM builds the Rust crate on boot and serves an OpenAI-compatible endpoint, and a local shim gives it a chat page and a second MCP server for demos. Python hosts the process and the HTTP server; the 35 layers, the ring cache and the decode loop are Rust.

https://github.com/xbill9/torchtpu-dev/tree/main/tpu-pytorchrust-v5e1-2b-q4w4a16emb4

The Rust model reaches TorchTPU through a 154-line C++ shim over ATen, so it builds against whichever PyTorch is installed: here Python 3.14, PyTorch 2.14.1 and a nightly Rust. It matches Hugging Face Transformers on 64 of 64 greedy tokens on CPU, boots on a fresh TPU VM in 4 minutes 29 seconds from installing Python to starting the service, and decodes at a median 12.20 tokens per second on the TPU.

What Is Being Built?

Gemma 4 in Rust on TorchTPU: Hands On With Colab ran the same Rust engine in a Colab notebook. This rig takes it to a TPU VM of its own and keeps it there:

Layer	Where	What
Capacity	Claude Code	An MCP server that creates a v5e-1 queued resource and reads its boot stages
Model	TPU VM	`torchtpu_rs`, a Rust crate built on the VM as a PyO3 extension
Kernels	TPU VM	ATen calls through PyTorch's dispatcher to TorchTPU
API	TPU VM, port 8000	`rust_openai_server.py`, OpenAI-compatible, streaming
Demo	This laptop	A shim on `127.0.0.1:8015` with a chat page, and a demo MCP server

The checkpoint is the W4A16 repack xbill9/gemma-4-E2B-it-qat-q4_0-w4a16-ct-text with the int4 vocabulary tables from its -text-emb4 twin. The directory name follows the rig naming scheme: tpu (Cloud TPU API), pytorchrust (the forward pass is Rust over PyTorch's ATen), v5e1, 2b, and q4w4a16emb4 for the encoding.

Versions on the VM, read from its /v1/engine: Python 3.14.10, PyTorch 2.14.1, torch_tpu 0.2.0, libtpu 0.0.51, Transformers 5.20.2, rustc 1.101.0-nightly.

Step 1 — Install the Rig

The rig is a Claude Code plugin with its own MCP server and skill:

git clone https://github.com/xbill9/torchtpu-dev
cd torchtpu-dev/tpu-pytorchrust-v5e1-2b-q4w4a16emb4
./project-setup.sh . --project <gcp-project-id>

Every setting lives in tpu.env: the zone, the two checkpoints, the Python and PyTorch versions the VM installs, the cache length and the firewall tags. PyTorch and TorchTPU come from public PyPI:

PYTHON_BIN=python3.14
TORCH_PIP_SPEC=torch==2.14.1 torch_tpu==0.2.0 transformers==5.20.2 compressed-tensors==0.19.0 'jinja2>=3.1' huggingface_hub fastapi uvicorn
SERVE_CONTEXT=8192
SERVE_DEFER_AND_FUSE=1

Python 3.14 is the newest version torch_tpu 0.2.0 ships wheels for. The Rust toolchain is named in the crate's rust-toolchain.toml, channel = "nightly", and the crate is on edition 2024.

Step 2 — Reach TorchTPU Without `tch`

TorchTPU registers its kernels on libtorch's dispatcher and ships no C++ API of its own, so Rust reaches it through ATen. The tch crate wraps every ATen operation in generated C++ that compiles against PyTorch 2.13.0 only. The model uses a few dozen operations, so the crate wraps those itself, in rust/src/aten.cpp:

#define RET(expr) TRY(return box(expr);) return nullptr;

extern "C" {
Tensor* aten_matmul(Tensor* a, Tensor* b) { RET(at::matmul(*a, *b)) }
Tensor* aten_linear(Tensor* x, Tensor* w) { RET(at::linear(*x, *w)) }
Tensor* aten_softmax(Tensor* a, int64_t d, int k) { RET(at::softmax(*a, d, kind(k))) }
Tensor* aten_index_copy_(Tensor* t, int64_t d, Tensor* idx, Tensor* src) { RET(t->index_copy_(d, *idx, *src)) }
// torch.Tensor <-> at::Tensor, through libtorch_python. Both directions share storage.
Tensor* aten_from_python(PyObject* o) { RET(THPVariable_Unpack(o)) }
PyObject* aten_to_python(Tensor* t) { TRY(return THPVariable_Wrap(*t);) return nullptr; }
}

rust/src/aten.rs gives them tch's method names, so the model code reads as before. build.rs asks the installed torch for its headers, libraries, C++ ABI and version, and compiles the shim against them with C++20, which PyTorch 2.14's headers require. The module records what built it:

>>> torchtpu_rs.BUILD
{'torch': '2.14.1+cpu', 'rustc': 'rustc 1.101.0-nightly (3e92e1a2d 2026-11-01)', 'python': '3.14.7', 'crate': '0.1.0'}

A module built for one PyTorch refuses to load under another, because the C++ ABI differs: import torchtpu_rs compares torch.__version__ with the version it was built against and raises ImportError on a mismatch.

Step 3 — The PyO3 Bindings

Every PyO3 item is in rust/src/bindings.rs; the model and weight code never see a Python object. rust/PYO3.md lists each crossing, and rust/python/torchtpu_rs.pyi is the type stub. Four rules shape the code.

Token ids are lists, tensors share storage. decode(ids: list[int], ...) returns list[int]; generated tokens never cross as tensors. The one tensor that crosses on every load is like, a torch.zeros(1, device="tpu") that names the device: Rust builds each weight on the host and moves it with type_as(like).

Long calls release the GIL; the callback takes it back. The server streams through a callback that runs once per token batch:

let done = py.detach(|| {
    aten::no_grad(|| {
        gemma.decode(&ids, max_new, &stop, read_every, |batch| {
            let Some(cb) = &on_tokens else { return true };
            Python::attach(|py| {
                let call = PyList::new(py, batch).and_then(|l| cb.call1(py, (l,)));
                match call.and_then(|r| r.is_truthy(py)) {
                    Ok(go_on) => go_on,
                    Err(e) => { failed = Some(e); false }
                }
            })
        })
    })
});

A false return ends the decode with finish="cancelled", and an exception in the callback is raised from decode. Generation stops at the checkpoint's eos_token_id list, [1, 106, 50].

The engine is #[pyclass(unsendable)]. It owns TPU tensors and one ring cache, so PyO3 checks the calling thread on every access. A call from another thread raises pyo3_runtime.PanicException, which derives from BaseException. The server builds the engine on one dedicated thread and queues every request to it:

class EngineWorker(threading.Thread):
    """Owns the one `torchtpu_rs.Engine`. Nothing outside this thread may touch it."""

Errors a caller can cause are ValueError. An empty prompt, max_new below 1, or a prompt longer than the cache are checked in bindings.rs before the model's own asserts.

tests/test_rust_bindings.py runs all of these on CPU against the real checkpoint:

test_bad_input_is_value_error ... ok
test_callback_exception_propagates ... ok
test_cancel_and_length ... ok
test_decode_stops_and_streams ... ok
test_gil_released_during_decode ... ok
test_module_surface ... ok
test_tensor_functions ... ok
test_unsendable ... ok
Ran 8 tests in 247.568s
OK

Step 4 — Check It Against Transformers

Before any TPU time, the Rust module and Hugging Face Transformers 5.20.2 run the same prompt on CPU, both on PyTorch 2.14.1, one process per side:

top5 rust [818, 8409, 102500, 4420, 16520] hf [818, 102500, 8409, 4420, 16520]
KL(hf||rs) 1.078e-05; same 64 of 64; first diff None

The five top candidates are the same set, the KL divergence of the first token's distribution is 1.1e-5, and all 64 greedy tokens match.

Step 5 — Provision the TPU

From Claude Code, the rig's MCP server creates a flex-start v5e-1 queued resource in us-west4-a. The startup script travels in the instance metadata with the crate and the server packed inside it, 47 KiB rendered, so the VM needs no SSH to get its code:

> create_tpu_queued_resource
🚀 Queued Resource gemma4-rust-qr creation initiated in us-west4-a, serving `xbill9/gemma-4-E2B-it-qat-q4_0-w4a16-ct-text-emb4`.

A TPU VM has no serial-console command, so the startup script writes each stage to a guest attribute and get_rig_progress reads it back without SSH. The boot on 2026-11-02:

STAGE: python
g++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
STAGE: torch
torch 2.14.1+cpu
STAGE: rust
STAGE: crate
rustc 1.101.0-nightly (3e92e1a2d 2026-11-01)
    Finished `release` profile [optimized] target(s) in 21.03s
STAGE: weights
STAGE: smoke
torchtpu_rs 0.1.0 matmul_sum 512.0 on tpu:0
TPU smoke OK
STAGE: unit
STAGE: serving

From installing Python to starting the service took 4 minutes 29 seconds. The smoke test is a Rust call on a tensor on the TPU, through the shim, checked for the right value and a result on tpu:0. The service then loads the weights in 6.8 seconds and runs a warmup decode in 18.8 seconds before /health answers.

🔎 Tip: Pin `jinja2>=3.1` on a TPU VM

The TPU image's own jinja2 3.0.3 sits in /usr/lib/python3/dist-packages, which is on the path of the deadsnakes Python. Transformers' chat templates need 3.1 or later and refuse to render with the older one:

ImportError: apply_chat_template requires jinja2>=3.1.0 to be installed. Your version is 3.0.3.

'jinja2>=3.1' in the pip spec puts a newer copy in /usr/local/lib/python3.14/dist-packages, which comes first on the path.

Step 6 — The OpenAI Endpoint

The VM serves /v1/chat/completions and /v1/completions, streamed or not, plus /v1/models, /health and /metrics. /v1/engine reports what is loaded and how it is bound:

"torch_tpu_defer_and_fuse": "1",
"load_seconds": 6.83,
"warmup_seconds": 18.8,
"layout": {"full_rings": 3, "sliding_rings": 12, "shared_layers": 20, "context": 8192},
"engine_class": "Engine — #[pyclass(unsendable)], owned by thread torchtpu-rs-engine"

Decoding is greedy. Each non-streamed response carries the engine's own seconds for reading the prompt and for decoding, and system_fingerprint reads torchtpu_rs-greedy.

Step 7 — Measure It

The same prompt and length as a vLLM run on the same chip type, Write a short paragraph about the ocean. at 100 tokens, five requests one at a time, rate from the engine's own timer:

request 1: 83 tokens, finish stop, read 0.088 s, decode 6.651 s, 12.33 tok/s, client 6.83 s
request 2: 83 tokens, finish stop, read 0.086 s, decode 6.720 s, 12.20 tok/s, client 6.89 s
request 3: 83 tokens, finish stop, read 0.086 s, decode 6.734 s, 12.18 tok/s, client 6.91 s
request 4: 83 tokens, finish stop, read 0.087 s, decode 6.762 s, 12.13 tok/s, client 6.94 s
request 5: 83 tokens, finish stop, read 0.083 s, decode 6.561 s, 12.50 tok/s, client 6.74 s
median decode tok/s 12.20, min 12.13, max 12.50

Streaming eight prompts through the shim from a laptop, run_rust_benchmark measured a first token in 0.175 to 0.256 seconds and 11.5 to 12.6 tokens per second as the client saw them.

Step 8 — Demo It From a Laptop

demo/install.sh installs two things on the laptop: a shim on 127.0.0.1:8015 as a systemd user service, and a demo MCP server registered with Claude Code:

demo/install.sh --project <gcp-project-id>

> endpoint_status
🟢 local http://127.0.0.1:8015/v1 → http://136.85.219.123:8000
- TPU /health: 200
- queued resource `gemma4-rust-qr` in us-west4-a, project aisprint-491218

The shim finds the VM's address with gcloud and looks it up again when the VM stops answering, so a new flex-start VM needs no reconfiguration. It passes every request through, streams included, so any OpenAI client works against http://127.0.0.1:8015/v1. Its root is a chat page: replies stream, the conversation carries over, and Stop cancels the decode on the TPU. A client that disconnected after 10 streamed chunks moved the server's cancel counter from 1 to 2, and the engine went idle.

The demo MCP server, torchtpu-rs-demo, has five tools, all through the shim:

> ask_gemma_rust "What is a TPU? Two sentences."
A TPU stands for **Tensor Processing Unit**, which is a specialized type of integrated circuit designed to accelerate machine learning workloads. ...
---
xbill9/gemma-4-E2B-it-qat-q4_0-w4a16-ct-text-emb4 · finish `stop` · 17 prompt + 55 completion tokens · prompt read 0.088 s · decode 12.2 tok/s on the TPU · 4.59 s round trip from here

stream_gemma_rust, engine_info, endpoint_status and serving_metrics cover streaming, the build, the shim's view of the VM and the server's counters.

Compare and Contrast

One TPU v5e chip, Gemma 4 E2B, decode tokens per second for one request at a time. Cost per million output tokens is computed from each rate and the v5e-1 flex-start list price, USD 0.60 an hour (arithmetic, evidence/arithmetic.txt):

One v5e chip	Decode tok/s	Model code	Stack	USD per M tokens
vLLM 0.30.0 🥇	144.1	Python	vLLM TPU, bf16 checkpoint	1.16
Python `torch.compile`, Colab 🥈	33.5 streamed, 45.1 batched	Python	TorchTPU, PyTorch 2.14.1
Rust, `tch`, Colab 🥉	15.1 streamed	Rust	TorchTPU defer-and-fuse, PyTorch 2.13.0
Rust, ATen shim, this rig	12.20	Rust	TorchTPU defer-and-fuse, PyTorch 2.14.1	13.66

The vLLM figure is from the gcp-inference demo on 2026-10-12, read from vLLM's own metrics. vLLM decodes 11.8x faster than this rig on the same chip type. The compiled Python step decodes 2.75x faster when streaming.

torch.compile traces Python and cannot see a Rust model, so the Rust ways run TorchTPU's defer-and-fuse mode, which fuses operations below ATen. This rig streams at 0.81x the Colab tch run. The two differ in PyTorch version, ATen layer, cache length (8,192 tokens against 33,024), a Python callback per streamed token, and where the timer sits; which of those accounts for the gap is not measured.

So, Which One?

For throughput on a v5e, serve with vLLM: 144.1 tokens per second on one request and USD 1.16 per million tokens.

For a model written in Rust on TorchTPU, this rig is the working shape: a crate that builds against the installed PyTorch, PyO3 bindings that keep the event loop free while the TPU works, and an OpenAI endpoint any client can use. It decodes at about 12 tokens per second.

For a demo, the shim gives one local address for a browser, an OpenAI client and Claude Code, whichever VM is up.

Summary

The goal of this article was to serve Gemma 4 E2B from a TPU VM with the model in Rust, through TorchTPU and without JAX, and to operate it from Claude Code. The key to the solution was a C++ shim over the few dozen ATen operations the model uses, compiled against the installed PyTorch, with PyO3 bindings that release the GIL during decoding and confine the engine to one thread. The results on one TPU v5e chip were:

🟢 Rust matches Transformers on CPU on 64 of 64 tokens, with a KL divergence of 1.1e-5, on PyTorch 2.14.1
🟢 No pinned PyTorch: Python 3.14, PyTorch 2.14.1 and nightly Rust on the VM
🟢 4 minutes 29 seconds from installing Python to starting the service, with the crate built on the VM in 21 seconds
🟢 OpenAI endpoint, streaming, cancel on disconnect, an MCP server for the TPU and one for demos, and a local chat page
⚠️ 12.20 tokens per second median decode, 11.8x slower than vLLM on the same chip type
⚠️ One request at a time and greedy decoding only; no tool-call parsing
❌ No compile path from Rust: torch.compile traces Python, and TorchTPU 0.2.0 has no C++ API

Scope: Gemma 4 E2B-it QAT W4A16 with int4 vocabulary tables, text only, one flex-start v5e-1 queued resource in us-west4-a on 2026-11-02, an 8,192-token cache, greedy decoding, one request at a time. The vLLM and torch.compile figures come from other runs on the same chip type (gcp-inference on 2026-10-12, Colab notebook 06 on 2026-11-01) with other checkpoints and cache lengths. Batched host reads, a longer cache and concurrent requests were not run on this rig. The parity check and the binding tests ran on a laptop CPU.

The strategy for serving a Rust model on TorchTPU was validated with an incremental step by step approach.

References

The rig: https://github.com/xbill9/torchtpu-dev/tree/main/tpu-pytorchrust-v5e1-2b-q4w4a16emb4
PyO3 bindings: https://github.com/xbill9/torchtpu-dev/blob/main/tpu-pytorchrust-v5e1-2b-q4w4a16emb4/rust/PYO3.md
Demo shim, chat page and MCP server: https://github.com/xbill9/torchtpu-dev/tree/main/tpu-pytorchrust-v5e1-2b-q4w4a16emb4/demo
Gemma 4 in Rust on TorchTPU, Hands On With Colab: https://dev.to/gde/gemma-4-in-rust-on-torchtpu-hands-on-with-colab-44hi
gcp-inference: https://github.com/xbill9/gcp-inference
PyO3: https://github.com/PyO3/pyo3
W4A16 repack: https://huggingface.co/xbill9/gemma-4-E2B-it-qat-q4_0-w4a16-ct-text
4-bit-tables repack: https://huggingface.co/xbill9/gemma-4-E2B-it-qat-q4_0-w4a16-ct-text-emb4

Gemma 4 in Rust on TorchTPU: Hands On With Colab

xbill — Mon, 02 Nov 2026 14:52:55 +0000

This article provides a step by step guide to running Gemma 4 E2B with the model written in Rust and every operation executed by TorchTPU, on a Colab v5e-1 runtime: one TPU v5e chip. A Rust crate reads the checkpoint, unpacks the 4-bit weights, runs the 35 layers and the ring cache, and drives the decode loop. Python hosts the backend and starts the engine. No JAX is involved.

https://github.com/xbill9/torchtpu-dev/tree/main/colab/rust

Rust writes the same tokens as compiled Python at about half the streaming speed and less than half the peak memory. Same chip, same checkpoint, same 19-token prompt, 64 greedy tokens, 33,024-token context:

One Colab v5e-1 chip	Streamed tok/s	Batched tok/s	First call	Peak memory
Rust, defer-and-fuse	14.7 to 15.7	15.2 to 16.0	82 to 84 s	7.75 GB
Rust, strict eager	4.8 to 5.0	4.8	23 to 25 s	8.62 GB
Python `torch.compile`	33.4 to 33.5	45.1 to 45.2	98 to 101 s	16.91 GB

Both Rust ways match the compiled Python model on all 64 tokens and on all three long-prompt answers.

What Is Being Measured?

TorchTPU (torch_tpu) is a PyTorch backend for TPUs. The question here is how far a model can move out of Python and still run on it. The Rust ways run on PyTorch 2.13.0 and the Python baseline on 2.14.1. The notebook swaps versions between them, for a reason Step 3 covers.

The model is Gemma 4 E2B in the W4A16 repack xbill9/gemma-4-E2B-it-qat-q4_0-w4a16-ct-text, with the per-layer table from its -text-emb4 twin. The Python baseline comes from notebook 05, the ring cache in PyTorch on TPU: Hands On With Colab.

Versions on the Colab image: Python 3.13.18, torch_tpu 0.2.0, libtpu 0.0.50, Transformers 5.20.2, compressed-tensors 0.19.0, rustc 1.99.1. Each way runs in a fresh Python process, so none inherits another's compiled programs.

Step 1 — Run the Notebook

Open notebook 06 in Colab, pick Runtime → Change runtime type → v5e-1 TPU, then Runtime → Run all. No Hugging Face token is needed. The notebook installs Rust and builds the crate on the VM:

Finished `release` profile [optimized] target(s) in 1m 44s
rustc 1.99.1 (58cf8f9c3 2026-10-13)
Rust installed and crate built in 122 s

From a terminal, the Colab CLI runs it on a real v5e-1 and keeps the executed copy:

git clone https://github.com/xbill9/torchtpu-dev
cd torchtpu-dev/colab
make setup
python3 tools/verify_on_tpu.py 06_e2b_rust_on_torchtpu

OK: 06_e2b_rust_on_torchtpu ran clean on v5e1 -> runs/2026-11-01/06_e2b_rust_on_torchtpu_output.ipynb

1/1 notebooks verified

Step 2 — How Rust Reaches the Chip

TorchTPU 0.2.0 ships no supported C++ API, and it links libtorch_python, so Python has to host the process. Importing torch_tpu registers TorchTPU's kernels on libtorch's dispatcher. The tch crate calls that same libtorch, loaded once in the same process, so a Rust call on a tpu tensor lands on the TorchTPU kernel.

The bridge is a PyO3 extension. PyO3 0.30 builds the crate as a native Python module, torchtpu_rs, and turns Rust functions and structs into Python functions and classes. tch's python-extension feature converts a Python torch.Tensor to a tch::Tensor and back, in two small helpers, take and give. A function unpacks a Python tensor, runs ATen calls and hands the result back:

#[pyfunction]
fn matmul_sum(py: Python<'_>, x: &Bound<'_, PyAny>) -> PyResult<Py<PyAny>> {
    let x = take(x)?;
    give(py, x.matmul(&x).sum(Kind::Float))
}

Run on the chip:

Rust matmul then sum: 64.0 on tpu:0
built in Rust, moved: [0.0, 1.0, 2.0, 3.0, 4.0] on tpu:0
bf16 kept: torch.bfloat16

The result stays on tpu:0, and bf16 survives the trip.

🔎 Tip: `tch` Has No TPU Device

tch::Device has no tpu variant. type_as follows any tensor already on the chip, so the crate builds on the host and moves by example:

/// Host to device without naming the device: tch has no `tpu`, but `type_as` follows the tensor.
fn on(t: &Tensor, like: &Tensor) -> Tensor {
    t.type_as(&like.to_kind(t.kind()))
}

Python passes in torch.zeros(1, device="tpu") and every tensor follows it. Avoid Tensor::device() on these tensors for the same reason.

Step 3 — Pin PyTorch to What `tch` Generates

tch 0.26 is generated for PyTorch 2.13.0, and its bindings fail to compile on 2.14.1:

'qr' is not a member of 'torch'

So the crate builds against 2.13.0, which TorchTPU 0.2.0 supports, with LIBTORCH_USE_PYTORCH=1 pointing the build at the installed PyTorch:

[dependencies]
tch = { version = "0.26.0", features = ["python-extension"] }
torch-sys = { version = "0.26.0", features = ["python-extension"] }
pyo3 = { version = "0.30", features = ["extension-module"] }
safetensors = "0.8.0"
memmap2 = "0.9.12"

Step 4 — The Model in Rust

The crate is about 580 lines. weights.rs memory-maps the safetensors files and unpacks the compressed-tensors 4-bit format to bf16. gemma.rs holds the model: 35 layers, of which 15 own keys and values and 20 reuse them, the same layout as the Python ring.

The ring's attention masks every slot that is empty or out of the window, so a step is the same program at every position:

fn attend(&self, q: &Tensor, pos: &Tensor) -> Tensor {
    let scores = q.matmul(&self.k.tr()).to_kind(Kind::Float);
    let (kpos, qpos) = (self.kpos.unsqueeze(0), pos.unsqueeze(1));
    let mut keep = kpos.ge(0).logical_and(&kpos.less_equal_tensor(&qpos));
    if let Some(w) = self.window {
        keep = keep.logical_and(&kpos.greater_tensor(&(&qpos - w)));
    }
    let weights = scores.masked_fill(&keep.logical_not().unsqueeze(1), f64::NEG_INFINITY);
    // softmax, then the values

Python starts it with a directory and a tensor on the chip:

import torch_tpu
import torchtpu_rs

engine = torchtpu_rs.Engine(MODEL_DIR, EMB4_DIR, torch.zeros(1, device="tpu"), 33024)
tokens, read_s, decode_s = engine.generate(ids, 64, 1)

The last argument sets how often tokens come back to the host: 1 streams, 8 reads in batches of eight.

PyO3 is what makes Engine a Python class. The Rust model sits behind a #[pyclass], and its methods take and return plain Python lists and numbers, so token ids cross the boundary with no tensor at all:

#[pyclass(unsendable)]
struct Engine(gemma::Gemma);

#[pymethods]
impl Engine {
    // new, layout and logits omitted
    #[pyo3(signature = (ids, max_new, read_every=1))]
    fn generate(&mut self, ids: Vec<i64>, max_new: i64, read_every: i64) -> (Vec<i64>, f64, f64) {
        tch::no_grad(|| self.0.generate(&ids, max_new, read_every))
    }
}

#[pymodule]
fn torchtpu_rs(m: &Bound<'_, PyModule>) -> PyResult<()> {
    // the two test functions omitted
    m.add_class::<Engine>()?;
    Ok(())
}

unsendable makes PyO3 refuse calls on the engine from any thread but the one that created it.

Step 5 — Check It Against Transformers

Before any TPU time, the Rust engine runs on CPU beside Hugging Face Transformers, on PyTorch 2.13.0 with the real checkpoint:

load 6.8 (3, 12, 20, 1024)
top5 rust [818, 102500, 4420, 16520, 18318] [25.75, 18.875, 18.75, 18.5, 18.125]
top5 hf   [818, 102500, 4420, 16520, 18318] [25.625, 18.75, 18.625, 18.25, 18.0]
max abs diff 0.375 KL(hf||rs) 1.8714923498919234e-05
same 64 of 64
first diff None

The top five candidates are the same, the KL divergence is 1.9e-5, and all 64 tokens match. The layout line reads 3 full-attention rings, 12 sliding rings and 20 shared layers.

Step 6 — The Three Ways on the Chip

Final run of notebook 06:

way                       load s  first s  graphs  breaks  streamed  in batches  peak GB
Rust fused, torch 2.13      23.1     83.5       -       -      15.1        15.2     7.75
Rust eager, torch 2.13      23.8     22.8       -       -       4.8         4.8     8.62
Python compile, 2.14.1      25.9    101.1       2       0      33.5        45.1    16.91
Rust fused, torch 2.13: first 64 of 64 tokens identical to Python compile on 2.14.1
Rust eager, torch 2.13: first 64 of 64 tokens identical to Python compile on 2.14.1

Defer-and-fuse is the fast mode for Rust. torch.compile traces Python, so it cannot see Rust. TORCH_TPU_DEFER_AND_FUSE=1 works below ATen and needs no change to the crate. It takes Rust from 4.8 to 15.1 tokens per second streamed, about 3.2x, at the same tokens. In this run Rust fused streams at 0.45x the compiled Python step and batched Python reaches 2.97x Rust (arithmetic, evidence/arithmetic.txt).

Compiling costs Python more memory: its 16.91 GB peak against 7.75 GB for Rust fused is 2.18x, close to the chip's 16 GiB.

🔎 Tip: Don't Run `torch.compile` on 2.13 For This Model

The compiled Python ring needs 2.14.1. On 2.13.0 the same source behaves differently:

Python compile, 2.13        25.6     34.6      65      64      15.9        16.2    11.26
Python compile, 2.13: first 21 of 64 tokens identical to Python compile on 2.14.1

It splits into 65 programs with 64 graph breaks and writes a different reply from token 21. Changing Transformers and compressed-tensors versions made no difference, and changing the PyTorch version did. Its cause is untested. The Rust ways on 2.13.0 are unaffected and match 2.14.1.

Step 7 — Long Prompts

Notebook 05's prompts fill the context with numbered records, each with a random code, and ask about one record:

way                       prompt  read s read tok/s  decode   answer  found
Rust fused, torch 2.13        61     0.1        776    14.6   915965    yes
Rust fused, torch 2.13      4081     1.0       4172    14.8   610395    yes
Rust fused, torch 2.13     32761     8.0       4116    15.6   319550     no
Rust eager, torch 2.13     32761    28.5       1148     4.8   319550     no
Python compile, 2.14.1     32761     5.0       6555    45.1   319550     no

Rust fused reads the 32,761-token prompt in 7.6 to 8.0 s, the compiled ring in 5.0 s and Rust eager in about 29 s. All three ways give the same three answers. The third is wrong in every way, the same code the Python ring gives, so the limit belongs to the model at this depth. Decode speed holds steady at every depth, because each step attends over every slot of a cache of the same size.

Compare and Contrast

One v5e chip, Gemma 4 E2B	Streamed tok/s	Model code	PyTorch	Peak memory
Python `torch.compile` 🥇	33.4 to 33.5	Python	2.14.1	16.91 GB
Rust, defer-and-fuse 🥈	14.7 to 15.7	Rust	2.13.0	7.75 GB
Rust, strict eager 🥉	4.8 to 5.0	Rust	2.13.0	8.62 GB

So, Which One?

For speed on this chip, stay with the compiled Python ring on PyTorch 2.14.1: 45 tokens per second in batches and 33 streamed.

For a model in Rust, set TORCH_TPU_DEFER_AND_FUSE=1: about 15 tokens per second streamed, the same tokens and answers, and less than half the peak memory. TorchTPU marks that variable experimental, so it may change.

Keep strict eager for debugging, at about 5 tokens per second.

Summary

The goal of this article was to run Gemma 4 E2B on a TPU with the model written in Rust, through TorchTPU and without JAX. The key to the solution was tch sharing the one libtorch that TorchTPU registers its kernels with, so a Rust ATen call on a tpu tensor reaches a TorchTPU kernel, with PyO3 exposing the Rust engine to Python as a module. The Colab v5e-1 results were:

🟢 Rust matches compiled Python on 64 of 64 tokens and 3 of 3 long-prompt answers
🟢 Rust matches Transformers on CPU on 64 of 64 tokens, with a KL divergence of 1.9e-5
🟢 One environment variable takes Rust from about 5 to about 15 tokens per second streamed
🟢 Rust peaks at 7.75 GB against 16.91 GB for compile
⚠️ Rust fused streams at 0.45x compiled Python, because torch.compile cannot trace Rust
⚠️ tch 0.26 builds against PyTorch 2.13.0 only
⚠️ Python still hosts the process, builds the prompts and tokenizes
❌ torch.compile on PyTorch 2.13.0 splits this model into 65 programs and changes the reply

Scope: Gemma 4 E2B-it QAT only, text only, one Colab v5e-1 runtime per run on 2026-11-01, greedy decoding, one sequence at a time, four prompts, a 33,024-token context. Rust ranges cover three runs and compiled Python ranges two. The earliest Rust run used Transformers 5.21.0 and compressed-tensors 0.20.1, and the rest 5.20.2 and 0.19.0. The Rust ways ran on PyTorch 2.13.0 and the Python baseline on 2.14.1, one version apart. The 131,328-token context, a serving endpoint, a Rust Colab kernel, and where the time goes between Rust fused and compile were not run. The final crate formatting commit was built and checked on CPU, and the TPU runs used the source before it.

The strategy for running a Rust model on TorchTPU was validated with an incremental step by step approach.

References

Rust crate: https://github.com/xbill9/torchtpu-dev/tree/main/colab/rust
Notebook 06: https://github.com/xbill9/torchtpu-dev/blob/main/colab/notebooks/06_e2b_rust_on_torchtpu.ipynb
Executed run: https://github.com/xbill9/torchtpu-dev/tree/main/colab/runs/2026-11-01
PyTorch on TPU: Hands On With Colab: https://dev.to/gde/pytorch-on-tpu-hands-on-with-colab-5b5g
tch: https://github.com/LaurentMazare/tch-rs
PyO3: https://github.com/PyO3/pyo3
W4A16 repack: https://huggingface.co/xbill9/gemma-4-E2B-it-qat-q4_0-w4a16-ct-text
4-bit-tables repack: https://huggingface.co/xbill9/gemma-4-E2B-it-qat-q4_0-w4a16-ct-text-emb4
Colab CLI: https://github.com/googlecolab/google-colab-cli

Profile Sprawl: Solving a Problem You Didn't Know You Had

xbill — Sun, 01 Nov 2026 14:35:31 +0000

This article provides a step by step guide to keeping a public web presence consistent across many sites. An agent skill for Claude Code is built that treats a bio as source code: one file holds the facts, one table records where it has landed, and scripts read the live sites to confirm it.

https://github.com/xbill9/profile-sync

Every profile you own is a copy of your bio, and copies drift. The fix is the one software already uses: a single source, a status table and a check that reads production.

What Is Profile Sprawl?

A working developer collects profiles. LinkedIn, GitHub, Medium, dev.to, Substack, a certification wallet, a community portal, a program dashboard. Each one was accurate on the day you filled it in.

Profile sprawl is what happens next. A new role or credential goes on the site you happened to have open, and the rest keep the old text. No site reports an error, because each one is internally fine. The inconsistency exists only across sites, and nobody reads them side by side except a recruiter, a conference organizer or a reviewer.

How Bad Could It Be?

One audit of ten profiles belonging to one person, read on 2026-10-31, the day after a new certification was issued:

Site	Certifications	Years of experience
LinkedIn headline	10
LinkedIn About	9	over 30+
Medium	9	20+
Google Developers profile	9	20+
Credly	9	20+
Advocu	9

The same LinkedIn profile disagreed with itself. The headline said 10 and the About said 9, and each line looked fine alone.

The other four sites had a different problem. AWS Builder Center listed 0 credentials, dev.to held a tagline, GitHub held an unexplained string, and Substack was empty apart from the platform's default text.

At This Point You Should Have

A list of your public profiles and a browser where you are logged in to them
Python 3, with no extra packages
Claude Code with the Chrome extension, or the patience to run the same steps by hand
A git repository, which can be public as long as it holds public text only

Step 1 — Write One Source File

bio.md holds everything a site could need, in plain Markdown with no site-specific syntax. Facts are lists. Text comes in lengths, written from those facts.

---
version: 2026-10-31
---

# Bio

## Headline
## Tagline
## Short
## Summary
## Long
## Roles
## Community
## Certifications
## Education
## Links

Three lengths cover every form measured here: Short at 133 characters, Summary at 561 and Long at 735. A site gets its own text only when a limit forces it, and even then the shorter text becomes a section of the file. Per-site copies are how sprawl starts.

The rule that keeps this working: a change starts in bio.md, never on a site. If a site was edited directly, that change is copied back first.

Step 2 — Track Where It Has Landed

tracker.md is one table with one row per site. Status takes one of four values.

Status	Meaning
Current	Synced version matches `bio.md` and the live profile was checked
Behind	The site has an older version
Unverified	Nobody has compared this site to `bio.md` yet
Blocked	Something outside your control stops the update, explained in Notes

The Text used column records which sections a site takes. A change to Long reaches only the sites that use Long, and every other row moves to the new version with no edit.

Step 3 — Make the Computer Do the Counting

preflight.py compares the two files and exits 1 on any disagreement. It counts the Certifications list and compares that count with every claim in the text, so a number in a headline can never drift from the list it summarizes.

$ python3 .claude/skills/profile-sync/scripts/preflight.py
bio.md 2026-10-31, 10 sites: {'Behind': 0, 'Blocked': 1, 'Current': 9, 'Unverified': 0}
preflight OK

It also fails on a text longer than a site's form allows, a Current row with no check date, and a status outside the four values. The repository is public, so it scans both files for email addresses and phone numbers too.

Arithmetic belongs in code. A model or a person can miscount a list. A script compares 10 with 10.

Step 4 — Read the Live Sites

A tracker row is a claim. check-live.py fetches what a visitor gets from each site with a public source and compares it with bio.md.

$ python3 .claude/skills/profile-sync/scripts/check-live.py
  SKIP  LinkedIn: browser check
  OK    GitHub: live text matches Short
  SKIP  Medium: browser check
  OK    dev.to: live text matches Short
  OK    Substack: live text matches Short
  SKIP  AWS Builder Center: browser check
  SKIP  Google Developers profile: browser check
  SKIP  Google Cloud Skills Boost: browser check
  SKIP  Credly: browser check
  SKIP  Advocu: browser check
check-live OK

A site marked Current that differs fails the run. A check that can only pass proves nothing, so the same script was run against a stale value to confirm it exits 1.

Step 5 — Measure Every Form

Each site has its own field, limit and save button. These are read from the forms themselves, and they live in one reference file so the next run starts with them.

Site	Field	Limit
Medium	Short bio	160
dev.to	Bio	200
AWS Builder Center	About	255
Credly	Personal Summary	600
Google Developers profile	Bio	750
LinkedIn	About	2,600

A property of one site is evidence about that site only. The Long text at 735 characters fits 2 of these fields and fails the other 4, so the limits go in biolib.py and preflight.py enforces them before any browser opens.

🔎 Tip: Reading and Writing Are Different Routes

dev.to has a good API, and a key reads the profile with no trouble. Profile fields are read-only there: PUT and PATCH on /api/users/me return 404, so the bio is saved through the settings page. Test both directions before planning a rollout around an API.

Step 6 — Compare Before Saving

Browser automation can drop a character while reporting success, and a page can show a success message for the wrong text. So each save follows the same order:

Scroll to the field and confirm it has focus.
Select the old text and type the new text.
Compare the field with the exact payload from bio.md.
Read the position of the save button after scrolling.
Save, reload and compare the live text.

payload.py supplies step 3. It prints the text for a site, or a small JavaScript function that compares a field with it. The text travels base64-encoded, which survives quotes, emoji and line breaks.

$ python3 .claude/skills/profile-sync/scripts/payload.py Credly > /dev/null
Summary: 561 characters

🔎 Tip: Decline the Announcement

Adding a certification on LinkedIn opens a prompt to post about it to your network. Updating a profile and announcing to it are separate decisions. The skill clicks Skip unless an announcement was requested.

Step 7 — Package It as a Skill

The steps above become an agent skill, in the shape of publishing-kit:

.claude/skills/profile-sync/
  SKILL.md               the workflow
  references/sites.md    route, field, limit and save button per site
  scripts/biolib.py      reads bio.md and tracker.md
  scripts/preflight.py   the two files agree
  scripts/check-live.py  the live sites agree
  scripts/payload.py     the exact text for a field
tests/test_preflight.py  one case per failure, with a passing case beside it

The tests are the guard on the guards. Each failure gets a case that must fail and a case that must pass.

$ python3 -m unittest discover -s tests
Ran 18 tests in 0.004s

OK

The passing cases carry weight too. One holds a date, so a phone number pattern that matches 2026-10-31 fails the suite before it can fail every row in the tracker.

Step 8 — Roll Out One Change

The certification that started this run shows the whole loop.

# 1. Edit bio.md, set version to today
# 2. Confirm the files agree
python3 .claude/skills/profile-sync/scripts/preflight.py
# 3. In Claude Code, run /profile-sync to roll it out
# 4. Confirm the live sites agree
python3 .claude/skills/profile-sync/scripts/check-live.py

The skill works down the tracker, saves each field in Chrome in the logged-in session, reloads each page, records the check date, then commits and pushes. Credentials stay in the browser, so the repository holds public text only.

Before starting, confirm the fact at its source. Here that source was the Credly badge, issued 2026-10-30, which settled whether 10 or 9 was correct.

Compare and Contrast

	By hand	A notes file	profile-sync
Source of truth	Whichever site is newest	A notes file	`bio.md` in git
Knows which sites are behind	No	If you update it	🥇 Yes
Catches a wrong count	No	No	🥇 Yes
Reads the live site	When you look	When you look	🥇 3 sites by script, the rest by browser

So, Which One?

Start with Step 1 and Step 2, even with no automation. The first side-by-side read will show you what your profiles have been saying.

Add the skill once you hold more profiles than you can check in one sitting. That number is small, and a new credential is a good day to start.

Summary

The goal of this article was to keep one person's public profiles consistent across many sites. The key to the solution was treating a bio as source code, with one source file, a status table and a check that reads the live sites. The profile-sync results were:

🟢 9 of the 10 sites are Current, each confirmed on its live page
🟢 The certification count is computed from a list, so the headline and the About can no longer disagree
🟢 18 tests keep each failure beside a passing case
❌ 5 sites said 9 certifications the day after a tenth was issued, and 3 said 20+ years
⚠️ Google Skills stays Blocked: it has no bio field and its public profile page returns an error
⚠️ check-live.py reads 3 sites and only the Short text, so the other sites still need a browser check

Scope: one person's ten profiles on 2026-10-31, in one Chrome session. Form limits were read from each site's own form on that date and can change. Two edits on the same day share one version, so version equality alone does not show that a site has the latest text, and the live comparison covers that gap.

The strategy for keeping a public profile consistent was validated with an incremental step by step approach.

References

Profile Sprawl: Solving a Problem You Didn't Know You Had

xbill — Sun, 01 Nov 2026 14:35:26 +0000

https://github.com/xbill9/profile-sync

Every profile you own is a copy of your bio, and copies drift. The fix is the one software already uses: a single source, a status table and a check that reads production.

What Is Profile Sprawl?

A working developer collects profiles. LinkedIn, GitHub, Medium, dev.to, Substack, a certification wallet, a community portal, a program dashboard. Each one was accurate on the day you filled it in.

How Bad Could It Be?

One audit of ten profiles belonging to one person, read on 2026-10-31, the day after a new certification was issued:

Site	Certifications	Years of experience
LinkedIn headline	10
LinkedIn About	9	over 30+
Medium	9	20+
Google Developers profile	9	20+
Credly	9	20+
Advocu	9

The same LinkedIn profile disagreed with itself. The headline said 10 and the About said 9, and each line looked fine alone.

At This Point You Should Have

A list of your public profiles and a browser where you are logged in to them
Python 3, with no extra packages
Claude Code with the Chrome extension, or the patience to run the same steps by hand
A git repository, which can be public as long as it holds public text only

Step 1 — Write One Source File

bio.md holds everything a site could need, in plain Markdown with no site-specific syntax. Facts are lists. Text comes in lengths, written from those facts.

---
version: 2026-10-31
---

# Bio

## Headline
## Tagline
## Short
## Summary
## Long
## Roles
## Community
## Certifications
## Education
## Links

The rule that keeps this working: a change starts in bio.md, never on a site. If a site was edited directly, that change is copied back first.

Step 2 — Track Where It Has Landed

tracker.md is one table with one row per site. Status takes one of four values.

Status	Meaning
Current	Synced version matches `bio.md` and the live profile was checked
Behind	The site has an older version
Unverified	Nobody has compared this site to `bio.md` yet
Blocked	Something outside your control stops the update, explained in Notes

The Text used column records which sections a site takes. A change to Long reaches only the sites that use Long, and every other row moves to the new version with no edit.

Step 3 — Make the Computer Do the Counting

$ python3 .claude/skills/profile-sync/scripts/preflight.py
bio.md 2026-10-31, 10 sites: {'Behind': 0, 'Blocked': 1, 'Current': 9, 'Unverified': 0}
preflight OK

Arithmetic belongs in code. A model or a person can miscount a list. A script compares 10 with 10.

Step 4 — Read the Live Sites

A tracker row is a claim. check-live.py fetches what a visitor gets from each site with a public source and compares it with bio.md.

$ python3 .claude/skills/profile-sync/scripts/check-live.py
  SKIP  LinkedIn: browser check
  OK    GitHub: live text matches Short
  SKIP  Medium: browser check
  OK    dev.to: live text matches Short
  OK    Substack: live text matches Short
  SKIP  AWS Builder Center: browser check
  SKIP  Google Developers profile: browser check
  SKIP  Google Cloud Skills Boost: browser check
  SKIP  Credly: browser check
  SKIP  Advocu: browser check
check-live OK

A site marked Current that differs fails the run. A check that can only pass proves nothing, so the same script was run against a stale value to confirm it exits 1.

Step 5 — Measure Every Form

Each site has its own field, limit and save button. These are read from the forms themselves, and they live in one reference file so the next run starts with them.

Site	Field	Limit
Medium	Short bio	160
dev.to	Bio	200
AWS Builder Center	About	255
Credly	Personal Summary	600
Google Developers profile	Bio	750
LinkedIn	About	2,600

🔎 Tip: Reading and Writing Are Different Routes

Step 6 — Compare Before Saving

Browser automation can drop a character while reporting success, and a page can show a success message for the wrong text. So each save follows the same order:

Scroll to the field and confirm it has focus.
Select the old text and type the new text.
Compare the field with the exact payload from bio.md.
Read the position of the save button after scrolling.
Save, reload and compare the live text.

$ python3 .claude/skills/profile-sync/scripts/payload.py Credly > /dev/null
Summary: 561 characters

🔎 Tip: Decline the Announcement

Step 7 — Package It as a Skill

The steps above become an agent skill, in the shape of publishing-kit:

.claude/skills/profile-sync/
  SKILL.md               the workflow
  references/sites.md    route, field, limit and save button per site
  scripts/biolib.py      reads bio.md and tracker.md
  scripts/preflight.py   the two files agree
  scripts/check-live.py  the live sites agree
  scripts/payload.py     the exact text for a field
tests/test_preflight.py  one case per failure, with a passing case beside it

The tests are the guard on the guards. Each failure gets a case that must fail and a case that must pass.

$ python3 -m unittest discover -s tests
Ran 18 tests in 0.004s

OK

The passing cases carry weight too. One holds a date, so a phone number pattern that matches 2026-10-31 fails the suite before it can fail every row in the tracker.

Step 8 — Roll Out One Change

The certification that started this run shows the whole loop.

# 1. Edit bio.md, set version to today
# 2. Confirm the files agree
python3 .claude/skills/profile-sync/scripts/preflight.py
# 3. In Claude Code, run /profile-sync to roll it out
# 4. Confirm the live sites agree
python3 .claude/skills/profile-sync/scripts/check-live.py

Before starting, confirm the fact at its source. Here that source was the Credly badge, issued 2026-10-30, which settled whether 10 or 9 was correct.

Compare and Contrast

	By hand	A notes file	profile-sync
Source of truth	Whichever site is newest	A notes file	`bio.md` in git
Knows which sites are behind	No	If you update it	🥇 Yes
Catches a wrong count	No	No	🥇 Yes
Reads the live site	When you look	When you look	🥇 3 sites by script, the rest by browser

So, Which One?

Start with Step 1 and Step 2, even with no automation. The first side-by-side read will show you what your profiles have been saying.

Add the skill once you hold more profiles than you can check in one sitting. That number is small, and a new credential is a good day to start.

Summary

🟢 9 of the 10 sites are Current, each confirmed on its live page
🟢 The certification count is computed from a list, so the headline and the About can no longer disagree
🟢 18 tests keep each failure beside a passing case
❌ 5 sites said 9 certifications the day after a tenth was issued, and 3 said 20+ years
⚠️ Google Skills stays Blocked: it has no bio field and its public profile page returns an error
⚠️ check-live.py reads 3 sites and only the Short text, so the other sites still need a browser check

The strategy for keeping a public profile consistent was validated with an incremental step by step approach.