DEV Community: xbill

Build an MCP server in Rust with rmcp: a walk-through 🦀

xbill — Sun, 16 Aug 2026 12:32:56 +0000

This tutorial walks through building an MCP server in Rust with
rmcp, the official Model Context Protocol Rust SDK.

The example is a real one: a devops agent that manages AWS EC2 G5g instances — Graviton2
boxes with NVIDIA T4G GPUs — serving Gemma 4 under vLLM. It launches instances, drives them
over SSM, and health-checks the model. There's an existing Python version, so at the end we
can put the two side by side.

Follow along and you'll have a working, registerable MCP server. 🦀

Why Rust for this?

Worth answering properly, because the weak version of the argument is easy to make and easy to
demolish — and the real one is better anyway.

Start with what it isn't: these tools are I/O bound. Every one is an AWS API call —
describe_instances, send_command, polling SSM — so 100–500 ms of network per call. The
caller's language contributes nothing measurable there. Anyone selling you a Rust rewrite on
raw speed for this workload is selling something.

Three claims that don't hold, so nobody has to make them in the comments:

Claim	Why it fails
"462 ms startup is slow"	stdio servers spawn once per session, not per call
"Rust is faster"	the work is network round-trips to AWS
"smaller supply chain"	241 crates vs 34 Python packages — it's worse

What actually justifies it, for this codebase:

1. It's a fleet, not a server. This monorepo has 16 rigs, each with its own MCP
server. That changes the units:

All loaded together	🐍 Python	🦀 Rust
Resident memory	16 × 83 MB ≈ 1.33 GB	16 × 12 MB ≈ 192 MB
Session startup	16 × 462 ms ≈ 7.4 s	16 × 2.5 ms ≈ 40 ms

A gigabyte of resident Python to expose sixteen tool lists is a real cost.

2. No shared interpreter. These rigs install system-wide — no virtualenvs, by policy — so
all sixteen share one Python. Sixteen servers with independently drifting boto3 and mcp
pins in one interpreter is a standing conflict risk. A static binary has no such coupling;
each rig pins whatever it likes in its own Cargo.lock.

3. The schema can't drift from the code. More on this at Step 3, but it's the one that
survives longest: schemars generates the tool schema from the same struct the handler
destructures.

So: distribution and correctness, not speed. ✅ If you have one MCP server and it works,
this is not a reason to rewrite it.

How does this all fit together?

Two halves. The agent and the MCP server run on your machine; the GPU box is remote, and it
has no inbound SSH — everything goes through the AWS APIs.

   YOUR MACHINE                                     AWS  us-east-1
┌──────────────────────────────┐       ┌───────────────────────────────────────┐
│                              │       │                                       │
│  Claude Code / IDE           │       │  ┌─ EC2 g5g.4xlarge ───────────────┐  │
│         |                    │       │  │  Graviton2 (aarch64)            │  │
│         | MCP · JSON-RPC 2.0 │       │  │  + NVIDIA T4G (SM 7.5)          │  │
│         | over stdio         │       │  │                                 │  │
│         v                    │  EC2  │  │  [PY] vLLM + [RUST] vllm-rs     │  │
│  ┌────────────────────────┐  │  API  │  │  listening on :8000             │  │
│  │ [RUST]                 │──┼──────>│  │                                 │  │
│  │ gpu-vllm-g5g-2b        │  │       │  │  Gemma 4 E2B                    │  │
│  │                        │  │  SSM  │  └─────────────────────────────────┘  │
│  │ rmcp 3.1.2             │──┼──────>│           ^                           │
│  │ tokio · schemars       │  │  Run  │           |  no inbound SSH,          │
│  │ aws-sdk-ec2 / -ssm     │  │  Cmd  │           |  no key pair,             │
│  │ 1 binary · 2.5 ms      │  │       │           |  no port 22 rule          │
│  └────────────────────────┘  │       │                                       │
│        9 tools               │       │  IAM instance profile carries         │
│  list / start / stop /       │       │  AmazonSSMManagedInstanceCore         │
│  terminate / endpoint /      │       │                                       │
│  run_remote / health ...     │       │                                       │
└──────────────────────────────┘       └───────────────────────────────────────┘

The agent never talks to the GPU box directly. It calls a tool; the tool calls EC2 to
manage the instance's lifecycle, or SSM Run Command to execute something on it. That's
what lets the box run with no inbound rules at all — which is the main reason this is worth
building as a server rather than a pile of shell scripts.

The [RUST] on the right-hand side is vLLM's own Rust frontend — the other article in this
series. This one is the [RUST] on the left: the Rust that drives the box.

What is MCP, in one paragraph?

Model Context Protocol is how an AI agent discovers and calls your tools. Your server
advertises a list of tools with JSON Schemas; the client (Claude Code, an IDE, whatever)
calls them over JSON-RPC 2.0. Transport is usually stdio — the client spawns your binary
and talks over stdin/stdout.

That last detail matters for the Rust pitch: if the client spawns your process on every
session, process startup is a user-visible cost.

Step 1 — Scaffold

cargo new --bin rust-mcp --name gpu-vllm-g5g-2b-mcp
cd rust-mcp

Now the dependencies. Feature flags are the thing to get right here — cargo add rmcp
on its own compiles fine and gives you almost nothing:

cargo add rmcp --features server,macros,transport-io

Feature	What it brings
`server`	the `ServerHandler` trait and router types
`macros`	`#[tool]`, `#[tool_router]`, `#[tool_handler]`
`transport-io`	stdio transport

The crate also ships client, auth, elicitation, transport-streamable-http-server and
more, all off by default. Add them when you need them.

Then the rest:

cargo add tokio --features rt-multi-thread,macros,process,time
cargo add serde serde_json anyhow schemars
cargo add aws-config aws-sdk-ec2 aws-sdk-ssm aws-sdk-secretsmanager

Resulting Cargo.toml:

[package]
name = "gpu-vllm-g5g-2b-mcp"
version = "0.1.0"
edition = "2024"

[dependencies]
rmcp = { version = "3.1.2", features = ["server", "macros", "transport-io"] }
tokio = { version = "1.53.1", features = ["rt-multi-thread", "macros", "process", "time"] }
aws-config = "1.10.1"
aws-sdk-ec2 = "1.246.0"
aws-sdk-ssm = "1.118.0"
serde = "1.0.229"
serde_json = "1.0.151"
schemars = "1.2.2"
anyhow = "1.0.104"

🔎 Tip: where the canonical examples live

rmcp moves fast, and rendered docs lag. The vendored tests on your own disk are
compiled against the exact version you resolved:

ls ~/.cargo/registry/src/*/rmcp-3.1.2/tests/

tests/test_tool_macros.rs is a complete, working server in about 60 lines. When an API
question comes up, that file answers it faster and more reliably than anything else. ⚡

Step 2 — The server struct

An rmcp server is a struct that owns a ToolRouter:

use rmcp::{
    ErrorData, ServerHandler, ServiceExt,
    handler::server::{router::tool::ToolRouter, wrapper::Parameters},
    model::{CallToolResult, ContentBlock, Implementation, ServerCapabilities, ServerInfo},
    tool, tool_handler, tool_router,
    transport::stdio,
};
use schemars::JsonSchema;
use serde::{Deserialize, Serialize};

#[derive(Clone)]
struct G5gServer {
    tool_router: ToolRouter<Self>,
}

Step 3 — Describe your inputs as types

This is the part that sold me on the whole exercise. Your tool's input is a plain struct, and
schemars turns it into the JSON Schema the agent sees — doc comments and all:

#[derive(Debug, Serialize, Deserialize, JsonSchema)]
struct InstanceId {
    /// EC2 instance id, e.g. `i-0123456789abcdef0`.
    instance_id: String,
}

That doc comment becomes the field's description in the tool schema. Rename the field and
the schema follows. The compiler checks the type your handler destructures. There is no
second artifact to keep in sync. ✅

Step 4 — Write the tools

#[tool_router] on the impl block, #[tool] on each method:

#[tool_router(router = tool_router)]
impl G5gServer {
    fn new() -> Self {
        Self { tool_router: Self::tool_router() }
    }

    #[tool(description = "List EC2 instances tagged ManagedBy=gpu-vllm-g5g-2b.")]
    async fn list_g5g_instances(&self) -> Result<CallToolResult, ErrorData> {
        let conf = aws_config::defaults(aws_config::BehaviorVersion::latest())
            .region(aws_config::Region::new("us-east-1"))
            .load()
            .await;
        let ec2 = aws_sdk_ec2::Client::new(&conf);

        let resp = match ec2.describe_instances()
            .filters(Filter::builder()
                .name("tag:ManagedBy").values("gpu-vllm-g5g-2b").build())
            .send().await
        {
            Ok(r) => r,
            Err(e) => return ok(format!("❌ describe_instances failed: {e}")),
        };

        let mut rows = Vec::new();
        for res in resp.reservations() {
            for inst in res.instances() {
                rows.push(format!("| `{}` | {} | {} |",
                    inst.instance_id().unwrap_or("?"),
                    inst.instance_type().map(|t| t.as_str()).unwrap_or("?"),
                    inst.state().and_then(|s| s.name())
                        .map(|n| n.as_str()).unwrap_or("unknown"),
                ));
            }
        }
        ok(format!("📡 Instances\n\n| id | type | state |\n|---|---|---|\n{}",
                   rows.join("\n")))
    }
}

Tools that take arguments wrap them in Parameters<T>:

    #[tool(description = "Terminate a G5g instance. Permanent — destroys the root volume.")]
    async fn terminate_g5g_instance(
        &self,
        Parameters(args): Parameters<InstanceId>,
    ) -> Result<CallToolResult, ErrorData> {
        // …
    }

And a small helper, since every tool returns the same shape:

fn ok(text: String) -> Result<CallToolResult, ErrorData> {
    Ok(CallToolResult::success(vec![ContentBlock::text(text)]))
}

Step 5 — Implement ServerHandler

#[tool_handler] wires the router in, so you never write a dispatch match:

#[tool_handler(router = self.tool_router)]
impl ServerHandler for G5gServer {
    fn get_info(&self) -> ServerInfo {
        let mut info = ServerInfo::new(
            ServerCapabilities::builder().enable_tools().build()
        );
        info.server_info = Implementation::new(
            "gpu-vllm-g5g-2b", env!("CARGO_PKG_VERSION")
        );
        info.instructions = Some(
            "Devops agent for AWS EC2 G5g (Graviton2 + NVIDIA T4G) serving Gemma 4 \
             under vLLM. Remote administration goes through SSM; there is no inbound SSH."
                .to_string(),
        );
        info
    }
}

💡 These model structs are #[non_exhaustive], so use the constructors (ServerInfo::new,
Implementation::new) and then assign fields — a struct literal won't compile, even with
..Default::default().

Step 6 — main

Four lines:

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let service = G5gServer::new().serve(stdio()).await?;
    service.waiting().await?;
    Ok(())
}

cargo build --release

Step 7 — Test the protocol by hand

An MCP server is a protocol implementation, so test it with a protocol transcript. Three
JSON-RPC lines on stdin — no client required:

printf '%s\n' \
'{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2026-07-28","capabilities":{},"clientInfo":{"name":"probe","version":"0"}}}' \
'{"jsonrpc":"2.0","method":"notifications/initialized"}' \
'{"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}}' \
'{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"get_help","arguments":{}}}' \
| ./target/release/gpu-vllm-g5g-2b-mcp

initialize OK: gpu-vllm-g5g-2b 0.1.0 proto 2026-07-28
tools/list OK: 9 tools -> get_deployment_config, get_endpoint, get_help,
  list_g5g_instances, run_remote, start_g5g_instance, stop_g5g_instance,
  terminate_g5g_instance, verify_model_health
tools/call get_help OK

🟢 rmcp 3.1.2 negotiates the 2026-07-28 spec version by default.

Keep this snippet. It's the fastest way to tell "my server is broken" from "my client config
is broken."

Step 8 — Register it

Point your MCP client at the binary. For Claude Code, .mcp.json:

{
  "mcpServers": {
    "gpu-vllm-g5g-2b": {
      "command": "/abs/path/to/rust-mcp/target/release/gpu-vllm-g5g-2b-mcp",
      "env": { "AWS_REGION": "us-east-1" }
    }
  }
}

The server name prefixes every tool — mcp__gpu-vllm-g5g-2b__list_g5g_instances — so name it
after the thing it manages, especially if you run several.

Credentials come from the standard AWS provider chain, so whatever
aws sts get-caller-identity resolves is what the server gets. Set AWS_PROFILE to pick one.

Tests worth writing

The interesting assertions aren't about the code, they're about the machine. Turing has no
bf16 datapath and no fp8, so the serving flags must differ from every L4-class box — exactly
the sort of thing that silently reverts when someone copies a flag set from a neighbour:

#[test]
fn serve_flags_are_turing_shaped() {
    let f = serve_flags("google/gemma-4-E2B-it", "g5g.2xlarge");
    assert!(f.contains("--dtype float16"), "Turing has no bf16 datapath");
    assert!(f.contains("--kv-cache-dtype auto"), "Turing has no fp8 datapath");
    assert!(!f.contains("attention-backend"));   // not a real vLLM v0.27 variable
}

#[test]
fn unknown_types_are_rejected_and_never_need_swap() {
    assert!(validate_instance_type("t4g.2xlarge").is_err());  // burstable CPU box, no GPU
    assert!(!needs_swap("t4g.2xlarge"));                      // 0 GiB must not read as "tiny"
}

That second one earns its keep: host_memory_gb returns 0 for an unknown instance type,
and a naive ram < 16 would decide an unrecognised machine needs a swapfile.

running 5 tests
test result: ok. 5 passed; 0 failed; finished in 0.00s

🐍 vs 🦀 — the scoreboard

Cold start measured the way a client experiences it: spawn the process, send initialize +
initialized + tools/list, stop the clock when the tool list comes back. Seven runs, median.

	🐍 Python (FastMCP)	🦀 Rust (rmcp)
Cold start to `tools/list`	462.0 ms (437–530)	2.5 ms (1.8–2.9)
Peak RSS	83 MB	12 MB
Artifact	Python runtime + 34 packages	one binary, 39.4 MB (19.9 MB stripped)
Direct dependencies	3	14
Total resolved packages	34	241
Tools implemented	15	9
Source	759 lines	560 + 52 of tests
Clean release build	n/a	5 m 28 s

185x on cold start — but per the section up top, resist quoting that on its own. One
server spawned once a session, 460 ms, nobody notices. It only becomes a number worth having
when you multiply it by sixteen rigs, and even then it's the 12 MB vs 83 MB row that does
the heavier lifting.

Read the rest honestly too. 241 resolved packages against 34 means the static binary is
not a smaller supply chain, just the same one audited in Cargo.lock. The port covers 9
tools to Python's 15 — the provisioning path (cloud-init rendering, AMI resolution, spot
options) is the fiddly half and isn't ported. And 5 m 28 s of clean build against an
interpreter that starts instantly is a real cost while you're iterating. 📊

So, worth it?

For a single MCP server that already works: no. Don't rewrite it.

For sixteen of them sharing one system Python, shipped to machines that shouldn't need a
Python environment at all: yes — and note that neither half of that sentence is about speed.
It's a packaging answer.

The part that'll still be true next year is schemars. The tool schema the agent sees is
generated from the same struct the handler destructures, checked by the compiler, documented
by the doc comments on its fields. In the Python version the schema, the runtime types and the
docs are three artifacts that agree by convention — and go quiet when they stop agreeing.

The 462 ms is a bonus. The schema not being able to lie about the code is the reason. ✅

Cheat sheet

# scaffold
cargo new --bin my-mcp && cd my-mcp
cargo add rmcp --features server,macros,transport-io
cargo add tokio --features rt-multi-thread,macros
cargo add serde serde_json schemars anyhow

# canonical examples for YOUR resolved version
ls ~/.cargo/registry/src/*/rmcp-*/tests/test_tool_macros.rs

# build + smoke test
cargo build --release
printf '%s\n' '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2026-07-28","capabilities":{},"clientInfo":{"name":"p","version":"0"}}}' \
              '{"jsonrpc":"2.0","method":"notifications/initialized"}' \
              '{"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}}' \
  | ./target/release/my-mcp

Four macros to remember: #[tool_router] on the impl, #[tool] on each method,
#[tool_handler] on the ServerHandler impl, Parameters<T> around your input struct.

Rust 1.97.1, rmcp 3.1.2, aws-sdk-ec2 1.246.0, edition 2024. Startup measured on the dev host
— it's a comparison of two MCP servers, not a hardware result. Single machine, seven runs per
side, median reported.

Build an MCP server in Rust with rmcp: a walk-through 🦀

xbill — Sun, 16 Aug 2026 12:32:18 +0000

This tutorial walks through building an MCP server in Rust with
rmcp, the official Model Context Protocol Rust SDK.

Follow along and you'll have a working, registerable MCP server. 🦀

Why Rust for this?

Worth answering properly, because the weak version of the argument is easy to make and easy to
demolish — and the real one is better anyway.

Three claims that don't hold, so nobody has to make them in the comments:

Claim	Why it fails
"462 ms startup is slow"	stdio servers spawn once per session, not per call
"Rust is faster"	the work is network round-trips to AWS
"smaller supply chain"	241 crates vs 34 Python packages — it's worse

What actually justifies it, for this codebase:

1. It's a fleet, not a server. This monorepo has 16 rigs, each with its own MCP
server. That changes the units:

All loaded together	🐍 Python	🦀 Rust
Resident memory	16 × 83 MB ≈ 1.33 GB	16 × 12 MB ≈ 192 MB
Session startup	16 × 462 ms ≈ 7.4 s	16 × 2.5 ms ≈ 40 ms

A gigabyte of resident Python to expose sixteen tool lists is a real cost.

3. The schema can't drift from the code. More on this at Step 3, but it's the one that
survives longest: schemars generates the tool schema from the same struct the handler
destructures.

So: distribution and correctness, not speed. ✅ If you have one MCP server and it works,
this is not a reason to rewrite it.

How does this all fit together?

Two halves. The agent and the MCP server run on your machine; the GPU box is remote, and it
has no inbound SSH — everything goes through the AWS APIs.

   YOUR MACHINE                                     AWS  us-east-1
┌──────────────────────────────┐       ┌───────────────────────────────────────┐
│                              │       │                                       │
│  Claude Code / IDE           │       │  ┌─ EC2 g5g.4xlarge ───────────────┐  │
│         |                    │       │  │  Graviton2 (aarch64)            │  │
│         | MCP · JSON-RPC 2.0 │       │  │  + NVIDIA T4G (SM 7.5)          │  │
│         | over stdio         │       │  │                                 │  │
│         v                    │  EC2  │  │  [PY] vLLM + [RUST] vllm-rs     │  │
│  ┌────────────────────────┐  │  API  │  │  listening on :8000             │  │
│  │ [RUST]                 │──┼──────>│  │                                 │  │
│  │ gpu-vllm-g5g-2b        │  │       │  │  Gemma 4 E2B                    │  │
│  │                        │  │  SSM  │  └─────────────────────────────────┘  │
│  │ rmcp 3.1.2             │──┼──────>│           ^                           │
│  │ tokio · schemars       │  │  Run  │           |  no inbound SSH,          │
│  │ aws-sdk-ec2 / -ssm     │  │  Cmd  │           |  no key pair,             │
│  │ 1 binary · 2.5 ms      │  │       │           |  no port 22 rule          │
│  └────────────────────────┘  │       │                                       │
│        9 tools               │       │  IAM instance profile carries         │
│  list / start / stop /       │       │  AmazonSSMManagedInstanceCore         │
│  terminate / endpoint /      │       │                                       │
│  run_remote / health ...     │       │                                       │
└──────────────────────────────┘       └───────────────────────────────────────┘

The [RUST] on the right-hand side is vLLM's own Rust frontend — the other article in this
series. This one is the [RUST] on the left: the Rust that drives the box.

What is MCP, in one paragraph?

That last detail matters for the Rust pitch: if the client spawns your process on every
session, process startup is a user-visible cost.

Step 1 — Scaffold

cargo new --bin rust-mcp --name gpu-vllm-g5g-2b-mcp
cd rust-mcp

Now the dependencies. Feature flags are the thing to get right here — cargo add rmcp
on its own compiles fine and gives you almost nothing:

cargo add rmcp --features server,macros,transport-io

Feature	What it brings
`server`	the `ServerHandler` trait and router types
`macros`	`#[tool]`, `#[tool_router]`, `#[tool_handler]`
`transport-io`	stdio transport

The crate also ships client, auth, elicitation, transport-streamable-http-server and
more, all off by default. Add them when you need them.

Then the rest:

cargo add tokio --features rt-multi-thread,macros,process,time
cargo add serde serde_json anyhow schemars
cargo add aws-config aws-sdk-ec2 aws-sdk-ssm aws-sdk-secretsmanager

Resulting Cargo.toml:

[package]
name = "gpu-vllm-g5g-2b-mcp"
version = "0.1.0"
edition = "2024"

[dependencies]
rmcp = { version = "3.1.2", features = ["server", "macros", "transport-io"] }
tokio = { version = "1.53.1", features = ["rt-multi-thread", "macros", "process", "time"] }
aws-config = "1.10.1"
aws-sdk-ec2 = "1.246.0"
aws-sdk-ssm = "1.118.0"
serde = "1.0.229"
serde_json = "1.0.151"
schemars = "1.2.2"
anyhow = "1.0.104"

🔎 Tip: where the canonical examples live

rmcp moves fast, and rendered docs lag. The vendored tests on your own disk are
compiled against the exact version you resolved:

ls ~/.cargo/registry/src/*/rmcp-3.1.2/tests/

tests/test_tool_macros.rs is a complete, working server in about 60 lines. When an API
question comes up, that file answers it faster and more reliably than anything else. ⚡

Step 2 — The server struct

An rmcp server is a struct that owns a ToolRouter:

use rmcp::{
    ErrorData, ServerHandler, ServiceExt,
    handler::server::{router::tool::ToolRouter, wrapper::Parameters},
    model::{CallToolResult, ContentBlock, Implementation, ServerCapabilities, ServerInfo},
    tool, tool_handler, tool_router,
    transport::stdio,
};
use schemars::JsonSchema;
use serde::{Deserialize, Serialize};

#[derive(Clone)]
struct G5gServer {
    tool_router: ToolRouter<Self>,
}

Step 3 — Describe your inputs as types

This is the part that sold me on the whole exercise. Your tool's input is a plain struct, and
schemars turns it into the JSON Schema the agent sees — doc comments and all:

#[derive(Debug, Serialize, Deserialize, JsonSchema)]
struct InstanceId {
    /// EC2 instance id, e.g. `i-0123456789abcdef0`.
    instance_id: String,
}

Step 4 — Write the tools

#[tool_router] on the impl block, #[tool] on each method:

#[tool_router(router = tool_router)]
impl G5gServer {
    fn new() -> Self {
        Self { tool_router: Self::tool_router() }
    }

    #[tool(description = "List EC2 instances tagged ManagedBy=gpu-vllm-g5g-2b.")]
    async fn list_g5g_instances(&self) -> Result<CallToolResult, ErrorData> {
        let conf = aws_config::defaults(aws_config::BehaviorVersion::latest())
            .region(aws_config::Region::new("us-east-1"))
            .load()
            .await;
        let ec2 = aws_sdk_ec2::Client::new(&conf);

        let resp = match ec2.describe_instances()
            .filters(Filter::builder()
                .name("tag:ManagedBy").values("gpu-vllm-g5g-2b").build())
            .send().await
        {
            Ok(r) => r,
            Err(e) => return ok(format!("❌ describe_instances failed: {e}")),
        };

        let mut rows = Vec::new();
        for res in resp.reservations() {
            for inst in res.instances() {
                rows.push(format!("| `{}` | {} | {} |",
                    inst.instance_id().unwrap_or("?"),
                    inst.instance_type().map(|t| t.as_str()).unwrap_or("?"),
                    inst.state().and_then(|s| s.name())
                        .map(|n| n.as_str()).unwrap_or("unknown"),
                ));
            }
        }
        ok(format!("📡 Instances\n\n| id | type | state |\n|---|---|---|\n{}",
                   rows.join("\n")))
    }
}

Tools that take arguments wrap them in Parameters<T>:

    #[tool(description = "Terminate a G5g instance. Permanent — destroys the root volume.")]
    async fn terminate_g5g_instance(
        &self,
        Parameters(args): Parameters<InstanceId>,
    ) -> Result<CallToolResult, ErrorData> {
        // …
    }

And a small helper, since every tool returns the same shape:

fn ok(text: String) -> Result<CallToolResult, ErrorData> {
    Ok(CallToolResult::success(vec![ContentBlock::text(text)]))
}

Step 5 — Implement ServerHandler

#[tool_handler] wires the router in, so you never write a dispatch match:

#[tool_handler(router = self.tool_router)]
impl ServerHandler for G5gServer {
    fn get_info(&self) -> ServerInfo {
        let mut info = ServerInfo::new(
            ServerCapabilities::builder().enable_tools().build()
        );
        info.server_info = Implementation::new(
            "gpu-vllm-g5g-2b", env!("CARGO_PKG_VERSION")
        );
        info.instructions = Some(
            "Devops agent for AWS EC2 G5g (Graviton2 + NVIDIA T4G) serving Gemma 4 \
             under vLLM. Remote administration goes through SSM; there is no inbound SSH."
                .to_string(),
        );
        info
    }
}

Step 6 — main

Four lines:

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let service = G5gServer::new().serve(stdio()).await?;
    service.waiting().await?;
    Ok(())
}

cargo build --release

Step 7 — Test the protocol by hand

An MCP server is a protocol implementation, so test it with a protocol transcript. Three
JSON-RPC lines on stdin — no client required:

printf '%s\n' \
'{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2026-07-28","capabilities":{},"clientInfo":{"name":"probe","version":"0"}}}' \
'{"jsonrpc":"2.0","method":"notifications/initialized"}' \
'{"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}}' \
'{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"get_help","arguments":{}}}' \
| ./target/release/gpu-vllm-g5g-2b-mcp

initialize OK: gpu-vllm-g5g-2b 0.1.0 proto 2026-07-28
tools/list OK: 9 tools -> get_deployment_config, get_endpoint, get_help,
  list_g5g_instances, run_remote, start_g5g_instance, stop_g5g_instance,
  terminate_g5g_instance, verify_model_health
tools/call get_help OK

🟢 rmcp 3.1.2 negotiates the 2026-07-28 spec version by default.

Keep this snippet. It's the fastest way to tell "my server is broken" from "my client config
is broken."

Step 8 — Register it

Point your MCP client at the binary. For Claude Code, .mcp.json:

{
  "mcpServers": {
    "gpu-vllm-g5g-2b": {
      "command": "/abs/path/to/rust-mcp/target/release/gpu-vllm-g5g-2b-mcp",
      "env": { "AWS_REGION": "us-east-1" }
    }
  }
}

The server name prefixes every tool — mcp__gpu-vllm-g5g-2b__list_g5g_instances — so name it
after the thing it manages, especially if you run several.

Credentials come from the standard AWS provider chain, so whatever
aws sts get-caller-identity resolves is what the server gets. Set AWS_PROFILE to pick one.

Tests worth writing

#[test]
fn serve_flags_are_turing_shaped() {
    let f = serve_flags("google/gemma-4-E2B-it", "g5g.2xlarge");
    assert!(f.contains("--dtype float16"), "Turing has no bf16 datapath");
    assert!(f.contains("--kv-cache-dtype auto"), "Turing has no fp8 datapath");
    assert!(!f.contains("attention-backend"));   // not a real vLLM v0.27 variable
}

#[test]
fn unknown_types_are_rejected_and_never_need_swap() {
    assert!(validate_instance_type("t4g.2xlarge").is_err());  // burstable CPU box, no GPU
    assert!(!needs_swap("t4g.2xlarge"));                      // 0 GiB must not read as "tiny"
}

That second one earns its keep: host_memory_gb returns 0 for an unknown instance type,
and a naive ram < 16 would decide an unrecognised machine needs a swapfile.

running 5 tests
test result: ok. 5 passed; 0 failed; finished in 0.00s

🐍 vs 🦀 — the scoreboard

Cold start measured the way a client experiences it: spawn the process, send initialize +
initialized + tools/list, stop the clock when the tool list comes back. Seven runs, median.

	🐍 Python (FastMCP)	🦀 Rust (rmcp)
Cold start to `tools/list`	462.0 ms (437–530)	2.5 ms (1.8–2.9)
Peak RSS	83 MB	12 MB
Artifact	Python runtime + 34 packages	one binary, 39.4 MB (19.9 MB stripped)
Direct dependencies	3	14
Total resolved packages	34	241
Tools implemented	15	9
Source	759 lines	560 + 52 of tests
Clean release build	n/a	5 m 28 s

So, worth it?

For a single MCP server that already works: no. Don't rewrite it.

The 462 ms is a bonus. The schema not being able to lie about the code is the reason. ✅

Cheat sheet

# scaffold
cargo new --bin my-mcp && cd my-mcp
cargo add rmcp --features server,macros,transport-io
cargo add tokio --features rt-multi-thread,macros
cargo add serde serde_json schemars anyhow

# canonical examples for YOUR resolved version
ls ~/.cargo/registry/src/*/rmcp-*/tests/test_tool_macros.rs

# build + smoke test
cargo build --release
printf '%s\n' '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2026-07-28","capabilities":{},"clientInfo":{"name":"p","version":"0"}}}' \
              '{"jsonrpc":"2.0","method":"notifications/initialized"}' \
              '{"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}}' \
  | ./target/release/my-mcp

Four macros to remember: #[tool_router] on the impl, #[tool] on each method,
#[tool_handler] on the ServerHandler impl, Parameters<T> around your input struct.

Shipping a vision-model verdict on Bedrock and Lightsail

xbill — Sun, 16 Aug 2026 12:24:12 +0000

Built 2026-08-15 against us.amazon.nova-lite-v1:0 via the Bedrock Converse API. FastAPI on Python 3.13, deployed to an Amazon Lightsail container service (nano, scale 1) in us-east-1. Scored against the live deployment, not localhost: 20/20 on the fixture set, median 880 ms per scan.

Live: Dog or Not: Lite · Source: github.com/xbill9/dog-or-not-lite · Built for the AWS Weekend Challenge: Build a Creative App.

TL;DR

Make the model fill in a schema instead of writing a sentence. The Converse API's toolConfig plus toolChoice forces a named function call, so is_dog arrives as a boolean because it was declared as one. Every image comes back in the same shape — including the ambiguous ones, which is exactly where free-text output gets creative and a string-matching parser gets it wrong.

The app is a webcam scanner that tells you whether the thing you are holding up is a dog. One HTML page, one POST /api/scan, one model call, no build step, no framework. The whole backend is 285 lines.

Three AWS specifics are worth the price of admission:

Lightsail container services have no IAM task role. There is nothing to attach a policy to, so the container needs a real access key as an environment variable. The mitigation is scope, not secrecy.
A cross-region inference profile is authorized against every region it routes to. With the policy pinned to us-east-1, a call made to us-east-1 was denied naming us-west-2. Measured, not inferred.
--platform linux/amd64 is not optional. An arm64 image builds, pushes and deploys cleanly, then crash-loops with an exec format error that never mentions architecture.

And a mock mode that answers every scan locally is what made the frontend free to build — no credentials, no model access, no bill.

1. The shape: one route, one call

The classification rule is the only opinionated part. is_dog is true only for a living domestic dog: a wolf is not a dog, nor is a coyote, fox, plush toy, bronze statue, cartoon, or person in a costume. That is a choice rather than a fact, and it is what makes the thing measurable — "is this a dog" is solved zero-shot by any modern vision model and has nothing to measure.

The second rule keeps it usable: judge the subject depicted, never the medium carrying it. People test this by holding a photo up on their phone, so a photograph of a real dog is a dog.

Everything else is plumbing:

browser ──HTTPS──> Lightsail container service ──> Amazon Bedrock
 camera             (nano, 1 node, FastAPI)          Nova Lite
 or upload           serves the page AND              vision + tool use
                     the /api/scan route

Two services total. One container, one model, one IAM user. No load balancer, no bucket, no API Gateway, no CDN to invalidate.

Nova Lite is the cheapest Bedrock model that takes an image and supports tool use, which is the exact intersection this needs. The us. prefix matters: it is an inference profile, and in several regions Nova is only served through one. Invoking the bare amazon.nova-lite-v1:0 there fails with a ValidationException that never mentions profiles.

2. Force the verdict into a schema

Declare the tool, then require it. The schema is where the classification rule actually lives — the field descriptions do more work than the system prompt:

TOOL_CONFIG = {
    "tools": [{
        "toolSpec": {
            "name": "report_verdict",
            "description": "Report whether the subject presented to the "
                           "scanner is a dog. Call this exactly once, for "
                           "every image, always.",
            "inputSchema": {"json": {
                "type": "object",
                "properties": {
                    "is_dog": {
                        "type": "boolean",
                        "description": "True only for an actual living dog. "
                                       "False for a wolf, coyote, fox, plush "
                                       "toy, statue, drawing, cartoon or costume.",
                    },
                    "confidence": {"type": "integer", "description": "0-100."},
                    "subject": {
                        "type": "string",
                        "description": "What it actually is, three words or fewer: "
                                       "'golden retriever', 'grey wolf', "
                                       "'ceramic figurine'.",
                    },
                    "is_cat": {
                        "type": "boolean",
                        "description": "True if the subject is a cat. Separate "
                                       "field because a cat is not merely a non-dog.",
                    },
                },
                "required": ["is_dog", "confidence", "subject", "is_cat"],
            }},
        }
    }],
    # Without this, Nova will sometimes narrate instead of calling the tool.
    "toolChoice": {"tool": {"name": "report_verdict"}},
}

The call itself, with the image as raw bytes — Converse takes bytes directly, so no base64 round trip on this side:

response = bedrock().converse(
    modelId="us.amazon.nova-lite-v1:0",
    system=[{"text": SYSTEM_PROMPT}],
    messages=[{"role": "user", "content": [
        {"image": {"format": "jpeg", "source": {"bytes": raw}}},
        {"text": "Identify the subject."},
    ]}],
    toolConfig=TOOL_CONFIG,
    inferenceConfig={"maxTokens": 256, "temperature": 0.2},
)

Pulling the answer out is a loop over content blocks, not a regex:

for block in response["output"]["message"]["content"]:
    use = block.get("toolUse")
    if use and use.get("name") == "report_verdict":
        return Verdict(**use["input"])
raise HTTPException(status_code=502, detail="model did not return a verdict")

toolChoice makes that last line close to unreachable. Keep it anyway — a 502 naming the cause beats a KeyError traceback.

Two things worth stealing:

Put the rule in the field description, not only the prompt. The is_dog description enumerating wolf/coyote/fox/plush/statue is read at the point of decision.
is_cat is a separate boolean, not a value of subject. Anything the UI branches on should be its own typed field. Parsing subject == "tabby cat" to decide whether to show a different state is how you end up back in string-matching.

3. Build the whole frontend with a mock

One environment variable short-circuits the model call and cycles four canned verdicts:

MOCK=1 ./run.sh          # http://127.0.0.1:8080, no credentials, no bill

MOCK_VERDICTS = [
    {"is_dog": True,  "confidence": 97, "subject": "golden retriever", "is_cat": False},
    {"is_dog": False, "confidence": 84, "subject": "grey wolf",        "is_cat": False},
    {"is_dog": False, "confidence": 91, "subject": "tabby cat",        "is_cat": True},
    {"is_dog": False, "confidence": 62, "subject": "plush dachshund",  "is_cat": False},
]

Pick the four deliberately: the happy path, the case that makes the rule interesting, the special state, and a low-confidence one. This is the only way to reach every UI state on demand rather than by going and finding a wolf.

Build the client lazily or this mode does not work at all — constructing a boto3 client at import time fails on a machine that has never authenticated, which is precisely the machine MOCK=1 is for:

_bedrock = None

def bedrock():
    global _bedrock
    if _bedrock is None:
        _bedrock = boto3.client("bedrock-runtime", region_name=AWS_REGION, ...)
    return _bedrock

It paid for itself on one bug. The confidence meter had:

transition: width 0.5s ease, background 0.3s ease;

That background transition left the bar showing the previous verdict's colour indefinitely — green under NOT A DOG, red under the cat state — while the verdict text, which had no transition, switched correctly. Two halves of the same readout disagreeing, permanently, each looking fine alone. You only see it if you can fire three verdicts in two seconds. A state indicator should snap to the state; only the fill level was ever worth animating.

4. Scope the IAM user, and expect a region you never asked for

Lightsail container services cannot assume an IAM role. On ECS or Lambda you attach a policy to a role and the SDK finds credentials. Lightsail has nothing to attach to, so the container gets a real key pair as environment variables — and deployment environment variables are readable afterwards through get-container-services. There is no way to make that elegant. The honest response is to scope the key until it is boring:

./iam-setup.sh    # creates the user, writes ~/dogornot-lite.key, chmod 600

It mints a new access key every run and deletes the old ones, so a re-run rotates rather than fails. The policy is the interesting part:

{
  "Effect": "Allow",
  "Action": ["bedrock:InvokeModel"],
  "Resource": [
    "arn:aws:bedrock:*::foundation-model/amazon.nova-lite-v1:0",
    "arn:aws:bedrock:*:<ACCOUNT_ID>:inference-profile/us.amazon.nova-lite-v1:0"
  ]
}

Note the wildcard regions, and note that you need both ARNs. A cross-region inference profile routes across regions, and Bedrock authorizes InvokeModel against the underlying foundation-model ARN in each one. With the policy pinned to us-east-1, a call made to us-east-1 fails like this:

AccessDeniedException: ... not authorized to perform: bedrock:InvokeModel
on resource: arn:aws:bedrock:us-west-2::foundation-model/amazon.nova-lite-v1:0

us-west-2 was never requested and never configured. I only state this because I narrowed the policy on purpose to watch it fail and then restored it. If you are staring at a Bedrock denial naming a region you never asked for, this is why.

One more denial worth naming: an AccessDeniedException at the first call, before any of the above, usually means Bedrock model access has not been granted for Nova in that account. The console calls it "Model access", and it is per-region.

5. Deploy: one container service

aws login          # or any credential source
./iam-setup.sh     # one-time
./deploy.sh        # build, push, deploy, print the URL

deploy.sh is idempotent — re-running it ships a new deployment version to the same service on the same URL. Budget 5–10 minutes on the first run, most of it Lightsail provisioning the service before it will accept a deployment at all.

Four things in it that are not obvious:

Build for x86 explicitly. Lightsail nodes are amd64. On an arm64 laptop the default build deploys cleanly and then crash-loops with an exec format error that never says "architecture".

docker build --platform linux/amd64 -t dog-or-not-lite:latest .

lightsailctl is a separate binary. aws lightsail push-container-image is a thin wrapper around it and fails confusingly without it. Check with command -v lightsailctl, not by looking in /usr/local/bin — it may be installed under ~/.local/bin.

Read the pushed image reference back from the API, rather than scraping the push output, which prints it in prose:

IMAGE_REF=$(aws lightsail get-container-images --service-name dog-or-not-lite \
    --query 'containerImages[0].image' --output text)

Do not be alarmed that the version is not sequential from 1. A brand-new service's first image came back as :dog-or-not-lite.scanner.219. That is normal.

Keep the secret off the command line. Build the containers JSON with python so the key is JSON-escaped rather than shell-interpolated, write it to a chmod 700 tempdir, and pass it by reference — ps never sees it:

aws lightsail create-container-service-deployment \
    --service-name dog-or-not-lite \
    --containers "file://$TMP_DIR/containers.json" \
    --public-endpoint "file://$TMP_DIR/endpoint.json"

Point the health check at a route that exists (/healthz, intervalSeconds: 10, healthyThreshold: 2). And if you serve static files from the same process, mount them last — a StaticFiles at / mounted before your routes shadows every one of them:

app.mount("/", StaticFiles(directory="static"), name="static")  # must stay last

6. Measure it before you believe it

/healthz passing proves the container booted, not that Bedrock is reachable from inside it. The only real verification is scoring the deployed URL:

./check.py --url https://<service>.us-east-1.cs.amazonlightsail.com --min-rate 0.9

20 images, each compared against a hand-checked is_dog in fixtures/fixtures.json. Stdlib only, so it runs against a deployed URL from anywhere without installing anything. Two details make it worth having:

The fixtures are stored at 640×480 q70 — the exact format the browser sends. The harness exercises the same payload the real client does, not a pristine 4000px original the app will never see.
The expectations were verified by eye before being committed. Generate the input, never the expectation. An eval whose ground truth came out of a model is measuring agreement, not accuracy.

Metric	Against the deployed service
Correct	20/20
Median latency	880 ms
Wolves, foxes, statues, cats	all classified correctly
Breeds named unprompted	"beagle dog", "corgi dog", "german shepherd"

Read that as a smoke test, not a benchmark. Twenty clean, well-lit, subject-fills-frame images say the prompt works and the plumbing is right. They say very little about a dog photographed badly at dusk, and I would not claim from this that the rule is robust.

7. What it costs, and how to turn it off

The standing cost is the container service, not the model. A Lightsail container service bills for as long as it exists, whether or not anyone uses it — roughly $7/month at nano. Deleting the deployment does not stop that. You have to delete the service:

aws lightsail delete-container-service --service-name dog-or-not-lite
aws iam delete-user-policy --user-name dog-or-not-lite --policy-name InvokeNovaLite
aws iam list-access-keys --user-name dog-or-not-lite   # delete each, then:
aws iam delete-user --user-name dog-or-not-lite

Per-scan Bedrock cost is small by comparison — a 640×480 q70 JPEG lands at 40–60 KB — but I have not metered it over a long enough run to publish a number. The app logs inputTokens and outputTokens on every scan if you want your own.

8. Reproduction

git clone https://github.com/xbill9/dog-or-not-lite
cd dog-or-not-lite
pip install -r requirements.txt

# Frontend work: no credentials, no model access, no bill
MOCK=1 ./run.sh                      # http://127.0.0.1:8080

# Real model calls locally (needs Bedrock model access for Nova in your region)
./run.sh
./check.py                           # score 20 fixtures against localhost

# Deploy
./iam-setup.sh                       # scoped IAM user + key at ~/dogornot-lite.key
./deploy.sh                          # build (amd64) → push → deploy → print URL
./check.py --url <printed url> --min-rate 0.9

One caveat if you use aws login locally: those credentials require botocore[crt], which is deliberately not in requirements.txt. Without it boto3 reports Missing Dependency: Using the login credential provider requires ... botocore[crt] and the app returns 502 bedrock unreachable. Install it in your virtualenv only — the container authenticates with a static key pair, which needs no CRT, and adding it would put ~20 MB in the image for nothing.

I built a security scanner that checks if you are a dog

xbill — Sat, 15 Aug 2026 14:11:53 +0000

This is a submission for Weekend Challenge: Dog Days Edition

What I Built

I went looking for a dog in the GPU family tree. I found fish, dead physicists, and a bridge in Florence.

NVIDIA names its architectures after scientists — Tesla, Kepler, Hopper, Blackwell. AMD names its GPUs after fish: Sienna Cichlid, Navy Flounder, Hotpink Bonefish. Intel uses Italian bridges. There is exactly one dog in the entire lineage: Husky, the CPU cores in AMD's 2011 Llano APU. And because Llano was an APU, those Husky cores shared a die with an integrated Radeon — codenamed Sumo. The closest a dog has ever gotten to a GPU is sitting next to one, and even then the graphics half got named after a wrestler.

So I put one there myself.

Dog or Not is a live-video security scanner with one job. You hold something up to the camera, say "scan", and it tells you whether it is a dog. If it is, it barks.

It is deliberately not charming about it. The scanner is a cold threat-assessment system that happens to have been pointed at dogs, and the entire joke is that it does not know it is making one. Hold up a golden retriever and it says, in the flattest voice available:

"Woof."

Hold up a wolf and it says "Negative. Grey wolf." Hold up a cat and it suffers a fatal system error. Get three dogs in frame and it declares a containment breach — in whichever of nine languages you picked.

The classification line is where it gets interesting. A wolf is not a dog. Neither is a coyote, a fox, a plush toy, a bronze statue, a cartoon, or a person in a costume. That is a choice rather than a fact, and it is the choice that makes the thing measurable — "is this a dog" is otherwise solved zero-shot and there is nothing to find out.

Demo

Live: https://dog-or-not-289270257791.us-central1.run.app

Grant camera access, press INITIATE, then hold something up and say "scan".
Chrome or Edge for the voice command — everywhere else, use the SCAN button,
which does exactly the same thing.

Code

xbill9 / devto-dog

A live-video security scanner that checks whether you are holding up a dog, and barks if you are. DEV Weekend Challenge: Dog Days Edition.

Dog or Not

A live-video security scanner with one job: you hold something up to the camera say "scan", and it tells you whether it is a dog. If it is, it barks.

It is deliberately not charming about it. The scanner is a cold threat-assessment system that happens to have been pointed at dogs, and the entire joke is that it does not know it is making one.

Built for the DEV Weekend Challenge: Dog Days Edition.

Live: https://dog-or-not-289270257791.us-central1.run.app

Grant camera access, press INITIATE, hold something up and say "scan" Chrome or Edge for the voice command; everywhere else the SCAN button does the same thing.

Credit where it is due

This is a fork of way-back-home/level_3_new which was the same scanner counting fingers for a biometric handshake. The multimodal plumbing — bidirectional WebSocket, 1 FPS video, local wake-word detection, the accuracy harness, the Cloud…

View on GitHub

Running it yourself is four commands, and only one of them costs anything:

./scripts/install_deps.sh      # NOT a bare pip install -r; see below
make frontend                  # npm ci && vite build — the backend serves dist/
make mock                      # offline fake server on :8080, free, no API key
make run                       # the real Live API — this one bills

The dependency script exists because pip install -r requirements.txt hard-fails here with ResolutionImpossible. requirements.txt pins websockets==17.0.1, deliberately above the caps google-adk (<16) and google-genai (<17) declare — those bounds are "last version we tested", not a real incompatibility, and overrides.txt overrides them. uv applies it with --override; pip has no equivalent, so the script installs the tree without websockets and then forces the pin with --no-deps. Do not "fix" this with a bare --no-deps on the whole file: it skips every transitive dependency too and leaves an app that cannot import.

Forked from way-back-home, which was the same scanner counting fingers for a biometric handshake. The multimodal plumbing came from there — bidirectional WebSocket, 1 FPS video, local wake-word detection, the accuracy harness, the Cloud Run chain — and it is the reason this exists at all in a weekend.

How I Built It

So where does the model actually sit?

Nothing here is a request/response call. The browser holds one WebSocket open for the life of a scan, and the same socket carries frames up and speech down.

Browser (Chrome / Edge)                        Cloud Run  --min-instances=1
┌─────────────────────────────┐                ┌──────────────────────────────┐
│ getUserMedia                │                │ FastAPI  main.py             │
│  ├ video → canvas → JPEG    │  binary  2  →  │  ├ origin allowlist          │
│  │    640×480 q60 @ 1 FPS   │                │  │   (CORS does not apply    │
│  └ audio → Web Speech API   │  {"type":      │  │    to WebSockets)         │
│      never leaves the tab   │   "text"}  →   │  ├ LiveRequestQueue          │
│                             │                │  └ ADK Runner  run_live()    │
│ audioStreamer.js            │  ←  binary  3  │           │                  │
│  └ mu-law → PCM → worklet   │     (mu-law)   │           ▼                  │
│                             │                │  biometric_agent             │
│ DogScanner.jsx              │  ←  {"type":   │   report_verdict             │
│  └ DOG / NOT A DOG          │      "match"}  │   trigger_system_error       │
└─────────────────────────────┘                │   trigger_heavy_metal_mode   │
                                               └──────────────┬───────────────┘
                                                              │ bidiGenerateContent
                                                              ▼
                                                     Gemini Live API

Two decisions in that picture cost the most to arrive at.

The microphone does not stream. Its entire job is to catch the word "scan", and doing that over the wire costs 256 kbit/s of raw PCM — about two thirds of the uplink. Worse, continuous audio is what stops the model taking turns: measured 0/5 with speech in the room against 5/5 for the identical prompts sent as text. So the audio stays in the browser and the wake word sends the same text frame the offline harness sends. Uplink drops from ~385 to ~128 kbit/s, all of it video.

Model audio comes back as mu-law on a binary frame, not base64 in the JSON. Base64 inflates binary by a third, and over a 14-second session that was measured at 220 KB → 95 KB, downlink 128 → 55 kbit/s. The encoder is 40 lines of pure Python; the decoder is a 256-entry table in the browser. Change one and you must change the other — a mismatch is static, not an error.

I did not write this. A self-paced loop did, and I want to talk about what that was actually like — because the interesting part is not that it worked.

Loop-driven development, concretely: instead of a conversation, the agent schedules its own next wake-up. It reads a build log, picks the highest-priority unblocked task, does it, and appends an honest entry about what moved and what broke. Then it decides when to come back — fifteen minutes while there was work, thirty once it ran out — and arms a file watcher so it wakes immediately if the thing it is waiting on arrives.

That build log is in the repo, tick by tick, written as it happened rather than reconstructed afterwards. This section is drawn from it.

Every green checkmark was, at some point, green over something broken

This is the thing I did not expect, and it is the whole reason the post is worth reading.

35 passing tests over an app that could not work. The backend started sending {is_dog, subject, confidence}. The tests went green. The frontend was still reading msg.count || msg.digit off that same frame, so the verdict never reached the UI. Nothing covers that seam — the Python suite stops at the socket and there were no frontend tests at all. The tests measured exactly what they cover, which was not the broken part.

There was a nastier bug hiding in the same line. is_dog is a boolean, so the old msg.count || ... idiom would have silently discarded every NOT-A-DOG verdict — half the answers, and the more interesting half.

Clean lint over a failed build. Deleting a dead function took a live one out with it, four lines below. ESLint passed — it does not resolve cross-module imports by default — and the failure only appeared in vite build, buried under twelve lines of rollup stack trace.

A green test asserting the exact thing it was supposed to prevent. The project keeps a non-public model id out of the repo, enforced by a gate wired into make test and make deploy. One test asserted that id as a string literal. A leak with a checkmark on it.

A gate that caught itself. That same gate failed on its first run — on its own list of patterns, which necessarily contains every string it searches for. Funny once, an infinite loop thereafter.

The same bug, three times, in three places

A route that exists in the real backend and not in the place people actually develop:

/api/config 404ing under the Vite dev proxy, which forwarded only /ws. Symptom: the header read AWAITING LINK — the exact bug that endpoint was written to fix.
/api/config and /api/fixtures missing from the mock server — the documented way to work on the UI without billing a session. The fixture portal could not load a single image there.
Waiting to happen on the next endpoint.

Every one failed soft. Nothing crashed, nothing logged an error, and the feature just looked unfinished. The graceful fallback hid its own cause. There is now a test that diffs the real app's /api/* routes against the mock's and fails naming the difference — and I checked it is not vacuous, because two empty sets compare equal and a parity test over nothing passes forever.

The model that narrates calling a tool and then doesn't

The whole architecture hangs off one tool call: the model sees a subject, calls report_verdict(is_dog, confidence, subject), and everything downstream — UI, bark, scoring — follows from that.

Exactly one non-preview model offers the Live API. On it, every single trial came back SILENT. Meanwhile its own thinking said:

"A visual identification scan reveals a clear image of a Golden Retriever. The subject is verified as a real dog with 95% confidence, as is_dog is true... I'm executing the report_verdict tool with the dog's details, then I will say 'Woof.'"

It sees the dog. It narrates the call. Zero tool calls, ever. Meanwhile the fallback model inherited from the earlier build could not open a Live session at all, under a comment claiming it could — because until the first real session, nothing had ever opened one.

One fixture in twenty was poisoned

The eval set came from Wikimedia Commons, sourced by search term. One of them — "Dog with Goofy plush toy" — is a real dog chewing a toy, filed as not-a-dog. It would have scored every correct answer as a failure.

There is no test for that. The only way to catch it is to look at it. The project's own older documentation already says so: generate the input, never the expectation.

And the harness was scoring obedience as failure

One dog fixture came back SILENT twice, consistently. It is a Halloween dog park — beagle, cavalier, retriever in the background — and the model had called trigger_heavy_metal_mode(). The containment breach fired unprompted, on an unposed real-world photograph.

The harness only scored report_verdict. But the instruction gives the easter eggs absolute priority over report_verdict — so the harness was marking the model doing exactly as it was told as a failure.

Is it actually right, though?

There is exactly one thing in this repo that measures the model rather than the plumbing, and it needs no human in front of the camera:

python scripts/scan_accuracy.py --blur-prob 0.3 --jitter 2 --min-rate 0.8

It drives the real deployed endpoint with fixture images at 640×480 JPEG q60 — the exact format the browser sends — puts a {"type":"text"} frame in where you would say "scan", and scores every report_verdict against a count verified by eye before it was committed. One billed session per run. --blur-prob and --jitter approximate a real webcam; --min-rate turns it into a gate that can fail a build.

What it cannot see: it holds one static fixture per trial, so it is structurally incapable of catching a sampling miss — a pose held for less than a second at 1 FPS that lands in no frame at all. That blind spot is how "1 FPS and 2 FPS score identically" got measured, believed, and written down.

The numbers

Twenty fixtures, one session each:

outcome	n
correct verdict	17	six breeds, wolves, coyotes, foxes, two bronze statues
containment breach	1	correct — three dogs in frame
cat alarm	2	both cats triggered the system error
called a non-dog a dog	0
refused a real dog	0

Latency 0.68–1.56s, mostly around 0.7. It correctly said "bronze statue" for a sculpture of a man with a bronze dog beside him.

What the loop got wrong

It reported the wrong blocker for four hours. It kept saying it was waiting on dog photos, while (a) there was no API key, which it had never checked, and (b) twenty public-domain fixtures with generated attribution turned out to be about ten minutes of work.

It optimised what it could reach instead of what mattered, and reported a blocker it had not verified. That is a very human failure mode and it is worth naming, because the honest version of "I let an AI build this over a weekend" includes it.

Prize Categories

Best Use of Google AI — Gemini Live via the Agent Development Kit. Bidirectional WebSocket streaming video at 1 FPS, structured verdicts through tool calls rather than parsed prose, and a nine-language session config where the model translates its own lines rather than reading a shipped phrasebook. That last part is the cheapest possible proof the model is really being called: a recording cannot answer in Japanese.

Best Use of ElevenLabs — the bark pack, generated with the Sound Effects API at build time, never at runtime. The clips are fetched and decoded once and held in memory, so the bark adds zero latency to the response path and cannot fail during a session. This project had already measured what a second audio stream does to a Live session — 0/5 against 5/5 — and the way to use a sound API here was to make the app touch the network less, not more.

Sound effects generated with ElevenLabs. Fixture images from Wikimedia Commons, attributed in tests/fixtures/ATTRIBUTION.md.

Measured on: gemini-3.1-flash-live-preview via google-adk==2.6.3 / google-genai==2.17.0, Python 3.13, deployed to Cloud Run in us-central1 with --min-instances=1 and --timeout=3600 (a WebSocket is one long request; the default 300s cap would end every session at five minutes). Video 640×480 JPEG q60 at 1 FPS, audio 16 kHz PCM up / 24 kHz down. Client tested in Chrome — the wake word is SpeechRecognition, which is Chrome and Edge only; everywhere else the SCAN button does the identical thing.

Serving Gemma4 with Rust on vLLM 🦀

xbill — Fri, 14 Aug 2026 20:43:34 +0000

This tutorial walks through installing and setting up the Rust toolchain for vLLM on an
AWS EC2 G5g instance — Graviton2 (aarch64) with an NVIDIA T4G GPU — and getting vLLM's
Rust frontend (vllm-rs) built, running, and verified.

This paper is a follow-on to the original G5g Gemma 4 build.

Everything below was run on the box. 🦀

Wait, vLLM has Rust in it?

You betcha. Since PR #40848 (merged
2026-05-21), vLLM vendors a 14-crate Rust workspace:

bench  chat  cmd  engine-core-client  llm  managed-engine  metrics
mock-engine  parser  parser/python  server  text  tokenizer  tracing

Edition 2024, resolver 3. Straight from the vendored rust/Cargo.toml:

Crate	Version	Job
`axum`	0.8.8	the HTTP server
`tokio`	1.47.1	async runtime
`zeromq`	0.6.0	talks to the Python engine
`rmp-serde` / `rmpv`	1.3.1	msgpack on the wire
`minijinja`	2.22	chat templates
`tonic` / `prost`	0.14.6 / 0.14.3	gRPC — remember this one

It's a drop-in replacement for the Python FastAPI server. Two artifacts get built:

🦀 vllm-rs — the axum frontend binary
🐍 vllm._rust_tool_parser — a PyO3 extension module

Rust is a build requirement now

That's the headline, and it's reason enough on its own: you cannot build vLLM from source at
v0.27.2rc0 without Rust in the picture. setup.py imports it at module scope, line 21,
unguarded:

from setuptools_rust.build import build_rust

No try, no feature flag, no opt-out. Metadata generation doesn't happen without it.

And this isn't a quirk of one release. vLLM's Rust surface is 14 crates covering the HTTP
frontend, the tool parser, the tokenizer and the benchmark client, and it has been growing
since it landed. If you build inference infrastructure from source, a Rust toolchain is
becoming table stakes — so it's worth knowing how to drive it properly rather than working
around it.

Three things do get conflated, though, and they have different scopes:

Component	Needed to build vLLM?	Needed to serve?
`setuptools_rust` (Python pkg)	yes, always	no
`cargo` / `rustc` toolchain	for working Rust artifacts	no
`protoc`	for `vllm-rs` specifically	no

Then why doesn't `pip install vllm` need this?

Because normally pip installs it for you. pyproject.toml declares it:

[build-system]
requires = [
    "cmake>=3.26.1", "ninja", "packaging>=24.2",
    "setuptools>=77.0.3,<81.0.0", "setuptools-scm>=8.0",
    "setuptools-rust>=1.9.0",          # <- pip grabs this automatically
    "torch == 2.13.0",                 # <- ...and this. Which is the problem.
    "wheel", "jinja2",
]

Under normal build isolation, pip creates a clean env, installs that list, and builds.
You never see setuptools_rust because you never had to think about it.

But look at the torch pin. Building in isolation means pip installs torch 2.13.0 from
PyPI — and the PyPI aarch64 wheels are built for sm_80 and up. No sm_75. Which
destroys the entire reason for building from source on a T4G.

So on this box you must build against the DLAMI's own torch, and that means:

python use_existing_torch.py
pip install -e . --no-build-isolation

--no-build-isolation turns off the automatic install of everything in that requires
list. From that moment on, every build dependency is yours to supply by hand — including
setuptools_rust, which is why it turns up as a bare ModuleNotFoundError minutes into a
build that has nothing visibly to do with Rust.

So the toolchain was always required; isolation was just hiding it. Building this way means
you own the dependency list, which is the rest of this walk-through. ⚡

What the DLAMI gives you, and what it doesn't

The AWS Deep Learning ARM64 AMI ships a runtime, not a build environment. On a fresh box:

Thing	Present?
PyTorch 2.12 with `sm_75`	✅
NVIDIA driver	✅
`nvcc` / CUDA toolkit	❌
Rust toolchain	❌
`setuptools_rust`	❌
`protoc`	❌

Four of those six are on you. Let's install them.

Step 1 — Rust itself

Standard rustup, nothing aarch64-specific about it:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
. "$HOME/.cargo/env"

stable-aarch64-unknown-linux-gnu installed - rustc 1.97.1 (8bab26f4f 2026-07-14)

Rust is installed now. Great!

Note the triple: stable-aarch64-unknown-linux-gnu. Rust's aarch64 support is a complete
non-event, which is a lovely change of pace on this hardware. ⚡

Step 2 — setuptools-rust

python3 -m pip install setuptools_rust

Per the section above: --no-build-isolation means pip won't do this for you. Do it early
— the failure lands during metadata generation, minutes into a build, as a bare
ModuleNotFoundError: No module named 'setuptools_rust' nowhere near anything that looks
like Rust.

⚠️ Install it into the same interpreter you'll build with. On the DLAMI that's
/opt/pytorch/bin/python3, not the system python3 — they're different, and the one that
matters is whichever owns the torch you're building against.

Step 3 — protoc 🔎

This is the one nobody documents:

apt-get install -y protobuf-compiler
protoc --version

libprotoc 3.21.12

Why: vllm-rs depends on the vllm-server crate, vllm-server builds gRPC stubs with
tonic/prost, and prost-build shells out to protoc. Skip it and the frontend binary
does not get built — see the summary at the end for how loudly that doesn't fail.

The tool parser has no protobuf dependency, which is why it builds either way.

Step 4 — the CUDA toolkit, while you're here

Not Rust, but the same class of problem, and you need it for vLLM's kernels:

# NVIDIA's **sbsa** repo — not the x86 one, easy reflex to get wrong on Arm
apt-get install -y cuda-toolkit-13-2

Step 5 — build the Rust artifacts

cd /opt/vllm-src
python tools/build_rust.py --release

⚠️ Do not omit --release. setuptools-rust builds inplace targets in debug by default,
and pip install -e . is an inplace build. The difference is not subtle:

Artifact	Debug	Release
`_rust_tool_parser.abi3.so`	100,913,216 B	1,009,080 B

100x. The debug artifact is four times the size of every CUDA kernel in vLLM combined.

Timing on a g5g.xlarge (4 vCPU), cold:

real    9m1.746s
user    25m9.199s
sys     1m35.023s

501 crates. Zero warnings. Exit 0. 🟢

Rust's aarch64 support does not put up a fight here — which is a pleasant contrast with the
CUDA side of this box, where SM 7.5 on Graviton needs a custom arch list and a patched
kernel.

Step 6 — check what you got

ls -la vllm/vllm-rs vllm/_rust_tool_parser.abi3.so

-rwxr-xr-x 1 root root 50039024 vllm/vllm-rs
-rwxr-xr-x 1 root root  1009080 vllm/_rust_tool_parser.abi3.so

file vllm/vllm-rs

ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV),
dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, not stripped

vllm/vllm-rs --help

Rust frontend and managed-engine CLI for vLLM.

Commands:
  frontend  Run the Rust OpenAI frontend as a Python-supervised worker
  serve     Launch a managed Python headless engine, then run the Rust OpenAI frontend
  bench     Run vLLM benchmarks
  render    Run engine-free request rendering and preprocessing

If vllm/vllm-rs isn't there, go back to Step 3.

Step 7 — run it, and mind the entrypoint ⚠️

VLLM_USE_RUST_FRONTEND=1 vllm serve google/gemma-4-E2B-it \
  --dtype float16 \
  --kv-cache-dtype auto \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 8 \
  --tensor-parallel-size 1 \
  --host 0.0.0.0 --port 8000

It must be vllm serve. If you launch the module directly —

# ❌ VLLM_USE_RUST_FRONTEND is IGNORED here
python -m vllm.entrypoints.openai.api_server --model … --host 0.0.0.0 --port 8000

— the variable does nothing. No warning, no Unknown vLLM environment variable line. The
server comes up healthy and serves happily on the Python frontend, and a benchmark run
against it looks entirely normal.

The flag is read in exactly two places:

vllm/entrypoints/cli/serve.py:62        envs.VLLM_RUST_FRONTEND_PATH if envs.VLLM_USE_RUST_FRONTEND else None
vllm/entrypoints/openai/dp_supervisor.py:261   if envs.VLLM_USE_RUST_FRONTEND and envs.VLLM_RUST_FRONTEND_PATH:

api_server.py never mentions it.

How do I know it's actually Rust? 🔎

Three checks. Do all three the first time.

1. The server: header:

curl -si localhost:8000/health | grep -i '^server:'

Frontend	Response
🐍 Python	`server: uvicorn`
🦀 Rust	(no `server:` header at all)

2. The process:

pgrep -af vllm-rs

26588 /opt/vllm-src/vllm/vllm-rs frontend --listen-fd 17
  --input-address  ipc:///tmp/f60f3962-d45b-4bcd-9026-c0dc32736028
  --output-address ipc:///tmp/5f75411d-2787-43bb-b4fc-14bf504a1cce
  --engine-start-index 0 --engine-count 1 --data-parallel-size 1

3. The log prefix — (RustFrontend pid=…) instead of (APIServer pid=…):

INFO [utils.py:392] Launching Rust frontend: /opt/vllm-src/vllm/vllm-rs frontend --listen-fd 17 …

So where does Rust actually sit?

In two places, and they're quite different. One is a separate process; the other is a
shared object loaded inside the Python process. Here's the whole VM:

┌─ EC2 g5g.4xlarge ── Graviton2, aarch64 ─────────────────────────────────────┐
│                                                                             │
│  Deep Learning ARM64 AMI · Ubuntu 24.04 · NVIDIA driver 595.71.05           │
│  you add > cuda-toolkit-13-2 (sbsa) · rustup 1.97.1 · protobuf-compiler     │
│                                                                             │
│      HTTP :8000                                                             │
│          |                                                                  │
│          v                                                                  │
│  ┌───────────────────────────────┐                                          │
│  │ [RUST] vllm-rs                │  50 MB aarch64 ELF, its OWN process      │
│  │        axum 0.8.8 · tokio     │  built from the vendored rust/ workspace │
│  │        minijinja · fastokens  │  <- Step 5                               │
│  └────────┬─────────────▲────────┘                                          │
│           |             |                                                   │
│  ipc://   | ROUTER      | PULL     msgpack (rmp-serde / rmpv)               │
│           v             |                                                   │
│  ┌────────┴─────────────┴────────┐                                          │
│  │ [PY]   vLLM supervisor        │  `vllm serve` opens the socket, then     │
│  │                               │  hands listen-fd 17 down to vllm-rs      │
│  └────────┬──────────────────────┘                                          │
│           | spawns                                                          │
│           v                                                                 │
│  ┌───────────────────────────────┐                                          │
│  │ [PY]   EngineCore             │  torch 2.12.0+cu132, arch list has sm_75 │
│  │  ┌─────────────────────────┐  │                                          │
│  │  │ [RUST] _rust_tool_parser│  │  PyO3 .so LOADED INTO the Python         │
│  │  │        1.0 MB release   │  │  process — not a process of its own      │
│  │  └─────────────────────────┘  │                                          │
│  └────────┬──────────────────────┘                                          │
│           | CUDA                                                            │
│           v                                                                 │
│  ┌───────────────────────────────┐                                          │
│  │ NVIDIA T4G · SM 7.5           │  15,360 MiB GDDR6 · 277 GB/s measured    │
│  │ TRITON_ATTN kernels           │  weights 9.94 GiB · KV 2.95 GiB          │
│  └───────────────────────────────┘                                          │
└─────────────────────────────────────────────────────────────────────────────┘

Two things worth pulling out of that picture:

vllm-rs is not a sidecar you point at a port. The Python side opens the listening socket and passes the file descriptor down. It's a worker the supervisor forks and feeds.
_rust_tool_parser is Rust living inside Python. It's the one that always builds (no protoc needed), which is why a broken install still leaves Rust on the box — just not the Rust you wanted.

And note where the GPU sits relative to all of this: at the bottom, behind everything. That's
the reason the benchmark below comes out the way it does.

Bonus: there's a Rust benchmark client too

VLLM_USE_RUST_BENCH=1 vllm bench serve …

Same binary, bench subcommand. Requires VLLM_RUST_FRONTEND_PATH to resolve, so it needs
the same Step 3 → Step 5 you just did.

Two warnings you should not scroll past 🔴

The server came up healthy. These went by in the startup log anyway.

Gemma 4 defeats the fast tokenizer:

INFO    [hf.rs:200] loading tokenizer with fastokens
WARNING [hf.rs:221] failed to load tokenizer with fastokens; falling back to
        HuggingFace tokenizers
        error=tokenizer error: normalizer error: unsupported normalizer type: Replace

fastokens 0.2.1 doesn't implement the Replace normalizer that Gemma 4's tokenizer.json
uses, so it falls back to the same HuggingFace tokenizers the Python path uses. Note the
fallback is graceful and correct — you just don't get the fast path on this model yet. It's
a coverage gap in a young crate, and one normalizer away from closing.

Multimodal isn't wired up for this model:

WARNING [multimodal.rs:446] multimodal model spec is not registered; disabling
        image/video support   model_id="google/gemma-4-E2B-it" model_type="gemma4"

Gemma 4 E2B is a vision model, and gemma4 isn't in the Rust multimodal spec table yet.
Text requests behave identically and the endpoint is healthy, so nothing in a normal check
reveals it. Also a registration gap rather than a design problem — but check it for your model
before you switch, because a healthy endpoint won't tell you.

Is it faster?

On a T4G, no. Output token throughput, same engine config, client on the box against
localhost:

Concurrency	🐍 Python	🦀 Rust
1	28.65	29.30
4	97.48	97.26
8	168.33	169.39
16	169.96	170.19
32	170.99	170.34

Median TTFT tracks just as tightly — 14305 ms against 14311 ms at concurrency 32.

That's the expected result, and worth saying plainly: decode on this card is
bandwidth-bound at a measured 277 GB/s, and the engine saturates at --max-num-seqs 8.
A frontend rewrite targets CPU-side per-request overhead. Here that overhead hides behind the
GPU, so swapping it can't move a bottleneck-limited number. If you want the Rust frontend
to buy you tokens per second on a small GPU, it won't.

One signal does appear, in median inter-token latency at high concurrency:

Concurrency	🐍 Python	🦀 Rust	Δ
16	38.55	36.18	−6.4%
32	38.41	36.23	−5.9%

Mean TPOT barely moves, so this is the middle of the distribution tightening rather than
everything speeding up — the shape you'd expect from a frontend scheduling streaming work
more evenly once many streams are in flight. Worth knowing if you serve at concurrency; not
worth switching for on its own. 📊

If a plain `pip install -e .` already ran

A from-source vLLM install done without the steps above succeeds, exits 0, and leaves you
with a 96 MB debug tool parser and no frontend binary. Four defaults stack up to make that
silent:

Symptom	Cause	Fix
No `vllm/vllm-rs` after a clean build	`protoc` absent ⇒ `vllm-server` fails with code 101	Step 3
`pip install` exits 0 anyway	`optional=not should_require_rust_frontend()` — setuptools-rust swallows it	`VLLM_REQUIRE_RUST_FRONTEND=1`
`_rust_tool_parser.abi3.so` is ~96 MB	editable ⇒ inplace ⇒ debug profile	`--release`
`FileNotFoundError: … vllm-rs was not found`	the above, discovered at import time	Steps 3 + 5
Healthy server, but `server: uvicorn`	flag set on the `api_server` module, which never reads it	`vllm serve`

VLLM_REQUIRE_RUST_FRONTEND=1 turns the second row into a hard build failure, which is what
you want on any machine you plan to serve from.

Cheat sheet

# toolchain — you supply these by hand because the sm_75 requirement
# forces --no-build-isolation, which disables pip's automatic build deps
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
. "$HOME/.cargo/env"
/opt/pytorch/bin/python3 -m pip install setuptools_rust   # the BUILD interpreter
apt-get install -y protobuf-compiler cuda-toolkit-13-2

# build against the DLAMI's torch, not a PyPI one (PyPI aarch64 has no sm_75)
cd /path/to/vllm
python use_existing_torch.py
TORCH_CUDA_ARCH_LIST=7.5 VLLM_REQUIRE_RUST_FRONTEND=1 \
  pip install -e . --no-build-isolation

# Rust artifacts, release profile (editable installs default to debug: 100x bigger)
VLLM_REQUIRE_RUST_FRONTEND=1 python tools/build_rust.py --release

# confirm
ls -la vllm/vllm-rs && vllm/vllm-rs --help

# run — `vllm serve`, NOT the api_server module
VLLM_USE_RUST_FRONTEND=1 vllm serve <model> --host 0.0.0.0 --port 8000

# verify it's really Rust
curl -si localhost:8000/health | grep -i '^server:'   # Rust sends none
pgrep -af vllm-rs

Run on EC2 g5g.xlarge and g5g.4xlarge, us-east-1a, NVIDIA T4G (SM 7.5). vLLM
0.27.2rc1.dev0+g7f7a32cfe, rustc 1.97.1, setuptools-rust 1.13.0, libprotoc 3.21.12,
torch 2.12.0+cu132. Benchmarks are one run per cell for Rust and two for Python; treat the
TPOT delta as suggestive.

Installing Rust for vLLM on Graviton: a G5g walk-through 🦀

xbill — Fri, 14 Aug 2026 20:37:16 +0000

This paper is a follow-on to the original G5g Gemma 4 build.

Everything below was run on the box. 🦀

Wait, vLLM has Rust in it?

You betcha. Since PR #40848 (merged
2026-05-21), vLLM vendors a 14-crate Rust workspace:

bench  chat  cmd  engine-core-client  llm  managed-engine  metrics
mock-engine  parser  parser/python  server  text  tokenizer  tracing

Edition 2024, resolver 3. Straight from the vendored rust/Cargo.toml:

Crate	Version	Job
`axum`	0.8.8	the HTTP server
`tokio`	1.47.1	async runtime
`zeromq`	0.6.0	talks to the Python engine
`rmp-serde` / `rmpv`	1.3.1	msgpack on the wire
`minijinja`	2.22	chat templates
`tonic` / `prost`	0.14.6 / 0.14.3	gRPC — remember this one

It's a drop-in replacement for the Python FastAPI server. Two artifacts get built:

🦀 vllm-rs — the axum frontend binary
🐍 vllm._rust_tool_parser — a PyO3 extension module

Rust is a build requirement now

That's the headline, and it's reason enough on its own: you cannot build vLLM from source at
v0.27.2rc0 without Rust in the picture. setup.py imports it at module scope, line 21,
unguarded:

from setuptools_rust.build import build_rust

No try, no feature flag, no opt-out. Metadata generation doesn't happen without it.

Three things do get conflated, though, and they have different scopes:

Component	Needed to build vLLM?	Needed to serve?
`setuptools_rust` (Python pkg)	yes, always	no
`cargo` / `rustc` toolchain	for working Rust artifacts	no
`protoc`	for `vllm-rs` specifically	no

Then why doesn't `pip install vllm` need this?

Because normally pip installs it for you. pyproject.toml declares it:

[build-system]
requires = [
    "cmake>=3.26.1", "ninja", "packaging>=24.2",
    "setuptools>=77.0.3,<81.0.0", "setuptools-scm>=8.0",
    "setuptools-rust>=1.9.0",          # <- pip grabs this automatically
    "torch == 2.13.0",                 # <- ...and this. Which is the problem.
    "wheel", "jinja2",
]

Under normal build isolation, pip creates a clean env, installs that list, and builds.
You never see setuptools_rust because you never had to think about it.

So on this box you must build against the DLAMI's own torch, and that means:

python use_existing_torch.py
pip install -e . --no-build-isolation

So the toolchain was always required; isolation was just hiding it. Building this way means
you own the dependency list, which is the rest of this walk-through. ⚡

What the DLAMI gives you, and what it doesn't

The AWS Deep Learning ARM64 AMI ships a runtime, not a build environment. On a fresh box:

Thing	Present?
PyTorch 2.12 with `sm_75`	✅
NVIDIA driver	✅
`nvcc` / CUDA toolkit	❌
Rust toolchain	❌
`setuptools_rust`	❌
`protoc`	❌

Four of those six are on you. Let's install them.

Step 1 — Rust itself

Standard rustup, nothing aarch64-specific about it:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
. "$HOME/.cargo/env"

stable-aarch64-unknown-linux-gnu installed - rustc 1.97.1 (8bab26f4f 2026-07-14)

Rust is installed now. Great!

Note the triple: stable-aarch64-unknown-linux-gnu. Rust's aarch64 support is a complete
non-event, which is a lovely change of pace on this hardware. ⚡

Step 2 — setuptools-rust

python3 -m pip install setuptools_rust

Step 3 — protoc 🔎

This is the one nobody documents:

apt-get install -y protobuf-compiler
protoc --version

libprotoc 3.21.12

The tool parser has no protobuf dependency, which is why it builds either way.

Step 4 — the CUDA toolkit, while you're here

Not Rust, but the same class of problem, and you need it for vLLM's kernels:

# NVIDIA's **sbsa** repo — not the x86 one, easy reflex to get wrong on Arm
apt-get install -y cuda-toolkit-13-2

Step 5 — build the Rust artifacts

cd /opt/vllm-src
python tools/build_rust.py --release

⚠️ Do not omit --release. setuptools-rust builds inplace targets in debug by default,
and pip install -e . is an inplace build. The difference is not subtle:

Artifact	Debug	Release
`_rust_tool_parser.abi3.so`	100,913,216 B	1,009,080 B

100x. The debug artifact is four times the size of every CUDA kernel in vLLM combined.

Timing on a g5g.xlarge (4 vCPU), cold:

real    9m1.746s
user    25m9.199s
sys     1m35.023s

501 crates. Zero warnings. Exit 0. 🟢

Rust's aarch64 support does not put up a fight here — which is a pleasant contrast with the
CUDA side of this box, where SM 7.5 on Graviton needs a custom arch list and a patched
kernel.

Step 6 — check what you got

ls -la vllm/vllm-rs vllm/_rust_tool_parser.abi3.so

-rwxr-xr-x 1 root root 50039024 vllm/vllm-rs
-rwxr-xr-x 1 root root  1009080 vllm/_rust_tool_parser.abi3.so

file vllm/vllm-rs

ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV),
dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, not stripped

vllm/vllm-rs --help

Rust frontend and managed-engine CLI for vLLM.

Commands:
  frontend  Run the Rust OpenAI frontend as a Python-supervised worker
  serve     Launch a managed Python headless engine, then run the Rust OpenAI frontend
  bench     Run vLLM benchmarks
  render    Run engine-free request rendering and preprocessing

If vllm/vllm-rs isn't there, go back to Step 3.

Step 7 — run it, and mind the entrypoint ⚠️

VLLM_USE_RUST_FRONTEND=1 vllm serve google/gemma-4-E2B-it \
  --dtype float16 \
  --kv-cache-dtype auto \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 8 \
  --tensor-parallel-size 1 \
  --host 0.0.0.0 --port 8000

It must be vllm serve. If you launch the module directly —

# ❌ VLLM_USE_RUST_FRONTEND is IGNORED here
python -m vllm.entrypoints.openai.api_server --model … --host 0.0.0.0 --port 8000

The flag is read in exactly two places:

vllm/entrypoints/cli/serve.py:62        envs.VLLM_RUST_FRONTEND_PATH if envs.VLLM_USE_RUST_FRONTEND else None
vllm/entrypoints/openai/dp_supervisor.py:261   if envs.VLLM_USE_RUST_FRONTEND and envs.VLLM_RUST_FRONTEND_PATH:

api_server.py never mentions it.

How do I know it's actually Rust? 🔎

Three checks. Do all three the first time.

1. The server: header:

curl -si localhost:8000/health | grep -i '^server:'

Frontend	Response
🐍 Python	`server: uvicorn`
🦀 Rust	(no `server:` header at all)

2. The process:

pgrep -af vllm-rs

26588 /opt/vllm-src/vllm/vllm-rs frontend --listen-fd 17
  --input-address  ipc:///tmp/f60f3962-d45b-4bcd-9026-c0dc32736028
  --output-address ipc:///tmp/5f75411d-2787-43bb-b4fc-14bf504a1cce
  --engine-start-index 0 --engine-count 1 --data-parallel-size 1

3. The log prefix — (RustFrontend pid=…) instead of (APIServer pid=…):

INFO [utils.py:392] Launching Rust frontend: /opt/vllm-src/vllm/vllm-rs frontend --listen-fd 17 …

So where does Rust actually sit?

In two places, and they're quite different. One is a separate process; the other is a
shared object loaded inside the Python process. Here's the whole VM:

┌─ EC2 g5g.4xlarge ── Graviton2, aarch64 ─────────────────────────────────────┐
│                                                                             │
│  Deep Learning ARM64 AMI · Ubuntu 24.04 · NVIDIA driver 595.71.05           │
│  you add > cuda-toolkit-13-2 (sbsa) · rustup 1.97.1 · protobuf-compiler     │
│                                                                             │
│      HTTP :8000                                                             │
│          |                                                                  │
│          v                                                                  │
│  ┌───────────────────────────────┐                                          │
│  │ [RUST] vllm-rs                │  50 MB aarch64 ELF, its OWN process      │
│  │        axum 0.8.8 · tokio     │  built from the vendored rust/ workspace │
│  │        minijinja · fastokens  │  <- Step 5                               │
│  └────────┬─────────────▲────────┘                                          │
│           |             |                                                   │
│  ipc://   | ROUTER      | PULL     msgpack (rmp-serde / rmpv)               │
│           v             |                                                   │
│  ┌────────┴─────────────┴────────┐                                          │
│  │ [PY]   vLLM supervisor        │  `vllm serve` opens the socket, then     │
│  │                               │  hands listen-fd 17 down to vllm-rs      │
│  └────────┬──────────────────────┘                                          │
│           | spawns                                                          │
│           v                                                                 │
│  ┌───────────────────────────────┐                                          │
│  │ [PY]   EngineCore             │  torch 2.12.0+cu132, arch list has sm_75 │
│  │  ┌─────────────────────────┐  │                                          │
│  │  │ [RUST] _rust_tool_parser│  │  PyO3 .so LOADED INTO the Python         │
│  │  │        1.0 MB release   │  │  process — not a process of its own      │
│  │  └─────────────────────────┘  │                                          │
│  └────────┬──────────────────────┘                                          │
│           | CUDA                                                            │
│           v                                                                 │
│  ┌───────────────────────────────┐                                          │
│  │ NVIDIA T4G · SM 7.5           │  15,360 MiB GDDR6 · 277 GB/s measured    │
│  │ TRITON_ATTN kernels           │  weights 9.94 GiB · KV 2.95 GiB          │
│  └───────────────────────────────┘                                          │
└─────────────────────────────────────────────────────────────────────────────┘

Two things worth pulling out of that picture:

vllm-rs is not a sidecar you point at a port. The Python side opens the listening socket and passes the file descriptor down. It's a worker the supervisor forks and feeds.
_rust_tool_parser is Rust living inside Python. It's the one that always builds (no protoc needed), which is why a broken install still leaves Rust on the box — just not the Rust you wanted.

And note where the GPU sits relative to all of this: at the bottom, behind everything. That's
the reason the benchmark below comes out the way it does.

Bonus: there's a Rust benchmark client too

VLLM_USE_RUST_BENCH=1 vllm bench serve …

Same binary, bench subcommand. Requires VLLM_RUST_FRONTEND_PATH to resolve, so it needs
the same Step 3 → Step 5 you just did.

Two warnings you should not scroll past 🔴

The server came up healthy. These went by in the startup log anyway.

Gemma 4 defeats the fast tokenizer:

INFO    [hf.rs:200] loading tokenizer with fastokens
WARNING [hf.rs:221] failed to load tokenizer with fastokens; falling back to
        HuggingFace tokenizers
        error=tokenizer error: normalizer error: unsupported normalizer type: Replace

Multimodal isn't wired up for this model:

WARNING [multimodal.rs:446] multimodal model spec is not registered; disabling
        image/video support   model_id="google/gemma-4-E2B-it" model_type="gemma4"

Is it faster?

On a T4G, no. Output token throughput, same engine config, client on the box against
localhost:

Concurrency	🐍 Python	🦀 Rust
1	28.65	29.30
4	97.48	97.26
8	168.33	169.39
16	169.96	170.19
32	170.99	170.34

Median TTFT tracks just as tightly — 14305 ms against 14311 ms at concurrency 32.

One signal does appear, in median inter-token latency at high concurrency:

Concurrency	🐍 Python	🦀 Rust	Δ
16	38.55	36.18	−6.4%
32	38.41	36.23	−5.9%

If a plain `pip install -e .` already ran

A from-source vLLM install done without the steps above succeeds, exits 0, and leaves you
with a 96 MB debug tool parser and no frontend binary. Four defaults stack up to make that
silent:

Symptom	Cause	Fix
No `vllm/vllm-rs` after a clean build	`protoc` absent ⇒ `vllm-server` fails with code 101	Step 3
`pip install` exits 0 anyway	`optional=not should_require_rust_frontend()` — setuptools-rust swallows it	`VLLM_REQUIRE_RUST_FRONTEND=1`
`_rust_tool_parser.abi3.so` is ~96 MB	editable ⇒ inplace ⇒ debug profile	`--release`
`FileNotFoundError: … vllm-rs was not found`	the above, discovered at import time	Steps 3 + 5
Healthy server, but `server: uvicorn`	flag set on the `api_server` module, which never reads it	`vllm serve`

VLLM_REQUIRE_RUST_FRONTEND=1 turns the second row into a hard build failure, which is what
you want on any machine you plan to serve from.

Cheat sheet

# toolchain — you supply these by hand because the sm_75 requirement
# forces --no-build-isolation, which disables pip's automatic build deps
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
. "$HOME/.cargo/env"
/opt/pytorch/bin/python3 -m pip install setuptools_rust   # the BUILD interpreter
apt-get install -y protobuf-compiler cuda-toolkit-13-2

# build against the DLAMI's torch, not a PyPI one (PyPI aarch64 has no sm_75)
cd /path/to/vllm
python use_existing_torch.py
TORCH_CUDA_ARCH_LIST=7.5 VLLM_REQUIRE_RUST_FRONTEND=1 \
  pip install -e . --no-build-isolation

# Rust artifacts, release profile (editable installs default to debug: 100x bigger)
VLLM_REQUIRE_RUST_FRONTEND=1 python tools/build_rust.py --release

# confirm
ls -la vllm/vllm-rs && vllm/vllm-rs --help

# run — `vllm serve`, NOT the api_server module
VLLM_USE_RUST_FRONTEND=1 vllm serve <model> --host 0.0.0.0 --port 8000

# verify it's really Rust
curl -si localhost:8000/health | grep -i '^server:'   # Rust sends none
pgrep -af vllm-rs

Building and Serving vLLM with Rust

xbill — Fri, 14 Aug 2026 20:12:39 +0000

This tutorial walks through deploying vLLM and some of the key Rust tools used for building and deployment.

This tutorial walks through installing and setting up the Rust toolchain for vLLM on an AWS EC2 G5g instance — Graviton2 (aarch64) with an NVIDIA T4G GPU — and getting vLLM’s Rust frontend (vllm-rs) built, running, and verified.

If you build vLLM from source, some of this applies to you on any architecture: since v0.27.2rc0 the build has a hard Rust dependency. The aarch64 + Turing box is just where every sharp edge shows up at once.

Everything below was run on the box. 🦀

Wait, vLLM has Rust in it?

You betcha. Since PR #40848 (merged 2026–05–21), vLLM vendors a 14-crate Rust workspace :

bench chat cmd engine-core-client llm managed-engine metrics
mock-engine parser parser/python server text tokenizer tracing

Edition 2024, resolver 3. Straight from the vendored rust/Cargo.toml:

It’s a drop-in replacement for the Python FastAPI server. Two artifacts get built:

🦀 vllm-rs — the axum frontend binary
🐍 vllm._rust_tool_parser — a PyO3 extension module

Rust is a build requirement now

That’s the headline, and it’s reason enough on its own: you cannot build vLLM from source at v0.27.2rc0 without Rust in the picture. setup.py imports it at module scope, line 21, unguarded:

from setuptools_rust.build import build_rust

No try, no feature flag, no opt-out. Metadata generation doesn't happen without it.

And this isn’t a quirk of one release. vLLM’s Rust surface is 14 crates covering the HTTP frontend, the tool parser, the tokenizer and the benchmark client, and it has been growing since it landed. If you build inference infrastructure from source, a Rust toolchain is becoming table stakes — so it’s worth knowing how to drive it properly rather than working around it.

Three things do get conflated, though, and they have different scopes:

Then why doesn’t pip install vllm need this?

Because normally pip installs it for you. pyproject.toml declares it:

[build-system]
requires = [
    "cmake>=3.26.1", "ninja", "packaging>=24.2",
    "setuptools>=77.0.3,<81.0.0", "setuptools-scm>=8.0",
    "setuptools-rust>=1.9.0", # <- pip grabs this automatically
    "torch == 2.13.0", # <- ...and this. Which is the problem.
    "wheel", "jinja2",
]

Under normal build isolation , pip creates a clean env, installs that list, and builds. You never see setuptools_rust because you never had to think about it.

But look at the torch pin. Building in isolation means pip installs torch 2.13.0 from PyPI — and the PyPI aarch64 wheels are built for sm_80 and up. No sm_75. Which destroys the entire reason for building from source on a T4G.

So on this box you must build against the DLAMI’s own torch, and that means:

python use_existing_torch.py
pip install -e . --no-build-isolation

--no-build-isolation turns off the automatic install of everything in that requires list. From that moment on, every build dependency is yours to supply by hand — including setuptools_rust, which is why it turns up as a bare ModuleNotFoundError minutes into a build that has nothing visibly to do with Rust.

So the toolchain was always required; isolation was just hiding it. Building this way means you own the dependency list, which is the rest of this walk-through. ⚡

What the DLAMI gives you, and what it doesn’t

The AWS Deep Learning ARM64 AMI ships a runtime , not a build environment. On a fresh box:

Four of those six are on you. Let’s install them.

Step 1 — Rust itself

Standard rustup, nothing aarch64-specific about it:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
. "$HOME/.cargo/env"

stable-aarch64-unknown-linux-gnu installed - rustc 1.97.1 (8bab26f4f 2026-07-14)

Rust is installed now. Great!

Note the triple: stable-aarch64-unknown-linux-gnu. Rust's aarch64 support is a complete non-event , which is a lovely change of pace on this hardware. ⚡

Step 2 — setuptools-rust

python3 -m pip install setuptools_rust

Per the section above: --no-build-isolation means pip won't do this for you. Do it early — the failure lands during metadata generation, minutes into a build, as a bare ModuleNotFoundError: No module named 'setuptools_rust' nowhere near anything that looks like Rust.

⚠️ Install it into the same interpreter you’ll build with. On the DLAMI that’s /opt/pytorch/bin/python3, not the system python3 — they're different, and the one that matters is whichever owns the torch you're building against.

Step 3 — protoc 🔎

This is the one nobody documents:

apt-get install -y protobuf-compiler
protoc --version

libprotoc 3.21.12

Why: vllm-rs depends on the vllm-server crate, vllm-server builds gRPC stubs with tonic/prost, and prost-build shells out to protoc. Skip it and the frontend binary does not get built — see the summary at the end for how loudly that doesn't fail.

The tool parser has no protobuf dependency, which is why it builds either way.

Step 4 — the CUDA toolkit, while you’re here

Not Rust, but the same class of problem, and you need it for vLLM’s kernels:

# NVIDIA's **sbsa** repo — not the x86 one, easy reflex to get wrong on Arm
apt-get install -y cuda-toolkit-13-2

Step 5 — build the Rust artifacts

cd /opt/vllm-src
python tools/build_rust.py --release

⚠️ Do not omit --release. setuptools-rust builds inplace targets in debug by default, and pip install -e . is an inplace build. The difference is not subtle:

100x. The debug artifact is four times the size of every CUDA kernel in vLLM combined.

Timing on a g5g.xlarge (4 vCPU), cold:

real 9m1.746s
user 25m9.199s
sys 1m35.023s

501 crates. Zero warnings. Exit 0. 🟢

Rust’s aarch64 support does not put up a fight here — which is a pleasant contrast with the CUDA side of this box, where SM 7.5 on Graviton needs a custom arch list and a patched kernel.

Step 6 — check what you got

ls -la vllm/vllm-rs vllm/_rust_tool_parser.abi3.so

-rwxr-xr-x 1 root root 50039024 vllm/vllm-rs
-rwxr-xr-x 1 root root 1009080 vllm/_rust_tool_parser.abi3.so

file vllm/vllm-rs

ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV),
dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, not stripped

vllm/vllm-rs --help

Rust frontend and managed-engine CLI for vLLM.

Commands:
  frontend Run the Rust OpenAI frontend as a Python-supervised worker
  serve Launch a managed Python headless engine, then run the Rust OpenAI frontend
  bench Run vLLM benchmarks
  render Run engine-free request rendering and preprocessing

If vllm/vllm-rs isn't there, go back to Step 3.

Step 7 — run it, and mind the entrypoint ⚠️

VLLM_USE_RUST_FRONTEND=1 vllm serve google/gemma-4-E2B-it \
  --dtype float16 \
  --kv-cache-dtype auto \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 8 \
  --tensor-parallel-size 1 \
  --host 0.0.0.0 --port 8000

It must be vllm serve. If you launch the module directly —

# ❌ VLLM_USE_RUST_FRONTEND is IGNORED here
python -m vllm.entrypoints.openai.api_server --model … --host 0.0.0.0 --port 8000

— the variable does nothing. No warning, no Unknown vLLM environment variable line. The server comes up healthy and serves happily on the Python frontend, and a benchmark run against it looks entirely normal.

The flag is read in exactly two places:

vllm/entrypoints/cli/serve.py:62 envs.VLLM_RUST_FRONTEND_PATH if envs.VLLM_USE_RUST_FRONTEND else None
vllm/entrypoints/openai/dp_supervisor.py:261 if envs.VLLM_USE_RUST_FRONTEND and envs.VLLM_RUST_FRONTEND_PATH:

api_server.py never mentions it.

How do I know it’s actually Rust? 🔎

Three checks. Do all three the first time.

1. The server: header:

curl -si localhost:8000/health | grep -i '^server:'

2. The process:

pgrep -af vllm-rs

26588 /opt/vllm-src/vllm/vllm-rs frontend --listen-fd 17
  --input-address ipc:///tmp/f60f3962-d45b-4bcd-9026-c0dc32736028
  --output-address ipc:///tmp/5f75411d-2787-43bb-b4fc-14bf504a1cce
  --engine-start-index 0 --engine-count 1 --data-parallel-size 1

3. The log prefix — (RustFrontend pid=…) instead of (APIServer pid=…):

INFO [utils.py:392] Launching Rust frontend: /opt/vllm-src/vllm/vllm-rs frontend --listen-fd 17 …

So where does Rust actually sit?

In two places, and they’re quite different. One is a separate process; the other is a shared object loaded inside the Python process. Here’s the whole VM:

Two things worth pulling out of that picture:

vllm-rs is not a sidecar you point at a port. The Python side opens the listening socket and passes the file descriptor down. It's a worker the supervisor forks and feeds.
_rust_tool_parser is Rust living inside Python. It's the one that always builds (no protoc needed), which is why a broken install still leaves Rust on the box — just not the Rust you wanted.

And note where the GPU sits relative to all of this: at the bottom, behind everything. That’s the reason the benchmark below comes out the way it does.

Bonus: there’s a Rust benchmark client too

VLLM_USE_RUST_BENCH=1 vllm bench serve …

Same binary, bench subcommand. Requires VLLM_RUST_FRONTEND_PATH to resolve, so it needs the same Step 3 → Step 5 you just did.

Two warnings you should not scroll past 🔴

The server came up healthy. These went by in the startup log anyway.

Gemma 4 defeats the fast tokenizer:

INFO [hf.rs:200] loading tokenizer with fastokens
WARNING [hf.rs:221] failed to load tokenizer with fastokens; falling back to
        HuggingFace tokenizers
        error=tokenizer error: normalizer error: unsupported normalizer type: Replace

fastokens 0.2.1 doesn't implement the Replace normalizer that Gemma 4's tokenizer.json uses, so it falls back to the same HuggingFace tokenizers the Python path uses. Note the fallback is graceful and correct — you just don't get the fast path on this model yet. It's a coverage gap in a young crate, and one normalizer away from closing.

Multimodal isn’t wired up for this model:

WARNING [multimodal.rs:446] multimodal model spec is not registered; disabling
        image/video support model_id="google/gemma-4-E2B-it" model_type="gemma4"

Gemma 4 E2B is a vision model, and gemma4 isn't in the Rust multimodal spec table yet. Text requests behave identically and the endpoint is healthy, so nothing in a normal check reveals it. Also a registration gap rather than a design problem — but check it for your model before you switch, because a healthy endpoint won't tell you.

Is it faster?

On a T4G, no. Output token throughput, same engine config, client on the box against localhost:

Median TTFT tracks just as tightly — 14305 ms against 14311 ms at concurrency 32.

That’s the expected result, and worth saying plainly: decode on this card is bandwidth-bound at a measured 277 GB/s, and the engine saturates at --max-num-seqs 8. A frontend rewrite targets CPU-side per-request overhead. Here that overhead hides behind the GPU, so swapping it can't move a bottleneck-limited number. If you want the Rust frontend to buy you tokens per second on a small GPU, it won't.

One signal does appear, in median inter-token latency at high concurrency:

Mean TPOT barely moves, so this is the middle of the distribution tightening rather than everything speeding up — the shape you’d expect from a frontend scheduling streaming work more evenly once many streams are in flight. Worth knowing if you serve at concurrency; not worth switching for on its own. 📊

If a plain pip install -e . already ran

A from-source vLLM install done without the steps above succeeds, exits 0, and leaves you with a 96 MB debug tool parser and no frontend binary. Four defaults stack up to make that silent:

VLLM_REQUIRE_RUST_FRONTEND=1 turns the second row into a hard build failure, which is what you want on any machine you plan to serve from.

Cheat sheet

# toolchain — you supply these by hand because the sm_75 requirement
# forces --no-build-isolation, which disables pip's automatic build deps
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
. "$HOME/.cargo/env"
/opt/pytorch/bin/python3 -m pip install setuptools_rust # the BUILD interpreter
apt-get install -y protobuf-compiler cuda-toolkit-13-2

# build against the DLAMI's torch, not a PyPI one (PyPI aarch64 has no sm_75)
cd /path/to/vllm
python use_existing_torch.py
TORCH_CUDA_ARCH_LIST=7.5 VLLM_REQUIRE_RUST_FRONTEND=1 \
  pip install -e . --no-build-isolation

# Rust artifacts, release profile (editable installs default to debug: 100x bigger)
VLLM_REQUIRE_RUST_FRONTEND=1 python tools/build_rust.py --release

# confirm
ls -la vllm/vllm-rs && vllm/vllm-rs --help

# run — `vllm serve`, NOT the api_server module
VLLM_USE_RUST_FRONTEND=1 vllm serve <model> --host 0.0.0.0 --port 8000

# verify it's really Rust
curl -si localhost:8000/health | grep -i '^server:' # Rust sends none
pgrep -af vllm-rs

Run on EC2 g5g.xlarge and g5g.4xlarge, us-east-1a, NVIDIA T4G (SM 7.5). vLLM 0.27.2rc1.dev0+g7f7a32cfe, rustc 1.97.1, setuptools-rust 1.13.0, libprotoc 3.21.12, torch 2.12.0+cu132. Benchmarks are one run per cell for Rust and two for Python; treat the TPOT delta as suggestive.

Looker's Native MCP Server with Claude Code

xbill — Fri, 14 Aug 2026 14:00:55 +0000

Looker hosts its own MCP server now. This walks through connecting Claude Code to it, pairing it with the Looker CLI, and being clear-eyed about where the tool set stops.

The binary you no longer need

Until recently, connecting an agent to Looker meant running MCP Toolbox as a local binary. You downloaded 292 MB onto your laptop, taught it your API credentials, launched it as a stdio subprocess, and kept it updated forever. Every developer needed their own copy, and the server ran on your side of the wire.

Looker closed that gap. Every Looker (Google Cloud core) and Looker (original) instance now exposes an MCP endpoint on its own base URL:

https://780eb09e-7dab-4076-9ec1-ecf9d8414630.looker.app/mcp

That is the entire install. Ask the endpoint who it is and the joke lands:

curl -s -X POST "$LOOKER_MCP_URL" \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json, text/event-stream' \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{
        "protocolVersion":"2025-06-18","capabilities":{},
        "clientInfo":{"name":"probe","version":"0"}}}'

{"jsonrpc":"2.0","id":1,"result":{
  "protocolVersion":"2025-06-18",
  "capabilities":{"tools":{"listChanged":false},"prompts":{"listChanged":false}},
  "serverInfo":{"name":"Toolbox","version":"1.4.0+container.release.linux.amd64.d67cfbe"}}}

"name":"Toolbox". It is the same software. Google moved it to the other end of the connection and took over running it. Migrating is not a bet on new technology — it is the server you were already running, minus the operational burden.

Before you start

Three things, and only the first needs someone else.

An admin has to switch the server on. It lives at Admin → Platform → Model Context Protocol. That page also holds the allowlist of which tools agents may call. A tool switched off there does not exist as far as any client is concerned.

You need API3 credentials — Base URL, Client ID, Client Secret. Looker admin panel, Users → (your user) → Edit Keys.

Note the preview limits before you plan around them. Customer-hosted instances are not supported. There are no fine-grained scopes — tool access is one global allowlist, not per-user or per-group. Tool-list changes take about 30 seconds to reach clients, which then have to reconnect.

Setup, start to finish

1. Resolve credentials. The setup script prompts for anything it cannot find, writes .env at mode 600, and derives LOOKER_MCP_URL from your base URL:

source set_env.sh

Source it, do not execute it. The reason matters and catches everyone once — see step 4.

2. Install the Looker CLI. Checksum-verified into the project root:

make cli

looker-cli 0.4.8

3. Smoke-test the credentials before involving an agent. If this fails, nothing downstream will work:

./lk user me

+----+--------------+-------------+----------+
| ID | DISPLAY NAME | IS DISABLED | ROLE IDS |
+----+--------------+-------------+----------+
| 3  | xbill work   | false       | 8        |
|    |              |             | 4        |
|    |              |             | 149      |
+----+--------------+-------------+----------+

4. Register the server with Claude Code. The whole configuration is four lines, and it holds no secrets:

{
  "mcpServers": {
    "looker-managed": {
      "type": "http",
      "url": "${LOOKER_MCP_URL}",
      "headersHelper": "${CLAUDE_PROJECT_DIR:-.}/lk headers"
    }
  }
}

Compare that with what it replaced: a bash -c wrapper that sourced .env, checked three variables, and exec'd a 292 MB binary with --stdio --prebuilt looker,looker-dev. Also gone is startup_timeout_sec — it existed because the binary had to boot and handshake before the client would call it ready. An endpoint that is already running has nothing to wait for.

Two fields are doing real work here.

headersHelper names a command, not a static value. Claude Code runs it on every connection, and again automatically after a 401 or 403, retrying the call once with fresh headers. Looker access tokens live one hour; this makes expiry heal itself. A stale token becomes one 401 you never see.

${LOOKER_MCP_URL} is expanded by Claude Code itself, not by a shell. A stdio server could source .env inside its own wrapper; a remote server has no wrapper. The variable must exist in the environment of the process you launch claude from — which is exactly why step 1 says source, not execute. If your tools are missing, check this first.

5. Verify. Start Claude Code and run /mcp. You should see looker-managed connected with 40 tools. The endpoint will confirm the count itself, without credentials:

curl -s -X POST "$LOOKER_MCP_URL" \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json, text/event-stream' \
  -d '{"jsonrpc":"2.0","id":2,"method":"tools/list"}' | jq '.result.tools | length'

Metadata is open — initialize and tools/list answer unauthenticated, which is what lets a client discover the server before signing in. tools/call refuses without a token. Nothing touches data anonymously.

One note on what this authentication actually is. ./lk headers exchanges your API3 key for a short-lived Looker access token. It works, and it is the right call for evaluation, but every action is attributed to the API3 key's user rather than the human who asked. For a shared instance, register an OAuth client instead — the instance advertises the endpoints at /.well-known/oauth-authorization-server and uses PKCE, so there is no client secret to store.

An example: from question to file

Here is the pattern the whole setup exists to support. Start with a question, not a query.

Which product categories drive the most revenue?

Claude walks the semantic model. get_models returns the instance inventory:

basic_ecomm, intermediate_ecomm, advanced_ecomm       sample_thelook_ecommerce
london_bicycles                                       london_bicycles
gcp_billing_block                                     marketplace_gcp-billing
bq_agent_analytics                                    agent_events

get_explores on advanced_ecomm narrows it to two, and get_measures returns the aggregates that actually exist — order_items.total_sale_price, order_items.count, order_items.average_sale_price. No guessing at column names, and no SQL. The agent is reading the same governed definitions your dashboards use.

Then query runs it:

{"products.category":"Outerwear & Coats","order_items.total_sale_price":971454.48,
 "order_items.count":6711,"order_items.average_sale_price":144.76}
{"products.category":"Jeans","order_items.total_sale_price":924765.39,
 "order_items.count":9428,"order_items.average_sale_price":98.09}
{"products.category":"Sweaters","order_items.total_sale_price":630197.03,
 "order_items.count":8476,"order_items.average_sale_price":74.35}

Outerwear leads on revenue with a third fewer items sold than Jeans, because it carries a 48% higher average price. That is the kind of read worth having an agent for: the rows are in its context, so it can reason about them.

Now hand it to the CLI. The query shape is settled, so freeze it. Same fields, same sorts, in a file the CLI understands:

{
  "model": "advanced_ecomm",
  "view": "advanced_example_ecommerce",
  "fields": [
    "products.category",
    "order_items.total_sale_price",
    "order_items.count",
    "order_items.average_sale_price"
  ],
  "sorts": ["order_items.total_sale_price desc"],
  "limit": "6"
}

./lk query runquery --file q.json --format csv --output category-revenue.csv

Products Category,Order Items Sales,Order Items # of Order Items,Order Items Average Price
Outerwear & Coats,971454.4791278839,6711,144.75554747845126
Jeans,924765.3913908005,9428,98.08712254887557
Sweaters,630197.0301675797,8476,74.35075863232427

Identical numbers, different destination. That is the entire argument for running both.

The view key is the one spelling trap — the CLI calls the explore view, while model, fields, filters and sorts match MCP exactly.

Why bother with the second interface at all? Because the two differ in what happens to the answer, not in what they can reach. Same instance, same API3 key, same REST API underneath.

	Native MCP (`looker-managed`)	CLI (`./lk`)
Coverage	40 tools: query, content, LookML dev, health	The whole API — git, users, roles, schedules, connections, deploys
Results land	In the model's context	On disk
Cost	Every row consumes context	Free of context until you read the file
Good at	Discovery, judgement, structured content creation	Scale, files, determinism, repeatability
Bad at	Bulk output, anything admin-shaped	Deciding what to ask for

Discover and decide over MCP, execute and persist with the CLI. Asking an agent which explore holds revenue by cohort is worth twenty minutes of clicking through the Explore UI. Routing 40,000 rows through its context is not. Once the shape is settled, the CSV job runs forever with no agent in the loop and no tokens burned.

The two also meet at the credential: ./lk headers is what authenticates the MCP server in the first place. The CLI is not a second path bolted on beside MCP — it is what gets MCP connected.

Native vs MCP Toolbox

	MCP Toolbox (local binary)	Native (Looker-hosted)
Install	292 MB download per machine, updated forever	A URL
Transport	stdio subprocess	Streamable HTTP
Version	You pin it — currently v1.8.0	Google pins it — currently 1.4.0
Tools	46, from `--prebuilt looker,looker-dev`	40, from the admin allowlist
Governance	Client-side, per developer, advisory	Admin panel, instance-wide, enforced
Auth	API3 key in the subprocess environment	Bearer token, or OAuth 2.1 + PKCE
Failure mode	"Why won't the server start"	"Why is the endpoint slow"

Two rows deserve more than a table cell.

Version is the real trade. The hosted server reports 1.4.0 while the downloadable binary is on v1.8.0. You stop patching, and you also stop choosing. If you depend on something that landed in Toolbox after 1.4.0, stay where you are for now.

Governance is the real win. With a downloaded Toolbox, the tool set was whatever --prebuilt shipped, and any restriction had to be re-implemented in every client by every developer who installed it. Now a tool switched off in the admin panel does not exist for anyone. That is the difference between a policy and a suggestion.

Migrating is mostly deletion: add the new server, move your git workflow to the CLI, then delete the binary, the download step, the launcher script and the toolbox line in .gitignore. In this repo that removed 292 MB, a checksum routine, a stdio wrapper, and an entire class of "why won't the server start" support question.

What MCP cannot do

The gaps are not random. The server covers Looker as a semantic model and stops at the edge of Looker as an administered system.

Git, entirely. list_git_branches, get_git_branch, create_git_branch, switch_git_branch and delete_git_branch all shipped with the local binary. The managed server exposes none of them. File editing and dev_mode are present, so the agent can write LookML — it just cannot get itself onto a branch to write it safely. The CLI covers that half:

./lk api project create_git_branch --project_id my_project --name my_branch
./lk project checkout my_project my_branch

Commit is missing from both, and that one is not an MCP limitation. The Looker API has no commit endpoint at all. Search the CLI's entire surface and you get deploy verbs:

./lk meta search commit

Found 7 matching commands:
  looker-cli api project create_git_branch     - Checkout New Git Branch
  looker-cli api project deploy_to_production  - Deploy To Production
  looker-cli api project tag_ref               - Tag Ref
  looker-cli api project update_git_branch     - Update Project Git Branch
  ...

So the agent creates the branch, writes the files, validates them and runs the tests — then stops. Committing the workspace is an IDE operation. The loop that looks like it should close (branch → edit → validate → commit → deploy) closes at every step except the second to last, and that step needs a browser.

Plan around it rather than fighting it. The agent does the branch, the edits, the validation and the tests; a human commits in the Looker IDE; the terminal deploys:

./lk project deploy my_project

deploy_to_production never had an MCP equivalent either, so the deploy was always a CLI call. The commit is the only step neither interface can reach.

get_field_value_suggestions. Gone. To find valid filter values, query the field's suggest explore directly — get_dimensions names it in the suggest_explore and suggest_dimension attributes of any suggestable field.

The entire administrative surface. Users, groups, roles, permissions, user attributes, schedules, alerts, connections, themes and sessions have no MCP tools and never did. Every one is a ./lk api call.

Two permission failures look like bugs and are not: health_analyze and health_vacuum need System Activity access and return Access Denied without it, and plan ls returns 404 without see_schedules.

Everything else — discovery, querying, content creation, LookML files, validation, tests, health — is present and works.

Summary

The native install is a smaller thing to own than what it replaced. A 292 MB download, a stdio wrapper and a --prebuilt flag collapsed into a URL and a headersHelper. Google patches the server, the admin panel governs which tools exist, and System Activity logs what the agent did.

What you give up is specific and covered: five git tools and get_field_value_suggestions, all of which the CLI handles. What you should fix before production is the auth shortcut — swap the API3 token helper for a registered OAuth client so actions are attributed to a person rather than a key.

The durable lesson is the division of labor. The MCP server is for discovery and judgement; the CLI is for execution and persistence. Install only one and you will find the seam within a week — most likely on a Tuesday afternoon, halfway through a LookML change, at the commit step.

References

Looker’s Native MCP Server with Claude Code

xbill — Fri, 14 Aug 2026 13:57:39 +0000

Looker hosts its own MCP server now. This walks through connecting Claude Code to it, pairing it with the Looker CLI, and being clear-eyed about where the tool set stops.

The binary you no longer need

Looker closed that gap. Every Looker (Google Cloud core) and Looker (original) instance now exposes an MCP endpoint on its own base URL:

https://780eb09e-7dab-4076-9ec1-ecf9d8414630.looker.app/mcp

That is the entire install. Ask the endpoint who it is and the joke lands:

curl -s -X POST "$LOOKER_MCP_URL" \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json, text/event-stream' \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{
        "protocolVersion":"2025-06-18","capabilities":{},
        "clientInfo":{"name":"probe","version":"0"}}}'

{"jsonrpc":"2.0","id":1,"result":{
  "protocolVersion":"2025-06-18",
  "capabilities":{"tools":{"listChanged":false},"prompts":{"listChanged":false}},
  "serverInfo":{"name":"Toolbox","version":"1.4.0+container.release.linux.amd64.d67cfbe"}}}

"name":"Toolbox". It is the same software. Google moved it to the other end of the connection and took over running it. Migrating is not a bet on new technology — it is the server you were already running, minus the operational burden.

Before you start

Three things, and only the first needs someone else.

You need API3 credentials — Base URL, Client ID, Client Secret. Looker admin panel, Users → (your user) → Edit Keys.

Setup, start to finish

1. Resolve credentials. The setup script prompts for anything it cannot find, and derives LOOKER_MCP_URL from your base URL:

source set_env.sh

Source it, do not execute it. The reason matters and catches everyone once — see step 4.

2. Install the Looker CLI. Checksum-verified into the project root:

make cli

looker-cli 0.4.8

3. Smoke-test the credentials before involving an agent. If this fails, nothing downstream will work:

./lk user me

+----+--------------+-------------+----------+
| ID | DISPLAY NAME | IS DISABLED | ROLE IDS |
+----+--------------+-------------+----------+
| 3 | xbill work | false | 8 |
| | | | 4 |
| | | | 149 |
+----+--------------+-------------+----------+

4. Register the server with Claude Code. The whole configuration is four lines, and it holds no secrets:

{
  "mcpServers": {
    "looker-managed": {
      "type": "http",
      "url": "${LOOKER_MCP_URL}",
      "headersHelper": "${CLAUDE_PROJECT_DIR:-.}/lk headers"
    }
  }
}

Compare that with what it replaced: a bash -c wrapper that sourced .env, checked three variables, and exec’d a 292 MB binary with --stdio --prebuilt looker,looker-dev. Also gone is startup_timeout_sec — it existed because the binary had to boot and handshake before the client would call it ready. An endpoint that is already running has nothing to wait for.

Two fields are doing real work here.

headersHelper names a command, not a static value. Claude Code runs it on every connection, and again automatically after a 401 or 403, retrying the call once with fresh headers. Looker access tokens live one hour; this makes expiry heal itself. A stale token becomes one 401 you never see.

${LOOKER_MCP_URL} is expanded by Claude Code itself , not by a shell. A stdio server could source .env inside its own wrapper; a remote server has no wrapper. The variable must exist in the environment of the process you launch claude from — which is exactly why step 1 says source, not execute. If your tools are missing, check this first.

5. Verify. Start Claude Code and run /mcp. You should see looker-managed connected with 40 tools. The endpoint will confirm the count itself, without credentials:

curl -s -X POST "$LOOKER_MCP_URL" \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json, text/event-stream' \
  -d '{"jsonrpc":"2.0","id":2,"method":"tools/list"}' | jq '.result.tools | length'

40

Metadata is open — initialize and tools/list answer unauthenticated, which is what lets a client discover the server before signing in. tools/call refuses without a token. Nothing touches data anonymously.

One note on what this authentication actually is. ./lk headers exchanges your API3 key for a short-lived Looker access token. It works, and it is the right call for evaluation, but every action is attributed to the API3 key’s user rather than the human who asked. For a shared instance, register an OAuth client instead — the instance advertises the endpoints at /.well-known/oauth-authorization-server and uses PKCE, so there is no client secret to store.

An example: from question to file

Here is the pattern the whole setup exists to support. Start with a question, not a query.

Which product categories drive the most revenue?

Claude walks the semantic model. get_models returns the instance inventory:

basic_ecomm, intermediate_ecomm, advanced_ecomm sample_thelook_ecommerce
london_bicycles london_bicycles
gcp_billing_block marketplace_gcp-billing
bq_agent_analytics agent_events

get_explores on advanced_ecomm narrows it to two, and get_measures returns the aggregates that actually exist — order_items.total_sale_price, order_items.count, order_items.average_sale_price. No guessing at column names, and no SQL. The agent is reading the same governed definitions your dashboards use.

Then query runs it:

{"products.category":"Outerwear & Coats","order_items.total_sale_price":971454.48,
 "order_items.count":6711,"order_items.average_sale_price":144.76}
{"products.category":"Jeans","order_items.total_sale_price":924765.39,
 "order_items.count":9428,"order_items.average_sale_price":98.09}
{"products.category":"Sweaters","order_items.total_sale_price":630197.03,
 "order_items.count":8476,"order_items.average_sale_price":74.35}

Now hand it to the CLI. The query shape is settled, so freeze it. Same fields, same sorts, in a file the CLI understands:

{
  "model": "advanced_ecomm",
  "view": "advanced_example_ecommerce",
  "fields": [
    "products.category",
    "order_items.total_sale_price",
    "order_items.count",
    "order_items.average_sale_price"
  ],
  "sorts": ["order_items.total_sale_price desc"],
  "limit": "6"
}

./lk query runquery --file q.json --format csv --output category-revenue.csv

Products Category,Order Items Sales,Order Items # of Order Items,Order Items Average Price
Outerwear & Coats,971454.4791278839,6711,144.75554747845126
Jeans,924765.3913908005,9428,98.08712254887557
Sweaters,630197.0301675797,8476,74.35075863232427

Identical numbers, different destination. That is the entire argument for running both.

The view key is the one spelling trap — the CLI calls the explore view, while model, fields, filters and sorts match MCP exactly.

Why bother with the second interface at all? Because the two differ in what happens to the answer, not in what they can reach. Same instance, same API3 key, same REST API underneath.

Native MCP (looker-managed) - Coverage: 40 tools — query, content, LookML dev, health - Results land in the model’s context - Every row consumes context - Good at: discovery, judgement, structured content creation - Bad at: bulk output, anything admin-shaped

CLI (./lk) - Coverage: the whole API — git, users, roles, schedules, connections, deploys - Results land on disk - Free of context until you read the file - Good at: scale, files, determinism, repeatability - Bad at: deciding what to ask for

The two also meet at the credential: ./lk headers is what authenticates the MCP server in the first place. The CLI is not a second path bolted on beside MCP — it is what gets MCP connected.

Native vs MCP Toolbox

MCP Toolbox (local binary) — Install: 292 MB download per machine, updated forever — Transport: stdio subprocess — Version: you pin it — currently v1.8.0 — Tools: 46, from --prebuilt looker,looker-dev - Governance: client-side, per developer, advisory - Auth: API3 key in the subprocess environment - Failure mode: “why won’t the server start”

Native (Looker-hosted) — Install: a URL — Transport: streamable HTTP — Version: Google pins it — currently 1.4.0 — Tools: 40, from the admin allowlist — Governance: admin panel, instance-wide, enforced — Auth: bearer token, or OAuth 2.1 + PKCE — Failure mode: “why is the endpoint slow”

Two rows deserve more than a table cell.

Governance is the real win. With a downloaded Toolbox, the tool set was whatever --prebuilt shipped, and any restriction had to be re-implemented in every client by every developer who installed it. Now a tool switched off in the admin panel does not exist for anyone. That is the difference between a policy and a suggestion.

Migrating is mostly deletion: add the new server, move your git workflow to the CLI, then delete the binary, the download step, the launcher script and the toolbox line in .gitignore. In this repo that removed 292 MB, a checksum routine, a stdio wrapper, and an entire class of “why won’t the server start” support question.

What MCP cannot do

The gaps are not random. The server covers Looker as a semantic model and stops at the edge of Looker as an administered system.

Git, entirely. list_git_branches, get_git_branch, create_git_branch, switch_git_branch and delete_git_branch all shipped with the local binary. The managed server exposes none of them. File editing and dev_mode are present, so the agent can write LookML — it just cannot get itself onto a branch to write it safely. The CLI covers that half:

./lk api project create_git_branch --project_id my_project --name my_branch
./lk project checkout my_project my_branch

Commit is missing from both, and that one is not an MCP limitation. The Looker API has no commit endpoint at all. Search the CLI’s entire surface and you get deploy verbs:

./lk meta search commit

Found 7 matching commands:
  looker-cli api project create_git_branch - Checkout New Git Branch
  looker-cli api project deploy_to_production - Deploy To Production
  looker-cli api project tag_ref - Tag Ref
  looker-cli api project update_git_branch - Update Project Git Branch
  ...

Plan around it rather than fighting it. The agent does the branch, the edits, the validation and the tests; a human commits in the Looker IDE; the terminal deploys:

./lk project deploy my_project

deploy_to_production never had an MCP equivalent either, so the deploy was always a CLI call. The commit is the only step neither interface can reach.

get_field_value_suggestions. Gone. To find valid filter values, query the field’s suggest explore directly — get_dimensions names it in the suggest_explore and suggest_dimension attributes of any suggestable field.

Two permission failures look like bugs and are not: health_analyze and health_vacuum need System Activity access and return Access Denied without it, and plan ls returns 404 without see_schedules.

Everything else — discovery, querying, content creation, LookML files, validation, tests, health — is present and works.

Summary

The native install is a smaller thing to own than what it replaced. A 292 MB download, a stdio wrapper and a --prebuilt flag collapsed into a URL and a headersHelper. Google patches the server, the admin panel governs which tools exist, and System Activity logs what the agent did.

What you give up is specific and covered: five git tools and get_field_value_suggestions, all of which the CLI handles. What you should fix before production is the auth shortcut — swap the API3 token helper for a registered OAuth client so actions are attributed to a person rather than a key.

References

Running Gemma 4 on EC2 G5g: Graviton2 AMD with NVIDIA GPU

xbill — Thu, 13 Aug 2026 18:45:41 +0000

A field report on serving Google's Gemma 4 E2B on AWS EC2 **G5g* — a Graviton2 (aarch64)
host with an NVIDIA T4G (Turing, SM 7.5) GPU. Three obstacles: an arch list nobody
publishes for this combination, a version floor that only the newest vLLM clears, and
64 KiB of shared memory that stops the model dead. Plus the seven things I documented
wrong before I had a box.*


Model	`google/gemma-4-E2B-it` (reference bf16 release)
Hardware	AWS EC2 `g5g.4xlarge` — Graviton2 + 1x NVIDIA T4G, compute capability 7.5, 15,360 MiB
Base image	Deep Learning ARM64 AMI OSS Nvidia Driver GPU PyTorch 2.12 (Ubuntu 24.04)
Software	torch 2.12.0+cu132 · CUDA 13.2 · vLLM v0.27.2rc0 built from source for `sm_75`
Result	43.1 tok/s single-stream greedy, 329,579-token KV cache — after one patch to vLLM

G5g is the only instance AWS has ever shipped that puts an NVIDIA GPU behind a Graviton
host. It launched in 2020, it never got a successor, and Graviton is now on its fifth
generation without one.

That matters more than it sounds. The Arm-plus-CUDA world moved on to NVIDIA's own Arm CPU
— Grace, paired with SM 9.0 and 10.0 parts. Turing stayed well supported, on x86. G5g is
the only hardware that is aarch64 and compute capability 7.5, and almost nobody publishes
a build for that combination.

I put a rig on one anyway. The packaging problem was the quick part. Everything after it
— a compiler that was not there, a version floor I did not expect, and 32 KiB of shared
memory — took far longer, because none of it fails where you are looking.

No published build covers aarch64 and SM 7.5 together

Start with the obvious candidate. vllm/vllm-openai:v0.27.1 publishes both platforms under
one tag, and you can read the arch lists straight out of the image config without pulling a
layer:

docker buildx imagetools inspect vllm/vllm-openai:v0.27.1 --format '{{json .Image}}'

linux/amd64   7.5 8.0 8.6 8.9 9.0 10.0 12.0
linux/arm64       8.0 8.7 8.9 9.0 10.0 11.0 12.0

The one architecture this hardware needs is the only entry the two images disagree on. The
arm64 list is Ampere and up, because that is what ships as an Arm-plus-NVIDIA system: A100,
Jetson Orin, GH200, Blackwell. Turing is not on that list and never will be.

Normally a missing target degrades to JIT from embedded PTX. Not here. The Dockerfile says
so, with a comment:

# Do not add +PTX here: vLLM filters torch's top-level PTX flag when it
# converts global gencode flags into per-kernel arch lists.

So it does not run slowly. It fails outright, with no kernel image is available for execution on the device.

The rest of the ecosystem splits the same way. Check before you plan anything:

Artifact	7.5 on arm64	State
`vllm/vllm-openai` arm64	no	Current. Never had it.
`nvcr.io/nvidia/pytorch` arm64	through 24.10	Dropped by 24.12.
`drikster80/vllm-aarch64`	yes	Abandoned Sept 2024. vLLM 0.6.1, far too old for Gemma 4.
PyPI torch aarch64	no	Built for 9.0 / 10.0 / 12.0.
AWS ARM64 GPU DLAMI	yes	Maintained. PyTorch 2.2 through 2.12.

AWS ships the one PyTorch that still has Turing

This is the finding that saves the whole exercise, and I nearly wrote it off. I had assumed
PyTorch's aarch64 CUDA wheels lacked sm_75 and that a from-source PyTorch build was
coming. That is true of the PyPI wheels. It is not true of AWS.

Read on two different DLAMIs, on the box:

torch 2.7.0+cu128    ['sm_75', 'sm_90', 'sm_100', 'sm_120']
torch 2.12.0+cu132   ['sm_75', 'sm_80', 'sm_90', 'sm_100', 'sm_110', 'sm_120']

AWS sells G5g, so AWS keeps Turing in the build — right through PyTorch 2.12 on CUDA 13.2,
an image cut three months ago. PyTorch never needs building. Only vLLM's own kernels do,
and CMake takes the arch list without argument:

-- CUDA target architectures: 7.5
CMake Warning: Pytorch version 2.11.0 expected for CUDA build, saw 2.12.0 instead.

That warning is worth reading twice, and I come back to it below.

The PyTorch DLAMI has no compiler

Two things the DLAMI does not give you, neither of them documented anywhere I could find.

There is no nvcc. The image ships the driver and a torch built against CUDA, not the
toolkit. You need the keyring and cuda-toolkit-13-2 from NVIDIA's sbsa repo — not the
x86 one, which is an easy reflex to get wrong on an Arm box.

And vLLM now wants Rust. Its vllm-rs frontend needs setuptools_rust plus a toolchain,
and the failure is a bare ModuleNotFoundError: No module named 'setuptools_rust' thrown
from metadata generation, several minutes in.

The newest vLLM was the only one that worked

No vLLM tag pins torch 2.12. They go 2.11, then jump to 2.13. I reasoned that building older
code against a newer runtime was the safer direction, took v0.26.0, and spent an hour being
wrong about it.

It builds fine. It then dies on model load:

transformers.integrations.heterogeneity.configuration_utils.AmbiguousGlobalPerLayerAttributeError:
'head_dim' is a per-layer attribute and may vary across layers.

Gemma 4's head_dim is not one number, and current transformers refuses to hand out a
global value for it. vLLM's config converter was still doing a flat
getattr(config, "head_dim", 0). The per_layer_config handling that copes with it landed
in v0.27.2rc0 — not v0.27.1, which I also checked. The newest tag was the only one that
worked.

If you take one process lesson from this: reach for the latest release first, and make the
constraint say out loud what stopped you when you fall back.

Gemma 4's attention heads are not one size

With the build working the server still would not start, and this failure has nothing to do
with Arm or packaging. It is this model against this chip.

Gemma4 model has heterogeneous head dimensions
{'sliding_attention': 256, 'full_attention': 512}.
FA4 not available, forcing TRITON_ATTN backend.

Read that as a chain, because every link is load-bearing:

Gemma 4's sliding layers are 256 wide. Its global layers are 512.
Only FA4 or Triton support heterogeneous head dims at all.
FA4 is not available, so vLLM forces TRITON_ATTN.
That choice is not yours to make. VLLM_ATTENTION_BACKEND is not a recognised variable in v0.27 — it logs Unknown vLLM environment variable detected and carries on. I set it twice before I read the warning.
Triton's unified attention kernel at head_size=512 wants about 96 KiB of shared memory per block.

64 KiB is the whole problem

Turing's shared memory is two numbers, and both are real. The default static limit per block
is 48 KiB — that is what torch.cuda.get_device_properties().shared_memory_per_block reports,
49,152 bytes. A kernel that needs more has to opt in through the dynamic shared-memory
attribute, and even then it tops out at 64 KiB. Ampere and later have 164 KiB and up.

Triton opts in, so it is measuring against the 64 KiB ceiling. It still does not fit:

triton.runtime.errors.OutOfResources: out of resource: shared memory,
Required: 98304, Hardware limit: 65536

Refused outright. Not slow, not degraded — the kernel will not launch, and it takes the
engine down during CUDA graph capture, which is late enough that you have already watched
the weights load and the KV cache get sized.

The fix is small. Shrink the KV tile until the query block and the K/V tiles fit inside the
budget, and drop the software pipeline to one stage. Gate it on pre-Ampere so it is a no-op
on every other card:

if current_platform.get_device_capability()[0] < 8:
    _smem_budget = 60000
    _esz = q.element_size()
    def _fits(t): return (BLOCK_M + 2 * t) * head_size * _esz <= _smem_budget
    while TILE_SIZE_PREFILL > 16 and not _fits(TILE_SIZE_PREFILL): TILE_SIZE_PREFILL //= 2
    while TILE_SIZE_DECODE  > 16 and not _fits(TILE_SIZE_DECODE):  TILE_SIZE_DECODE  //= 2
    launch_num_stages = 1

With that in vllm/v1/attention/ops/triton_unified_attention.py, graphs capture, the engine
comes up in 76 seconds, and the model serves. This is not upstream. It lives on my
instance and has to be reapplied on any vLLM upgrade, which makes it the obvious thing to
send back.

Most of the build is kernels that can never load

67 minutes on a g5g.4xlarge at MAX_JOBS=12, and the majority of it is FlashAttention.
vLLM compiles FA2 and FA3 regardless of TORCH_CUDA_ARCH_LIST — I watched it grind
through hundreds of sm90 Hopper instantiations on a build targeting 7.5 only. FA2 needs
sm80, FA3 needs sm90. Neither can ever load on this card.

Constraining VLLM_FA_CMAKE_GPU_ARCHES should cut that dramatically. I did not try it,
because by the time I understood what I was looking at the build was 45 minutes in and
interrupting it would have cost more than finishing.

What I got wrong before I had hardware

I wrote the rig's documentation before provisioning anything. Seven claims in it were wrong,
and every correction came off the machine rather than out of an argument. This is the part I
would keep if I kept nothing else.

What I wrote	What the box said
PyTorch aarch64 lacks `sm_75`	AWS DLAMI has it, on both versions I checked
bfloat16 is a hard failure here	Torch upconverts; vLLM logs `Casting torch.bfloat16 to torch.float16` and proceeds
The backend is XFORMERS	`TRITON_ATTN`, forced, not selectable
`VLLM_ATTENTION_BACKEND` picks it	Not a recognised variable. I had shipped dead config.
w4a16 needs sm80+ Marlin	The build compiled `sm75_kernel_float16_u4b8_float16.cu.o`
The GPU has 16 GB	15,360 MiB
`/v1/completions` returns an empty body	It returns `': ok: ok: ok: ok'` — garbage, not silence

That last one has teeth. If you health-check by testing for an empty response, this endpoint
passes while producing nonsense. Use /v1/chat/completions and read the text.

One claim is still standing only because I never tested it: whether g5g.xlarge's 8 GiB of
host RAM can stage 9.5 GiB of weights. Safetensors loading is mmap-backed, so I suspect it
can. It is labelled untested rather than stated as fact, which is where it should have been
all along.

What it does once it runs

content: 'Site Reliability Engineering (SRE) is a discipline that applies
          software engineering principles to infrastructure and operations
          problems to create highly reliable, scalable, and efficient systems.'
finish_reason: stop      usage: 19 prompt / 32 completion / 51 total

Measure	Value
Throughput, single stream greedy	42.9 tok/s @ 64, 43.1 @ 256
KV cache	2.95 GiB, 329,579 tokens
Concurrency at 16k context	20.12x
GPU memory while serving	13,501 / 15,360 MiB
Engine init	76.4 s, graph capture 17 s
Memory bandwidth, measured	277.0 GB/s read · 234.3 GB/s copy (320.1 theoretical)

Before reading too much into 43 tok/s, note what the memory does. The T4G has GDDR6, not
HBM — 256-bit bus at 5,001 MHz, so 320 GB/s theoretical. I measured 277 GB/s on a
streaming read (87% of peak) and 234 GB/s on a read-modify-write. Decode is bandwidth-bound,
so 277 is the real ceiling. For scale, a TPU v5e is about 859 GB/s normalized and a v6e about
1,638 — this part has roughly a third of one and a sixth of the other. It is a bandwidth-limited
card behaving like a bandwidth-limited card.

Single run, single stream, no repeats and no variance figure. One sample per cell, and taken
with the clamped tiles, so it is a floor rather than a characterisation. My Inferentia port
measured about 44 tok/s for E2B on one core, which is the same neighbourhood — but that is a
different harness on different silicon and I would not put the two in one table.

Troubleshooting quick reference

Symptom	Cause
`no kernel image is available`	Stock arm64 image. No 7.5, no PTX. Build from source.
`OutOfResources: shared memory`	Turing's 64 KiB against a 512-wide head. Clamp the tiles.
`AmbiguousGlobalPerLayerAttributeError`	vLLM older than v0.27.2rc0.
`No module named 'setuptools_rust'`	Missing Rust toolchain for `vllm-rs`.
`nvcc: not found`	PyTorch DLAMI has no toolkit. Install `cuda-toolkit-13-2` (sbsa).
`Unknown vLLM environment variable`	You set `VLLM_ATTENTION_BACKEND`. It does nothing.
Healthy endpoint, nonsense output	You checked `/v1/completions`. Use chat completions.

The short version

Take the AWS ARM64 GPU PyTorch DLAMI — it is the only maintained aarch64 stack that still
carries sm_75. Add cuda-toolkit-13-2 from the sbsa repo and a Rust toolchain, because the
image ships neither. Build vLLM v0.27.2rc0 or newer from source with
TORCH_CUDA_ARCH_LIST=7.5 and use_existing_torch.py, and patch the Triton attention kernel
to fit Turing's shared memory before you try to start it. Serve with --dtype float16 and
--kv-cache-dtype auto.

Nothing here failed loudly, and nothing failed where I was looking. The packaging gap I built
the rig around was already solved by AWS; the thing that actually stopped me was 32 KiB of
shared memory and a model whose global attention heads are twice as wide as its sliding ones.
Hardware this far off the mainstream will keep producing that shape of surprise — the fix is
not to reason harder about it, but to get to a box sooner and let it tell you.

Measured on EC2 g5g.4xlarge spot, us-east-1a. NVIDIA T4G, compute capability 7.5,
15,360 MiB, driver 595.71.05. Deep Learning ARM64 AMI OSS Nvidia Driver GPU PyTorch 2.12
(Ubuntu 24.04). torch 2.12.0+cu132, CUDA 13.2. vLLM 0.27.2rc1.dev0+g7f7a32cfe built from
v0.27.2rc0.

Running Gemma 4 on EC2 G5g: Graviton2 AMD with NVIDIA GPU

xbill — Thu, 13 Aug 2026 18:42:09 +0000


Model	`google/gemma-4-E2B-it` (reference bf16 release)
Hardware	AWS EC2 `g5g.4xlarge` — Graviton2 + 1x NVIDIA T4G, compute capability 7.5, 15,360 MiB
Base image	Deep Learning ARM64 AMI OSS Nvidia Driver GPU PyTorch 2.12 (Ubuntu 24.04)
Software	torch 2.12.0+cu132 · CUDA 13.2 · vLLM v0.27.2rc0 built from source for `sm_75`
Result	43.1 tok/s single-stream greedy, 329,579-token KV cache — after one patch to vLLM

G5g is the only instance AWS has ever shipped that puts an NVIDIA GPU behind a Graviton
host. It launched in 2020, it never got a successor, and Graviton is now on its fifth
generation without one.

No published build covers aarch64 and SM 7.5 together

Start with the obvious candidate. vllm/vllm-openai:v0.27.1 publishes both platforms under
one tag, and you can read the arch lists straight out of the image config without pulling a
layer:

docker buildx imagetools inspect vllm/vllm-openai:v0.27.1 --format '{{json .Image}}'

linux/amd64   7.5 8.0 8.6 8.9 9.0 10.0 12.0
linux/arm64       8.0 8.7 8.9 9.0 10.0 11.0 12.0

Normally a missing target degrades to JIT from embedded PTX. Not here. The Dockerfile says
so, with a comment:

# Do not add +PTX here: vLLM filters torch's top-level PTX flag when it
# converts global gencode flags into per-kernel arch lists.

So it does not run slowly. It fails outright, with no kernel image is available for execution on the device.

The rest of the ecosystem splits the same way. Check before you plan anything:

Artifact	7.5 on arm64	State
`vllm/vllm-openai` arm64	no	Current. Never had it.
`nvcr.io/nvidia/pytorch` arm64	through 24.10	Dropped by 24.12.
`drikster80/vllm-aarch64`	yes	Abandoned Sept 2024. vLLM 0.6.1, far too old for Gemma 4.
PyPI torch aarch64	no	Built for 9.0 / 10.0 / 12.0.
AWS ARM64 GPU DLAMI	yes	Maintained. PyTorch 2.2 through 2.12.

AWS ships the one PyTorch that still has Turing

Read on two different DLAMIs, on the box:

torch 2.7.0+cu128    ['sm_75', 'sm_90', 'sm_100', 'sm_120']
torch 2.12.0+cu132   ['sm_75', 'sm_80', 'sm_90', 'sm_100', 'sm_110', 'sm_120']

-- CUDA target architectures: 7.5
CMake Warning: Pytorch version 2.11.0 expected for CUDA build, saw 2.12.0 instead.

That warning is worth reading twice, and I come back to it below.

The PyTorch DLAMI has no compiler

Two things the DLAMI does not give you, neither of them documented anywhere I could find.

The newest vLLM was the only one that worked

It builds fine. It then dies on model load:

transformers.integrations.heterogeneity.configuration_utils.AmbiguousGlobalPerLayerAttributeError:
'head_dim' is a per-layer attribute and may vary across layers.

If you take one process lesson from this: reach for the latest release first, and make the
constraint say out loud what stopped you when you fall back.

Gemma 4's attention heads are not one size

With the build working the server still would not start, and this failure has nothing to do
with Arm or packaging. It is this model against this chip.

Gemma4 model has heterogeneous head dimensions
{'sliding_attention': 256, 'full_attention': 512}.
FA4 not available, forcing TRITON_ATTN backend.

Read that as a chain, because every link is load-bearing:

Gemma 4's sliding layers are 256 wide. Its global layers are 512.
Only FA4 or Triton support heterogeneous head dims at all.
FA4 is not available, so vLLM forces TRITON_ATTN.
That choice is not yours to make. VLLM_ATTENTION_BACKEND is not a recognised variable in v0.27 — it logs Unknown vLLM environment variable detected and carries on. I set it twice before I read the warning.
Triton's unified attention kernel at head_size=512 wants about 96 KiB of shared memory per block.

64 KiB is the whole problem

Triton opts in, so it is measuring against the 64 KiB ceiling. It still does not fit:

triton.runtime.errors.OutOfResources: out of resource: shared memory,
Required: 98304, Hardware limit: 65536

if current_platform.get_device_capability()[0] < 8:
    _smem_budget = 60000
    _esz = q.element_size()
    def _fits(t): return (BLOCK_M + 2 * t) * head_size * _esz <= _smem_budget
    while TILE_SIZE_PREFILL > 16 and not _fits(TILE_SIZE_PREFILL): TILE_SIZE_PREFILL //= 2
    while TILE_SIZE_DECODE  > 16 and not _fits(TILE_SIZE_DECODE):  TILE_SIZE_DECODE  //= 2
    launch_num_stages = 1

Most of the build is kernels that can never load

What I got wrong before I had hardware

What I wrote	What the box said
PyTorch aarch64 lacks `sm_75`	AWS DLAMI has it, on both versions I checked
bfloat16 is a hard failure here	Torch upconverts; vLLM logs `Casting torch.bfloat16 to torch.float16` and proceeds
The backend is XFORMERS	`TRITON_ATTN`, forced, not selectable
`VLLM_ATTENTION_BACKEND` picks it	Not a recognised variable. I had shipped dead config.
w4a16 needs sm80+ Marlin	The build compiled `sm75_kernel_float16_u4b8_float16.cu.o`
The GPU has 16 GB	15,360 MiB
`/v1/completions` returns an empty body	It returns `': ok: ok: ok: ok'` — garbage, not silence

That last one has teeth. If you health-check by testing for an empty response, this endpoint
passes while producing nonsense. Use /v1/chat/completions and read the text.

What it does once it runs

content: 'Site Reliability Engineering (SRE) is a discipline that applies
          software engineering principles to infrastructure and operations
          problems to create highly reliable, scalable, and efficient systems.'
finish_reason: stop      usage: 19 prompt / 32 completion / 51 total

Measure	Value
Throughput, single stream greedy	42.9 tok/s @ 64, 43.1 @ 256
KV cache	2.95 GiB, 329,579 tokens
Concurrency at 16k context	20.12x
GPU memory while serving	13,501 / 15,360 MiB
Engine init	76.4 s, graph capture 17 s
Memory bandwidth, measured	277.0 GB/s read · 234.3 GB/s copy (320.1 theoretical)

Troubleshooting quick reference

Symptom	Cause
`no kernel image is available`	Stock arm64 image. No 7.5, no PTX. Build from source.
`OutOfResources: shared memory`	Turing's 64 KiB against a 512-wide head. Clamp the tiles.
`AmbiguousGlobalPerLayerAttributeError`	vLLM older than v0.27.2rc0.
`No module named 'setuptools_rust'`	Missing Rust toolchain for `vllm-rs`.
`nvcc: not found`	PyTorch DLAMI has no toolkit. Install `cuda-toolkit-13-2` (sbsa).
`Unknown vLLM environment variable`	You set `VLLM_ATTENTION_BACKEND`. It does nothing.
Healthy endpoint, nonsense output	You checked `/v1/completions`. Use chat completions.

The short version

Do I Still Need a Monkey Patch for Gemini Live?

xbill — Thu, 13 Aug 2026 01:41:40 +0000

No. And deleting 187 lines of it was the single biggest benefit of moving to ADK 2.x — but it was not the only one, and it was not the last thing that needed fixing.

What Does This Agent Do?

The project is a biometric security scanner, built to exercise the parts of the Gemini Live API that a text chatbot never touches. A browser captures webcam and microphone, streams both to a FastAPI backend over a single WebSocket, and the backend forwards them to Gemini 3.1 Flash Live through the Agent Development Kit. The model watches the video feed, counts the fingers being held up, and calls a tool.

Three tools are registered:

report_digit(count) — the detected finger count, which drives the UI
trigger_system_error() — fired on an offensive gesture, which terminates the session
trigger_heavy_metal_mode() — fired on the "Devil's Horns", a secret override

The transport is deliberately plain. Binary WebSocket frames carry a 1-byte type prefix — 1 for audio, 2 for JPEG — with 16 kHz PCM going up and 24 kHz PCM coming back, played through an AudioWorklet so the main thread stays free. Everything runs locally with make run, or on Cloud Run behind make deploy.

git clone https://github.com/xbill9/way-back-home
cd way-back-home/level_3_new

Two versions live side by side in that repo: level_3, the original design, and level_3_new, the current one. Most of this article is the diff between them.

What Level 3 Needed to Work

The original build ran on google-adk 1.27.2, and it worked — but only because a file named patch_adk.py sat next to it, 187 lines long, applied at import time before anything else could run.

The problem it solved was real. Gemini 3.1 deprecated media_chunks, the field a 1.x ADK used to send realtime media. A 1.x ADK talking to a 3.1 Live model would send the deprecated shape and get nothing useful back. The patch monkey-patched three separate call sites to translate:

# level_3_gemini/backend/app/patch_adk.py
if hasattr(rt_input, "media_chunks") and rt_input.media_chunks:
    logger.info("[PATCH] Unrolling 'media_chunks' from realtime_input.")
    for chunk in rt_input.media_chunks:
        ...
        await self.send_realtime_input(audio=chunk)
        ...
        await self.send_realtime_input(video=chunk)

The three targets were live.AsyncSession.send_realtime_input, which unrolled media_chunks into the new typed keywords; GeminiLlmConnection.send_realtime, which routed each blob to audio=, video= or text= by mime type; and AudioCacheManager.cache_audio, which was guarded against a NoneType blob that would otherwise raise.

It worked, and it was a liability. Monkey patching a framework means every upgrade is a gamble — the patch either becomes redundant, becomes wrong, or silently stops applying because the method it wraps was renamed. The closing recommendation in the original write-up was to delete it the moment the ADK supported the model natively.

What ADK 2.x Handles Natively

That moment arrived with google-adk 2.6.3, and patch_adk.py was deleted outright. The framework now does the routing itself, detecting the model generation and dispatching on it:

# google/adk/models/gemini_llm_connection.py, send_realtime()
if isinstance(input, types.Blob):
  if self._is_gemini_3_x_live or self._is_gemini_3_5_live_translate:
    if input.mime_type and input.mime_type.startswith('audio/'):
      await self._gemini_session.send_realtime_input(audio=input)
    elif input.mime_type and input.mime_type.startswith('image/'):
      await self._gemini_session.send_realtime_input(video=input)
    else:
      logger.warning(
          'Blob not sent. Unknown or empty mime type for'
          ' send_realtime_input: %s',
          input.mime_type,
      )
  else:
    await self._gemini_session.send_realtime_input(media=input)

Note the third branch. Audio and image mime types are dispatched explicitly, and anything else is dropped with a warning rather than guessed at — so a blob sent with a missing or unexpected mime type goes nowhere, and the only evidence is a log line.

Text is handled the same way. A single-part text Content is routed to send_realtime_input(text=...) for 3.x models rather than going out as client content, which matches the Live API's own guidance that send_client_content is only for seeding history.

That is the whole first patch target and the whole second one, upstream, maintained, and tested by someone else. The third — the NoneType guard on cache_audio — was not carried over, because upstream still calls len(audio_blob.data) unguarded. No path in this application produces a blob with data=None, so it stays deleted rather than being reintroduced as a precaution.

The One Thing That Broke

The patch was hiding a bug in the calling code, and deleting it exposed the bug rather than causing it.

LiveRequestQueue.send_realtime() accepts types.Blob and nothing else. The old patch had used model_construct internally, which skips Pydantic validation, so passing a bare string worked by accident. Without the patch it raises a ValidationError.

The call site that matters is the keepalive. This project sends a text stimulus every ten seconds when the client goes quiet, and under 1.x that stimulus was a string handed straight to send_realtime(). Text has to go through send_content() instead:

def send_text_stimulus(live_request_queue: LiveRequestQueue, text: str) -> None:
    live_request_queue.send_content(
        types.Content(role="user", parts=[types.Part(text=text)])
    )

This is the removal most likely to take an agent off the air quietly. It does not fail at startup. It fails the first time the keepalive fires, ten seconds into a session that otherwise looks healthy. Anyone migrating a Live agent off 1.x should check that call site before touching anything else.

What Else Was Updated

The migration was the headline, but it was not the end of the work. Reading the Live API documentation with the source open beside it turned up several things that had been wrong the whole time, none of which any build, test or lint run had ever objected to.

Video was running at twice the documented maximum. The capabilities guide is specific:

Video frames are sent as individual images (e.g., JPEG or PNG) at a specific frame rate (max 1 frame per second).

The project ran at 2 FPS and permitted up to 5 through an environment variable. Nothing rejects the surplus frames, which is why it went unnoticed — but they are billed, and they consume the session budget twice as fast. VIDEO_FPS now defaults to 1.0 and is hard-clamped there, so VIDEO_FPS=3 yields 1.0 rather than being honoured. A documented limit that is not enforced is how the 2 FPS crept in to begin with.

Audio-plus-video sessions cap at two minutes. From the session management guide:

audio-only sessions are limited to 15 minutes, and audio-video sessions are limited to 2 minutes

Context window compression removes the cap entirely. RunConfig.context_window_compression defaults to None, so it has to be asked for:

context_window_compression=types.ContextWindowCompressionConfig(
    sliding_window=types.SlidingWindow(),
),

This application streams both continuously, so it had been on the two-minute clock since the first version. Short test sessions never reached it. A demo where someone works through five gestures does.

Interruptions were documented and unhandled. When a user talks over the model, the model stops generating — but the audio it already sent is sitting in the client's ring buffer and keeps playing. The Live API guidance is to stop playback and clear the queue on interruption. ADK surfaces interrupted on the event, and because the backend forwards whole events as JSON, the flag was already arriving in the browser with nothing reading it. The clearing machinery already existed too. Three lines connected them.

The Log Line That Never Ran

Both input and output audio transcription were enabled in RunConfig from the very first version of this project. Neither ever produced a line of output.

input_transcription = getattr(event, "input_audio_transcription", None)
if input_transcription and input_transcription.final_transcript:
    logger.info(f"USER TRANSCRIPT: {input_transcription.final_transcript}")

Two mistakes are stacked here. input_audio_transcription is the RunConfig field that enables transcription — it is not the field on the event that transcription produces. And final_transcript is not a member of types.Transcription at all; the fields are text, finished, language_code, speaker_label and words.

Either mistake alone raises AttributeError and gets fixed in minutes. Together, behind the default on getattr, they produce silence. The condition evaluates to None and ..., which is falsy, forever.

The correct field names:

input_transcription = getattr(event, "input_transcription", None)
if input_transcription and input_transcription.finished:
    logger.info(f"USER TRANSCRIPT: {input_transcription.text}")

Gating on finished is deliberate. ADK emits partial transcription events with finished=False and one accumulated event with finished=True, so this produces one clean line per turn instead of one per fragment. The run_live() docstring is the reference: partial and non-partial events are both yielded to the caller, but only non-partial ones are saved to the session.

The fix produces this on connect, which is the exact opening line the agent instruction specifies:

INFO - GEMINI TRANSCRIPT: Scanner Online.

Video That Stops When You Look Away

The original design captured frames on a timer:

intervalRef.current = setInterval(() => { /* capture, send */ }, 500);

The rewrite replaced that with requestAnimationFrame and a manual elapsed-time check. On paper it is the better primitive — frame-aligned, idle when the compositor has nothing to do, and paired with toBlob instead of toDataURL it keeps JPEG encoding off the main thread.

In a backgrounded tab, requestAnimationFrame is throttled to zero.

The microphone is not. It runs in an AudioWorklet on the audio thread, which browsers keep alive so capture and playback survive a tab switch. The result is an asymmetric, silent failure: switch tabs and video stops completely while audio streams on. The WebSocket stays open, the session stays billed, and finger detection — the entire point of the application — stops with no error on either side. A 65-second session logged 8,050 audio packets and zero video frames.

Timers are throttled in background tabs as well, but to roughly one second rather than to zero, so the fix is a self-rescheduling timeout:

const captureFrame = () => {
    if (ws.current?.readyState === WebSocket.OPEN) { /* capture, send */ }
    if (intervalRef.current !== null) {
        intervalRef.current = setTimeout(captureFrame, frameIntervalRef.current);
    }
};

Degrading to roughly 1 FPS beats stopping. Re-reading the interval on each tick also keeps the server's config frame authoritative, which the rAF version did and a plain setInterval would not.

Code With No Caller

Four things were deleted outright, for ninety lines removed and one added.

The base64 JSON media path decoded type: "audio" and type: "image" payloads out of JSON. It was not speculative — it was the original design's entire wire protocol, orphaned when media moved to binary frames with a type prefix. Its two tests went with it; they were pinning an implementation, not a contract anyone relied on.

The proactivity and affective_dialog query parameters were declared on the WebSocket endpoint, documented in the docstring, and read by nothing. In the original design they were real, feeding a conditional RunConfig. Gemini 3.1 Flash Live then shipped without support for either, the config was removed, and the parameters outlived it. The Live API reference lists both under limitations — "Proactive audio — Not yet supported in Gemini 3.1 Flash Live" and the same line for affective dialogue — each followed by an instruction to remove any configuration for the feature.

A function-response scan walked server_content.model_turn.parts looking for function_response. model_turn carries model output; a functionResponse is something a client sends. The list was structurally always empty.

A second notification channel, lastMessage, was set on every match, system error and heavy-metal trigger, exported from the frontend hook, and read by no component. The callbacks drive the UI.

Rules for Staying on ADK 2.x

ADK 2.0 moved agents onto a graph engine, and the new constraints fail quietly rather than loudly. Four are worth knowing before writing anything:

Use the plain Agent. Custom BaseNode subclasses and _run_async_impl() or generate_content() overrides are silently bypassed. No error, no warning, no effect.
Never hand-append events to the session. It circumvents the graph engine and breaks determinism; run_live() owns the session.
Watch broad exception handlers. The framework catches exceptions for retries and human-in-the-loop pausing. A broad except inside a node masks that, and catching BaseException breaks pausing outright.
run_live(session=...) is deprecated. Pass user_id and session_id.

This project satisfies all four, which is why the upgrade was uneventful: a plain Agent, InMemorySessionService, no hand-built events, and its broad handlers sitting in the transport layer rather than inside the graph.

One 2.x addition is worth adopting. Runner gained auto_create_session, and run_live() already calls its internal get-or-create helper — without the flag a missing session is a ValueError, which is why the code hand-rolled get-then-create:

runner = Runner(
    app_name=APP_NAME,
    agent=root_agent,
    session_service=session_service,
    auto_create_session=True,
)

One is worth skipping. Runner(app=App(...)) is now described as the recommended construction, but Runner(agent=..., app_name=...) is still supported and gets wrapped into an App internally, with no deprecation warning. "Recommended" and "required" are different words.

Whether any of this is still true after the next ADK release is checkable rather than a matter of reading changelogs:

python -W error::DeprecationWarning -m pytest -q

Zero ADK warnings today. That command is the check after any bump.

Two Traps Worth Knowing

ADK's own docstring names a field that does not exist. run_live() refers to RunConfig.save_live_model_audio_to_session. In 2.6.3 the real fields are save_live_blob and save_live_audio. Where documentation and installed source disagree, the source wins — it is the thing that runs.

A green test suite proves less than it looks like. The suites stub run_live(), which is what makes them hermetic: no API key, no network, no charge. It also means session creation, resumption and teardown are never exercised, and the suite passes whether or not they work. The two changes most capable of breaking every session — auto_create_session and context_window_compression — were verified against the real API with a throwaway WebSocket client instead.

What Did Not Change

The wire protocol, the agent instruction, the tool definitions and the deployment path all came through the migration untouched: binary frames with a 1-byte prefix, 16 kHz PCM in and 24 kHz out, the three server-side tools, Secret Manager for the API key, and the model-id fallback to gemini-2.5-flash under adk run, since the Live preview model still 404s on generateContent.

A major-version framework upgrade that touches only the compatibility layer is the outcome to want. It is also the argument for keeping shims isolated in one file with a name that says what it is.

So What Really Changed?

patch_adk.py is deleted — 187 lines and three monkey-patched call sites, replaced by native routing in ADK 2.6.3.
Text goes through send_content(), because send_realtime() takes types.Blob only. The keepalive is the call site that catches people out.
Video runs at 1 FPS with a hard ceiling, matching the documented maximum instead of doubling it.
Context compression is enabled, lifting a two-minute cap on audio-plus-video sessions.
Transcript logging works, after reading a RunConfig field name off the event since the original design.
The capture loop is a timer again, because requestAnimationFrame stops dead in a background tab while the microphone does not.
Barge-in clears the playback queue, connecting a flag that was already arriving to machinery that already existed.
Ninety lines with no caller are gone, along with the two tests that covered them.

Summary

The Agent Development Kit 2.x release removed the need for a compatibility patch that had been carried since the original build, and deleting it was almost entirely subtractive — the pleasant kind of upgrade. The one removal that bites is send_realtime() rejecting anything that is not a types.Blob, which surfaces as a keepalive failure ten seconds into an otherwise healthy session rather than as a crash at startup.

The wider lesson came after the migration. A framework upgrade tells you what stopped compiling; it says nothing about what still compiles and is wrong. A dead branch compiles. A no-op log line runs. A throttled callback returns cleanly. An undocumented frame rate is accepted by the server. Every one of these survived a migration, a test suite, a lint config and a demo that visibly worked, and each was found by reading the documentation next to the code rather than by running anything.

For anyone running a Live agent of their own, three checks cost about an hour between them: confirm the transcript handler has ever produced output, confirm video keeps flowing with the tab hidden, and read the session-limit page next to your RunConfig.

DEV Community: xbill

Build an MCP server in Rust with rmcp: a walk-through 🦀

Why Rust for this?

How does this all fit together?

What is MCP, in one paragraph?

Step 1 — Scaffold

🔎 Tip: where the canonical examples live

Step 2 — The server struct

Step 3 — Describe your inputs as types

Step 4 — Write the tools

Step 5 — Implement ServerHandler

Step 6 — main

Step 7 — Test the protocol by hand

Step 8 — Register it

Tests worth writing

🐍 vs 🦀 — the scoreboard

So, worth it?

Cheat sheet

Build an MCP server in Rust with rmcp: a walk-through 🦀

Why Rust for this?

How does this all fit together?

What is MCP, in one paragraph?

Step 1 — Scaffold

🔎 Tip: where the canonical examples live

Step 2 — The server struct

Step 3 — Describe your inputs as types

Step 4 — Write the tools

Step 5 — Implement ServerHandler

Step 6 — main

Step 7 — Test the protocol by hand

Step 8 — Register it

Tests worth writing

🐍 vs 🦀 — the scoreboard

So, worth it?

Cheat sheet

Shipping a vision-model verdict on Bedrock and Lightsail

TL;DR

1. The shape: one route, one call

2. Force the verdict into a schema

3. Build the whole frontend with a mock

4. Scope the IAM user, and expect a region you never asked for

5. Deploy: one container service

6. Measure it before you believe it

7. What it costs, and how to turn it off

8. Reproduction

Links

I built a security scanner that checks if you are a dog

What I Built

Demo

Code

xbill9 / devto-dog

A live-video security scanner that checks whether you are holding up a dog, and barks if you are. DEV Weekend Challenge: Dog Days Edition.

Dog or Not

Credit where it is due

How I Built It

So where does the model actually sit?

Every green checkmark was, at some point, green over something broken

The same bug, three times, in three places

The model that narrates calling a tool and then doesn't

One fixture in twenty was poisoned

And the harness was scoring obedience as failure

Is it actually right, though?

The numbers

What the loop got wrong

Prize Categories

Serving Gemma4 with Rust on vLLM 🦀

Wait, vLLM has Rust in it?

Rust is a build requirement now

Then why doesn't pip install vllm need this?

What the DLAMI gives you, and what it doesn't

Step 1 — Rust itself

Step 2 — setuptools-rust

Step 3 — protoc 🔎

Step 4 — the CUDA toolkit, while you're here

Step 5 — build the Rust artifacts

Step 6 — check what you got

Step 7 — run it, and mind the entrypoint ⚠️

How do I know it's actually Rust? 🔎

So where does Rust actually sit?

Bonus: there's a Rust benchmark client too

Then why doesn't `pip install vllm` need this?

If a plain `pip install -e .` already ran

Then why doesn't `pip install vllm` need this?

If a plain `pip install -e .` already ran