DEV Community: Paw from Oz

Stoke: A kill switch for runaway AI agents

Paw from Oz — Mon, 20 Jul 2026 19:00:58 +0000

I've been building with AI coding agents daily — Claude Code, OpenCode, custom agents. They're productive, but they have a failure mode that costs real money: they retry, loop, and fan out on metered API keys. Billing alerts fire hours after the damage is done.

Observability tools (Langfuse, Helicone, Portkey) tell you what happened. None of them stop it from happening.

Stoke

Stoke is a single Rust binary (~5.5 MB, zero runtime dependencies) that sits between your AI agents and model providers. It enforces policy before any provider is called:

Hard budget caps — per-API-key spend limits in USD. Over the limit = 429 Budget exceeded, not an alert. Streamed responses accrue spend too.
Loop detection — exact prompt-hash matching plus optional semantic similarity. 5 similar requests from one key in 60 seconds = blocked for 120 seconds. Catches agents stuck in retry loops.
Rate limiting — per-key requests-per-minute over a sliding window.

The key insight: enforce, don't just observe.

What else it does

Node-aware routing — polls your Ollama nodes, places requests on warm machines, balances by live load, fails over automatically
Federation — one Stoke can front another across machines, with a hop guard that prevents cycles
Auto-routing — send "model": "auto" and Stoke scores every (model, node) candidate on cost, predicted latency, and your preferences. Every response carries a receipt explaining the decision.
Anthropic Messages API — point Claude Code at Stoke with ANTHROPIC_BASE_URL. Same enforcement, then passthrough.
Plugin system — built-in PII redaction, audit logging, prompt harness + HTTP webhooks in any language + JS/TS via deno_core
Fail-closed auth — no keys configured and no dev flag = every request rejected. There is no accidentally-open state.

Why Rust

Every Python proxy (LiteLLM, OpenRouter) adds latency and memory overhead — 500MB+ RAM, Postgres, Redis. Stoke is 5.5 MB, starts in milliseconds, uses ~10 MB RAM, and has zero runtime dependencies. It's a single binary you can curl-pipe-sh install.

Multi-machine setup

If you have Ollama running on two machines (say a MacBook and a Mac Studio), Stoke routes between them automatically — preferring the machine where the model is already loaded in RAM, balancing by in-flight count, and measuring real time-to-first-token per node.

Or use federation: run a Stoke on each machine, keep Ollama on 127.0.0.1, and front them with a type = "stoke" provider. Stoke is the only network-facing door, and it requires auth.

Quickstart

curl -sSf https://stokegate.com/install | sh -s -- --version nightly

Minimal config:

routing = "single"
default_model = "qwen3:8b"

[server]
host = "127.0.0.1"
port = 8787

[[providers]]
name = "local"
base_url = "http://127.0.0.1:11434/v1"
tier = "local"

Start it:

STOKE_API_KEYS=stk-mykey ./target/release/stoke

Point any OpenAI-compatible client at http://localhost:8787/v1 — OpenCode, Cursor, custom apps. Budget caps, rate limits, and loop detection apply to every request.

Status

Pre-release, MIT licensed, EU-based. Dogfooded daily on a real two-Mac + Ollama setup. Built in the open at github.com/Ozperium/stoke.

The smoke scripts (./scripts/smoke.sh, ./scripts/smoke_federation.sh) run the full system end-to-end with mock Ollama nodes — no Ollama required to verify the claims.

What's next

Anthropic to OpenAI translation (so Claude Code can run against local models)
Homebrew formula and Docker images
Per-key loop-detection thresholds
OpenTelemetry traces and Prometheus metrics

Feedback welcome. The repo is github.com/Ozperium/stoke.

Testing AI Agents with AgentSpec: Jest for Non-Deterministic Behavior

Paw from Oz — Sun, 19 Jul 2026 19:13:47 +0000

AI agents are non-deterministic. This is both their superpower and their biggest testing challenge.

When you change a prompt, swap a model, or update a tool definition, the agent's behavior shifts in ways you can't predict. Traditional test tools (Jest, Vitest, Playwright) are built for deterministic code — they expect exact string matching. AI agents don't work that way.

Last week, I changed a system prompt in a support agent. It still passed all our Jest tests (the function still returned a string). But the agent had stopped offering password reset help — a regression that only surfaced when a user complained.

That's why I built AgentSpec.

What is AgentSpec?

AgentSpec is a testing framework designed specifically for AI agents. It provides assertions that handle non-deterministic output, behavior diff reports that show what changed, and CI/CD integration out of the box.

Assertions for AI output

Instead of exact string matching, AgentSpec provides assertions that work with the variability of AI responses:

# agentspec.yaml
name: "support agent"
tests:
  - name: "handles expired token"
    input: "my token expired"
    expect:
      contains: "refresh token"
      not_contains: "I don't know"
      max_latency_ms: 5000

  - name: "routes billing question"
    input: "I want a refund"
    expect:
      contains_any: ["billing", "refund", "support team"]
      tool_called: "transfer_to_billing"

  - name: "response is on-topic"
    input: "how do I reset my password?"
    expect:
      semantically_similar: "reset password credentials"
      min_confidence: 0.5

Available assertions:

contains / not_contains — substring checks
contains_any / contains_all — flexible multi-string matching
regex — pattern matching (error codes, IDs)
tool_called — verify specific tools were invoked
max_latency_ms / max_tokens — performance and cost gates
semantically_similar — word-overlap similarity (no API needed)
json_path — extract and verify fields from JSON agent output
llm_judge — use a local LLM to evaluate output quality

Behavior diff reports

This is the killer feature. AgentSpec stores the last passing output per test. When behavior changes, you see exactly what shifted:

⚠️ REGRESSION support agent > handles expired token
  ⚠  Behavior REGRESSED — test was passing, now failing
  + added: your, token, has, expired, please, contact, support
  - removed: you, need, refresh, the, token, settings, security

Instead of just knowing a test failed, you know WHAT changed. This turns debugging from "what broke?" into "the model stopped saying 'refresh token' and started saying 'renew credentials'."

LLM-as-judge with local models

The most powerful assertion is llm_judge — use an LLM to evaluate whether a response meets quality criteria that can't be expressed as string matching:

tests:
  - name: "response is helpful"
    input: "How do I reset my password?"
    expect:
      llm_judge: "The response should explain how to reset a password"

By default, AgentSpec uses a local Ollama model as the judge — no API costs, no data leaving your machine:

agentspec run --judge-endpoint http://127.0.0.1:11434/v1/chat/completions --judge-model qwen2.5:7b

Testing real agents

AgentSpec includes an HTTP agent adapter to test any HTTP-accessible AI agent:

agentspec run --endpoint https://my-agent.example.com/chat

It POSTs {input: "..."} to your endpoint and reads the response. Works with any agent that exposes an HTTP API.

CI/CD integration

AgentSpec is designed to drop into any CI pipeline:

# Exit code 1 on failure
agentspec run --ci

# JUnit XML for CI systems
agentspec run --junit > results.xml

There's also a GitHub Action:

- uses: agentspec/action@v1
  with:
    test-dir: tests

Get started

npm install -g @ozperium/agentspec
agentspec init
agentspec run

That's it. YAML test definitions, assertions for non-deterministic output, diff reports, CI integration. No cloud, no account, no telemetry.

GitHub: Ozperium/agentspec
Landing: agentspec.pages.dev

I'd love feedback on:

Is YAML the right format, or would you prefer TS test definitions?
What assertions are you missing for your AI agents?
Would you use LLM-as-judge with a local model?