Cheaper Per Token Is Not Cheaper Per Outcome

Disclosure up front: I build agentproto, the daemon
in the setup section. The benchmark readings and their caveats hold whatever
you route with — the tool is just how I gate it. Corrections welcome, file an
issue.

A coding model that scores within a few points of the frontier and costs a
tenth as much per token exists right now. Several do — GLM-5.2, DeepSeek V4,
Kimi K2.7, all permissively licensed, all posting 76–83% on SWE-bench Verified
as of July 2026.

Route your mechanical work to one of them and you cut your inference bill by an
order of magnitude. Route it there with nothing checking the output, and you
haven't saved a cent — you've moved the cost somewhere you can't see it yet.

The one idea, if you remember nothing else:
Cheaper per token is not cheaper per outcome. The judge is what makes the
arbitrage net-positive.

That's the whole essay. The cheap model is real, the savings are real, and they
evaporate on the first bad merge nobody caught. Everything below is how to keep
the savings and not the bill.

The arithmetic that turns a saving into a loss

Start with the number people quote and stop: a model 10–30× cheaper per token.
It's true, and it's the wrong unit. The unit that pays your salary is cost per
successful task, and a cheap model that fails one task in ten changes that
number in a hurry.

Compute cost per successful task, not per token. A model 10–30× cheaper
per token that fails 1 in 10 of your tasks isn't cheaper once you count the
wrong answers and the developer hours spent unwinding them — and for the
hardest reasoning loops, the pricier frontier model can win on total cost
outright.

Run the numbers on a real fleet and the leverage is brutal in both directions.
Say the cheap executor saves you $20 of frontier tokens per task. One uncaught
bad merge costs you an afternoon of debugging plus the revert — call it fifty
tasks of savings, gone, on a single miss.

At cheap-executor volume, misses aren't rare events you can shrug off. They're a
steady drip, and unreviewed failures merge fast because the thing that made
execution cheap also made it high-throughput. The savings and the risk scale
on the exact same axis — token volume.

So the routing question was never "which model is cheapest." It's "what catches
the cheap model's 25% before it costs me the afternoon?" That's a question about
your fleet's shape, not your model list — so look at the fleet first.

What's your fleet? Three shapes, three bills

Strip the tooling and every setup routing by cost is one of three shapes. Find
yours, then find where it leaks money.

Shape 1 — single-model loyalty. One vendor, one model, everything. Every task
pays frontier prices, including the mechanical 80% that a $0.14 model would
nail. You're not exposed to the arbitrage trap because you're not taking the
arbitrage — you're overpaying for lint fixes and file renames as a matter of
policy.

The ceiling: you're leaving the whole saving on the table. High-volume,
low-complexity work — classification, extraction, boilerplate edits — is
dominated by inference cost, not peak capability. Flash-tier open models
(DeepSeek-V4-Flash at ~$0.14 per million input tokens, MiniMax at ~$0.30)
handle it at a fraction of frontier API cost. (Per Faros AI's open-weight-models
roundup.)

Shape 2 — mixed fleet, no gate. You took the advice, split the work, and now
a cheap model executes while a frontier model plans. Your token bill dropped.
This is the dangerous one, because the savings are visible on the invoice and
the cost is invisible until a bad merge surfaces in production three days later.

The ceiling here is the one this whole series keeps circling: the agent grades
its own work, and it grades leniently. You saved money per token and quietly
took on a failure rate nobody is measuring.

Shape 3 — mixed fleet with an independent gate. Cheap model executes, and
something outside that model — a test suite, a skeptical judge, ideally both —
decides whether the output ships, before it merges. This is the only shape where
the per-token saving survives contact with the per-outcome accounting.

Where are you? If your honest answer is "one model does everything,"
you're overpaying. If it's "cheap model, and I skim its diffs myself,"
you're the gate — and you don't scale, and you don't fork.
Only Shape 3 is actually banking the arbitrage. The rest of this piece is how
to stand there.

Read the benchmarks like a skeptic, or you'll route wrong

Before you route anything, you have to trust the scores — and the scores are
slipperier than the marketing lets on. This is the fair part: the open coders
really are frontier-band. It's the rankings that lie.

Here's the July 2026 receipt — the same SWE-bench spread introduced in piece
1, broken out model by model here —
with the permissively-licensed field sitting a short step behind closed
frontier:

Model	License	SWE-bench Verified	Where it actually tops
GPT-5.6 Sol	closed	96.2	frontier ceiling
Claude Fable 5	closed	95.0	frontier ceiling
GLM-5.2 (Z.ai)	MIT	82.8	open-intelligence aggregate
Kimi K2.7 (Moonshot)	open-weight	78.2	Kilo Bench (60.7%)
DeepSeek V4	MIT	76.2	LiveCodeBench (V4-Pro, 93.5%)
MiniMax M3	open-weight	—	SWE-Bench Pro (59.0%)

Two reading rules, because a vendor slide will mislead you politely:

Vendor scores are floors, not ranks. Self-reported numbers use each lab's
own scaffold, retry budget, and effort settings, so two vendors claiming #1
within 0.2 points are inside harness noise. Use vendor numbers to confirm a
model is in the frontier band; use a neutral harness (Artificial Analysis,
LMArena) — one fixed methodology across all models — to order them within it.

Rankings invert across suites, so a headline is not a verdict. Kimi K2.7
leads Kilo Bench (60.7%) while GLM-5.2 tops the overall ranking (53.0%);
DeepSeek V4-Pro leads LiveCodeBench (93.5%, beating closed frontier APIs)
while MiniMax M3 tops SWE-Bench Pro (59.0%). And check the fine print — GLM's
"first past 80% on Terminal-Bench" was scored on TB 2.1, the relaxed
revision with looser timeouts, not comparable to earlier runs. (Per Kilo's
open-source coding models page and Agyn's 2026 open-source LLM roundup.)

The benchmark that doesn't resemble your workload is trivia. SWE-bench
Verified is Python-heavy with well-scoped tickets and deterministic tests; if
your work is a TypeScript monorepo with flaky integration suites, its ranking
tells you almost nothing about which model to trust on your afternoon. The only
leaderboard that matters is a few dozen tasks from your own repo — which,
conveniently, is also the thing your gate runs on.

Plan expensive, execute cheap, verify independently

Once you read the numbers honestly, the rational fleet falls out of the price
sheet almost mechanically. It has three legs, and the third is the one everyone
skips.

• A frontier model plans. Decomposition, architecture, the 20% of judgment that's genuinely worth frontier prices. This is a handful of expensive tokens that shape everything downstream.
• Cheap open models execute. The mechanical 80%: apply the plan, write the tests, fix the lint. This is where the token volume lives, so this is where the money is saved.
• A separate judge verifies. Because a model that succeeds 75% of the time fails 25% of the time, and at execution volume those failures merge before you notice.

This isn't a novel invention; it's what everyone who's shipped a serious harness
converged on, under three different vocabularies.

Three labs, three names, one architecture. Cursor landed on
planner/worker/judge, Anthropic on planner/generator/evaluator, the harness
literature on a separate grading agent with a rubric that "has not seen the
task agent's reasoning." When three teams independently split the same seam,
it's a requirement, not a style. (Per Jinyan Su's essay on planner/generator/
evaluator splits.)

The third leg is load-bearing and non-negotiable, and it's precisely the leg the
cost-savings pitch drops. The reason it can't be the cheap model checking itself
is the whole subject of the supervision ladder:
an agent grading its own work is a defendant marking their own exam. Skepticism
is cheap to install in a second agent and nearly impossible to install in the
first one.

So the mixed fleet isn't "cheap model plus a prompt telling it to be careful."
It's cheap model plus a check it can't see, reach, or talk out of failing. Which
raises the obvious question: can't you just buy that from a vendor?

Why no vendor will sell you the mixed fleet

You can't, and the reason is structural, not a product gap someone will patch
next quarter. Every vendor's orchestration stack is single-vendor by
construction, because no lab has an incentive to make its cheap tier easy to pair
with a rival's judge.

The sharpest example is Anthropic's Advisor tool — genuinely clever, and exactly
the shape the economics want: a cheap executor consults a stronger model
mid-turn, which is routing. It has the same single-vendor, encrypted-advice
catch as Agent Teams — see the landscape
piece for the compatibility-table
breakdown — which puts the configuration the price sheet actually wants (a GLM
executor judged by a Claude reviewer) structurally outside every vendor
offering.

Concede the depth, because it's real: for an all-Claude shop on a Team plan, the
vendor-native path integrates closer to the model than any third party ever
will, and if your whole fleet is one vendor's models, take it. Depth of
integration is a genuine strength, and pretending otherwise would cost this piece
its credibility.

But the mixed fleet is defined by not being one vendor's models. The moment
your executor and your judge come from different labs — the moment you route by
cost across vendors — the routing and the checking layer have to sit outside all
of them, in infrastructure you run. The layer that arbitrages models can't
itself belong to one of the models.

A working setup, with the cost notes

Here's the concrete version — one daemon, three roles, any vendors. agentproto
shown because it's what I build, but the shape ports to anything that can run all
three legs and physically gate the merge.

npm i -g @agentproto/cli && agentproto serve

# 1. PLAN — frontier model writes the plan to a file, implements nothing
agentproto sessions start claude-code --model claude-opus-4-8 \
  --prompt "Read the issue in TASK.md. Write a step-by-step implementation plan to PLAN.md. Do not write code."

# 2. EXECUTE — cheap open model implements the plan (hermes → OpenRouter)
agentproto sessions start hermes --model z-ai/glm-5.2 \
  --prompt "Implement PLAN.md exactly. Run the tests as you go."

Then gate the executor with an independent judge before anything lands — a
completion policy attached over MCP from any client wired to the daemon
(policy_attach). One call chains a shell gate → a skeptical judge → a commit
staged behind your ack:

{
  "sessionId": "<executor-session>",
  "gate": { "command": "pnpm", "args": ["test"] },
  "onFail": { "maxRetries": 2, "nudge": "Gate failed (exit {code}). Fix and finish." },
  "then": "emit",
  "next": {
    "gate": { "judge": {
      "adapter": "claude-code", "model": "claude-sonnet-5",
      "prompt": "Skeptical review: does this diff implement PLAN.md without weakening or skipping tests? Try to refute it."
    }},
    "then": "commit",
    "commit": { "paths": ["src"], "message": "implement PLAN.md", "requireHumanAck": true }
  }
}

The expensive model reads and writes plans and verdicts; it never grinds
tokens on mechanical edits. Trace the bill through the pipeline: the plan is
one frontier call, the execution — where 90%+ of tokens burn — runs at
open-model prices, and the judge is one mid-tier call per turn-end. Frontier
quality touches only the two places judgment actually pays for itself.

Two notes worth pinning down before you copy this — one a receipt already spent
elsewhere, one a routing rule:

Watch the defaults — they route to the expensive model more than you'd
think. We hit this in the Paseo session: its default Claude model is Opus,
and a one-word test prompt billed $0.29 before we'd typed anything real —
see the hands-on review for the full log.
Sessions inherit whatever model you configured, so pin --model explicitly in
anything scripted, or the arbitrage quietly runs in reverse.

• Route by task class, not by loyalty or by mood. Mechanical single-file fixes → cheap executor, gate only. Cross-cutting refactors → frontier plans first, cheap execution under a strict gate. Anything touching auth or payments → frontier end-to-end with human review regardless, because that's the one place where a saved dollar can cost you a breach.

The gate itself is the top rung of the supervision ladder — a check in a
long-lived daemon that survives your laptop closing — and the full climb up to
it is its own piece. The point here is narrower:
without that third leg, Shape 2 is just a cheaper way to ship bugs.

The bill you can see and the bill you can't

Open weights made execution cheap. Reading the benchmarks honestly tells you
which cheap model to trust for which job. And no vendor will sell you the
cross-vendor fleet the price sheet is begging for — so the layer that routes,
and the layer that checks, both have to live in infrastructure you run.

Do the routing without the judge and you're not saving money. You're taking the
saving that shows up on this month's invoice and paying it back, with interest,
on a bad merge you'll debug next month. Cheaper per token is a line item.
Cheaper per outcome is a system — and the judge is the part you can't skip.

This is the interop problem in miniature, by the way: a plan file written by one
model, executed by another, judged by a third, all speaking through files with
contracts instead of one vendor's SDK. That shared substrate — the layer every
mixed fleet quietly reinvents — is where this series goes next.

If your fleet already banks the arbitrage a cleaner way, or I've mis-priced a
model or mis-read a benchmark, tell me where — I'll fix the piece.

---

The series — Orchestration, Honestly

Ten pieces, one argument. Start anywhere; each one cross-links the rest.

	Piece	The one idea
1	You can't parallelize the trust	Amdahl's Law: why your fifth agent slows you down
2	Harness engineering	you rent the model; the harness is the part you own
3	The supervision ladder	five rungs of trusting an agent you don't watch
4	The approval plane	auto-approve reads, gate writes — wire the line between
5	Kill the loop	why "keep going until done" compounds a wrong turn
6	Route by cost (you're here)	plan expensive, execute cheap, verify independently
7	Files with contracts	the interop layer every agent system reinvents
8	Knowledge is power	give your agent your knowledge, not the internet's average
9	Paseo, hands-on	a full real-session review of the daemon
10	9 orchestrators, compared	the tool-by-tool teardown + a decision table

Written by the maintainer of agentproto (Apache-2.0, source). Same contract as our /compare page — dated facts, named strengths, corrections by issue. Got something wrong? File an issue.

Building agentproto in the open — follow @theagentproto and @agentik_ai on X.