Your AI Agent Has a Burn Rate — So I Benchmarked the Cost of a Correct Answer

My submission to the Kaggle Benchmarking Challenge.
Benchmark: https://www.kaggle.com/benchmarks/karmendra8386/cost-per-correct
Tooling: tokenecon — my open-source cost accounting for agent workflows · Preprint: https://doi.org/10.5281/zenodo.23195511

A single model call has a predictable price. A looping agent does not. Every tool call, retry, and retrieved document adds another priced round-trip, and most teams find out the total from the cloud bill at the end of the month.

So I built a benchmark that asks a different question. Not "what does the model cost?" — but "what does each correct answer cost?"

What I built

A suite of 24 small agent-style tasks in three difficulty bands — easy (arithmetic, extraction, classification), medium (analysis, drafting, prioritization), and hard (multi-hop synthesis, trade-offs, derivations). The kind of work production agents actually do, not trivia. Every task has a deterministic checker: the answer is right or it isn't. No LLM judge, no vibes.

The first five easy tasks are live on Kaggle now, run against five hosted models: Gemini 2.5 Flash, Gemini 3.7 Flash, DeepSeek R1, GPT-OSS 120B, and Qwen3 235B.

The Kaggle results (real, from the live task pages)

Task	Gemini 2.5 Flash	Gemini 3.7 Flash	DeepSeek R1	GPT-OSS 120B	Qwen3 235B
e1 — arithmetic: 47 × 38 + 112	Pass	–	–	–	–
e2 — extraction: founding year	Pass	–	Error	Error	Error
e3 — sentiment classification	Pass	Pass	Pass	Pass	Pass
e4 — arithmetic: requests in 3 hours	Pass	Pass	Pass	Pass	Pass
e5 — extraction: Eiffel Tower height	Pass	Pass	Pass	Pass	Fail

Three things stand out:

1. One model finished everything. Gemini 2.5 Flash is the only model with a result on all five tasks — 5 for 5 — and it tops the benchmark leaderboard at 100.00. On the three tasks where all five models ran (e3–e5), 14 of 15 rows pass.

2. The only genuine failure is the interesting one. GPT-OSS 120B, a flagship open-weights model, failed e5 — a one-line extraction ("how tall is the Eiffel Tower, in meters?") that every other model passed. Easy tiers are supposed to be boring. They mostly are — until a model drops a task it had no business dropping, and in production that dropped task is a real user's request.

3. When accuracy ties, cost is the whole game. On easy work, nearly every modern model is right nearly all the time. If accuracy can't separate them, the differentiator is what each correct answer costs — which is exactly the number most teams never compute.

One honest note on e2: its three "Error" rows are Kaggle platform errors from my first build pattern, not model answers — the runs completed but the platform wouldn't register swapped-in models. I fixed the pattern for e3 onward and left e2 as-is rather than quietly rebuilding it, because a benchmark you can silently edit isn't worth much.

The other half: routing (local simulation)

The Kaggle tasks above test models one at a time. Production agents don't work that way — a router decides which tier handles each task. So locally, I ran the full 24-task suite through four routing strategies using tokenecon, with deterministic scoring:

Strategy	Accuracy	Total cost	Cost per correct answer
Tiered routing	100%	$0.1661	$0.0069
Large model only	100%	$0.2334	$0.0097
Medium model only	67%	$0.0274	$0.0017
Small model only	33%	$0.0036	$0.0004

Tiered routing matched the big model's accuracy at 29% lower cost — easy tasks went to the cheap tier, hard ones escalated. But look at the last row: the cheapest tier has the lowest cost per correct answer and gets two-thirds of the work wrong. Cost-per-correct without an accuracy floor is a trap. With a ≥95% accuracy requirement, tiered routing is the cheapest viable strategy — and that floor is the piece most cost dashboards are missing.

To be clear about what this table is: a deterministic local simulation with illustrative prices from tokenecon's price table. Kaggle's task pages report pass/fail only, so no dollar figures above come from Kaggle. The simulation measures the routing policy, which is the part teams actually control.

Disclosures, because benchmarks should have them

• Only the 5 easy tasks are live on Kaggle so far. The medium and hard tiers exist in the suite and in the local runs; their Kaggle versions come next.
• Kaggle's model catalog offers Qwen3 235B in its 2506 build, so that's what ran — my original spec said 2507.
• Gemini 3.7 Flash appears because it's Kaggle's default placeholder model for benchmark tasks built from its template. Its rows are real runs, so I kept them.
• The stray private task page some may spot ("What is Kaggle?") is an untouched template I built by accident while learning the UI. It's not part of the benchmark.

What's next

Medium and hard tiers go up on Kaggle next, and that's where this gets interesting — easy tasks saturate, but multi-hop work is where models separate and where routing earns its keep. If you run agents in production, fork the harness, plug in your own tasks and your own prices, and find out what your correct answers actually cost.

Comments

[Dean Lee]

Treating failed attempts as zero-cost in the cost-per-correct denominator is the core reason the small model looks artificially cheap. In production, an incorrect answer is never costless; it carries an asymmetric penalty in human review queues, retries, or downstream reconciliation. Once you price that failure cost into the objective function, the apparent efficiency of the low-accuracy tier evaporates immediately.

The second variable that erodes the 29 percent savings in tiered routing is the classification error of the router itself. If the router misclassifies a complex request as simple, the workflow either delivers a silent failure or triggers an escalation retry where you pay the small model token burn on top of the large model fallback. The margin between single-tier and multi-tier architectures usually gets determined by the cost of those cascading retries rather than nominal token rates.

[karmendra pandey]

You're right on both counts. A failed attempt is never zero-cost in production — price in even a $0.01 human-review penalty per wrong answer and the small model's math flips: 16 failures on 24 tasks takes it from $0.0004 to ~$0.020 per correct answer, while tiered routing (100% accuracy) stays at $0.0069. The accuracy floor in the post was a crude version of exactly your point; a failure penalty is the cleaner way to express it. On the router: mine was deterministic in the local run, so misclassification was zero by construction — in production, a misrouted hard task costs you the small-model burn plus the large-model fallback, and that retry cost is where the 29% saving lives or dies. Both go into the next version of the benchmark. Appreciate the push.

[Wilfredo Lassalle]

Karmendra, I’d price all three, but distinguish a mistake caught in review from one that reaches a customer or changes a decision. The first costs review time and rework. The second can add rollback, remediation and customer harm. A bad summary and a wrong claims decision aren’t the same risk.

From a release-gate standpoint, some failures should block autonomous execution regardless of average cost. I’d test routing against those failure types first, then compare costs among the options that clear the gate. Cheaper only matters after the workflow is safe enough to run.

[karmendra pandey]

That's the framing I was missing, Will. Failure cost isn't one number, it's a ladder — caught in review is review time and rework; escaped is rollback, remediation, and customer harm; and some failures shouldn't be priced at all, they should be gated out of autonomous execution entirely. That turns my simple accuracy floor into something better: a risk gate first, then cost per correct only among the routings that clear it. A bad summary and a wrong claims decision sat in the same bucket in my benchmark. They shouldn't. The next version splits failures by type before it compares costs. Thank you — this genuinely changes the design.