My submission to the Kaggle Benchmarking Challenge.
Benchmark: https://www.kaggle.com/benchmarks/karmendra8386/cost-per-correct
Tooling: tokenecon — my open-source cost accounting for agent workflows · Preprint: https://doi.org/10.5281/zenodo.23195511
A single model call has a predictable price. A looping agent does not. Every tool call, retry, and retrieved document adds another priced round-trip, and most teams find out the total from the cloud bill at the end of the month.
So I built a benchmark that asks a different question. Not "what does the model cost?" — but "what does each correct answer cost?"
What I built
A suite of 24 small agent-style tasks in three difficulty bands — easy (arithmetic, extraction, classification), medium (analysis, drafting, prioritization), and hard (multi-hop synthesis, trade-offs, derivations). The kind of work production agents actually do, not trivia. Every task has a deterministic checker: the answer is right or it isn't. No LLM judge, no vibes.
The first five easy tasks are live on Kaggle now, run against five hosted models: Gemini 2.5 Flash, Gemini 3.7 Flash, DeepSeek R1, GPT-OSS 120B, and Qwen3 235B.
The Kaggle results (real, from the live task pages)
| Task | Gemini 2.5 Flash | Gemini 3.7 Flash | DeepSeek R1 | GPT-OSS 120B | Qwen3 235B |
|---|---|---|---|---|---|
| e1 — arithmetic: 47 × 38 + 112 | Pass | – | – | – | – |
| e2 — extraction: founding year | Pass | – | Error | Error | Error |
| e3 — sentiment classification | Pass | Pass | Pass | Pass | Pass |
| e4 — arithmetic: requests in 3 hours | Pass | Pass | Pass | Pass | Pass |
| e5 — extraction: Eiffel Tower height | Pass | Pass | Pass | Pass | Fail |
Three things stand out:
1. One model finished everything. Gemini 2.5 Flash is the only model with a result on all five tasks — 5 for 5 — and it tops the benchmark leaderboard at 100.00. On the three tasks where all five models ran (e3–e5), 14 of 15 rows pass.
2. The only genuine failure is the interesting one. GPT-OSS 120B, a flagship open-weights model, failed e5 — a one-line extraction ("how tall is the Eiffel Tower, in meters?") that every other model passed. Easy tiers are supposed to be boring. They mostly are — until a model drops a task it had no business dropping, and in production that dropped task is a real user's request.
3. When accuracy ties, cost is the whole game. On easy work, nearly every modern model is right nearly all the time. If accuracy can't separate them, the differentiator is what each correct answer costs — which is exactly the number most teams never compute.
One honest note on e2: its three "Error" rows are Kaggle platform errors from my first build pattern, not model answers — the runs completed but the platform wouldn't register swapped-in models. I fixed the pattern for e3 onward and left e2 as-is rather than quietly rebuilding it, because a benchmark you can silently edit isn't worth much.
The other half: routing (local simulation)
The Kaggle tasks above test models one at a time. Production agents don't work that way — a router decides which tier handles each task. So locally, I ran the full 24-task suite through four routing strategies using tokenecon, with deterministic scoring:
| Strategy | Accuracy | Total cost | Cost per correct answer |
|---|---|---|---|
| Tiered routing | 100% | $0.1661 | $0.0069 |
| Large model only | 100% | $0.2334 | $0.0097 |
| Medium model only | 67% | $0.0274 | $0.0017 |
| Small model only | 33% | $0.0036 | $0.0004 |
Tiered routing matched the big model's accuracy at 29% lower cost — easy tasks went to the cheap tier, hard ones escalated. But look at the last row: the cheapest tier has the lowest cost per correct answer and gets two-thirds of the work wrong. Cost-per-correct without an accuracy floor is a trap. With a ≥95% accuracy requirement, tiered routing is the cheapest viable strategy — and that floor is the piece most cost dashboards are missing.
To be clear about what this table is: a deterministic local simulation with illustrative prices from tokenecon's price table. Kaggle's task pages report pass/fail only, so no dollar figures above come from Kaggle. The simulation measures the routing policy, which is the part teams actually control.
Disclosures, because benchmarks should have them
- Only the 5 easy tasks are live on Kaggle so far. The medium and hard tiers exist in the suite and in the local runs; their Kaggle versions come next.
- Kaggle's model catalog offers Qwen3 235B in its 2506 build, so that's what ran — my original spec said 2507.
- Gemini 3.7 Flash appears because it's Kaggle's default placeholder model for benchmark tasks built from its template. Its rows are real runs, so I kept them.
- The stray private task page some may spot ("What is Kaggle?") is an untouched template I built by accident while learning the UI. It's not part of the benchmark.
What's next
Medium and hard tiers go up on Kaggle next, and that's where this gets interesting — easy tasks saturate, but multi-hop work is where models separate and where routing earns its keep. If you run agents in production, fork the harness, plug in your own tasks and your own prices, and find out what your correct answers actually cost.
Top comments (2)
Treating failed attempts as zero-cost in the cost-per-correct denominator is the core reason the small model looks artificially cheap. In production, an incorrect answer is never costless; it carries an asymmetric penalty in human review queues, retries, or downstream reconciliation. Once you price that failure cost into the objective function, the apparent efficiency of the low-accuracy tier evaporates immediately.
The second variable that erodes the 29 percent savings in tiered routing is the classification error of the router itself. If the router misclassifies a complex request as simple, the workflow either delivers a silent failure or triggers an escalation retry where you pay the small model token burn on top of the large model fallback. The margin between single-tier and multi-tier architectures usually gets determined by the cost of those cascading retries rather than nominal token rates.
tr.ee/dev-to