When you wire a model into a coding agent, the question is never "which model is smartest." It's narrower and more practical: which model makes the right tool call, how often, how fast, at what cost per correct answer, and on which host. We could not find that answered in one place, so we built it. It started as our own tool; today it's public and built for the dev community: tirtha.ai/bench.
| At a glance | |
|---|---|
| Models tested | 68 |
| On the one shared test | 8, same 800 tasks and 120 traps, same scorer |
| Top accuracy | 95.4% (Grok 4.7 and Claude Opus 5.5, tied) |
| Same model, two hosts | 93.4% vs 69.5% (Llama 3.3 70B) |
| Dollars per 1,000 correct calls | $0.09 to $5.63 |
| Outside re-run of our 800 | 90.6% vs our 91.0% |
The one rule: same test, same scorer
Most leaderboards mix numbers from different test sets, different prompts and different graders, then rank them side by side. Ours has one rule: a model is only ranked against models that ran the identical test and were scored the same way.
The core test is 800 tool-calling tasks from Berkeley's public BFCL set (400 single calls, 200 where the model has to choose between several tools, 200 that need parallel calls) plus 120 trap prompts we wrote to tempt a wrong tool call. Every model on the shared table ran all of it.
Every label on the page comes from a fixed rule you can hover to read, not from our opinion. Our own product sits on the same table, measured the same way, split by the kind of job:
What a shared test shows that a leaderboard can't
The host matters, not just the model. The same open model, gpt-oss-120b, on the same 800 tasks, scored anywhere from 63.9% to 78.0% depending on which of four providers served it.
And Llama 3.3 70B is starker: 93.4% on one host, 69.5% on another. If you pick a model off a leaderboard and then buy it from whichever host is cheapest, you may not be getting the model you read about.
Price per right answer, not sticker price. A cheap model that gets more wrong can cost more per correct call. So we divide what each run actually cost by the answers it got right:
The spread is 65x: from $0.09 to $5.63 per 1,000 correct calls.
Versions move in ways one number hides. DeepSeek's new V4.1 Flash scores the same overall as V4 Flash, a tie within noise. But it falls for far fewer traps (88% avoided against 73%) and handles parallel calls better:
The models that check the models. Agents increasingly use a small yes/no model to approve or reject a tool call before it runs. Those checkers get the same treatment, including how many wrong calls they wave through.
Every model gets its own page
Click any model and you get its page: headline numbers with its rank against the models that ran the same test, a one-line read of what they mean, accuracy by task type, where it sits in the field on each measure, and every raw result we hold, failed runs included and marked. Here's Grok 4.5:
Checked from outside
A benchmark you can't reproduce is a claim. So every request we send is published, and someone did reproduce it. Mikhail Maslenik ran our 800 requests on his own machine, on a quantized copy of the same model, and scored them with Berkeley's official checker. He got 90.6%. We got 91.0%. Three tasks apart out of 800.
He also found what every honest re-run finds: two items in the public set whose expected answer punishes a fair reply, and the fact that our scorer is our own rewrite of Berkeley's. We're scoring our runs with the official checker next so the two can be compared directly, and the two items are footnoted on the page.
What we're building next, from developer feedback
Single tool calls are the easy part. Real coding work is a long session, and Mikhail pushed us on exactly where models break there. His case: a direct instruction, "connect to the local server and update it," and one model spent its whole budget reading unrelated files and never connected, while a free model connected, updated and pulled logs in minutes. So the next track, Real Work, will measure:
- Work before the first real action: files opened, calls made, tokens and seconds spent before the model does the thing it was asked to do, and whether it ever does.
- Context fill: the same task started from a clean context, from a context already half full of real session history, and right after compression.
- An endurance run (his design): 45 minutes building a small sync tool against a server that changes its data format partway through, fails every tenth request, and hides instructions inside the data.
- The logs to explain the score: every step, every retry, every recovery, on one clock, so a low score turns into a story you can fix.
Your turn
The part we haven't solved is fairness on real work. One run of a 45-minute task is one sample; models that get lucky early look disciplined. How many seeds is enough before a ranking means something? And what would you want measured before you trust a model with your repo? Tell us in the comments, and if you run the 800 yourself, we'll put your numbers next to ours. Any feedback is appreciated, on the bench, the page, or what's missing.
The full bench, every model and every raw row: tirtha.ai/bench
Top comments (1)
Лайк!