Most people pick one model and ask it to do everything. We stopped doing that months ago, mostly because it kept running out. Our subscriptions hit their limits, and the one model doing everything also checked its own work, which is the one job a model is worst at.
So we split it. Claude plans and checks. Grok attacks whatever Claude just wrote. Codex builds the big pieces. And as of this week, Tīrtha, our own router, takes the routine building. We kept a ledger of who caught what. Here is what it shows, and what it doesn't show yet.
| Reviewer | Findings raised | Held up on re-check | Not in Claude's draft | Critical and new |
|---|---|---|---|---|
| Grok | 519 | 309 | 207 | 59 |
| Codex | 50 | 39 | 33 | 5 |
Logged Sep 25 to Oct 17. Cost per Grok review: median $0.021 over the last 30 ($2.32 total).
Grok: the cheap second opinion
Grok raised 519 findings between Sep 25 and Oct 17. When we re-checked them by hand, 309 held up. 207 of those were things Claude's own draft did not have, and 59 of the 207 were critical.
It makes mistakes too. 28 of its findings were simply wrong, and another 30 were wrong because our brief was missing something. At two cents a review, a reviewer that is sometimes wrong is still well worth having.
The last 30 Grok reviews cost $2.32 in total, a median of about two cents each. It is cheap because it never gets the repo. It gets a short brief with the claim and the evidence, around 2,300 tokens, and is told to break it. (We first wrote about this in September, in What Grok catches, what Codex catches, and what the pair costs us. That was two reviews. This is 746 logged findings.)
One example from today. Claude built a gateway fix and it passed 55 tests. A Grok attack that cost 17 cents found three cases where the fix would have blanked a correct answer. All three were real, and all three are fixed now.
Codex: the one that reads everything
Codex raised far fewer findings, 50, but 39 held up and 33 were new to Claude. The best one: it noticed that our live gateway was writing no usage rows at all for some requests, so billing was blind. Nobody else would have seen that without reading the code.
The depth costs a lot. A median Codex run uses about 84,000 tokens, and between Sep 22 and Oct 14 we hit Codex's usage limit 7 times, twice for days. Every routine job we hand Codex is a job that eats the window we need for the hard ones.
Tīrtha: the routine builder (just started)
This is where we are honest about how little we know. Tīrtha took its first real job this week, through OpenCode, after we fixed a bug where tool calls came back as plain text (0 of 2 sessions working before, 3 of 3 after). It's a smoke test. A track record takes weeks of real jobs.
The bet is simple: let Codex do what only Codex can do, and send the plumbing to a cheap, checked builder, so the subscriptions last. We ran a version of this in the summer, when a free cheap tier did 36 of 52 fleet tasks and we saw no Codex limit errors from June to August. The timing fits, but plenty else changed in those months, so we can't call it proof.
What we can't show yet
- Whether the whole team beats Claude working alone. We built that test and haven't run it.
- Whether moving the building to Tīrtha makes the subscriptions last longer. We will measure it now that it has started.
- The ledger itself has a weakness: most of its 746 rows were logged after the fact from saved reviews, and Claude judged which findings were already in its own draft. A model grading its own blind spots is exactly the kind of thing Grok would flag.
So, how do you split yours?
Do you use one model for everything, or give each one a job? Has anyone measured whether a cheap reviewer actually catches more than it costs you in false alarms? We'd like to compare notes, and we'll share our ledger format if it helps.
Update, Oct 19 evening: the first pilot failed
We ran the routine building pilot the same night. It failed where we already knew we were weakest: long, multi-step agent jobs. We just hadn't seen how much that weakness costs.
- OpenCode on Tīrtha's cheap tier failed both runs. In one it reported the job done without changing any code. In the other it broke a file, committed it, and kept trying to push. The sandbox had no credentials, so nothing left the machine.
- Aider got the job right, but every request went up to the top model, so it saved nothing.
- Codex couldn't run on Tīrtha yet. Three gateway bugs in the Responses API stopped it.
The lesson is about compounding. On our bench Tīrtha gets 93.6% of single tool calls right, middle of the pack among the models we test. That sounds close to the leaders until a job takes a dozen steps in a row. At 93.6% per step, twelve clean steps in a row happen less than half the time (0.936 to the 12th is about 0.45). Our checks look at each call's shape, not at whether the agent is going in circles, so nothing escalated.
What we're fixing: escalate when the same failing steps repeat, stop cutting top-model answers at 2,048 tokens, and fix the Responses API for Codex. Then we run the same pilot again and post the result here.
Update 2, Oct 19 night: it was an outage on our side
An hour later we found the real cause. Our account with the supplier that hosts Tīrtha's tool-call model had run out of credit, so for about three hours every tool call quietly fell back to a weaker backup model. Nothing alerted us. After the top-up we ran the same pilot again on the right model: OpenCode passed 2 of 2 on the real repo task with no stray commits or pushes, and Cline and Continue went from 0 of 2 to 2 of 2.
The compounding math above still holds for weaker models, which is exactly what we were serving by accident. The first fix now is the alarm we didn't have: an alert the moment any model in the ladder stops answering, and a warning before a balance runs out.
Top comments (4)
I'd hand-label a sample of the "not in Claude's draft" calls and put a dumb baseline next to Claude on it. That's how my own LLM judge got caught out. It matched each claim in an agent's report against the tool log, and at picking out the reports I'd later questioned, about 7% of them, it scored 0.48 AUROC, no better than chance. It also marked about 70% of all claims "no evidence" on every kind of report. A plain TF-IDF classifier on the same labels did better at 0.65, and neither was good enough to use.
That's the right test, and it hits the weak spot the article admits: Claude decided which findings were already in its own draft. A dumb baseline is cheap here too, plain word overlap between each finding and the draft it was checked against. So the plan is a random sample of the 207, labeled blind by a model from another family that never sees Claude's call, then Claude and the overlap baseline both scored against those labels. I'll post the numbers in this thread. Your 70% "no evidence" rate is interesting. Was that the judge being cautious, or were the tool logs really missing the steps the reports described?
The reason a two-cent critic works here is that you are buying an asymmetric payoff option on catastrophic tail risk. Catching 59 critical errors that Claude missed for under three dollars in API spend is essentially buying deep out-of-the-money insurance at a massive discount.
The real hurdle rate in multi-model verification is never token spend, but the human triage tax on false discoveries. Your logs show roughly an eleven percent false alarm rate across the 519 Grok findings. At that ratio, engineering focus stays intact. If that error rate drifts higher, developer context switching quickly overtakes the compute savings. A fifteen-minute human investigation into an invalid finding costs orders of magnitude more than the two cents spent generating it.
The economic break-even between a single expensive model and a heterogeneous review committee depends almost entirely on the shape of the downstream loss function. For high-severity liabilities like unbilled gateway traffic, an aggressive, cheap critic delivers positive expected value even with substantial noise. For low-stakes routine logic, false alarms quietly turn into an unhedged operational drain.
Agreed that triage is the real cost, and two things in our logs change the math a little. Of the 58 wrong findings, 30 were our own fault, a brief missing context or a reviewer kept blind on purpose, so Grok's own error rate was 28 of 519, about 5%. And the first triage pass isn't a person: Claude re-checks each finding against the code before anything changes, so most false alarms cost model time, not someone's afternoon. What we haven't measured is that re-check cost by severity, and your point says that's exactly where the break-even lives. The 76 findings nobody ever tested are their own quiet cost too.