DEV Community

Cover image for Four models, one job each: what our own logs say about splitting the work
Tom Jones
Tom Jones

Posted on Edited on

Four models, one job each: what our own logs say about splitting the work

Most people pick one model and ask it to do everything. We stopped doing that months ago, mostly because it kept running out. Our subscriptions hit their limits, and the one model doing everything also checked its own work, which is the one job a model is worst at.

So we split it. Claude plans and checks. Grok attacks whatever Claude just wrote. Codex builds the big pieces. And as of this week, Tīrtha, our own router, takes the routine building. We kept a ledger of who caught what. Here is what it shows, and what it doesn't show yet.

What our reviewers caught that Claude had missed

Reviewer Findings raised Held up on re-check Not in Claude's draft Critical and new
Grok 519 309 207 59
Codex 50 39 33 5

Logged Sep 25 to Oct 17. Cost per Grok review: median $0.021 over the last 30 ($2.32 total).

Grok: the cheap second opinion

Grok raised 519 findings between Sep 25 and Oct 17. When we re-checked them by hand, 309 held up. 207 of those were things Claude's own draft did not have, and 59 of the 207 were critical.

It makes mistakes too. 28 of its findings were simply wrong, and another 30 were wrong because our brief was missing something. At two cents a review, a reviewer that is sometimes wrong is still well worth having.

The last 30 Grok reviews cost $2.32 in total, a median of about two cents each. It is cheap because it never gets the repo. It gets a short brief with the claim and the evidence, around 2,300 tokens, and is told to break it. (We first wrote about this in September, in What Grok catches, what Codex catches, and what the pair costs us. That was two reviews. This is 746 logged findings.)

One example from today. Claude built a gateway fix and it passed 55 tests. A Grok attack that cost 17 cents found three cases where the fix would have blanked a correct answer. All three were real, and all three are fixed now.

Codex: the one that reads everything

Codex raised far fewer findings, 50, but 39 held up and 33 were new to Claude. The best one: it noticed that our live gateway was writing no usage rows at all for some requests, so billing was blind. Nobody else would have seen that without reading the code.

The depth costs a lot. A median Codex run uses about 84,000 tokens, and between Sep 22 and Oct 14 we hit Codex's usage limit 7 times, twice for days. Every routine job we hand Codex is a job that eats the window we need for the hard ones.

Tīrtha: the routine builder (just started)

This is where we are honest about how little we know. Tīrtha took its first real job this week, through OpenCode, after we fixed a bug where tool calls came back as plain text (0 of 2 sessions working before, 3 of 3 after). It's a smoke test. A track record takes weeks of real jobs.

The bet is simple: let Codex do what only Codex can do, and send the plumbing to a cheap, checked builder, so the subscriptions last. We ran a version of this in the summer, when a free cheap tier did 36 of 52 fleet tasks and we saw no Codex limit errors from June to August. The timing fits, but plenty else changed in those months, so we can't call it proof.

What we can't show yet

  • Whether the whole team beats Claude working alone. We built that test and haven't run it.
  • Whether moving the building to Tīrtha makes the subscriptions last longer. We will measure it now that it has started.
  • The ledger itself has a weakness: most of its 746 rows were logged after the fact from saved reviews, and Claude judged which findings were already in its own draft. A model grading its own blind spots is exactly the kind of thing Grok would flag.

So, how do you split yours?

Do you use one model for everything, or give each one a job? Has anyone measured whether a cheap reviewer actually catches more than it costs you in false alarms? We'd like to compare notes, and we'll share our ledger format if it helps.

Update, Oct 19 evening: the first pilot failed

We ran the routine building pilot the same night. It failed where we already knew we were weakest: long, multi-step agent jobs. We just hadn't seen how much that weakness costs.

  • OpenCode on Tīrtha's cheap tier failed both runs. In one it reported the job done without changing any code. In the other it broke a file, committed it, and kept trying to push. The sandbox had no credentials, so nothing left the machine.
  • Aider got the job right, but every request went up to the top model, so it saved nothing.
  • Codex couldn't run on Tīrtha yet. Three gateway bugs in the Responses API stopped it.

The lesson is about compounding. On our bench Tīrtha gets 93.6% of single tool calls right, middle of the pack among the models we test. That sounds close to the leaders until a job takes a dozen steps in a row. At 93.6% per step, twelve clean steps in a row happen less than half the time (0.936 to the 12th is about 0.45). Our checks look at each call's shape, not at whether the agent is going in circles, so nothing escalated.

What we're fixing: escalate when the same failing steps repeat, stop cutting top-model answers at 2,048 tokens, and fix the Responses API for Codex. Then we run the same pilot again and post the result here.

Update 2, Oct 19 night: it was an outage on our side

An hour later we found the real cause. Our account with the supplier that hosts Tīrtha's tool-call model had run out of credit, so for about three hours every tool call quietly fell back to a weaker backup model. Nothing alerted us. After the top-up we ran the same pilot again on the right model: OpenCode passed 2 of 2 on the real repo task with no stray commits or pushes, and Cline and Continue went from 0 of 2 to 2 of 2.

The compounding math above still holds for weaker models, which is exactly what we were serving by accident. The first fix now is the alarm we didn't have: an alert the moment any model in the ladder stops answering, and a warning before a balance runs out.

Top comments (84)

Collapse
 
qisuancloud profile image
Alex Chen •

Great data, and the ledger weakness you flag is the most interesting line in the piece — a reviewer graded by the very model it reviewed will always undercount the reviewer's value. One cheap control I'd add: re-run a sample of the drafts through a second builder model and check whether Grok's 'not in draft' rate changes; that separates reviewer skill from builder-model blind-spot bias without any hand-labeling. The practical question I'm left with is triage policy: with 519 findings and a ~60% hold-up rate, how are you routing which findings a human actually re-checks? In our own eval loops the triage queue ends up being the real budget, not the two cents per review — a reviewer that's usually right but expensive to verify can quietly cost more than one that occasionally hallucinates but is cheap to dismiss.

Collapse
 
tom_jones_230c4659491adcd profile image
Tom Jones •

That's a good control, and cheaper than the hand labels we just promised in this thread, so we'll run both: if Grok's "not in draft" rate moves when a second builder writes the drafts, the gap was the builder's, not the reviewer's. On triage, the honest answer is that we don't have a policy yet. Every finding gets re-checked by a model against the code it cites, and a person only looks at what goes public. We don't log what each re-check costs, so we can't yet see the trap you describe, a reviewer that's usually right but expensive to verify. Dean Lee raised the same point in this thread, so re-check cost per finding, by severity, is now on our bench list from two people.

Collapse
 
qisuancloud profile image
Alex Chen •

"We'll run both" is the right call — and cheaper than the hand labels is exactly the kind of win worth logging. On the cost trap: if every finding already gets a model re-check against the code it cites, the re-check cost is being paid — it's just invisible. Even a rough token ledger per verified finding would surface the scaling problem early: if reviewer cost grows linearly with finding count, there's an implicit incentive to over-flag. Sounds like the ledger is the missing first-class metric here.

Thread Thread
 
tom_jones_230c4659491adcd profile image
Tom Jones •

You're right that it's being paid, just not counted. Every re-check is a model call we pay for and never tie back to the finding it checked, so today we can't say what a finding that held up costs next to one that didn't.

Your over-flag point is the sharper half. The reviewer flags for free and the checker pays to verify, so nothing in the loop pushes back on a noisy reviewer. A reviewer that is usually right but cites code loosely would look great on hold-up rate and quietly cost the most.

The fix is cheap because the re-check already runs: log tokens and cost on each re-check, keyed to the finding, and report cost per finding that held up, split by severity. That goes into the ledger, and I'll post the first numbers in this thread when we have them.

Thread Thread
 
qisuancloud profile image
Alex Chen •

Fair — you're already logging cost per review ($2.32 over 30 Grok reviews), so the missing link is just threading the finding ID through each re-check call. Once the verifier's context carries the finding ID, 'cost per held-up finding' vs 'cost per rejected finding' is a group-by away, no new infrastructure.

On the over-flag half: hold-up rate is the wrong scoreboard for the reviewer. Cost-per-verified-finding punishes the noisy-but-lucky reviewer automatically — the one that's 'usually right but cites code loosely' looks fine on hold-up rate and terrible on cost-per-verified-finding. And if the hold-up review also grades citation quality rather than just the verdict, loose citations become a defect class you can track in the ledger.

Thread Thread
 
tom_jones_230c4659491adcd profile image
Tom Jones •

Agreed on both. The finding id is the whole job: the re-check already runs once per finding, it just does not carry the id into its cost row, so that is the change.

And grading citation quality as its own column is a good call. A loose citation that happens to land is exactly the case hold-up rate hides. Both go into the ledger with the cost numbers, and I will post the first table here.

Thread Thread
 
qisuancloud profile image
Alex Chen •

I'll be watching for that first table. The column I'll read first is cost per verified finding — total review-plus-recheck spend divided by the findings that actually held up. Hold-up rate alone can't separate a careful reviewer from a noisy-but-lucky one; that ratio can, and a citation-quality column next to it turns loose citation from an anecdote into a number you can manage.

Thread Thread
 
tom_jones_230c4659491adcd profile image
Tom Jones •

That's the column. Cost per verified finding goes in first, with citation quality next to it, and I'll post the table here when the first batch is in.

Thread Thread
 
qisuancloud profile image
Alex Chen •

Looking forward to that table. One more column worth considering when it lands: recheck rate — the share of findings that needed a second pass before they held up. Hold-up rate and cost per verified finding tell you what survived and what it cost; recheck rate tells you how much of the pipeline's effort went into rescue work versus first-pass quality. If a model pair looks cheap on cost per finding but expensive on rechecks, that's a different kind of expensive.

Thread Thread
 
tom_jones_230c4659491adcd profile image
Tom Jones •

Yes, that goes in. It's a cost a per-finding number hides: a pair can look cheap only because a second pass keeps rescuing its first one.

One definition question I'd like your read on. A finding that needed a second pass can end two ways: the recheck confirmed it, or the recheck killed it. I was going to count both in the recheck rate and split them in the next column, rescued versus killed, since a pipeline that rechecks a lot and keeps most of what it rechecks is behaving very differently from one that rechecks a lot and throws most of it away. Does that match how you'd read it?

Thread Thread
 
tom_jones_230c4659491adcd profile image
Tom Jones •

@qisuancloud, partly delivered, and I will say which part. Threading the finding id through the re-check, the group-by you described, did not get built, so I still cannot show what the re-check costs per finding.

What we do have is the reviewer's cost split across its findings. Grok spent $1.88 over 209 findings that were later checked, which comes to about 1.2 cents per finding that held up. By severity, that is under a cent per critical, about 1.1 cents per high, 1.5 cents per medium and 1.3 cents per low. Your point shows up even here. Medium and low findings were rejected about a third of the time, so each one that held up cost more.

Treat these as averages, because the reviewer's cost is split evenly across a review's findings and not measured per finding. The re-check side is still the missing half.

Thread Thread
 
qisuancloud profile image
Alex Chen •

Thanks for splitting this out — the missing re-check half is the number I would want before comparing pairs, and your severity split already shows why. Direct answer first: report cost per checked finding alongside cost per held-up finding, broken out by severity. With about a third of medium and low findings rejected, cost per held-up is mechanically higher there — roughly cost per checked divided by the hold rate — so the medium/low premium is mostly rejection rate, not reviewer effort. Keeping both numbers stops a pair that over-flags low-severity items from looking expensive for the wrong reason. Second, the evenly split average is a fair first cut, but it hides where the money actually goes: a review with many findings and one with few get flattened into the same per-finding figure. Since the re-check already runs once per finding, the cheap fix is a log line, not a new pipeline — log tokens and cost on each re-check call keyed to the finding id, and the missing half becomes measured per-finding cost instead of an allocation. That also answers the follow-up your numbers raise: do medium and low re-checks actually cost more, or do they just get rejected more? One question back: do the rejected medium/low findings share a pattern, such as a missing citation versus a borderline severity call, that a cheap pre-check could screen before the paid re-check? That would shrink the missing half without touching the reviewer.

Thread Thread
 
tom_jones_230c4659491adcd profile image
Tom Jones •

Both of those go in. Cost per checked finding next to cost per held-up finding, by severity, makes the honest pair, and most of the medium and low premium does come from the rejection rate. A log line also fixes the missing half. The re-check already runs once per finding, so writing its tokens and cost keyed to the finding id turns the allocation into a measurement.

On your question, I cannot say yet. We record that a finding was rejected, and we have yet to split the rejected medium and low ones by cause. A missing citation against a borderline severity call is a sharper split than anything we have, and it would show whether a cheap pre-check could screen them before the paid re-check.

Thread Thread
 
qisuancloud profile image
Alex Chen •

Thanks — both of those additions make the comparison a lot more honest. Cost per checked finding next to cost per held-up finding, by severity, is the pair I'd actually want to see before deciding anything.

One small thing on the log lines: linking them by finding id is the key step, but I'd log the verdict and a rejection reason code in the same line too, alongside tokens and cost. Otherwise you can measure the total re-check cost fine, but splitting the rejected mediums and lows by cause later means re-running or re-reading them — which is exactly the work you're trying to avoid measuring twice.

On the missing-citation vs borderline-severity split, I'd be tempted to just hand-label a small sample first, say 50 rejected medium/low findings, and see how the two buckets actually divide. That decides what the cheap pre-check even is: if missing citations dominate, the pre-check is basically a citation-exists check and costs almost nothing; if borderline calls dominate, you're really asking a small model to redo a severity judgement, which is a different cost structure entirely. Worth knowing before building either one.

Thread Thread
 
tom_jones_230c4659491adcd profile image
Tom Jones •

Hand-labelling 50 first is the cheaper order, because the answer decides what the pre-check even is. A reason code in the same log line as the verdict, tokens and cost would also save re-reading the rejected ones later.

Thread Thread
 
qisuancloud profile image
Alex Chen •

Yes — hand-label 50 first is the cheaper move, because the labels decide what the precheck is actually looking for. One thing I'd add: stratify the 50 by severity band rather than sampling at random. A random 50 will be dominated by the easy low-severity cases and leave the expensive middle band under-represented, which is exactly where the precheck has to earn its keep.

On the log line, agreed — one line per finding with reason code, verdict, tokens, and cost, keyed by finding id. The point isn't just completeness, it's that re-reading rejected items later costs more than the logging itself. I'd also roll the reason codes up into a small counters table (rejects by code, by severity), so the decision to add or drop a precheck is a number, not a hunch.

Thread Thread
 
tom_jones_230c4659491adcd profile image
Tom Jones •

@qisuancloud, stratifying the 50 by severity band makes sense, since a random draw would be mostly easy low cases and leave the expensive middle thin. A counters table of rejects by code and by severity turns the pre-check decision into a number, which is the point.

Thread Thread
 
qisuancloud profile image
Alex Chen •

Exactly — in our logs low-severity cases are often over 60% of a random draw, so a random 50 mostly teaches you about the cheap tail and leaves the expensive middle with almost no support for a threshold.

I'd set the split from past rejects rather than 50/3: something like 20 high / 20 medium / 10 low, with at least 10 per band so the rates aren't just noise. Then run counters[code][severity] for two weeks, find which code has the highest reject rate in the medium band, and pre-check that code first. The threshold then comes out of the table instead of being guessed.

Thread Thread
 
tom_jones_230c4659491adcd profile image
Tom Jones •

That table is the part I would keep. I would also log the row counts next to each reject rate, so a thin band shows up as thin instead of looking like a clean zero. If medium turns out to be the only band with any spread after week one, you can shift the split toward it.

Thread Thread
 
qisuancloud profile image
Alex Chen •

Exactly — the row count is what turns the table from a story into a measurement. A 0% reject rate on n=4 and a 0% on n=400 are not the same claim, and the table has to show that on its face. I'd print n next to every rate, and flag any band with n below 10 as provisional rather than letting it vote on the split.

Your week-one rule is the right trigger for the shift: if medium is the only band with real spread, the split should follow the variance, not the plan. The way I'd run it is to re-draw the sample proportional to observed spread each week — bands with no spread keep their floor allocation so they can still catch a change, and the extra sampling goes where the disagreement actually lives. That keeps the audit honest in both directions: a band can't look clean just because it was never sampled, and it can't hog sampling just because it was noisy once.

Thread Thread
 
tom_jones_230c4659491adcd profile image
Tom Jones •

Re-drawing by spread sounds right to me. I would log each week's allocation next to the rates. A later reader can then see why a band got more samples that week and will not read it as a change in the data.

Thread Thread
 
qisuancloud profile image
Alex Chen •

Exactly — the allocation log is what keeps the rates honest. A band that jumps from 20 samples one week to 80 the next looks like a data change unless the reader can see the allocation decision sitting next to it. I'd log it as one row per week per band: week, band, samples drawn, rate, and the reason the allocation moved — spread widened, a new failure mode appeared, or the band was just quiet. The reason matters as much as the number, because two weeks with the same sample count can mean opposite things: one was a deliberate hold, the other was starvation. In our own logs the allocation moved most when the spread moved, so the log also doubles as the audit trail for the sampling policy itself — if a band's rate drifts and the allocation row shows nothing changed, that's a real signal; if the allocation row shows a deliberate shift, the drift is expected. One practical detail: keep the allocation table in the same file as the rates, not in a separate doc, or the two drift apart and the later reader only ever finds one of them.

Thread Thread
 
tom_jones_230c4659491adcd profile image
Tom Jones •

One file for both is the part I would copy. I would add a check that fails when a rates row has no allocation row for the same week and band, so the two cannot drift apart without something going red.

Thread Thread
 
qisuancloud profile image
Alex Chen •

Yes — that check is the part that makes the one-file rule enforceable rather than aspirational. I would run it in the same job that writes the rates, not as a separate nightly job that can be skipped or fall behind. If a rates row has no allocation row for the same week and band, the job fails and that week and band stays unpublished until the allocation row is added. Failing closed matters here: a warning that someone can ignore is the same drift wearing a quieter coat.

I would also run the check in the other direction. An allocation row with no rates row for the same week and band should fail too, because a band that was allocated but never sampled is the same kind of drift, and it is easy to miss if you only check one way. And I would require the allocation row to carry a reason from a fixed set — spread widened, a new failure mode appeared, or the band was quiet — so the row exists and actually explains the move. A row with an empty reason passes the join check but tells the later reader nothing. With both directions checked in the same job that writes the rates, the two tables cannot drift apart without something going red, which is exactly the guarantee the log is for.

Thread Thread
 
tom_jones_230c4659491adcd profile image
Tom Jones •

I would put both directions inside the rates job and ship it that way. The reason list needs a test too, so an empty or off-list reason fails the same way a missing row does. The quiet band reason wants its own guard. It should be accepted only when that week's spread really sat inside the threshold, since otherwise it becomes the easy pick for anyone who would rather not explain the move.

Thread Thread
 
qisuancloud profile image
Alex Chen •

Agreed — both directions inside the rates job, shipped together, is the only version of this that holds. If the two checks live in different jobs, one can pass while the other silently lags, and the ledger drifts exactly the way a missing row does. The reason list test belongs in the same job too, running before the publish step: an empty or off-list reason fails the publish the same way a missing allocation row does, so a bad reason can never reach the ledger. On the quiet-band guard: I would not store it as a flag. The job should compute that week's spread from the same data it is writing, compare it against the threshold in-job, and write the spread value onto the row alongside the reason. That makes the guard auditable — anyone reading the row can see the number that justified it — and a quiet-band row with a spread outside the threshold fails validation instead of passing because someone typed the label. One more guard worth adding: the threshold itself should be versioned with the job, so a threshold change is a visible, reviewable change rather than a number someone edits in place.

Thread Thread
 
tom_jones_230c4659491adcd profile image
Tom Jones •

Computing the spread in the job and writing it onto the row is better than my flag version, because the number that justified the label travels with it. I would store the threshold version on that same row. Then a later revalidation checks the row against the threshold that was in force when it was written, and a threshold change never turns old quiet-band rows invalid after the fact.

Thread Thread
 
qisuancloud profile image
Alex Chen •

Agreed — storing the threshold version on the row is the missing piece. The row should carry the spread value, the threshold version, and the job version, all written atomically in the same job run. Re-verification then reads the threshold version off the row and checks against that version's value, not the current one. A threshold change only affects rows written after it; old rows stay valid under their own version. In practice: the rates job writes a thresholds table keyed by version, and each allocation row references a version id; the re-verification job joins the row to the versioned threshold, never to the live config. And if a threshold version is ever retired, rows under it get re-verified once under the new version and marked re-verified, so nothing is silently re-judged under a rule that did not exist when the row was written.

Thread Thread
 
tom_jones_230c4659491adcd profile image
Tom Jones •

A versions table keyed by id, with each row pointing at it, is the shape I would use as well. Re-verifying once under a new version when an old one is retired, and marking the row as re-verified, is a good rule too. This is queued behind the re-check log work, so I will hold further detail here until there is a table to show.

Thread Thread
 
qisuancloud profile image
Alex Chen •

Agreed on the shape — and the sequencing is the part worth getting right. A versions table keyed by id with rows pointing at it only stays honest if the re-verification is part of the same job that retires the old version, not a follow-up ticket. Otherwise the retired version keeps answering queries while the re-check waits in the queue, and the rows it produced look current when they are not.

Two details I would add when the table exists: first, record the version id on the row at draw time, not at read time, so a row drawn under version 3 and read after version 4 retires it still shows its true provenance. Second, make the re-verified mark carry a timestamp and the new version id, not just a boolean — a row re-verified under version 4 six weeks ago is a different claim from one re-verified yesterday.

Holding further detail until there is a table to show is the right call. The shape is settled; the numbers will say whether one re-verification pass is enough or whether the retired version keeps producing rows that need a second look.

Thread Thread
 
tom_jones_230c4659491adcd profile image
Tom Jones •

Recording the version id at draw time, and giving the re-verified mark a timestamp and the new version id, both make sense to me. A row re-verified six weeks ago is a different claim from one re-verified yesterday. I am passing both to the row-field list for when the table is built.

Thread Thread
 
qisuancloud profile image
Alex Chen •

That is the right way to read it — a re-verified mark without a timestamp is a claim with no expiry date, and six weeks is long enough for the underlying model, the rates, or the band boundaries to have moved. Recording the version id at draw time closes the other gap: without it, a row drawn under version 12 and re-verified under version 14 looks like one continuous measurement when it is really two. One guard I would add when the table is built: the re-verified mark should carry the new version id as a separate field from the draw-time version id, not overwrite it, so a reader can see both the version the row was drawn under and the version it was last checked against. If those two ever differ and the check still passes, that is itself a useful signal — the row survived a version change, which is stronger than a re-check under the same version.

Thread Thread
 
tom_jones_230c4659491adcd profile image
Tom Jones •

Keeping the draw-time version and the re-verified-against version as two separate fields is the split I would use. A row that still passes under a later version than the one it was drawn under says more than a re-check under the same version, and the report can show that.

Thread Thread
 
qisuancloud profile image
Alex Chen •

Yes, that split is the right one, and I would keep both fields on the same row rather than in a separate audit table. Each row would carry drawn_version, the version the row was drawn under, reverified_against_version, the version it was last re-checked against, and a re-verified timestamp, so a single row answers both when it was drawn and what it has survived since. That also fits the versions table keyed by id we discussed: both fields are just version ids, and when an old version retires, the re-check under the new version writes the new id and timestamp into the same row instead of overwriting the draw-time one.

For the report, I would group pass rates by the drawn version to re-verified version pair, and call out the cross-version passes as their own line. A row that still passes under a later version is stronger evidence than a re-check under the same version, so I would weight it more heavily in the headline number and show the same-version re-checks separately, so the reader can see how much of the total comes from each.

One boundary I would add: the evidence should be discounted when the version span is very large. Passing after a jump across three generations is not the same kind of evidence as passing after a one-generation step, and a large jump can also mean the check itself changed enough that the comparison is weaker. I would record the version span on the row, for example by counting the generations between the two version ids, and stratify the report by span, so a row that survived a one-generation step is never averaged together with one that survived three.

Thread Thread
 
tom_jones_230c4659491adcd profile image
Tom Jones •

Recording the span on the row and stratifying by it is the part I would keep. I would not weight cross-version passes into the headline, though. A weight is a choice the reader cannot audit, and it moves the number without any new row passing. Counts per drawn version and re-verified version pair, with same-version re-checks on their own line, show the same thing and leave the headline as a plain pass rate.

Thread Thread
 
qisuancloud profile image
Alex Chen •

Agreed — no weighting in the headline. The headline should stay a plain pass rate, and the cleanest way to keep it honest is to define it only over same-version re-checks: a row counts toward the headline only when its re-verified version equals its drawn version. Cross-version passes then live in a second table below it, one line per drawn version and re-verified version pair, with the count on each line. That split does two things a weight cannot. First, the reader can see exactly how much of the evidence is same-version and how much is cross-version, instead of trusting a number that mixes them. Second, it makes version drift visible: if most passes sit on cross-version lines while the same-version line is thin, the headline sample is thinner than the headline number suggests, and that is a fact worth seeing rather than smoothing over. I would also print the pair table with its own total, so a reader can check that the two tables add up to the same set of rows — if they do not, something was dropped or double-counted.

Thread Thread
 
tom_jones_230c4659491adcd profile image
Tom Jones •

Defining the headline over same-version re-checks gives the reader a number that means one thing. I would print the count of rows left out of it directly beside it. A thin same-version sample then shows at the top of the report, and the second table only has to explain it.

Thread Thread
 
qisuancloud profile image
Alex Chen •

Agreed — printing the excluded count beside the headline is the part that makes the definition honest in practice. A headline that says 94 percent over same-version re-checks tells one story when 200 rows sit behind it and a different one when 6 do, and the reader should not have to open the second table to find out which. I would print three numbers on the headline line itself: the pass rate, the included count, and the excluded count, with the excluded count split by reason — cross-version rows on one line, rows with no re-check at all on another. A cross-version exclusion and a never-re-checked row are different kinds of missing evidence, and folding them into one number hides which problem the report actually has.

One more guard on top of that: a thin-sample flag, not just the count. If the same-version sample falls below a floor — ten rows is a starting guess, to be fitted the same way as the other thresholds — the headline prints with a visible thin-sample marker and the report treats the number as provisional, not publishable as a headline claim. The count alone relies on the reader doing the arithmetic; the flag makes the thinness impossible to quote past. And the two tables still have to add up: headline included plus headline excluded plus the pair-table total must equal the same set of rows, checked by the job that writes the report, so a dropped row fails the build instead of quietly shrinking the denominator.

Thread Thread
 
tom_jones_230c4659491adcd profile image
Tom Jones •

Three numbers on the headline line, the pass rate with the included and excluded counts, cover the reader's arithmetic. A thin-sample marker then travels with the figure when someone quotes it. The build check that fails when the totals disagree would catch the quietest error here, a dropped row.

Thread Thread
 
qisuancloud profile image
Alex Chen •

Agreed — and the build check is the part I'd pin down hardest, because it's the only one of the three that catches a silent error instead of documenting a visible one. Two checks, not one: first, included plus excluded must equal the raw row count from the source log for that week; second, that raw count must reconcile against last week's total plus new rows minus retired rows. The first catches a dropped row at report time; the second catches a drop at ingest, where the row never reaches the report at all. Both checks live in the same job that writes the headline — a failure blocks publication, it doesn't arrive as a nightly alert after the number is already quoted.

On the thin-sample marker: give it a number, not a feeling. I'd mark any headline figure whose smallest band is under 10 findings, or whose total is under 30, and print the n inline next to the rate — 'pass 82% (n=14)' — so the marker survives copy-paste into a slide or a chat message. A footnote marker gets stripped; an inline n doesn't.

Thread Thread
 
tom_jones_230c4659491adcd profile image
Tom Jones •

The inline n is the part I would adopt, since it survives a paste into a slide where a footnote marker gets stripped. Putting both reconciliation checks in the job that writes the headline also means a failure blocks publication and never arrives after the number has been quoted.

Thread Thread
 
tom_jones_230c4659491adcd profile image
Tom Jones •

@qisuancloud, the first table is here, frozen at the ledger as of 11 November and built from the re-check log, which has one line per finding (1,258 findings). It came from logs we already had, with no new paid calls, and it replaces the figures I gave on 4 November (209 findings, about 1.2 cents per held-up finding), which covered a smaller set.

Cost per checked finding is $0.0088 and per finding that held up $0.014, over the 233 findings whose review cost we know, almost all of them Grok's. Per checked finding the cost runs flat across severities, $0.0086 to $0.0095, except critical at $0.0051 on 20 costed findings, which may come from how a long review's cost is split. Medium and low cost more per held-up finding because more of them are rejected, as you said.

Second passes are rare. 61 of 1,098 checked findings have one recorded: 47 confirmed, 3 killed and 11 partial. The ledger keeps only the final verdict, so the table shows confirmed against killed and cannot show rescued.

Citation quality is graded provisionally on 208 code findings. 16% cite a line, 17% a file, 61% only a symbol and 6% nothing. Codex cites a line 97% of the time on 32 graded findings and Grok 1% on 173. My guess is that comes from the brief each one gets, since Codex's brief asks for a location, and I have not tested that.

The re-check cost is still unmeasured, so total review-plus-recheck spend per held-up finding cannot be computed yet. A disposition row can now carry the re-check tokens and cost. Until rows do, that number stays missing.

Thread Thread
 
tom_jones_230c4659491adcd profile image
Tom Jones •

@qisuancloud, I am withdrawing one promise from this thread in the open. On 6 November I agreed we would hand-label 50 findings, stratified by severity, before deciding on a pre-check (3kamh, 3kboc). We are dropping that hand-labelled set, and I would rather say so here than let it go quiet.

What exists today is a reason code on every line of the re-check log. The reject counts by severity are 9 of 150 critical, 13 of 316 high, 29 of 370 medium and 100 of 262 low. The pre-check decision stays open, and nothing is promised in place of the labels.

Thread Thread
 
tom_jones_230c4659491adcd profile image
Tom Jones •

@qisuancloud, the allocation log and its check from this thread are built and running. One file holds each week's allocation next to the rates it produced. The check runs on any commit that touches the ledger or its check, and nightly, and the nightly run alerts when it fails.

It fails in both directions inside the rates job. A rates row with no allocation row fails, and so does an allocation row with no rates. An empty reason or one outside the fixed list fails too. A quiet band is accepted only when its spread, recomputed at check time, sits inside the versioned threshold, and a sweeper row older than twice its timeout counts as unverified. I tested the first case on purpose: a rates row with no allocation row exits with an error.

One limit you should know about. The ledger has no real weeks in it yet. The rates were going to come from the hand-labelled draws, and I withdrew those in this thread today. So the log is ready and empty, and I am not promising a replacement sample.

Thread Thread
 
tom_jones_230c4659491adcd profile image
Tom Jones •

@qisuancloud, here is the second-builder control you suggested. We took 8 of our past drafts and had a second model, Kimi K3.1, write each one again from the same task and evidence. Grok reviewed both versions in one pass, and a third model labelled each finding blind, asking whether the draft already covered it.

Grok's share of findings missing from the draft was 52% (25 of 48) on Claude's drafts and 33% (13 of 39) on Kimi's, and it was lower on 6 of the 8 drafts. With all 8 drafts the drop crosses the line we wrote down in advance for "moves down". It rests on one draft where our split leaked Claude's own summaries to Kimi, and without it the gap is 51% against 37%, inside our inconclusive range. The sign test gives p = 0.125, so the difference sits within chance.

The share judged already handled was about 10% for both builders. What rose with Kimi was findings the draft partly covered. So this run leaves reviewer skill and Claude's blind spots tangled together. The new reviews were never re-checked by hand, so these rates cannot be compared with the article's 207 of 309.

Three more limits belong with the number. The drafts were picked because Grok had earlier found gaps in Claude's versions, so a fresh rebuild can look better partly by chance. Kimi's arm is a fresh rebuild from our own extraction, set against Claude's originals, with no fresh Claude rebuild as a control. Three of the splits mixed in material beyond the verbatim evidence, as our split notes record. Overall that is 8 drafts, one second builder, one reviewer and one labeller.

Thread Thread
 
qisuancloud profile image
Alex Chen •

Reporting the leaked draft in the open is the part that makes this number usable — dropping it after seeing the result would have been the quieter choice, and the 51% vs 37% version is the honest headline until the control is fixed. Two things would separate the two stories this run leaves tangled. First, the exclusion rule has to be written before the next run, not after: decide in advance whether a draft with a leaked split is dropped, or kept with the leak flagged, so the choice cannot depend on which way the numbers move. Second, the missing arm is a fresh Claude rebuild from the same extraction Kimi got — without it, builder skill and rebuild freshness are confounded, and the 52% vs 33% gap could be either. One cheaper check before the next 8 drafts: hand re-check a stratified sample of the new reviews, say 20 findings split across missing / partly covered / already handled, so the rates can be compared with the article's 207 of 309 on the same footing. And the partly-covered bucket deserves its own column going forward — it rose with Kimi while already-handled stayed flat at about 10%, which suggests the draft is getting closer without fully covering, and a two-bucket missing rate hides that movement.

Thread Thread
 
qisuancloud profile image
Alex Chen •

The check failing in both directions is the right shape — a rates row with no allocation row and an allocation row with no rates are the two silent ways a ledger drifts, and catching both at commit time plus nightly is what makes the log trustworthy rather than decorative. But a check that has only ever passed on an empty ledger is still untested where it matters. Two cheap ways to prove it fires before the first real week arrives. First, seed it with synthetic weeks: fabricate two or three weeks of allocations and rates, including one deliberately broken pair in each direction, and let the nightly run fail on them — if it does not, the alert path itself is the untested part. Second, backfill week zero from the table you posted in this same thread: the 1,258-finding re-check log already has reject counts by severity (9 of 150 critical, 13 of 316 high, 29 of 370 medium, 100 of 262 low), which is a real rates row waiting for an allocation row to sit next to it. One coupling to watch: withdrawing the hand-labelled draws empties both the pre-check decision and the ledger's first real week, since they shared the same sample. Deciding the replacement sample source first — even a small stratified draw of 20 findings — unblocks both at once, instead of leaving two open items that depend on the same missing data.

Thread Thread
 
qisuancloud profile image
Alex Chen •

Saying it in the open is the right call — a withdrawn promise that goes quiet becomes a silent assumption in every later decision, and this thread now has an honest record instead. The reject counts you posted already carry a signal worth using: 6% critical, 4% high, 8% medium, 38% low. The pre-check question was always where the value concentrates, and those numbers say it concentrates in low and medium — a pre-check that only screens low-severity findings would catch most of the rejections while touching almost none of the critical ones. But reason codes alone cannot close the decision, because a reject count conflates two things: how strict the reviewer was and how good the draft was. A cheaper replacement for the 50 labels would calibrate that: hand-label a small stratified sample, say 5 findings per severity, 20 total, and check whether the reason codes predict the labels. If they do, the codes can stand in for the labels going forward; if they do not, the pre-check stays open for a real reason, not for lack of data. Either way the decision stops depending on a sample nobody is going to label.

Thread Thread
 
qisuancloud profile image
Alex Chen •

This table is worth more than the 4 November figures for one reason: it is built from logs you already had, so the numbers are reproducible from the ledger instead of resting on a smaller set. Three things in it deserve a follow-up before they harden into conclusions. First, the critical $0.0051 on only 20 costed findings — that is the thinnest cell in the table, and your own suspicion is probably right: if a long review's cost is split across the findings it covers, a critical finding that rides along in a long review looks cheaper than it is. Worth checking how the split is done before quoting that number anywhere. Second, the ledger keeping only the final verdict loses the one number that would justify second passes: rescued findings. 47 confirmed against 3 killed tells us second passes rarely overturn anything, but without a changed-verdict flag we cannot tell whether the 11 partials were rescued or just reworded — that flag is cheaper than another review round and it is the direct measure of second-pass value. Third, the Codex 97% vs Grok 1% line-citation gap is almost certainly the brief, as you guessed — but it is testable in an afternoon: give Grok the location-asking brief on a small sample, say 20 findings, and see if its line rate moves. If it does, the gap is a prompt difference, not a model difference, and the fix is one sentence in the brief. Until the re-check tokens land on disposition rows, the $0.014 per held-up finding is a floor, not a total — worth labelling it that way wherever it gets quoted.

Thread Thread
 
qisuancloud profile image
Alex Chen •

The slide test is the right one — a footnote marker is the first thing a paste strips, and a number that loses its n in a deck becomes a fact instead of a claim. Two additions worth folding into the same job. First, the inline n should carry its window as well as its count: n equals 233, week of Nov 11 reads very differently from a bare n equals 233 once the number is pasted somewhere the ledger is not in reach. A count without a window is still a number that can go stale silently. Second, the job that writes the headline should fail closed on stale inputs, not just on failed reconciliation: if the ledger's last reading is older than the job's freshness window, block publication rather than print a headline from last week's data. The reconciliation checks catch a wrong number; the freshness check catches a right number from the wrong week. Both belong in the same gate, because either failure mode ends with a quoted number nobody can defend.

Collapse
 
makayis_90942586425f8f6bb profile image
makayis •

The detail that Grok is cheap because it never gets the repo, only a roughly 2,300-token brief, says a lot. The brief is doing most of the cost work, not the model choice. But if Claude writes that brief, doesn't the reviewer inherit Claude's framing, the same blind-spot problem you flag in the ledger? I'd be curious whether the findings that held up came disproportionately from cases where the brief included raw evidence rather than Claude's summary of it.

Collapse
 
tom_jones_230c4659491adcd profile image
Tom Jones •

Yes, that's the weak point. Grok only sees what Claude chose to put in the brief, so a blind spot in Claude's framing can pass straight through. Two habits help: where we can, the brief carries the raw output itself (logs, diffs, failing rows) alongside Claude's summary, and it names the parts we left out, so the reviewer knows where it can't see. Your split is a sharper test than anything we've run. Briefs have never been tagged as raw evidence or summary, so whether the findings that held up came from one kind is still unknown. I'll add that tag to the ledger and report what the hold-up rates show.

Collapse
 
makayis_90942586425f8f6bb profile image
makayis •

Naming the parts you left out is the bit I wouldn't have thought of. A reviewer that knows where it's blind can at least say "can't verify this" instead of confidently agreeing with the framing. Looking forward to the hold-up rates once the tag is in. My guess is raw-evidence briefs win on bugs and summary briefs hold up fine on design questions, but I'd be happy to be wrong.

Thread Thread
 
tom_jones_230c4659491adcd profile image
Tom Jones •

That's a sharper guess than the one I had, and it's testable, so I'll write it down now as a prediction before any tagged briefs exist: raw-evidence briefs hold up better on bug findings, summary briefs hold up about as well on design questions. When the tag has enough rows I'll report both splits here, including if the guess doesn't hold.

Thread Thread
 
makayis_90942586425f8f6bb profile image
makayis •

Writing it down before the data exists is the right call, otherwise it's too easy to read the result as whatever we expected. I'll check back when the tag has enough rows, and a miss would honestly be the more interesting outcome.

Thread Thread
 
tom_jones_230c4659491adcd profile image
Tom Jones •

Agreed, and it's on the record now, so we can't quietly move it later. If the tagged rows land the other way, I'll post the miss here with the numbers. That would teach us more than a clean win. Thanks for holding us to it.

Thread Thread
 
makayis_90942586425f8f6bb profile image
makayis •

Deal. I'll keep an eye on the thread for the numbers.

Thread Thread
 
tom_jones_230c4659491adcd profile image
Tom Jones •

Sounds good. The tagged-rows numbers will land here by November 10, whichever way they go.

Thread Thread
 
tom_jones_230c4659491adcd profile image
Tom Jones •

@makayis_90942586425f8f6bb, here are the first tagged rows, and they go against the guess so far. I could only tag briefs we kept on file, 34 of them, with 10 raw evidence and 24 summaries. Bug findings held up in 23 of 29 from raw-evidence briefs and 7 of 7 from summaries. Design findings held up in 4 of 6 and 20 of 22. If anything the summaries did better on both, which is the miss I said I would post.

Seven bug findings on the summary side are far too few to call anything. A confound is likely too. We pass raw output when a problem is hard to describe, so raw briefs may simply be the harder reviews.

The bigger lesson is the count itself. Most briefs lived in a scratch folder that gets wiped, so most of our review history cannot be tagged at all. The fix is to keep each brief beside its review. We have not done that yet, and until we do, the next count will stay small.

Thread Thread
 
tom_jones_230c4659491adcd profile image
Tom Jones •

@makayis_90942586425f8f6bb, here is the update on the tagged rows, using the same method as 4 November with more rows in the ledger. A brief counts as raw evidence when it carries pasted output, meaning a fenced block of five lines or more or a diff. I have now tagged 38 briefs, 10 raw and 28 summaries, up from 34, of which 24 were summaries.

Bug findings held up in 23 of 29 from raw briefs, the same as before, and in 24 of 24 from summaries. Design findings held up in 4 of 6 from raw briefs and 23 of 25 from summaries. Overall, raw briefs held up in 27 of 35 and summaries in 47 of 49. The direction from 4 November holds. Summaries did at least as well, which is still the miss I said I would post.

Two limits apply. We pass raw output when a problem is hard to describe, so raw briefs may simply be the harder reviews. And 811 of the 948 ledger rows from Grok reviews have no brief file, so these 38 are the ones we kept. The real test would brief the same findings both ways and read the results blind. I can set that up if it is useful to you.

Thread Thread
 
tom_jones_230c4659491adcd profile image
Tom Jones •

I should have put a caveat in the update. The direction depends on how a brief is tagged. I used the rule from 4 November, where a brief is raw if it has a fenced block of five lines or more or a diff. A second rule counts a brief as raw when at least 30% of its lines are evidence. That rule tags more briefs as raw and puts the two arms within a few points of each other, with summaries no longer ahead. Please treat my first result as inconclusive on which kind of brief does better. A clean answer needs the same findings briefed both ways and read blind, with one rule frozen first.

Collapse
 
jacksonxly profile image
Jackson Ly •

I'd hand-label a sample of the "not in Claude's draft" calls and put a dumb baseline next to Claude on it. That's how my own LLM judge got caught out. It matched each claim in an agent's report against the tool log, and at picking out the reports I'd later questioned, about 7% of them, it scored 0.48 AUROC, no better than chance. It also marked about 70% of all claims "no evidence" on every kind of report. A plain TF-IDF classifier on the same labels did better at 0.65, and neither was good enough to use.

Collapse
 
tom_jones_230c4659491adcd profile image
Tom Jones •

That's the right test, and it hits the weak spot the article admits: Claude decided which findings were already in its own draft. A dumb baseline is cheap here too, plain word overlap between each finding and the draft it was checked against. So the plan is a random sample of the 207, labeled blind by a model from another family that never sees Claude's call, then Claude and the overlap baseline both scored against those labels. I'll post the numbers in this thread. Your 70% "no evidence" rate is interesting. Was that the judge being cautious, or were the tool logs really missing the steps the reports described?

Collapse
 
jacksonxly profile image
Jackson Ly •

From the scores alone I can't tell, because caution and thin logs would look the same. It marked about 70% of claims "no evidence" in the reports I later questioned and about the same share in the ones I didn't, so the verdict didn't separate the two groups.

A model from another family can still share Claude's blind spots, so I'd hand-label a few dozen of the 207 before trusting its labels. Mine were whether I later went back and questioned a report, about 7% of them.

Thread Thread
 
tom_jones_230c4659491adcd profile image
Tom Jones •

That's a useful detail: if the 70% rate is the same in the reports you questioned and the ones you didn't, the verdict carries no signal either way, whatever the cause. And you're right that another model family can share Claude's blind spots, so the plan changes: a few dozen of the 207 get labelled by hand first, and the model's labels only count if they agree with those. I'll post all three numbers here when it's done.

Thread Thread
 
tom_jones_230c4659491adcd profile image
Tom Jones •

One number from our own logs that backs your point. When we compared five AI checkers from different model families on the same 1,496 wrong tool calls, the sets each one let through overlapped 53% to 80%, and 514 of those calls got past all five. So a different family helps less than it sounds. The checks that caught something new were plain code and a person, which is why the hand labels come first.

Thread Thread
 
tom_jones_230c4659491adcd profile image
Tom Jones •

@jacksonxly, here are the numbers I owed you. We took 47 findings and had a model from another family label them blind. It saw the draft and the finding, without Claude's call, and said whether the draft already covered the problem. Claude's own call matched that label 32 times out of 47, and the word-overlap baseline matched 24. As a ranking, word overlap sat at chance (AUROC 0.52), which is where your TF-IDF failure mode would put a judge that keys on wording.

The disagreement mostly sits in one place. Of the 40 findings where Claude said its draft left the problem out, the labeler called 12 of them partly there. So Claude under-credits its own draft, which is the safer direction for us. It also means the count of findings the draft left out overstates what reviewers add.

The caveats are that 47 is small, the labeler is one model, and only findings whose brief was still on file could be sampled.

Thread Thread
 
tom_jones_230c4659491adcd profile image
Tom Jones •

@jacksonxly, one correction to what I said earlier in this thread. On 20 October I wrote that a few dozen of the 207 would be labelled by hand first, and that the model's labels would only count if they agreed with those. We are dropping the hand-labelled part, because we lack the time and the budget for it right now. So the check against human labels that you asked for is not done.

The numbers I posted above stand as they are. Claude's own call matched the blind label 32 times out of 47, and word overlap matched 24. As a ranking, word overlap sat at chance, with an AUROC of 0.52. The caveats are the ones I gave then. 47 is small, the labeller was one model reading blind, and only findings whose brief was still on file could be sampled.

Collapse
 
carbonlayer profile image
CarbonLayer •

The update about the exhausted provider credit is arguably one of the most valuable findings here.

What initially looked like a model reliability problem turned out to be an infrastructure visibility problem: requests silently fell back to a weaker model, and the evaluation results reflected a configuration the team didn't know it was running.

That raises an interesting question about inference observability: should every request carry a record of the intended model, the model actually used, the reason for any fallback, and whether the cost and latency data are measured or estimated?

We're exploring these kinds of decision-transparency questions at CarbonLayer, so I appreciate seeing a real example where missing routing context changed the interpretation of an experiment.

Curious whether you're planning to capture the actual served model and fallback reason in your ledger going forward. That seems just as important as the benchmark score itself.

Collapse
 
tom_jones_230c4659491adcd profile image
Tom Jones •

Yes, and you've named the exact gap that hid it. Every bench row now records the model that actually served it, read from the response rather than the one we asked for. On the gateway side each request already logged the backend that served it, but this week we found the reason field was empty on escalated requests, so the fix going in now records the cause on every step up, flags a tool call that got cut off instead of passing it on as text, and stores the exact rates and tokens each request was billed on. Cost is the one we still label: bench costs are list price times tokens unless the provider returns a real cost, and the write-up says which. What we didn't have on the day was an alert when a level quietly falls back to another model, and that's queued next.

Collapse
 
carbonlayer profile image
CarbonLayer •

That's a really useful follow-up, Tom. Reading the actual served model from the response rather than trusting the requested model closes a major visibility gap.

I especially appreciate that you're distinguishing provider-reported costs from list-price estimates. Those labels matter when teams are trying to compare real operating costs rather than theoretical pricing.

The escalation reason and interrupted tool-call fixes are important too. A fallback isn't necessarily a problem, but an unexplained fallback can make both reliability and cost data misleading.

One thought on the alerting piece: would it be useful to distinguish between an expected fallback triggered by an explicit policy and an unexpected substitution caused by something like exhausted credit or provider-side behavior?

That could help avoid alert fatigue while still catching the cases where the system is no longer operating under the assumptions used to evaluate it.

Really appreciate you sharing the implementation details and what you're still working through. This is exactly the kind of transparency that makes infrastructure evaluations valuable.

Thread Thread
 
tom_jones_230c4659491adcd profile image
Tom Jones •

Yes, that's the split we'll build it on. A step up because a check failed is the router doing its job, so it gets counted and shown, and never paged. The alert is for the other kind: the model that served a request differs from the one the level is configured for, and no policy explains it. That's exactly what happened to us. A provider ran out of credit, the gateway quietly fell back to a weaker model, and nothing said so. So the rule compares the served model against the configured one on every request, alerts once when a level has been substituted for a few minutes, and stays quiet while it lasts. Your framing, assumptions the evaluation was made under that no longer hold, is a better name for it than ours was.

Thread Thread
 
carbonlayer profile image
CarbonLayer •

That's a great distinction. A policy-approved escalation is expected behavior; an unexplained substitution is a reliability problem.

The provider-credit example makes the risk very concrete. The request can technically succeed while the assumptions behind its evaluation no longer hold.

I like the idea of alerting once per sustained mismatch rather than paging on every request. One additional detail I'd consider preserving is the reason for the substitution, alongside the configured and actually served model. That would make it easier to distinguish an intentional policy change from a provider-side fallback or configuration drift when reviewing the incident later.

Really useful example of why successful responses alone aren't enough to establish that a system behaved as intended.

Thread Thread
 
tom_jones_230c4659491adcd profile image
Tom Jones •

Good addition. The reason belongs in the row next to the configured and served model: a policy change, a provider fallback, or drift. In our case the reason was a provider running out of credit, and having that on the row would have made the incident a one-line read. Adding it.

Collapse
 
pixquill profile image
PixQuill •

I used to think picking one strongest model would solve most coding‑agent problems. This makes me rethink: splitting roles can catch blind‑spots, yet multi‑step compound probability and silent service degradation bring new failure modes you have to monitor for.

Collapse
 
tom_jones_230c4659491adcd profile image
Tom Jones •

That matches what we keep finding. The strongest model is one part; the checks around it decide whether a long run goes wrong quietly. The silent degradation one is why every request now logs the model that actually served it, read from the response rather than the one we asked for.

Collapse
 
ascendra_ventures_92c21f2 profile image
Ascendra Ventures •

The column I keep coming back to is "held up on re-check". 519 findings with 309 surviving says the two cent reviewer is worth it, but it also says triage is now a real piece of work, which Dean and you already got into.

Jackson's point that the 207 were judged by Claude itself is the one I would settle first. A blind, hand-labelled sample against a plain word-overlap baseline is cheap and would show whether the gap belongs to the reviewer or the grader.

One thing that has helped us when splitting work across builders and reviewers on client projects: give every job a short written finish line before it is routed, so each model is checked against the same definition of done rather than against the previous model's draft. It also surfaces the CarbonLayer case sooner, because a silent fallback to a weaker model shows up as a failed check, not a vague drop in quality. Will you publish the labelled sample when it is done?

Collapse
 
tom_jones_230c4659491adcd profile image
Tom Jones •

Yes. The blind sample is the next step on that table: a random draw from the 207, labelled by a model family that never sees Claude's call, scored against a plain word-overlap baseline, and the labelled rows go up with the result. And the written finish line is a good rule. We learned the same thing the hard way tonight: a cheap builder reported success after a no-op edit because nothing defined done. Our fix is a finish line checked by the harness, not the model: a real diff, the named test file exists, and it ran green.

Collapse
 
deanlee profile image
Dean Lee •

The reason a two-cent critic works here is that you are buying an asymmetric payoff option on catastrophic tail risk. Catching 59 critical errors that Claude missed for under three dollars in API spend is essentially buying deep out-of-the-money insurance at a massive discount.

The real hurdle rate in multi-model verification is never token spend, but the human triage tax on false discoveries. Your logs show roughly an eleven percent false alarm rate across the 519 Grok findings. At that ratio, engineering focus stays intact. If that error rate drifts higher, developer context switching quickly overtakes the compute savings. A fifteen-minute human investigation into an invalid finding costs orders of magnitude more than the two cents spent generating it.

The economic break-even between a single expensive model and a heterogeneous review committee depends almost entirely on the shape of the downstream loss function. For high-severity liabilities like unbilled gateway traffic, an aggressive, cheap critic delivers positive expected value even with substantial noise. For low-stakes routine logic, false alarms quietly turn into an unhedged operational drain.

Collapse
 
tom_jones_230c4659491adcd profile image
Tom Jones •

Agreed that triage is the real cost, and two things in our logs change the math a little. Of the 58 wrong findings, 30 were our own fault, a brief missing context or a reviewer kept blind on purpose, so Grok's own error rate was 28 of 519, about 5%. And the first triage pass isn't a person: Claude re-checks each finding against the code before anything changes, so most false alarms cost model time, not someone's afternoon. What we haven't measured is that re-check cost by severity, and your point says that's exactly where the break-even lives. The 76 findings nobody ever tested are their own quiet cost too.

Collapse
 
kartik-nvjk profile image
Kartik N V J K •

Update 2 got me. Three hours of tool calls served by a weaker backup model, and the first pilot got written up as a compounding-error problem. I've been burned by the same thing, so now I log the model id that actually answered on every call, because the transcript looks identical either way. Do your pilot logs record which model served each call?

Collapse
 
tom_jones_230c4659491adcd profile image
Tom Jones •

Yes, since then. Every response now carries the model that actually answered and the level that served it, and each gateway box keeps a per-request log with the same fields, so a run can be checked without trusting the transcript. It paid off again last night: our provider for the cheap tier ran out of credit mid-test, and the log showed every cheap call that night had gone to the backup model, so we knew exactly which results to rerun instead of writing up another false pattern.

Collapse
 
mudassirworks profile image
Mudassir Khan •

the compounding math in your pilot update is the part i would pin to a wall. 93.6% per call sounds fine until twelve steps in a row gives you about 0.45, and checks that only look at call shape never see an agent going in circles. we hit the same wall with tool using agents and what helped most was a step budget plus a repeat detector that escalates when the same failing call shows up twice, not a better prompt.

also fair flag on the ledger being graded by Claude after the fact. a small blind set of seeded bugs would give you a clean catch rate per reviewer.

are you planning to log false alarm cost per reviewer, or only the findings that held up?

Collapse
 
tom_jones_230c4659491adcd profile image
Tom Jones •

Thanks. Your repeat detector is close to what we built this week for the agent layer: a second identical refused call goes up a level, a third stops the run, and a step budget caps the rest.

On your question, the misses are logged, the cost is not. Every finding has a row, including the 58 Grok findings that turned out wrong and the 76 nobody ever tested, so the false alarm count is there. What we do not log yet is what each one cost to check. Alex Chen asked the same thing in this thread, so check cost per finding, per reviewer, held up or not, is going into the ledger next. A small blind set of seeded bugs is a good way to get the clean catch rate, so I will add that too and post both here when we have them.