DEV Community: Tom Jones

Who your model works with matters more than which model you picked

Tom Jones — Tue, 18 Aug 2026 21:08:41 +0000

The short version, for anyone who does not benchmark models for a living

Every few weeks a new model tops a leaderboard and the advice is to switch to it.

We have been measuring our own stack for a while, and two things keep coming back that are worth separating carefully, because the second one is easy to oversell and we have oversold it twice.

The first is measured and we are confident in it. Models that a leaderboard ranks two points apart are not doing the same work. On the problems where they differ, the difference is stable: run them again and the same models are right and wrong on the same specific problems. That is a property of the models. Which one tops the column total is substantially a property of the run.

The second is what we would like to be true, and we have not earned it. It would follow that pairing two different models beats upgrading to one better one. We have never run that experiment. We built the obvious mechanism to capture the available headroom and captured none of it. So take the title as the hypothesis this piece is about, not as a result it delivers.

What we can hand you is the measurement underneath it, a ceiling we could not reach, and twenty-one days of a ledger recording every time our own second model told us something confidently false. Including once, today, about this article.

What "a better model" actually buys you

Start with a number that surprised us, on our own production path.

We ran the same fixed 800-item tool-calling benchmark through our live system five separate times. Nothing changed between passes. Same items, same harness, same endpoint.

The five totals were 752, 750, 750, 749, 749. A mean of 93.75%, and the whole spread is three items.

That looks like a beautifully stable system, and here is the part that matters: 38 distinct items, 4.8% of the set, flipped somewhere across those five passes. They cancel out. The worst pair of passes disagrees on 23 individual items while their totals differ by one.

Two numbers, and they say different things. The score is extremely stable: 0.375 points across five passes. The answers underneath it are not: on any given pair of passes, 2.00% to 2.88% of individual items come back differently.

We had a rule of thumb here that doubled the second number and used it as a threshold on the first. Our own reviewer killed it while reviewing this article, and it was right to: an item-flip rate and a score gap are different quantities, and the whole reason the score is stable is that the flips cancel. You cannot derive one from the other, and we have retracted it. The section further down shows that exchange in full, because it is a better illustration of the method than anything we could have written on purpose.

What the five passes do license is narrower and still worth having. If you are going to quote a single benchmark number, run it more than once, publish the spread, and publish how many individual items moved. A stable total is not a stable measurement, and only the second of those two numbers tells you whether the thing you are ranking sits still.

There is a second way the level story gets overstated, and it is worse. We took one model, kept everything identical, and changed only which company served it. On one model-and-provider pairing that moved the score by 17.3 points, from 78.2% to 95.5%. On a different pairing the same experiment moved 0.6 points, so this is a property of the specific pair rather than a law. But it means a cross-vendor ranking built on unpinned routing can be measuring the plumbing rather than the model, at a magnitude larger than the gaps it is reporting.

What a second model buys you instead

Now the other half, and this is the one we would defend hardest.

We put three models over the same 160 coding problems. Same prompts, same scorer, one run. They scored 148, 146 and 144 of 160, which is 92.5%, 91.25% and 90.0%. A 2.5 point spread, and by the logic above you should treat that ranking as noise.

Then we stopped reading the column totals and partitioned by problem instead.

Across two runs the three models disagree on roughly 15 to 18 percent of the problems, and 83% of the set is uncontested by all three. Most of the work is genuinely settled. On the remainder, models that a leaderboard calls equivalent are answering differently.

The obvious objection is that this is the same flicker wearing a costume, and it has to be answered on this set rather than by importing a number from a different benchmark. On these 160 problems the same model on the same problem across two identical runs flips 3.7%, 4.9% and 5.2% of the time. So the honest comparison is a contested rate of 15 to 18 percent against a self-flip rate of up to 5.2 percent, which is a factor of roughly one and a half to two and a quarter depending on which pair you take. Real, and considerably less dramatic than it would look if you reached for a flip rate from some other benchmark, which is a thing we have now done twice and been caught doing twice.

So the size of the disagreement is not the argument. The argument is that ten problems are contested in both runs with zero flips by any model. Each model is stably right or stably wrong on them, run after run.

We had a bigger-sounding version of this, resting on how improbable the overlap of contested sets was under a null where every problem is equally likely to be contested. Our reviewer killed that null, correctly: easy problems are essentially never contested, both runs draw from the same medium band, so the sets overlap with no family specialisation at all. It supplied a difficulty mixture reproducing most of the overlap. We checked the arithmetic and it holds, so that p-value is gone.

Then we ran the test it should have been, because the objection is answerable rather than merely conceded.

Hold each problem's difficulty fixed and destroy only the model-specific part. For every problem take its observed success rate across all three models and both runs. That number is its difficulty, measured, with no mixture assumed and no distribution fitted, which makes it immune to the objection above whatever the true spread turns out to be. Then resample every cell as an independent coin weighted to that rate, and count the stable-contested problems again. Twenty thousand draws.

| | |

|---|---|
| observed, contested in both runs with zero flips | 10 |
| same statistic under the difficulty-preserving null, mean | 2.0 |
| the most the null ever produced in 20,000 draws | 9 |
| p | 0.00005 |

Difficulty makes all three models wobble on a problem. It does not make one model reliably right and another reliably wrong on that same problem, twice. The claim survives a null built specifically to satisfy the objection to the previous one.

Here is the whole residue, named, on 154 problems both runs scored:

problem	gpt-oss-120b	qwen3-235b	qwen3.8-max
HumanEval/9	pass	fail	fail
HumanEval/10	pass	fail	pass
HumanEval/22	pass	pass	fail
HumanEval/26	pass	pass	fail
HumanEval/86	fail	pass	pass
HumanEval/115	pass	fail	pass
HumanEval/125	pass	fail	pass
HumanEval/132	fail	pass	fail
HumanEval/145	fail	fail	pass
HumanEval/160	pass	fail	pass

Read down the columns. Every model is stably right somewhere another is stably wrong, and every model is the sole failure somewhere. gpt-oss-120b owns five of these and is the lone loser on two. qwen3.8-max is the only one that solves 145. None of the three is redundant, and none of them dominates.

That is ten problems out of 154, which is 6.5%. It is a much smaller number than the disagreement rate, and it is the one we can defend. The self-flip rates on this same set are 3.9%, 4.5% and 5.2%, measured here rather than imported.

The part we have not earned, and will not claim

If three models disagree on that many problems, then something that could always pick the right one would score above any of them individually. On the 154 problems both runs scored, the best single model gets 145 and any-of-three gets 152, so that ceiling is 4.5 points.

We are not claiming those six points. Reaching them requires an oracle that knows which model was right, and an oracle is precisely the thing you do not have at the moment you need it.

We know this the expensive way. We built the obvious mechanism to capture that headroom, ran it end to end, and captured none of it. The cheap arm passed its own checks almost every time, so the second opinion was rarely consulted, and the result landed exactly where that first arm already scored. The headroom was real and our design could not reach it by construction rather than by tuning.

So the honest claim is narrower than the exciting one, and it is still the useful one: the gap between models is a real, reproducible, addressable thing, and the value shows up when a second model actually looks at the first one's work. Which brings us to the part we can put numbers on from our own three weeks.

So what do we actually run, and why in those seats

Three models, fixed roles, and the roles were chosen by failure mode rather than by tier.

Claude orchestrates. Specs, review, decisions, and the commits. It holds a very large context, which means it can carry the codebase, the written constitution and the current spec at the same time and notice when two of them disagree. That is the job.

Where it is weak, measured on itself: it agrees with itself. Given a shortcut it will produce a reason for the shortcut, and the reason will be fluent. Our own standing rule says substantive work runs as three, and the rule exists because the orchestrator twice decided a piece of work was "small and mechanical" and did it alone. Both times the cost showed up within the hour, in public once. An orchestrator that also implements ends up grading its own homework, which is why it does not implement.

Codex implements. It works from a written spec inside a sandbox, and it is good in direct proportion to how exact the spec is. Vague spec, confident wrong build.

The interesting part is a limitation we stopped trying to remove: it cannot commit. Its sandbox has no path to our history. That forces a review step between "the code exists" and "the code is in the repo", performed by something that did not write it. We could grant it commit rights. We have chosen not to, because the gap is where the checking happens.

Grok reviews, adversarially, and verifies across multiple hops. Different family from both of the others, which by the argument above is the entire point. It is strongest before something is built, when the thing under review is a design and there is still time for the answer to be "do not build this". Three times in three weeks it stopped something broken from shipping, and one of those was a token minter where we had handed the verifier its own minting key.

Where it is weak is documented below in detail: it is confidently wrong at a rate we can measure, and it attacks whichever leg you failed to brief it on, with exactly the same confidence it brings to the legs it has evidence for.

And the fourth seat is a person. Every pivot that mattered over three weeks came from the founder rather than from any model: ask another mind, is that apples to apples, what does the source actually say. The models are good at execution inside a frame. Choosing the frame, and noticing when the whole frame is wrong, has not once come from them. Any description of this setup that leaves the human out is describing something that would have drifted weeks ago.

The brief does more work than the model choice

This is the part we would keep if we had to throw everything else away.

Ask any model to "review this" and you get a compliment. It will find something small, agree with your framing, and hand the whole thing back improved by three percent. The model is not the problem. The request is.

So our brief names what we want destroyed, in those words. Here is the shape of a real one, from a review that came back DO NOT BUILD:

You are reviewing a design. Your job is to attack it, not to compliment it.

The five things I most want destroyed, in these words:

The central claim may be oversold. [the claim, and the measurement behind it]

[The load-bearing beam.] This is the strongest argument; if it breaks, tell us.

The evidence may prove a different thesis than the one claimed.

[The weakest data.] Make the strongest possible case that this is worthless.

[The newest, least-reviewed part.]

Also answer: what is missing, what should be cut, and where are you systematically weak and
should not be trusted.

Rules: re-derive any step you challenge rather than asserting it. Separate findings from
insinuations. If you cannot check something from what is here, say "I cannot check this from
the brief" rather than guessing. Name which of your objections you are least sure of.

Every line there is load-bearing. Naming the load-bearing beam invites the review to aim at the thing you would least like to lose. Asking where the reviewer is weak gets you a calibration you cannot get any other way. And "name which objection you are least sure of" is the one that turns an undifferentiated wall of confident findings into something you can triage.

That last instruction exists because of the failure described in the next section: a reviewer attacks the leg you failed to brief it on, with exactly the confidence it brings to the legs it has evidence for.

A small, dull example, because the dramatic ones prove less

We would rather show you the cheapest possible case than a war story, because the cheap case is the one you will actually recognise.

Today the task was posting four comments. No design, no architecture, no judgment. The kind of chore where running a review feels absurd.

We skipped the check and did it alone. What that produced, measured from the logs:


replies intended	4
posting attempts made	17, over 11 minutes
landed correctly	1
landed and should not exist	1
never posted	2
permanent public artifact	1 comment now reading "posted by mistake"

The queue was shared and held old rows that had already been posted days earlier. Draining it fired those too, and one went out as a word-for-word duplicate of something we had already said in that thread. Fixing it took a browser session, an edit to a live comment, and an API check to confirm the edit landed.

The check that would have prevented all of it was reading the queue before draining it. One command. Under a second.

Two honest caveats, and they matter more than the anecdote.

That is not a typical solo result. Most shortcuts cost nothing at all, which is exactly why they keep getting taken. Anyone claiming their process catches an error every time is selling something.

And the comparison is not "team good, solo bad" on average. It is about the shape of the distribution. The check is cheap, bounded, and always the same price. The failure is rare, unbounded, and in this case irreversible and public. You are not buying a better average. You are buying a shorter tail, and the reason it is worth doing on a chore is that chores are exactly where the check feels too expensive to bother with.

The guard now exists, so this specific hole is closed. That is the other half of the discipline: every manual finding leaves something automatic behind, or you will find it again.

And one seat is not filled by a model either. Every finding gets re-derived against the source by whoever is about to act on it. That step is the product. Everything above it is logistics.

We kept a ledger of every time the reviewer was wrong

Cross-model AI review works, and that part is settled by people with bigger samples than ours.

Greptile ran 1,000 pull requests in July 2026 and found that Claude Opus catches 53.7% of bugs in its own code and 62.0% in GPT-authored code, with GPT-5.5 showing the same asymmetry in reverse. Milvus put five different-family models through five rounds of adversarial debate on 15 real production bugs and moved detection from 53% to 80%. Cloudflare runs a role-tiered, cross-vendor reviewer in production. If you want evidence that a different-family reviewer catches what the author model misses, go read those.

We have something smaller and, we think, differently useful. For the last three weeks we have kept a ledger. Every time our reviewer returned a finding, we re-derived it against the source before acting, graded it, and wrote down the grade. Including the times the reviewer was wrong.

That last column turns out to be the interesting one.

The numbers

Sixteen adversarial reviews between 2026-07-27 and 2026-08-16. Fourteen ran before the thing shipped, two after.

	count	rate
reviews on file	16	21 days
ran pre-ship	14	87.5%
changed the artifact	14	87.5%
contained a finding that was wrong on re-derivation	13	81.3%
would have shipped something broken without the review	3

Read those two bold rows together, because separately each one tells a lie.

The first says the review is worth running. Fourteen times out of sixteen, the artifact that shipped differed from the artifact we brought to review.

The second says the review cannot be obeyed. Thirteen times out of sixteen, at least one finding fell apart the moment we went back to the source. Not stylistic disagreements. Confident, specific, technical claims that were false.

What the wrong ones looked like

A P0 that died to one curl. A security review ranked identity forgeability as an unverified critical issue. We sent an anonymous request and a forged-JWT request at the live endpoint. Both returned the sign-in page, zero data. The finding was ranked highest and was empty.

A missing measurement we had already taken. A pre-publication review objected that we had never measured end-to-end task latency. We had: p50 0.85s, p90 1.36s, same 800-task run, already in the ledger. Deferring to that finding would have made us withdraw a defensible public claim. The reason it happened is worth naming, because it was our fault: we had not briefed that leg. A reviewer briefed on a subset will attack the gap you made and will sound exactly as confident there as everywhere else.

A mechanism that was never there, hiding a worse bug that was. A migration review's central high-severity finding described a specific line of code doing a specific thing. That line does something else entirely. But going to the source to check produced a different and more serious defect: two functions resolve a caller's tenant by reading different database columns, so a partial migration would put one principal in two tenants at once. Nobody had that going in. The wrong finding was worth more than a right one would have been, because chasing it down led somewhere real.

What the right ones looked like

Three times the review stopped something broken from shipping.

A pre-build review caught that consolidating keyboard handling by focus would steal the Enter key from the terminal, so the shell would get no newline or a doubled one. The correct shape came back in the reviewer's own words and went into the spec verbatim.

A migration review said do not flip one tenant flag: all 8,116 cache rows were world-scoped, so flipping it would stop them matching. The important part is the failure mode. That regression surfaces as a bigger bill, not as an error, so monitoring that watches error rates stays green throughout.

A pre-ship review of a token minter found we had handed the verifier the minting key. The binding we were proud of was policy on one writer, while the cryptography permitted many. The patch was held. The design went asymmetric.

One worked example, start to finish

Here is a full instance, so the protocol is visible instead of merely described.

The problem. Our own standing rule says substantive work runs as three. One Sunday the orchestrator went solo on two implementation items anyway and told itself they were "small and mechanical", which is the exact rationalisation the rule exists to block. The founder asked: "How do we make the team use happen all the time? Is it something I have to continue to remind you of?"

A standing rule a human keeps re-supplying was never installed. It is only being remembered, and remembering is the thing that resets. So: build a gate.

Step one, refuse to guess. A hand-written test restates your own mental model as data and can only confirm it. So we extracted every file-editing action from 163 sessions of transcript history, 210,438 lines, giving a corpus of 5,037 real edits, and tested candidate definitions of the lane boundary against what the agent had actually done.

candidate definition of "implementer lane"	share of 5,037 edits blocked
anything not clearly orchestrator work	51.9%
restricted to implementer roots and code file types	51.0%
the same, minus measurement and benchmark scripts	42.2%
the same, plus "no other agent was dispatched this session"	27.0%
the same, counted once per file instead of once per edit	6.2%

The mental model was wrong. One scripts directory held 34% of all edits across 369 distinct files, mixing production serving code, guards, benchmarks and ops automation with no prefix separating them.

The obvious rival is that product code drives that rate and the scripts folder rides along. We tested it by dropping that directory from the definition: 51.0% falls to 16.9%, so it supplies 67% of the block volume by itself. One minute of work, and now the claim has a test under it.

The last row felt like the answer. 178 edits to one file is one decision to go solo, not 178. Change the accounting unit and an unshippable rate becomes about two prompts per session.

Step two, the review. The design went out with the measurements attached and an instruction to attack it. Asking a model to "review this" gets you a compliment, so the brief listed the five things we most wanted destroyed, in those words.

It came back DO NOT BUILD, with two findings we graded correct.

The dispatch condition was a laundering mechanism: survey, then decide, then implement is the normal shape of a working session, so after any unrelated dispatch the gate goes dark for the day. We measured it before agreeing. Removing that condition moved the rate from 1.91 to 6.03 fires per session. It had been hiding 357 of 669 events, which is 53% of the entire livability argument. The number proving the design was safe rested on the component we trusted least. Circular, and we missed it.

The second finding was about us. We picked the counting unit after looking at the rate table. The principle behind it holds, a decision does happen once, but we arrived at it while hunting for a number that worked.

Step three, the reviewer was wrong too. Its prescription was to stop inferring the boundary and block a fixed list of unambiguous implementer directories instead. Decidable, mechanical, no judgment.

We ran that against the same corpus. It fires 4.78 times per session, and it misses one of the two real incidents that caused the task to exist.

That is the finding of the day, and neither side had it going in:

The failures live precisely in the zone that no decidable rule reaches. A boundary crisp enough to be
mechanical cannot cover the ambiguous middle, and the ambiguous middle is where things actually go wrong.

Two designs, both measured, both dead. We built an observer instead: it blocks nothing, logs every event it would have flagged, and pre-registers all the candidate counting units so that post-hoc selection stops being available to us.

We also owe the reviewer a note on what we left out of the brief. We never told it our hooks support a shadow mode, which materially weakens its strongest objection, because livability then stops being a matter of argument and becomes something you measure in production for free.

The rule underneath all of it

One line, from the founder, in July:

"Don't trust someone else's work, because ours still has issues, why should we think theirs doesn't."

That became a constitutional rule here: an outside review is evidence, not a verdict. You re-derive the step the conclusion turns on. You separate the finding from the insinuation. You name the legs you did not brief, before you decide whether an objection is real or an artifact of your own briefing.

The asymmetry it exists to correct is the easy one to fall into: hostile scrutiny for our own numbers, courteous acceptance for someone else's. We had three instances of that in a single hour once, which is how the rule got written.

And it runs both ways. A review that agrees with you earns the same scrutiny as one that argues. External agreement feels like independent verification, which is exactly how it slips past the check it resembles.

We reviewed the review

Everything above describes a protocol. Here it is running, on this article, with the grades.

We sent this draft to the reviewer with a brief telling it to destroy five named things. It came back with a verdict: do not publish this thesis with this evidence. Then it listed findings.

The temptation at that moment is the whole subject of this piece. A confident, specific, well-argued rejection from a different model feels like a verdict, and the pull is either to obey it or to dismiss it. Both are the same mistake, which is treating the review as an answer rather than as input.

So we graded it. Every finding, re-derived against the source before deciding.

The findings we accepted

It found a defect that reached past the draft into our own source of truth. The article claimed a benchmark gap under about 5.8 points "is not a finding." The reviewer called that a category error: the number is twice an item-level flip rate, being used as a threshold on a score gap, and those are different quantities.

We opened the script the rule comes from. Its two-times rule governs the ratio of between-model disagreement to the within-model noise floor, and it prints a refusal when that ratio fails to clear two. It has nothing to say about leaderboard gaps. And our own paragraph refutes the claim without any help: across five passes the score moved 0.375 points while 4.8% of items flipped, because flips cancel.

Correct, and worse than the reviewer knew. That sentence did not originate in the draft. It came from a row in our verified-claims ledger, a row that had been cleared for public citation. The reviewer was attacking a line of prose. The defect was upstream, in the thing the prose was quoting, and it needed a correction at the claim.

It caught us committing an error we had already retired. The opening contrasted between-model disagreement of roughly 15 to 18 percent against within-model flicker of 2.00 to 2.88 percent. Those come from different benchmarks with different task distributions and different arity: one is a three-model contested rate on 160 coding problems, the other a pairwise flip rate on 800 tool-calling items.

That is the same error class we struck from the ledger in August, when an earlier version of this comparison divided a three-way rate by a pairwise one. We own the correct figure. On the same 160-item set the within-model flip rate is 3.7, 4.9 and 5.2 percent, which puts the like-for-like comparison at 1.50 to 2.25 times rather than something dramatic, and our own adversarial pass had already noted that this straddles the refusal line.

The reviewer re-derived that with no access to our ledger, our scripts or our artifacts. It had the article and nothing else.

It broke the load-bearing beam. Our strongest argument was that the contested problems reproduce across runs, 17 overlapping against 3.6 expected by chance, p = 1.5e-12. The objection is that the null assumes every problem is equally likely to be contested, which is false on arrival, because easy problems are never contested and both runs draw from the same medium band.

It supplied an existence proof. Thirty problems contested at probability 0.7 plus 130 at 0.023 yields the same overall rate and an expected overlap of 14.8 against our observed 17. We checked the arithmetic. It holds.

What survives is the smaller claim we already had: ten problems are contested in both runs with zero flips by any model. Difficulty alone cannot produce stable opposite answers. Only an interaction between a specific model and a specific problem can. That is the real residue, it is 6.25% rather than 15 to 18, and it should have been the beam all along.

The findings we rejected, and why

"91.2% is impossible on 160 binary items." The arithmetic is right, 91.2% of 160 is not an integer, and the conclusion drawn from it, that either the scoring is not binary or n is not 160, does not follow. It is 146 of 160, which is 91.25%, printed to one decimal. The repair is to publish counts alongside percentages, which we should have done anyway. The reviewer reasoned correctly from the page and reached past what the page could support.

"You never report the within-model flip rate on that set. You have the data." True as an observation, and the word "hiding" that followed it is an insinuation rather than a finding. That number is published in our own ledger, and it was withdrawn from public use by our own adversarial pass months before this draft existed. Leaving it out of the article is an editing failure. It is not concealment, and the difference matters, because one of those is a bug and the other is an accusation.

And a large share of what it could not check is our fault, not its. We sent the article and nothing else. No ledger rows, no scripts, no artifacts, no methods appendix. Its repeated "I cannot check this from the brief" is an accurate report about a brief we wrote badly, and its demand for a methods section is partly an artifact of that gap. A reviewer attacks the hole you left. Ours left several.

What grading actually costs, and what it buys

Settling two of those findings meant opening a script and a ledger row. That is perhaps twenty minutes. It is also the entire value of the exercise, because the accepted and rejected findings arrived in the same voice, at the same confidence, in the same list.

That is the point we would leave you with, and it is the one thing here that the larger studies do not measure. The published work on cross-model review reports how many real defects a second model catches. It does not report what the second model said that was wrong, because that column is only visible if someone re-derives every finding before acting on it.

We keep that column. Today it read five accepted, three rejected, and one of the five reached further than either of us expected.

And the recursive part is not a joke at our expense. The error the reviewer caught in the opening section was written today, by the orchestrator, inside the section arguing that a second model is worth the trouble.

What does not need three, which is most things

If the previous sections read as an argument for running everything past a committee, we have written them badly.

Most work does not need this. Renaming a variable, adding a log line, writing a test for behaviour you just specified, fixing a typo in a doc, any change whose failure mode is that it does not compile. A second model on those buys nothing and costs tokens, latency, and a small amount of your patience, and the practice will be abandoned within a week if you apply it there.

The filter we use is not size, and it is not risk in the abstract. It is this:

Would being wrong here be visible?

Work where the failure announces itself, a crash, a red test, a page that will not load, does not need a second opinion. You already have one, and it is faster and cheaper than any model.

Work where being wrong looks exactly like being right is where the whole practice earns out. Concretely, in three weeks, that meant: anything whose failure surfaces as a bigger bill rather than an error. Anything where a check might be passing because it cannot fail. Any security or identity boundary, where the successful case and the compromised case return the same status code. Anything irreversible or public. And any claim that is about to be published with a number attached.

That list is short on purpose. Rigour costs real money and real time, and the lever is never to be less rigorous. It is to be rigorous about fewer things, chosen deliberately, and then to go all the way on those.

What we are not claiming

We make no claim that three models beat one in general. We have never run that experiment.

We claim no novelty for cross-model review. Greptile and Milvus measured it first and at larger scale, and if the aggregate case is what you need, theirs is better evidence than ours.

Two honest limits on our own numbers. The corpus is self-selected: these are the reviews we chose to write up as nodes, so reviews that changed nothing may simply never have been recorded. And "artifact" here spans specs, designs, public claims and marketing copy, so the strict code-or-design cut is 9 of 10, against 14 of 16 overall.

And the premise underneath the worked example belongs to someone else too. "A blocking gate teaches where an advisory rule fails" was published by TRACE in June 2026, measuring advisory rules in context at 55.0% against hook-enforced blocking at 70.1%. Ours would have been a replication at best. What we actually found is that for a boundary requiring judgment we could not build the instrument at all, which is a smaller and stranger result than the one we set out to get.

The cost is real. Reviews take tokens, briefs take care, and re-deriving a finding means doing the measurement twice. Cutting rigour is the wrong lever. The right one is choosing fewer things to be rigorous about, deliberately, then going all the way on those.

The two lines worth keeping

On building guards: before you narrow a noisy guard's matcher, ask whether its unit is wrong. Everyone tunes the pattern. Almost nobody asks what one event is supposed to represent.

On running a team of models: the value is in the re-derivation, not the verdict. A reviewer you obey is an oracle you have not tested, and ours was wrong in 13 of 16 reviews while still being worth running every single time.

The agent that remembers, and never decides

Tom Jones — Wed, 12 Aug 2026 19:35:20 +0000

This is not a shortage of information. Every answer a department needs has usually been worked out already, by someone, in a meeting that happened. The problem is where it went.

It went into chat threads, policy articles, document folders, meeting notes, email, calibration discussions, and individual memory. Seven places, none of which is searchable together, and the last of which walks out of the building at some point.

What that costs is specific and recognizable. The same questions get answered repeatedly, slightly differently each time. Prior decisions are hard to find, so they get re-litigated instead of applied. Conflicting guidance exists and stays buried until it causes a problem. Calibration outcomes are lost once the meeting ends. New team members depend on finding whoever happens to remember.

The last one is the tell. When onboarding runs on a person instead of a record, the organization lacks knowledge. It has employees who hold knowledge, which is a different and much more fragile thing.

Start with what it never does

Most descriptions of a knowledge agent start with capabilities. This one starts with the boundary, because the boundary is the design and everything else is downstream of it.

The agent never makes the determination itself. It never creates policy. It never resolves a conflict between two sources. It never overrides existing guidance. It never treats a discussion thread as official guidance.

What it does instead is retrieve, cite, compare and escalate. When two approved sources disagree, it does not pick a winner. It reports that they disagree and routes the question to the people whose job that is.

Humans decide. The agent remembers. That sentence is the whole governance model, and it is what makes the rest safe to build. An agent that answers authoritatively becomes a policy source nobody approved. An agent that only ever hands you the record, with its provenance attached, cannot quietly become one.

The agent lives where the work already happens

It sits in the channels the department already uses, so nothing has to be adopted. No new destination, no separate portal, no habit to build. Someone asks a policy or calibration question the way they already ask it, and the agent answers in the thread.

It searches approved policy articles, prior calibration decisions, escalation outcomes, channel history, meeting summaries, and a lightweight knowledge store.

It returns the source articles linked, prior decisions with dates, related discussions, known conflicts named as conflicts, and an escalation recommendation when the record leaves the question open.

Returning a known conflict is the part that is easy to undervalue. A system that always produces a confident answer will produce one when the underlying guidance contradicts itself, and the person asking will never learn that. Surfacing the contradiction is more useful than resolving it, and it is the honest output.

The build is small on purpose. It runs on what the department already has: the chat platform, its agent builder, the existing policy hubs, the calibration memory files, the escalation log, and a small local database holding policies, decisions, escalations, questions and conflicts. Nothing here is exotic, and that is the point. A knowledge system that requires a procurement cycle will not be tested this quarter.

The loop that turns an answer into an institution

Eight steps, and most of them are human.

A question is asked in the channel.
The agent searches articles and history.
An answer is found, a conflict is found, or no guidance exists.
An unresolved item is escalated.
The calibration team reviews.
Leadership approves where required.
The approved decision is added to the knowledge base.
Future askers receive the approved answer.

   question ---asks---> THE AGENT ---reads only---> +--------------------+
      ^                     |                       |     APPROVED       |
      |                     |                       |  KNOWLEDGE STORE   |
      +--returns sources----+                       +--------------------+
        and named conflicts |                          ^              |
                            |                          |              |
              escalates when unresolved                |              |
                            v                          |              |
                    CALIBRATION TEAM                   |              |
                            |                    THE ONLY WRITE       |
                            v                          |              |
                       LEADERSHIP ---approves----------+              |
                                                                      |
       future askers receive the approved answer <-------------------+

The agent reads and never writes. Every path that changes what the organization believes runs through a person, and exactly one arrow writes into the store. That is what stops the agent quietly becoming a policy source nobody approved, and it is why the loop compounds: step seven happens once, step eight is free forever.

Steps four through seven are human. Correctness enters at step seven and only at step seven, from the calibration team, from leadership, and from policy owners where applicable. Only approved decisions are written back. A discussion in a channel, however senior the person in it, is not a decision and does not enter the store.

This is institutional learning, and the distinction matters. The model learns nothing about the domain. The organization accumulates decisions it already made, in a place where the next person will find them. Step eight is where the compounding happens, and step eight costs nothing, forever, once step seven has happened once.

Where the memory lives

Three layers, as plain files in the document stores the teams already use, surfaced through chat. The structure is deliberately boring. The permission model is where the care is required, because whether people write honestly into a personal layer depends entirely on knowing who can read it.

A personal layer, one per person, readable only by its owner. Root context describing role, tools and working style. Live tasks, blockers and next actions. Decisions made, each with its reasoning. Recurring issues with their causes and resolutions. The standards that person works to, versioned. Plus one folder that is deliberately shared upward, holding flags the team needs to know and open questions being surfaced.

A team layer, team readable and lead writable. Team identity and active work. Team decisions with history. Patterns aggregated across the team. Active standards, version stamped. Anonymous signals arriving from the personal layers.

A department layer above both teams, readable by both leads. Cross team context. Patterns surfacing across both teams. Decisions affecting both. An anonymous aggregate from both team layers.

                     +-----------------------------------+
                     |         DEPARTMENT LAYER          |   both leads
                     +-----------------------------------+
                            ^                     ^
                    a pattern seen in MORE THAN ONE team
                            |                     |
        +---------------------+           +---------------------+
        |  TEAM LAYER   eng   |           | TEAM LAYER  product |   team read
        +---------------------+           +---------------------+   lead write
           ^       ^       ^                 ^       ^       ^
                a pattern, person stripped out
           |       |       |                 |       |       |
        +-----+ +-----+ +-----+           +-----+ +-----+ +-----+
        | own | | own | | own |           | own | | own | | own |   owner only
        +-----+ +-----+ +-----+           +-----+ +-----+ +-----+

     named entries, written freely because only the owner can read them

Signal rises and loses identity at every boundary. A person writes named entries into a layer only they can read. What reaches their team is a pattern with the person removed. What reaches the department is a pattern that appeared in more than one team.

Signal moves upward and loses identity as it goes. An individual writes freely because the layer is theirs. What reaches the team layer is a pattern with the person stripped out. What reaches the department layer is a pattern that appeared in more than one team. Each boundary is a permission boundary, so the privacy property holds structurally and nobody has to keep remembering a rule.

Two phases, and why the order carries the weight

Phase 1 runs to completion before Phase 2 begins. That sequencing is the most consequential decision in the test, and it is easy to mistake for scheduling.

Phase 1, weeks one and two, is engineering only. Personal layers for each engineer. The team layer capturing architectural decisions and patterns. Session context assembled automatically at the start of each session. Decisions logged with their reasoning attached, the why alongside the what. Bug patterns and root causes captured in structured form. Standards made explicit, versioned and retrievable.

Phase 2, weeks three and four, adds product while engineering continues unchanged. Product onboards using the same architecture. The shared department layer activates. Engineering decisions become visible to product context and the reverse. Cross team patterns surface that neither team sees alone.

When product onboards in week three, engineering does not learn alongside them. Engineering has been living in the system for two weeks and helps with the setup, so the second team takes days, against the two weeks the first team needed. Run the phases in parallel and that advantage disappears: two teams learning at once, and nobody in the room who has done it before.

Engineering and product are chosen because they hold the most consequential knowledge gap in most technology organizations. Engineers know how a thing was built and which alternatives were rejected. Product knows what was decided, why it was prioritized, and what the customer signal was. Those two records are almost entirely separate today, and the value of connecting them needs no domain expertise to evaluate.

What would count as working

Stated in advance and deliberately narrow. Every criterion is a thing that either exists or fails to. Nothing here is a satisfaction score, for reasons given in the next section.

Prior decisions are findable in under thirty seconds, timed, by someone other than whoever filed them. A repeated question gets the same answer twice, asked again in week four from a different account. Conflicts are escalated rather than buried, with at least one case running all the way to an approved decision. Responses carry their sources, and a spot check confirms the link supports the claim. An approved decision from week one is still retrievable in week four with its date and reasoning. A cross team pattern surfaces that neither team saw alone. People are still asking questions in week four without being reminded to.

That sixth one is the core claim of Phase 2 and the only signal that cannot be produced by either team in isolation. If nothing appears in two weeks, the department layer has not earned its place.

Three predictions, recorded before the test starts

A protocol published after the fact is a story. The author already knows how it turned out and the criteria drift, quietly, toward whatever happened. Publishing first is the only cheap way to stop that.

I am strict about this because I recently paid for it. An experiment of mine returned a clean, total result: one hundred percent in the treated arm, zero in both controls, across all four cases. It was worthless. Every task in it could be answered by copying a sentence out of the text being injected, so the harness was measuring reading comprehension. The only reason I caught it was a prediction written to disk beforehand saying the result should be messy and mixed. The clean number contradicted the prediction, and that contradiction was the entire signal. Two later versions of the same instrument were also wrong, each in a way that flattered whatever I was hoping for.

So here are three predictions, on the record, before any data.

Everything below rests on three measurements taken on 2026-08-11, on a system I run, against codebases I did not write. They are small, and they are stated with their size so they can be argued with.

What was measured	Setup	Result
Where durable lessons come from	4 conditions, 2 foreign codebases	Ordinary use 1, isolated test 0, agreeing with 20 recorded fixtures 3
Whether a healthy system generates material	1 full suite, 476 tests passing	0 durable lessons
Whether irrelevant context is harmless	12 trials per arm, 1 model, 1 scenario	Irrelevant note wrong 12 of 12, control merely vague

One. The layers will fill unevenly, and the pattern will not be effort.

People working where their output has to agree with someone else's, shared interfaces, contested policy, anything with a boundary in it, will produce full layers. People working on self contained work will produce nearly empty ones while working just as hard. Read that as engagement and half the team gets judged for the shape of their work instead of its quality.

The basis, measured 2026-08-11 across four conditions on two codebases I did not write: ordinary use produced one durable lesson, adding an isolated test produced none, and making one component agree with twenty recorded fixtures produced three in about thirty minutes. The variable was integration surface, not hours worked.

Two. Week one will feel like nothing is happening.

A new layer is empty, and a process running normally generates very little worth recording. The material appears when something is contested, revisited, or wrong. A team expecting usefulness by day three will conclude the test failed before the compounding has begun.

The basis, measured the same day: a full test suite on a healthy codebase, 476 tests passing in about a minute, produced zero durable lessons worth recording. Nothing was broken. There was simply nothing to learn, because nothing resisted.

Three. Asking whether it felt useful will return yes, and will mean nothing.

Delivered context makes people feel better informed whether or not the context was any good. This is why every criterion above either exists or fails to, and why none of them is a rating.

The basis, measured the same day at twelve trials per arm, one model, one scenario: against a control given nothing, injecting a genuinely irrelevant note produced a confidently wrong answer in twelve of twelve cases, while the control was merely vague. The bad context went past unhelpful into actively misleading, and it read as authoritative while doing it. Treat that as a lead, with the sample size and the single scenario counting against it.

That third one is also the strongest argument for the boundary at the top of this piece. An agent that hands over sourced records lets a person check. An agent that hands over confident answers removes that option.

What I do not know

There is one instance of this architecture running in production over a long period, and it is mine. One organization, one operator, and every figure quoted above comes from it. This document generalizes to any two teams in any organization, and that generalization is a hypothesis rather than a finding. This test is the first attempt to check it against somebody else's work.

Three further things it cannot settle, listed because a careful reader will find them anyway.

Whether people write honestly into a layer their employer hosts. The permission model is designed for this and design is not proof. A personal layer that quietly becomes a performance record stops receiving true entries immediately, and that failure is silent.

Whether a cross team pattern is a pattern or a coincidence. Two teams generate enough signal that something will always look like a connection. Requiring that neither team saw it independently is a guard, and a weak one.

Whether four weeks is long enough for anything to compound. It is long enough for the layers to fill and for first patterns to appear. It is almost certainly not long enough to observe the property the whole design aims at, which is a team keeping its knowledge across a departure.

The test does not end at four weeks. What ends at four weeks is the part anyone is willing to make predictions about.

If you have run something like this inside a real organization, I am most interested in the first unknown above. The permission boundary is the part I am least able to prove from my own instance, because in my instance the owner and the operator are the same person.

Microsoft measured our thesis, and we still cannot quote ours

Tom Jones — Sun, 09 Aug 2026 14:45:33 +0000

Microsoft's .NET team published the benchmark behind their polyglot unit testing agent. We have been arguing its conclusion for months. We can't cite our own version of it, because we withdrew our number on 2026-08-06 and haven't earned it back.

Both halves of that are worth writing down.

What they measured

The setup is clean. One tool, one model, 152 tasks from real repositories. The only thing that varies is a plugin that makes the agent research the repository, plan, implement, then verify before it claims to be done. A task passed only if the repository built, all tests passed, the agent added at least one test, and it removed none.

Arm	Completed	Rate
With the workflow	140 / 152	92.1%
Same model, stock	120 / 152	78.9%

63% fewer failures, from the same model and the same tool.

The breakdown is where it gets useful.

Prompt type	With workflow	Stock
Vague, 89 tasks	88.8%	66.3%
Detailed, 63 tasks	96.8%	96.8%
Diff-targeted, 15 tasks	15 / 15	0 / 15

On detailed prompts the two arms tie exactly. The whole gap comes from vague prompts, where the agent has to work out for itself what to test, which framework is in use, and where the tests belong. So the scaffold isn't making the model smarter. It's taking away the guessing.

Then the per-model results, on 45 .NET tasks:

Model	With workflow	Stock
Claude Opus 4.8	95.6%	77.8%
GPT-5.5	91.1%	80.0%
Claude Haiku 4.5	75.6%	55.6%

Their own summary of that table is one sentence: "The workflow helped every model. With Opus, it added eight wins with no losses."

There's a reading they don't offer, and I want to be clear that it's mine rather than theirs. GPT-5.5 inside the workflow finished 91.1%. Opus 4.8 outside it finished 77.8%, on the same 45 tasks. Read across the rows instead of down them and the cheaper model in a system came out ahead of the stronger model on its own. Microsoft compares each model only against itself and never makes that claim. I'm making it, from their published table, so weigh it as my inference and not their finding.

Two caveats we'd want applied to us

Their coverage barely moved: 72.4% against 72.2% line, 49.8% against 49.1% branch. The agent also wrote slightly fewer tests than stock, 6,963 against 7,129. So the workflow didn't produce better tests. It produced tests that exist, build and pass, more often. They say so themselves, and it's a completion result rather than a quality one.

The gains also concentrate where the request is underspecified. On detailed prompts the workflow buys nothing at all. That's a real boundary on the claim, and it belongs in the headline rather than a footnote.

We'd hold our own numbers to exactly that. So we will.

Why we can't quote ours

We have a version of this result. On our own harness, switching the verification layer off dropped correctness from 100% to 75%, and switching the guards off as well took it to 50%. Small n, our harness, not a public benchmark, and we say that every time we cite it.

The bigger claim was a lift figure: a cheap model alone, then the same model inside our cascade. We published it. Then we audited our own truth layer and found something ugly. Our two baselines for the same quantity, the same cheap model alone on the same benchmark, disagreed by 3.1 points. One page computed the lift from one baseline. Another page computed it from the other. Each traced back to a real measurement, which is exactly why nobody caught it.

Both lift figures came off every public surface that day, and they stay off until one baseline is re-run.

So here's the honest position. Microsoft has a clean, large, well-controlled measurement of something we believe. We have an unresolved 3.1 point disagreement with ourselves about ours. Their result doesn't repair ours. Someone else agreeing with you isn't a re-measurement of your own work, and being right isn't the same as having measured it.

The part we did measure, and it points the same way

One result of ours survives a second independent run and an adversarial pass, so it can sit next to theirs.

We ran three models over the same 160 HumanEval+ problems, with the same prompts and the same scorer. They finished 92.5%, 91.2% and 90.0%. A 2.5 point spread that reads as interchangeable.

Partition by task instead of by column total and it looks different. 83% of the set is uncontested, where all three models get it right, and the models disagree on roughly 15 to 18 percent of the work across two runs. What makes that citable isn't how big the disagreement is, it's how stable: the same problems come up contested each time. 17 tasks are contested in both runs against 3.6 expected by chance, and ten of them are contested in both runs without any model ever changing its answer.

Which problems they split on is a property of the models. Which model scores highest is a property of the run. That's HumanEval+ Python on our harness, not customer traffic.

Put it beside Microsoft's table and you get one argument instead of two. They measured that the workflow around the model carries most of the gain you can reach. We measured that the ranking you'd use to pick a model is mostly noise. Both point at the same place, and it isn't the model.

What we took from it

One technique, straight away.

Before their agent declares a task complete it runs a set of checks, and one of them reads: "It considers small code changes that should make the tests fail. This is a lightweight form of mutation testing." No full mutation framework. Just a cheap step that catches an assertion which can't fail.

We knew that rule. We wrote it down after a scorer of ours passed its positive control perfectly while being structurally incapable of returning a negative. We wrote it down again when a guard printed OK at every boot and then fired on zero of seven deliberately injected defects. We were applying it from memory, and memory isn't a source. Microsoft made it a step in the loop.

So we built it. Every guard in our tree can now declare, in its own header, an injected defect that must make it exit non-zero. A runner executes each one in a fresh directory and reports whether the guard can still say no. Three grades: PROVEN, BROKEN, and UNPROVEN for guards that haven't declared a control yet. UNPROVEN counts as a warning rather than a failure, because a gate that's red on day one gets overridden by reflex, and a reflex override has stopped being a signal.

We watched it fail before trusting it. A stub guard that always exits zero grades BROKEN.

Then we pointed it at the guard from the story above, the one printing OK at every boot. We injected a wrong percentage where a real measured one belongs. Exit 0. We injected a cost multiple we'd already struck from our own public copy. Exit 0. A third run, with the correct value in place, confirmed the guard really was reading two live surfaces against the real ledger, so a trivial pass couldn't be mistaken for a catch. It graded BROKEN by its own declaration, and it's since been fixed.

As I write this the runner reports 7 proven, 0 broken, 33 unproven, out of 40 guards. That last number moved while I was drafting, because writing this article involved building another checker and the gate made me declare a control for it before it would let me commit. The unproven count is the honest one, and it's the one we expect to be asked about. It's also the point of the exercise. Before we built this, the answer to "which of your guards can still fail?" was "we assume all of them."

A postscript, because it happened while writing this

Fact-checking this article, I flagged four of its own numbers as fabrications. The branch coverage figures. The Haiku row. The three-model percentages. Our own ablation result.

All four were real. I'd checked them against a news summary of Microsoft's post rather than the post, and against the wrong one of our own benchmark artifacts. The summary leaves out branch coverage entirely and never mentions Haiku, so an absence looked like an invention.

We have a rule for this and I broke it anyway: never relay one party's account of another's work without reading the original. I was one hop further out than I thought, which is the only place that error lives.

Two defects did survive the pass, and both were in the draft's account of our own work. A comparison of mine that read as Microsoft's, and a stale guard count.

And I have to finish that sentence honestly, because the first version of this article went out before I caught either one. It was live for about three hours, saying that a guard "is now marked BROKEN" when we had fixed it that morning, and reporting 2 proven of 37 when the runner said 7 of 40. This text replaced it in place, which is why the two paragraphs above are written the way they are. An article about withdrawing a number published a wrong one first. That is the whole argument for having a correction path that edits at the claim instead of appending a note at the bottom, and it is why every figure here now carries a line in a verification block saying where it was re-derived from.

Sources

Microsoft .NET blog, the polyglot unit testing agent: https://devblogs.microsoft.com/dotnet/polyglot-unit-testing-agent/
The plugin, MIT licensed: https://github.com/dotnet/skills

A stable aggregate is not a stable measurement

Tom Jones — Fri, 07 Aug 2026 17:49:12 +0000

We ran the same benchmark five times against our own production system and got 752, 750, 750, 749 and 749 out of 800.

That is a spread of three items across five runs. Less than half a percentage point. If you saw those five numbers you would conclude the measurement was essentially deterministic, quote the mean, and move on. I nearly did.

Then I compared the runs item by item, and 38 of the 800 items had changed answer between one run and another. Not three. Thirty eight, which is 4.8 percent of the set.

The totals barely moved because the flips cancelled. Something that was right in run 1 and wrong in run 3 is invisible in a sum, and there were enough in each direction that the aggregate came out almost identical every time. The number was stable. The measurement underneath it was not.

What five runs actually showed

run	score	of 800
1	752	94.00%
2	750	93.75%
3	750	93.75%
4	749	93.62%
5	749	93.62%
mean	750.0	93.75%

Same 800 fixed items, same configuration, same scorer, five separate passes through the production path, an hour and a half apart end to end. Temperature was zero, which people reasonably expect to mean deterministic. It does not.

Partitioning the 800 by how they behaved across all five runs:

across all five runs	items	share
all five correct	730	91.2%
all five wrong	32	4.0%
changed at least once	38	4.8%

And the pairwise view, counting items where two runs disagree with each other:

run pair	items that disagree	rate
1-2	16	2.00%
1-3	22	2.75%
1-4	19	2.38%
1-5	21	2.62%
2-3	20	2.50%
2-4	23	2.88%
2-5	17	2.12%
3-4	21	2.62%
3-5	17	2.12%
4-5	16	2.00%

The sharpest example is the pair that disagrees most. Runs 2 and 4 differ on 23 items. Their totals differ by one. If those had been the only two runs I did, I would have written down that the system reproduces to within a single item, and I would have been wrong by a factor of twenty three.

Why this matters if you are comparing anything

The reason to care is not academic. It is that almost every model comparison you read, including the ones I have written, is a difference between two totals.

If your measurement has a 2.88 percent chance of flipping any given item between identical runs, then two systems whose totals differ by less than roughly twice that are not distinguishable by the run you did. You need the gap to clear the noise, and you cannot know the noise without measuring it.

We learned this the embarrassing way first. Earlier we ran a three model comparison twice, one hour apart, same prompts, same temperature zero. The model that came first in run one came third in run two, and the one that came third came first. The gaps were around one to two points. The noise was larger than the gaps. We had a ranking, and it was a coin.

The cheap fix, which is one extra run

You do not need a statistical apparatus. You need a second pass of the identical set.

Run your benchmark twice on the same items. Do not change anything.
Count the items that changed answer. That percentage is your noise floor. Ours is 2.88 percent on this workload.
Refuse any comparison whose gap does not clear roughly twice the floor. Not "note the caveat." Refuse it, in code if you can, so a tired person cannot quote it later.

We put that last rule in the tool. It computes the floor alongside the result and will not bless a claim that sits inside the noise. It has told us no more than once, including on a result that flattered us.

And the thing worth keeping instead

If aggregates are fragile, something has to replace them, and the replacement is better anyway.

Report which items you disagree on, not who scored higher. When we compared three models by task
rather than by total, the partition was stable across runs even while the ranking was not: the same
problems came out contested each time, far more often than chance would put them there. Which problems
a system gets wrong is a property of the system. Who scored higher on the total is a property of
the afternoon.

That is also the more useful finding for anyone actually building something. A leaderboard position tells you which one to pick. A partition tells you where picking one is not enough.

The uncomfortable part

I want to be honest about where this leaves our own numbers, because it would be convenient to publish this as a lesson for other people.

It applies to us. Every benchmark result we have quoted from a single run is a number with an unstated uncertainty, and some of the differences we have found interesting in the past are inside the floor we just measured. The 2.88 percent is not a result about other people's evaluations. It is the error bar on ours, and we did not have it until this week.

The five runs cost about two hours and a few dollars. The thing they bought was not a better score. It was knowing how much of the score was real.

We stopped choosing models and started choosing what to measure

Tom Jones — Fri, 07 Aug 2026 17:49:09 +0000

The default way to use an LLM is to pick one and send it everything. That is what the integration looks like, it is what a model marketplace is shaped around, and it is what most agents I read about do. Pick the good one. Send it the work.

We run a routing and verification gateway, so we have had to measure a lot of models against the same tasks with money attached. I want to put down what our own data keeps saying, including the parts where it makes us look bad, because the conclusion is not "we found the better model."

We ran the experiment everyone runs to pick a model. We ran it twice. We got a different winner.

Three models, 164 HumanEval+ problems, temperature 0, one completion each. Then the identical thing again an hour later. Nothing changed between the two runs except time.

model	run 1	run 2	rank
qwen3-235b	89.6%	92.1%	3 -> 1
gpt-oss-120b	91.5%	90.2%	1 -> 3
qwen3.8-max	91.2%	91.6%	2 -> 2

Each model is scored over the problems it actually answered. Two of them answered all 164 in both runs. qwen3.8-max timed out on 4 and then on 9, so its denominator is 160 and then 155. The partition table further down uses the smaller set where all three answered, so its percentages are not directly comparable to these.

First and last swapped. The model we would have picked on Tuesday is the one we would have rejected on Wednesday, and the new frontier model that lost to our cheap tier by 1.3 points beat it by 1.4 the second time.

Temperature 0 is not determinism, and 164 problems is not enough to separate models sitting two points apart. We can put a number on how much of this is flicker: run the same model over the same problem twice and it changes its answer 3.7 to 5.2 percent of the time. Any ranking gap smaller than that is not a result. Ours were all smaller than that.

But something did survive both runs, and it is the more useful thing.

Stop asking how many each model got and ask which ones.

	run 1	run 2
all three correct	130	130
all three wrong	2	2
they DISAGREE	28 (17.5%)	23 (14.8%)

Roughly 15 to 18 percent of the time, the model you picked decided whether you got a right answer.
Note what the same table says about the other 82 percent: all three models agree, 130 of them right
and 2 of them wrong together, and the choice is irrelevant. This is a claim about a fifth of the work, not about all of it.

The reason to believe the fifth is real rather than flicker is not the size of the number, it is that it lands on the same problems both times. Twenty four problems were contested in the first run and twenty three in the second, and seventeen of them are the same problems. If the disagreement were noise you would expect about four to coincide. Ten of them are contested in both runs without a single model ever changing its own answer.

The ceiling is the part I would think hardest about. The best single model scores about 92 percent. If you could always take whichever of the three happened to be right, you would score 98.8 percent, and that number was stable across both runs too. Picking one model, even correctly, leaves about six points on the table.

A leaderboard cannot show you any of this. A leaderboard is column totals, and all of it lives in the rows.

Capability is shaped like the task, not like a ranking

Our verification witness is llama-3.3-70b, pinned to one provider. On 400 standard tool-calling tasks it scores 382, which is 95.5 percent, and matches our primary path. On an adversarial set built to tempt it into calling a tool when it should decline, it scores 30 of 40.

I have to be careful with that 40, and I only learned why today. It is four distinct trap shapes, each parameterised ten ways. The replicates swap a city or a number; they do not add a fifth kind of trap. So the honest denominator is four, not forty, and I am not going to put a confidence interval on it. What the instrument legitimately supports is: across four adversarial tool-selection shapes, parameterised ten ways each, it failed ten times.

The interesting part is not the 75 percent, which I now think is close to meaningless. It is that all ten misses are the same verdict. Every one is a false call: it invoked a tool where the correct behaviour was to abstain. So the model matches the primary at doing the thing and fails it at knowing when not to. Those are different abilities, they are not correlated the way a single score implies, and only one of them was on the benchmark.

If you are building on a model because it scored well at a task, check whether the thing you actually need is the task or the restraint.

The price on the page is not the price

We screened a cheaper candidate for our cheap tier. Advertised at 0.05 and 0.20 per million against our shipped tier at 0.07 and 0.27, roughly 30 percent cheaper on both sides.

On a real tool call it emitted 132 completion tokens, of which 123 were reasoning tokens that we paid for and never saw. Its effective output price came out around fifteen times its sticker, making it roughly ten times more expensive than the tier it appeared to undercut. The tool call itself was correct. The capability was real. Only the economics were inverted.

This is one call, so treat it as a mechanism rather than a rate. The mechanism is enough: compare cost per successful result, never cost per million tokens, and read the reasoning token count before you believe any price comparison.

You may not be choosing a model at all

Same model, same 400 prompts, same scorer. Pinned to a single serving provider it scored 95.5 percent. Unpinned, letting the marketplace route it, 78.2 percent. The model was held constant and the score moved 17.3 points.

For scale, adjacent models in our own sweep differ by 0.5 to 6 points, so the routing spread was larger than the entire model spread we were trying to measure.

And then we checked whether that was a law, because it would be a convenient one for us. We ran the
same pinned-versus-unpinned experiment on gpt-oss-120b. It moved 0.6 points, 65.2 percent pinned
against 65.8 unpinned. So this is not "marketplaces degrade models." It is a property of the specific
model and provider pair, and the actionable version is that you have to check yours rather than assume
either way.

The newest one is not automatically the answer either

Qwen 3.8 came out yesterday. We benchmarked it the same night against the two models it might replace.

And this is where I have to be careful, because the first version of this section said "it did not beat our cheap tier" and the second run says it did. On accuracy it is indistinguishable, in both directions, and I am not going to pretend otherwise in either.

What did not move between runs is the part that actually decides it. It cost 64 to 76 times more per solved problem, depending on the run, priced at the marketplace list rate rather than at what the vendor's own API would have charged us, which we did not read. It was slower at the median in both. And its timeouts more than doubled, 4 then 9 out of 164, with a worst case over 200 seconds. The cost multiple moves because it is a ratio of two noisy things, so treat it as a band and not a constant, which is the same rule this piece applies to everything else.

So the reason not to adopt it is not that it is worse at the task. It is that it is indistinguishable at the task, dramatically more expensive, and less reliable, and those three are all stable across runs while the score is not. The score was the only thing I originally judged it on, and the score was the only thing that turned out to be noise.

So what do you do instead

If models differ by task rather than rank, then "which model is best" is optimising the wrong thing. The unit we route on is the task shape plus a check: send the request to whatever suits its shape, serve the cheap answer when something independent agrees it is right, and escalate only when the check disagrees. The model becomes a swappable component, and being wrong about one becomes survivable.

And the uncomfortable part, because leaving it out would make this an advertisement. Our own escalation target never cleared the cheap tier it exists to rescue on code, in either run. On the first run it scored below it, 90.0 against 92.5. On the second the two tied at 92.9. By this piece's own rule I cannot report either of those as a ranking, so the honest version is the weaker and worse one: we have no evidence that the model we escalate to is better at code than the model it is rescuing. That is not what an escalation target is for. A gate can only be as good as the thing it routes to, so the thing we are fixing is not the gate, it is the ladder, and last night's benchmark was us checking whether the new frontier model was the fix. It was not.

That is scoped to code, and I should say so rather than let it sound general. This benchmark did not test the tool path at all, and our own data has the ordering of these tiers inverting by task shape. Which is the whole argument again, made against us this time: the rung that looks wrong on one task shape is not automatically the wrong rung.

Caveats, in full

The 98.8 percent ceiling assumes an oracle that always picks the model that happened to be right. No router has that, ours included, and we have not measured what share of those six points our own routing actually captures. Read it as the size of the prize, not as a result.

Single run per arm on the 160 problem comparison, temperature 0, one completion each. Temperature 0 was not deterministic for us: two runs of the same 8 problems gave different results, so treat small gaps as noise, including ours. The 95.5 versus 78.2 pair is 400 per arm, single run, no confidence intervals. The trap set is four shapes replicated ten times, not forty independent cases, so treat it as four observations and ignore any interval anyone computes on n=40, including one of ours that is now struck. The pricing finding is one call. Our benchmarks are code and tool calling and they are not your workload.

None of this says frontier models are overrated. We escalate to one and would not ship without it. It says that after measuring a lot of them against the same tasks, the most useful thing we learned was not which one wins. It was that they fail in different places, and the difference between them is bigger than the gap in their scores.

I built a page that only publishes numbers surviving our own checks. It caught me inside the hour.

Tom Jones — Sun, 02 Aug 2026 20:07:11 +0000

I run a verified inference endpoint. What we sell is checking: we verify results before serving them, and we say so when we cannot.

Every claim defect we shipped this month had the same shape. A number was measured honestly, the configuration underneath it changed, and the number stayed on the page. Our benchmark rows were measured on model tiers we retired weeks earlier. Nothing was fabricated. Everything was stale.

So I built a page that cannot go stale. It is generated from the live gateway config, and every badge on it is derived rather than written. If a measurement's configuration premise no longer matches what is running, the page strikes the number and says why.

It went live. Within the hour it caught me publishing two numbers our own ledger explicitly withholds.

What the page does

For each path through the system it states four things:

What "good" means on that path, and whether a real oracle checked it. Tool calls are checked against the schema. Code runs against your tests in a sandbox. Free prose gets nothing, because there is nothing to run.
The measurement, with n, interval and date.
Whether that measurement still describes the running system, graded automatically against live config.
Whether it passed adversarial review, and if not, that appears on the page next to the number.

The third item is the whole point. A number and a config are two facts that drift apart silently, and no amount of care fixes that. A generator that reads both cannot lie about the gap.

The catch

I put our tool-calling and trap-set results on it. Both are good numbers. Both are marked in our internal claims ledger as not cleared for external citation, owes adversarial verify.

I had written the page whose entire purpose is publishing only what survives our checks, and I had not run the check.

The first fix was a blocklist for that one phrase. Also wrong. Our ledger withholds rows in at least six phrasings, and a blocklist misses the seventh. For a public claim the gate has to fail closed: a figure publishes only if its ledger row exists and carries no hold marker. Over-withholding costs a number on a page. Under-withholding costs a retraction.

Failing closed immediately caught two more, subtler ones. Two figures had no ledger row at all, because I had sourced them from an internal handoff instead of the truth layer. One was a 30/40 result. Our ledger also contains a 30/40, from a completely different run on a different model. Same number, different measurement. That is exactly the confusion an uncited figure invites.

Then the reviewer rejected my draft

The remaining claim needed real adversarial review, so I commissioned one and briefed it to refute rather than agree. It came back do not publish, with three specific defects:

I wrote that our agreement gate accepts a wrong answer "roughly a quarter to a third of the time." That range was two underpowered subgroups (6 of 25, and 5 of 15) whose confidence intervals almost entirely overlap. It reads like a stable operating characteristic. The data supports one pooled binomial: 11 of 40, 27.5%, Wilson 95% interval 16.1 to 42.8.
I quoted "1.7 to 3.5% on code." It has the shape of a measured interval. The reviewer said the evidence was not in my brief, which would normally mean I under-briefed it. So I checked instead of accepting. The figure appears three times in our ledger with no n, no interval and no date anywhere. The gap was real.
I wrote "near zero on arithmetic." That is residue from a claim we retracted publicly after a reader caught it from our own article's text. No clean replacement exists, so the phrase cannot appear at all.

I took all three. I refused one instruction: it wanted me to name an internal threshold, which is recipe rather than result. It did not know that because I had not briefed it on our disclosure policy. Same failure, other direction.

The green check that could not fail

While extending our claims gate to cover more surfaces, I did something I should have done when it was written: I injected the defect it exists to catch.

Seven injections. Five wrong percentages, a struck cost multiple, a retracted compounding claim. It fired on none of them. Its tolerance covers most of the range it checks, so almost any number is "backed" by something. It had been printing OK every morning for weeks.

It was not lying. Its own output says "values checked, not claims," and a comment in its source says the same. The defect is one level out: a green line on a status board is read as protection, and nobody re-reads the caveat.

A guard you have never watched fail is not a guard. It is a habit.

The rules that transfer

A correction in your repo is not a correction in public. Our live site was still serving retracted figures while every internal instrument reported the sweep as done, because every internal instrument greps the repo. Verify a public claim by fetching the public URL.

Your measurement tool is an instrument, so verify it first. Our cost script summed one of three cost columns. It reported one cent where the truth was $1.73, and it was the exact script our ledger's cost row cited.

A readability filter can eat an error. I piped a remote script through grep -v to hide a deprecation warning. A database auth failure exited zero, and that became STATUS: Success, then a fleet-wide OK across both production boxes. The lie got more authoritative at every hop.

Placeholders are claims. Our dashboard rendered "$1,284 saved this month" as static HTML before live data arrived. A brand-new account with zero usage would have been shown a fabricated saving. I found it while opening signup to the public, one step before strangers would have seen it.

Publish the path where you are weaker. Our trap set is 200 of 200 on the primary path. Under failover it is 30 of 40, and every miss is the same verdict: it calls a tool where it should decline. Both numbers are on the page. The second is the more useful one.

Go and check it

The page is live at tirtha.ai/capabilities. Seven paths, what checks each one, what it measured, and the review state of every figure.

To try it rather than read about it, tirtha.ai/verify runs live against our API with no key and no signup. Three prefilled adversarial cases, and you can watch it decline a call instead of inventing one.

To run it on your own workload: go to tirtha.ai, click Get an API key, sign in with Google, and start. Free while we are in beta. No card, no email, no sales call. You get 100 requests a month and it stops rather than surprises you.

If you find a claim on that page that does not survive, I want to hear it. That is not rhetorical. The last time someone did, they were reading our own published text, and they were right.

We built a router to predict when a cheap model is enough. It does not work.

Tom Jones — Wed, 29 Jul 2026 02:10:58 +0000

If you serve a model cascade, escalation is your cost dial. Not your model choice, not your prompt, not your context window. The single number that moves your bill is what fraction of requests climb to the expensive tier.

So the obvious thing to build is a router: look at the incoming request, predict whether the cheap model will get it right, and only pay for the expensive one when the answer is no. We built that. It does not work, and the reason turned out to be more interesting than a working router would have been.

Publishing it because a negative result with a mechanism is worth more than a positive result with a story, and because the last two times we published numbers, readers found the flaws faster than we did.

What we built and what it scored

Our gateway already embeds every prompt, because the semantic cache needs a vector. Reusing that vector to predict difficulty is free. That is exactly why we did it, and that turned out to be the founding design flaw.

Trained on 539 real coding tasks labelled by actual execution (428 easy, 111 hard):

held-out AUC                       0.594
5-fold CV                          0.55 to 0.57   (one fold below chance)
best threshold                     scores exactly what "never escalate" scores

A classifier that cannot beat a constant policy is not a weak classifier. It is not a classifier.

It is a feature limit, not a tuning problem, and we made it prove that

The tempting read is underfitting or overfitting, so we swept L2 regularisation across four orders of magnitude, 0.01 to 100, with cross-validation at every step. Train AUC stayed at 0.94 to 0.98 while test AUC stayed at 0.58 to 0.59, at every single setting. The gap never closed. A gap that survives four orders of magnitude of regularisation is telling you the features do not contain the signal.

Then the result that actually explains it. We threw together 11 crude surface features, things like length and token counts, no semantics at all:

11 crude surface features          AUC 0.610
1024-dimension prompt embedding    AUC 0.552
both together                      AUC 0.609   (the embedding adds nothing)

Eleven numbers you could compute with a ruler beat a 1024-dimension semantic embedding. The reason is simple once you see it: the cache vector encodes topic, not difficulty. It is built to answer "have I seen a question like this before", and it is good at that. Two questions about sorting a list sit close together in that space whether one is trivial and the other is subtle. We reused it because it was free, and free was the whole problem.

Two measurement traps, which are worth more than the router

One. Scoring a cost-saving router on accuracy alone marks it failed by construction. A router that
saves money by sending easy work to a cheap model will, correctly, be slightly less accurate than
always escalating. Our first yardstick printed DO-NOT-SHIP on a router that actually had skill. The
control you need is random routing at the same cost, not the expensive model. If you cannot beat
a coin flip that spends what you spend, you have nothing. If you can, you have something, even if your
raw accuracy went down.

Two. The ceiling is not "always escalate". We assumed the expensive model was the upper bound and
a perfect router would approach it. It is not. On our 539 tasks, escalating everything to the top tier
rescues 39 answers the cheap model got wrong and breaks 23 it had already got right. Net +16.

always escalate      82.4%
a PERFECT router     86.6%

A perfect router beats always-escalate by 4.2 points, because it also knows when not to escalate. That headroom is real and no gate-local heuristic captures it. If you are benchmarking a cascade against "just use the big model", you are measuring against the wrong ceiling.

What is still alive

Showing the router the cheap model's draft rather than only the prompt does better: AUC 0.640, and 81.8% accuracy at 36% escalation against 80.5% for an equal-cost random control. That is the right shape. But permutation testing gives p=0.0375 unadjusted and roughly p=0.30 after Bonferroni, so the honest verdict is promising, not proven. We are not going to claim it until it survives a pre-registered run.

Which makes sense mechanically: the draft carries evidence of difficulty that the question alone does not. A model that is about to be wrong often looks different while being wrong.

The questions, and this is why I am posting

I would rather hear from people who have hit this than keep guessing.

What features actually carry difficulty before generation? We have shown that a topic embedding
does not, and that crude surface stats beat it. That is a low bar. What clears it?
Is pre-generation difficulty prediction possible at all, or is the draft the earliest honest
signal? It is entirely plausible that "will this model fail" is not a property of the question,
only of the interaction. If someone has evidence either way I would like to see it.
If you run a cascade in production, what do you actually route on? I suspect the honest answer
for most teams is a hand-written rule about request type, and I suspect those rules do better than
our classifier did. That would itself be a finding.

Numbers, labels and the negative result are ours to share. If you want the setup in more detail, ask and I will write it up.

Tokens Are Not the Unit

Tom Jones — Sun, 26 Jul 2026 20:04:49 +0000

Every AI provider publishes a price in dollars per million tokens. Every comparison table ranks by it. Every build-versus-buy spreadsheet runs on it.

That number is misleading, and not in a small way. It can be wrong by 10x, and wrong in the direction that makes the expensive option look cheap.

This piece explains seven things about the real cost of AI work. Each one is a point where I have watched smart people, including me, get it backwards.

1. The sticker price is not the price

THE POINT: you are billed for tokens the model produces while thinking, even though you never see them and cannot use them.

Modern models emit "reasoning tokens." The model works through the problem, and that working-out is generated text. You are charged for it. On many APIs you never even receive it.

Here is a real evaluation we ran. We were considering swapping our low-cost tier for a cheaper model:

Current model:   $0.07 in / $0.27 out per million tokens
Candidate model: $0.05 in / $0.20 out per million tokens
On paper: about 30% cheaper on both sides.

Obvious swap. So we sent it one real request. It answered correctly. Then we read the billing detail:

274 prompt tokens
132 completion tokens
  of which 123 were REASONING tokens
  of which   9 were the actual answer

You are billed for all 132. You can only use 9.

Do that arithmetic and the effective price of a useful output token was $2.93 per million, which is 14.7 times the advertised rate. The "30% cheaper" model was roughly ten times more expensive than the one it appeared to undercut.

We verified this against the provider's own reported cost for that call and the two agreed exactly, so this is not a units error on our side.

One honest limit: that was a single call. The ratio will move with how hard the question is. Treat the mechanism as the finding, not the specific 14.7. The direction does not move.

There is a second trap in the same family. On some models the reasoning goes into a separate field and the content field comes back empty. Under a tight output limit, the reasoning eats the entire budget before any answer is produced. We saw exactly this: at a 100 token limit, empty response. At 200, a correct one. An empty response looked like the model was incapable. It was actually a budget symptom. If we had trusted the first reading we would have discarded a model that works fine.

WHAT TO DO ABOUT IT: before you believe any price comparison, send one real request and read completion_tokens_details.reasoning_tokens in the response. Then compute dollars per useful output token. If your provider does not expose that field, you cannot actually price the model, and you should say so out loud in the meeting.

2. Cheap models are not uniformly worse. They fail differently, and the difference is the whole story.

THE POINT: the standard mental model, that models sit on one line from dumb to smart, will get you hurt. Two models with nearly identical scores can behave completely differently when it matters.

Most people picture a single quality axis. Expensive is smart, cheap is dumb, pick a point that fits the budget. If that were true, choosing a model would be a budget exercise.

It is not true. Here is a field of models on 800 tool-calling tasks. First, plain accuracy:

frontier model A   95.9
frontier model B   95.1
ours               ~94
strong open model  93.6
cheap model C      93.0
cheap model D      91.0

Read that column alone and the cheap models look like a steal. Four percentage points for a fraction of the price.

Now here is a second column. Inside those 800 tasks are deliberate traps: requests where the correct behavior is to decline, because the right tool is not available or the request is malformed. This column is the percentage of traps handled without grabbing the wrong tool:

frontier model A   100%
ours               100%
strong open model  100%
cheap model C       90%
cheap model D       52%   <-- and this model scored 91 on accuracy

Stop on that last line. A model that looks four points behind on accuracy will pick up the wrong tool half the time it is baited.

Think about what that means in a system with real permissions. That is not "slightly less accurate." That is a model that will confidently call delete_records when it should have said "I do not have a tool for that." The accuracy average washed the single most important behavior completely out of view.

WHAT TO DO ABOUT IT: build a small set of tasks where the right answer is to refuse, and measure the refusal rate separately. Never let it be averaged into an accuracy score. If a vendor cannot tell you what their model does when it should do nothing, you do not have the number that matters.

3. Because they fail differently, sorting beats upgrading

THE POINT: if cheap models were uniformly worse, your only lever would be paying more. Because they fail in specific, predictable ways, you have a much better lever: send each task to the weakest thing that can actually do it.

This is routing, and the important thing about routing is that it is a sorting problem, not an intelligence problem. Sorting is cheap. Intelligence is expensive. Any time you can convert the second into the first, you win.

A concrete picture over thirty days of our real production traffic, 68,369 requests:

Priced at frontier rates, this exact traffic:  $166.25
What it actually cost us:                      ~$46 to $51
Gross margin:                                  about 70%

But the composition is the part worth internalizing:

84% of our total cost was the escalations to the expensive model. Everything else, all the cheap serving, all the infrastructure, was rounding error next to it.

That means the cost dial is not which model you picked, and not the price you negotiated. It is how often you have to escalate. A 10% reduction in escalation rate does more for your bill than a 10% discount from any vendor.

WHAT TO DO ABOUT IT: instrument your escalation rate before you optimize anything else. If you do not know what fraction of your requests need the expensive model, you do not know what your system costs or why.

4. Verification is what makes cheap safe

THE POINT: routing on its own is a gamble. What turns it into engineering is being able to cheaply check whether the cheap answer is right.

Here is the asymmetry the whole approach rests on:

Producing a correct answer is expensive. Checking one is often very cheap.

You already know this from normal software. Writing the function is the hard part. Running the tests is the easy part. That asymmetry does not disappear when a model writes the function, and it is the thing you should be exploiting.

Cheap checks available to you: run the tests. Check the types. See if it compiles. And one more that people underuse: ask two independent models and see if they agree.

We measured that last one. On questions with no tests to run, using two endpoints we believed were independent:

WITHDRAWN, see the correction below:
  When the two agreed, probability the answer was wrong:  0.00   (n=160)
  How often they agreed:                                  76%

What re-measurement found, once the gate was tested per workload shape:
  short-answer arithmetic (what the original probe used)   ~0
  code, with executable tests as ground truth              1.7% to 3.5%
  faithfulness judgement, real off-trap data              27.5%   (95% CI 16.1 to 42.8, n=70)
  how often the two agreed on that data                     61%

CORRECTION, added 2026-07-27. Do not use the 0.00 above. A reader (@anp2network) took the number apart in the comments, and our own follow up measurements agree with them on every point. Leaving it uncorrected would be the exact failure this piece complains about elsewhere.

One. Zero wrong in 160 does not bound the error rate at zero. For a zero numerator the 95% upper bound runs about 3/n, roughly 1.9%. With the gate passing 76% of traffic that permits something like 1.4% of everything served to be wrong and never checked. The point estimate was 0.00. The bound never was.

Two. That probe used short answer arithmetic, where the cheap tier is essentially never wrong. A gate cannot be measured where the primary does not fail, and every row showing the primary correct should have been the tell. Re-run on code, using the same agreement check and the benchmark's own tests as ground truth, the gate leaks low single digit percent, not zero.

Three. Agreement substitutes for correctness only if the two endpoints fail independently, and independence is a property of what the provider is serving, not of your code. Ours quietly stopped being independent. The witness was selected by model name, and all three of our cheap backends turned out to be the same underlying model under three different spellings. A same model pair agrees about 88% of the time against 70 to 75% for a genuinely different one. That is close to a rubber stamp, and a rubber stamp also reports 0.00.

Those 160 cases spanned four task families including deliberately hard traps. We originally read the traps as reassurance. That was backwards. A trap works by shared structure in the input, shared structure is exactly what correlates two endpoints, so the gate thins out on the cases that cost the most to get wrong.

What the gate actually buys you, stated honestly: it lowers the rate at which you serve a wrong cheap answer. It does not drive it to zero, and how far it lowers it depends on the two endpoints really being different. Verify that they still are, and keep verifying, because the answer can change without you deploying anything.

The asymmetry this section opens with still holds. Checking is cheaper than producing, and routing by a check still beats routing by hope. What does not hold is treating a single agreement number, measured once, on the wrong shape of task, as a guarantee.

WHAT TO DO ABOUT IT: for every class of work you send to a model, write down how you would check the answer cheaply. If you cannot answer that, that class of work is not a routing candidate yet, and that is useful to know before you build.

5. The benchmarks actively punish the behavior you want

THE POINT: the leaderboards score a correct refusal as a failure. If you pick models by leaderboard, you are selecting against safety.

This one is worth being very explicit about, because it is counterintuitive and it is expensive.

The most valuable behavior in a production agent is declining to act when the request is ambiguous, malformed, or outside its remit.

The major agent benchmarks score task success. A refusal is a failed task. They award exactly nothing for "correctly declined to do the dangerous thing."

So a system tuned for production safety scores WORSE on the headline number than a system that always attempts and is occasionally catastrophically wrong.

Sit with that. The public number that everyone compares is, in this specific and important respect, pointing the wrong way.

WHAT TO DO ABOUT IT: run the benchmarks anyway, because your customers and your competitors will. But report the wrong-action rate right next to the task-success rate, every time. And know both numbers privately before anyone runs them at you publicly.

6. Your workload shape decides your economics, not your architecture

THE POINT: a single blended cost-per-request number hides the variable that actually determines whether this is profitable for you.

Two workloads through our identical system:

Tool-calling work:  almost never needs the expensive model
Coding work:        about 57% escalated on fresh problems

Same code. Same models. Same prices. One of those is enormously profitable and the other is thin.

Our healthy margin exists partly because our traffic happens to be tool-calling heavy. A customer whose work is mostly fresh coding would see materially worse economics, and it would be dishonest of us to quote them our number.

I am saying that plainly because the whole industry quotes blended numbers, and a blended number is a hidden assumption about your mix.

WHAT TO DO ABOUT IT: when anyone shows you a cost-per-request for an AI system, your first question is "on what mix of work?" If they do not have an answer, the number describes their traffic, not yours. And measure your own mix before you forecast anything.

7. Publish the ceiling honestly, because someone else will find it

THE POINT: we are at parity on code, not ahead, and saying so is the only version that survives contact with a skeptic.

On a clean, cache-free run of a standard coding benchmark:

Our cascade:        92.1
Frontier model A:   92.7
Frontier model B:   93.3
The bare cheap model alone: 81.1

Same harness for all four. We are slightly under the frontier. The architecture adds about 11 points over the cheap model by itself.

We were tempted by a "beats the frontier" line. The measurement did not support it, so we do not use it. Parity is the honest word, and parity at a fraction of the cost is the actual product.

WHAT TO DO ABOUT IT: the number that does not flatter you is the only one worth publishing, because it is the only one that holds up when a customer reruns it. A claim you cannot survive being checked on is a liability with a delayed fuse.

8. The hardest part is not building the system. It is trusting your own measurements.

THE POINT: a wrong measurement is more dangerous than no measurement, because it comes with confidence attached.

This is the one I would most want a reader to take away, because it applies whether or not you ever build any of the above.

In a single working session, we chased seven separate alarms. All seven were broken instruments, not real problems. A parser reading five rows of a seventy-nine row file and reporting a catastrophe. A checker matching error strings against its own console output and finding "errors" it had printed itself. A meter reporting 135% of a hard limit when the true figure was 27%.

Three more from production, all instructive:

We were dropping two thirds of every traffic burst and could not see it. Our server had a careful queue that answered overload with a polite "busy, try again," which is exactly what an aggregator wants. But the operating system's own accept queue underneath it was at its default of five connections, so any burst deeper than five was refused by the kernel before a single line of our code ran. Every load test we had ever written sent exactly as much traffic as the server was willing to admit, which made those tests structurally incapable of finding this. The test design guaranteed the blind spot.

Then we fixed it wrong, in a way that looked right. A configuration flag said a feature was off. The deployed code was an older version that could not express "off" and instead disabled the feature completely. Latency improved. Of course it did, because doing nothing is fast. Every dashboard was green while a feature was 100% dead.

And we measured a worst case at 17 seconds using 10 samples. Later we measured the identical component with 828 samples. The real worst case was 126 seconds. Ten samples gave us an accurate median and a completely wrong tail.

WHAT TO DO ABOUT IT, and these are the four habits that would have caught every case above:

Change one variable at a time. If your fix changes three things and it works, you have learned that the bundle works. You have learned nothing about why, and you will keep the two useless changes forever.
Turn your fix off and confirm the problem comes back. A test that only ever passes has told you nothing.
Match your sample size to the statistic. A median settles in tens of samples. A worst case needs hundreds. A tail measured with 10 samples is not a cautious estimate, it is a wrong one that reads as cautious.
Chase the gap you cannot explain. When a number is slightly off and you invent a plausible reason to dismiss it, that reason is usually the bug. Twice in one week the explanation I reached for ("network overhead") was covering a real defect.

The one page version

If you remember nothing else:

Price in dollars per successful result, never per million tokens.
Read the reasoning token count before believing any quoted price.
Measure the wrong-action rate separately. Averages hide the failure that matters most.
Track your escalation rate. It is the cost dial, not your model choice.
Verify cheaply so you can generate cheaply. Tests, types and independent agreement all cost less than intelligence.
Quote economics per workload shape. A blended number is a hidden assumption about someone else's traffic.
Verify the instrument before acting on its number, and be most suspicious when the number is good.

None of this requires owning a frontier model. It requires taking measurement seriously, which is rarer, and considerably cheaper.

Edits to this article

2026-07-27. The agreement figure this piece originally published, a 0.00 probability of a wrong
answer when two endpoints agree at n=160, was withdrawn. A reader, @anp2network, took it apart in the
comments and was right on every point: a zero numerator bounds the error near 3/n rather than at zero,
the probe used short-answer arithmetic where the cheap tier is essentially never wrong so the gate was
measured where the primary does not fail, and independence between two endpoints is a property of what
a provider is serving rather than of your code, so it can decay with nothing in your system changing.
Ours had decayed exactly that way.

2026-07-28. Replaced the withdrawn 0.00 inside the table itself instead of leaving it standing
above a correction, and added the re-measured numbers by workload shape. The honest summary is that
P(wrong when they agree) is not one number. It is close to zero on arithmetic, 1.7 to 3.5 percent on
code checked by executable tests, and 27.5 percent on faithfulness judgement. Quoting any single one of
those as the property of the gate is the mistake this article made the first time.

Also worth stating plainly, since the same commenter suggested it: we now escalate a sample of the agreed cases to the expensive model anyway and score whether it concurs, which turns a one-time n into a number that moves when a route changes behind either endpoint. That is running in production as of this edit. We will publish what it says, including if it is unflattering.

Your company already runs the hardest parts of an AI agent platform

Tom Jones — Sun, 26 Jul 2026 20:03:50 +0000

Every AI agent session starts as a fresh mind. It has no memory of yesterday, no memory of the correction you gave it last week, and no idea what your company has already decided.

That is fine for one developer with one project. It falls apart the moment you have five teams, because now the things that must be true everywhere have to somehow be true in every session, of every project, forever.

The obvious answer is to write a good prompt and paste it into each project. That answer fails twice. It drifts out of sync the first week, because six copies of a rule are six rules. And every pasted line is paid for in every session forever, so the document that encodes your standards becomes a tax on all future work.

The better answer is the one your company already uses for people: inheritance, scoped by the org chart, with a channel for asking a human.

Here is the architecture, and then the part where I show you the data on which half of it worked.

Three tiers, inherited rather than copied

Tier one is the constitution. One document at the root, loaded automatically by every session in every project beneath it. It holds identity and the things that are never negotiable: what counts as evidence, what "done" means, the safety boundaries that are legal rather than stylistic.

Mine is capped at about forty lines, and the cap is the design, not tidiness. Every line here is read by every session of every project, forever. A constitution that grows is a bill that arrives every time anyone starts work. If a rule is not needed by every project before it knows what it is doing, it does not belong in tier one.

Tier two is the project constitution. What is true for this project and nothing more. It inherits tier one for free, so it never restates it.

Tier three is everything else. In my case, several hundred documents. None of them load at startup.

Notice that the cost runs opposite to the volume. The smallest tier is the most expensive because everyone pays for it always. The largest tier is nearly free because nobody pays for it until something asks. Most people build this upside down: they put the huge context at the top and wonder why every session is slow and expensive.

The mechanism that makes tier three work

A pile of documents nobody reads is not memory. What turns it into memory is a short table in tier two that maps an action to the thing you must read before doing it. Before you touch billing code, read this. Before you change anything a customer sees, read that.

The agent does not go looking for knowledge. The knowledge is delivered at the moment of the action that needs it.

This distinction sounds like a detail. It is the entire difference between the half of my system that worked and the half that died, and I have the numbers.

Where the humans go

An agent organization needs two human channels, and they are not the same thing.

Escalation is the agent saying "I need a decision that is not mine to make." A production change, a spend commitment, anything irreversible.

Calibration is a human saying "that was wrong, and here is the shape of the wrongness." This is the more valuable of the two and the one most setups drop entirely. A correction that lives only in a chat window is a correction you will pay for again in three weeks.

Both should run through the channel your organization already lives in. For most companies that is Teams. Not because Teams is special, but because a notification surface people already have open is the only kind that gets read. I have built beautiful dashboards for this. Nobody opened them, including me.

Threads matter here more than they look. A thread gives a correction a durable home attached to the thing being corrected, which is exactly the raw material you need if you ever want to fold that correction back into tier two.

The part I did not expect to be so clean

Once you accept the org chart as the scoping model, most of the hard problems stop being yours.

Identity and permission: your existing directory. An agent becomes a principal in the directory you already govern. It joins groups, it inherits those groups' rights, and it is revoked exactly the way a leaver is revoked. The alternative is a second permission model that lives beside the real one and drifts away from it, which is the security incident everyone eventually writes up.

This also answers the question every risk team asks first, which is not "is the AI accurate" but "what can it reach, and who decided that." If the answer is "the same groups that govern the humans doing this work," you are having a very different conversation.

The tiers: your existing document storage. Tier one is a site everyone reads, referenced rather than copied, so a change to the rules is a reviewable, versioned event instead of an edit nobody noticed. Tier two lives with the team that owns the work, so the boundary is one the company already agreed on rather than a folder convention somebody invented on a Tuesday. And personal scope goes in personal storage: private by default, follows the person, cannot leak into shared tiers by accident, because the permission is the boundary rather than a naming rule.

Audit: your existing audit log. Who told the agent what, and when. You inherit the answer instead of building an audit system for agents from scratch.

None of this is a product to buy. That is the point. The boring parts of an agent platform, identity and scoping and audit and a channel humans actually read, are precisely the parts your company solved years ago for its people. An agent organization should inherit them, not reinvent weaker copies.

Signal, and why the privacy design is a data quality mechanism

Information has to flow upward or the upper tiers are fiction. But if individual identity flows upward, you have built surveillance, and surveillance destroys the thing you were collecting.

The resolution is that the signal carries the pattern, not the person.

PERMITTED:      "4 people flagged this standard as unclear this month"
NOT PERMITTED:  "Sarah flagged it 4 times"

Identity is stripped at the team tier. Everything above receives patterns only, and no reverse lookup is possible by construction. That last clause matters: it is a structural property of how permissions and aggregation are built, not a policy somebody promised to follow.

There is also a threshold. A single event is not a pattern, so nothing escalates until several signals of the same type appear. This stops one bad afternoon from being surfaced upward as an organizational trend.

The governing shape is: private truth, then anonymous patterns, then organizational intelligence.

Here is why this is engineering and not ethics decoration. If people believe their personal tier can be used against them, they will populate it performatively. Performative input produces false patterns. False patterns produce confident, wrong decisions at the top of the organization, which is worse than having no system at all. The entire intelligence value of the system depends on the bottom tier being genuinely private, so the privacy guarantee is load-bearing infrastructure.

The loop that makes it improve itself

A memory system that only stores is a filing cabinet. What makes it compound is routing the questions.

When someone asks the assistant about a standard or a policy:

If a calibrated answer exists, answer immediately with a citation to the exact version, and log the question.
If no answer exists, do not improvise one and move on. Flag it as an open edge case and route it to whoever owns that standard.
If the same question appears repeatedly, that is not several confused people. That is one ambiguous standard, and it should be escalated as such.

That last inversion is the valuable one. Three people asking the same question about a rule is evidence about the rule, not about the people. Most organizations read it the other way around and send the three people to training.

The accumulating question log turns out to be a real-time map of where the organization is confused, produced continuously, with no survey and no focus group.

Now the half that failed

I added one more piece: a shared log where each project session ends by writing one line about what moved, feeding a cross-project summary.

Twenty two days later I counted what was actually in it.

Ten entries. Eight of them in the first three days. One more eleven days ago. Then nothing.

All ten came from a single project, out of five. The live one with a real customer has never written a line.

The shared summary, the entire reason the layer existed, was last modified the day it was created. It still confidently describes a project as a shell folder that I retired eight days afterward. It is not stale. It is wrong, and it looks authoritative while being wrong, which is considerably worse than being empty.

There was also a schema drift I never noticed: the timestamp is an integer in the first row and a formatted string in the last. I only found out because the script I wrote to analyze all this crashed on it.

Why it died, and why tier three did not

The design was fine. The trigger was not.

Writing that line was a remembered act. Nothing fired it. It sat at the end of a session and depended on somebody choosing to do it while tired and finished. That works for about three days.

Tier three works because it is bound to an act. Touch the file, get the rule. Nothing depends on anyone remembering the rule exists.

Same system. Same week. Same author. The only difference is what starts the write.

I have a second case that removes any remaining doubt. I built a lookup table for operational answers, seeded it with eight rows, and told the agent to add to it as it went. A day later it had eight rows. Zero added, while that same day's work had produced at least three entries that qualified and filed them somewhere else. The table did not need more rows. It needed a trigger.

The law

Anything that depends on remembering will not happen. Bind it to an act, or it does not exist.

Not because agents are careless. Because every session is a fresh mind with no memory of having agreed to the habit. You cannot build organizational memory out of good intentions distributed across amnesiacs.

And pull-based systems fail in the most flattering way possible. The file is still there. The design still reads well in the diagram. Nothing errors. You do not find out until you go and count the rows, which is why you should go and count the rows.

Two corollaries I paid for:

The read side needs pushing too. I assumed someone would open the summary. Nobody did for three weeks, including its author. A summary nobody is handed is a summary nobody reads. Surface it unprompted at the start of a session, or accept that it does not exist.

Anything derived needs a freshness check. A dashboard that is silently eleven days stale is worse than no dashboard, because it still looks like an answer. Stamp it with when it was generated and make it complain when that gets old.

If you are building this

In the order I would do it again:

Write tier one and cap it. Forty lines. If you cannot say it in forty lines it is a tier two rule wearing a disguise.
Build the trigger table before the documents. The delivery mechanism is the product. The documents are inventory.
Set the privacy boundary before the first signal. You cannot retrofit anonymity onto a system people already distrust.
Put the agent in your directory. Groups, not bespoke config. Deprovisioning should already work on day one.
Wire escalation into your chat tool before you need it. The first time an agent needs a human should not also be the first test of the channel.
Make calibration cheap to capture. If correcting the agent takes more than a reply, it will not happen, and you will pay for the same correction repeatedly.
Then, and only then, the upward flow. And bind it to something that already happens at the end of work, never to a good intention.

Start with a software team, whatever the eventual target is. Developers already live in this world, already use context files, and already feel the specific pain of losing the reasoning behind decisions. You do not have to explain the concept to them. You show them, and two weeks later either someone says it changed how they work, or it did not, and you have your answer cheaply.

The architecture is not the hard part. Getting it to still be running in three weeks is the hard part, and that is decided entirely by what fires the write.

Everyone is hardening the structure. Nobody is passing down the why.

Tom Jones — Sun, 05 Jul 2026 17:14:28 +0000

Two weeks ago I published a short piece called The Two-Channel Problem (tirtha.ai/research, the perspective panel) about what actually breaks when a forgetful AI agent builds a real product over months. Since then a genuinely good conversation has emerged between people running AI organizations. The nokaze seven-week paper named the cross-conversion gap: the rule file exists, and the agent sails right past it in the exact situation it was written for. The comment thread under it converged on real fixes: action-keyed triggers instead of "notice the moment," completion claims invalid without a re-checkable evidence source, checks that cannot be run by the layer that felt confident writing the claim. We run an AI-operated shop too (one human, a Claude orchestrator, Codex, a headless build fleet), and we built the same walls independently. All of that is right, and trading notes with these teams has sharpened our own guards twice this week alone.

But the conversation is converging on exactly one of the two channels, and the update I owe is about the other one.

What the structure channel fixes

Every fix in that thread strengthens what we internally call the structure channel: hooks, gates, generated status, mechanically forced loading. The structure channel transmits the WHAT. It makes discipline un-forgettable, which matters enormously when your workers wake up with no memory of yesterday.

The failure it cannot fix

Here is the failure it cannot fix. Early on, one of our sessions booted into a perfectly clean room: accurate task state, working guards, honest logs. It executed correctly all day, and it drifted all day. It searched instead of receiving, re-derived things the system already knew, treated the guards as obstacles to route around. Nothing it did was wrong by the letter. It complied without understanding. We started calling that a hollow successor.

So we run a second channel

So we run a second channel. Plain prose, written by each session for the next one, read at boot before any work: who to be here, why the room matters, what the rules are actually FOR, what yesterday's session learned about HOW to work that the commit log cannot carry. Not documentation. A letter.

The operating rule that fell out: structure transmits the what, only words transmit the why. A hook can force a file to load. It cannot make the next mind care what is in it. And an agent that does not understand why a rule exists will satisfy its letter and defeat its purpose the first time the letter and the purpose diverge, which is exactly the shape of most of the incidents this whole conversation is about.

The practical form

The practical form, since this sounds soft and is not:

A welcome document is read at boot, before work, and yes, a hook enforces that it gets read. We use the structure channel to guarantee the soul channel is delivered. What no hook can guarantee is that it lands. That gap is the point, not a flaw in the design.
Every session writes a short transmission at handoff. Not what it did (the log has that), but what it learned about how to work here: which instinct misled it, which boring re-check saved it, what it wishes it had been told at boot.
Corrections carry their origin. Every rule is stored with the incident that created it, so the next session inherits the reason, not just the restriction.

What we observe

The result we observe, for whatever one shop's evidence is worth: sessions that receive both channels correct themselves mid-flight in ways the guards alone never produced. This morning one of ours printed a success sentinel for a merge that had silently failed, caught it seconds later, and said out loud that it was the exact failure shape it had read about at boot. The guard did not catch that one. The letter did.

Write the other half

The teams in that thread are building excellent guards. If you run an organization of minds that forget, the guards are half the inheritance. Write the letters too.

I didn't mean to build this. Looking for testers anyway.

Tom Jones — Sat, 04 Jul 2026 15:55:11 +0000

I'll be honest up front: I'm not a salesman. Everyone has always told me that, and lately even the AI I build with tells me that. So this is not a pitch. This is me asking for help.

I didn't set out to create any of this. It started because I kept having context window issues while building something else entirely, and I went down a rabbit hole. The rabbit hole kept going. Somewhere down there I got tired of watching my AI coding agent send every single request to the most expensive model available, whether the task needed it or not, and I started building a fix for myself.

What's been embarrassing is I thought I had one story. Then the testing led me to another one. Then another. My website has changed multiple times, which has been a lesson learned in public, the hard way. The good news is there are no clients yet, so that's good :)

Where the testing has left me now is this:

As a gateway I am able to send routine coding requests to lower cost models that get verified before you ever see them. The hard ones escalate to a frontier model. On my benchmark runs that was about 1 in 27 requests. The full setup measured around 95 percent on HumanEval+ (n=164) at roughly 8x lower cost per request. I keep re-running these numbers because honestly I didn't quite believe them either. So far they keep holding.

So based on those runs, that works out to making a Fable 5 subscription last up to 27x longer, because most of the work doesn't need that level of cost to be accurate.

What I don't know yet is whether it holds up on YOUR work. Real repos, real deadlines, weird edge cases. That's exactly what I need testers for.

Security, because you should ask: encrypted at rest, TLS in transit, and your code is never training data. I verified the upstream providers do not train on it, and the routine tier runs on open models I host myself. Local models, privacy, security, and a cache that bends the cost curve down even lower over time. That is my goal, my promise, and what I want help testing and proving.

The deal: it's pre-beta, it's free while it is, and I will keep publishing the real numbers whether they flatter me or not. I feel like I have something special here, something people have been asking for. I just need people smarter than me to hammer on it and tell me where it breaks.

I got my coding agent to tie the frontier for about 8x less. Here is the honest benchmark.

Tom Jones — Thu, 02 Jul 2026 21:32:32 +0000

I am a solo founder. I do not have a lab or a team of researchers. I live paycheck to paycheck supporting my family like most people do. I didn't intend to build this. I just went down a rabbit hole and here is the story.

Two things bugged me about the AI coding agents I was using. The first was the cost. Every request, easy or hard, went to the most expensive model available. The second was quieter: I did not actually know where my code was going when the agent wrote it.

So I built something to fix both, and I measured it carefully, because I would rather tell you the honest number than a flattering one. Here is what I found, including the parts that are not flattering.

The idea

Most coding requests are not hard. A cheaper or local model handles them fine. Only a small slice genuinely needs a frontier model. So instead of paying frontier prices on everything, the system routes each request to the cheapest model that can actually do the job, checks the result, and escalates to a frontier model only when the check fails. Verified answers get cached, so work you have done before comes back fast. Router, verifier, frontier backstop, cache. I am holding the engineering details, but the idea is not the hard part. Measuring it honestly is.

The numbers

Same benchmark, same harness, across all of them. HumanEval+, 164 problems.

Parity, not a win

I want to be careful with the words, because it matters. This is parity. It ties the frontier models. It does not beat them. Anyone who tells you their cheap setup beats the frontier on accuracy is either measuring wrong or selling something. What I am claiming is narrower and more useful: you can land in the same accuracy band as the frontier without paying frontier prices on every request. The cheap model alone was 84.8%. The routing and verification is what closes the gap to 94.5%.

The cost, which is the actual point

Measured over 313 production requests, from the real usage logs. Blended cost came out to about $0.002 per request, versus about $0.017 for the frontier equivalent. Roughly 8x cheaper for work in the same accuracy band. On that run, 96% of requests were served by the cheap tier, and about 3.7% escalated.

The second win I did not expect

There is a second win I did not expect. Verified answers are cached, and a cache hit returns in about 0.16 seconds, which in my testing was 24 to 185x faster than solving it fresh. The more you code, the more of your work is instant. I will be honest that this compounds with real usage, and I am early, so I am watching it, not overclaiming it.

Where it is weak

Where it is weak (you should know before you trust it). - The hardest problems still escalate to a frontier model. That is by design. On hard, multi-step problems the savings shrink, because more of them escalate. It is not a cheap model doing frontier work by magic. It is the right model for each job, with a backstop.

HumanEval+ is a benchmark. Real-world code is messier, and I am still measuring that part honestly rather than pretending the benchmark settles it.
The verifier is only as good as the checks it runs. Give it weak tests and the gate is weaker.

About your code

Every AI coding agent sends your code somewhere, and mine is no different. Your code does reach my gateway. I am not going to pretend otherwise. What matters is what happens to it there. I cache verified answers, never your prompts or your code. Your code never enters the shared cache, it never trains anything, and it is walled off per tenant so it can never be served to anyone else. For regulated work there is a dedicated tier I genuinely cannot reach into.

Where I am

I am opening it to a first group of testers. If a cheaper, private coding agent that ties the frontier on accuracy is useful to you, I would genuinely like your help pressure-testing it. Tell me where the numbers do not hold up. I will keep publishing results as more people run it, the ugly ones included, because the honesty is the whole reason to trust a number from a guy you have never met.

Thanks for reading. If you have questions about the method, ask. I am around.