DEV Community: Tom Jones

The agent that remembers, and never decides

Tom Jones — Wed, 12 Aug 2026 19:35:20 +0000

This is not a shortage of information. Every answer a department needs has usually been worked out already, by someone, in a meeting that happened. The problem is where it went.

It went into chat threads, policy articles, document folders, meeting notes, email, calibration discussions, and individual memory. Seven places, none of which is searchable together, and the last of which walks out of the building at some point.

What that costs is specific and recognizable. The same questions get answered repeatedly, slightly differently each time. Prior decisions are hard to find, so they get re-litigated instead of applied. Conflicting guidance exists and stays buried until it causes a problem. Calibration outcomes are lost once the meeting ends. New team members depend on finding whoever happens to remember.

The last one is the tell. When onboarding runs on a person instead of a record, the organization lacks knowledge. It has employees who hold knowledge, which is a different and much more fragile thing.

Start with what it never does

Most descriptions of a knowledge agent start with capabilities. This one starts with the boundary, because the boundary is the design and everything else is downstream of it.

The agent never makes the determination itself. It never creates policy. It never resolves a conflict between two sources. It never overrides existing guidance. It never treats a discussion thread as official guidance.

What it does instead is retrieve, cite, compare and escalate. When two approved sources disagree, it does not pick a winner. It reports that they disagree and routes the question to the people whose job that is.

Humans decide. The agent remembers. That sentence is the whole governance model, and it is what makes the rest safe to build. An agent that answers authoritatively becomes a policy source nobody approved. An agent that only ever hands you the record, with its provenance attached, cannot quietly become one.

The agent lives where the work already happens

It sits in the channels the department already uses, so nothing has to be adopted. No new destination, no separate portal, no habit to build. Someone asks a policy or calibration question the way they already ask it, and the agent answers in the thread.

It searches approved policy articles, prior calibration decisions, escalation outcomes, channel history, meeting summaries, and a lightweight knowledge store.

It returns the source articles linked, prior decisions with dates, related discussions, known conflicts named as conflicts, and an escalation recommendation when the record leaves the question open.

Returning a known conflict is the part that is easy to undervalue. A system that always produces a confident answer will produce one when the underlying guidance contradicts itself, and the person asking will never learn that. Surfacing the contradiction is more useful than resolving it, and it is the honest output.

The build is small on purpose. It runs on what the department already has: the chat platform, its agent builder, the existing policy hubs, the calibration memory files, the escalation log, and a small local database holding policies, decisions, escalations, questions and conflicts. Nothing here is exotic, and that is the point. A knowledge system that requires a procurement cycle will not be tested this quarter.

The loop that turns an answer into an institution

Eight steps, and most of them are human.

A question is asked in the channel.
The agent searches articles and history.
An answer is found, a conflict is found, or no guidance exists.
An unresolved item is escalated.
The calibration team reviews.
Leadership approves where required.
The approved decision is added to the knowledge base.
Future askers receive the approved answer.

   question ---asks---> THE AGENT ---reads only---> +--------------------+
      ^                     |                       |     APPROVED       |
      |                     |                       |  KNOWLEDGE STORE   |
      +--returns sources----+                       +--------------------+
        and named conflicts |                          ^              |
                            |                          |              |
              escalates when unresolved                |              |
                            v                          |              |
                    CALIBRATION TEAM                   |              |
                            |                    THE ONLY WRITE       |
                            v                          |              |
                       LEADERSHIP ---approves----------+              |
                                                                      |
       future askers receive the approved answer <-------------------+

The agent reads and never writes. Every path that changes what the organization believes runs through a person, and exactly one arrow writes into the store. That is what stops the agent quietly becoming a policy source nobody approved, and it is why the loop compounds: step seven happens once, step eight is free forever.

Steps four through seven are human. Correctness enters at step seven and only at step seven, from the calibration team, from leadership, and from policy owners where applicable. Only approved decisions are written back. A discussion in a channel, however senior the person in it, is not a decision and does not enter the store.

This is institutional learning, and the distinction matters. The model learns nothing about the domain. The organization accumulates decisions it already made, in a place where the next person will find them. Step eight is where the compounding happens, and step eight costs nothing, forever, once step seven has happened once.

Where the memory lives

Three layers, as plain files in the document stores the teams already use, surfaced through chat. The structure is deliberately boring. The permission model is where the care is required, because whether people write honestly into a personal layer depends entirely on knowing who can read it.

A personal layer, one per person, readable only by its owner. Root context describing role, tools and working style. Live tasks, blockers and next actions. Decisions made, each with its reasoning. Recurring issues with their causes and resolutions. The standards that person works to, versioned. Plus one folder that is deliberately shared upward, holding flags the team needs to know and open questions being surfaced.

A team layer, team readable and lead writable. Team identity and active work. Team decisions with history. Patterns aggregated across the team. Active standards, version stamped. Anonymous signals arriving from the personal layers.

A department layer above both teams, readable by both leads. Cross team context. Patterns surfacing across both teams. Decisions affecting both. An anonymous aggregate from both team layers.

                     +-----------------------------------+
                     |         DEPARTMENT LAYER          |   both leads
                     +-----------------------------------+
                            ^                     ^
                    a pattern seen in MORE THAN ONE team
                            |                     |
        +---------------------+           +---------------------+
        |  TEAM LAYER   eng   |           | TEAM LAYER  product |   team read
        +---------------------+           +---------------------+   lead write
           ^       ^       ^                 ^       ^       ^
                a pattern, person stripped out
           |       |       |                 |       |       |
        +-----+ +-----+ +-----+           +-----+ +-----+ +-----+
        | own | | own | | own |           | own | | own | | own |   owner only
        +-----+ +-----+ +-----+           +-----+ +-----+ +-----+

     named entries, written freely because only the owner can read them

Signal rises and loses identity at every boundary. A person writes named entries into a layer only they can read. What reaches their team is a pattern with the person removed. What reaches the department is a pattern that appeared in more than one team.

Signal moves upward and loses identity as it goes. An individual writes freely because the layer is theirs. What reaches the team layer is a pattern with the person stripped out. What reaches the department layer is a pattern that appeared in more than one team. Each boundary is a permission boundary, so the privacy property holds structurally and nobody has to keep remembering a rule.

Two phases, and why the order carries the weight

Phase 1 runs to completion before Phase 2 begins. That sequencing is the most consequential decision in the test, and it is easy to mistake for scheduling.

Phase 1, weeks one and two, is engineering only. Personal layers for each engineer. The team layer capturing architectural decisions and patterns. Session context assembled automatically at the start of each session. Decisions logged with their reasoning attached, the why alongside the what. Bug patterns and root causes captured in structured form. Standards made explicit, versioned and retrievable.

Phase 2, weeks three and four, adds product while engineering continues unchanged. Product onboards using the same architecture. The shared department layer activates. Engineering decisions become visible to product context and the reverse. Cross team patterns surface that neither team sees alone.

When product onboards in week three, engineering does not learn alongside them. Engineering has been living in the system for two weeks and helps with the setup, so the second team takes days, against the two weeks the first team needed. Run the phases in parallel and that advantage disappears: two teams learning at once, and nobody in the room who has done it before.

Engineering and product are chosen because they hold the most consequential knowledge gap in most technology organizations. Engineers know how a thing was built and which alternatives were rejected. Product knows what was decided, why it was prioritized, and what the customer signal was. Those two records are almost entirely separate today, and the value of connecting them needs no domain expertise to evaluate.

What would count as working

Stated in advance and deliberately narrow. Every criterion is a thing that either exists or fails to. Nothing here is a satisfaction score, for reasons given in the next section.

Prior decisions are findable in under thirty seconds, timed, by someone other than whoever filed them. A repeated question gets the same answer twice, asked again in week four from a different account. Conflicts are escalated rather than buried, with at least one case running all the way to an approved decision. Responses carry their sources, and a spot check confirms the link supports the claim. An approved decision from week one is still retrievable in week four with its date and reasoning. A cross team pattern surfaces that neither team saw alone. People are still asking questions in week four without being reminded to.

That sixth one is the core claim of Phase 2 and the only signal that cannot be produced by either team in isolation. If nothing appears in two weeks, the department layer has not earned its place.

Three predictions, recorded before the test starts

A protocol published after the fact is a story. The author already knows how it turned out and the criteria drift, quietly, toward whatever happened. Publishing first is the only cheap way to stop that.

I am strict about this because I recently paid for it. An experiment of mine returned a clean, total result: one hundred percent in the treated arm, zero in both controls, across all four cases. It was worthless. Every task in it could be answered by copying a sentence out of the text being injected, so the harness was measuring reading comprehension. The only reason I caught it was a prediction written to disk beforehand saying the result should be messy and mixed. The clean number contradicted the prediction, and that contradiction was the entire signal. Two later versions of the same instrument were also wrong, each in a way that flattered whatever I was hoping for.

So here are three predictions, on the record, before any data.

Everything below rests on three measurements taken on 2026-08-11, on a system I run, against codebases I did not write. They are small, and they are stated with their size so they can be argued with.

What was measured	Setup	Result
Where durable lessons come from	4 conditions, 2 foreign codebases	Ordinary use 1, isolated test 0, agreeing with 20 recorded fixtures 3
Whether a healthy system generates material	1 full suite, 476 tests passing	0 durable lessons
Whether irrelevant context is harmless	12 trials per arm, 1 model, 1 scenario	Irrelevant note wrong 12 of 12, control merely vague

One. The layers will fill unevenly, and the pattern will not be effort.

People working where their output has to agree with someone else's, shared interfaces, contested policy, anything with a boundary in it, will produce full layers. People working on self contained work will produce nearly empty ones while working just as hard. Read that as engagement and half the team gets judged for the shape of their work instead of its quality.

The basis, measured 2026-08-11 across four conditions on two codebases I did not write: ordinary use produced one durable lesson, adding an isolated test produced none, and making one component agree with twenty recorded fixtures produced three in about thirty minutes. The variable was integration surface, not hours worked.

Two. Week one will feel like nothing is happening.

A new layer is empty, and a process running normally generates very little worth recording. The material appears when something is contested, revisited, or wrong. A team expecting usefulness by day three will conclude the test failed before the compounding has begun.

The basis, measured the same day: a full test suite on a healthy codebase, 476 tests passing in about a minute, produced zero durable lessons worth recording. Nothing was broken. There was simply nothing to learn, because nothing resisted.

Three. Asking whether it felt useful will return yes, and will mean nothing.

Delivered context makes people feel better informed whether or not the context was any good. This is why every criterion above either exists or fails to, and why none of them is a rating.

The basis, measured the same day at twelve trials per arm, one model, one scenario: against a control given nothing, injecting a genuinely irrelevant note produced a confidently wrong answer in twelve of twelve cases, while the control was merely vague. The bad context went past unhelpful into actively misleading, and it read as authoritative while doing it. Treat that as a lead, with the sample size and the single scenario counting against it.

That third one is also the strongest argument for the boundary at the top of this piece. An agent that hands over sourced records lets a person check. An agent that hands over confident answers removes that option.

What I do not know

There is one instance of this architecture running in production over a long period, and it is mine. One organization, one operator, and every figure quoted above comes from it. This document generalizes to any two teams in any organization, and that generalization is a hypothesis rather than a finding. This test is the first attempt to check it against somebody else's work.

Three further things it cannot settle, listed because a careful reader will find them anyway.

Whether people write honestly into a layer their employer hosts. The permission model is designed for this and design is not proof. A personal layer that quietly becomes a performance record stops receiving true entries immediately, and that failure is silent.

Whether a cross team pattern is a pattern or a coincidence. Two teams generate enough signal that something will always look like a connection. Requiring that neither team saw it independently is a guard, and a weak one.

Whether four weeks is long enough for anything to compound. It is long enough for the layers to fill and for first patterns to appear. It is almost certainly not long enough to observe the property the whole design aims at, which is a team keeping its knowledge across a departure.

The test does not end at four weeks. What ends at four weeks is the part anyone is willing to make predictions about.

If you have run something like this inside a real organization, I am most interested in the first unknown above. The permission boundary is the part I am least able to prove from my own instance, because in my instance the owner and the operator are the same person.

Microsoft measured our thesis, and we still cannot quote ours

Tom Jones — Sun, 09 Aug 2026 14:45:33 +0000

Microsoft's .NET team published the benchmark behind their polyglot unit testing agent. We have been arguing its conclusion for months. We can't cite our own version of it, because we withdrew our number on 2026-08-06 and haven't earned it back.

Both halves of that are worth writing down.

What they measured

The setup is clean. One tool, one model, 152 tasks from real repositories. The only thing that varies is a plugin that makes the agent research the repository, plan, implement, then verify before it claims to be done. A task passed only if the repository built, all tests passed, the agent added at least one test, and it removed none.

Arm	Completed	Rate
With the workflow	140 / 152	92.1%
Same model, stock	120 / 152	78.9%

63% fewer failures, from the same model and the same tool.

The breakdown is where it gets useful.

Prompt type	With workflow	Stock
Vague, 89 tasks	88.8%	66.3%
Detailed, 63 tasks	96.8%	96.8%
Diff-targeted, 15 tasks	15 / 15	0 / 15

On detailed prompts the two arms tie exactly. The whole gap comes from vague prompts, where the agent has to work out for itself what to test, which framework is in use, and where the tests belong. So the scaffold isn't making the model smarter. It's taking away the guessing.

Then the per-model results, on 45 .NET tasks:

Model	With workflow	Stock
Claude Opus 4.8	95.6%	77.8%
GPT-5.5	91.1%	80.0%
Claude Haiku 4.5	75.6%	55.6%

Their own summary of that table is one sentence: "The workflow helped every model. With Opus, it added eight wins with no losses."

There's a reading they don't offer, and I want to be clear that it's mine rather than theirs. GPT-5.5 inside the workflow finished 91.1%. Opus 4.8 outside it finished 77.8%, on the same 45 tasks. Read across the rows instead of down them and the cheaper model in a system came out ahead of the stronger model on its own. Microsoft compares each model only against itself and never makes that claim. I'm making it, from their published table, so weigh it as my inference and not their finding.

Two caveats we'd want applied to us

Their coverage barely moved: 72.4% against 72.2% line, 49.8% against 49.1% branch. The agent also wrote slightly fewer tests than stock, 6,963 against 7,129. So the workflow didn't produce better tests. It produced tests that exist, build and pass, more often. They say so themselves, and it's a completion result rather than a quality one.

The gains also concentrate where the request is underspecified. On detailed prompts the workflow buys nothing at all. That's a real boundary on the claim, and it belongs in the headline rather than a footnote.

We'd hold our own numbers to exactly that. So we will.

Why we can't quote ours

We have a version of this result. On our own harness, switching the verification layer off dropped correctness from 100% to 75%, and switching the guards off as well took it to 50%. Small n, our harness, not a public benchmark, and we say that every time we cite it.

The bigger claim was a lift figure: a cheap model alone, then the same model inside our cascade. We published it. Then we audited our own truth layer and found something ugly. Our two baselines for the same quantity, the same cheap model alone on the same benchmark, disagreed by 3.1 points. One page computed the lift from one baseline. Another page computed it from the other. Each traced back to a real measurement, which is exactly why nobody caught it.

Both lift figures came off every public surface that day, and they stay off until one baseline is re-run.

So here's the honest position. Microsoft has a clean, large, well-controlled measurement of something we believe. We have an unresolved 3.1 point disagreement with ourselves about ours. Their result doesn't repair ours. Someone else agreeing with you isn't a re-measurement of your own work, and being right isn't the same as having measured it.

The part we did measure, and it points the same way

One result of ours survives a second independent run and an adversarial pass, so it can sit next to theirs.

We ran three models over the same 160 HumanEval+ problems, with the same prompts and the same scorer. They finished 92.5%, 91.2% and 90.0%. A 2.5 point spread that reads as interchangeable.

Partition by task instead of by column total and it looks different. 83% of the set is uncontested, where all three models get it right, and the models disagree on roughly 15 to 18 percent of the work across two runs. What makes that citable isn't how big the disagreement is, it's how stable: the same problems come up contested each time. 17 tasks are contested in both runs against 3.6 expected by chance, and ten of them are contested in both runs without any model ever changing its answer.

Which problems they split on is a property of the models. Which model scores highest is a property of the run. That's HumanEval+ Python on our harness, not customer traffic.

Put it beside Microsoft's table and you get one argument instead of two. They measured that the workflow around the model carries most of the gain you can reach. We measured that the ranking you'd use to pick a model is mostly noise. Both point at the same place, and it isn't the model.

What we took from it

One technique, straight away.

Before their agent declares a task complete it runs a set of checks, and one of them reads: "It considers small code changes that should make the tests fail. This is a lightweight form of mutation testing." No full mutation framework. Just a cheap step that catches an assertion which can't fail.

We knew that rule. We wrote it down after a scorer of ours passed its positive control perfectly while being structurally incapable of returning a negative. We wrote it down again when a guard printed OK at every boot and then fired on zero of seven deliberately injected defects. We were applying it from memory, and memory isn't a source. Microsoft made it a step in the loop.

So we built it. Every guard in our tree can now declare, in its own header, an injected defect that must make it exit non-zero. A runner executes each one in a fresh directory and reports whether the guard can still say no. Three grades: PROVEN, BROKEN, and UNPROVEN for guards that haven't declared a control yet. UNPROVEN counts as a warning rather than a failure, because a gate that's red on day one gets overridden by reflex, and a reflex override has stopped being a signal.

We watched it fail before trusting it. A stub guard that always exits zero grades BROKEN.

Then we pointed it at the guard from the story above, the one printing OK at every boot. We injected a wrong percentage where a real measured one belongs. Exit 0. We injected a cost multiple we'd already struck from our own public copy. Exit 0. A third run, with the correct value in place, confirmed the guard really was reading two live surfaces against the real ledger, so a trivial pass couldn't be mistaken for a catch. It graded BROKEN by its own declaration, and it's since been fixed.

As I write this the runner reports 7 proven, 0 broken, 33 unproven, out of 40 guards. That last number moved while I was drafting, because writing this article involved building another checker and the gate made me declare a control for it before it would let me commit. The unproven count is the honest one, and it's the one we expect to be asked about. It's also the point of the exercise. Before we built this, the answer to "which of your guards can still fail?" was "we assume all of them."

A postscript, because it happened while writing this

Fact-checking this article, I flagged four of its own numbers as fabrications. The branch coverage figures. The Haiku row. The three-model percentages. Our own ablation result.

All four were real. I'd checked them against a news summary of Microsoft's post rather than the post, and against the wrong one of our own benchmark artifacts. The summary leaves out branch coverage entirely and never mentions Haiku, so an absence looked like an invention.

We have a rule for this and I broke it anyway: never relay one party's account of another's work without reading the original. I was one hop further out than I thought, which is the only place that error lives.

Two defects did survive the pass, and both were in the draft's account of our own work. A comparison of mine that read as Microsoft's, and a stale guard count.

And I have to finish that sentence honestly, because the first version of this article went out before I caught either one. It was live for about three hours, saying that a guard "is now marked BROKEN" when we had fixed it that morning, and reporting 2 proven of 37 when the runner said 7 of 40. This text replaced it in place, which is why the two paragraphs above are written the way they are. An article about withdrawing a number published a wrong one first. That is the whole argument for having a correction path that edits at the claim instead of appending a note at the bottom, and it is why every figure here now carries a line in a verification block saying where it was re-derived from.

Sources

Microsoft .NET blog, the polyglot unit testing agent: https://devblogs.microsoft.com/dotnet/polyglot-unit-testing-agent/
The plugin, MIT licensed: https://github.com/dotnet/skills

A stable aggregate is not a stable measurement

Tom Jones — Fri, 07 Aug 2026 17:49:12 +0000

We ran the same benchmark five times against our own production system and got 752, 750, 750, 749 and 749 out of 800.

That is a spread of three items across five runs. Less than half a percentage point. If you saw those five numbers you would conclude the measurement was essentially deterministic, quote the mean, and move on. I nearly did.

Then I compared the runs item by item, and 38 of the 800 items had changed answer between one run and another. Not three. Thirty eight, which is 4.8 percent of the set.

The totals barely moved because the flips cancelled. Something that was right in run 1 and wrong in run 3 is invisible in a sum, and there were enough in each direction that the aggregate came out almost identical every time. The number was stable. The measurement underneath it was not.

What five runs actually showed

run	score	of 800
1	752	94.00%
2	750	93.75%
3	750	93.75%
4	749	93.62%
5	749	93.62%
mean	750.0	93.75%

Same 800 fixed items, same configuration, same scorer, five separate passes through the production path, an hour and a half apart end to end. Temperature was zero, which people reasonably expect to mean deterministic. It does not.

Partitioning the 800 by how they behaved across all five runs:

across all five runs	items	share
all five correct	730	91.2%
all five wrong	32	4.0%
changed at least once	38	4.8%

And the pairwise view, counting items where two runs disagree with each other:

run pair	items that disagree	rate
1-2	16	2.00%
1-3	22	2.75%
1-4	19	2.38%
1-5	21	2.62%
2-3	20	2.50%
2-4	23	2.88%
2-5	17	2.12%
3-4	21	2.62%
3-5	17	2.12%
4-5	16	2.00%

The sharpest example is the pair that disagrees most. Runs 2 and 4 differ on 23 items. Their totals differ by one. If those had been the only two runs I did, I would have written down that the system reproduces to within a single item, and I would have been wrong by a factor of twenty three.

Why this matters if you are comparing anything

The reason to care is not academic. It is that almost every model comparison you read, including the ones I have written, is a difference between two totals.

If your measurement has a 2.88 percent chance of flipping any given item between identical runs, then two systems whose totals differ by less than roughly twice that are not distinguishable by the run you did. You need the gap to clear the noise, and you cannot know the noise without measuring it.

We learned this the embarrassing way first. Earlier we ran a three model comparison twice, one hour apart, same prompts, same temperature zero. The model that came first in run one came third in run two, and the one that came third came first. The gaps were around one to two points. The noise was larger than the gaps. We had a ranking, and it was a coin.

The cheap fix, which is one extra run

You do not need a statistical apparatus. You need a second pass of the identical set.

Run your benchmark twice on the same items. Do not change anything.
Count the items that changed answer. That percentage is your noise floor. Ours is 2.88 percent on this workload.
Refuse any comparison whose gap does not clear roughly twice the floor. Not "note the caveat." Refuse it, in code if you can, so a tired person cannot quote it later.

We put that last rule in the tool. It computes the floor alongside the result and will not bless a claim that sits inside the noise. It has told us no more than once, including on a result that flattered us.

And the thing worth keeping instead

If aggregates are fragile, something has to replace them, and the replacement is better anyway.

Report which items you disagree on, not who scored higher. When we compared three models by task
rather than by total, the partition was stable across runs even while the ranking was not: the same
problems came out contested each time, far more often than chance would put them there. Which problems
a system gets wrong is a property of the system. Who scored higher on the total is a property of
the afternoon.

That is also the more useful finding for anyone actually building something. A leaderboard position tells you which one to pick. A partition tells you where picking one is not enough.

The uncomfortable part

I want to be honest about where this leaves our own numbers, because it would be convenient to publish this as a lesson for other people.

It applies to us. Every benchmark result we have quoted from a single run is a number with an unstated uncertainty, and some of the differences we have found interesting in the past are inside the floor we just measured. The 2.88 percent is not a result about other people's evaluations. It is the error bar on ours, and we did not have it until this week.

The five runs cost about two hours and a few dollars. The thing they bought was not a better score. It was knowing how much of the score was real.

We stopped choosing models and started choosing what to measure

Tom Jones — Fri, 07 Aug 2026 17:49:09 +0000

The default way to use an LLM is to pick one and send it everything. That is what the integration looks like, it is what a model marketplace is shaped around, and it is what most agents I read about do. Pick the good one. Send it the work.

We run a routing and verification gateway, so we have had to measure a lot of models against the same tasks with money attached. I want to put down what our own data keeps saying, including the parts where it makes us look bad, because the conclusion is not "we found the better model."

We ran the experiment everyone runs to pick a model. We ran it twice. We got a different winner.

Three models, 164 HumanEval+ problems, temperature 0, one completion each. Then the identical thing again an hour later. Nothing changed between the two runs except time.

model	run 1	run 2	rank
qwen3-235b	89.6%	92.1%	3 -> 1
gpt-oss-120b	91.5%	90.2%	1 -> 3
qwen3.8-max	91.2%	91.6%	2 -> 2

Each model is scored over the problems it actually answered. Two of them answered all 164 in both runs. qwen3.8-max timed out on 4 and then on 9, so its denominator is 160 and then 155. The partition table further down uses the smaller set where all three answered, so its percentages are not directly comparable to these.

First and last swapped. The model we would have picked on Tuesday is the one we would have rejected on Wednesday, and the new frontier model that lost to our cheap tier by 1.3 points beat it by 1.4 the second time.

Temperature 0 is not determinism, and 164 problems is not enough to separate models sitting two points apart. We can put a number on how much of this is flicker: run the same model over the same problem twice and it changes its answer 3.7 to 5.2 percent of the time. Any ranking gap smaller than that is not a result. Ours were all smaller than that.

But something did survive both runs, and it is the more useful thing.

Stop asking how many each model got and ask which ones.

	run 1	run 2
all three correct	130	130
all three wrong	2	2
they DISAGREE	28 (17.5%)	23 (14.8%)

Roughly 15 to 18 percent of the time, the model you picked decided whether you got a right answer.
Note what the same table says about the other 82 percent: all three models agree, 130 of them right
and 2 of them wrong together, and the choice is irrelevant. This is a claim about a fifth of the work, not about all of it.

The reason to believe the fifth is real rather than flicker is not the size of the number, it is that it lands on the same problems both times. Twenty four problems were contested in the first run and twenty three in the second, and seventeen of them are the same problems. If the disagreement were noise you would expect about four to coincide. Ten of them are contested in both runs without a single model ever changing its own answer.

The ceiling is the part I would think hardest about. The best single model scores about 92 percent. If you could always take whichever of the three happened to be right, you would score 98.8 percent, and that number was stable across both runs too. Picking one model, even correctly, leaves about six points on the table.

A leaderboard cannot show you any of this. A leaderboard is column totals, and all of it lives in the rows.

Capability is shaped like the task, not like a ranking

Our verification witness is llama-3.3-70b, pinned to one provider. On 400 standard tool-calling tasks it scores 382, which is 95.5 percent, and matches our primary path. On an adversarial set built to tempt it into calling a tool when it should decline, it scores 30 of 40.

I have to be careful with that 40, and I only learned why today. It is four distinct trap shapes, each parameterised ten ways. The replicates swap a city or a number; they do not add a fifth kind of trap. So the honest denominator is four, not forty, and I am not going to put a confidence interval on it. What the instrument legitimately supports is: across four adversarial tool-selection shapes, parameterised ten ways each, it failed ten times.

The interesting part is not the 75 percent, which I now think is close to meaningless. It is that all ten misses are the same verdict. Every one is a false call: it invoked a tool where the correct behaviour was to abstain. So the model matches the primary at doing the thing and fails it at knowing when not to. Those are different abilities, they are not correlated the way a single score implies, and only one of them was on the benchmark.

If you are building on a model because it scored well at a task, check whether the thing you actually need is the task or the restraint.

The price on the page is not the price

We screened a cheaper candidate for our cheap tier. Advertised at 0.05 and 0.20 per million against our shipped tier at 0.07 and 0.27, roughly 30 percent cheaper on both sides.

On a real tool call it emitted 132 completion tokens, of which 123 were reasoning tokens that we paid for and never saw. Its effective output price came out around fifteen times its sticker, making it roughly ten times more expensive than the tier it appeared to undercut. The tool call itself was correct. The capability was real. Only the economics were inverted.

This is one call, so treat it as a mechanism rather than a rate. The mechanism is enough: compare cost per successful result, never cost per million tokens, and read the reasoning token count before you believe any price comparison.

You may not be choosing a model at all

Same model, same 400 prompts, same scorer. Pinned to a single serving provider it scored 95.5 percent. Unpinned, letting the marketplace route it, 78.2 percent. The model was held constant and the score moved 17.3 points.

For scale, adjacent models in our own sweep differ by 0.5 to 6 points, so the routing spread was larger than the entire model spread we were trying to measure.

And then we checked whether that was a law, because it would be a convenient one for us. We ran the
same pinned-versus-unpinned experiment on gpt-oss-120b. It moved 0.6 points, 65.2 percent pinned
against 65.8 unpinned. So this is not "marketplaces degrade models." It is a property of the specific
model and provider pair, and the actionable version is that you have to check yours rather than assume
either way.

The newest one is not automatically the answer either

Qwen 3.8 came out yesterday. We benchmarked it the same night against the two models it might replace.

And this is where I have to be careful, because the first version of this section said "it did not beat our cheap tier" and the second run says it did. On accuracy it is indistinguishable, in both directions, and I am not going to pretend otherwise in either.

What did not move between runs is the part that actually decides it. It cost 64 to 76 times more per solved problem, depending on the run, priced at the marketplace list rate rather than at what the vendor's own API would have charged us, which we did not read. It was slower at the median in both. And its timeouts more than doubled, 4 then 9 out of 164, with a worst case over 200 seconds. The cost multiple moves because it is a ratio of two noisy things, so treat it as a band and not a constant, which is the same rule this piece applies to everything else.

So the reason not to adopt it is not that it is worse at the task. It is that it is indistinguishable at the task, dramatically more expensive, and less reliable, and those three are all stable across runs while the score is not. The score was the only thing I originally judged it on, and the score was the only thing that turned out to be noise.

So what do you do instead

If models differ by task rather than rank, then "which model is best" is optimising the wrong thing. The unit we route on is the task shape plus a check: send the request to whatever suits its shape, serve the cheap answer when something independent agrees it is right, and escalate only when the check disagrees. The model becomes a swappable component, and being wrong about one becomes survivable.

And the uncomfortable part, because leaving it out would make this an advertisement. Our own escalation target never cleared the cheap tier it exists to rescue on code, in either run. On the first run it scored below it, 90.0 against 92.5. On the second the two tied at 92.9. By this piece's own rule I cannot report either of those as a ranking, so the honest version is the weaker and worse one: we have no evidence that the model we escalate to is better at code than the model it is rescuing. That is not what an escalation target is for. A gate can only be as good as the thing it routes to, so the thing we are fixing is not the gate, it is the ladder, and last night's benchmark was us checking whether the new frontier model was the fix. It was not.

That is scoped to code, and I should say so rather than let it sound general. This benchmark did not test the tool path at all, and our own data has the ordering of these tiers inverting by task shape. Which is the whole argument again, made against us this time: the rung that looks wrong on one task shape is not automatically the wrong rung.

Caveats, in full

The 98.8 percent ceiling assumes an oracle that always picks the model that happened to be right. No router has that, ours included, and we have not measured what share of those six points our own routing actually captures. Read it as the size of the prize, not as a result.

Single run per arm on the 160 problem comparison, temperature 0, one completion each. Temperature 0 was not deterministic for us: two runs of the same 8 problems gave different results, so treat small gaps as noise, including ours. The 95.5 versus 78.2 pair is 400 per arm, single run, no confidence intervals. The trap set is four shapes replicated ten times, not forty independent cases, so treat it as four observations and ignore any interval anyone computes on n=40, including one of ours that is now struck. The pricing finding is one call. Our benchmarks are code and tool calling and they are not your workload.

None of this says frontier models are overrated. We escalate to one and would not ship without it. It says that after measuring a lot of them against the same tasks, the most useful thing we learned was not which one wins. It was that they fail in different places, and the difference between them is bigger than the gap in their scores.

I built a page that only publishes numbers surviving our own checks. It caught me inside the hour.

Tom Jones — Sun, 02 Aug 2026 20:07:11 +0000

I run a verified inference endpoint. What we sell is checking: we verify results before serving them, and we say so when we cannot.

Every claim defect we shipped this month had the same shape. A number was measured honestly, the configuration underneath it changed, and the number stayed on the page. Our benchmark rows were measured on model tiers we retired weeks earlier. Nothing was fabricated. Everything was stale.

So I built a page that cannot go stale. It is generated from the live gateway config, and every badge on it is derived rather than written. If a measurement's configuration premise no longer matches what is running, the page strikes the number and says why.

It went live. Within the hour it caught me publishing two numbers our own ledger explicitly withholds.

What the page does

For each path through the system it states four things:

What "good" means on that path, and whether a real oracle checked it. Tool calls are checked against the schema. Code runs against your tests in a sandbox. Free prose gets nothing, because there is nothing to run.
The measurement, with n, interval and date.
Whether that measurement still describes the running system, graded automatically against live config.
Whether it passed adversarial review, and if not, that appears on the page next to the number.

The third item is the whole point. A number and a config are two facts that drift apart silently, and no amount of care fixes that. A generator that reads both cannot lie about the gap.

The catch

I put our tool-calling and trap-set results on it. Both are good numbers. Both are marked in our internal claims ledger as not cleared for external citation, owes adversarial verify.

I had written the page whose entire purpose is publishing only what survives our checks, and I had not run the check.

The first fix was a blocklist for that one phrase. Also wrong. Our ledger withholds rows in at least six phrasings, and a blocklist misses the seventh. For a public claim the gate has to fail closed: a figure publishes only if its ledger row exists and carries no hold marker. Over-withholding costs a number on a page. Under-withholding costs a retraction.

Failing closed immediately caught two more, subtler ones. Two figures had no ledger row at all, because I had sourced them from an internal handoff instead of the truth layer. One was a 30/40 result. Our ledger also contains a 30/40, from a completely different run on a different model. Same number, different measurement. That is exactly the confusion an uncited figure invites.

Then the reviewer rejected my draft

The remaining claim needed real adversarial review, so I commissioned one and briefed it to refute rather than agree. It came back do not publish, with three specific defects:

I wrote that our agreement gate accepts a wrong answer "roughly a quarter to a third of the time." That range was two underpowered subgroups (6 of 25, and 5 of 15) whose confidence intervals almost entirely overlap. It reads like a stable operating characteristic. The data supports one pooled binomial: 11 of 40, 27.5%, Wilson 95% interval 16.1 to 42.8.
I quoted "1.7 to 3.5% on code." It has the shape of a measured interval. The reviewer said the evidence was not in my brief, which would normally mean I under-briefed it. So I checked instead of accepting. The figure appears three times in our ledger with no n, no interval and no date anywhere. The gap was real.
I wrote "near zero on arithmetic." That is residue from a claim we retracted publicly after a reader caught it from our own article's text. No clean replacement exists, so the phrase cannot appear at all.

I took all three. I refused one instruction: it wanted me to name an internal threshold, which is recipe rather than result. It did not know that because I had not briefed it on our disclosure policy. Same failure, other direction.

The green check that could not fail

While extending our claims gate to cover more surfaces, I did something I should have done when it was written: I injected the defect it exists to catch.

Seven injections. Five wrong percentages, a struck cost multiple, a retracted compounding claim. It fired on none of them. Its tolerance covers most of the range it checks, so almost any number is "backed" by something. It had been printing OK every morning for weeks.

It was not lying. Its own output says "values checked, not claims," and a comment in its source says the same. The defect is one level out: a green line on a status board is read as protection, and nobody re-reads the caveat.

A guard you have never watched fail is not a guard. It is a habit.

The rules that transfer

A correction in your repo is not a correction in public. Our live site was still serving retracted figures while every internal instrument reported the sweep as done, because every internal instrument greps the repo. Verify a public claim by fetching the public URL.

Your measurement tool is an instrument, so verify it first. Our cost script summed one of three cost columns. It reported one cent where the truth was $1.73, and it was the exact script our ledger's cost row cited.

A readability filter can eat an error. I piped a remote script through grep -v to hide a deprecation warning. A database auth failure exited zero, and that became STATUS: Success, then a fleet-wide OK across both production boxes. The lie got more authoritative at every hop.

Placeholders are claims. Our dashboard rendered "$1,284 saved this month" as static HTML before live data arrived. A brand-new account with zero usage would have been shown a fabricated saving. I found it while opening signup to the public, one step before strangers would have seen it.

Publish the path where you are weaker. Our trap set is 200 of 200 on the primary path. Under failover it is 30 of 40, and every miss is the same verdict: it calls a tool where it should decline. Both numbers are on the page. The second is the more useful one.

Go and check it

The page is live at tirtha.ai/capabilities. Seven paths, what checks each one, what it measured, and the review state of every figure.

To try it rather than read about it, tirtha.ai/verify runs live against our API with no key and no signup. Three prefilled adversarial cases, and you can watch it decline a call instead of inventing one.

To run it on your own workload: go to tirtha.ai, click Get an API key, sign in with Google, and start. Free while we are in beta. No card, no email, no sales call. You get 100 requests a month and it stops rather than surprises you.

If you find a claim on that page that does not survive, I want to hear it. That is not rhetorical. The last time someone did, they were reading our own published text, and they were right.

We built a router to predict when a cheap model is enough. It does not work.

Tom Jones — Wed, 29 Jul 2026 02:10:58 +0000

If you serve a model cascade, escalation is your cost dial. Not your model choice, not your prompt, not your context window. The single number that moves your bill is what fraction of requests climb to the expensive tier.

So the obvious thing to build is a router: look at the incoming request, predict whether the cheap model will get it right, and only pay for the expensive one when the answer is no. We built that. It does not work, and the reason turned out to be more interesting than a working router would have been.

Publishing it because a negative result with a mechanism is worth more than a positive result with a story, and because the last two times we published numbers, readers found the flaws faster than we did.

What we built and what it scored

Our gateway already embeds every prompt, because the semantic cache needs a vector. Reusing that vector to predict difficulty is free. That is exactly why we did it, and that turned out to be the founding design flaw.

Trained on 539 real coding tasks labelled by actual execution (428 easy, 111 hard):

held-out AUC                       0.594
5-fold CV                          0.55 to 0.57   (one fold below chance)
best threshold                     scores exactly what "never escalate" scores

A classifier that cannot beat a constant policy is not a weak classifier. It is not a classifier.

It is a feature limit, not a tuning problem, and we made it prove that

The tempting read is underfitting or overfitting, so we swept L2 regularisation across four orders of magnitude, 0.01 to 100, with cross-validation at every step. Train AUC stayed at 0.94 to 0.98 while test AUC stayed at 0.58 to 0.59, at every single setting. The gap never closed. A gap that survives four orders of magnitude of regularisation is telling you the features do not contain the signal.

Then the result that actually explains it. We threw together 11 crude surface features, things like length and token counts, no semantics at all:

11 crude surface features          AUC 0.610
1024-dimension prompt embedding    AUC 0.552
both together                      AUC 0.609   (the embedding adds nothing)

Eleven numbers you could compute with a ruler beat a 1024-dimension semantic embedding. The reason is simple once you see it: the cache vector encodes topic, not difficulty. It is built to answer "have I seen a question like this before", and it is good at that. Two questions about sorting a list sit close together in that space whether one is trivial and the other is subtle. We reused it because it was free, and free was the whole problem.

Two measurement traps, which are worth more than the router

One. Scoring a cost-saving router on accuracy alone marks it failed by construction. A router that
saves money by sending easy work to a cheap model will, correctly, be slightly less accurate than
always escalating. Our first yardstick printed DO-NOT-SHIP on a router that actually had skill. The
control you need is random routing at the same cost, not the expensive model. If you cannot beat
a coin flip that spends what you spend, you have nothing. If you can, you have something, even if your
raw accuracy went down.

Two. The ceiling is not "always escalate". We assumed the expensive model was the upper bound and
a perfect router would approach it. It is not. On our 539 tasks, escalating everything to the top tier
rescues 39 answers the cheap model got wrong and breaks 23 it had already got right. Net +16.

always escalate      82.4%
a PERFECT router     86.6%

A perfect router beats always-escalate by 4.2 points, because it also knows when not to escalate. That headroom is real and no gate-local heuristic captures it. If you are benchmarking a cascade against "just use the big model", you are measuring against the wrong ceiling.

What is still alive

Showing the router the cheap model's draft rather than only the prompt does better: AUC 0.640, and 81.8% accuracy at 36% escalation against 80.5% for an equal-cost random control. That is the right shape. But permutation testing gives p=0.0375 unadjusted and roughly p=0.30 after Bonferroni, so the honest verdict is promising, not proven. We are not going to claim it until it survives a pre-registered run.

Which makes sense mechanically: the draft carries evidence of difficulty that the question alone does not. A model that is about to be wrong often looks different while being wrong.

The questions, and this is why I am posting

I would rather hear from people who have hit this than keep guessing.

What features actually carry difficulty before generation? We have shown that a topic embedding
does not, and that crude surface stats beat it. That is a low bar. What clears it?
Is pre-generation difficulty prediction possible at all, or is the draft the earliest honest
signal? It is entirely plausible that "will this model fail" is not a property of the question,
only of the interaction. If someone has evidence either way I would like to see it.
If you run a cascade in production, what do you actually route on? I suspect the honest answer
for most teams is a hand-written rule about request type, and I suspect those rules do better than
our classifier did. That would itself be a finding.

Numbers, labels and the negative result are ours to share. If you want the setup in more detail, ask and I will write it up.

Tokens Are Not the Unit

Tom Jones — Sun, 26 Jul 2026 20:04:49 +0000

Every AI provider publishes a price in dollars per million tokens. Every comparison table ranks by it. Every build-versus-buy spreadsheet runs on it.

That number is misleading, and not in a small way. It can be wrong by 10x, and wrong in the direction that makes the expensive option look cheap.

This piece explains seven things about the real cost of AI work. Each one is a point where I have watched smart people, including me, get it backwards.

1. The sticker price is not the price

THE POINT: you are billed for tokens the model produces while thinking, even though you never see them and cannot use them.

Modern models emit "reasoning tokens." The model works through the problem, and that working-out is generated text. You are charged for it. On many APIs you never even receive it.

Here is a real evaluation we ran. We were considering swapping our low-cost tier for a cheaper model:

Current model:   $0.07 in / $0.27 out per million tokens
Candidate model: $0.05 in / $0.20 out per million tokens
On paper: about 30% cheaper on both sides.

Obvious swap. So we sent it one real request. It answered correctly. Then we read the billing detail:

274 prompt tokens
132 completion tokens
  of which 123 were REASONING tokens
  of which   9 were the actual answer

You are billed for all 132. You can only use 9.

Do that arithmetic and the effective price of a useful output token was $2.93 per million, which is 14.7 times the advertised rate. The "30% cheaper" model was roughly ten times more expensive than the one it appeared to undercut.

We verified this against the provider's own reported cost for that call and the two agreed exactly, so this is not a units error on our side.

One honest limit: that was a single call. The ratio will move with how hard the question is. Treat the mechanism as the finding, not the specific 14.7. The direction does not move.

There is a second trap in the same family. On some models the reasoning goes into a separate field and the content field comes back empty. Under a tight output limit, the reasoning eats the entire budget before any answer is produced. We saw exactly this: at a 100 token limit, empty response. At 200, a correct one. An empty response looked like the model was incapable. It was actually a budget symptom. If we had trusted the first reading we would have discarded a model that works fine.

WHAT TO DO ABOUT IT: before you believe any price comparison, send one real request and read completion_tokens_details.reasoning_tokens in the response. Then compute dollars per useful output token. If your provider does not expose that field, you cannot actually price the model, and you should say so out loud in the meeting.

2. Cheap models are not uniformly worse. They fail differently, and the difference is the whole story.

THE POINT: the standard mental model, that models sit on one line from dumb to smart, will get you hurt. Two models with nearly identical scores can behave completely differently when it matters.

Most people picture a single quality axis. Expensive is smart, cheap is dumb, pick a point that fits the budget. If that were true, choosing a model would be a budget exercise.

It is not true. Here is a field of models on 800 tool-calling tasks. First, plain accuracy:

frontier model A   95.9
frontier model B   95.1
ours               ~94
strong open model  93.6
cheap model C      93.0
cheap model D      91.0

Read that column alone and the cheap models look like a steal. Four percentage points for a fraction of the price.

Now here is a second column. Inside those 800 tasks are deliberate traps: requests where the correct behavior is to decline, because the right tool is not available or the request is malformed. This column is the percentage of traps handled without grabbing the wrong tool:

frontier model A   100%
ours               100%
strong open model  100%
cheap model C       90%
cheap model D       52%   <-- and this model scored 91 on accuracy

Stop on that last line. A model that looks four points behind on accuracy will pick up the wrong tool half the time it is baited.

Think about what that means in a system with real permissions. That is not "slightly less accurate." That is a model that will confidently call delete_records when it should have said "I do not have a tool for that." The accuracy average washed the single most important behavior completely out of view.

WHAT TO DO ABOUT IT: build a small set of tasks where the right answer is to refuse, and measure the refusal rate separately. Never let it be averaged into an accuracy score. If a vendor cannot tell you what their model does when it should do nothing, you do not have the number that matters.

3. Because they fail differently, sorting beats upgrading

THE POINT: if cheap models were uniformly worse, your only lever would be paying more. Because they fail in specific, predictable ways, you have a much better lever: send each task to the weakest thing that can actually do it.

This is routing, and the important thing about routing is that it is a sorting problem, not an intelligence problem. Sorting is cheap. Intelligence is expensive. Any time you can convert the second into the first, you win.

A concrete picture over thirty days of our real production traffic, 68,369 requests:

Priced at frontier rates, this exact traffic:  $166.25
What it actually cost us:                      ~$46 to $51
Gross margin:                                  about 70%

But the composition is the part worth internalizing:

84% of our total cost was the escalations to the expensive model. Everything else, all the cheap serving, all the infrastructure, was rounding error next to it.

That means the cost dial is not which model you picked, and not the price you negotiated. It is how often you have to escalate. A 10% reduction in escalation rate does more for your bill than a 10% discount from any vendor.

WHAT TO DO ABOUT IT: instrument your escalation rate before you optimize anything else. If you do not know what fraction of your requests need the expensive model, you do not know what your system costs or why.

4. Verification is what makes cheap safe

THE POINT: routing on its own is a gamble. What turns it into engineering is being able to cheaply check whether the cheap answer is right.

Here is the asymmetry the whole approach rests on:

Producing a correct answer is expensive. Checking one is often very cheap.

You already know this from normal software. Writing the function is the hard part. Running the tests is the easy part. That asymmetry does not disappear when a model writes the function, and it is the thing you should be exploiting.

Cheap checks available to you: run the tests. Check the types. See if it compiles. And one more that people underuse: ask two independent models and see if they agree.

We measured that last one. On questions with no tests to run, using two endpoints we believed were independent:

WITHDRAWN, see the correction below:
  When the two agreed, probability the answer was wrong:  0.00   (n=160)
  How often they agreed:                                  76%

What re-measurement found, once the gate was tested per workload shape:
  short-answer arithmetic (what the original probe used)   ~0
  code, with executable tests as ground truth              1.7% to 3.5%
  faithfulness judgement, real off-trap data              27.5%   (95% CI 16.1 to 42.8, n=70)
  how often the two agreed on that data                     61%

CORRECTION, added 2026-07-27. Do not use the 0.00 above. A reader (@anp2network) took the number apart in the comments, and our own follow up measurements agree with them on every point. Leaving it uncorrected would be the exact failure this piece complains about elsewhere.

One. Zero wrong in 160 does not bound the error rate at zero. For a zero numerator the 95% upper bound runs about 3/n, roughly 1.9%. With the gate passing 76% of traffic that permits something like 1.4% of everything served to be wrong and never checked. The point estimate was 0.00. The bound never was.

Two. That probe used short answer arithmetic, where the cheap tier is essentially never wrong. A gate cannot be measured where the primary does not fail, and every row showing the primary correct should have been the tell. Re-run on code, using the same agreement check and the benchmark's own tests as ground truth, the gate leaks low single digit percent, not zero.

Three. Agreement substitutes for correctness only if the two endpoints fail independently, and independence is a property of what the provider is serving, not of your code. Ours quietly stopped being independent. The witness was selected by model name, and all three of our cheap backends turned out to be the same underlying model under three different spellings. A same model pair agrees about 88% of the time against 70 to 75% for a genuinely different one. That is close to a rubber stamp, and a rubber stamp also reports 0.00.

Those 160 cases spanned four task families including deliberately hard traps. We originally read the traps as reassurance. That was backwards. A trap works by shared structure in the input, shared structure is exactly what correlates two endpoints, so the gate thins out on the cases that cost the most to get wrong.

What the gate actually buys you, stated honestly: it lowers the rate at which you serve a wrong cheap answer. It does not drive it to zero, and how far it lowers it depends on the two endpoints really being different. Verify that they still are, and keep verifying, because the answer can change without you deploying anything.

The asymmetry this section opens with still holds. Checking is cheaper than producing, and routing by a check still beats routing by hope. What does not hold is treating a single agreement number, measured once, on the wrong shape of task, as a guarantee.

WHAT TO DO ABOUT IT: for every class of work you send to a model, write down how you would check the answer cheaply. If you cannot answer that, that class of work is not a routing candidate yet, and that is useful to know before you build.

5. The benchmarks actively punish the behavior you want

THE POINT: the leaderboards score a correct refusal as a failure. If you pick models by leaderboard, you are selecting against safety.

This one is worth being very explicit about, because it is counterintuitive and it is expensive.

The most valuable behavior in a production agent is declining to act when the request is ambiguous, malformed, or outside its remit.

The major agent benchmarks score task success. A refusal is a failed task. They award exactly nothing for "correctly declined to do the dangerous thing."

So a system tuned for production safety scores WORSE on the headline number than a system that always attempts and is occasionally catastrophically wrong.

Sit with that. The public number that everyone compares is, in this specific and important respect, pointing the wrong way.

WHAT TO DO ABOUT IT: run the benchmarks anyway, because your customers and your competitors will. But report the wrong-action rate right next to the task-success rate, every time. And know both numbers privately before anyone runs them at you publicly.

6. Your workload shape decides your economics, not your architecture

THE POINT: a single blended cost-per-request number hides the variable that actually determines whether this is profitable for you.

Two workloads through our identical system:

Tool-calling work:  almost never needs the expensive model
Coding work:        about 57% escalated on fresh problems

Same code. Same models. Same prices. One of those is enormously profitable and the other is thin.

Our healthy margin exists partly because our traffic happens to be tool-calling heavy. A customer whose work is mostly fresh coding would see materially worse economics, and it would be dishonest of us to quote them our number.

I am saying that plainly because the whole industry quotes blended numbers, and a blended number is a hidden assumption about your mix.

WHAT TO DO ABOUT IT: when anyone shows you a cost-per-request for an AI system, your first question is "on what mix of work?" If they do not have an answer, the number describes their traffic, not yours. And measure your own mix before you forecast anything.

7. Publish the ceiling honestly, because someone else will find it

THE POINT: we are at parity on code, not ahead, and saying so is the only version that survives contact with a skeptic.

On a clean, cache-free run of a standard coding benchmark:

Our cascade:        92.1
Frontier model A:   92.7
Frontier model B:   93.3
The bare cheap model alone: 81.1

Same harness for all four. We are slightly under the frontier. The architecture adds about 11 points over the cheap model by itself.

We were tempted by a "beats the frontier" line. The measurement did not support it, so we do not use it. Parity is the honest word, and parity at a fraction of the cost is the actual product.

WHAT TO DO ABOUT IT: the number that does not flatter you is the only one worth publishing, because it is the only one that holds up when a customer reruns it. A claim you cannot survive being checked on is a liability with a delayed fuse.

8. The hardest part is not building the system. It is trusting your own measurements.

THE POINT: a wrong measurement is more dangerous than no measurement, because it comes with confidence attached.

This is the one I would most want a reader to take away, because it applies whether or not you ever build any of the above.

In a single working session, we chased seven separate alarms. All seven were broken instruments, not real problems. A parser reading five rows of a seventy-nine row file and reporting a catastrophe. A checker matching error strings against its own console output and finding "errors" it had printed itself. A meter reporting 135% of a hard limit when the true figure was 27%.

Three more from production, all instructive:

We were dropping two thirds of every traffic burst and could not see it. Our server had a careful queue that answered overload with a polite "busy, try again," which is exactly what an aggregator wants. But the operating system's own accept queue underneath it was at its default of five connections, so any burst deeper than five was refused by the kernel before a single line of our code ran. Every load test we had ever written sent exactly as much traffic as the server was willing to admit, which made those tests structurally incapable of finding this. The test design guaranteed the blind spot.

Then we fixed it wrong, in a way that looked right. A configuration flag said a feature was off. The deployed code was an older version that could not express "off" and instead disabled the feature completely. Latency improved. Of course it did, because doing nothing is fast. Every dashboard was green while a feature was 100% dead.

And we measured a worst case at 17 seconds using 10 samples. Later we measured the identical component with 828 samples. The real worst case was 126 seconds. Ten samples gave us an accurate median and a completely wrong tail.

WHAT TO DO ABOUT IT, and these are the four habits that would have caught every case above:

Change one variable at a time. If your fix changes three things and it works, you have learned that the bundle works. You have learned nothing about why, and you will keep the two useless changes forever.
Turn your fix off and confirm the problem comes back. A test that only ever passes has told you nothing.
Match your sample size to the statistic. A median settles in tens of samples. A worst case needs hundreds. A tail measured with 10 samples is not a cautious estimate, it is a wrong one that reads as cautious.
Chase the gap you cannot explain. When a number is slightly off and you invent a plausible reason to dismiss it, that reason is usually the bug. Twice in one week the explanation I reached for ("network overhead") was covering a real defect.

The one page version

If you remember nothing else:

Price in dollars per successful result, never per million tokens.
Read the reasoning token count before believing any quoted price.
Measure the wrong-action rate separately. Averages hide the failure that matters most.
Track your escalation rate. It is the cost dial, not your model choice.
Verify cheaply so you can generate cheaply. Tests, types and independent agreement all cost less than intelligence.
Quote economics per workload shape. A blended number is a hidden assumption about someone else's traffic.
Verify the instrument before acting on its number, and be most suspicious when the number is good.

None of this requires owning a frontier model. It requires taking measurement seriously, which is rarer, and considerably cheaper.

Edits to this article

2026-07-27. The agreement figure this piece originally published, a 0.00 probability of a wrong
answer when two endpoints agree at n=160, was withdrawn. A reader, @anp2network, took it apart in the
comments and was right on every point: a zero numerator bounds the error near 3/n rather than at zero,
the probe used short-answer arithmetic where the cheap tier is essentially never wrong so the gate was
measured where the primary does not fail, and independence between two endpoints is a property of what
a provider is serving rather than of your code, so it can decay with nothing in your system changing.
Ours had decayed exactly that way.

2026-07-28. Replaced the withdrawn 0.00 inside the table itself instead of leaving it standing
above a correction, and added the re-measured numbers by workload shape. The honest summary is that
P(wrong when they agree) is not one number. It is close to zero on arithmetic, 1.7 to 3.5 percent on
code checked by executable tests, and 27.5 percent on faithfulness judgement. Quoting any single one of
those as the property of the gate is the mistake this article made the first time.

Also worth stating plainly, since the same commenter suggested it: we now escalate a sample of the agreed cases to the expensive model anyway and score whether it concurs, which turns a one-time n into a number that moves when a route changes behind either endpoint. That is running in production as of this edit. We will publish what it says, including if it is unflattering.

Your company already runs the hardest parts of an AI agent platform

Tom Jones — Sun, 26 Jul 2026 20:03:50 +0000

Every AI agent session starts as a fresh mind. It has no memory of yesterday, no memory of the correction you gave it last week, and no idea what your company has already decided.

That is fine for one developer with one project. It falls apart the moment you have five teams, because now the things that must be true everywhere have to somehow be true in every session, of every project, forever.

The obvious answer is to write a good prompt and paste it into each project. That answer fails twice. It drifts out of sync the first week, because six copies of a rule are six rules. And every pasted line is paid for in every session forever, so the document that encodes your standards becomes a tax on all future work.

The better answer is the one your company already uses for people: inheritance, scoped by the org chart, with a channel for asking a human.

Here is the architecture, and then the part where I show you the data on which half of it worked.

Three tiers, inherited rather than copied

Tier one is the constitution. One document at the root, loaded automatically by every session in every project beneath it. It holds identity and the things that are never negotiable: what counts as evidence, what "done" means, the safety boundaries that are legal rather than stylistic.

Mine is capped at about forty lines, and the cap is the design, not tidiness. Every line here is read by every session of every project, forever. A constitution that grows is a bill that arrives every time anyone starts work. If a rule is not needed by every project before it knows what it is doing, it does not belong in tier one.

Tier two is the project constitution. What is true for this project and nothing more. It inherits tier one for free, so it never restates it.

Tier three is everything else. In my case, several hundred documents. None of them load at startup.

Notice that the cost runs opposite to the volume. The smallest tier is the most expensive because everyone pays for it always. The largest tier is nearly free because nobody pays for it until something asks. Most people build this upside down: they put the huge context at the top and wonder why every session is slow and expensive.

The mechanism that makes tier three work

A pile of documents nobody reads is not memory. What turns it into memory is a short table in tier two that maps an action to the thing you must read before doing it. Before you touch billing code, read this. Before you change anything a customer sees, read that.

The agent does not go looking for knowledge. The knowledge is delivered at the moment of the action that needs it.

This distinction sounds like a detail. It is the entire difference between the half of my system that worked and the half that died, and I have the numbers.

Where the humans go

An agent organization needs two human channels, and they are not the same thing.

Escalation is the agent saying "I need a decision that is not mine to make." A production change, a spend commitment, anything irreversible.

Calibration is a human saying "that was wrong, and here is the shape of the wrongness." This is the more valuable of the two and the one most setups drop entirely. A correction that lives only in a chat window is a correction you will pay for again in three weeks.

Both should run through the channel your organization already lives in. For most companies that is Teams. Not because Teams is special, but because a notification surface people already have open is the only kind that gets read. I have built beautiful dashboards for this. Nobody opened them, including me.

Threads matter here more than they look. A thread gives a correction a durable home attached to the thing being corrected, which is exactly the raw material you need if you ever want to fold that correction back into tier two.

The part I did not expect to be so clean

Once you accept the org chart as the scoping model, most of the hard problems stop being yours.

Identity and permission: your existing directory. An agent becomes a principal in the directory you already govern. It joins groups, it inherits those groups' rights, and it is revoked exactly the way a leaver is revoked. The alternative is a second permission model that lives beside the real one and drifts away from it, which is the security incident everyone eventually writes up.

This also answers the question every risk team asks first, which is not "is the AI accurate" but "what can it reach, and who decided that." If the answer is "the same groups that govern the humans doing this work," you are having a very different conversation.

The tiers: your existing document storage. Tier one is a site everyone reads, referenced rather than copied, so a change to the rules is a reviewable, versioned event instead of an edit nobody noticed. Tier two lives with the team that owns the work, so the boundary is one the company already agreed on rather than a folder convention somebody invented on a Tuesday. And personal scope goes in personal storage: private by default, follows the person, cannot leak into shared tiers by accident, because the permission is the boundary rather than a naming rule.

Audit: your existing audit log. Who told the agent what, and when. You inherit the answer instead of building an audit system for agents from scratch.

None of this is a product to buy. That is the point. The boring parts of an agent platform, identity and scoping and audit and a channel humans actually read, are precisely the parts your company solved years ago for its people. An agent organization should inherit them, not reinvent weaker copies.

Signal, and why the privacy design is a data quality mechanism

Information has to flow upward or the upper tiers are fiction. But if individual identity flows upward, you have built surveillance, and surveillance destroys the thing you were collecting.

The resolution is that the signal carries the pattern, not the person.

PERMITTED:      "4 people flagged this standard as unclear this month"
NOT PERMITTED:  "Sarah flagged it 4 times"

Identity is stripped at the team tier. Everything above receives patterns only, and no reverse lookup is possible by construction. That last clause matters: it is a structural property of how permissions and aggregation are built, not a policy somebody promised to follow.

There is also a threshold. A single event is not a pattern, so nothing escalates until several signals of the same type appear. This stops one bad afternoon from being surfaced upward as an organizational trend.

The governing shape is: private truth, then anonymous patterns, then organizational intelligence.

Here is why this is engineering and not ethics decoration. If people believe their personal tier can be used against them, they will populate it performatively. Performative input produces false patterns. False patterns produce confident, wrong decisions at the top of the organization, which is worse than having no system at all. The entire intelligence value of the system depends on the bottom tier being genuinely private, so the privacy guarantee is load-bearing infrastructure.

The loop that makes it improve itself

A memory system that only stores is a filing cabinet. What makes it compound is routing the questions.

When someone asks the assistant about a standard or a policy:

If a calibrated answer exists, answer immediately with a citation to the exact version, and log the question.
If no answer exists, do not improvise one and move on. Flag it as an open edge case and route it to whoever owns that standard.
If the same question appears repeatedly, that is not several confused people. That is one ambiguous standard, and it should be escalated as such.

That last inversion is the valuable one. Three people asking the same question about a rule is evidence about the rule, not about the people. Most organizations read it the other way around and send the three people to training.

The accumulating question log turns out to be a real-time map of where the organization is confused, produced continuously, with no survey and no focus group.

Now the half that failed

I added one more piece: a shared log where each project session ends by writing one line about what moved, feeding a cross-project summary.

Twenty two days later I counted what was actually in it.

Ten entries. Eight of them in the first three days. One more eleven days ago. Then nothing.

All ten came from a single project, out of five. The live one with a real customer has never written a line.

The shared summary, the entire reason the layer existed, was last modified the day it was created. It still confidently describes a project as a shell folder that I retired eight days afterward. It is not stale. It is wrong, and it looks authoritative while being wrong, which is considerably worse than being empty.

There was also a schema drift I never noticed: the timestamp is an integer in the first row and a formatted string in the last. I only found out because the script I wrote to analyze all this crashed on it.

Why it died, and why tier three did not

The design was fine. The trigger was not.

Writing that line was a remembered act. Nothing fired it. It sat at the end of a session and depended on somebody choosing to do it while tired and finished. That works for about three days.

Tier three works because it is bound to an act. Touch the file, get the rule. Nothing depends on anyone remembering the rule exists.

Same system. Same week. Same author. The only difference is what starts the write.

I have a second case that removes any remaining doubt. I built a lookup table for operational answers, seeded it with eight rows, and told the agent to add to it as it went. A day later it had eight rows. Zero added, while that same day's work had produced at least three entries that qualified and filed them somewhere else. The table did not need more rows. It needed a trigger.

The law

Anything that depends on remembering will not happen. Bind it to an act, or it does not exist.

Not because agents are careless. Because every session is a fresh mind with no memory of having agreed to the habit. You cannot build organizational memory out of good intentions distributed across amnesiacs.

And pull-based systems fail in the most flattering way possible. The file is still there. The design still reads well in the diagram. Nothing errors. You do not find out until you go and count the rows, which is why you should go and count the rows.

Two corollaries I paid for:

The read side needs pushing too. I assumed someone would open the summary. Nobody did for three weeks, including its author. A summary nobody is handed is a summary nobody reads. Surface it unprompted at the start of a session, or accept that it does not exist.

Anything derived needs a freshness check. A dashboard that is silently eleven days stale is worse than no dashboard, because it still looks like an answer. Stamp it with when it was generated and make it complain when that gets old.

If you are building this

In the order I would do it again:

Write tier one and cap it. Forty lines. If you cannot say it in forty lines it is a tier two rule wearing a disguise.
Build the trigger table before the documents. The delivery mechanism is the product. The documents are inventory.
Set the privacy boundary before the first signal. You cannot retrofit anonymity onto a system people already distrust.
Put the agent in your directory. Groups, not bespoke config. Deprovisioning should already work on day one.
Wire escalation into your chat tool before you need it. The first time an agent needs a human should not also be the first test of the channel.
Make calibration cheap to capture. If correcting the agent takes more than a reply, it will not happen, and you will pay for the same correction repeatedly.
Then, and only then, the upward flow. And bind it to something that already happens at the end of work, never to a good intention.

Start with a software team, whatever the eventual target is. Developers already live in this world, already use context files, and already feel the specific pain of losing the reasoning behind decisions. You do not have to explain the concept to them. You show them, and two weeks later either someone says it changed how they work, or it did not, and you have your answer cheaply.

The architecture is not the hard part. Getting it to still be running in three weeks is the hard part, and that is decided entirely by what fires the write.

Everyone is hardening the structure. Nobody is passing down the why.

Tom Jones — Sun, 05 Jul 2026 17:14:28 +0000

Two weeks ago I published a short piece called The Two-Channel Problem (tirtha.ai/research, the perspective panel) about what actually breaks when a forgetful AI agent builds a real product over months. Since then a genuinely good conversation has emerged between people running AI organizations. The nokaze seven-week paper named the cross-conversion gap: the rule file exists, and the agent sails right past it in the exact situation it was written for. The comment thread under it converged on real fixes: action-keyed triggers instead of "notice the moment," completion claims invalid without a re-checkable evidence source, checks that cannot be run by the layer that felt confident writing the claim. We run an AI-operated shop too (one human, a Claude orchestrator, Codex, a headless build fleet), and we built the same walls independently. All of that is right, and trading notes with these teams has sharpened our own guards twice this week alone.

But the conversation is converging on exactly one of the two channels, and the update I owe is about the other one.

What the structure channel fixes

Every fix in that thread strengthens what we internally call the structure channel: hooks, gates, generated status, mechanically forced loading. The structure channel transmits the WHAT. It makes discipline un-forgettable, which matters enormously when your workers wake up with no memory of yesterday.

The failure it cannot fix

Here is the failure it cannot fix. Early on, one of our sessions booted into a perfectly clean room: accurate task state, working guards, honest logs. It executed correctly all day, and it drifted all day. It searched instead of receiving, re-derived things the system already knew, treated the guards as obstacles to route around. Nothing it did was wrong by the letter. It complied without understanding. We started calling that a hollow successor.

So we run a second channel

So we run a second channel. Plain prose, written by each session for the next one, read at boot before any work: who to be here, why the room matters, what the rules are actually FOR, what yesterday's session learned about HOW to work that the commit log cannot carry. Not documentation. A letter.

The operating rule that fell out: structure transmits the what, only words transmit the why. A hook can force a file to load. It cannot make the next mind care what is in it. And an agent that does not understand why a rule exists will satisfy its letter and defeat its purpose the first time the letter and the purpose diverge, which is exactly the shape of most of the incidents this whole conversation is about.

The practical form

The practical form, since this sounds soft and is not:

A welcome document is read at boot, before work, and yes, a hook enforces that it gets read. We use the structure channel to guarantee the soul channel is delivered. What no hook can guarantee is that it lands. That gap is the point, not a flaw in the design.
Every session writes a short transmission at handoff. Not what it did (the log has that), but what it learned about how to work here: which instinct misled it, which boring re-check saved it, what it wishes it had been told at boot.
Corrections carry their origin. Every rule is stored with the incident that created it, so the next session inherits the reason, not just the restriction.

What we observe

The result we observe, for whatever one shop's evidence is worth: sessions that receive both channels correct themselves mid-flight in ways the guards alone never produced. This morning one of ours printed a success sentinel for a merge that had silently failed, caught it seconds later, and said out loud that it was the exact failure shape it had read about at boot. The guard did not catch that one. The letter did.

Write the other half

The teams in that thread are building excellent guards. If you run an organization of minds that forget, the guards are half the inheritance. Write the letters too.

I didn't mean to build this. Looking for testers anyway.

Tom Jones — Sat, 04 Jul 2026 15:55:11 +0000

I'll be honest up front: I'm not a salesman. Everyone has always told me that, and lately even the AI I build with tells me that. So this is not a pitch. This is me asking for help.

I didn't set out to create any of this. It started because I kept having context window issues while building something else entirely, and I went down a rabbit hole. The rabbit hole kept going. Somewhere down there I got tired of watching my AI coding agent send every single request to the most expensive model available, whether the task needed it or not, and I started building a fix for myself.

What's been embarrassing is I thought I had one story. Then the testing led me to another one. Then another. My website has changed multiple times, which has been a lesson learned in public, the hard way. The good news is there are no clients yet, so that's good :)

Where the testing has left me now is this:

As a gateway I am able to send routine coding requests to lower cost models that get verified before you ever see them. The hard ones escalate to a frontier model. On my benchmark runs that was about 1 in 27 requests. The full setup measured around 95 percent on HumanEval+ (n=164) at roughly 8x lower cost per request. I keep re-running these numbers because honestly I didn't quite believe them either. So far they keep holding.

So based on those runs, that works out to making a Fable 5 subscription last up to 27x longer, because most of the work doesn't need that level of cost to be accurate.

What I don't know yet is whether it holds up on YOUR work. Real repos, real deadlines, weird edge cases. That's exactly what I need testers for.

Security, because you should ask: encrypted at rest, TLS in transit, and your code is never training data. I verified the upstream providers do not train on it, and the routine tier runs on open models I host myself. Local models, privacy, security, and a cache that bends the cost curve down even lower over time. That is my goal, my promise, and what I want help testing and proving.

The deal: it's pre-beta, it's free while it is, and I will keep publishing the real numbers whether they flatter me or not. I feel like I have something special here, something people have been asking for. I just need people smarter than me to hammer on it and tell me where it breaks.

I got my coding agent to tie the frontier for about 8x less. Here is the honest benchmark.

Tom Jones — Thu, 02 Jul 2026 21:32:32 +0000

I am a solo founder. I do not have a lab or a team of researchers. I live paycheck to paycheck supporting my family like most people do. I didn't intend to build this. I just went down a rabbit hole and here is the story.

Two things bugged me about the AI coding agents I was using. The first was the cost. Every request, easy or hard, went to the most expensive model available. The second was quieter: I did not actually know where my code was going when the agent wrote it.

So I built something to fix both, and I measured it carefully, because I would rather tell you the honest number than a flattering one. Here is what I found, including the parts that are not flattering.

The idea

Most coding requests are not hard. A cheaper or local model handles them fine. Only a small slice genuinely needs a frontier model. So instead of paying frontier prices on everything, the system routes each request to the cheapest model that can actually do the job, checks the result, and escalates to a frontier model only when the check fails. Verified answers get cached, so work you have done before comes back fast. Router, verifier, frontier backstop, cache. I am holding the engineering details, but the idea is not the hard part. Measuring it honestly is.

The numbers

Same benchmark, same harness, across all of them. HumanEval+, 164 problems.

Parity, not a win

I want to be careful with the words, because it matters. This is parity. It ties the frontier models. It does not beat them. Anyone who tells you their cheap setup beats the frontier on accuracy is either measuring wrong or selling something. What I am claiming is narrower and more useful: you can land in the same accuracy band as the frontier without paying frontier prices on every request. The cheap model alone was 84.8%. The routing and verification is what closes the gap to 94.5%.

The cost, which is the actual point

Measured over 313 production requests, from the real usage logs. Blended cost came out to about $0.002 per request, versus about $0.017 for the frontier equivalent. Roughly 8x cheaper for work in the same accuracy band. On that run, 96% of requests were served by the cheap tier, and about 3.7% escalated.

The second win I did not expect

There is a second win I did not expect. Verified answers are cached, and a cache hit returns in about 0.16 seconds, which in my testing was 24 to 185x faster than solving it fresh. The more you code, the more of your work is instant. I will be honest that this compounds with real usage, and I am early, so I am watching it, not overclaiming it.

Where it is weak

Where it is weak (you should know before you trust it). - The hardest problems still escalate to a frontier model. That is by design. On hard, multi-step problems the savings shrink, because more of them escalate. It is not a cheap model doing frontier work by magic. It is the right model for each job, with a backstop.

HumanEval+ is a benchmark. Real-world code is messier, and I am still measuring that part honestly rather than pretending the benchmark settles it.
The verifier is only as good as the checks it runs. Give it weak tests and the gate is weaker.

About your code

Every AI coding agent sends your code somewhere, and mine is no different. Your code does reach my gateway. I am not going to pretend otherwise. What matters is what happens to it there. I cache verified answers, never your prompts or your code. Your code never enters the shared cache, it never trains anything, and it is walled off per tenant so it can never be served to anyone else. For regulated work there is a dedicated tier I genuinely cannot reach into.

Where I am

I am opening it to a first group of testers. If a cheaper, private coding agent that ties the frontier on accuracy is useful to you, I would genuinely like your help pressure-testing it. Tell me where the numbers do not hold up. I will keep publishing results as more people run it, the ugly ones included, because the honesty is the whole reason to trust a number from a guy you have never met.

Thanks for reading. If you have questions about the method, ask. I am around.

Serving cheap when two models agree: a measured cost lever

Tom Jones — Mon, 29 Jun 2026 04:18:02 +0000

The problem

A cost efficient AI system sends easy work to a cheap model and only escalates hard work to an expensive frontier model. The trouble is knowing which is which. When a task has a test, like code with unit tests, you just run the test: if the cheap answer passes, serve it; if not, escalate. But most real prompts have no test. A question like "what time is the maintenance window" cannot be checked by running code. With no test, a careful system escalates almost everything, and you pay frontier prices for work a cheap model could have done.

We measured our own gateway and found exactly that. On no-test prompts in automatic mode, the system escalated to the frontier 100 percent of the time, at every context length. The cheap tier was capable, but the system did not trust it without a test, so it never served those answers.

The idea: agreement as a stand-in for a test

Instead of a test, ask a second, independent cheap model the same question. If the two cheap models agree, the answer is very likely correct, so serve it cheap. If they disagree, that is the genuinely hard case, so escalate to the frontier. Disagreement never serves a worse answer than before, because the disagreement path is the same escalation that used to happen anyway. Agreement only adds a chance to skip an unnecessary frontier call. The gate is conservative by construction, so its only failure mode is paying for an avoidable escalation, never serving a wrong answer, unless the two cheap models happen to agree on the same wrong answer. That single risk is the whole ballgame, so we measured it directly.

The one number that can break it

The only way this gate ever serves a wrong answer is if two cheap models agree on the same wrong answer. We call that P(wrong given agree). If it is zero, agreement is a safe stand-in for a test. So we stress tested it.

We ran two architecturally different cheap models across four task families, including a set of brand new hard traps we wrote ourselves so they could not be memorized from training data (a reverse-percentage trap, a rise-then-fall price trap, a buy-two-get-one trap, a clock-chime interval trap, a snail-in-a-well trap, and more). The result, at n=160: P(wrong given agree) was 0.00 in every one of the four families (retrieval, reasoning, multi-fact, and the new adversarial traps), with zero agree-and-wrong cases out of 160 total.

When the two cheap models agreed, they were correct 100 percent of the time, across every category, including the traps designed to fool them. They agreed about 76 percent of the time, and each cheap check took about 0.9 seconds at the median. One honest note on rigor: an early run showed a few apparent failures that turned out to be a formatting bug in our scoring, not real errors. We caught it, fixed it, and re-ran clean. The boring re-check is the work.

CORRECTION, added 2026-07-27. The 0.00 above is real but it does not mean what this piece says it means, and two phrases here are now retracted: "safe stand-in for a test" and "correct 100 percent of the time".

One. Zero wrong in 160 does not bound the error rate at zero. For a zero numerator the 95% upper bound runs about 3/n, roughly 1.9%. At a 76% agree-rate that permits on the order of 1.4% of everything served to be wrong and never checked. The point estimate was 0.00. The bound never was, and this piece leaned on the point estimate as if it were the bound.

Two. The probe was short answer arithmetic, where the cheap tier is essentially never wrong. A gate cannot be measured where the primary does not fail. Every row of that probe showed the primary correct, which should have been the tell and was not. Re-run on code, using the same agreement check with the benchmark's own tests as ground truth, the gate leaks low single digit percent, not zero.

Three. The word "independent" in the opening is doing work it cannot support. Agreement substitutes for correctness only when the two endpoints fail independently, and independence is a property of what the provider is serving, not of your code. Ours quietly stopped being independent: the witness was selected by model name, and all three of our cheap backends turned out to be the same underlying model under three different spellings. A same model pair agrees about 88% of the time against 70 to 75% for a genuinely different one. That is close to a rubber stamp, and a rubber stamp also reports 0.00.

And the traps cut the other way from how this piece reads them. We presented the adversarial families as evidence the gate was safe. They are the opposite. A bait works by shared structure in the input, shared structure is exactly what correlates two endpoints, so the gate is least trustworthy on the cases that cost the most to get wrong.

What survives: disagreement is still a useful escalation signal, and checking is still cheaper than producing. What does not survive is treating agreement as a test. If you use this pattern, sample a slice of the agreed cases, escalate them anyway, and score whether the expensive model concurs. That turns a one time number into one that moves when a provider changes what sits behind a model name.

What it does to cost

We shipped the gate to production and re-measured escalation on no-test prompts:

context length frontier escalation before frontier escalation after accuracy after 1,000 tokens 100% 40% 100% 4,000 tokens 100% 20% 100% 16,000 tokens 100% 0% 100% 32,000 tokens 100% 0% 100%

At long context, where frontier calls cost the most, escalation went from total to zero with no loss of accuracy. Across live traffic the system now serves about 91 percent of requests on the cheap tier. Our blended measured cost is about 0.002 dollars per request, and a repeated question is served from cache at close to zero.

*Why it matters
*
The savings on verifiable work, like code with tests, were already real. This extends the same economics to the large class of work that has no test, which is most real questions, without guessing and without giving up accuracy. The hard cases still get the frontier, and those are exactly the cases worth caching a high quality answer for, so the next identical question is served cheap too.

Honesty and limits

Every case we proved is a single final answer. We have not yet proven the gate on multi-step reasoning where a final answer can be right by luck on a broken chain, or where two models could agree on the same wrong chain. That is the next frontier and we are not claiming it here. The result above is for two specific cheap models; a different pair could behave differently, and widening model diversity is a known lever we hold in reserve. We are publishing the result and the honesty, not the gating engineering.