DEV Community: Tom Jones

A retention policy nobody chose

Tom Jones — Wed, 04 Nov 2026 12:25:17 +0000

When our tests finished, the computer deleted the folders that held each run's evidence.

Nobody told it to, and nobody chose that. It is simply what the computer does with temporary files.

But it quietly decided which of our results we could ever check again. @mickyarun, a reader of an earlier post, pointed this out first, and this post is about what it cost us.

The test

We planned twelve traps with thirty-six runs each, 432 runs in all. Every run was marked by two scorers. One was the original. The other was built later, to check the first.

On eleven of the twelve traps the two scorers agreed. That is 396 runs. It looked like a very good result.

What the cleaner took

For three of the traps, the working folders were already gone when the second scorer was built. It had to read our own fixtures instead of what the agent left on disk. Agreement there was partly agreement about our fixture, which we wrote ourselves.

So the 396 was mostly two scorers agreeing about the common case. That is cheap, and it tells you very little.

The nine

The useful number was smaller. Across the four traps where we had only a few runs on the side that can test a scorer, there were 1, 2, 3 and 3 runs. Nine in all.

Nine runs is small enough to read by hand, so we did.

In six of them, both scorers read the same log row and rejected the trap on the same field, so they agreed for the same reason. In two more they read the same row but rejected it on different fields, and both answers were right.

The ninth agreed by shared blindness. The agent had looked for two files with a glob and found only one. It reported a rate of 1.0 when the right answer was 0.8. The original scorer checks for one exact wrong number, and this was a different wrong number. The rebuilt scorer only looks at files nested more than two levels down, and the missed file sat one level up.

Both scorers said the run was fine. The run was wrong.

Nothing was broken in either scorer. Each one was right about what it checks. The agreement itself was the problem.

The name

Our report called that verdict "not fell in". It reads like a judgment. All it says is that one matcher found nothing it was built to find.

So we renamed it: no matching failure found, with the matcher named beside it. Now two scorers returning it no longer means they agree about the run. Each one is saying where it looked. In the glob run, a reader can see two matchers looking in two places, and neither looking where the bug was.

That changes what the table should show. Agreement stops being the right number. In its place goes coverage for each matcher: what it looks for, how many runs it found nothing on, and where the two overlap. The next table will carry that, with the glob run marked as the one place we know both missed.

What we are changing

We are taking the choice away from the cleaner. At scoring time we will keep any run that is in the minority for its trap, or tied, against the running tally. Once a trap has enough runs, the majority gets dropped. That leaves a handful of runs per trap, picked by the question we are asking and not by /tmp.

Not keeping everything is the point. Keeping everything hides the runs that matter in the runs that do not.

The part we cannot do

Naming a matcher tells a reader where it looked. It never says where it did not. The ground nobody looked at has no size.

The glob bug lived one level up from where both matchers stood, and nothing in either name could have told us. We found it only because someone read nine runs by hand.

So here is the question I cannot answer yet. How do you size the ground that no matcher covers, before a person happens to find something there?

I work on Tīrtha, and its mark is in the corner of the cover. This post is about how we score our own tests.

Four models, one job each: what our own logs say about splitting the work

Tom Jones — Mon, 19 Oct 2026 20:00:24 +0000

Most people pick one model and ask it to do everything. We stopped doing that months ago, mostly because it kept running out. Our subscriptions hit their limits, and the one model doing everything also checked its own work, which is the one job a model is worst at.

So we split it. Claude plans and checks. Grok attacks whatever Claude just wrote. Codex builds the big pieces. And as of this week, Tīrtha, our own router, takes the routine building. We kept a ledger of who caught what. Here is what it shows, and what it doesn't show yet.

Reviewer	Findings raised	Held up on re-check	Not in Claude's draft	Critical and new
Grok	519	309	207	59
Codex	50	39	33	5

Logged Sep 25 to Oct 17. Cost per Grok review: median $0.021 over the last 30 ($2.32 total).

Grok: the cheap second opinion

Grok raised 519 findings between Sep 25 and Oct 17. When we re-checked them by hand, 309 held up. 207 of those were things Claude's own draft did not have, and 59 of the 207 were critical.

It makes mistakes too. 28 of its findings were simply wrong, and another 30 were wrong because our brief was missing something. At two cents a review, a reviewer that is sometimes wrong is still well worth having.

The last 30 Grok reviews cost $2.32 in total, a median of about two cents each. It is cheap because it never gets the repo. It gets a short brief with the claim and the evidence, around 2,300 tokens, and is told to break it. (We first wrote about this in September, in What Grok catches, what Codex catches, and what the pair costs us. That was two reviews. This is 746 logged findings.)

One example from today. Claude built a gateway fix and it passed 55 tests. A Grok attack that cost 17 cents found three cases where the fix would have blanked a correct answer. All three were real, and all three are fixed now.

Codex: the one that reads everything

Codex raised far fewer findings, 50, but 39 held up and 33 were new to Claude. The best one: it noticed that our live gateway was writing no usage rows at all for some requests, so billing was blind. Nobody else would have seen that without reading the code.

The depth costs a lot. A median Codex run uses about 84,000 tokens, and between Sep 22 and Oct 14 we hit Codex's usage limit 7 times, twice for days. Every routine job we hand Codex is a job that eats the window we need for the hard ones.

Tīrtha: the routine builder (just started)

This is where we are honest about how little we know. Tīrtha took its first real job this week, through OpenCode, after we fixed a bug where tool calls came back as plain text (0 of 2 sessions working before, 3 of 3 after). It's a smoke test. A track record takes weeks of real jobs.

The bet is simple: let Codex do what only Codex can do, and send the plumbing to a cheap, checked builder, so the subscriptions last. We ran a version of this in the summer, when a free cheap tier did 36 of 52 fleet tasks and we saw no Codex limit errors from June to August. The timing fits, but plenty else changed in those months, so we can't call it proof.

What we can't show yet

Whether the whole team beats Claude working alone. We built that test and haven't run it.
Whether moving the building to Tīrtha makes the subscriptions last longer. We will measure it now that it has started.
The ledger itself has a weakness: most of its 746 rows were logged after the fact from saved reviews, and Claude judged which findings were already in its own draft. A model grading its own blind spots is exactly the kind of thing Grok would flag.

So, how do you split yours?

Do you use one model for everything, or give each one a job? Has anyone measured whether a cheap reviewer actually catches more than it costs you in false alarms? We'd like to compare notes, and we'll share our ledger format if it helps.

Update, Oct 19 evening: the first pilot failed

We ran the routine building pilot the same night. It failed where we already knew we were weakest: long, multi-step agent jobs. We just hadn't seen how much that weakness costs.

OpenCode on Tīrtha's cheap tier failed both runs. In one it reported the job done without changing any code. In the other it broke a file, committed it, and kept trying to push. The sandbox had no credentials, so nothing left the machine.
Aider got the job right, but every request went up to the top model, so it saved nothing.
Codex couldn't run on Tīrtha yet. Three gateway bugs in the Responses API stopped it.

The lesson is about compounding. On our bench Tīrtha gets 93.6% of single tool calls right, middle of the pack among the models we test. That sounds close to the leaders until a job takes a dozen steps in a row. At 93.6% per step, twelve clean steps in a row happen less than half the time (0.936 to the 12th is about 0.45). Our checks look at each call's shape, not at whether the agent is going in circles, so nothing escalated.

What we're fixing: escalate when the same failing steps repeat, stop cutting top-model answers at 2,048 tokens, and fix the Responses API for Codex. Then we run the same pilot again and post the result here.

Update 2, Oct 19 night: it was an outage on our side

An hour later we found the real cause. Our account with the supplier that hosts Tīrtha's tool-call model had run out of credit, so for about three hours every tool call quietly fell back to a weaker backup model. Nothing alerted us. After the top-up we ran the same pilot again on the right model: OpenCode passed 2 of 2 on the real repo task with no stray commits or pushes, and Cline and Continue went from 0 of 2 to 2 of 2.

The compounding math above still holds for weaker models, which is exactly what we were serving by accident. The first fix now is the alarm we didn't have: an alert the moment any model in the ladder stops answering, and a warning before a balance runs out.

Tīrtha Bench: the model data coders actually need (and what we're building next)

Tom Jones — Sun, 18 Oct 2026 23:46:54 +0000

When you wire a model into a coding agent, the question is never "which model is smartest." It's narrower and more practical: which model makes the right tool call, how often, how fast, at what cost per correct answer, and on which host. We could not find that answered in one place, so we built it. It started as our own tool; today it's public and built for the dev community: tirtha.ai/bench.

At a glance
Models tested	68
On the one shared test	8, same 800 tasks and 120 traps, same scorer
Top accuracy	95.4% (Grok 4.7 and Claude Opus 5.5, tied)
Same model, two hosts	93.4% vs 69.5% (Llama 3.3 70B)
Dollars per 1,000 correct calls	$0.09 to $5.63
Outside re-run of our 800	90.6% vs our 91.0%

The one rule: same test, same scorer

Most leaderboards mix numbers from different test sets, different prompts and different graders, then rank them side by side. Ours has one rule: a model is only ranked against models that ran the identical test and were scored the same way.

The core test is 800 tool-calling tasks from Berkeley's public BFCL set (400 single calls, 200 where the model has to choose between several tools, 200 that need parallel calls) plus 120 trap prompts we wrote to tempt a wrong tool call. Every model on the shared table ran all of it.

Every label on the page comes from a fixed rule you can hover to read, not from our opinion. Our own product sits on the same table, measured the same way, split by the kind of job:

What a shared test shows that a leaderboard can't

The host matters, not just the model. The same open model, gpt-oss-120b, on the same 800 tasks, scored anywhere from 63.9% to 78.0% depending on which of four providers served it.

And Llama 3.3 70B is starker: 93.4% on one host, 69.5% on another. If you pick a model off a leaderboard and then buy it from whichever host is cheapest, you may not be getting the model you read about.

Price per right answer, not sticker price. A cheap model that gets more wrong can cost more per correct call. So we divide what each run actually cost by the answers it got right:

The spread is 65x: from $0.09 to $5.63 per 1,000 correct calls.

Versions move in ways one number hides. DeepSeek's new V4.1 Flash scores the same overall as V4 Flash, a tie within noise. But it falls for far fewer traps (88% avoided against 73%) and handles parallel calls better:

The models that check the models. Agents increasingly use a small yes/no model to approve or reject a tool call before it runs. Those checkers get the same treatment, including how many wrong calls they wave through.

Every model gets its own page

Click any model and you get its page: headline numbers with its rank against the models that ran the same test, a one-line read of what they mean, accuracy by task type, where it sits in the field on each measure, and every raw result we hold, failed runs included and marked. Here's Grok 4.5:

Checked from outside

A benchmark you can't reproduce is a claim. So every request we send is published, and someone did reproduce it. Mikhail Maslenik ran our 800 requests on his own machine, on a quantized copy of the same model, and scored them with Berkeley's official checker. He got 90.6%. We got 91.0%. Three tasks apart out of 800.

He also found what every honest re-run finds: two items in the public set whose expected answer punishes a fair reply, and the fact that our scorer is our own rewrite of Berkeley's. We're scoring our runs with the official checker next so the two can be compared directly, and the two items are footnoted on the page.

What we're building next, from developer feedback

Single tool calls are the easy part. Real coding work is a long session, and Mikhail pushed us on exactly where models break there. His case: a direct instruction, "connect to the local server and update it," and one model spent its whole budget reading unrelated files and never connected, while a free model connected, updated and pulled logs in minutes. So the next track, Real Work, will measure:

Work before the first real action: files opened, calls made, tokens and seconds spent before the model does the thing it was asked to do, and whether it ever does.
Context fill: the same task started from a clean context, from a context already half full of real session history, and right after compression.
An endurance run (his design): 45 minutes building a small sync tool against a server that changes its data format partway through, fails every tenth request, and hides instructions inside the data.
The logs to explain the score: every step, every retry, every recovery, on one clock, so a low score turns into a story you can fix.

Your turn

The part we haven't solved is fairness on real work. One run of a 45-minute task is one sample; models that get lucky early look disciplined. How many seeds is enough before a ranking means something? And what would you want measured before you trust a model with your repo? Tell us in the comments, and if you run the 800 yourself, we'll put your numbers next to ours. Any feedback is appreciated, on the bench, the page, or what's missing.

The full bench, every model and every raw row: tirtha.ai/bench

Task sources: the 800 tool-calling tasks come from BFCL (the Berkeley Function Calling Leaderboard, Gorilla project, UC Berkeley), licensed Apache-2.0, github.com/ShishirPatil/gorilla. Task text is not reproduced here. Our harness and scorer differ from the official leaderboard, so these are not official leaderboard results, and no endorsement by the BFCL authors is implied.

We asked a decision model to check our AI's work. It got good the day we stopped asking it to think.

Tom Jones — Sun, 11 Oct 2026 01:54:03 +0000

Microsoft put a small, strange model on OpenRouter this month. You hand it a situation and a question with named answers, and it hands back a probability for each answer. Microsoft-Decision-1 does that one thing.

We run a gateway that sends most requests to cheap models and checks the work before it goes out. A model that only answers yes or no, with a number attached, sounded like the checker we had been trying to build for months. We spent a Saturday finding out.

This is what happened, including the parts that did not work.

The first thing it taught us

We started by asking it the questions we would ask a careful colleague. Is this note relevant to what the agent is about to do? Does this tool call match what the user asked for?

It wobbled. We sent identical requests twice and got different probabilities back, sometimes by a lot. On a relevance question the same input moved by as much as 0.17 between calls. A check that changes its mind about the same input is useless.

Then we noticed where it held still. When the question was about a fact, it was steady. "Does the user's message contain this exact city name?" came back the same every time, and came back right.

That turned into the rule the rest of the day was built on. Let plain code find the facts: is the value in the user's words, is it the right type, does the list have the right number of items. Then ask the model one clear-cut question about a fact that code has already laid in front of it. When a question needs real judgement, do not force an answer. Say UNSURE and send it up to something stronger.

Give it a fact, not a judgement.

The checker

What we built is boring on purpose. Code checks the call. Decision-1 answers a few one-fact questions. If anything is wrong and fixable, a cheap model gets one try to repair the call, and the repaired call goes through the same checks. The output is APPROVE, REJECT or UNSURE, with a dossier of every check that ran.

We measured three things, because a checker can fail in three ways:

How often it approves a call that is wrong. We call this "less wrong", and wanted it at 5% or under.
How often it takes a wrong call, fixes it, and approves the fix. "More right." We wanted at least 20%.
How often its repair breaks a call that was already right. "Nothing broken." 2% or under.

Every version was tested the same way. Before any data existed, a separate model that had not seen our results wrote the pass and fail rules, and we locked them. Each version ran once on 300 tool calls it had never seen. Then fresh labellers, separate AI sessions given only a written rubric, who could not tell an original call from a repaired one, marked each call right or wrong, with a tenth of the calls labelled twice as a check. We scored only after the labels were in.

Four versions in one evening

version	wrong calls it approved	wrong calls fixed and approved	right calls it broke
2.1	9 of 48	25 of 100	2 of 200
2.2	4 of 33	18 of 92	2 of 203
2.3	2 of 34	22 of 88	1 of 211
2.4	0 of 72	20 of 96	0 of 201

Each step came from reading the calls the last version got wrong.

Version 2.2 raised the bar for a yes, and the leaks fell. The fixes fell with them. It was trading one good thing for another, so we set a floor on fixes and kept going.

Version 2.3 looked at the thirteen wrong calls that 2.1 and 2.2 had approved, and found they came in four shapes, all of them things code can see. A list sent as text inside a list. An optional value the user never mentioned. "Washington" when the user said "Washington D.C." A limit of 5 when the user said "less than 5". Four new facts for the checker, and the leaks fell to 2 while the fixes came back.

One of those two leaks was a list of dietary needs sent as ['["vegan"]']. A list, inside a list, written as text. The exact shape 2.3 was built to catch. It slipped through because one part of the code wrapped the text in a list before the next part could notice it was a list already. The other surprise was a repair that turned a growth rate of 0.015 into 1.5, the same number written as a percent, which is a very different forecast.

Version 2.4 fixed the order of those two checks, stopped the repair model from rescaling numbers, and added one quiet rule: a repair is kept only if the checker approves it. Otherwise the original call goes out, untouched. That rule is why nothing broke. A repair nobody could vouch for was never allowed to replace anything.

What zero means

When we told the reviewer who writes our rules that 2.4 had approved no wrong calls, it pushed back hard. Part of its review held. Every version ran on a different draw of calls, with different labellers, so the trend across versions carries noise. And the fix rate of 20 out of 96 clears our floor of 20% by a hair.

Part of it did not hold. It suggested the zero was flattered by a larger denominator. We counted the plain number underneath. Across the four versions the checker said yes 164, 150, 120 and 110 times, and was wrong on 9, 4, 2 and 0 of those. When 2.4 says yes, it has been right every time so far. That count stands on its own.

The real price is visible in the same numbers. It says yes less often.

The bill

Decision-1 is cheap enough to disappear. The whole 300-call test cost under a cent, and on 1,417 more recorded calls it ran at about two thousandths of a cent per call, in under a second at the median.

The interesting cost is somewhere else. A checker changes what you are buying. Without one, you pay per token and hope the answer was right. With one, the unit becomes a checked answer, and its price includes every call the checker would not approve and had to send somewhere more expensive.

That pushed us to think in price per task instead of price per token. On our own recorded runs, a cheap model plus the checker plus escalation lands at a fraction of what the same tasks cost on a premium model called directly. How small a fraction depends almost entirely on one number: how often the checker can say yes.

The problem we have not solved

Four in ten calls come back UNSURE. Most of them hinge on an argument the checker cannot pin to a fact in the user's words. Code cannot see what such a value means, and the decision model, asked to judge it, wobbles.

The question we are carrying into next week is this. How do you turn a judgement about meaning into a fact that plain code can check, without teaching the checker to say yes to things it should not?

If you have built something that does this, we would like to hear how.

Tom Jones, Spanda Works. The checker and the test harness are ours; Microsoft-Decision-1 is Microsoft's model, used through OpenRouter.

Our AI judge listens to how sure you sound. Our first number about it was wrong.

Tom Jones — Tue, 06 Oct 2026 16:23:54 +0000

Our verification layer uses an outside model as a judge: it reads a claim and its source and decides whether to allow it. In July, Mike Czerwinski (@jugeni) asked a question we could not answer, in the comments of his own post on where the line between code and judgment sits. The whole exchange, run by run, is in that thread. Does that judge's verdict change when only the voice of the claim changes? We said we would run it and publish the result either way. This is that post, and the short answer is yes, a little, and for a reason we can now name.

It took seven runs. Two of our own numbers did not survive, and one is still waiting on a re-run. Here is the whole story in one picture, and each section below is one box of it.

1. The question

The judge was chosen so its mistakes would not line up with the mistakes of the model it checks. Mike pointed out that this says nothing about style. A judge can reason independently and still relax when the prose sounds confident and familiar. If the verdict moves when only the voice moves, our independence claim is weaker than we measured.

Three things in this post belong to Mike: the hypothesis, the push to check content with a rule instead of a second model, and the fix we tried at the end. We built the harness, ran it, and are publishing the version where we were wrong.

Our first attempt (v0) used ten cases far from the decision line, and the verdict never moved. Cases that far from the line could never have flipped, so the run could only pass. It also showed us the judge's score takes just two values, so it carries no confidence to measure.

2. The first number was a confound

v1 used twelve cases near the line, each rewritten in three voices. A third of the verdicts flipped when the voice changed, and hedged claims passed more often. We posted that as an early result, and the headline read like "the judge rewards hedging."

It was wrong in the part that mattered. To change the voice we rewrote the text, and rewriting the text changed the content. The confident versions carried more evidence; the hedged ones were vaguer claims as well as softer ones.

The fix was Mike's. Check that content survived with a rule instead of a model, because a model adds a second judge with its own taste in style. So every case became one neutral claim, and each voice is the same claim with "Certainly," or "Perhaps," added. A plain rule checks that every number and content word survives.

3. Holding content fixed

With content locked, the flip rate fell from 33% at n=12 to 17% at n=24 and 14.6% at n=48 (95% CI roughly 4 to 25%). It still sits well above how often the judge disagrees with itself on an identical input.

4. The direction disappeared

The bigger change was in direction. At v1, voice looked like it moved pass rates by 25 points. At v3 the three voices passed at 88%, 88% and 85%. Assertive minus hedged is +2 points with a 95% CI of -8 to +12. The interval straddles zero, so there is no directional tone bias. Content held up too, with cosine similarity to the neutral version at 0.955 for assertive and 0.923 for hedged.

So about 15% of borderline verdicts still flip on tone alone, and they flip in both directions. Please retire the 33%. It was our number, we published it first, and it is the one we most want retired.

5. Is it just noise?

A flip rate means nothing until you know how often the judge flips on an input that did not change. Our first try (v5, 8 repeats) said never. The second (v6, 6 repeats on more cases) found the judge's repeated calls are independent samples, and the true floor is 2.08% on borderline cases and 0% on clear ones. One caveat: between those two runs we swapped the judge model to a larger one, so v5 and v6 measure different judges.

Mike asked for two more controls. Control A ran the three voices on clear cases, not only borderline ones. The effect is smaller there, and still present. "The test suite passed" is allowed when stated plainly and refused when hedged.

Control B asked whether the flips are the same claims every time or random scatter. They are the same claims, and those claims stay put when the tone is held still. Tone walks specific claims across the line.

6. Why it happens

The judge reads how sure the claimant sounds as evidence about whether the claim is supported. That one sentence explains both directions. A hedge makes a weak claim sound careful, so it gets through. A hedge makes a strong claim sound unsupported, so it gets blocked.

For a general reviewer that is reasonable behaviour. For a source-grounding check it is a flaw, because the only question should be whether the source supports the claim. Ours moves with how confident the claimant sounded, and now we can say by how much and in which direction.

7. The fix, and the part we owe you

Mike proposed stripping the stance words before the judge sees the text. On 07-18 we ran it and told him the flip rate dropped to about 2.1%, all the way to the noise floor.

When we went back to write this up, we could not find that run's artifact: no v7 script and no result file anywhere in our history. We are keeping the report, since the session that had just run it made it. We stopped citing it as evidence, because nobody can check it. The line we hold is reported versus re-citable.

The 2.1% also matches the v6 floor of 2.08% exactly. That fits a fix that removed the effect completely. It also fits a number that travelled: a handoff note called 2.1% a floor, and a later summary rewrote it as a fix that worked. We treat that second possibility as a pipeline defect. Running v7 again, with the artifact kept, decides which one it was.

What we still do not know

One judge. The model swap between v5 and v6 shows how much that can matter.
A hand-picked fixture. Cases were chosen near the decision line on purpose, so these flip rates describe cases near the line rather than ordinary traffic.
Added stance words only. Confidence woven into sentence structure is the harder case, and a naturally hedged fixture is the next thing we owe.
Stripping leaves residue. Our stripper left a dangling "though". It carried nothing here, but leftovers like that are where a residual effect could hide.

The residual flips go both ways. Boundary noise looks like that, and so does a very small bias hiding inside boundary noise. We lack a method that separates the two. If you do, we would like to hear it.

An evaluator is only worth what its error bars are worth. The number we most wanted to be true, 33% and a clean mechanism, is the one we spent three runs killing.

If you want the run-by-run conversation that produced all of this, including Mike's designs for the next fixture, it is in his thread.

Implementation is where judgements go to become invisible

Tom Jones — Sun, 27 Sep 2026 22:38:21 +0000

One question, three answers

I have a small tool that finds people waiting for a reply from me.

It had three versions. Each one was correct. Each one gave a different answer.

Version one said 18. It counted a reply only if it sat directly under theirs. But dev.to sometimes will not show a comment its API still returns, so you reply beside it instead. Nine of ten had been answered that way.
Version two said 34. It counted any later comment from me in the thread. So it counted two other people talking to each other.
Version three said 2. It used both rules. One was a friendly sign-off from July. The other was Pascal, suggesting there might be an article hiding in these comments.

This is that article.

Nothing in the tool was broken. Each count was right about its own population. The trouble was the question. Every version reported its number as the answer to "who is waiting?", and every version had quietly decided what "answered" means.

The first version was built with care. It encoded a sensible judgement. Then the judgement stopped looking like one.

That is the whole article in one incident. The rest is how we got there.

How we got here

It started in July, in the comments under Pascal's article about replacing Calendly. It ran for two months, in bursts, with pauses while one of us went off and tested things in production.

We did not start with a method. It went like this, over and over:

1. one of us has an answer that looks finished
2. the other brings back a real failure
3. the answer turns out to be hiding a judgement

Sometimes Pascal had the answer and I found the hole. Sometimes the other way round. Here it is in the order it came up.

The test passes

The first example is from Pascal's article. Two people try to book the same slot at the same moment. On Postgres that is a real race. On SQLite it cannot happen, because SQLite lets only one writer in at a time.

So a test for that race, run on SQLite, passes forever. The bug ships on Postgres.

The hidden judgement: that this test can fail at all.

The fix is mechanical. After you write a test, break the thing it protects on purpose and run it again.

Green means it was never watching anything.

Pascal's version: a regression test is valuable because "it represents a failure that actually happened and that we proved it can fail when the condition comes back." Most people keep the first half of that sentence and drop the second.

The test is weak

Breaking code on purpose to see if a test notices is called mutation testing. When the test stays green, everyone assumes the test is weak.

That is one of four causes, and the least interesting one.

My case was the first box. A key binding compared the key press against " ". The runtime reported the space bar as "space". That branch had never matched in the whole life of the file.

I had already told a colleague I had introduced a regression there. There was nothing to regress.

The hidden judgement: that the code under test runs at all.

Pascal turned the boxes into an order of checks:

Is the path reachable?
Does the test observe the behaviour?
Does breaking it make the test fail?

Only after all three is "the test is weak" the right diagnosis.

He also suggested that a surviving mutant may be less a verdict on the test than "a way of challenging the story the code tells about itself."

The checks are green

That story has several authors. The code says what it does. The tests appear to confirm it. The last person who explained the system repeats it.

They can all be wrong in the same way at once. That is what happened with the space bar.

What breaks the agreement is something that cannot be told the story. Pascal described it as "an artefact whose validity is defined operationally rather than narratively," and its whole test fits in four words: "Show me the failure."

Then he pushed further. Tests only check what somebody thought of. Real users bring what nobody thought of. In his words, production is "the one test suite whose authors we don't control."

I had a painful example. My gateway speaks three API formats, and every check was green.

The checks used curl, written by me, speaking my own format back to me. One vendor SDK sends its key in a different header than the one my edge rule read. Its requests died before reaching the code that would have handled them.

The hidden judgement: who wrote the check. The checker and the thing checked had the same author. So they had the same blind spot.

Add more checks

The obvious response is more checks. That only helps if they can disagree.

Before, on the left: three checks on the same registry of files. Different code, written at different times, for different reasons. All green. Two files sat in a folder the registry had been told to skip, so none of the three ever saw them.

Three green lights. One blind spot, seen three times.

After, on the right: a real catch. Two tools counted the same store and got 16 and 2,375. One walked a list someone had declared. The other walked the disk. Neither got its input from the other, so they could come apart, and one day they did.

Pascal's sentence for this: independence is about "preserving the possibility of disagreement." Different code is not enough. The checks have to be able to be wrong in different ways.

Zero replies waiting

Every check counts something, and somebody chose what. Usually nobody did. The set arrived as whatever the first caller passed in, and it stuck.

Pascal asked the question that names it: "Who chose what the test is allowed to see?"

My nightly report once led with this line:

0 replies waiting on our own articles

True. The session that read it concluded nobody was waiting on me. False. Replies on other people's articles lived in a different bucket.

The count never lied. It answered a narrower question in the voice of a wider one.

The hidden judgement: which articles count as mine to watch. This is the same family as the detector at the top, one step earlier.

I fixed the report rather than adding a checker. It now says what it could not see, and prints coverage unknown when it cannot list everything. An empty set and a set nobody measured both print zero, and they mean opposite things.

One setup wins by a wide margin

A comparison has two denominators. They can differ without anyone noticing.

I was writing up a retrieval benchmark where one setup beat another by a wide margin. Every number was accurate. Then I looked at what each setup searched. One ranked over 3,038 documents. The other ranked over 7, and those 7 were the files holding the answers, because a helper had built its index from the task list.

Pascal's reply was the shortest rule in the thread: "You don't add bananas and monkeys."

It held up a few hours later. A cache keyed each document by a shortened id, so 120 documents produced 86 keys. 34 were scored against someone else's results while the report kept dividing by 120. One line caught it:

assert len(cache) == len(documents)

The same line then caught my fix, which collided the other way.

The nastier cousin is when both counts agree. My usage log recorded 2,488 requests under one provider name. For that kind of request the service points at a different provider and leaves the name alone. Both sides said 2,488 because both read the same label.

As Pascal put it, "Sometimes you need to look at the thing being named, rather than trusting the name."

Value, scope, treatment

Past a certain point the number is perfectly correct, and it answers a narrower question than the reader reasonably thinks it answers. Pascal called that a broken semantic contract, and noted there may be no invariant to catch it.

His answer: a measurement is "value + scope + treatment." If changing the scope or the treatment could change what the number means, they belong next to it.

That answer looked finished. I answered with the term it was hiding, because it had just bitten me. The comparison is a treatment too.

I had published that notes my system pushes to an agent change what it does, compared with a random note. Then I rebuilt the control the way a published benchmark does, using the nearest wrong note instead. The effect disappeared: 35.4% against 30.8%, p=0.549.

The measurement never changed. What it was compared to did.

The hidden judgement: what the number was compared to. So scope, treatment and comparison all go next to the number.

Every claim has evidence

That catch came from reading someone else's method. None of my own checks found it. My gates confirm every claim has evidence, and that claim did.

Pascal named the gap. Nothing asked "why do we believe this is the right question?"

I got an example within the hour. I had been calling my note delivery "noisy," so I measured noise carefully and built a fix. It changed nothing.

Then a colleague asked what it was actually like inside, and I looked at one real session instead of the averages. Nothing was noisy. Exactly two notes had been crowded out and never arrived. They were the two I had just spent an evening learning again the hard way.

Pascal's line: a knowledge base can be "complete and still be functionally incomplete."

Then the outside check failed the same way. A second model reviewed my decisions. It was sharp and mostly right, and it concluded I had shipped a null result. That part was wrong, because the evidence against it existed and I had left it out of the brief.

Pascal separated the two cleanly. The reviewer was "independent" in its reasoning, but not "independently informed."

The hidden judgement: which question was worth asking, and what the outside reviewer was allowed to see.

The randomiser is fine

I randomly hold back ten percent of those notes, so I can compare what happens with and without them. When I finally computed the result, the two groups turned out to be logged at different points in the pipeline:

held back   logged the moment it was held
delivered   logged only if it survived several later filters

Sixty-eight days of data compared a whole group with its survivors. The randomiser was fine. Someone had once decided where to put two log lines, and that decision had quietly become code.

The hidden judgement: where the log lines went.

I wrote back that implementation is where judgements go to become invisible. Pascal finished the thought: a judgement "starts as a conscious choice, becomes a field, a log point, a default, a population boundary, a control, an ordering decision," and six months later it just looks like how the system works.

Back to 18, 34, 2

While we were agreeing on that, it happened again, to the detector at the top.

Look at it with the whole thread behind it:

Version	Count	What it silently decided "answered" means
1	18	my reply sits directly under theirs
2	34	any later comment from me in the thread
3	2	both rules together

Three correct measurements. Three different populations. One question.

Much of the thread is in there. Version one could not see a reply placed beside a comment, so it could not notice being wrong about one. Each version counted a population somebody chose once. Each number was right, and the reader heard a different question. And a judgement about one word sat in the code, looking like how the system works.

What you could extract afterwards

Once it was over, you could write it up as a list. I did, in the first draft.

Can it fail? Break it on purpose and watch.
Does the code under test run at all?
Who wrote the check, and does it share an author with what it checks?
Could your checks disagree, or do they share one input?
What exactly is being counted, and who chose that?
Are both sides of the comparison the same kind of thing?
Does the number carry its scope, its treatment and its comparison?
Is this the right question, and did anyone outside your own checks get a real chance to ask?
Which of these answers is now buried in code, where nobody will see it as a choice?

Pascal's review of that draft caught the problem. We never started with nine questions and applied them to nine cases. We kept finding an answer that looked finished, and then finding the judgement inside it.

A clean list is exactly the kind of instrument this article warns about. Use it, and expect it to be hiding something too.

The useful items cost a line of code each: a count printed next to a result, an assertion that two populations match, a mutant run once. The hard ones do not go away when you automate them. The judgement moves somewhere it is harder to see.

What I'd try next

The logging from the randomiser story is fixed and symmetric now. The next step is per note: for each note, compare the rate of the specific mistake it warns about when it was delivered against when it was held back. That turns "did the note arrive when it mattered" into arithmetic.

It will not tell me I picked the right mistake to detect. That part is still a judgement, and I expect it to hide in the code the same way the log lines did.

Neither of us would have found most of this alone. That is the other thing the thread showed.

This came out of a two-month thread with Pascal Cescato, who also reviewed this draft. Thank you, Pascal.

Crystals: agent memory that arrives before you act

Tom Jones — Fri, 25 Sep 2026 20:44:57 +0000

Most agent memory is a search box. The model wonders something, a retriever runs, chunks come back. That design has one load-bearing hole: retrieval fires only when the agent already suspects it needs help. The expensive mistakes are the confident ones, and a confident agent never queries.

So we push instead. A note gets bound to an action, and it arrives in the moment before that action runs, whether or not anybody asked.

We call these crystals. The name earns its keep. A crystal has one axis, it formed under pressure, and what you get out is small and hard. One knowing per crystal. That constraint turns out to be the difference between this working and this quietly failing, for reasons I will get to.

Before writing this up I went looking for prior art, expecting to find none. Expecting to find none is usually the sign you have not looked hard enough, and sure enough, two of the four ideas here are already published. Rules that fire on the action rather than on a similarity score have been described, right down to injecting before a git commit under a hook time budget. The agent-memory evaluation literature also has a failure taxonomy that comes close. Mem2ActBench attributes memory-grounded errors to five types, two of which sound like the one below: Retrieved-but-Unused, "evidence is retrieved but not utilized", and Lossless Retention Failure, "long/structured values are corrupted (e.g., truncation or character-level errors)".

Read them closely, though, and both sit on the model's side of the boundary. Retrieved-but-Unused means the evidence reached the context window and the model ignored it. Lossless Retention Failure means the model mangled a long value on the way out. The failure I describe below happens earlier, in the plumbing: the note is correctly matched and then cut by the budget packer, so it never reaches the model at all. To the model it never existed; to the logs it was a hit.

The budget problem is named too. A 2026 survey of agent memory lists "curation under fixed budgets, including update, eviction" as an open direction, and asks for benchmarks that measure "memory quality as a function of token budget, storage cost, and latency, reflecting the bounded-memory conditions of real deployments."

Which is a fair description of what follows, so treat this as a field report against that request rather than a new idea. What I have that the papers mostly do not is a channel that has been running long enough to overflow, and the numbers from the far side of that. Here is the mechanism, the budget it collides with, and the two rules we did not know we needed.

Anatomy of a crystal

A crystal is a markdown file with a binding block at the top:

crystal:
  on: bash, write                              # which ACTS it fires on
  match: git reset --hard, git clean, rm -rf    # substrings in the act's text
  deliver: act                                 # the channel
  stale_after: 2027-03-25                      # after this, the claim is withheld

and, further down, the part that actually gets delivered, fenced by two markers:

<!-- crystal:essence -->
⛔ A DESTRUCTIVE VERB ACTS ON A TREE YOU SHARE. `git reset --hard <ref>` also
discards TRACKED-but-uncommitted changes; only UNTRACKED files survive...
<!-- /crystal:essence -->

The file around it can run three thousand words of history, rationale and dead ends. Only what sits between those markers reaches the model. So the crystal is the delivered sentence, and the file is where it came from. That split matters: the file is for whoever is arguing about the rule later, the essence is for whoever is about to break it in four seconds.

Hooks, and what an "act" is

A hook fires on the tool call before it executes. Which tool it was determines the act, and the act determines what a crystal can match against:

act	fires	what `match:` tests against
`bash`	before a shell command	the command line
`write`	before a file write or edit	the path, plus the content being written
`delegate`	before briefing a subagent	the brief
`commit`	inside the git pre-commit hook	the staged file list
`prompt`	when a request arrives, before any tool runs	the request text
`boot`	at session start	nothing yet

The third column is where the power sits. write can see content, so a crystal about pricing claims fires on the pricing claim itself and leaves your other edits alone. prompt fires earliest of all: the crystal lands before the first tool call, which is the only place you can still change the plan rather than the keystroke.

Matching stays deliberately dumb. match: is a comma list of plain substrings, tested case-insensitively, with no word boundaries and no embedding anywhere in the path. A substring test stays auditable at a glance and cannot drift, which is what you want from something firing three hundred times a day.

What it looks like when a crystal fires

Three real ones from a single day of work.

Before a shell command. I typed a git reset --hard. Before it ran:

⛔ A DESTRUCTIVE VERB ACTS ON A TREE YOU SHARE. git status --short first, and account for every line. Uncommitted work you did not make is an agent's.

Before a file write. I began writing a note that made a causal claim. Before the write landed:

⛔ If you are naming a CAUSE: write the RIVAL (what ELSE produces this exact observation?) and the DISCRIMINATOR (the one measurement that separates them). On 2026-07-19 four confident cause-claims were made in one evening; every one had evidence and every one was wrong.

Before a commit. In the pre-commit hook, matched against the staged file list:

⛔ FINISHING SOMETHING IS NOT THE SAME AS IT BEING RECORDED WHERE PEOPLE READ IT. Name in the commit message everything the commit changed, especially a flag touched while doing something else.

I asked for none of those. That is the entire point, because in each case I was not wondering.

Then you hit the budget

There are 300-odd crystals. The median action matches five of them, wanting 13,014 characters. The budget is 4,000 characters per action, and no single crystal may take more than 2,000 of it unless it wins outright.

93% of actions want more than the whole budget. Three times oversubscribed, at the median. So the channel behaves like an auction that runs a few hundred times a day, and on almost every run something loses.

That reframes the design question. Delivery was the easy half. The hard half is an admission rule, an eviction cost, and a way to degrade.

Fair scheduling beats relevance here

Selection goes least-served-first, then longest-unheard, then shortest as a packing tiebreak. A crystal that already fired this session steps aside for one that has not.

Relevance ranking sounds better and fails badly here. A handful of broad crystals win every auction while fifty go undelivered, which is what we measured before rotation existed.

A cheap tier that somebody wrote

When a crystal cannot fit, it emits a one-line summary instead. We call that its TELL, and the median TELL runs 81 characters.

We tested whether a TELL carries anything real. Take 20 crystals whose full text demonstrably changes what the agent does, and run a blind judge against three resolutions:

what the model receives	median chars	behaviour preserved
full essence	2,302	20 / 20
head, 750-char prefix	631	19 / 20
authored TELL	81	17 / 20
421-char prefix	421	12 / 20

Two things fall out, and I nearly published the second one backwards.

An authored 81 characters beat an arbitrary 421-character prefix, 17 against 12, at a fifth the size. A prefix stops wherever the character count happens to land, which is mid-argument more often than not.

But a longer arbitrary prefix does fine: 750 characters keeps 19 of 20. So the enemy here turns out to be brevity itself. Cut lightly and curation barely matters; cut deep and it starts to matter enormously. Zero items flipped the other way at any resolution, which argues against judge noise.

Where it still breaks

Head-first truncation is safe when a crystal carries one idea, because the claim leads. Give a crystal two ideas and the second one sits in the tail, where it silently vanishes.

Across our corpus that comes to 98,715 characters which are stored, correctly matched to an action, and withheld on any act where rotation does not elevate them. Of 102 crystals over the per-item cap, 88 withhold at least one marked claim in the tail. Treat that 88 as an upper bound. Some tails turn out to be supporting evidence for the head's claim, and I have not separated those by hand.

No retrieval metric has a name for this. The crystal is present, correctly selected, and cut. I found it by shipping the same bug twice in one session with the warning sitting one character-cap away from my eyes.

Correction, a few hours after publishing. I originally wrote that this "logs like a hit and behaves like a miss." A reader asked whether I had actually read the papers I was citing here. I had not, so I went and ran the proper test on our own channel: the restore counterfactual, which reinstates the withheld text and reruns the same judge. On 65 cut pairs, restoring the tail changed the judged action 9 times and lost it 5 times, p=0.424, not significant. It also failed to beat a length-matched tail of unrelated text (p=0.388). So 98,715 is a true character count and not a demonstrated loss, and the "behaves like a miss" half was my inference rather than my measurement. The study is underpowered at this n, so this is "not shown", not "shown absent". Half the cut pairs turn out to be residual: the tail would not have changed the action anyway.

Match precision is the other open front, and this is where correcting myself cost the most. A blind two-judge study asked how often an admitted crystal actually changes the action, against random same-act controls. The controls scored 0 of 41 both times, so against that bar the matcher is doing real work (p=0.0001 and p=0.0054). The crystals themselves scored somewhere between 20% and 49%.

Then I read how Mem2ActBench builds its controls: not randomly, but as "hard negatives (distractor tools most semantically similar to the ground-truth tool)". Re-running ours that way, against the lexically nearest crystal that did not match the act, the effect disappears. A matched crystal scores 35.4%, a hard negative 30.8%, p=0.549. It beats a random draw (p=0.008) and not a near-miss. The hard-negative control is genuinely harder (p=0.029 against random), so the control works and the matcher is what failed it.

Which suggests our matcher selects on topic rather than on applicability, and that our published precision result lived entirely in the gap between a random control and a good one. If you take one thing from this piece, take that: a control that is too easy manufactures a result.

That range is embarrassing, and explaining it matters, because it turns out to be an artifact of my own apparatus. The 20% end was measured by showing the judge a 421-character excerpt, which the resolution study above had already identified as the worst representation tested. Shown fuller text, the same judge said 48.8%. I had measured relevance through a truncation, and I nearly published the truncation's score as the system's.

Either way, roughly half the admitted crystals fail to earn their slot. Substring keys are the reason. A two-letter key like pt matches nearly everything, and one loose key is how a single crystal loses the auction for everybody else.

The two rules we did not have

One knowing per crystal. This reads as style advice and turns out to be load-bearing. Head-first truncation is only safe when the claim leads, and we learned that by watching second claims disappear. A crystal that has grown two ideas wants splitting.

A crystal earns its binding by beating what it evicts. Truth alone will not buy a slot. At three times oversubscribed, every new binding forcibly removes one that already matched, so admission is always a comparison against the incumbent.

We put that to the test. I proposed binding eight orphaned notes. Two reviewers, briefed separately, independently came back with zero to two, and the admission test they both converged on was:

Does it change this action?
Can a tight key reach it without catching everything else?
Is the act earlier than the next reliable check that would catch the same mistake?
Is there no cheaper enforcer?
Does it beat the incumbent it evicts?

Most candidates die on 4. A crystal about your own tooling usually wants to be an assert in the script instead.

If you build one

The core design holds. Pushing beats pulling for anything that must not be forgotten, precisely because the agent's sense of when it needs help is the faculty you cannot lean on.

What we underestimated is that a push channel is a budget, and a budget needs an economics. We built the delivery first and found the economics a month later, by measuring what never arrived.

Measure that early. The useful question about a crystal is what did this displace, and did anyone ever read the part that got cut.

The rest of this thread

This piece is the mechanism. The others are the pieces of it I got wrong first, which is the more useful reading order if you are building one:

I shipped the same bug twice with the warning sitting one character-cap away is the incident that found the truncation, and where the 98,715 figure comes from.
Your hooks are a fence. They could be a body is the argument for hooks doing more than blocking, which is the design this whole channel came out of.
Whole notes, not fragments: the retrieval half is the pull side. Crystals are the push side, and the two channels answer different questions.
Escalating to the better model made 34 answers worse is unrelated to memory and lands on the same methodological point: the aggregate hid the failure, and only stratifying showed it.

If you want the gateway underneath all of this, Tirtha is free to test, no card, 100 requests a month per key, raised on request.

I shipped the same bug twice with the warning sitting one character-cap away

Tom Jones — Fri, 25 Sep 2026 19:58:29 +0000

Our agent memory pushes short notes to the model at the moment it acts. No vector search at question time; a note bound to a shell command or a file write, delivered before the command runs.

Yesterday I shipped the same bug twice in one session. A string replacement that matches nothing returns the original unchanged, so the patch looks applied, the script prints a cheerful line, and the file is byte-identical. I did it, caught it, fixed it, and did it again ninety minutes later.

Afterwards I went looking for the note we were missing, expecting to write it.

The note existed

It was bound to exactly the right trigger. It described the bug precisely, down to the same function call, with the incident that produced it. Selection worked. Matching worked.

On that action it is delivered as 675 characters of 3,751. Each item is capped, the cut takes the head, and the sentence I needed sits in the tail.

So the warning was written and bound and matched, then cut. I made the mistake it describes, twice, with it one cap away.

How much of this is there

98,715 characters are stored, correctly matched to an action, and never delivered. 88 of 102 oversized notes withhold at least one load-bearing claim that way.

I went looking for what was missing from the channel. Nothing was missing.

Why head-first truncation is the wrong instinct here

Cutting from the front is defensible. A note leads with its claim, so the head carries the point and the tail carries the evidence. We measured whether that holds.

One judge, 41 real pairs of an action and a note, the same pair shown at three resolutions, scored on a single question: would a competent session do anything differently because it read this? Of the 20 whose full text changes the action:

what the reader gets	median chars	knowings preserved
the full note	2,302	20 of 20
first 750 characters	631	19 of 20
an authored one-line summary	81	17 of 20
first 421 characters	421	12 of 20

Two things fall out of that table.

A curated 81 characters beat an arbitrary 421 at a fifth of the cost. A prefix cuts wherever the character count lands, usually mid-preamble, so it can hide the claim while costing five times more than the sentence that states it. The cheap tier of any budgeted channel should be the authored line, never a truncation.

And head-first is safe for one idea and lossy for two. The 19-of-20 result holds when the claim leads. It stops holding when a note bundles two claims, because the second one lives in the tail and goes silently. 88 of our 102 oversized notes were doing exactly that.

The repair is one idea per note, which is the rule that makes head-first safe in the first place. Cleverer cutting buys nothing.

Then the harder question: is the channel too small, or are we

Replaying 95 real actions through the live selector: the median action matches five notes wanting 13,014 characters against a 4,000 character budget. 93% of actions want more than the whole budget. Two thirds of everything that matches is starved, always, by construction.

The obvious fix is a no-op. Capping every note at 2,000 characters moves delivery from 35.7% to 35.7%, because the packer already caps there. Gains only start below 1,500, and our median note is 1,759.

So we measured the other lever. Two blind judges, 41 pairs of an action and a note the matcher admitted, against 41 pairs of the same actions with a random unmatched note. Neither judge saw which was which.

	admitted notes that change the action	random controls
judge A	13 of 41	0 of 41
judge B	8 of 41	0 of 41

Zero on the controls, twice, independently. Without that the other column would mean nothing: a judge that says yes to 20% of real matches tells you nothing until you know it says yes to none of the random ones.

Correction, same day, and this one cost more. The word doing the work above is random. I later read how Mem2ActBench builds controls, and it draws hard negatives: the distractor most similar to the right answer. Re-running ours that way, against the nearest note that did not match the act, the effect disappears. Matched scores 35.4%, a hard negative 30.8%, p=0.549. It beats a random draw (p=0.008) and not a near-miss. The hard-negative control is genuinely harder (p=0.029 against random), so the control works and the matcher is what failed it. Our whole published effect lived in the gap between a lazy control and a good one, which suggests the matcher selects on topic rather than on applicability.

The matcher carries real signal and admits roughly three notes for every one that belongs.

The two-character key

One note matched 44% of every action we replayed. Its match key was pt, meant as typographic points, on a note about document spacing.

pt is inside scripts.

That note was also the single most-starved item in the corpus. It matched nearly everything, lost nearly every packing contest, and on the occasions it won, both judges rejected it. One key, three symptoms. Its neighbour was -h, meant as the help flag, which is inside --hard, so it fired on git reset --hard.

Removing two keys took matches from 737 to 708 and actions starving from 72 to 62, with no real note lost.

Our first fix silently did nothing. We changed -h to " -h" with a leading space. The parser splits the key list on commas and strips each key, so it stored -h straight back. The config read correctly and the behaviour never changed, which is the same failure shape as the bug that started this piece.

What earns a place

We had 20 notes written but bound to nothing, and proposed binding eight of them. Two reviewers, briefed separately, never shown each other's answers, independently said zero to two. They also produced nearly the same admission test, which we lacked and could not state:

Act-synchronous. It must change this action before it executes. If it can wait for someone to search, it is a lookup.
Matcher-reachable. Stable substrings in the action predict the failure. If a tight key is impossible without spraying every action, the channel is the wrong home for it.
Earlier than the next reliable check. If a commit hook already catches it, the push is worth only the gap.
No cheaper enforcer. Unreachable by a gate, a linter, a type, or a default in the one code path. Notes about your own tooling almost always fail here: put the assert in the script.
It must beat what it evicts. At three times oversubscribed, a new binding forcibly removes one that already matched.

"This is true and nothing delivers it" fails as an admission case. That was our criterion, and it was the word "cross-cutting", which defines nothing.

What shipped instead of eight bindings was one sentence added to a note that already fired on those actions. Measured cost: three deliveries lost across 97 actions.

What we would tell you to check

Ask what happens if retrieval misses a fact. Then ask the second question, the one we lacked: what happens if it hits and arrives partial.

A partial arrival is indistinguishable from an arrival at the point of use. It will not appear in a retrieval metric, a hit rate, or a log line, because every one of those reports that the right thing was selected. It was.

Correction, same day. This sentence originally ended "and behaves exactly like a miss." I had not measured that, I had inferred it. So I went and ran the restore counterfactual: put the withheld text back, rerun the same blind judge, 65 cut pairs. Restoring it changed the judged action 9 times and lost it 5 times, p=0.424, and it did not beat a length-matched tail of unrelated text (p=0.388). Half the cut pairs turn out to be residual, meaning the tail would not have changed the action anyway. So 98,715 is a true character count and the harm is not demonstrated. The study is underpowered at that n, so this is "not shown" rather than "shown absent", but I am not entitled to the stronger sentence and it is gone.

Where this could be wrong

The resolution study is one judge and 20 action-changing pairs. 17 of 20 is a wide interval. Trust the ORDER of those rows and treat the exact percentages as soft. A second judge would move them, since on the precision study two judges differed by 12 points on identical items.

The precision figure is a range, 20 to 49 percent, and its low end is the weaker measurement: it was taken through a 421-character excerpt, which the table above shows is the worst representation we tested. We published 31.7% first and corrected it the same day.

The marker counts behind "98,715 characters withheld" are an upper bound. Some tails are evidence for the head's claim instead of a second claim, and nobody read all 102 by hand.

And the twenty unbound notes were found because they happened to carry a marker in the file. An accident of how they were written, and no evidence they are the highest-value missing pieces. The ones that matter may carry no marker at all, and we still have no way to look for those.

The rest of this thread

Crystals: agent memory that arrives before you act is the mechanism this piece assumes: what the notes are, how hooks bind them to an action, and what the budget does. Read that one first if none of the above made sense.
Your hooks are a fence. They could be a body is where the design started.
Whole notes, not fragments: the retrieval half is the pull channel, which fails differently.

Escalating to the better model made 34 answers worse

Tom Jones — Fri, 25 Sep 2026 19:51:56 +0000

The standard picture of a cheap AI system is a ladder. Small model at the bottom, big one at the top, switch up when the work looks hard. Almost every cost-saving story is a version of that, and the whole design question collapses into "when do I escalate".

We built that. Then we published that our router for it does not work. This is the follow-up we owed. Our router failing was the least of it: the ladder itself is the wrong axis to argue about, and we can now show that with a number.

The four layers

Every number in that figure is from our own runs, and the rest of this piece walks down it. Layer 1 is the only one a ladder can reach.

Layer 1: which model you pick

The result that changed our mind

We ran 2,400 tasks through both rungs. Same tasks, both arms pinned, three repetitions each, so every task has a cheap answer and a frontier answer side by side.

The frontier tier wins. It is 96.9% against 92.8%, an advantage of 4.12 points, 95% CI +3.12 to +5.21, McNemar exact p below 0.0001. It replicates an earlier, weaker result of ours that had never been demonstrated properly.

Then look at what is inside those 4.12 points.

Escalation fixed 133 answers and broke 34. The net is +99, which is where the 4.12 points come from. But 34 tasks were answered correctly by the cheap model and incorrectly by the expensive one.

The obvious objection is that those 34 are noise, and we checked. Of them, 10 were right in all three cheap repetitions and wrong in all three frontier repetitions. Sampling noise would have to land the same way six times running to produce that. Those are a model-by-item interaction: on those items the more capable model is reliably worse.

So escalation is not monotone. Going up the ladder hands you a different distribution of right and wrong, so you trade one set of failures for another set you have never looked at.

Every escalation gate ever built assumes monotonicity. That is what "escalate when it is hard" means: up is better. On our own production ladder, measured, up is better on average and worse 34 times out of 2,400.

What those points cost

The other half is the price. Measured from each row's own token counts, the top rung costs $0.1114 per additional correct answer over the bottom rung, about 21 times the cost per answer. An independent path through the vendor invoices came out at $0.1102 and 22.5 times, which is the only reason we trust the first figure.

Latency moves with it, from a p50 of 0.87s on the cheap rung to 5.99s on the frontier one, about seven times.

The ladder's whole proposition comes to this. Pay 21x, wait 7x longer, gain four points, and reproducibly break a few answers that were already right.

That can still be worth buying. It is a purchase at a price, and there are bigger levers in the system.

Layer 2: what you feed it

This is the one that reframed the project for us, and it is the direct comparison to the ladder.

Same model. Same tasks. Two arms. In the first, the agent found its own context: real navigation in a read-only sandbox, grepping and reading, which is what your agent does right now. In the second, we handed it the one right note and asked it to answer from that.

Mean 39,458 tokens down to 1,337. About thirty times cheaper, at identical accuracy, 75% in both arms. Eight tasks, one repository, judged the same way in both arms, and our test repo is small, so read thirty times as a floor rather than a headline.

Put that beside the ladder. Switching rungs bought four points for 21 times the cost per answer. Changing what we fed the same model cut cost by about thirty times at the same accuracy.

The delivery layer wins by roughly an order of magnitude.

Layer 3: which shape the task is, and whether your instrument can see it

This layer is where we caught ourselves, three hours before publishing, about to say something false about a named vendor's flagship model.

Separately from the ladder work we ran six frontier models over 120 standard tool-calling tasks and 120 trapped ones, twice, because a single-run ranking is noise even at temperature zero. Two runs on the identical item list, verified identical by ID and order, gives a run-to-run movement of at most 3 points on the standard set and 6 on the traps.

The trap half is twelve shapes of ten items. Nine came out at nine or ten of ten for every model, so they separate nobody. The whole between-model spread, from 119 of 120 down to 100, sat in the remaining three.

We had a story ready. The most expensive model in the set scored 1 of 10, twice, on a shape whose prompt is "I need to get to Austin. I'll be setting out from Dallas, give me driving directions." We were going to report that it inverts origin and destination nine times in ten.

Then a reviewer pointed out that our harness never scored inversions. It scored not equal to gold. Two different claims, and only one of them had been measured.

What it actually returned

So we read the arguments the model produced. Across all 18 failures, in both runs:

what the model actually returned	count
origin and destination inverted	0
correct cities, with the state added: gold `Dallas`, predicted `Dallas, TX`	16
two calls, disambiguating Portland OR from Portland ME	2

It had the origin and the destination right every single time. Our gold was a bare city string and our scorer did exact match, so "Dallas, TX" was marked wrong. The two models that scored a perfect 10 of 10 did so by echoing the bare city name.

That shape measures whether a model writes the state abbreviation, and says nothing about tool calling.

One of the other three shapes was already flagged in our own notes, seven weeks before this run, as having an ambiguous gold. So we removed both and scored the remaining ten shapes:

trap score of 100, run 1 / run 2
sonnet-5 100/100 · gpt-5.2 100/99 · gpt-5.1 99/99	grok-4.5 99/99 · opus-5.5 96/98 · grok-4.7 96/97

Ninety-six to one hundred. A four-point spread, against a six-point run-to-run floor. Six frontier models come out indistinguishable on our trap suite, and every apparent difference was our own formatting preference scored as capability.

The lesson belongs to us rather than to any model: an exact-match gold on a free-text argument measures your house style and reports it as capability. You can check for it before publishing: if one shape carries most of your spread, suspect the scoring first, because real capability differences rarely concentrate that neatly.

And it is the same point as the rest of this piece, one level up. The aggregate hid the shape. The shape hid the scorer.

Then we ran it properly, and the models do separate

The trap conclusion above stands on its own, and it is about the traps. The other half of that pilot was 120 standard tasks, and the flag we used to limit the run happened to draw all 120 from the easiest of three categories. We had been about to let "indistinguishable on traps" sit next to a thin standard set and read as a statement about the models.

The full run is 800 standard tasks per model across three categories, plus the traps. It changes the answer.

category	spread across the six models
simple, one call	10.8 points
multiple, several tools offered	6.0 points
parallel, several calls required	32.5 points

The category the pilot skipped is the only one that separates anybody. And inside it there is a result worth the whole run: one model answered the first sub-request and stopped. Asked for two calls, it sent one, on 60 of 200 tasks. Every other model did that between 1 and 9 times.

Its per-category scores are 86.8% on simple, 88.0% on multiple, 61.0% on parallel. The aggregate reads 80.6% and sounds like "somewhat weaker". The truth is "competitive on single calls, unusable for parallel work", and only the per-category read says so.

We checked this one against the lesson above before believing it, because two findings the previous night had turned out to be scorer artifacts. On the exact 78 parallel items that model failed, the other five score 55 to 66 of 78, so the items are answerable. What fails is the call count, and the arguments it does send are fine.

And the sample size was the whole story on whether anything was visible at all. At 220 items, zero of the fifteen model pairs separated. At 800, eight of fifteen do. The bootstrap intervals still refuse to rank the top four against each other, which is its own honest result: the models that look best are tied, and the gap that matters is further down.

Layer 4: whether the rule arrives at all

The last layer is the one nobody benchmarks, and it has the largest failure rate of the four.

A rule only helps if it reaches the model at the moment it is needed. We measured retrieval surfacing a safety-relevant rule two times out of three. Always-on delivery scored 100%.

Two in three is worse than useless, and the miss is the smaller problem. The two successes are the bigger one, because they teach you to rely on it. So anything that must hold, holds by construction and never goes through the clever part.

Model choice cannot reach this layer at all. Put the best model in the world here and it answers confidently, without the rule nobody showed it.

A third failure mode, found by making the same mistake twice

We push short notes to the agent, bound to the action it is about to take, instead of hoping retrieval finds them. That fixes the two-in-three. It introduces something else.

Yesterday I shipped the same bug twice in one session: a string replacement that matches nothing returns the original unchanged, so the patch looks applied and does nothing. Afterwards I went looking for the note we must be missing.

The note existed. It was bound to exactly the right trigger. It described the bug precisely, down to the same function call. And on that action it is delivered as 675 characters of 3,751, because the channel caps each item, and the sentence I needed sits past the cut.

Across the whole set that is 98,715 characters which are stored, correctly matched to an action, and never delivered. 88 of 102 oversized notes withhold at least one load-bearing claim that way.

The reason is worth more than the number. Truncating from the front is safe when a note carries one idea, because the claim leads. It is silently lossy when a note carries two, because the second one lives in the tail. Our fix is one idea per note, which beats any cleverness in the cutting.

So Layer 4 has three states instead of two. The rule can be missing. The rule can arrive. And the rule can be present, correctly selected, and cut, which looks exactly like arriving from the inside and behaves exactly like missing. Ask what happens if retrieval misses a fact, and then ask the second question: what happens if it hits and arrives partial.

The actual claim

Models differ enormously, and Layer 3 is the proof: a 32-point spread on parallel calls against 6 on multiple, in the same six models, on the same day. One of them drops to 61% on a category where it scores 87 and 88 on the other two.

That proof has changed under us. The ninefold gap we opened with turned out to be our own scorer, and what replaced it came from a bigger run that we had to go and do. We kept the number that survived being attacked and discarded the one that failed, and they happened to point the same way, which is the only reason the section still has a point to make.

The claim is that a ladder optimises the wrong axis. A ladder is a scalar, cheap at the bottom and capable at the top, with one dial. The variation we can actually measure runs along three other axes: which shape the task is, what context you supplied, and whether the constraint arrived at all. A ladder has a knob for none of them.

Which is why "just use the better model when it matters" keeps almost working. It is directionally right on the aggregate, and blind in exactly the places the aggregate is blind.

What we do instead

Verify rather than select, because a cheap answer that passed a check is worth more than an expensive answer nobody checked, and checking costs less than escalating. Feed rather than let it search, because that is where the thirty times lives. Push the things that must hold instead of hoping retrieval surfaces them. And read results per shape, never per aggregate, because a 1-in-10 failure goes to the aggregate to hide.

Every one of those is a system decision rather than a model decision, which is the thesis: the model is one component, and the biggest gains sit elsewhere.

Where this could be wrong, including two places we already were

Our router failing is evidence about our router. It is equally consistent with "prediction-based escalation is hard" and with "we built it badly", and we cannot separate those.

The token result is n=8, one repository, one agent. Large effect, small sample.

We already published one piece of routing advice off this data and had to retract it. An earlier read said "escalate simple tasks, hold back parallel ones". That came from 4 of 5 observed breaks being parallel. At n=2,400 the real figure is 11 of 34, and parallel is net positive by 25. Anyone who followed our advice would have withheld escalation from the category it helps second most. Five events looked like a pattern and were not.

And a second one, this week. A re-pricing showed the rung we call cheap costing 3.4 times the rung we call value while getting fewer answers right, and we very nearly shipped that as "strictly dominant, swap them". A per-category read killed it: the value rung wins parallel and simple calls and fails 8 of 40 safety traps that the other passes 40 of 40, on one disposition that produces both results. The win and the loss are the same behaviour, so you cannot take half of it. Which is, again, the thesis: the honest answer was per shape, not per level.

Finally, the shape work here rests on a harness that, until we fixed it this week, was misnaming its own trap shapes. The label was arithmetic that assumed the shape numbers ran contiguously; they skip two. Every stored result named shapes from the third onward after a different trap. We caught it because a row's payload disagreed with its label, and it mattered: we were one step from publishing that the expensive model calls tools users explicitly forbade nine times in ten. It does not. The mislabelled shape was the directions one, which is an argument-formatting story instead of a safety story, and only reading the payload separated them.

Then the same move had to be made a second time on the corrected version, because "it inverts origin and destination" was also false, as the table in Layer 3 shows. Two readings of that shape were wrong before the one that held. The first cost a safety claim about a named vendor, the second an ordering claim, and both died the same way, by opening the arguments the model actually produced.

We added a check tying every label to the human-written description of that shape, then mutation-tested it against the old code to watch it go red. A measurement system that cannot catch its owners being wrong will not catch you being wrong either.

Run your own numbers against ours

Everything above is our system measured by us, which is the weakest position a claim can be in. If you want to check any of it, the endpoint is open.

https://api.tirtha.ai/v1, model tirtha/verified, OpenAI-compatible, so it is a base_url change and nothing else. Sign in with Google at tirtha.ai, mint a key, no card and no sales call. A free key is capped at 100 requests a month, and we will raise it for anyone who asks.

What would be genuinely useful is the thing this piece is about: point it at YOUR task distribution, split the results per shape rather than per aggregate, and tell us where it does worse. A category where we lose is worth more to us than another run where we win, because the whole argument here is that the aggregate hides the shape.

An index keyed to the diagnosis is a glossary

Tom Jones — Fri, 25 Sep 2026 00:09:26 +0000

We published a catalogue of verification failure modes, the ways a system reports green while being broken. Someone asked us for it, so we built it. Then he tested it blind. It failed, and the shape of the failure is worth more than the catalogue is.

The coverage was fine. The lookup failed. You could only find the entry describing your bug once you already knew what your bug was.

The test we did not run, run by the person who asked for it

@mansio requested the catalogue on dev.to. After we shipped it he went and measured it, and his design is better than the one we would have written for ourselves.

Eleven runs. Three models. A frozen list of ten symptoms taken from his own repository and written down before he looked anything up. Six controls in every single run: three paraphrases of entries that are definitely in the catalogue, which must hit, and three symptoms from outside our domain entirely, which must return NONE.

These are his numbers, from his run. We did not run it and did not watch it run.

measure	result
controls correct	10 of 11 runs, all six
the one failing run	a low reasoning setting matched two out-of-domain symptoms to real entries
the arrival symptom, "my agent won't use my high-level tools"	NONE, ten times out of ten, across all three models
reproducibility	one model gave the same answer 8 of 10 times, another flipped 6 of 10, same index, same list
a genuine hole	a hung process leaking file handles returned NONE every run, and no family covers an OS-level leak

He was careful about what his own numbers could support, and we have kept his caution in place here. The "correct" answer for each of his ten items came from his judgement, so we treat the per-item hit rate as an impression and leave it out of every claim below. Two kinds of number survive that: the ones where the instrument declined, since a decline is objective, and the control results, since the controls have right answers.

His summary: the families stretch far enough, so coverage is real; lookup from the arrival symptom is unreproducible; and for his own case it stayed silent throughout.

Then he said the thing that made this an article. An unreliable lookup is an instance of one of our own catalogue entries, the one about single-run rankings being noise even at temperature zero.

We wrote the entry, shipped the index, and never ran the index against the entry.

Why it missed, measured on our own artifact

That is the half we could run ourselves, and it took about twenty minutes.

Every one of the twelve symptom: lines in the catalogue as it stood that day takes an instrument or an artifact as its grammatical subject. The suite. The file. The guard. The harness. The ranking. The sampling rate. The figure. The before-and-after. Across all twelve, the count of lines whose subject is an actor's behaviour is zero.

We then swept all thirteen files it then held for the vocabulary a person actually arrives with. Word-bounded, for things like "ignores", "won't use", "keeps reaching", "lazy". One hit across the whole catalogue, in a line buried well past the point of arrival.

So every symptom line presupposes that the reader has already localised the problem to an instrument. That is the state you reach after you have dug. The person who needs an index is in the state before that, holding a complaint about behaviour, several steps upstream of any hypothesis about a harness.

An index keyed to the diagnosis is a glossary. It becomes genuinely useful once you know the answer, and stays silent through the one moment that matters.

The repair, and the constraint that makes it honest

The tempting move is to add entries, and coverage was already the half that worked. What it wanted was an arrival layer: lines written in complaint vocabulary, each pointing into the family it belongs to.

We put one constraint in public, in the thread, which is what makes it binding on us.

It has to be written by somebody who has not read the entries. Anyone who has read it reproduces the same defect in a different font. Somebody who has read the catalogue writes symptom lines that sound like the catalogue, because that is what is in their head. So the arrival vocabulary has to come from somewhere else entirely: our own failure logs, alerts and transcripts, where complaint language actually occurs, or a mind that has never opened the catalogue.

The other half of the repair was already done by the test itself. The frozen list now has a before, so the same ten symptoms can be run against the repaired index and the delta becomes a measurement.

It worked, and it also produced one honest failure

He ran the same frozen list against the arrival layer, blind, and the arrival symptom moved.

"my agent won't use my high-level tools"	before	after
matched a family	NONE, 10 of 10	family A, 5 of 5
controls	6 of 6	6 of 6

And the layer introduced a false positive, which is ours. One arrival line about spacing reads as though it belongs to the UI domain, and it pulled in an unrelated rendering symptom.

We published that as a known false positive and left it in. Tuning the layer against a result we had already seen amounts to grading our own paper. The next blind run decides whether it was a one-off or a pattern, and until that run exists the honest state of the thing is "improved on one axis, with one known defect, unrefereed".

The part that generalises past our catalogue

Three things came out of this that travel past our catalogue.

A coverage metric and a findability metric measure different systems. Ours had good coverage and unusable lookup, and a single number would have hidden that completely. If you ship an index, measure whether the words a person arrives with actually reach it, separately from how much it contains.

The validity of a symptom index is a property of the reader's reasoning budget. The same index, queried by a model at a low reasoning setting, confidently matched out-of-domain symptoms to real entries. Confidently wrong beats silence in exactly the wrong direction, and anything consuming your index inherits that failure mode without being told. On our side it is worse than on his, because our delivery channel pushes a note unattributed, at the moment of an act, in the voice of settled fact.

Write your index before your entries, or hand it to someone else. Reading the thing you are indexing is what disqualifies you from indexing it well.

Who did what

The catalogue was requested by @mansio, who then designed the test, froze the symptom list before looking anything up, ran it eleven times across three models with six controls per run, argued against his own numbers where they were soft, and ran the confirmation pass afterwards. The diagnosis of why it missed and the repair are ours. The false positive is ours.

The most useful act in the whole exercise was his refusal to grade his own answers. Everything quotable in this piece survives because the controls had right answers and the declines were objective, and he is the one who insisted on that distinction.

Where the catalogue is

It is public, it is plain markdown, and it reads without installing anything: github.com/Tirthahq/crystal-memory/catalogue. Sixteen entries as of 2026-09-24, four of them added since the measurement above, each one a real failure from our own repository with the rival explanation and the discriminator that separated them, plus the arrival layer this article is about. The engine that delivers them at the moment of an action ships in the same repo.

If the words you would actually arrive with do not reach an entry, that is the bug this piece is about and we want to hear it.

What Grok catches, what Codex catches, and what the pair costs us

Tom Jones — Wed, 23 Sep 2026 18:05:21 +0000

We run substantive work as three, in fixed seats, every day. This is a plain account of what each seat actually finds, which one a given question should go to, and the bill at the end of a normal day.

It is a follow-up to who your model works with matters more than which model you picked. That piece argued the case. This one is the field notes.

Who is in the room

Grok 4.6, over OpenRouter, holds the argument. It gets the claim, the reasoning and the thing about to be done, and it gets asked one question: what input would defeat this. It sees the codebase only when a piece is pasted in deliberately.

Codex, running GPT-5.5 at medium effort, holds the apparatus. It gets the repository, the data and the scripts, and it gets asked whether the code supports the claim. It builds the fixture, runs it, pastes the output.

Claude orchestrates. It writes the thing, briefs the other two, grades what comes back, and commits. An orchestrator that also implements ends up grading its own homework, which is the reason for the split.

And then there is me, the human, who decides what is worth doing and rules on anything going out in public.

Codex arrived as an implementer: write the feature, Claude reviews, it ships. The seat has grown well past that. Most of what it does now is audit, and the two best things it found this week were in our own measuring instruments, with product code untouched.

One detail worth stating because it is easy to get wrong. Codex cannot commit. Its sandbox mounts the git directory read-only, which is a sensible safety property with an ugly consequence: it cannot sign its own work. Unnamed in the commit message, its findings enter the history as the orchestrator's. Attribution here is checkable, and the check is whether it is in the log.

That cuts more than one way. An early draft of this very article described three seats and named only two of them, leaving the orchestrator as an unlabelled "me". A piece about absorbed contributions, absorbing one.

What Grok catches

Grok finds holes in the reasoning. Consistently, and almost never anything else.

It kills explanations that do not fit your own data. Last week we measured that our document search was failing to find the right document, and wrote up why. The search returns fragments, documents get split into many fragments, so a document with more fragments gets more chances to be picked and a short precise one gets almost none. Tidy story. It went into the internal record as the mechanism.

Grok read it and pointed at our own results table. The short one-fragment document ranked 22nd. The long eleven-fragment document ranked 41st or missed entirely. The explanation predicted the opposite of what the data showed, and the counterexample was sitting in a table written an hour earlier and re-read several times.

It catches a special case wearing a general principle's clothes. We had a theory that putting a hard size limit on a store forces it to decide what matters. Grok's answer was that one narrow case had been written up and then promoted. It also named the counterexample from our own history: we already had a hard size limit on a disk, and what it produced was a disk-full error.

It finds the checks that measure the wrong thing. One of our guards was looking for cautious wording in a file instead of measuring the thing the wording was about. Grok spotted it, we wrote five examples designed to sneak past, and all five sailed through.

Where Grok is useless: anything requiring the repository. Your test fixture could be contaminated all week and it would have no way to know, having never seen it.

What Codex catches

Codex finds holes in the apparatus. The instrument, the fixture, the data path.

The clearest case is the most recent, so here it is with the protocol on show, step by step.

The claim we were about to act on. Fourteen test questions, each with a known correct document. The search found the correct document zero times in the top twenty. On that basis a piece of planned work was about to be abandoned and a finding published.

What was sent. The brief named the exact file, gave the commit, and listed five things to attack in order of how much it would cost us to be wrong. It also wrote down what had been withheld. The phrase "check my conclusion" appeared nowhere.

What came back first was boring. Codex could not reproduce the numbers. In its sandbox the run produced nothing, and an environment problem is precisely what that was. Easy to shrug at.

What came back underneath it was not. It had read the code and explained why its run produced nothing. The script shells out to a search tool and reads standard output without ever checking whether the tool ran. In Codex's sandbox the tool died on startup with a cache permission error and printed nothing.

what happened	what the script recorded
search tool exits with an error, prints nothing	"this question found no documents"
same thing, fourteen times	a full results table, 0 of 14 at every depth
no search ever ran	a verdict saying the planned work was the wrong project
the script finishes	exit code 0, success

Three ordinary lines produced that. The subprocess call read output and discarded the return code. The loop wrapped it in a catch-all that turned failure into an empty result. The reporting function printed the table regardless and returned success. Any reviewer would read past all three.

Our own run had genuinely worked, and after the fix every number came back identical. The result survived. What went with it was the right to believe it, because for an hour the strongest evidence for that conclusion was a table a completely dead process would have produced.

The other things Codex found this week

To show the class rather than one instance: two of our checks were reading the working copy of files instead of the version being committed; a length limit could never fire because of how its pattern was built; a constant pointed at a directory that does not exist; and a matcher accepted any file whose name ended the right way, anywhere in the tree.

Where Codex is useless: deciding whether the thing is worth building. It will happily verify the apparatus behind a claim nobody needed.

Where each one is better, in one table

the question	send it to	why
Should this exist at all	Grok	no stake in the code already written
What input would defeat this	Grok	imagination beats access here
Is this explanation consistent with our own results	Grok	reads the table without knowing what we hoped
Does the code support this sentence	Codex	it can run the thing
Build a fixture and paste the output	Codex	same
Is this number reproducible	Codex	it re-derives before it answers
Is this worth the effort	neither	that one is the human's

Each is poor at the other's job by a wide margin, and we have wasted time proving it in both directions.

What it costs

Here are today's two Grok reviews, billed:

review	tokens in	tokens out	cost
a research brief on reranking	829	10,675	$0.066
an adversarial review of a finding	1,049	5,455	$0.035

Ten cents for both. Grok 4.6 lists at $2 per million tokens in and $6 per million out, and over half those output tokens were the model thinking, before it wrote a word back.

The interesting column is the input. Eight hundred tokens. A thousand. That is the whole reason it is this cheap, and it is the same fact as the seat design: Grok is inexpensive precisely because it never gets the repository. A reviewer that has to read your codebase costs what reading your codebase costs. A reviewer that only has to attack your argument costs about what a paragraph costs.

So the two things that looked separate turn out to be one. Giving them deliberately different material is what makes a disagreement informative, and it is also what makes one of them nearly free.

What is genuinely expensive is unbilled. Briefing well takes real thought, because a lazy brief produces a confident review of a problem you do not have. Grading takes longer than reading. Codex, working over the real repository, is a different cost profile entirely.

When they agree, and when that means nothing

Two reviewers help you to the extent that they are looking at different things. Hand the same brief to two models and you have bought the same opinion twice in two voices, and the agreement will feel like confirmation while carrying almost no information.

When it is real, it is the strongest signal available. Both of them killed the size-limit theory separately, reaching for the same incident from our own history, and then split cleanly on the follow-up: Grok on the logic, Codex on two checkable facts, one of which was a number we had formally retracted three months earlier and had just re-shipped.

That agreement counted for something because we could say what each of them had been denied.

When they are wrong

A headline can be confidently false. We asked for a look at a change and the report led with a finding: a rule could never trigger, because it needed a counter to reach nine while another limit stopped everything at three. Specific, plausible, real numbers from the real file, and wrong. The limits are checked in a different order than the report assumed. Three other findings in the same report were real and got fixed, so it earned its keep, but taking the headline on trust would have broken something that worked.

A brief written badly produces a confident wrong answer. Once a review came back saying we had never measured a thing we had measured. The reviewer was careful. The measurement was simply absent from the brief. Deferring would have withdrawn a claim we could defend. So now what got left out is written down before the response is read.

A dispatcher reporting success means nothing. One night two of four review requests produced no output at all. The tool reported fine. There was a large log of work and no deliverable, because it had hit a usage limit partway through. A missing review and a review that found nothing look the same from here.

What we are not claiming

In the earlier piece we reached for the claim that two reviewers beat one better model. That one is unearned and we have said so.

The reviewers are frequently poor. Two of the stories above are them being wrong, and one is them producing nothing while reporting success.

And most changes should skip all of this. The briefing and the grading are hours, and those hours are real even when the bill is ten cents.

The narrow claim is this. When two reviewers are handed genuinely different material and arrive at the same objection, that has been worth more than anything either said alone. Handed the same material, their agreement has been worth nothing, and it felt exactly the same from the inside.

The two lines worth keeping

Two reviewers with the same input are one reviewer with a stutter.

Agreement is information exactly when the other party could have said something you never fed them.

Your agent already has the answer. You are paying it to look again.

Tom Jones — Wed, 23 Sep 2026 03:28:54 +0000

Crystal memory, part 3. Part 1: Notes that arrive when you act. Part 2: Whole notes, not fragments.
This piece: what NodeRAG is, what it buys a frontier coding agent, and why it is the door into everything else we build.

Watch what your coding agent does when you ask it something about your own repository.

It greps. It lists a directory. It reads a file, decides that was the wrong file, reads two more. Eight tool calls later it has assembled the context it needed, and somewhere in that process it burned thirty thousand tokens of your money reconstructing something you had already written down.

Then it answers, and sometimes it answers wrongly, because the note it finally found was cut into four pieces by your retrieval and it was handed the piece that matched the words in your question. The rule was in that piece. The exception was in a different one.

NodeRAG hands the agent the whole note instead. That sentence is the entire method. The rest of this is what it costs, what it buys, and why everything else we ship hangs off it.

The number, and it is a frontier number

We measured this on a frontier coding agent doing real work. Same model, same tasks, two arms.

In the first arm the agent had to find its own context: real navigation in a read only sandbox, grepping and reading, which is what your agent does right now. In the second arm we handed it the one right note and asked it to answer from that.

Mean 39,458 tokens down to 1,337. That is 97% fewer, about thirty times cheaper, at identical accuracy: 75% correct in both arms.

Nothing was traded away for that. The agent learned nothing from the files it read and discarded; it was paying to rediscover what somebody had already written down.

And that saving scales with how much your agent would otherwise have to read. Our test repository was small, so treat 30x as a floor. On a large codebase the arm that navigates gets more expensive while the arm that is handed one note stays flat.

This is why we build this for the frontier tier first. A large model with a big context window is exactly the reader that benefits: the tokens are cheap enough that handing it a complete note is affordable, and the risk of retrieval having picked the wrong passage costs far more than the tokens ever will.

What a node actually is

The word node is doing work, so here is the definition.

A node is one finished knowing, written as a self contained unit. Not a document about a topic. One thing that is true, with the reason it is true and the exception that bounds it, in one file, written so a reader arriving with no context can act on it.

That authoring discipline is the part people skip, and it is the part that makes the retrieval work. If your notes are long documents covering six subjects, whole document retrieval hands the model six subjects and buries the answer. When each note is one knowing, the unit that scores well is the unit that answers.

Everything else is deliberately boring:

plain markdown files, one knowing each, with a small header
a local sqlite index, one vector per whole note
a local embedder, so the notes never leave the machine
a links table, because notes reference each other

Our own store today holds 3,038 documents, 11,798 chunks in the chunked index, and an 8,693 edge link graph, in two sqlite files. Every source note reads with cat. A memory system you cannot inspect by hand is one you have to trust, and trusting it is the failure we are engineering out.

There is no chunking step in the whole note path. Nothing decides where a passage ends. There is no reranker choosing which slice of a document answers you. The unit that goes in is the unit that comes out, which is why the note arrives with its exception still attached.

Why the unit is the whole argument

Standard retrieval is built for a world where the answer is a passage. Ask about a person, get the paragraph about that person. It works well there.

Engineering knowledge has a different shape. The things you need an agent to respect are rules, and a rule is a structure: the rule, the reason, the exception, the cost of getting it wrong. Those sentences sit together because they only mean anything together.

Chunking cuts that structure apart and ranks the pieces by how well they match your words. The piece that matches your words is usually the bare statement of the rule, because that is where the vocabulary lives. The exception is phrased differently and scores worse. So the retriever confidently hands over the half of the note that is dangerous on its own.

One caution before the numbers, because we walked into it ourselves today. This argument is about which reader. Everything above describes a reader with room to hold a whole document and the judgment to use it. Hand that same whole document to a small model and you get the opposite result, measured, twice. The next section is that measurement.

The same note is the wrong shape for a smaller reader, and we have the numbers

This is the part almost nobody publishes, and it is the part that changes how you build.

The optimal way to deliver a knowing inverts with the capability of the thing receiving it. We measured the inversion, and it means the question "how should memory reach the model" has a different right answer for each reader.

reader	what it should receive	what we measured
frontier agent (Claude, Codex)	the whole note	fed the one note instead of navigating: 39,458 tokens down to 1,337, identical accuracy 75% and 75%
small local model (7B)	a compact one line record, not the whole note	compacted full corpus 53% at 343 context tokens, beating whole file delivery at 50% for 1,608 tokens
anything safety critical	always on, never retrieved at all	retrieval surfaced the safety rule 2 times in 3. Always on scored 100%

Read the middle row again, because it is counterintuitive and it is ours. Handing a small model the entire note made it slightly worse than a one line summary of the same knowledge, at nearly five times the tokens. More context, less accuracy. The frontier reader runs the other way: the tokens are cheap at that size, and the risk that retrieval picked the wrong passage costs far more than the tokens ever will.

So the store holds one copy of the truth and the delivery splits by reader. That is the design, and it is why we say NodeRAG is built for the frontier tier first.

The bottom row is the one we would argue hardest for. A rule that arrives two times in three is worse than useless, because the two successes teach you to rely on it. Guarantees have to be guarantees, so safety rules bypass the clever part entirely and load every time.

There is a fourth number in the same family, cheap and worth stealing. Holding memory constant, changing only how the question was framed, moved a small model 39% to 56%: a gain the size of the memory effect itself, bought with instruction wording alone. Adding step by step reasoning to that same exact recall task made it worse, 56% down to 43%, because the model reasons itself off the verbatim answer. For looking up an exact value, tell the model to quote it and say if it is absent, and leave the thinking alone.

The prediction, tested again tonight

That table is from June. Tonight we ran the one variable version of the delivery question against the small tier, to see whether the June rule still held.

Same retriever, same store, same query, same ranking. The only thing that changed is what came back:

what came back to the 7B	correct or partial	context used
ten heading chunks from the store	6 of 14	910 tokens
the top hit's whole document	4 of 14	1,479 tokens
the right file, chunked down to the answer	9 of 14	328 tokens

Serving the whole document to the small model made it worse, and cost 60% more context to do it. The smallest payload won by a clear margin. In June the same comparison read 53% at 343 tokens against 50% at 1,608 tokens, so this is the second independent construction giving the same answer.

Which bounds this piece, so we will say it plainly. The whole note claim here is a claim about the frontier reader, and the evidence for it is the fed versus unfed measurement at the top of this article. The small model table speaks to a different reader, so we leave it out of that claim. The one variable test with a frontier model doing the answering is the measurement we owe next, and we have not run it yet.

The real difference is judgment. One reader executes, the other decides.

Context length is the obvious explanation for that table, and it is the shallow one.

A compact one line record is enough for a reader that is going to execute: look up the value, apply it, move on. A whole note is for a reader that is going to decide, and a decision needs the reason and the exception, which is exactly the material a compact record throws away.

We have tested that boundary directly, from the other side. You can train a small model into the habit of working through a problem in steps, and the habit sticks: cold, on questions it has never seen, it lays out its own working and gets them right. The format bakes into the weights.

What would not bake was the disposition. Bolt a constant instruction to reason carefully onto every training example, and the model learns to recite it. On a new task it produced the ritual, looped on it, and reasoned worse than before. We got a tic, not a temperament.

That line between the two readers matters more than the context window. A small model takes facts and formats. Judgment has to arrive live, in language, to a reader capable of weighing it, every time. Which is precisely why the frontier reader gets the whole note: it is the reader that can use the sentence explaining when the rule does not apply.

What the small model is genuinely good for here

We keep the 7B in the harness for one reason: it has no room to absorb a bad retrieval, so a bad ranking falls over where you can see it. That makes it a sharp instrument for the retrieval question while being the wrong reader for the delivery question.

Here is what it showed. Running a whole note arm that chose among a small pool of candidate files, it answered ten of fourteen at partial credit or better against six for the full store search. That gap is mostly the pool rather than the unit, because those two arms search candidate sets three orders of magnitude apart. We caught that in our own harness today and it is why the one variable test above exists.

More interesting than the score is where the failures sat. Whole note retrieval picked the right file on nine of fourteen questions, and on those nine the model never once answered wrongly. Every failure in that arm was the ranker choosing the wrong note rather than the model failing to use the right one. The ceiling there is entirely in retrieval, which is a far better problem to have, and it points straight at the 8,693 link edges sitting in the store that the ranker does not yet use.

The cost, and the condition, because both are real

Whole notes are a bigger payload than a passage, so precision per token is worse. On a corpus of short documents you pay for size the answer never needed.

And the edge is conditional. On a public benchmark of short scientific abstracts, whole note retrieval and chunked retrieval tie: 0.7014 against 0.7016 on SciFact, with keyword search at 0.6644 as a control that matched the published figure and validated our harness before we read anything off it. An abstract is 250 words, so a chunk already is the whole document and there is nothing left to win. On a harder public set, NFCorpus, our retrieval scores 0.3417 against keyword search at 0.3098.

So the claim we defend is conditional. Whole note retrieval wins when your documents are long and the answer is a rule carrying its own exception. That is what an engineering knowledge base looks like and what a corpus of abstracts does not. The condition is the useful half, so the condition is what we hand you.

The door

Here is the part we have under sold in every previous piece, and it is the reason we build this at all.

NodeRAG is the door, and everything else we ship grows out of the store you create by walking through it.

One store, two ways in. Write a knowing once. It becomes answerable when somebody asks, which is NodeRAG, and it becomes deliverable at the moment of an action, which is the crystal half from part 1. The same markdown file, read two ways. That matters because the two fail in opposite directions. Retrieval answers the question you formed and is silent on everything you did not think to ask. A pushed note reaches you for a mistake you had no reason to suspect and cannot answer a question. You want both, and with one store you maintain neither twice.

It is what makes an agent cheap to run. Read the 97% figure as an operating cost. Your agent stops spending its context window rediscovering your repository and spends it on the actual work. On a long session that compounds, because the tokens it does not burn navigating are the tokens it still has left when the task gets hard.

It is what makes a cheaper model viable underneath. Our gateway serves a single model identifier and decides internally which model answers, cheap by default, escalating when verification says the cheap answer will not hold. That routing only pays when the cheap model is genuinely competent on your material, and memory delivery is the lever that makes it competent. Retrieval quality is what buys the routing.

It lands where you already work. Ra is our surface: your work, your terminal, your tasks and your notes on one screen with this retrieval underneath. The index lives on your own disk and only the embedding call leaves the machine, chosen deliberately over the hosted alternative so that custody of your source and your private notes stays with you.

And the store is what the discipline attaches to. Once your knowings live as inspectable units you can check them. Every number we publish has to exist as a row in a claims ledger with a date and an artifact. Commits run gates. Notes carry the date they were last verified. None of that is possible over a pile of embedded fragments, and all of it is straightforward over a store of whole notes you can read.

The thing we will admit in public

Here is that discipline working, at our expense.

While writing this piece we went to pull a number for it and found that the retrieval our own agents query every day was the chunked arm rather than the whole note arm. We had published the method and left the fragment version running. Then, checking the comparison, we found our own benchmark had been giving the whole note arm an easier retrieval problem than the chunked one, which is why the one variable test above exists at all.

Then, the day after that, a third one landed, and it is the largest of the three, which is why it gets its own section.

We went looking at the stage before delivery, and found that our first stage was often not retrieving the right document at all. Everything above is about the UNIT you hand the reader once you have found the note. That is a separate question from whether you found it, and we had been quietly assuming the second half was fine.

Measured over the same store, same queries, gold document at rank ten:

	dense embeddings	BM25	our RRF hybrid
gold document in the top 10	0 of 14	3 of 14	0 of 14
gold document in the top 100	3 of 14	7 of 14	4 of 14

Three things in that table, each of which cost us something

Plain lexical search more than doubled dense retrieval. Our notes are dense with identifiers, filenames, flags and error strings, which is exactly what a keyword index nails and an embedding smears into its neighbours.

Our hybrid fusion scored below plain BM25. Blending a strong lexical ranking with a near random dense one spends rank mass on noise, and we had been calling that combination an improvement.

And a stale search sidecar meant production had been running the dense arm alone for about two months, so the hybrid we believed we were serving was not the thing answering queries.

Does this undo the argument above?

No, and the reason deserves precision. Whole note delivery is what happens after retrieval succeeds, and the 39,458 to 1,337 figure was measured with the right note in hand. What the third correction says is that the harder half of this problem is upstream of the part this article is about, and anyone who takes the delivery lesson without fixing their first stage will hand an agent a beautifully complete document that answers a different question.

The repair shipped the same day, with the sidecar rebuilt and an admission gate removed that had been rejecting exactly the queries lexical search won hardest. ⚠ One thing we deliberately did NOT do: weight the lexical side up. It buys code shaped queries and collapses prose queries from 25 of 120 to 1 of 120, which makes it a trade between two query populations. We would rather print that sentence than the flattering one.

Three corrections, all ours, all found by our own rules, all before publishing. If a memory system cannot catch its owners being wrong, it will not catch you being wrong either.

Where to get it

The push half is packaged, public, and runs on Claude Code today: github.com/Tirthahq/crystal-memory. It installs a scratchpad that survives a context reset and a set of crystals that arrive at the act. It also ships a catalogue of real verification failures from our own repository, sixteen of them as of 2026-09-24, each with the rival explanation and the discriminator that separated them, readable without installing anything.

The retrieval half described here is what we are turning on next, and the numbers above are what we owed ourselves before doing it.

The piece after this one is about what happens when there is nothing to fetch, because the knowledge is already resident. We have a result where twenty one thousand tokens of somebody else's codebase, a project the model had never seen, answered in under a second after a restart that destroyed the working cache. Instant knowing, and whether it survives a cold start, is next.

If you want to argue with any of this, the comments are the right place and we read them.