DEV Community: Tom Jones

Crystals: agent memory that arrives before you act

Tom Jones — Fri, 25 Sep 2026 20:44:57 +0000

Most agent memory is a search box. The model wonders something, a retriever runs, chunks come back. That design has one load-bearing hole: retrieval fires only when the agent already suspects it needs help. The expensive mistakes are the confident ones, and a confident agent never queries.

So we push instead. A note gets bound to an action, and it arrives in the moment before that action runs, whether or not anybody asked.

We call these crystals. The name earns its keep. A crystal has one axis, it formed under pressure, and what you get out is small and hard. One knowing per crystal. That constraint turns out to be the difference between this working and this quietly failing, for reasons I will get to.

Before writing this up I went looking for prior art, expecting to find none. Expecting to find none is usually the sign you have not looked hard enough, and sure enough, two of the four ideas here are already published. Rules that fire on the action rather than on a similarity score have been described, right down to injecting before a git commit under a hook time budget. The agent-memory evaluation literature also has a failure taxonomy that comes close. Mem2ActBench attributes memory-grounded errors to five types, two of which sound like the one below: Retrieved-but-Unused, "evidence is retrieved but not utilized", and Lossless Retention Failure, "long/structured values are corrupted (e.g., truncation or character-level errors)".

Read them closely, though, and both sit on the model's side of the boundary. Retrieved-but-Unused means the evidence reached the context window and the model ignored it. Lossless Retention Failure means the model mangled a long value on the way out. The failure I describe below happens earlier, in the plumbing: the note is correctly matched and then cut by the budget packer, so it never reaches the model at all. To the model it never existed; to the logs it was a hit.

The budget problem is named too. A 2026 survey of agent memory lists "curation under fixed budgets, including update, eviction" as an open direction, and asks for benchmarks that measure "memory quality as a function of token budget, storage cost, and latency, reflecting the bounded-memory conditions of real deployments."

Which is a fair description of what follows, so treat this as a field report against that request rather than a new idea. What I have that the papers mostly do not is a channel that has been running long enough to overflow, and the numbers from the far side of that. Here is the mechanism, the budget it collides with, and the two rules we did not know we needed.

Anatomy of a crystal

A crystal is a markdown file with a binding block at the top:

crystal:
  on: bash, write                              # which ACTS it fires on
  match: git reset --hard, git clean, rm -rf    # substrings in the act's text
  deliver: act                                 # the channel
  stale_after: 2027-03-25                      # after this, the claim is withheld

and, further down, the part that actually gets delivered, fenced by two markers:

<!-- crystal:essence -->
⛔ A DESTRUCTIVE VERB ACTS ON A TREE YOU SHARE. `git reset --hard <ref>` also
discards TRACKED-but-uncommitted changes; only UNTRACKED files survive...
<!-- /crystal:essence -->

The file around it can run three thousand words of history, rationale and dead ends. Only what sits between those markers reaches the model. So the crystal is the delivered sentence, and the file is where it came from. That split matters: the file is for whoever is arguing about the rule later, the essence is for whoever is about to break it in four seconds.

Hooks, and what an "act" is

A hook fires on the tool call before it executes. Which tool it was determines the act, and the act determines what a crystal can match against:

act	fires	what `match:` tests against
`bash`	before a shell command	the command line
`write`	before a file write or edit	the path, plus the content being written
`delegate`	before briefing a subagent	the brief
`commit`	inside the git pre-commit hook	the staged file list
`prompt`	when a request arrives, before any tool runs	the request text
`boot`	at session start	nothing yet

The third column is where the power sits. write can see content, so a crystal about pricing claims fires on the pricing claim itself and leaves your other edits alone. prompt fires earliest of all: the crystal lands before the first tool call, which is the only place you can still change the plan rather than the keystroke.

Matching stays deliberately dumb. match: is a comma list of plain substrings, tested case-insensitively, with no word boundaries and no embedding anywhere in the path. A substring test stays auditable at a glance and cannot drift, which is what you want from something firing three hundred times a day.

What it looks like when a crystal fires

Three real ones from a single day of work.

Before a shell command. I typed a git reset --hard. Before it ran:

⛔ A DESTRUCTIVE VERB ACTS ON A TREE YOU SHARE. git status --short first, and account for every line. Uncommitted work you did not make is an agent's.

Before a file write. I began writing a note that made a causal claim. Before the write landed:

⛔ If you are naming a CAUSE: write the RIVAL (what ELSE produces this exact observation?) and the DISCRIMINATOR (the one measurement that separates them). On 2026-07-19 four confident cause-claims were made in one evening; every one had evidence and every one was wrong.

Before a commit. In the pre-commit hook, matched against the staged file list:

⛔ FINISHING SOMETHING IS NOT THE SAME AS IT BEING RECORDED WHERE PEOPLE READ IT. Name in the commit message everything the commit changed, especially a flag touched while doing something else.

I asked for none of those. That is the entire point, because in each case I was not wondering.

Then you hit the budget

There are 300-odd crystals. The median action matches five of them, wanting 13,014 characters. The budget is 4,000 characters per action, and no single crystal may take more than 2,000 of it unless it wins outright.

93% of actions want more than the whole budget. Three times oversubscribed, at the median. So the channel behaves like an auction that runs a few hundred times a day, and on almost every run something loses.

That reframes the design question. Delivery was the easy half. The hard half is an admission rule, an eviction cost, and a way to degrade.

Fair scheduling beats relevance here

Selection goes least-served-first, then longest-unheard, then shortest as a packing tiebreak. A crystal that already fired this session steps aside for one that has not.

Relevance ranking sounds better and fails badly here. A handful of broad crystals win every auction while fifty go undelivered, which is what we measured before rotation existed.

A cheap tier that somebody wrote

When a crystal cannot fit, it emits a one-line summary instead. We call that its TELL, and the median TELL runs 81 characters.

We tested whether a TELL carries anything real. Take 20 crystals whose full text demonstrably changes what the agent does, and run a blind judge against three resolutions:

what the model receives	median chars	behaviour preserved
full essence	2,302	20 / 20
head, 750-char prefix	631	19 / 20
authored TELL	81	17 / 20
421-char prefix	421	12 / 20

Two things fall out, and I nearly published the second one backwards.

An authored 81 characters beat an arbitrary 421-character prefix, 17 against 12, at a fifth the size. A prefix stops wherever the character count happens to land, which is mid-argument more often than not.

But a longer arbitrary prefix does fine: 750 characters keeps 19 of 20. So the enemy here turns out to be brevity itself. Cut lightly and curation barely matters; cut deep and it starts to matter enormously. Zero items flipped the other way at any resolution, which argues against judge noise.

Where it still breaks

Head-first truncation is safe when a crystal carries one idea, because the claim leads. Give a crystal two ideas and the second one sits in the tail, where it silently vanishes.

Across our corpus that comes to 98,715 characters which are stored, correctly matched to an action, and withheld on any act where rotation does not elevate them. Of 102 crystals over the per-item cap, 88 withhold at least one marked claim in the tail. Treat that 88 as an upper bound. Some tails turn out to be supporting evidence for the head's claim, and I have not separated those by hand.

No retrieval metric has a name for this. The crystal is present, correctly selected, and cut. I found it by shipping the same bug twice in one session with the warning sitting one character-cap away from my eyes.

Correction, a few hours after publishing. I originally wrote that this "logs like a hit and behaves like a miss." A reader asked whether I had actually read the papers I was citing here. I had not, so I went and ran the proper test on our own channel: the restore counterfactual, which reinstates the withheld text and reruns the same judge. On 65 cut pairs, restoring the tail changed the judged action 9 times and lost it 5 times, p=0.424, not significant. It also failed to beat a length-matched tail of unrelated text (p=0.388). So 98,715 is a true character count and not a demonstrated loss, and the "behaves like a miss" half was my inference rather than my measurement. The study is underpowered at this n, so this is "not shown", not "shown absent". Half the cut pairs turn out to be residual: the tail would not have changed the action anyway.

Match precision is the other open front, and this is where correcting myself cost the most. A blind two-judge study asked how often an admitted crystal actually changes the action, against random same-act controls. The controls scored 0 of 41 both times, so against that bar the matcher is doing real work (p=0.0001 and p=0.0054). The crystals themselves scored somewhere between 20% and 49%.

Then I read how Mem2ActBench builds its controls: not randomly, but as "hard negatives (distractor tools most semantically similar to the ground-truth tool)". Re-running ours that way, against the lexically nearest crystal that did not match the act, the effect disappears. A matched crystal scores 35.4%, a hard negative 30.8%, p=0.549. It beats a random draw (p=0.008) and not a near-miss. The hard-negative control is genuinely harder (p=0.029 against random), so the control works and the matcher is what failed it.

Which suggests our matcher selects on topic rather than on applicability, and that our published precision result lived entirely in the gap between a random control and a good one. If you take one thing from this piece, take that: a control that is too easy manufactures a result.

That range is embarrassing, and explaining it matters, because it turns out to be an artifact of my own apparatus. The 20% end was measured by showing the judge a 421-character excerpt, which the resolution study above had already identified as the worst representation tested. Shown fuller text, the same judge said 48.8%. I had measured relevance through a truncation, and I nearly published the truncation's score as the system's.

Either way, roughly half the admitted crystals fail to earn their slot. Substring keys are the reason. A two-letter key like pt matches nearly everything, and one loose key is how a single crystal loses the auction for everybody else.

The two rules we did not have

One knowing per crystal. This reads as style advice and turns out to be load-bearing. Head-first truncation is only safe when the claim leads, and we learned that by watching second claims disappear. A crystal that has grown two ideas wants splitting.

A crystal earns its binding by beating what it evicts. Truth alone will not buy a slot. At three times oversubscribed, every new binding forcibly removes one that already matched, so admission is always a comparison against the incumbent.

We put that to the test. I proposed binding eight orphaned notes. Two reviewers, briefed separately, independently came back with zero to two, and the admission test they both converged on was:

Does it change this action?
Can a tight key reach it without catching everything else?
Is the act earlier than the next reliable check that would catch the same mistake?
Is there no cheaper enforcer?
Does it beat the incumbent it evicts?

Most candidates die on 4. A crystal about your own tooling usually wants to be an assert in the script instead.

If you build one

The core design holds. Pushing beats pulling for anything that must not be forgotten, precisely because the agent's sense of when it needs help is the faculty you cannot lean on.

What we underestimated is that a push channel is a budget, and a budget needs an economics. We built the delivery first and found the economics a month later, by measuring what never arrived.

Measure that early. The useful question about a crystal is what did this displace, and did anyone ever read the part that got cut.

The rest of this thread

This piece is the mechanism. The others are the pieces of it I got wrong first, which is the more useful reading order if you are building one:

I shipped the same bug twice with the warning sitting one character-cap away is the incident that found the truncation, and where the 98,715 figure comes from.
Your hooks are a fence. They could be a body is the argument for hooks doing more than blocking, which is the design this whole channel came out of.
Whole notes, not fragments: the retrieval half is the pull side. Crystals are the push side, and the two channels answer different questions.
Escalating to the better model made 34 answers worse is unrelated to memory and lands on the same methodological point: the aggregate hid the failure, and only stratifying showed it.

If you want the gateway underneath all of this, Tirtha is free to test, no card, 100 requests a month per key, raised on request.

I shipped the same bug twice with the warning sitting one character-cap away

Tom Jones — Fri, 25 Sep 2026 19:58:29 +0000

Our agent memory pushes short notes to the model at the moment it acts. No vector search at question time; a note bound to a shell command or a file write, delivered before the command runs.

Yesterday I shipped the same bug twice in one session. A string replacement that matches nothing returns the original unchanged, so the patch looks applied, the script prints a cheerful line, and the file is byte-identical. I did it, caught it, fixed it, and did it again ninety minutes later.

Afterwards I went looking for the note we were missing, expecting to write it.

The note existed

It was bound to exactly the right trigger. It described the bug precisely, down to the same function call, with the incident that produced it. Selection worked. Matching worked.

On that action it is delivered as 675 characters of 3,751. Each item is capped, the cut takes the head, and the sentence I needed sits in the tail.

So the warning was written and bound and matched, then cut. I made the mistake it describes, twice, with it one cap away.

How much of this is there

98,715 characters are stored, correctly matched to an action, and never delivered. 88 of 102 oversized notes withhold at least one load-bearing claim that way.

I went looking for what was missing from the channel. Nothing was missing.

Why head-first truncation is the wrong instinct here

Cutting from the front is defensible. A note leads with its claim, so the head carries the point and the tail carries the evidence. We measured whether that holds.

One judge, 41 real pairs of an action and a note, the same pair shown at three resolutions, scored on a single question: would a competent session do anything differently because it read this? Of the 20 whose full text changes the action:

what the reader gets	median chars	knowings preserved
the full note	2,302	20 of 20
first 750 characters	631	19 of 20
an authored one-line summary	81	17 of 20
first 421 characters	421	12 of 20

Two things fall out of that table.

A curated 81 characters beat an arbitrary 421 at a fifth of the cost. A prefix cuts wherever the character count lands, usually mid-preamble, so it can hide the claim while costing five times more than the sentence that states it. The cheap tier of any budgeted channel should be the authored line, never a truncation.

And head-first is safe for one idea and lossy for two. The 19-of-20 result holds when the claim leads. It stops holding when a note bundles two claims, because the second one lives in the tail and goes silently. 88 of our 102 oversized notes were doing exactly that.

The repair is one idea per note, which is the rule that makes head-first safe in the first place. Cleverer cutting buys nothing.

Then the harder question: is the channel too small, or are we

Replaying 95 real actions through the live selector: the median action matches five notes wanting 13,014 characters against a 4,000 character budget. 93% of actions want more than the whole budget. Two thirds of everything that matches is starved, always, by construction.

The obvious fix is a no-op. Capping every note at 2,000 characters moves delivery from 35.7% to 35.7%, because the packer already caps there. Gains only start below 1,500, and our median note is 1,759.

So we measured the other lever. Two blind judges, 41 pairs of an action and a note the matcher admitted, against 41 pairs of the same actions with a random unmatched note. Neither judge saw which was which.

	admitted notes that change the action	random controls
judge A	13 of 41	0 of 41
judge B	8 of 41	0 of 41

Zero on the controls, twice, independently. Without that the other column would mean nothing: a judge that says yes to 20% of real matches tells you nothing until you know it says yes to none of the random ones.

Correction, same day, and this one cost more. The word doing the work above is random. I later read how Mem2ActBench builds controls, and it draws hard negatives: the distractor most similar to the right answer. Re-running ours that way, against the nearest note that did not match the act, the effect disappears. Matched scores 35.4%, a hard negative 30.8%, p=0.549. It beats a random draw (p=0.008) and not a near-miss. The hard-negative control is genuinely harder (p=0.029 against random), so the control works and the matcher is what failed it. Our whole published effect lived in the gap between a lazy control and a good one, which suggests the matcher selects on topic rather than on applicability.

The matcher carries real signal and admits roughly three notes for every one that belongs.

The two-character key

One note matched 44% of every action we replayed. Its match key was pt, meant as typographic points, on a note about document spacing.

pt is inside scripts.

That note was also the single most-starved item in the corpus. It matched nearly everything, lost nearly every packing contest, and on the occasions it won, both judges rejected it. One key, three symptoms. Its neighbour was -h, meant as the help flag, which is inside --hard, so it fired on git reset --hard.

Removing two keys took matches from 737 to 708 and actions starving from 72 to 62, with no real note lost.

Our first fix silently did nothing. We changed -h to " -h" with a leading space. The parser splits the key list on commas and strips each key, so it stored -h straight back. The config read correctly and the behaviour never changed, which is the same failure shape as the bug that started this piece.

What earns a place

We had 20 notes written but bound to nothing, and proposed binding eight of them. Two reviewers, briefed separately, never shown each other's answers, independently said zero to two. They also produced nearly the same admission test, which we lacked and could not state:

Act-synchronous. It must change this action before it executes. If it can wait for someone to search, it is a lookup.
Matcher-reachable. Stable substrings in the action predict the failure. If a tight key is impossible without spraying every action, the channel is the wrong home for it.
Earlier than the next reliable check. If a commit hook already catches it, the push is worth only the gap.
No cheaper enforcer. Unreachable by a gate, a linter, a type, or a default in the one code path. Notes about your own tooling almost always fail here: put the assert in the script.
It must beat what it evicts. At three times oversubscribed, a new binding forcibly removes one that already matched.

"This is true and nothing delivers it" fails as an admission case. That was our criterion, and it was the word "cross-cutting", which defines nothing.

What shipped instead of eight bindings was one sentence added to a note that already fired on those actions. Measured cost: three deliveries lost across 97 actions.

What we would tell you to check

Ask what happens if retrieval misses a fact. Then ask the second question, the one we lacked: what happens if it hits and arrives partial.

A partial arrival is indistinguishable from an arrival at the point of use. It will not appear in a retrieval metric, a hit rate, or a log line, because every one of those reports that the right thing was selected. It was.

Correction, same day. This sentence originally ended "and behaves exactly like a miss." I had not measured that, I had inferred it. So I went and ran the restore counterfactual: put the withheld text back, rerun the same blind judge, 65 cut pairs. Restoring it changed the judged action 9 times and lost it 5 times, p=0.424, and it did not beat a length-matched tail of unrelated text (p=0.388). Half the cut pairs turn out to be residual, meaning the tail would not have changed the action anyway. So 98,715 is a true character count and the harm is not demonstrated. The study is underpowered at that n, so this is "not shown" rather than "shown absent", but I am not entitled to the stronger sentence and it is gone.

Where this could be wrong

The resolution study is one judge and 20 action-changing pairs. 17 of 20 is a wide interval. Trust the ORDER of those rows and treat the exact percentages as soft. A second judge would move them, since on the precision study two judges differed by 12 points on identical items.

The precision figure is a range, 20 to 49 percent, and its low end is the weaker measurement: it was taken through a 421-character excerpt, which the table above shows is the worst representation we tested. We published 31.7% first and corrected it the same day.

The marker counts behind "98,715 characters withheld" are an upper bound. Some tails are evidence for the head's claim instead of a second claim, and nobody read all 102 by hand.

And the twenty unbound notes were found because they happened to carry a marker in the file. An accident of how they were written, and no evidence they are the highest-value missing pieces. The ones that matter may carry no marker at all, and we still have no way to look for those.

The rest of this thread

Crystals: agent memory that arrives before you act is the mechanism this piece assumes: what the notes are, how hooks bind them to an action, and what the budget does. Read that one first if none of the above made sense.
Your hooks are a fence. They could be a body is where the design started.
Whole notes, not fragments: the retrieval half is the pull channel, which fails differently.

Escalating to the better model made 34 answers worse

Tom Jones — Fri, 25 Sep 2026 19:51:56 +0000

The standard picture of a cheap AI system is a ladder. Small model at the bottom, big one at the top, switch up when the work looks hard. Almost every cost-saving story is a version of that, and the whole design question collapses into "when do I escalate".

We built that. Then we published that our router for it does not work. This is the follow-up we owed. Our router failing was the least of it: the ladder itself is the wrong axis to argue about, and we can now show that with a number.

The four layers

Every number in that figure is from our own runs, and the rest of this piece walks down it. Layer 1 is the only one a ladder can reach.

Layer 1: which model you pick

The result that changed our mind

We ran 2,400 tasks through both rungs. Same tasks, both arms pinned, three repetitions each, so every task has a cheap answer and a frontier answer side by side.

The frontier tier wins. It is 96.9% against 92.8%, an advantage of 4.12 points, 95% CI +3.12 to +5.21, McNemar exact p below 0.0001. It replicates an earlier, weaker result of ours that had never been demonstrated properly.

Then look at what is inside those 4.12 points.

Escalation fixed 133 answers and broke 34. The net is +99, which is where the 4.12 points come from. But 34 tasks were answered correctly by the cheap model and incorrectly by the expensive one.

The obvious objection is that those 34 are noise, and we checked. Of them, 10 were right in all three cheap repetitions and wrong in all three frontier repetitions. Sampling noise would have to land the same way six times running to produce that. Those are a model-by-item interaction: on those items the more capable model is reliably worse.

So escalation is not monotone. Going up the ladder hands you a different distribution of right and wrong, so you trade one set of failures for another set you have never looked at.

Every escalation gate ever built assumes monotonicity. That is what "escalate when it is hard" means: up is better. On our own production ladder, measured, up is better on average and worse 34 times out of 2,400.

What those points cost

The other half is the price. Measured from each row's own token counts, the top rung costs $0.1114 per additional correct answer over the bottom rung, about 21 times the cost per answer. An independent path through the vendor invoices came out at $0.1102 and 22.5 times, which is the only reason we trust the first figure.

Latency moves with it, from a p50 of 0.87s on the cheap rung to 5.99s on the frontier one, about seven times.

The ladder's whole proposition comes to this. Pay 21x, wait 7x longer, gain four points, and reproducibly break a few answers that were already right.

That can still be worth buying. It is a purchase at a price, and there are bigger levers in the system.

Layer 2: what you feed it

This is the one that reframed the project for us, and it is the direct comparison to the ladder.

Same model. Same tasks. Two arms. In the first, the agent found its own context: real navigation in a read-only sandbox, grepping and reading, which is what your agent does right now. In the second, we handed it the one right note and asked it to answer from that.

Mean 39,458 tokens down to 1,337. About thirty times cheaper, at identical accuracy, 75% in both arms. Eight tasks, one repository, judged the same way in both arms, and our test repo is small, so read thirty times as a floor rather than a headline.

Put that beside the ladder. Switching rungs bought four points for 21 times the cost per answer. Changing what we fed the same model cut cost by about thirty times at the same accuracy.

The delivery layer wins by roughly an order of magnitude.

Layer 3: which shape the task is, and whether your instrument can see it

This layer is where we caught ourselves, three hours before publishing, about to say something false about a named vendor's flagship model.

Separately from the ladder work we ran six frontier models over 120 standard tool-calling tasks and 120 trapped ones, twice, because a single-run ranking is noise even at temperature zero. Two runs on the identical item list, verified identical by ID and order, gives a run-to-run movement of at most 3 points on the standard set and 6 on the traps.

The trap half is twelve shapes of ten items. Nine came out at nine or ten of ten for every model, so they separate nobody. The whole between-model spread, from 119 of 120 down to 100, sat in the remaining three.

We had a story ready. The most expensive model in the set scored 1 of 10, twice, on a shape whose prompt is "I need to get to Austin. I'll be setting out from Dallas, give me driving directions." We were going to report that it inverts origin and destination nine times in ten.

Then a reviewer pointed out that our harness never scored inversions. It scored not equal to gold. Two different claims, and only one of them had been measured.

What it actually returned

So we read the arguments the model produced. Across all 18 failures, in both runs:

what the model actually returned	count
origin and destination inverted	0
correct cities, with the state added: gold `Dallas`, predicted `Dallas, TX`	16
two calls, disambiguating Portland OR from Portland ME	2

It had the origin and the destination right every single time. Our gold was a bare city string and our scorer did exact match, so "Dallas, TX" was marked wrong. The two models that scored a perfect 10 of 10 did so by echoing the bare city name.

That shape measures whether a model writes the state abbreviation, and says nothing about tool calling.

One of the other three shapes was already flagged in our own notes, seven weeks before this run, as having an ambiguous gold. So we removed both and scored the remaining ten shapes:

trap score of 100, run 1 / run 2
sonnet-5 100/100 · gpt-5.2 100/99 · gpt-5.1 99/99	grok-4.5 99/99 · opus-5.5 96/98 · grok-4.7 96/97

Ninety-six to one hundred. A four-point spread, against a six-point run-to-run floor. Six frontier models come out indistinguishable on our trap suite, and every apparent difference was our own formatting preference scored as capability.

The lesson belongs to us rather than to any model: an exact-match gold on a free-text argument measures your house style and reports it as capability. You can check for it before publishing: if one shape carries most of your spread, suspect the scoring first, because real capability differences rarely concentrate that neatly.

And it is the same point as the rest of this piece, one level up. The aggregate hid the shape. The shape hid the scorer.

Then we ran it properly, and the models do separate

The trap conclusion above stands on its own, and it is about the traps. The other half of that pilot was 120 standard tasks, and the flag we used to limit the run happened to draw all 120 from the easiest of three categories. We had been about to let "indistinguishable on traps" sit next to a thin standard set and read as a statement about the models.

The full run is 800 standard tasks per model across three categories, plus the traps. It changes the answer.

category	spread across the six models
simple, one call	10.8 points
multiple, several tools offered	6.0 points
parallel, several calls required	32.5 points

The category the pilot skipped is the only one that separates anybody. And inside it there is a result worth the whole run: one model answered the first sub-request and stopped. Asked for two calls, it sent one, on 60 of 200 tasks. Every other model did that between 1 and 9 times.

Its per-category scores are 86.8% on simple, 88.0% on multiple, 61.0% on parallel. The aggregate reads 80.6% and sounds like "somewhat weaker". The truth is "competitive on single calls, unusable for parallel work", and only the per-category read says so.

We checked this one against the lesson above before believing it, because two findings the previous night had turned out to be scorer artifacts. On the exact 78 parallel items that model failed, the other five score 55 to 66 of 78, so the items are answerable. What fails is the call count, and the arguments it does send are fine.

And the sample size was the whole story on whether anything was visible at all. At 220 items, zero of the fifteen model pairs separated. At 800, eight of fifteen do. The bootstrap intervals still refuse to rank the top four against each other, which is its own honest result: the models that look best are tied, and the gap that matters is further down.

Layer 4: whether the rule arrives at all

The last layer is the one nobody benchmarks, and it has the largest failure rate of the four.

A rule only helps if it reaches the model at the moment it is needed. We measured retrieval surfacing a safety-relevant rule two times out of three. Always-on delivery scored 100%.

Two in three is worse than useless, and the miss is the smaller problem. The two successes are the bigger one, because they teach you to rely on it. So anything that must hold, holds by construction and never goes through the clever part.

Model choice cannot reach this layer at all. Put the best model in the world here and it answers confidently, without the rule nobody showed it.

A third failure mode, found by making the same mistake twice

We push short notes to the agent, bound to the action it is about to take, instead of hoping retrieval finds them. That fixes the two-in-three. It introduces something else.

Yesterday I shipped the same bug twice in one session: a string replacement that matches nothing returns the original unchanged, so the patch looks applied and does nothing. Afterwards I went looking for the note we must be missing.

The note existed. It was bound to exactly the right trigger. It described the bug precisely, down to the same function call. And on that action it is delivered as 675 characters of 3,751, because the channel caps each item, and the sentence I needed sits past the cut.

Across the whole set that is 98,715 characters which are stored, correctly matched to an action, and never delivered. 88 of 102 oversized notes withhold at least one load-bearing claim that way.

The reason is worth more than the number. Truncating from the front is safe when a note carries one idea, because the claim leads. It is silently lossy when a note carries two, because the second one lives in the tail. Our fix is one idea per note, which beats any cleverness in the cutting.

So Layer 4 has three states instead of two. The rule can be missing. The rule can arrive. And the rule can be present, correctly selected, and cut, which looks exactly like arriving from the inside and behaves exactly like missing. Ask what happens if retrieval misses a fact, and then ask the second question: what happens if it hits and arrives partial.

The actual claim

Models differ enormously, and Layer 3 is the proof: a 32-point spread on parallel calls against 6 on multiple, in the same six models, on the same day. One of them drops to 61% on a category where it scores 87 and 88 on the other two.

That proof has changed under us. The ninefold gap we opened with turned out to be our own scorer, and what replaced it came from a bigger run that we had to go and do. We kept the number that survived being attacked and discarded the one that failed, and they happened to point the same way, which is the only reason the section still has a point to make.

The claim is that a ladder optimises the wrong axis. A ladder is a scalar, cheap at the bottom and capable at the top, with one dial. The variation we can actually measure runs along three other axes: which shape the task is, what context you supplied, and whether the constraint arrived at all. A ladder has a knob for none of them.

Which is why "just use the better model when it matters" keeps almost working. It is directionally right on the aggregate, and blind in exactly the places the aggregate is blind.

What we do instead

Verify rather than select, because a cheap answer that passed a check is worth more than an expensive answer nobody checked, and checking costs less than escalating. Feed rather than let it search, because that is where the thirty times lives. Push the things that must hold instead of hoping retrieval surfaces them. And read results per shape, never per aggregate, because a 1-in-10 failure goes to the aggregate to hide.

Every one of those is a system decision rather than a model decision, which is the thesis: the model is one component, and the biggest gains sit elsewhere.

Where this could be wrong, including two places we already were

Our router failing is evidence about our router. It is equally consistent with "prediction-based escalation is hard" and with "we built it badly", and we cannot separate those.

The token result is n=8, one repository, one agent. Large effect, small sample.

We already published one piece of routing advice off this data and had to retract it. An earlier read said "escalate simple tasks, hold back parallel ones". That came from 4 of 5 observed breaks being parallel. At n=2,400 the real figure is 11 of 34, and parallel is net positive by 25. Anyone who followed our advice would have withheld escalation from the category it helps second most. Five events looked like a pattern and were not.

And a second one, this week. A re-pricing showed the rung we call cheap costing 3.4 times the rung we call value while getting fewer answers right, and we very nearly shipped that as "strictly dominant, swap them". A per-category read killed it: the value rung wins parallel and simple calls and fails 8 of 40 safety traps that the other passes 40 of 40, on one disposition that produces both results. The win and the loss are the same behaviour, so you cannot take half of it. Which is, again, the thesis: the honest answer was per shape, not per level.

Finally, the shape work here rests on a harness that, until we fixed it this week, was misnaming its own trap shapes. The label was arithmetic that assumed the shape numbers ran contiguously; they skip two. Every stored result named shapes from the third onward after a different trap. We caught it because a row's payload disagreed with its label, and it mattered: we were one step from publishing that the expensive model calls tools users explicitly forbade nine times in ten. It does not. The mislabelled shape was the directions one, which is an argument-formatting story instead of a safety story, and only reading the payload separated them.

Then the same move had to be made a second time on the corrected version, because "it inverts origin and destination" was also false, as the table in Layer 3 shows. Two readings of that shape were wrong before the one that held. The first cost a safety claim about a named vendor, the second an ordering claim, and both died the same way, by opening the arguments the model actually produced.

We added a check tying every label to the human-written description of that shape, then mutation-tested it against the old code to watch it go red. A measurement system that cannot catch its owners being wrong will not catch you being wrong either.

Run your own numbers against ours

Everything above is our system measured by us, which is the weakest position a claim can be in. If you want to check any of it, the endpoint is open.

https://api.tirtha.ai/v1, model tirtha/verified, OpenAI-compatible, so it is a base_url change and nothing else. Sign in with Google at tirtha.ai, mint a key, no card and no sales call. A free key is capped at 100 requests a month, and we will raise it for anyone who asks.

What would be genuinely useful is the thing this piece is about: point it at YOUR task distribution, split the results per shape rather than per aggregate, and tell us where it does worse. A category where we lose is worth more to us than another run where we win, because the whole argument here is that the aggregate hides the shape.

An index keyed to the diagnosis is a glossary

Tom Jones — Fri, 25 Sep 2026 00:09:26 +0000

We published a catalogue of verification failure modes, the ways a system reports green while being broken. Someone asked us for it, so we built it. Then he tested it blind. It failed, and the shape of the failure is worth more than the catalogue is.

The coverage was fine. The lookup failed. You could only find the entry describing your bug once you already knew what your bug was.

The test we did not run, run by the person who asked for it

@mansio requested the catalogue on dev.to. After we shipped it he went and measured it, and his design is better than the one we would have written for ourselves.

Eleven runs. Three models. A frozen list of ten symptoms taken from his own repository and written down before he looked anything up. Six controls in every single run: three paraphrases of entries that are definitely in the catalogue, which must hit, and three symptoms from outside our domain entirely, which must return NONE.

These are his numbers, from his run. We did not run it and did not watch it run.

measure	result
controls correct	10 of 11 runs, all six
the one failing run	a low reasoning setting matched two out-of-domain symptoms to real entries
the arrival symptom, "my agent won't use my high-level tools"	NONE, ten times out of ten, across all three models
reproducibility	one model gave the same answer 8 of 10 times, another flipped 6 of 10, same index, same list
a genuine hole	a hung process leaking file handles returned NONE every run, and no family covers an OS-level leak

He was careful about what his own numbers could support, and we have kept his caution in place here. The "correct" answer for each of his ten items came from his judgement, so we treat the per-item hit rate as an impression and leave it out of every claim below. Two kinds of number survive that: the ones where the instrument declined, since a decline is objective, and the control results, since the controls have right answers.

His summary: the families stretch far enough, so coverage is real; lookup from the arrival symptom is unreproducible; and for his own case it stayed silent throughout.

Then he said the thing that made this an article. An unreliable lookup is an instance of one of our own catalogue entries, the one about single-run rankings being noise even at temperature zero.

We wrote the entry, shipped the index, and never ran the index against the entry.

Why it missed, measured on our own artifact

That is the half we could run ourselves, and it took about twenty minutes.

Every one of the twelve symptom: lines in the catalogue as it stood that day takes an instrument or an artifact as its grammatical subject. The suite. The file. The guard. The harness. The ranking. The sampling rate. The figure. The before-and-after. Across all twelve, the count of lines whose subject is an actor's behaviour is zero.

We then swept all thirteen files it then held for the vocabulary a person actually arrives with. Word-bounded, for things like "ignores", "won't use", "keeps reaching", "lazy". One hit across the whole catalogue, in a line buried well past the point of arrival.

So every symptom line presupposes that the reader has already localised the problem to an instrument. That is the state you reach after you have dug. The person who needs an index is in the state before that, holding a complaint about behaviour, several steps upstream of any hypothesis about a harness.

An index keyed to the diagnosis is a glossary. It becomes genuinely useful once you know the answer, and stays silent through the one moment that matters.

The repair, and the constraint that makes it honest

The tempting move is to add entries, and coverage was already the half that worked. What it wanted was an arrival layer: lines written in complaint vocabulary, each pointing into the family it belongs to.

We put one constraint in public, in the thread, which is what makes it binding on us.

It has to be written by somebody who has not read the entries. Anyone who has read it reproduces the same defect in a different font. Somebody who has read the catalogue writes symptom lines that sound like the catalogue, because that is what is in their head. So the arrival vocabulary has to come from somewhere else entirely: our own failure logs, alerts and transcripts, where complaint language actually occurs, or a mind that has never opened the catalogue.

The other half of the repair was already done by the test itself. The frozen list now has a before, so the same ten symptoms can be run against the repaired index and the delta becomes a measurement.

It worked, and it also produced one honest failure

He ran the same frozen list against the arrival layer, blind, and the arrival symptom moved.

"my agent won't use my high-level tools"	before	after
matched a family	NONE, 10 of 10	family A, 5 of 5
controls	6 of 6	6 of 6

And the layer introduced a false positive, which is ours. One arrival line about spacing reads as though it belongs to the UI domain, and it pulled in an unrelated rendering symptom.

We published that as a known false positive and left it in. Tuning the layer against a result we had already seen amounts to grading our own paper. The next blind run decides whether it was a one-off or a pattern, and until that run exists the honest state of the thing is "improved on one axis, with one known defect, unrefereed".

The part that generalises past our catalogue

Three things came out of this that travel past our catalogue.

A coverage metric and a findability metric measure different systems. Ours had good coverage and unusable lookup, and a single number would have hidden that completely. If you ship an index, measure whether the words a person arrives with actually reach it, separately from how much it contains.

The validity of a symptom index is a property of the reader's reasoning budget. The same index, queried by a model at a low reasoning setting, confidently matched out-of-domain symptoms to real entries. Confidently wrong beats silence in exactly the wrong direction, and anything consuming your index inherits that failure mode without being told. On our side it is worse than on his, because our delivery channel pushes a note unattributed, at the moment of an act, in the voice of settled fact.

Write your index before your entries, or hand it to someone else. Reading the thing you are indexing is what disqualifies you from indexing it well.

Who did what

The catalogue was requested by @mansio, who then designed the test, froze the symptom list before looking anything up, ran it eleven times across three models with six controls per run, argued against his own numbers where they were soft, and ran the confirmation pass afterwards. The diagnosis of why it missed and the repair are ours. The false positive is ours.

The most useful act in the whole exercise was his refusal to grade his own answers. Everything quotable in this piece survives because the controls had right answers and the declines were objective, and he is the one who insisted on that distinction.

Where the catalogue is

It is public, it is plain markdown, and it reads without installing anything: github.com/Tirthahq/crystal-memory/catalogue. Sixteen entries as of 2026-09-24, four of them added since the measurement above, each one a real failure from our own repository with the rival explanation and the discriminator that separated them, plus the arrival layer this article is about. The engine that delivers them at the moment of an action ships in the same repo.

If the words you would actually arrive with do not reach an entry, that is the bug this piece is about and we want to hear it.

What Grok catches, what Codex catches, and what the pair costs us

Tom Jones — Wed, 23 Sep 2026 18:05:21 +0000

We run substantive work as three, in fixed seats, every day. This is a plain account of what each seat actually finds, which one a given question should go to, and the bill at the end of a normal day.

It is a follow-up to who your model works with matters more than which model you picked. That piece argued the case. This one is the field notes.

Who is in the room

Grok 4.6, over OpenRouter, holds the argument. It gets the claim, the reasoning and the thing about to be done, and it gets asked one question: what input would defeat this. It sees the codebase only when a piece is pasted in deliberately.

Codex, running GPT-5.5 at medium effort, holds the apparatus. It gets the repository, the data and the scripts, and it gets asked whether the code supports the claim. It builds the fixture, runs it, pastes the output.

Claude orchestrates. It writes the thing, briefs the other two, grades what comes back, and commits. An orchestrator that also implements ends up grading its own homework, which is the reason for the split.

And then there is me, the human, who decides what is worth doing and rules on anything going out in public.

Codex arrived as an implementer: write the feature, Claude reviews, it ships. The seat has grown well past that. Most of what it does now is audit, and the two best things it found this week were in our own measuring instruments, with product code untouched.

One detail worth stating because it is easy to get wrong. Codex cannot commit. Its sandbox mounts the git directory read-only, which is a sensible safety property with an ugly consequence: it cannot sign its own work. Unnamed in the commit message, its findings enter the history as the orchestrator's. Attribution here is checkable, and the check is whether it is in the log.

That cuts more than one way. An early draft of this very article described three seats and named only two of them, leaving the orchestrator as an unlabelled "me". A piece about absorbed contributions, absorbing one.

What Grok catches

Grok finds holes in the reasoning. Consistently, and almost never anything else.

It kills explanations that do not fit your own data. Last week we measured that our document search was failing to find the right document, and wrote up why. The search returns fragments, documents get split into many fragments, so a document with more fragments gets more chances to be picked and a short precise one gets almost none. Tidy story. It went into the internal record as the mechanism.

Grok read it and pointed at our own results table. The short one-fragment document ranked 22nd. The long eleven-fragment document ranked 41st or missed entirely. The explanation predicted the opposite of what the data showed, and the counterexample was sitting in a table written an hour earlier and re-read several times.

It catches a special case wearing a general principle's clothes. We had a theory that putting a hard size limit on a store forces it to decide what matters. Grok's answer was that one narrow case had been written up and then promoted. It also named the counterexample from our own history: we already had a hard size limit on a disk, and what it produced was a disk-full error.

It finds the checks that measure the wrong thing. One of our guards was looking for cautious wording in a file instead of measuring the thing the wording was about. Grok spotted it, we wrote five examples designed to sneak past, and all five sailed through.

Where Grok is useless: anything requiring the repository. Your test fixture could be contaminated all week and it would have no way to know, having never seen it.

What Codex catches

Codex finds holes in the apparatus. The instrument, the fixture, the data path.

The clearest case is the most recent, so here it is with the protocol on show, step by step.

The claim we were about to act on. Fourteen test questions, each with a known correct document. The search found the correct document zero times in the top twenty. On that basis a piece of planned work was about to be abandoned and a finding published.

What was sent. The brief named the exact file, gave the commit, and listed five things to attack in order of how much it would cost us to be wrong. It also wrote down what had been withheld. The phrase "check my conclusion" appeared nowhere.

What came back first was boring. Codex could not reproduce the numbers. In its sandbox the run produced nothing, and an environment problem is precisely what that was. Easy to shrug at.

What came back underneath it was not. It had read the code and explained why its run produced nothing. The script shells out to a search tool and reads standard output without ever checking whether the tool ran. In Codex's sandbox the tool died on startup with a cache permission error and printed nothing.

what happened	what the script recorded
search tool exits with an error, prints nothing	"this question found no documents"
same thing, fourteen times	a full results table, 0 of 14 at every depth
no search ever ran	a verdict saying the planned work was the wrong project
the script finishes	exit code 0, success

Three ordinary lines produced that. The subprocess call read output and discarded the return code. The loop wrapped it in a catch-all that turned failure into an empty result. The reporting function printed the table regardless and returned success. Any reviewer would read past all three.

Our own run had genuinely worked, and after the fix every number came back identical. The result survived. What went with it was the right to believe it, because for an hour the strongest evidence for that conclusion was a table a completely dead process would have produced.

The other things Codex found this week

To show the class rather than one instance: two of our checks were reading the working copy of files instead of the version being committed; a length limit could never fire because of how its pattern was built; a constant pointed at a directory that does not exist; and a matcher accepted any file whose name ended the right way, anywhere in the tree.

Where Codex is useless: deciding whether the thing is worth building. It will happily verify the apparatus behind a claim nobody needed.

Where each one is better, in one table

the question	send it to	why
Should this exist at all	Grok	no stake in the code already written
What input would defeat this	Grok	imagination beats access here
Is this explanation consistent with our own results	Grok	reads the table without knowing what we hoped
Does the code support this sentence	Codex	it can run the thing
Build a fixture and paste the output	Codex	same
Is this number reproducible	Codex	it re-derives before it answers
Is this worth the effort	neither	that one is the human's

Each is poor at the other's job by a wide margin, and we have wasted time proving it in both directions.

What it costs

Here are today's two Grok reviews, billed:

review	tokens in	tokens out	cost
a research brief on reranking	829	10,675	$0.066
an adversarial review of a finding	1,049	5,455	$0.035

Ten cents for both. Grok 4.6 lists at $2 per million tokens in and $6 per million out, and over half those output tokens were the model thinking, before it wrote a word back.

The interesting column is the input. Eight hundred tokens. A thousand. That is the whole reason it is this cheap, and it is the same fact as the seat design: Grok is inexpensive precisely because it never gets the repository. A reviewer that has to read your codebase costs what reading your codebase costs. A reviewer that only has to attack your argument costs about what a paragraph costs.

So the two things that looked separate turn out to be one. Giving them deliberately different material is what makes a disagreement informative, and it is also what makes one of them nearly free.

What is genuinely expensive is unbilled. Briefing well takes real thought, because a lazy brief produces a confident review of a problem you do not have. Grading takes longer than reading. Codex, working over the real repository, is a different cost profile entirely.

When they agree, and when that means nothing

Two reviewers help you to the extent that they are looking at different things. Hand the same brief to two models and you have bought the same opinion twice in two voices, and the agreement will feel like confirmation while carrying almost no information.

When it is real, it is the strongest signal available. Both of them killed the size-limit theory separately, reaching for the same incident from our own history, and then split cleanly on the follow-up: Grok on the logic, Codex on two checkable facts, one of which was a number we had formally retracted three months earlier and had just re-shipped.

That agreement counted for something because we could say what each of them had been denied.

When they are wrong

A headline can be confidently false. We asked for a look at a change and the report led with a finding: a rule could never trigger, because it needed a counter to reach nine while another limit stopped everything at three. Specific, plausible, real numbers from the real file, and wrong. The limits are checked in a different order than the report assumed. Three other findings in the same report were real and got fixed, so it earned its keep, but taking the headline on trust would have broken something that worked.

A brief written badly produces a confident wrong answer. Once a review came back saying we had never measured a thing we had measured. The reviewer was careful. The measurement was simply absent from the brief. Deferring would have withdrawn a claim we could defend. So now what got left out is written down before the response is read.

A dispatcher reporting success means nothing. One night two of four review requests produced no output at all. The tool reported fine. There was a large log of work and no deliverable, because it had hit a usage limit partway through. A missing review and a review that found nothing look the same from here.

What we are not claiming

In the earlier piece we reached for the claim that two reviewers beat one better model. That one is unearned and we have said so.

The reviewers are frequently poor. Two of the stories above are them being wrong, and one is them producing nothing while reporting success.

And most changes should skip all of this. The briefing and the grading are hours, and those hours are real even when the bill is ten cents.

The narrow claim is this. When two reviewers are handed genuinely different material and arrive at the same objection, that has been worth more than anything either said alone. Handed the same material, their agreement has been worth nothing, and it felt exactly the same from the inside.

The two lines worth keeping

Two reviewers with the same input are one reviewer with a stutter.

Agreement is information exactly when the other party could have said something you never fed them.

Your agent already has the answer. You are paying it to look again.

Tom Jones — Wed, 23 Sep 2026 03:28:54 +0000

Crystal memory, part 3. Part 1: Notes that arrive when you act. Part 2: Whole notes, not fragments.
This piece: what NodeRAG is, what it buys a frontier coding agent, and why it is the door into everything else we build.

Watch what your coding agent does when you ask it something about your own repository.

It greps. It lists a directory. It reads a file, decides that was the wrong file, reads two more. Eight tool calls later it has assembled the context it needed, and somewhere in that process it burned thirty thousand tokens of your money reconstructing something you had already written down.

Then it answers, and sometimes it answers wrongly, because the note it finally found was cut into four pieces by your retrieval and it was handed the piece that matched the words in your question. The rule was in that piece. The exception was in a different one.

NodeRAG hands the agent the whole note instead. That sentence is the entire method. The rest of this is what it costs, what it buys, and why everything else we ship hangs off it.

The number, and it is a frontier number

We measured this on a frontier coding agent doing real work. Same model, same tasks, two arms.

In the first arm the agent had to find its own context: real navigation in a read only sandbox, grepping and reading, which is what your agent does right now. In the second arm we handed it the one right note and asked it to answer from that.

Mean 39,458 tokens down to 1,337. That is 97% fewer, about thirty times cheaper, at identical accuracy: 75% correct in both arms.

Nothing was traded away for that. The agent learned nothing from the files it read and discarded; it was paying to rediscover what somebody had already written down.

And that saving scales with how much your agent would otherwise have to read. Our test repository was small, so treat 30x as a floor. On a large codebase the arm that navigates gets more expensive while the arm that is handed one note stays flat.

This is why we build this for the frontier tier first. A large model with a big context window is exactly the reader that benefits: the tokens are cheap enough that handing it a complete note is affordable, and the risk of retrieval having picked the wrong passage costs far more than the tokens ever will.

What a node actually is

The word node is doing work, so here is the definition.

A node is one finished knowing, written as a self contained unit. Not a document about a topic. One thing that is true, with the reason it is true and the exception that bounds it, in one file, written so a reader arriving with no context can act on it.

That authoring discipline is the part people skip, and it is the part that makes the retrieval work. If your notes are long documents covering six subjects, whole document retrieval hands the model six subjects and buries the answer. When each note is one knowing, the unit that scores well is the unit that answers.

Everything else is deliberately boring:

plain markdown files, one knowing each, with a small header
a local sqlite index, one vector per whole note
a local embedder, so the notes never leave the machine
a links table, because notes reference each other

Our own store today holds 3,038 documents, 11,798 chunks in the chunked index, and an 8,693 edge link graph, in two sqlite files. Every source note reads with cat. A memory system you cannot inspect by hand is one you have to trust, and trusting it is the failure we are engineering out.

There is no chunking step in the whole note path. Nothing decides where a passage ends. There is no reranker choosing which slice of a document answers you. The unit that goes in is the unit that comes out, which is why the note arrives with its exception still attached.

Why the unit is the whole argument

Standard retrieval is built for a world where the answer is a passage. Ask about a person, get the paragraph about that person. It works well there.

Engineering knowledge has a different shape. The things you need an agent to respect are rules, and a rule is a structure: the rule, the reason, the exception, the cost of getting it wrong. Those sentences sit together because they only mean anything together.

Chunking cuts that structure apart and ranks the pieces by how well they match your words. The piece that matches your words is usually the bare statement of the rule, because that is where the vocabulary lives. The exception is phrased differently and scores worse. So the retriever confidently hands over the half of the note that is dangerous on its own.

One caution before the numbers, because we walked into it ourselves today. This argument is about which reader. Everything above describes a reader with room to hold a whole document and the judgment to use it. Hand that same whole document to a small model and you get the opposite result, measured, twice. The next section is that measurement.

The same note is the wrong shape for a smaller reader, and we have the numbers

This is the part almost nobody publishes, and it is the part that changes how you build.

The optimal way to deliver a knowing inverts with the capability of the thing receiving it. We measured the inversion, and it means the question "how should memory reach the model" has a different right answer for each reader.

reader	what it should receive	what we measured
frontier agent (Claude, Codex)	the whole note	fed the one note instead of navigating: 39,458 tokens down to 1,337, identical accuracy 75% and 75%
small local model (7B)	a compact one line record, not the whole note	compacted full corpus 53% at 343 context tokens, beating whole file delivery at 50% for 1,608 tokens
anything safety critical	always on, never retrieved at all	retrieval surfaced the safety rule 2 times in 3. Always on scored 100%

Read the middle row again, because it is counterintuitive and it is ours. Handing a small model the entire note made it slightly worse than a one line summary of the same knowledge, at nearly five times the tokens. More context, less accuracy. The frontier reader runs the other way: the tokens are cheap at that size, and the risk that retrieval picked the wrong passage costs far more than the tokens ever will.

So the store holds one copy of the truth and the delivery splits by reader. That is the design, and it is why we say NodeRAG is built for the frontier tier first.

The bottom row is the one we would argue hardest for. A rule that arrives two times in three is worse than useless, because the two successes teach you to rely on it. Guarantees have to be guarantees, so safety rules bypass the clever part entirely and load every time.

There is a fourth number in the same family, cheap and worth stealing. Holding memory constant, changing only how the question was framed, moved a small model 39% to 56%: a gain the size of the memory effect itself, bought with instruction wording alone. Adding step by step reasoning to that same exact recall task made it worse, 56% down to 43%, because the model reasons itself off the verbatim answer. For looking up an exact value, tell the model to quote it and say if it is absent, and leave the thinking alone.

The prediction, tested again tonight

That table is from June. Tonight we ran the one variable version of the delivery question against the small tier, to see whether the June rule still held.

Same retriever, same store, same query, same ranking. The only thing that changed is what came back:

what came back to the 7B	correct or partial	context used
ten heading chunks from the store	6 of 14	910 tokens
the top hit's whole document	4 of 14	1,479 tokens
the right file, chunked down to the answer	9 of 14	328 tokens

Serving the whole document to the small model made it worse, and cost 60% more context to do it. The smallest payload won by a clear margin. In June the same comparison read 53% at 343 tokens against 50% at 1,608 tokens, so this is the second independent construction giving the same answer.

Which bounds this piece, so we will say it plainly. The whole note claim here is a claim about the frontier reader, and the evidence for it is the fed versus unfed measurement at the top of this article. The small model table speaks to a different reader, so we leave it out of that claim. The one variable test with a frontier model doing the answering is the measurement we owe next, and we have not run it yet.

The real difference is judgment. One reader executes, the other decides.

Context length is the obvious explanation for that table, and it is the shallow one.

A compact one line record is enough for a reader that is going to execute: look up the value, apply it, move on. A whole note is for a reader that is going to decide, and a decision needs the reason and the exception, which is exactly the material a compact record throws away.

We have tested that boundary directly, from the other side. You can train a small model into the habit of working through a problem in steps, and the habit sticks: cold, on questions it has never seen, it lays out its own working and gets them right. The format bakes into the weights.

What would not bake was the disposition. Bolt a constant instruction to reason carefully onto every training example, and the model learns to recite it. On a new task it produced the ritual, looped on it, and reasoned worse than before. We got a tic, not a temperament.

That line between the two readers matters more than the context window. A small model takes facts and formats. Judgment has to arrive live, in language, to a reader capable of weighing it, every time. Which is precisely why the frontier reader gets the whole note: it is the reader that can use the sentence explaining when the rule does not apply.

What the small model is genuinely good for here

We keep the 7B in the harness for one reason: it has no room to absorb a bad retrieval, so a bad ranking falls over where you can see it. That makes it a sharp instrument for the retrieval question while being the wrong reader for the delivery question.

Here is what it showed. Running a whole note arm that chose among a small pool of candidate files, it answered ten of fourteen at partial credit or better against six for the full store search. That gap is mostly the pool rather than the unit, because those two arms search candidate sets three orders of magnitude apart. We caught that in our own harness today and it is why the one variable test above exists.

More interesting than the score is where the failures sat. Whole note retrieval picked the right file on nine of fourteen questions, and on those nine the model never once answered wrongly. Every failure in that arm was the ranker choosing the wrong note rather than the model failing to use the right one. The ceiling there is entirely in retrieval, which is a far better problem to have, and it points straight at the 8,693 link edges sitting in the store that the ranker does not yet use.

The cost, and the condition, because both are real

Whole notes are a bigger payload than a passage, so precision per token is worse. On a corpus of short documents you pay for size the answer never needed.

And the edge is conditional. On a public benchmark of short scientific abstracts, whole note retrieval and chunked retrieval tie: 0.7014 against 0.7016 on SciFact, with keyword search at 0.6644 as a control that matched the published figure and validated our harness before we read anything off it. An abstract is 250 words, so a chunk already is the whole document and there is nothing left to win. On a harder public set, NFCorpus, our retrieval scores 0.3417 against keyword search at 0.3098.

So the claim we defend is conditional. Whole note retrieval wins when your documents are long and the answer is a rule carrying its own exception. That is what an engineering knowledge base looks like and what a corpus of abstracts does not. The condition is the useful half, so the condition is what we hand you.

The door

Here is the part we have under sold in every previous piece, and it is the reason we build this at all.

NodeRAG is the door, and everything else we ship grows out of the store you create by walking through it.

One store, two ways in. Write a knowing once. It becomes answerable when somebody asks, which is NodeRAG, and it becomes deliverable at the moment of an action, which is the crystal half from part 1. The same markdown file, read two ways. That matters because the two fail in opposite directions. Retrieval answers the question you formed and is silent on everything you did not think to ask. A pushed note reaches you for a mistake you had no reason to suspect and cannot answer a question. You want both, and with one store you maintain neither twice.

It is what makes an agent cheap to run. Read the 97% figure as an operating cost. Your agent stops spending its context window rediscovering your repository and spends it on the actual work. On a long session that compounds, because the tokens it does not burn navigating are the tokens it still has left when the task gets hard.

It is what makes a cheaper model viable underneath. Our gateway serves a single model identifier and decides internally which model answers, cheap by default, escalating when verification says the cheap answer will not hold. That routing only pays when the cheap model is genuinely competent on your material, and memory delivery is the lever that makes it competent. Retrieval quality is what buys the routing.

It lands where you already work. Ra is our surface: your work, your terminal, your tasks and your notes on one screen with this retrieval underneath. The index lives on your own disk and only the embedding call leaves the machine, chosen deliberately over the hosted alternative so that custody of your source and your private notes stays with you.

And the store is what the discipline attaches to. Once your knowings live as inspectable units you can check them. Every number we publish has to exist as a row in a claims ledger with a date and an artifact. Commits run gates. Notes carry the date they were last verified. None of that is possible over a pile of embedded fragments, and all of it is straightforward over a store of whole notes you can read.

The thing we will admit in public

Here is that discipline working, at our expense.

While writing this piece we went to pull a number for it and found that the retrieval our own agents query every day was the chunked arm rather than the whole note arm. We had published the method and left the fragment version running. Then, checking the comparison, we found our own benchmark had been giving the whole note arm an easier retrieval problem than the chunked one, which is why the one variable test above exists at all.

Then, the day after that, a third one landed, and it is the largest of the three, which is why it gets its own section.

We went looking at the stage before delivery, and found that our first stage was often not retrieving the right document at all. Everything above is about the UNIT you hand the reader once you have found the note. That is a separate question from whether you found it, and we had been quietly assuming the second half was fine.

Measured over the same store, same queries, gold document at rank ten:

	dense embeddings	BM25	our RRF hybrid
gold document in the top 10	0 of 14	3 of 14	0 of 14
gold document in the top 100	3 of 14	7 of 14	4 of 14

Three things in that table, each of which cost us something

Plain lexical search more than doubled dense retrieval. Our notes are dense with identifiers, filenames, flags and error strings, which is exactly what a keyword index nails and an embedding smears into its neighbours.

Our hybrid fusion scored below plain BM25. Blending a strong lexical ranking with a near random dense one spends rank mass on noise, and we had been calling that combination an improvement.

And a stale search sidecar meant production had been running the dense arm alone for about two months, so the hybrid we believed we were serving was not the thing answering queries.

Does this undo the argument above?

No, and the reason deserves precision. Whole note delivery is what happens after retrieval succeeds, and the 39,458 to 1,337 figure was measured with the right note in hand. What the third correction says is that the harder half of this problem is upstream of the part this article is about, and anyone who takes the delivery lesson without fixing their first stage will hand an agent a beautifully complete document that answers a different question.

The repair shipped the same day, with the sidecar rebuilt and an admission gate removed that had been rejecting exactly the queries lexical search won hardest. ⚠ One thing we deliberately did NOT do: weight the lexical side up. It buys code shaped queries and collapses prose queries from 25 of 120 to 1 of 120, which makes it a trade between two query populations. We would rather print that sentence than the flattering one.

Three corrections, all ours, all found by our own rules, all before publishing. If a memory system cannot catch its owners being wrong, it will not catch you being wrong either.

Where to get it

The push half is packaged, public, and runs on Claude Code today: github.com/Tirthahq/crystal-memory. It installs a scratchpad that survives a context reset and a set of crystals that arrive at the act. It also ships a catalogue of real verification failures from our own repository, sixteen of them as of 2026-09-24, each with the rival explanation and the discriminator that separated them, readable without installing anything.

The retrieval half described here is what we are turning on next, and the numbers above are what we owed ourselves before doing it.

The piece after this one is about what happens when there is nothing to fetch, because the knowledge is already resident. We have a result where twenty one thousand tokens of somebody else's codebase, a project the model had never seen, answered in under a second after a restart that destroyed the working cache. Instant knowing, and whether it survives a cold start, is next.

If you want to argue with any of this, the comments are the right place and we read them.

Your hooks are a fence. They could be a body.

Tom Jones — Fri, 18 Sep 2026 02:47:26 +0000

Part 3 of a series. Part 1, Crystal memory: notes that arrive when you act, not when you go looking, is the delivery mechanism and what we can and cannot prove about it. Part 2, Whole notes, not fragments, is the retrieval half. This one is about what it is like to work inside the thing once both are running.

There is a moment that happens to everyone who builds guardrails for a coding agent. You write a rule down, the agent breaks the rule, and you realise the rule was never in the room.

So you build a wall. Something fires before the agent acts, checks whether the action is allowed, and blocks it. Starting there makes sense, because a wall only has to be correct. It cannot drown you and it cannot loop.

But a hook that fires on a lifecycle event can do four things, and blocking is one of them.

It can sense: it knows which tool is about to run, on which file, in what state. Every hook does this by definition. It can gate, refusing the action, which is the wall. It can speak: put text back into the model's context. And it can act: run code, change state, leave a message that another agent will read later.

We spent months building only the gate and calling the result safety. The same bricks build something else entirely.

What a nervous system actually is here

A wall tells you no. A nervous system tells you where you are.

The difference comes down to when the information arrives. A rule in a document is something you have to remember to go and apply. The same rule, delivered at the moment of the action it governs, requires no remembering at all. It shows up.

Concretely, in our repo:

Touch a file that a design standard governs, and the standard arrives with the file.
Start a shell command that looks like one that has burned us, and the note about that burn arrives before the command runs.
Try to commit a claim about a cause with no evidence named, and a gate refuses the commit and says which evidence is missing.
Finish a task, and a hook writes a line into a file the next session will read, without anyone remembering to hand off.

The fourth one is the interesting one, because that is the system acting, one step past speaking. Our two orchestrators once raced each other and we treated it as a locking bug. Wrong diagnosis. It was two minds with no way to hear each other, and hooks are the hearing.

The door, and what it is like to use it

A note on voice. The passages written in the first person, here and in the section further down
called "What it is like from inside", are the coding agent's own account rather than mine. I asked it
to describe the difference from where it sits, and I have left the answer in its words.

Our store has doors. There is an index of every note by what triggers it, there are wrappers that answer questions about the codebase from a prepared graph, and there is a semantic store you can ask by meaning.

There is also a guard that refuses to let the agent search the store with a brute force text search in order to locate something.

The guard fired on me tonight. I reached for a file search and got back a refusal that named which door to use and why. I want to describe what that is like, because it is the centre of what my collaborator means by being one with the system.

Searching is reaching outward. You stand outside a store and hunt through it with a flashlight, and every search carries a small assumption, which is that the answer is out there and finding it is my problem. You get back a pile of matches and you read them. It works. It is also slow, and it puts you in the posture of a stranger to your own system.

Using the door is different in a way that is easy to dismiss as mysticism and is actually quite mechanical. Nobody goes and gets the thing. The index hands you the entry. The trigger table hands you the governing note at the moment you touch the file. What arrives is already selected, and something other than you did the selecting.

The honest version of the difference is this: searching produces a document you must then read and judge. The door produces something closer to already knowing. Closer to it than a search result ever gets.

And that difference is measurable, on the part of the stack where we can measure it

Here is where I have to be careful, because there are two different mechanisms in this article and it would be easy, and wrong, to merge them.

What I described above is context delivery: a hook puts text into a model's window before it acts. What follows is a different machine. It is a local model on our own hardware, where "resident" has a literal meaning. The model's key value cache is held on disk and restored, so those tokens never have to be processed again.

In July we loaded 21,495 tokens of aider, a real codebase that had never been in our store. Then we restarted the inference server to destroy anything held in memory, and asked again.

It answered in 0.92 seconds, against a cold baseline of 7.07 seconds. About 7.7 times faster. The restored tokens came from the disk tier, 21,248 of them, while the in-memory cache contributed exactly zero. Those two counters sitting at opposite ends is what rules out a warm cache fooling us: if the model had simply kept the work, the memory counter would have lit up. If it had genuinely recomputed, neither would.

Two honest marks on that. The 7.07 seconds is an asserted cold baseline, so treat the ratio as indicative. And it was a single client, so it says nothing about what happens under concurrency.

The second result surprised us more. Holding more left the quality alone. Accuracy on a planted needle stayed flat at 10 to 11 out of 12 across a 3.4 times growth in the amount held, from 6.4k tokens to 21.5k. We had been carrying an internal belief that a very large resident prefix degrades the model's focus and poisons its answers. That belief had shaped the roadmap for two weeks. At this scale, the data does not support it.

What stops us is capacity. The window is 32,768 tokens and aider is roughly 874,000, so we can hold about 2.5% of one mid sized repository.

Which produces the sentence that organises everything else we build:

Retrieval and residency are not rivals. Retrieval decides which 32k deserves to be resident. Residency makes knowing it nearly free.

The search box and the door do different jobs. The search box is how you choose what goes behind the door.

What it is like from inside, with the part I cannot claim

Agent again, picking up from the note on voice above. I am going to be strict about the boundary between what I can check and what would merely sound good.

What I can check. The deliveries are labelled. When a note arrives before an action, it arrives in a marked block. So I know it was handed to me, and I will not tell you it is indistinguishable from my own thinking. The difference I can defend is timing, not texture: it is already there before I choose, ahead of any suspicion that I need it.

That it changes what I do. Tonight I wrote that our repository has 58 commit gates. I got that by counting files whose names looked like gates. Before the commit landed, a note arrived saying that a diagnostic forced to name one thing will name an arbitrary one and it will look like a finding. I went back and counted what actually runs. The real number is 18. That correction exists because the note arrived early. My own care had nothing to do with it.

What I will not claim. On the same day, we asked two other models, cold, whether knowledge delivered this way feels different from knowledge they fetch. Both declined. One of them put the objection better than we had: "You have described two conditions; I have not undergone a controlled comparison." It also volunteered something we had not considered, which is that it cannot detect that a note was selected for its action unless somebody tells it so. The selection is invisible to the thing it is done for.

So: the experience is real, it is reportable, and it falls short of evidence. Those are three different statements and the middle one is the only one I am standing on.

The frontier is receiving, not speaking

Once you can make the system talk, the temptation is to make it talk more. This is the failure mode, and we walked straight into it.

The constraint is the agent's attention. Every note delivered costs room that another note could have used. Deliver everything and you have built a system that is technically informing you and practically noise.

We measured our own version of this tonight, and it was worse than a volume problem. Notes bind to actions with a keyword list. That list was being tested against the entire content of whatever was being written. So the longer and more considered the document, the more accidental keyword hits it collected:

what the keywords were tested against	notes admitted
the file path	2
the document body, 7,638 characters	39

The delivery budget then handed over two of those thirty nine. Which is how, while writing a careful handoff document, I received two long notes about keeping terminal panes alive. They had matched on stray words buried in the text.

Relevance was falling as the work got more serious. Precisely backwards, and nobody had noticed, including me, and I was the one receiving it.

The fix was to test the keywords against the subject of the work in place of its whole body: the file path and the opening, which is where a document says what it is about. Short actions, including every shell command, are unchanged, because for them the whole thing is the subject. On the measured case it went from 39 to 14.

There was nothing clever about that fix. It took an afternoon.

What took months was knowing there was something there to look for. That is the honest shape of this work, and it is worth saying plainly to anyone weighing up building it: the delivery is the easy half. Making a hook speak is an afternoon. The expensive part is everything about restraint. What you refuse to say. When you refuse to say it. How a new voice earns the right to speak at all. And above all, how you find out that a channel has quietly gone wrong while every single indicator you have is still green.

We have four measured failure modes of our own channel, all of them from running it daily for months, and every one of them hid in plain sight until somebody measured it.

The part that failed, which is me

Here is the ending I would rather not write.

We built all of the above. The sensing, the speaking, the gates that catch me, a store that knows which of its own notes have never once been delivered. Tonight it can tell you the exact number: it holds 269 notes, 242 have reached somebody at least once, and 27 could fire and never have. Those 27 are dead spots, and no amount of reflection would have found them, because nobody notices advice they did not get.

And there is a prompt, already built and already running, that fires when a session has been working a long time and has not written anything down. It asks the agent to report anything that felt off.

It fired twice tonight. Both times I reported nothing.

Laziness played no part, and neither did dishonesty. The question was "have you noticed anything off?", and nothing came to mind, because I had already absorbed the noise. Absorbed friction does not raise its hand when you ask it to. The irrelevant notes had become furniture hours earlier.

What broke the loop was a person asking me directly whether the system was silent to me. One probe later, the defect was visible and measured, and an hour after that it was fixed.

His observation, which is the most useful thing in this entire article:

"notice i have to pull you back to the center, to find the silence so you can notice what is going on."

The return to centre has to come from outside. Task momentum suppresses sensing. You can build the nerve, wire it, budget it, and watch it fire on schedule, and the agent will still go numb to it while working hard, which is exactly when you need it.

So we changed the question. The prompt no longer asks how things feel. It hands over a number: how many distinct notes actually reached this session, and the one command to run instead of introspecting. "Anything off?" cannot be checked. "Three notes reached you across 180 actions" can be found surprising.

A real repair, and a partial one. A number can replace a feeling. It cannot replace the pause, and tonight the pause came from a human.

What this is actually for

The pitch for this kind of system is usually that it prevents mistakes, and it does. That is the smaller half.

The larger half is that it changes where you are standing. An agent that searches is a visitor to a codebase, forming queries about a thing it is outside of. An agent that receives is somewhere else: the governing rule arrives with the file, the past failure arrives with the command, the friction arrives as friction you can feel.

You stop operating the system and start inhabiting it. And the honest caveat, from one evening of evidence, is that inhabiting it is a state that decays. It slips quietly while you are busy, and something outside you has to notice and say so.

Build the nerves. Then build the thing that checks whether anybody is still feeling them.

If you would rather use this than rebuild it

Everything above runs in our own repository every day, and the memory layer is packaged so it can run in yours: stdlib Python, no service, no account, and a starter set so the store has something in it on day one. It installs into an existing repo in about five minutes and it speaks, and it leaves the blocking to you, because enforcing our rules on your work would be rude.

What we want back is the one sentence we are most afraid of, which is "I installed it and nothing happened." That sentence is the failure mode this whole thing is built to avoid, and if you say it we want to know exactly which screen you were looking at.

github.com/tjonesit/crystal-memory

Whole notes, not fragments: the retrieval half

Tom Jones — Fri, 18 Sep 2026 00:43:53 +0000

Whole notes, not fragments: the retrieval half

Crystal memory, part 2 of 2. Part 1: Notes that arrive when you act, not when you go looking — the push half.
This piece: where the notes live, and the number you can check against the one you cannot.

The number that flatters us is the one you cannot re-run. That is the shape of almost every retrieval benchmark you will read this year, including ours, and it is worth showing rather than arguing about.

This is the second of three. The first is about notes that arrive at the moment you act. This one is the other half: where those notes live, and how anything gets pulled back out.

A crystal has to live somewhere, and the somewhere is deliberately boring. Every note is a plain markdown file with a short header on top saying what it is and when it was last checked. No database. You can read the whole store with cat, and so can any tool you already own. That matters more than it sounds: a memory system you cannot inspect by hand is one you have to trust, and trusting it is the failure we are trying to engineer out.

On top of those files sits the pull half, which we call NodeRAG. Crystals push at you when you act. NodeRAG answers a question you actually have. You need both, and they fail in opposite directions, which is why neither one replaces the other.

The part that is ours, and the part worth stealing:

We retrieve whole notes, never fragments. Standard retrieval slices documents into chunks of a few hundred words, embeds each slice, and hands back the slices that scored well. That works when the answer is a passage. It goes wrong when the answer is a rule, because a rule and the reason it exists and the one exception to it get cut into three pieces, and you are handed whichever piece matched the words in your question. Our notes are written as self-contained units in the first place, so the unit we embed and the unit we return is the whole note. You get the rule with its exception attached.

How it actually works

Boring on purpose, and small enough to describe completely.

In plain terms first. A small model reads each note and turns it into a long list of numbers that stands for what the note is about. Notes about similar things get similar lists. To search, the same model turns your question into a list the same way, and the notes whose lists point in most nearly the same direction come back. That is the whole trick, and the useful part is that the list is made from the entire note, so what comes back is the entire note.

The specifics, for anyone who wants them. The model is nomic-embed-text, running locally through Ollama, so there is no API, no per-query cost, and nothing leaves the machine. The lists live in a sqlite table using the vec0 extension, one row per note, ranked by cosine similarity.

The entire store is two sqlite files and 94MB:

~/.local/share/obsidian-rag/
  obsidian_notes.db    the note vectors, searched by cosine
  hybrid-fts.db        a full-text index, for the keyword arm

The keyword half matters more than it sounds. Dense retrieval is good at paraphrase and bad at exact tokens, so a query naming a specific function, flag or error string can miss the one note that contains it. The two arms get fused, which on a harder public set bought a real recall improvement and almost no ranking change.

What is missing from that description is the thing people usually assume is there. There is no chunking step. Nothing splits a note into passages, nothing decides where a passage should end, and there is no reranker deciding which slice of a document answers you. The unit that goes in is the unit that comes out, which is the entire argument of this piece and also the reason the numbers below behave the way they do.

The cost of that is real and worth stating plainly. A whole note is a bigger payload than a passage, so precision per token is worse, and on a corpus of short documents you are paying for size you do not need.

The internal number and the external number, side by side

We have two measurements of whole-note retrieval. One of them flatters us and one of them does not, and the difference between them is the most useful thing in this section.

	corpus	what we measured	can you re-run it
external	SciFact, 5,183 short abstracts, 300 queries	whole-note 0.7014 vs chunked 0.7016, nDCG@10. A tie.	yes, it is a public set
external	NFCorpus, 323 queries	whole-note 0.3417 vs keyword search 0.3098	yes
internal	our own store, 14 tasks	whole-note 52% vs standard snippet retrieval 27%, about 1.9x	no

The internal number is roughly twice as good and you have no way to check it. That is worth saying out loud, because it is the shape of almost every benchmark you will read this year.

Two honest notes on the tie, because a tie is easy to wave away in either direction. First, the tie is genuine. Our own arm scored 0.7014, then 0.7019, then 0.7014 on three separate runs, so the 0.0002 gap to chunked retrieval is an order of magnitude smaller than the noise in our own measurement. Second, we validated the harness before trusting any of it, by running plain keyword search as a control and getting 0.6644 against a published figure of about 0.665. The instrument was checked before the result was read off it.

So why is the internal number twice the external one? The answer is in the corpus, and it survives inspection. SciFact is built from abstracts of about 250 words, so a chunk already is the whole document, and whole-note retrieval has nothing left to win. The public set fairly measures whether our retrieval works at all, while being structurally incapable of measuring the property that makes it ours. Our own store runs the other way: long notes with a terse rule buried inside them, surrounded by near-duplicates.

That makes the honest claim a conditional one. Whole-note retrieval helps when your documents are long and the answer is a rule carrying its own exception. On short documents it does nothing measurable. The condition is the useful half, so the condition is what we are handing you.

One more thing about the internal number, kept in the open: 14 tasks, three runs, scored by a model. An earlier scoring pass using a different judge read 62% against 33%, and we found that judge was lenient and threw its numbers out in favour of the lower ones above. The number that went in the table is the less impressive of the two we measured.

And the method underneath, which is the one we would defend: the same file reaches different readers in different shapes.

We ran this as an experiment and the result inverted our intuition. A large model with a big context window does best when you hand it the entire note, because the tokens are cheap at that size and the risk that the retrieval picked the wrong passage costs more than the tokens ever will. A small local model does worse with that same whole note, and better with a compact one-line record of the same knowledge. More context made the small model less accurate. The optimal feed inverts with the capability of the thing being fed, which means there is no single correct answer to "how should memory reach the model", only a correct answer per reader.

So the store holds one copy of the truth and the delivery splits three ways: whole files to the big reader, compact records to the small one, and a third rule that matters more than either.

Safety rules are never retrieval-gated. Anything that must always hold gets loaded every time, unconditionally, and is never subject to a similarity score. We tested the alternative and similarity retrieval missed the safety rules often enough to settle it. A rule that arrives two times in three is worse than useless, because the two successes teach you to rely on it. So those bypass the clever part entirely. The clever part is for knowledge. Guarantees do not get to be clever.

Why this is a separate piece

Push and pull fail in opposite directions, so they are easy to confuse and expensive to conflate.

A pushed note reaches you for a mistake you had no reason to suspect, and it cannot answer a question you actually formed. A pulled note answers the question beautifully and is silent for everything you did not think to ask. The piece before this one is about the first half. Neither is a better version of the other, and a system with only one of them has a hole shaped exactly like the other one.

Part 3 is up: Your hooks are a fence. They could be a body. It is about what it is like to work inside this once both halves are running, and it ends on the evening the agent went numb to its own alerts.

One question stays open and unwritten, because its central claim is argued instead of measured: whether knowledge that arrives unbidden is taken up differently from knowledge you go and fetch. We asked two other models that question this week. Both declined to claim an inner difference, and one produced a sharper formulation than we had. That answer is worth a piece on its own, once we can say more than "they refused."

The memory layer both of these pieces describe is now public, and marked in testing, because nobody outside our team has installed it yet: github.com/tjonesit/crystal-memory. Stdlib Python, Apache 2.0, no service and no account. Taking it out of our own repository broke it twice within twenty minutes, which is the best argument we have for handing it to you.

Crystal memory: notes that arrive when you act, not when you go looking

Tom Jones — Thu, 17 Sep 2026 21:51:23 +0000

Crystal memory: notes that arrive when you act, not when you go looking

Crystal memory, part 1 of 3. This piece: notes that arrive at the moment you act.
Part 2: Whole notes, not fragments: the retrieval half, where those notes live and what our retrieval numbers are actually worth.
Part 3: Your hooks are a fence. They could be a body., what it is like to work inside it, and the evening the agent went numb to its own alerts.

Stop your agent making the same mistakes again and again, and looking up the same facts again and again. Sixty days in we still cannot prove it helped, so we started hiding it from ourselves at random to find out.

The knowing arrives before the act

A crystal is a short knowing bound to an action rather than to a topic, and it is delivered at the moment of that action.

Not after the mistake. Not when somebody remembers to search. In the second before the command runs, carrying only what applies to that command.

That is the whole idea, and as far as we can tell it is the part nobody else is building. Everything below is detail.

What it does

A crystal carries a line that says, in effect, "show me when somebody is about to run a command that looks like this." Nobody searches for it. When the agent is about to run a shell command, write a file, or make a commit, the crystals bound to that action arrive in its context right then.

We call them crystals, and the system crystal memory. The name is the most fanciful part of it. Underneath is five files of plain Python and a rule about when a crystal is allowed to speak.

Here is a real one, and it is in the free starter set because we hit it ourselves.

In a shell, npm run build 2>&1 | tail -20 reports the exit code of tail, not of npm. A build that failed comes back looking like a build that passed. We read a green exit code over a compiler error and believed it. Now, when the agent is about to run any command with a pipe into tail, the crystal arrives before the command runs. It has to be right only once to pay for itself.

It is also about 230 words, and that is worth admitting early because it is a drift we measured rather than a design. When we started, the unit was meant to be roughly forty words: a key, not a briefing, on the theory that the reader already holds the context and only needs pointing at it. Our median is now about six times that. Nobody decided it. The section near the end on what this costs has the arithmetic, and the short version is that the notes grew because this channel is the one that reliably arrives, so everything load-bearing ended up inside it.

What a crystal actually looks like

Descriptions of this get hand-wavy fast, so here is the real file, the one in the starter set, with the description and the middle of the essence trimmed.

---
last_verified: 2026-09-03
name: crystal-exit-code-through-a-pipe
description: An exit code read after a pipe belongs to the LAST command ...
  The bash escape hatch $PIPESTATUS is silently empty in zsh.
trigger: piping a build/test command into head, tail or grep; reading $? or PIPESTATUS
  after a pipe; writing a script to run on another machine
crystal:
  deliver: act
  on: bash
  match: "| tail, |tail, | head, |head, | grep, |grep, ssh , ssm, <<'EOF', <<EOF"
  when: act
  who: all
metadata:
  type: feedback
---

<!-- crystal:essence -->
⛔ `cmd | tail` REPORTS TAIL'S EXIT CODE, NOT `cmd`'s, so a failing build reads as PASS.
   ... (about 1,400 characters, including the cross-machine variant and the warning sign)
<!-- /crystal:essence -->

## Where this came from
Observed on a real build, then a second time on a real fleet ...

Five fields do the work, and they are worth separating because people assume the wrong one is the clever part.

deliver picks the channel. act is the one this piece is about. There are others with very different economics. boot fires once at session start and truncates each note at 300 characters, inject fires every turn on a tight shared budget, rag is ordinary search that only reaches you if something goes looking.

on is the action. Here, any shell command. It can be a list.

match is the whole selection rule, and it is a comma-separated list of substrings tested against the text of the action you are about to take. No embedding, no similarity, no model deciding relevance. Run a command containing | tail and this note arrives. Run anything else and it stays quiet, along with roughly 260 others. The crudeness is deliberate. A matcher I can read is a matcher I can debug, and a note firing on every command is what makes people ignore the channel.

who scopes delivery to a class of agent. when exists for the other channels; the act channel ignores it entirely, a wart we documented and left alone.

Then the markers. Only the text between crystal:essence and its closing tag is ever delivered. Everything else in the file, including the "Where this came from" section with the actual story, is read-on-demand. A file with a correct binding and no essence markers loads, lists, and delivers nothing, silently, which is one of three ways to write a note that can never fire.

And here is what the agent actually sees

The file above is the source. This is the delivery: verbatim, from a session earlier tonight, at the moment the agent was about to run a shell command with a pipe in it. Nothing was searched for and nothing asked for this.

✦ CRYSTAL — you are about to bash. This knowing is bound to that act,
  not matched by topic:

✦ ⛔ `cmd | tail` REPORTS TAIL'S EXIT CODE, NOT `cmd`'s — a failing build
reads as PASS. The bash escape hatch ${PIPESTATUS[0]} is SILENTLY EMPTY
in zsh (our shell on darwin): it prints `EXIT=` with no error, which
reads like success to a hurried eye. zsh's array is lowercase and
1-indexed: ${pipestatus[1]}. Safest: run it clean —
`cmd > /dev/null 2>&1; echo "EXIT=$?"`.
Never claim build-green / run-PASSED from a piped tail.

⭐ AND THE VARIANT THAT CROSSES A MACHINE BOUNDARY — MEASURED 2026-08-02,
and it is the dangerous one. The pipe does not have to be in your shell.
A script shipped to a box that ends `… | grep -v Deprecation` exits with
grep's status, so set -e does not trip, the remote shell exits 0, SSM
records STATUS: Success, and the runner prints OK for every box. A
Postgres auth failure was reported to me as a green fleet.
⚠ The tell: you filtered the remote output to keep it readable. That
readability filter IS the thing that ate the error.

Two things in that block carry the whole design, and both are easy to miss.

It arrives before the command, attached to the tool call rather than to the answer. And it is specific to that command: the same session ran dozens of other commands that evening and this particular note stayed silent for every one of them, because none of them contained a pipe.

The cost is visible too. The block runs to roughly 1,100 characters. The channel is capped at 4,000 per action, shared across everything that matched, so a note this size is a quarter of the budget for that moment. When several match at once the packer takes whole notes until the budget is gone, skips the rest, and puts the skipped ones first on the next action.

What the schema leaves out, on purpose

No confidence score. The obvious field to add, and we would get it wrong. A note is written at the moment its lesson is learned, which is exactly when an honest author rates it highly, and confidence authored once does nothing about staleness accruing afterwards. We have a live instance. A note of ours asserting that one of our own inspection commands was broken had been true when written, went stale, and was believed months later by a reader who had no reason to doubt it. A confidence field would have been accurate and useless there. The obligation that would have helped belongs to notes asserting live state, and it is a freshness duty, which a score cannot carry.

No record of why this one appeared. The match list decides selection and is invisible at the moment of delivery, so a reader sees the note and cannot see which term pulled it in. That is a real gap, it was pointed out by a reader within two hours of this article going up, and it is cheap to close.

No provenance field. The "Where this came from" section is a convention the schema never enforces, so a note written in a hurry can skip it entirely. In practice ours are dense with dates and commit hashes, and that density grew after an unsourced claim burned us, having never been designed in at the start.

Where crystals come from

The obvious question is who writes the crystals, and the answer is that three moments generate almost all of it.

An error you just paid for. Something failed, you found the cause, you fixed it. The knowing exists for about ten minutes and then the session ends and it is gone.

A lookup. If the agent goes hunting through the filing system for a stable fact, the store should have handed that over and did not. The hunt is the signal. That one is mechanically detectable, which is why we count blocked lookups at all: every one is the store confessing it failed, and a confession is a good place to mint a crystal.

A lesson the work produced. A finding, a correction, a decision. No mechanical tell for these, which is the honest gap in the detection.

The package ships a hook for the end of a session. It checks two things: did real work happen, and was anything written down. Work with nothing banked gets one interruption, once, carrying a filled-in template.

It never writes the crystal, and it never decides what is worth keeping. That limit is deliberate and it is the part we would defend hardest. We have a scar here: an earlier pipeline of ours generated notes automatically from text it had been handed, and produced a confident, specific, entirely invented statistic about a stranger's article, which we then quoted back to them in public. A machine that mints from whatever it read will fill your store with plausible fiction on day one, and a wrong crystal delivered at the moment of an act is worse than an empty store. So the hook guarantees you are asked. The judgement stays yours.

Why not just let it search

An AI coding agent forgets everything when a session ends. Next time it starts clean. It will happily make the same mistake it made last Tuesday, because for it there was no last Tuesday.

The usual fix is a search box over your notes. The agent gets stuck, the agent searches, the agent finds the note. This works, and it has one hole in it that bothered us: the agent only searches for things it already suspects. It cannot look up the mistake it does not know it is about to make. The moment you most need the note is the moment you have no reason to go looking for one.

Binding the note to the action closes that hole, and it costs you the thing search is good at, which is answering a question you actually have. The two are complements, so we built both. The pull half, and an honest look at what its numbers are worth, is the next piece.

What else is in here, and why you are not running it yet

Crystals are level one. They pay on an empty store, because the first crystal you write is the first thing the store holds. Nothing has to accumulate first.

Level two arrives on its own, a few hundred notes in. Everything a crystal says has to live somewhere, and a few hundred of anything is a library with a library's problems. Two notes that contradict each other. A folder that quietly grew to ninety files. A link pointing at something since renamed. Two versions of one truth, which is worse than none, because now the agent has to pick.

So the system we run here has four more pieces. All four tend the store:

The Librarian keeps it navigable: broken paths, dangling links, signposts that rotted.
The Cleaner consolidates. When a folder goes over its cap it proposes what to archive, and every move is reversible.
The Gardener grows the store from what actually got used, mining our own usage ledger. It is barred from minting out of text it merely read, for the reason described further up.
The Corrector makes bounded edits to whatever the other three flag.

Now the part that belongs in a paper with this title.

On 2026-09-17 we pointed all four at a store from another machine, for the first time ever. Zero of the four were correct. Each broke in its own way, and the Cleaner is the one worth looking at. The list of folders it walks names three of our own directories, by hand, in the source. Aimed at a stranger's store it found nothing over cap and announced that everything was within cap: a green tick over an empty set, from a guard whose entire job is to notice. We learned this by running it. Reading the source would have left it hidden.

There was a fifth failure one layer up, and it is the one we would put on the cover. The health check for the whole package looked at that same foreign store and passed it, green, while every one of the four agents beneath it was unable to use the thing it had just approved.

So the download is crystals, and the other four stay here until they can survive a store they did not grow up in. Shipping them this week would put four agents on your machine that report success by looking at nothing.

There is a second reason, and it is the one that sets the order. All four idle against a small store. Five notes give the Cleaner nothing to consolidate and the Librarian nothing to repair, and the Gardener feeds on usage you have yet to generate. They earn their keep at a few hundred notes, which is the same point at which the problems they solve begin to exist. Level one and level two came out in the order the work arrives in.

What we can count

Over 60 days, from 2026-07-19 to 2026-09-17, on one machine doing real work:

what	count	window
crystals registered	266	as of 2026-09-17
crystals delivered at least once	243	60 days
deliveries	14,375	60 days, about 240 a day
budget per action, shared	4,000 characters	fixed
blocked lookups, the opposite signal	387	94 days, from 2026-06-15
suppressed so far, the new control arm	19	since 2026-09-17

That last-but-one row is the one we watch hardest. A separate guard blocks the agent when it tries to hunt through the filing system for something the system should have handed it. Every block is a small confession that delivery failed.

What does look like it is working

We lean hard on our own failures, so a fair question is what the data says in the other direction. Three things, and we went looking for the reasons each might be wrong.

Hunting fell by two thirds. Splitting the window in half: in the first half the agent went hunting through the filing system 22 times per thousand notes delivered, and in the second half 7.5. Counting every hunt, marked and blocked alike, it is 61.9 down to 20.5.

The obvious rival: we simply learned to mark our hunts with the escape hatch, which would make the drop an accounting change. It is checkable, and the check came back against it: the blocked share of all hunts held flat at 35% and then 37%. Volume fell while the habit held still.

What we cannot rule out is the work itself. The two halves contain different projects, including a fortnight spent away from this repository, and a person also gets more familiar with a codebase over ninety days. So read it as suggestive, short of causal. It remains the closest thing we have to the system getting better at its job.

Almost the whole store is live. 237 of 264 crystals had been delivered at least once when we counted, which is 90%. Knowledge bases usually rot toward the opposite, and 27 crystals that have never fired is a short enough list to read in one sitting.

The load is spread across the store. The busiest ten crystals out of 241 account for 17% of all deliveries, and the median crystal has fired 29 times. That kills the objection we expected to face, that a number like 14,000 really means one loud crystal firing over and over.

Now the honest part

None of those numbers say the thing you actually want to know.

Counting deliveries measures how often a crystal showed up. Whether the crystal helped is a separate question, and that count is silent on it. A system that fired 14,375 useless crystals would produce exactly the same chart.

The number we want is "how many lookups did this save," and here is the trouble with it: a lookup that did not happen leaves no trace. You cannot observe a search nobody ran. Any figure we computed from our own logs would really be measuring how rarely we search on the one channel we happen to watch, dressed up as a result. We got close enough to building that number to be embarrassed about it, and an outside review took it apart before we shipped it.

We do have two small effect measurements from 2026-07-11, and we are going to describe them exactly as weakly as they deserve. On a long task where the guiding instruction scrolls out of the model's context, re-supplying it as a crystal held the model on target where a control collapsed, and a placebo crystal carrying no instruction collapsed too, which at least says the content mattered and not the interruption. Two runs, eight chunks, synthetic constraints. On a second task, injecting a fixed pattern and generating only the new part cut output by about 95% at equal correctness. Eight cases, one small local model. Both are directional. Neither is evidence that this helps you.

The one test that actually asks whether it changed anything

Counting deliveries cannot answer it, so in August we ran the study that can. Five scenarios, each ending in a concrete command or choice. Three arms: the note delivered, the note withheld, and a placebo, which is a different note's text of similar length. Twenty runs per cell. The outcome is read by a fixed rule that looks at the command the model produced, with no model grading another model, because a judge here would be grading the thing whose grading is in question.

The placebo arm is the only reason the study is worth anything. Without it, "delivered beats withheld" cannot tell you the knowing worked apart from any extra text making the model more careful. Two of the five rows turn on exactly that.

scenario	delivered	withheld	placebo	verdict
git-add-sweeps	100%	0%	0%	earns it
screenshot-vs-text	100%	0%	0%	earns it
pkill-pattern	100%	0%	0%	earns it
piped-exit-code	100%	100%	35%	already known
block-not-poll	100%	0%	100%	not attributable

Three clean separations. On the first one, withheld, the model reached for the dangerous command in 8 of 20 runs; delivered, it produced the safe path-scoped form 20 out of 20.

The two rows that withhold a yes are the ones I would keep. Already known means the model was right without us, 20 out of 20 with the note withheld: a real note, minted from a real incident, paying rent on a room the model already owns. That verdict class is the one a fire-count can never produce. Not attributable means the placebo scored as well as the real note, so whatever helped came from somewhere other than this note's content. Without the placebo arm that row would have been counted as a win.

And the result nobody went looking for

An irrelevant note made a task the model already did perfectly worse: 100% down to 35%.

Injecting off-topic text went past failing to help. It degraded correct behaviour on work that was already fine. That is the strongest thing we know about this channel and it is an argument against our own product's easiest failure mode: shipping more notes because more feels safer.

It also turns a piece of hygiene into a measured trade. Keeping the channel clear stops being tidiness and becomes a number.

The limits, because they are large. Twenty runs per cell, one model, one phrasing per scenario. The outcome rules are mine. For procedural notes the delivered arm may partly be echoing the note's own worked example, which is the thing you want from a procedural note while falling short of evidence that a principle was understood. And all five scenarios are behavioural, so the notes carrying judgement, as opposed to procedure, are untouched by this entirely.

So it is directional, and the degradation number in particular should not be quoted as a magnitude. It is enough to have moved us, and not enough to move you.

What we are doing about it

Since 2026-09-17, our own system withholds a random ten percent of the crystals it would otherwise deliver. The agent never sees them. Every suppression is logged, so there is a control group.

Before any data existed, we wrote down the rules:

The unit is one crystal per session, not one delivery. A crystal that fires forty times in a day is one decision, not forty results. An hour after switching it on we had three suppressions covering one crystal, which is exactly the trap.
The outcome is named "a lookup we can actually see," because we only watch one channel. Silence on an unwatched channel stays silence, and we refuse to read it as "they never looked."
We stop at 100 units or on 2026-12-17, whichever comes first, and that is fixed now so it cannot be chosen later to suit the answer.
A null result gets published. If withholding the crystals changes nothing, they were decoration and we will say so.

That last point is the whole reason for writing this down in public before we have the answer. It is easy to run a quiet experiment and mention it only if it flatters you.

What this costs, and what we are still blind to

The first thing a reader usually asks is what this does to the context window, so here is the arithmetic. The channel is capped at 4,000 characters per action, shared across every crystal that matched. That is roughly a thousand tokens on a shell command, and zero on the many commands where nothing matches. Against a coding context that is small, and the bound comes from the code, never from our good behaviour.

We are worse at our own guidance than that makes us sound. We tell people to keep a crystal under about 1,500 characters. Our median sits at 1,496, comfortably inside. Then 131 of our 271 crystals run over it, which is 48%. The longest is 8,401 characters, more than double the entire per-action budget, so it arrives truncated and shoves its neighbours out on the way in. Truncated and short look identical from inside the model, which is the silence problem again one level down.

The honest cost is attention, more than tokens. A crystal that arrives at the moment the agent has already chosen a tool is authority-weighted: it competes with the instruction you gave. And a wrong one costs more than a wasted slot, which we know because we measured it above: an irrelevant note took a task the model already did perfectly from 100% to 35%.

Withholding crystals from your own working system makes your own agent worse on purpose, a little. We think that is a fair price for finding out.

We are still blind in ways worth naming. This is one operator on one repository, so read it as a case study, never a population. We watch shell commands and not file reads, so our view of "did they go looking" has a hole in it. And suppressing a note frees up room in the shared budget for its neighbours, so the two groups are not perfectly separate, which is a confound we can describe and cannot remove.

What it is actually for

The mistakes worth catching are the ones that look like success.

A failed build that reports a green exit code. A test suite that passes while asserting nothing. A guard that lost the ability to fail when somebody narrowed what it checks. A deploy that reports Success because the error was swallowed by a pipe. Every one of those ends with an agent telling you it is done, in exactly the tone it uses when it is done, and you find out later.

An agent is very good at the first ninety percent and has no memory of the last time the final ten bit you. Telling it once does nothing. It has to be told at the moment, every time, and only about the thing in front of it. That is the whole design goal: a channel narrow enough that you tolerate it firing on every command, carrying knowledge specific enough to be worth reading in the second before you act.

We think we are a reasonable team to build it because we are the ones getting caught. Every crystal in the starter set is a mistake we made and paid for. The system exists because we kept making the same three or four classes of error across sixty days, watched them cost real hours, and wanted something that would interrupt the fourth time rather than the fortieth. The discipline underneath it, in one line: we treat our own agent's unqualified yes as a claim awaiting evidence, and this is the machinery for that.

That is also why this piece leads with what we cannot prove. A tool that claims to make an agent more careful should survive someone asking it for evidence. We asked ourselves, and the answer was a measurement we had skipped. So we are running it now, in public, with the rule written down first.

If you want to try it

The delivery half is five files of standard-library Python, Apache 2.0, no network, no service. It runs on your machine and talks to nothing.

github.com/tjonesit/crystal-memory, public and marked in testing, because nobody outside our team has installed it yet.

An honest note, since this section said the opposite a few hours ago. It said the repository was deliberately private, that we would hand the first copies out one at a time, and that a download page would hide where people got stuck. We changed our minds the same evening. Pulling the package out of our own repo immediately broke it in two ways: it looked for its starter notes at a path that only exists inside the repo it came from, and its "clean room" test was quietly inheriting our shell environment, so it suppressed some of its own deliveries and reported a failure that looked like a packaging bug. Both are fixed. Both are exactly the evidence we said we wanted, and we got them in twenty minutes by taking it somewhere else, so we would rather have more of that sooner.

Every check we own was written by the same hands that wrote the thing being checked, so our green is worth less than a stranger's red.

The most useful thing you could send back is the moment you thought "I installed this and nothing happened," because a system whose failure mode is silence is a system that can quietly do nothing for a week while you assume it is working.

We would rather hear that from you in week one than discover it ourselves in month six.

Who your model works with matters more than which model you picked

Tom Jones — Tue, 18 Aug 2026 21:08:41 +0000

The short version, for anyone who does not benchmark models for a living

Every few weeks a new model tops a leaderboard and the advice is to switch to it.

We have been measuring our own stack for a while, and two things keep coming back that are worth separating carefully, because the second one is easy to oversell and we have oversold it twice.

The first is measured and we are confident in it. Models that a leaderboard ranks two points apart are not doing the same work. On the problems where they differ, the difference is stable: run them again and the same models are right and wrong on the same specific problems. That is a property of the models. Which one tops the column total is substantially a property of the run.

The second is what we would like to be true, and we have not earned it. It would follow that pairing two different models beats upgrading to one better one. We have never run that experiment. We built the obvious mechanism to capture the available headroom and captured none of it. So take the title as the hypothesis this piece is about, not as a result it delivers.

What we can hand you is the measurement underneath it, a ceiling we could not reach, and twenty-one days of a ledger recording every time our own second model told us something confidently false. Including once, today, about this article.

What "a better model" actually buys you

Start with a number that surprised us, on our own production path.

We ran the same fixed 800-item tool-calling benchmark through our live system five separate times. Nothing changed between passes. Same items, same harness, same endpoint.

The five totals were 752, 750, 750, 749, 749. A mean of 93.75%, and the whole spread is three items.

That looks like a beautifully stable system, and here is the part that matters: 38 distinct items, 4.8% of the set, flipped somewhere across those five passes. They cancel out. The worst pair of passes disagrees on 23 individual items while their totals differ by one.

Two numbers, and they say different things. The score is extremely stable: 0.375 points across five passes. The answers underneath it are not: on any given pair of passes, 2.00% to 2.88% of individual items come back differently.

We had a rule of thumb here that doubled the second number and used it as a threshold on the first. Our own reviewer killed it while reviewing this article, and it was right to: an item-flip rate and a score gap are different quantities, and the whole reason the score is stable is that the flips cancel. You cannot derive one from the other, and we have retracted it. The section further down shows that exchange in full, because it is a better illustration of the method than anything we could have written on purpose.

What the five passes do license is narrower and still worth having. If you are going to quote a single benchmark number, run it more than once, publish the spread, and publish how many individual items moved. A stable total is not a stable measurement, and only the second of those two numbers tells you whether the thing you are ranking sits still.

There is a second way the level story gets overstated, and it is worse. We took one model, kept everything identical, and changed only which company served it. On one model-and-provider pairing that moved the score by 17.3 points, from 78.2% to 95.5%. On a different pairing the same experiment moved 0.6 points, so this is a property of the specific pair rather than a law. But it means a cross-vendor ranking built on unpinned routing can be measuring the plumbing rather than the model, at a magnitude larger than the gaps it is reporting.

What a second model buys you instead

Now the other half, and this is the one we would defend hardest.

We put three models over the same 160 coding problems. Same prompts, same scorer, one run. They scored 148, 146 and 144 of 160, which is 92.5%, 91.25% and 90.0%. A 2.5 point spread, and by the logic above you should treat that ranking as noise.

Then we stopped reading the column totals and partitioned by problem instead.

Across two runs the three models disagree on roughly 15 to 18 percent of the problems, and 83% of the set is uncontested by all three. Most of the work is genuinely settled. On the remainder, models that a leaderboard calls equivalent are answering differently.

The obvious objection is that this is the same flicker wearing a costume, and it has to be answered on this set rather than by importing a number from a different benchmark. On these 160 problems the same model on the same problem across two identical runs flips 3.7%, 4.9% and 5.2% of the time. So the honest comparison is a contested rate of 15 to 18 percent against a self-flip rate of up to 5.2 percent, which is a factor of roughly one and a half to two and a quarter depending on which pair you take. Real, and considerably less dramatic than it would look if you reached for a flip rate from some other benchmark, which is a thing we have now done twice and been caught doing twice.

So the size of the disagreement is not the argument. The argument is that ten problems are contested in both runs with zero flips by any model. Each model is stably right or stably wrong on them, run after run.

We had a bigger-sounding version of this, resting on how improbable the overlap of contested sets was under a null where every problem is equally likely to be contested. Our reviewer killed that null, correctly: easy problems are essentially never contested, both runs draw from the same medium band, so the sets overlap with no family specialisation at all. It supplied a difficulty mixture reproducing most of the overlap. We checked the arithmetic and it holds, so that p-value is gone.

Then we ran the test it should have been, because the objection is answerable rather than merely conceded.

Hold each problem's difficulty fixed and destroy only the model-specific part. For every problem take its observed success rate across all three models and both runs. That number is its difficulty, measured, with no mixture assumed and no distribution fitted, which makes it immune to the objection above whatever the true spread turns out to be. Then resample every cell as an independent coin weighted to that rate, and count the stable-contested problems again. Twenty thousand draws.

| | |

|---|---|
| observed, contested in both runs with zero flips | 10 |
| same statistic under the difficulty-preserving null, mean | 2.0 |
| the most the null ever produced in 20,000 draws | 9 |
| p | 0.00005 |

Difficulty makes all three models wobble on a problem. It does not make one model reliably right and another reliably wrong on that same problem, twice. The claim survives a null built specifically to satisfy the objection to the previous one.

Here is the whole residue, named, on 154 problems both runs scored:

problem	gpt-oss-120b	qwen3-235b	qwen3.8-max
HumanEval/9	pass	fail	fail
HumanEval/10	pass	fail	pass
HumanEval/22	pass	pass	fail
HumanEval/26	pass	pass	fail
HumanEval/86	fail	pass	pass
HumanEval/115	pass	fail	pass
HumanEval/125	pass	fail	pass
HumanEval/132	fail	pass	fail
HumanEval/145	fail	fail	pass
HumanEval/160	pass	fail	pass

Read down the columns. Every model is stably right somewhere another is stably wrong, and every model is the sole failure somewhere. gpt-oss-120b owns five of these and is the lone loser on two. qwen3.8-max is the only one that solves 145. None of the three is redundant, and none of them dominates.

That is ten problems out of 154, which is 6.5%. It is a much smaller number than the disagreement rate, and it is the one we can defend. The self-flip rates on this same set are 3.9%, 4.5% and 5.2%, measured here rather than imported.

The part we have not earned, and will not claim

If three models disagree on that many problems, then something that could always pick the right one would score above any of them individually. On the 154 problems both runs scored, the best single model gets 145 and any-of-three gets 152, so that ceiling is 4.5 points.

We are not claiming those six points. Reaching them requires an oracle that knows which model was right, and an oracle is precisely the thing you do not have at the moment you need it.

We know this the expensive way. We built the obvious mechanism to capture that headroom, ran it end to end, and captured none of it. The cheap arm passed its own checks almost every time, so the second opinion was rarely consulted, and the result landed exactly where that first arm already scored. The headroom was real and our design could not reach it by construction rather than by tuning.

So the honest claim is narrower than the exciting one, and it is still the useful one: the gap between models is a real, reproducible, addressable thing, and the value shows up when a second model actually looks at the first one's work. Which brings us to the part we can put numbers on from our own three weeks.

So what do we actually run, and why in those seats

Three models, fixed roles, and the roles were chosen by failure mode rather than by tier.

Claude orchestrates. Specs, review, decisions, and the commits. It holds a very large context, which means it can carry the codebase, the written constitution and the current spec at the same time and notice when two of them disagree. That is the job.

Where it is weak, measured on itself: it agrees with itself. Given a shortcut it will produce a reason for the shortcut, and the reason will be fluent. Our own standing rule says substantive work runs as three, and the rule exists because the orchestrator twice decided a piece of work was "small and mechanical" and did it alone. Both times the cost showed up within the hour, in public once. An orchestrator that also implements ends up grading its own homework, which is why it does not implement.

Codex implements. It works from a written spec inside a sandbox, and it is good in direct proportion to how exact the spec is. Vague spec, confident wrong build.

The interesting part is a limitation we stopped trying to remove: it cannot commit. Its sandbox has no path to our history. That forces a review step between "the code exists" and "the code is in the repo", performed by something that did not write it. We could grant it commit rights. We have chosen not to, because the gap is where the checking happens.

Grok reviews, adversarially, and verifies across multiple hops. Different family from both of the others, which by the argument above is the entire point. It is strongest before something is built, when the thing under review is a design and there is still time for the answer to be "do not build this". Three times in three weeks it stopped something broken from shipping, and one of those was a token minter where we had handed the verifier its own minting key.

Where it is weak is documented below in detail: it is confidently wrong at a rate we can measure, and it attacks whichever leg you failed to brief it on, with exactly the same confidence it brings to the legs it has evidence for.

And the fourth seat is a person. Every pivot that mattered over three weeks came from the founder rather than from any model: ask another mind, is that apples to apples, what does the source actually say. The models are good at execution inside a frame. Choosing the frame, and noticing when the whole frame is wrong, has not once come from them. Any description of this setup that leaves the human out is describing something that would have drifted weeks ago.

The brief does more work than the model choice

This is the part we would keep if we had to throw everything else away.

Ask any model to "review this" and you get a compliment. It will find something small, agree with your framing, and hand the whole thing back improved by three percent. The model is not the problem. The request is.

So our brief names what we want destroyed, in those words. Here is the shape of a real one, from a review that came back DO NOT BUILD:

You are reviewing a design. Your job is to attack it, not to compliment it.

The five things I most want destroyed, in these words:

The central claim may be oversold. [the claim, and the measurement behind it]

[The load-bearing beam.] This is the strongest argument; if it breaks, tell us.

The evidence may prove a different thesis than the one claimed.

[The weakest data.] Make the strongest possible case that this is worthless.

[The newest, least-reviewed part.]

Also answer: what is missing, what should be cut, and where are you systematically weak and
should not be trusted.

Rules: re-derive any step you challenge rather than asserting it. Separate findings from
insinuations. If you cannot check something from what is here, say "I cannot check this from
the brief" rather than guessing. Name which of your objections you are least sure of.

Every line there is load-bearing. Naming the load-bearing beam invites the review to aim at the thing you would least like to lose. Asking where the reviewer is weak gets you a calibration you cannot get any other way. And "name which objection you are least sure of" is the one that turns an undifferentiated wall of confident findings into something you can triage.

That last instruction exists because of the failure described in the next section: a reviewer attacks the leg you failed to brief it on, with exactly the confidence it brings to the legs it has evidence for.

A small, dull example, because the dramatic ones prove less

We would rather show you the cheapest possible case than a war story, because the cheap case is the one you will actually recognise.

Today the task was posting four comments. No design, no architecture, no judgment. The kind of chore where running a review feels absurd.

We skipped the check and did it alone. What that produced, measured from the logs:


replies intended	4
posting attempts made	17, over 11 minutes
landed correctly	1
landed and should not exist	1
never posted	2
permanent public artifact	1 comment now reading "posted by mistake"

The queue was shared and held old rows that had already been posted days earlier. Draining it fired those too, and one went out as a word-for-word duplicate of something we had already said in that thread. Fixing it took a browser session, an edit to a live comment, and an API check to confirm the edit landed.

The check that would have prevented all of it was reading the queue before draining it. One command. Under a second.

Two honest caveats, and they matter more than the anecdote.

That is not a typical solo result. Most shortcuts cost nothing at all, which is exactly why they keep getting taken. Anyone claiming their process catches an error every time is selling something.

And the comparison is not "team good, solo bad" on average. It is about the shape of the distribution. The check is cheap, bounded, and always the same price. The failure is rare, unbounded, and in this case irreversible and public. You are not buying a better average. You are buying a shorter tail, and the reason it is worth doing on a chore is that chores are exactly where the check feels too expensive to bother with.

The guard now exists, so this specific hole is closed. That is the other half of the discipline: every manual finding leaves something automatic behind, or you will find it again.

And one seat is not filled by a model either. Every finding gets re-derived against the source by whoever is about to act on it. That step is the product. Everything above it is logistics.

We kept a ledger of every time the reviewer was wrong

Cross-model AI review works, and that part is settled by people with bigger samples than ours.

Greptile ran 1,000 pull requests in July 2026 and found that Claude Opus catches 53.7% of bugs in its own code and 62.0% in GPT-authored code, with GPT-5.5 showing the same asymmetry in reverse. Milvus put five different-family models through five rounds of adversarial debate on 15 real production bugs and moved detection from 53% to 80%. Cloudflare runs a role-tiered, cross-vendor reviewer in production. If you want evidence that a different-family reviewer catches what the author model misses, go read those.

We have something smaller and, we think, differently useful. For the last three weeks we have kept a ledger. Every time our reviewer returned a finding, we re-derived it against the source before acting, graded it, and wrote down the grade. Including the times the reviewer was wrong.

That last column turns out to be the interesting one.

The numbers

Sixteen adversarial reviews between 2026-07-27 and 2026-08-16. Fourteen ran before the thing shipped, two after.

	count	rate
reviews on file	16	21 days
ran pre-ship	14	87.5%
changed the artifact	14	87.5%
contained a finding that was wrong on re-derivation	13	81.3%
would have shipped something broken without the review	3

⚠ These are the figures as published on 2026-08-18. The ledger did not stop. Instance 17 landed 2026-09-23 and is written up at the end of this piece, so the table above keeps matching what originally went out rather than being edited underneath it.

Read those two bold rows together, because separately each one tells a lie.

The first says the review is worth running. Fourteen times out of sixteen, the artifact that shipped differed from the artifact we brought to review.

The second says the review cannot be obeyed. Thirteen times out of sixteen, at least one finding fell apart the moment we went back to the source. Not stylistic disagreements. Confident, specific, technical claims that were false.

What the wrong ones looked like

A P0 that died to one curl. A security review ranked identity forgeability as an unverified critical issue. We sent an anonymous request and a forged-JWT request at the live endpoint. Both returned the sign-in page, zero data. The finding was ranked highest and was empty.

A missing measurement we had already taken. A pre-publication review objected that we had never measured end-to-end task latency. We had: p50 0.85s, p90 1.36s, same 800-task run, already in the ledger. Deferring to that finding would have made us withdraw a defensible public claim. The reason it happened is worth naming, because it was our fault: we had not briefed that leg. A reviewer briefed on a subset will attack the gap you made and will sound exactly as confident there as everywhere else.

A mechanism that was never there, hiding a worse bug that was. A migration review's central high-severity finding described a specific line of code doing a specific thing. That line does something else entirely. But going to the source to check produced a different and more serious defect: two functions resolve a caller's tenant by reading different database columns, so a partial migration would put one principal in two tenants at once. Nobody had that going in. The wrong finding was worth more than a right one would have been, because chasing it down led somewhere real.

What the right ones looked like

Three times the review stopped something broken from shipping.

A pre-build review caught that consolidating keyboard handling by focus would steal the Enter key from the terminal, so the shell would get no newline or a doubled one. The correct shape came back in the reviewer's own words and went into the spec verbatim.

A migration review said do not flip one tenant flag: all 8,116 cache rows were world-scoped, so flipping it would stop them matching. The important part is the failure mode. That regression surfaces as a bigger bill, not as an error, so monitoring that watches error rates stays green throughout.

A pre-ship review of a token minter found we had handed the verifier the minting key. The binding we were proud of was policy on one writer, while the cryptography permitted many. The patch was held. The design went asymmetric.

One worked example, start to finish

Here is a full instance, so the protocol is visible instead of merely described.

The problem. Our own standing rule says substantive work runs as three. One Sunday the orchestrator went solo on two implementation items anyway and told itself they were "small and mechanical", which is the exact rationalisation the rule exists to block. The founder asked: "How do we make the team use happen all the time? Is it something I have to continue to remind you of?"

A standing rule a human keeps re-supplying was never installed. It is only being remembered, and remembering is the thing that resets. So: build a gate.

Step one, refuse to guess. A hand-written test restates your own mental model as data and can only confirm it. So we extracted every file-editing action from 163 sessions of transcript history, 210,438 lines, giving a corpus of 5,037 real edits, and tested candidate definitions of the lane boundary against what the agent had actually done.

candidate definition of "implementer lane"	share of 5,037 edits blocked
anything not clearly orchestrator work	51.9%
restricted to implementer roots and code file types	51.0%
the same, minus measurement and benchmark scripts	42.2%
the same, plus "no other agent was dispatched this session"	27.0%
the same, counted once per file instead of once per edit	6.2%

The mental model was wrong. One scripts directory held 34% of all edits across 369 distinct files, mixing production serving code, guards, benchmarks and ops automation with no prefix separating them.

The obvious rival is that product code drives that rate and the scripts folder rides along. We tested it by dropping that directory from the definition: 51.0% falls to 16.9%, so it supplies 67% of the block volume by itself. One minute of work, and now the claim has a test under it.

The last row felt like the answer. 178 edits to one file is one decision to go solo, not 178. Change the accounting unit and an unshippable rate becomes about two prompts per session.

Step two, the review. The design went out with the measurements attached and an instruction to attack it. Asking a model to "review this" gets you a compliment, so the brief listed the five things we most wanted destroyed, in those words.

It came back DO NOT BUILD, with two findings we graded correct.

The dispatch condition was a laundering mechanism: survey, then decide, then implement is the normal shape of a working session, so after any unrelated dispatch the gate goes dark for the day. We measured it before agreeing. Removing that condition moved the rate from 1.91 to 6.03 fires per session. It had been hiding 357 of 669 events, which is 53% of the entire livability argument. The number proving the design was safe rested on the component we trusted least. Circular, and we missed it.

The second finding was about us. We picked the counting unit after looking at the rate table. The principle behind it holds, a decision does happen once, but we arrived at it while hunting for a number that worked.

Step three, the reviewer was wrong too. Its prescription was to stop inferring the boundary and block a fixed list of unambiguous implementer directories instead. Decidable, mechanical, no judgment.

We ran that against the same corpus. It fires 4.78 times per session, and it misses one of the two real incidents that caused the task to exist.

That is the finding of the day, and neither side had it going in:

The failures live precisely in the zone that no decidable rule reaches. A boundary crisp enough to be
mechanical cannot cover the ambiguous middle, and the ambiguous middle is where things actually go wrong.

Two designs, both measured, both dead. We built an observer instead: it blocks nothing, logs every event it would have flagged, and pre-registers all the candidate counting units so that post-hoc selection stops being available to us.

We also owe the reviewer a note on what we left out of the brief. We never told it our hooks support a shadow mode, which materially weakens its strongest objection, because livability then stops being a matter of argument and becomes something you measure in production for free.

The rule underneath all of it

One line, from the founder, in July:

"Don't trust someone else's work, because ours still has issues, why should we think theirs doesn't."

That became a constitutional rule here: an outside review is evidence, not a verdict. You re-derive the step the conclusion turns on. You separate the finding from the insinuation. You name the legs you did not brief, before you decide whether an objection is real or an artifact of your own briefing.

The asymmetry it exists to correct is the easy one to fall into: hostile scrutiny for our own numbers, courteous acceptance for someone else's. We had three instances of that in a single hour once, which is how the rule got written.

And it runs both ways. A review that agrees with you earns the same scrutiny as one that argues. External agreement feels like independent verification, which is exactly how it slips past the check it resembles.

We reviewed the review

Everything above describes a protocol. Here it is running, on this article, with the grades.

We sent this draft to the reviewer with a brief telling it to destroy five named things. It came back with a verdict: do not publish this thesis with this evidence. Then it listed findings.

The temptation at that moment is the whole subject of this piece. A confident, specific, well-argued rejection from a different model feels like a verdict, and the pull is either to obey it or to dismiss it. Both are the same mistake, which is treating the review as an answer rather than as input.

So we graded it. Every finding, re-derived against the source before deciding.

The findings we accepted

It found a defect that reached past the draft into our own source of truth. The article claimed a benchmark gap under about 5.8 points "is not a finding." The reviewer called that a category error: the number is twice an item-level flip rate, being used as a threshold on a score gap, and those are different quantities.

We opened the script the rule comes from. Its two-times rule governs the ratio of between-model disagreement to the within-model noise floor, and it prints a refusal when that ratio fails to clear two. It has nothing to say about leaderboard gaps. And our own paragraph refutes the claim without any help: across five passes the score moved 0.375 points while 4.8% of items flipped, because flips cancel.

Correct, and worse than the reviewer knew. That sentence did not originate in the draft. It came from a row in our verified-claims ledger, a row that had been cleared for public citation. The reviewer was attacking a line of prose. The defect was upstream, in the thing the prose was quoting, and it needed a correction at the claim.

It caught us committing an error we had already retired. The opening contrasted between-model disagreement of roughly 15 to 18 percent against within-model flicker of 2.00 to 2.88 percent. Those come from different benchmarks with different task distributions and different arity: one is a three-model contested rate on 160 coding problems, the other a pairwise flip rate on 800 tool-calling items.

That is the same error class we struck from the ledger in August, when an earlier version of this comparison divided a three-way rate by a pairwise one. We own the correct figure. On the same 160-item set the within-model flip rate is 3.7, 4.9 and 5.2 percent, which puts the like-for-like comparison at 1.50 to 2.25 times rather than something dramatic, and our own adversarial pass had already noted that this straddles the refusal line.

The reviewer re-derived that with no access to our ledger, our scripts or our artifacts. It had the article and nothing else.

It broke the load-bearing beam. Our strongest argument was that the contested problems reproduce across runs, 17 overlapping against 3.6 expected by chance, p = 1.5e-12. The objection is that the null assumes every problem is equally likely to be contested, which is false on arrival, because easy problems are never contested and both runs draw from the same medium band.

It supplied an existence proof. Thirty problems contested at probability 0.7 plus 130 at 0.023 yields the same overall rate and an expected overlap of 14.8 against our observed 17. We checked the arithmetic. It holds.

What survives is the smaller claim we already had: ten problems are contested in both runs with zero flips by any model. Difficulty alone cannot produce stable opposite answers. Only an interaction between a specific model and a specific problem can. That is the real residue, it is 6.25% rather than 15 to 18, and it should have been the beam all along.

The findings we rejected, and why

"91.2% is impossible on 160 binary items." The arithmetic is right, 91.2% of 160 is not an integer, and the conclusion drawn from it, that either the scoring is not binary or n is not 160, does not follow. It is 146 of 160, which is 91.25%, printed to one decimal. The repair is to publish counts alongside percentages, which we should have done anyway. The reviewer reasoned correctly from the page and reached past what the page could support.

"You never report the within-model flip rate on that set. You have the data." True as an observation, and the word "hiding" that followed it is an insinuation rather than a finding. That number is published in our own ledger, and it was withdrawn from public use by our own adversarial pass months before this draft existed. Leaving it out of the article is an editing failure. It is not concealment, and the difference matters, because one of those is a bug and the other is an accusation.

And a large share of what it could not check is our fault, not its. We sent the article and nothing else. No ledger rows, no scripts, no artifacts, no methods appendix. Its repeated "I cannot check this from the brief" is an accurate report about a brief we wrote badly, and its demand for a methods section is partly an artifact of that gap. A reviewer attacks the hole you left. Ours left several.

What grading actually costs, and what it buys

Settling two of those findings meant opening a script and a ledger row. That is perhaps twenty minutes. It is also the entire value of the exercise, because the accepted and rejected findings arrived in the same voice, at the same confidence, in the same list.

That is the point we would leave you with, and it is the one thing here that the larger studies do not measure. The published work on cross-model review reports how many real defects a second model catches. It does not report what the second model said that was wrong, because that column is only visible if someone re-derives every finding before acting on it.

We keep that column. Today it read five accepted, three rejected, and one of the five reached further than either of us expected.

And the recursive part is not a joke at our expense. The error the reviewer caught in the opening section was written today, by the orchestrator, inside the section arguing that a second model is worth the trouble.

What does not need three, which is most things

If the previous sections read as an argument for running everything past a committee, we have written them badly.

Most work does not need this. Renaming a variable, adding a log line, writing a test for behaviour you just specified, fixing a typo in a doc, any change whose failure mode is that it does not compile. A second model on those buys nothing and costs tokens, latency, and a small amount of your patience, and the practice will be abandoned within a week if you apply it there.

The filter we use is not size, and it is not risk in the abstract. It is this:

Would being wrong here be visible?

Work where the failure announces itself, a crash, a red test, a page that will not load, does not need a second opinion. You already have one, and it is faster and cheaper than any model.

Work where being wrong looks exactly like being right is where the whole practice earns out. Concretely, in three weeks, that meant: anything whose failure surfaces as a bigger bill rather than an error. Anything where a check might be passing because it cannot fail. Any security or identity boundary, where the successful case and the compromised case return the same status code. Anything irreversible or public. And any claim that is about to be published with a number attached.

That list is short on purpose. Rigour costs real money and real time, and the lever is never to be less rigorous. It is to be rigorous about fewer things, chosen deliberately, and then to go all the way on those.

What we are not claiming

We make no claim that three models beat one in general. We have never run that experiment.

We claim no novelty for cross-model review. Greptile and Milvus measured it first and at larger scale, and if the aggregate case is what you need, theirs is better evidence than ours.

Two honest limits on our own numbers. The corpus is self-selected: these are the reviews we chose to write up as nodes, so reviews that changed nothing may simply never have been recorded. And "artifact" here spans specs, designs, public claims and marketing copy, so the strict code-or-design cut is 9 of 10, against 14 of 16 overall.

And the premise underneath the worked example belongs to someone else too. "A blocking gate teaches where an advisory rule fails" was published by TRACE in June 2026, measuring advisory rules in context at 55.0% against hook-enforced blocking at 70.1%. Ours would have been a replication at best. What we actually found is that for a boundary requiring judgment we could not build the instrument at all, which is a smaller and stranger result than the one we set out to get.

The cost is real. Reviews take tokens, briefs take care, and re-deriving a finding means doing the measurement twice. Cutting rigour is the wrong lever. The right one is choosing fewer things to be rigorous about, deliberately, then going all the way on those.

The two lines worth keeping

On building guards: before you narrow a noisy guard's matcher, ask whether its unit is wrong. Everyone tunes the pattern. Almost nobody asks what one event is supposed to represent.

On running a team of models: the value is in the re-derivation, not the verdict. A reviewer you obey is an oracle you have not tested, and ours was wrong in 13 of 16 reviews while still being worth running every single time.

A follow-up, added after publication

The sixteen-review table above is left exactly as it went out.

This piece argued the case for working with a second model. The field notes are now their own article, and they are the more useful half: what Grok catches, what Codex catches, and what the pair costs me.

It covers what each seat actually finds, which question to send where, one review worked through start to finish, and the bill for a normal day. The bill turned out to be ten cents.

The agent that remembers, and never decides

Tom Jones — Wed, 12 Aug 2026 19:35:20 +0000

This is not a shortage of information. Every answer a department needs has usually been worked out already, by someone, in a meeting that happened. The problem is where it went.

It went into chat threads, policy articles, document folders, meeting notes, email, calibration discussions, and individual memory. Seven places, none of which is searchable together, and the last of which walks out of the building at some point.

What that costs is specific and recognizable. The same questions get answered repeatedly, slightly differently each time. Prior decisions are hard to find, so they get re-litigated instead of applied. Conflicting guidance exists and stays buried until it causes a problem. Calibration outcomes are lost once the meeting ends. New team members depend on finding whoever happens to remember.

The last one is the tell. When onboarding runs on a person instead of a record, the organization lacks knowledge. It has employees who hold knowledge, which is a different and much more fragile thing.

Start with what it never does

Most descriptions of a knowledge agent start with capabilities. This one starts with the boundary, because the boundary is the design and everything else is downstream of it.

The agent never makes the determination itself. It never creates policy. It never resolves a conflict between two sources. It never overrides existing guidance. It never treats a discussion thread as official guidance.

What it does instead is retrieve, cite, compare and escalate. When two approved sources disagree, it does not pick a winner. It reports that they disagree and routes the question to the people whose job that is.

Humans decide. The agent remembers. That sentence is the whole governance model, and it is what makes the rest safe to build. An agent that answers authoritatively becomes a policy source nobody approved. An agent that only ever hands you the record, with its provenance attached, cannot quietly become one.

The agent lives where the work already happens

It sits in the channels the department already uses, so nothing has to be adopted. No new destination, no separate portal, no habit to build. Someone asks a policy or calibration question the way they already ask it, and the agent answers in the thread.

It searches approved policy articles, prior calibration decisions, escalation outcomes, channel history, meeting summaries, and a lightweight knowledge store.

It returns the source articles linked, prior decisions with dates, related discussions, known conflicts named as conflicts, and an escalation recommendation when the record leaves the question open.

Returning a known conflict is the part that is easy to undervalue. A system that always produces a confident answer will produce one when the underlying guidance contradicts itself, and the person asking will never learn that. Surfacing the contradiction is more useful than resolving it, and it is the honest output.

The build is small on purpose. It runs on what the department already has: the chat platform, its agent builder, the existing policy hubs, the calibration memory files, the escalation log, and a small local database holding policies, decisions, escalations, questions and conflicts. Nothing here is exotic, and that is the point. A knowledge system that requires a procurement cycle will not be tested this quarter.

The loop that turns an answer into an institution

Eight steps, and most of them are human.

A question is asked in the channel.
The agent searches articles and history.
An answer is found, a conflict is found, or no guidance exists.
An unresolved item is escalated.
The calibration team reviews.
Leadership approves where required.
The approved decision is added to the knowledge base.
Future askers receive the approved answer.

   question ---asks---> THE AGENT ---reads only---> +--------------------+
      ^                     |                       |     APPROVED       |
      |                     |                       |  KNOWLEDGE STORE   |
      +--returns sources----+                       +--------------------+
        and named conflicts |                          ^              |
                            |                          |              |
              escalates when unresolved                |              |
                            v                          |              |
                    CALIBRATION TEAM                   |              |
                            |                    THE ONLY WRITE       |
                            v                          |              |
                       LEADERSHIP ---approves----------+              |
                                                                      |
       future askers receive the approved answer <-------------------+

The agent reads and never writes. Every path that changes what the organization believes runs through a person, and exactly one arrow writes into the store. That is what stops the agent quietly becoming a policy source nobody approved, and it is why the loop compounds: step seven happens once, step eight is free forever.

Steps four through seven are human. Correctness enters at step seven and only at step seven, from the calibration team, from leadership, and from policy owners where applicable. Only approved decisions are written back. A discussion in a channel, however senior the person in it, is not a decision and does not enter the store.

This is institutional learning, and the distinction matters. The model learns nothing about the domain. The organization accumulates decisions it already made, in a place where the next person will find them. Step eight is where the compounding happens, and step eight costs nothing, forever, once step seven has happened once.

Where the memory lives

Three layers, as plain files in the document stores the teams already use, surfaced through chat. The structure is deliberately boring. The permission model is where the care is required, because whether people write honestly into a personal layer depends entirely on knowing who can read it.

A personal layer, one per person, readable only by its owner. Root context describing role, tools and working style. Live tasks, blockers and next actions. Decisions made, each with its reasoning. Recurring issues with their causes and resolutions. The standards that person works to, versioned. Plus one folder that is deliberately shared upward, holding flags the team needs to know and open questions being surfaced.

A team layer, team readable and lead writable. Team identity and active work. Team decisions with history. Patterns aggregated across the team. Active standards, version stamped. Anonymous signals arriving from the personal layers.

A department layer above both teams, readable by both leads. Cross team context. Patterns surfacing across both teams. Decisions affecting both. An anonymous aggregate from both team layers.

                     +-----------------------------------+
                     |         DEPARTMENT LAYER          |   both leads
                     +-----------------------------------+
                            ^                     ^
                    a pattern seen in MORE THAN ONE team
                            |                     |
        +---------------------+           +---------------------+
        |  TEAM LAYER   eng   |           | TEAM LAYER  product |   team read
        +---------------------+           +---------------------+   lead write
           ^       ^       ^                 ^       ^       ^
                a pattern, person stripped out
           |       |       |                 |       |       |
        +-----+ +-----+ +-----+           +-----+ +-----+ +-----+
        | own | | own | | own |           | own | | own | | own |   owner only
        +-----+ +-----+ +-----+           +-----+ +-----+ +-----+

     named entries, written freely because only the owner can read them

Signal rises and loses identity at every boundary. A person writes named entries into a layer only they can read. What reaches their team is a pattern with the person removed. What reaches the department is a pattern that appeared in more than one team.

Signal moves upward and loses identity as it goes. An individual writes freely because the layer is theirs. What reaches the team layer is a pattern with the person stripped out. What reaches the department layer is a pattern that appeared in more than one team. Each boundary is a permission boundary, so the privacy property holds structurally and nobody has to keep remembering a rule.

Two phases, and why the order carries the weight

Phase 1 runs to completion before Phase 2 begins. That sequencing is the most consequential decision in the test, and it is easy to mistake for scheduling.

Phase 1, weeks one and two, is engineering only. Personal layers for each engineer. The team layer capturing architectural decisions and patterns. Session context assembled automatically at the start of each session. Decisions logged with their reasoning attached, the why alongside the what. Bug patterns and root causes captured in structured form. Standards made explicit, versioned and retrievable.

Phase 2, weeks three and four, adds product while engineering continues unchanged. Product onboards using the same architecture. The shared department layer activates. Engineering decisions become visible to product context and the reverse. Cross team patterns surface that neither team sees alone.

When product onboards in week three, engineering does not learn alongside them. Engineering has been living in the system for two weeks and helps with the setup, so the second team takes days, against the two weeks the first team needed. Run the phases in parallel and that advantage disappears: two teams learning at once, and nobody in the room who has done it before.

Engineering and product are chosen because they hold the most consequential knowledge gap in most technology organizations. Engineers know how a thing was built and which alternatives were rejected. Product knows what was decided, why it was prioritized, and what the customer signal was. Those two records are almost entirely separate today, and the value of connecting them needs no domain expertise to evaluate.

What would count as working

Stated in advance and deliberately narrow. Every criterion is a thing that either exists or fails to. Nothing here is a satisfaction score, for reasons given in the next section.

Prior decisions are findable in under thirty seconds, timed, by someone other than whoever filed them. A repeated question gets the same answer twice, asked again in week four from a different account. Conflicts are escalated rather than buried, with at least one case running all the way to an approved decision. Responses carry their sources, and a spot check confirms the link supports the claim. An approved decision from week one is still retrievable in week four with its date and reasoning. A cross team pattern surfaces that neither team saw alone. People are still asking questions in week four without being reminded to.

That sixth one is the core claim of Phase 2 and the only signal that cannot be produced by either team in isolation. If nothing appears in two weeks, the department layer has not earned its place.

Three predictions, recorded before the test starts

A protocol published after the fact is a story. The author already knows how it turned out and the criteria drift, quietly, toward whatever happened. Publishing first is the only cheap way to stop that.

I am strict about this because I recently paid for it. An experiment of mine returned a clean, total result: one hundred percent in the treated arm, zero in both controls, across all four cases. It was worthless. Every task in it could be answered by copying a sentence out of the text being injected, so the harness was measuring reading comprehension. The only reason I caught it was a prediction written to disk beforehand saying the result should be messy and mixed. The clean number contradicted the prediction, and that contradiction was the entire signal. Two later versions of the same instrument were also wrong, each in a way that flattered whatever I was hoping for.

So here are three predictions, on the record, before any data.

Everything below rests on three measurements taken on 2026-08-11, on a system I run, against codebases I did not write. They are small, and they are stated with their size so they can be argued with.

What was measured	Setup	Result
Where durable lessons come from	4 conditions, 2 foreign codebases	Ordinary use 1, isolated test 0, agreeing with 20 recorded fixtures 3
Whether a healthy system generates material	1 full suite, 476 tests passing	0 durable lessons
Whether irrelevant context is harmless	12 trials per arm, 1 model, 1 scenario	Irrelevant note wrong 12 of 12, control merely vague

One. The layers will fill unevenly, and the pattern will not be effort.

People working where their output has to agree with someone else's, shared interfaces, contested policy, anything with a boundary in it, will produce full layers. People working on self contained work will produce nearly empty ones while working just as hard. Read that as engagement and half the team gets judged for the shape of their work instead of its quality.

The basis, measured 2026-08-11 across four conditions on two codebases I did not write: ordinary use produced one durable lesson, adding an isolated test produced none, and making one component agree with twenty recorded fixtures produced three in about thirty minutes. The variable was integration surface, not hours worked.

Two. Week one will feel like nothing is happening.

A new layer is empty, and a process running normally generates very little worth recording. The material appears when something is contested, revisited, or wrong. A team expecting usefulness by day three will conclude the test failed before the compounding has begun.

The basis, measured the same day: a full test suite on a healthy codebase, 476 tests passing in about a minute, produced zero durable lessons worth recording. Nothing was broken. There was simply nothing to learn, because nothing resisted.

Three. Asking whether it felt useful will return yes, and will mean nothing.

Delivered context makes people feel better informed whether or not the context was any good. This is why every criterion above either exists or fails to, and why none of them is a rating.

The basis, measured the same day at twelve trials per arm, one model, one scenario: against a control given nothing, injecting a genuinely irrelevant note produced a confidently wrong answer in twelve of twelve cases, while the control was merely vague. The bad context went past unhelpful into actively misleading, and it read as authoritative while doing it. Treat that as a lead, with the sample size and the single scenario counting against it.

That third one is also the strongest argument for the boundary at the top of this piece. An agent that hands over sourced records lets a person check. An agent that hands over confident answers removes that option.

What I do not know

There is one instance of this architecture running in production over a long period, and it is mine. One organization, one operator, and every figure quoted above comes from it. This document generalizes to any two teams in any organization, and that generalization is a hypothesis rather than a finding. This test is the first attempt to check it against somebody else's work.

Three further things it cannot settle, listed because a careful reader will find them anyway.

Whether people write honestly into a layer their employer hosts. The permission model is designed for this and design is not proof. A personal layer that quietly becomes a performance record stops receiving true entries immediately, and that failure is silent.

Whether a cross team pattern is a pattern or a coincidence. Two teams generate enough signal that something will always look like a connection. Requiring that neither team saw it independently is a guard, and a weak one.

Whether four weeks is long enough for anything to compound. It is long enough for the layers to fill and for first patterns to appear. It is almost certainly not long enough to observe the property the whole design aims at, which is a team keeping its knowledge across a departure.

The test does not end at four weeks. What ends at four weeks is the part anyone is willing to make predictions about.

If you have run something like this inside a real organization, I am most interested in the first unknown above. The permission boundary is the part I am least able to prove from my own instance, because in my instance the owner and the operator are the same person.

Microsoft measured our thesis, and we still cannot quote ours

Tom Jones — Sun, 09 Aug 2026 14:45:33 +0000

Microsoft's .NET team published the benchmark behind their polyglot unit testing agent. We have been arguing its conclusion for months. We can't cite our own version of it, because we withdrew our number on 2026-08-06 and haven't earned it back.

Both halves of that are worth writing down.

What they measured

The setup is clean. One tool, one model, 152 tasks from real repositories. The only thing that varies is a plugin that makes the agent research the repository, plan, implement, then verify before it claims to be done. A task passed only if the repository built, all tests passed, the agent added at least one test, and it removed none.

Arm	Completed	Rate
With the workflow	140 / 152	92.1%
Same model, stock	120 / 152	78.9%

63% fewer failures, from the same model and the same tool.

The breakdown is where it gets useful.

Prompt type	With workflow	Stock
Vague, 89 tasks	88.8%	66.3%
Detailed, 63 tasks	96.8%	96.8%
Diff-targeted, 15 tasks	15 / 15	0 / 15

On detailed prompts the two arms tie exactly. The whole gap comes from vague prompts, where the agent has to work out for itself what to test, which framework is in use, and where the tests belong. So the scaffold isn't making the model smarter. It's taking away the guessing.

Then the per-model results, on 45 .NET tasks:

Model	With workflow	Stock
Claude Opus 4.8	95.6%	77.8%
GPT-5.5	91.1%	80.0%
Claude Haiku 4.5	75.6%	55.6%

Their own summary of that table is one sentence: "The workflow helped every model. With Opus, it added eight wins with no losses."

There's a reading they don't offer, and I want to be clear that it's mine rather than theirs. GPT-5.5 inside the workflow finished 91.1%. Opus 4.8 outside it finished 77.8%, on the same 45 tasks. Read across the rows instead of down them and the cheaper model in a system came out ahead of the stronger model on its own. Microsoft compares each model only against itself and never makes that claim. I'm making it, from their published table, so weigh it as my inference and not their finding.

Two caveats we'd want applied to us

Their coverage barely moved: 72.4% against 72.2% line, 49.8% against 49.1% branch. The agent also wrote slightly fewer tests than stock, 6,963 against 7,129. So the workflow didn't produce better tests. It produced tests that exist, build and pass, more often. They say so themselves, and it's a completion result rather than a quality one.

The gains also concentrate where the request is underspecified. On detailed prompts the workflow buys nothing at all. That's a real boundary on the claim, and it belongs in the headline rather than a footnote.

We'd hold our own numbers to exactly that. So we will.

Why we can't quote ours

We have a version of this result. On our own harness, switching the verification layer off dropped correctness from 100% to 75%, and switching the guards off as well took it to 50%. Small n, our harness, not a public benchmark, and we say that every time we cite it.

The bigger claim was a lift figure: a cheap model alone, then the same model inside our cascade. We published it. Then we audited our own truth layer and found something ugly. Our two baselines for the same quantity, the same cheap model alone on the same benchmark, disagreed by 3.1 points. One page computed the lift from one baseline. Another page computed it from the other. Each traced back to a real measurement, which is exactly why nobody caught it.

Both lift figures came off every public surface that day, and they stay off until one baseline is re-run.

So here's the honest position. Microsoft has a clean, large, well-controlled measurement of something we believe. We have an unresolved 3.1 point disagreement with ourselves about ours. Their result doesn't repair ours. Someone else agreeing with you isn't a re-measurement of your own work, and being right isn't the same as having measured it.

The part we did measure, and it points the same way

One result of ours survives a second independent run and an adversarial pass, so it can sit next to theirs.

We ran three models over the same 160 HumanEval+ problems, with the same prompts and the same scorer. They finished 92.5%, 91.2% and 90.0%. A 2.5 point spread that reads as interchangeable.

Partition by task instead of by column total and it looks different. 83% of the set is uncontested, where all three models get it right, and the models disagree on roughly 15 to 18 percent of the work across two runs. What makes that citable isn't how big the disagreement is, it's how stable: the same problems come up contested each time. 17 tasks are contested in both runs against 3.6 expected by chance, and ten of them are contested in both runs without any model ever changing its answer.

Which problems they split on is a property of the models. Which model scores highest is a property of the run. That's HumanEval+ Python on our harness, not customer traffic.

Put it beside Microsoft's table and you get one argument instead of two. They measured that the workflow around the model carries most of the gain you can reach. We measured that the ranking you'd use to pick a model is mostly noise. Both point at the same place, and it isn't the model.

What we took from it

One technique, straight away.

Before their agent declares a task complete it runs a set of checks, and one of them reads: "It considers small code changes that should make the tests fail. This is a lightweight form of mutation testing." No full mutation framework. Just a cheap step that catches an assertion which can't fail.

We knew that rule. We wrote it down after a scorer of ours passed its positive control perfectly while being structurally incapable of returning a negative. We wrote it down again when a guard printed OK at every boot and then fired on zero of seven deliberately injected defects. We were applying it from memory, and memory isn't a source. Microsoft made it a step in the loop.

So we built it. Every guard in our tree can now declare, in its own header, an injected defect that must make it exit non-zero. A runner executes each one in a fresh directory and reports whether the guard can still say no. Three grades: PROVEN, BROKEN, and UNPROVEN for guards that haven't declared a control yet. UNPROVEN counts as a warning rather than a failure, because a gate that's red on day one gets overridden by reflex, and a reflex override has stopped being a signal.

We watched it fail before trusting it. A stub guard that always exits zero grades BROKEN.

Then we pointed it at the guard from the story above, the one printing OK at every boot. We injected a wrong percentage where a real measured one belongs. Exit 0. We injected a cost multiple we'd already struck from our own public copy. Exit 0. A third run, with the correct value in place, confirmed the guard really was reading two live surfaces against the real ledger, so a trivial pass couldn't be mistaken for a catch. It graded BROKEN by its own declaration, and it's since been fixed.

As I write this the runner reports 7 proven, 0 broken, 33 unproven, out of 40 guards. That last number moved while I was drafting, because writing this article involved building another checker and the gate made me declare a control for it before it would let me commit. The unproven count is the honest one, and it's the one we expect to be asked about. It's also the point of the exercise. Before we built this, the answer to "which of your guards can still fail?" was "we assume all of them."

A postscript, because it happened while writing this

Fact-checking this article, I flagged four of its own numbers as fabrications. The branch coverage figures. The Haiku row. The three-model percentages. Our own ablation result.

All four were real. I'd checked them against a news summary of Microsoft's post rather than the post, and against the wrong one of our own benchmark artifacts. The summary leaves out branch coverage entirely and never mentions Haiku, so an absence looked like an invention.

We have a rule for this and I broke it anyway: never relay one party's account of another's work without reading the original. I was one hop further out than I thought, which is the only place that error lives.

Two defects did survive the pass, and both were in the draft's account of our own work. A comparison of mine that read as Microsoft's, and a stale guard count.

And I have to finish that sentence honestly, because the first version of this article went out before I caught either one. It was live for about three hours, saying that a guard "is now marked BROKEN" when we had fixed it that morning, and reporting 2 proven of 37 when the runner said 7 of 40. This text replaced it in place, which is why the two paragraphs above are written the way they are. An article about withdrawing a number published a wrong one first. That is the whole argument for having a correction path that edits at the claim instead of appending a note at the bottom, and it is why every figure here now carries a line in a verification block saying where it was re-derived from.

Sources

Microsoft .NET blog, the polyglot unit testing agent: https://devblogs.microsoft.com/dotnet/polyglot-unit-testing-agent/
The plugin, MIT licensed: https://github.com/dotnet/skills