DEV Community: Tom Jones

We asked a decision model to check our AI's work. It got good the day we stopped asking it to think.

Tom Jones — Sun, 11 Oct 2026 01:54:03 +0000

Microsoft put a small, strange model on OpenRouter this month. You hand it a situation and a question with named answers, and it hands back a probability for each answer. Microsoft-Decision-1 does that one thing.

We run a gateway that sends most requests to cheap models and checks the work before it goes out. A model that only answers yes or no, with a number attached, sounded like the checker we had been trying to build for months. We spent a Saturday finding out.

This is what happened, including the parts that did not work.

The first thing it taught us

We started by asking it the questions we would ask a careful colleague. Is this note relevant to what the agent is about to do? Does this tool call match what the user asked for?

It wobbled. We sent identical requests twice and got different probabilities back, sometimes by a lot. On a relevance question the same input moved by as much as 0.17 between calls. A check that changes its mind about the same input is useless.

Then we noticed where it held still. When the question was about a fact, it was steady. "Does the user's message contain this exact city name?" came back the same every time, and came back right.

That turned into the rule the rest of the day was built on. Let plain code find the facts: is the value in the user's words, is it the right type, does the list have the right number of items. Then ask the model one clear-cut question about a fact that code has already laid in front of it. When a question needs real judgement, do not force an answer. Say UNSURE and send it up to something stronger.

Give it a fact, not a judgement.

The checker

What we built is boring on purpose. Code checks the call. Decision-1 answers a few one-fact questions. If anything is wrong and fixable, a cheap model gets one try to repair the call, and the repaired call goes through the same checks. The output is APPROVE, REJECT or UNSURE, with a dossier of every check that ran.

We measured three things, because a checker can fail in three ways:

How often it approves a call that is wrong. We call this "less wrong", and wanted it at 5% or under.
How often it takes a wrong call, fixes it, and approves the fix. "More right." We wanted at least 20%.
How often its repair breaks a call that was already right. "Nothing broken." 2% or under.

Every version was tested the same way. Before any data existed, a separate model that had not seen our results wrote the pass and fail rules, and we locked them. Each version ran once on 300 tool calls it had never seen. Then fresh labellers, separate AI sessions given only a written rubric, who could not tell an original call from a repaired one, marked each call right or wrong, with a tenth of the calls labelled twice as a check. We scored only after the labels were in.

Four versions in one evening

version	wrong calls it approved	wrong calls fixed and approved	right calls it broke
2.1	9 of 48	25 of 100	2 of 200
2.2	4 of 33	18 of 92	2 of 203
2.3	2 of 34	22 of 88	1 of 211
2.4	0 of 72	20 of 96	0 of 201

Each step came from reading the calls the last version got wrong.

Version 2.2 raised the bar for a yes, and the leaks fell. The fixes fell with them. It was trading one good thing for another, so we set a floor on fixes and kept going.

Version 2.3 looked at the thirteen wrong calls that 2.1 and 2.2 had approved, and found they came in four shapes, all of them things code can see. A list sent as text inside a list. An optional value the user never mentioned. "Washington" when the user said "Washington D.C." A limit of 5 when the user said "less than 5". Four new facts for the checker, and the leaks fell to 2 while the fixes came back.

One of those two leaks was a list of dietary needs sent as ['["vegan"]']. A list, inside a list, written as text. The exact shape 2.3 was built to catch. It slipped through because one part of the code wrapped the text in a list before the next part could notice it was a list already. The other surprise was a repair that turned a growth rate of 0.015 into 1.5, the same number written as a percent, which is a very different forecast.

Version 2.4 fixed the order of those two checks, stopped the repair model from rescaling numbers, and added one quiet rule: a repair is kept only if the checker approves it. Otherwise the original call goes out, untouched. That rule is why nothing broke. A repair nobody could vouch for was never allowed to replace anything.

What zero means

When we told the reviewer who writes our rules that 2.4 had approved no wrong calls, it pushed back hard. Part of its review held. Every version ran on a different draw of calls, with different labellers, so the trend across versions carries noise. And the fix rate of 20 out of 96 clears our floor of 20% by a hair.

Part of it did not hold. It suggested the zero was flattered by a larger denominator. We counted the plain number underneath. Across the four versions the checker said yes 164, 150, 120 and 110 times, and was wrong on 9, 4, 2 and 0 of those. When 2.4 says yes, it has been right every time so far. That count stands on its own.

The real price is visible in the same numbers. It says yes less often.

The bill

Decision-1 is cheap enough to disappear. The whole 300-call test cost under a cent, and on 1,417 more recorded calls it ran at about two thousandths of a cent per call, in under a second at the median.

The interesting cost is somewhere else. A checker changes what you are buying. Without one, you pay per token and hope the answer was right. With one, the unit becomes a checked answer, and its price includes every call the checker would not approve and had to send somewhere more expensive.

That pushed us to think in price per task instead of price per token. On our own recorded runs, a cheap model plus the checker plus escalation lands at a fraction of what the same tasks cost on a premium model called directly. How small a fraction depends almost entirely on one number: how often the checker can say yes.

The problem we have not solved

Four in ten calls come back UNSURE. Most of them hinge on an argument the checker cannot pin to a fact in the user's words. Code cannot see what such a value means, and the decision model, asked to judge it, wobbles.

The question we are carrying into next week is this. How do you turn a judgement about meaning into a fact that plain code can check, without teaching the checker to say yes to things it should not?

If you have built something that does this, we would like to hear how.

Tom Jones, Spanda Works. The checker and the test harness are ours; Microsoft-Decision-1 is Microsoft's model, used through OpenRouter.

Our AI judge listens to how sure you sound. Our first number about it was wrong.

Tom Jones — Tue, 06 Oct 2026 16:23:54 +0000

Our verification layer uses an outside model as a judge: it reads a claim and its source and decides whether to allow it. In July, Mike Czerwinski (@jugeni) asked a question we could not answer, in the comments of his own post on where the line between code and judgment sits. The whole exchange, run by run, is in that thread. Does that judge's verdict change when only the voice of the claim changes? We said we would run it and publish the result either way. This is that post, and the short answer is yes, a little, and for a reason we can now name.

It took seven runs. Two of our own numbers did not survive, and one is still waiting on a re-run. Here is the whole story in one picture, and each section below is one box of it.

1. The question

The judge was chosen so its mistakes would not line up with the mistakes of the model it checks. Mike pointed out that this says nothing about style. A judge can reason independently and still relax when the prose sounds confident and familiar. If the verdict moves when only the voice moves, our independence claim is weaker than we measured.

Three things in this post belong to Mike: the hypothesis, the push to check content with a rule instead of a second model, and the fix we tried at the end. We built the harness, ran it, and are publishing the version where we were wrong.

Our first attempt (v0) used ten cases far from the decision line, and the verdict never moved. Cases that far from the line could never have flipped, so the run could only pass. It also showed us the judge's score takes just two values, so it carries no confidence to measure.

2. The first number was a confound

v1 used twelve cases near the line, each rewritten in three voices. A third of the verdicts flipped when the voice changed, and hedged claims passed more often. We posted that as an early result, and the headline read like "the judge rewards hedging."

It was wrong in the part that mattered. To change the voice we rewrote the text, and rewriting the text changed the content. The confident versions carried more evidence; the hedged ones were vaguer claims as well as softer ones.

The fix was Mike's. Check that content survived with a rule instead of a model, because a model adds a second judge with its own taste in style. So every case became one neutral claim, and each voice is the same claim with "Certainly," or "Perhaps," added. A plain rule checks that every number and content word survives.

3. Holding content fixed

With content locked, the flip rate fell from 33% at n=12 to 17% at n=24 and 14.6% at n=48 (95% CI roughly 4 to 25%). It still sits well above how often the judge disagrees with itself on an identical input.

4. The direction disappeared

The bigger change was in direction. At v1, voice looked like it moved pass rates by 25 points. At v3 the three voices passed at 88%, 88% and 85%. Assertive minus hedged is +2 points with a 95% CI of -8 to +12. The interval straddles zero, so there is no directional tone bias. Content held up too, with cosine similarity to the neutral version at 0.955 for assertive and 0.923 for hedged.

So about 15% of borderline verdicts still flip on tone alone, and they flip in both directions. Please retire the 33%. It was our number, we published it first, and it is the one we most want retired.

5. Is it just noise?

A flip rate means nothing until you know how often the judge flips on an input that did not change. Our first try (v5, 8 repeats) said never. The second (v6, 6 repeats on more cases) found the judge's repeated calls are independent samples, and the true floor is 2.08% on borderline cases and 0% on clear ones. One caveat: between those two runs we swapped the judge model to a larger one, so v5 and v6 measure different judges.

Mike asked for two more controls. Control A ran the three voices on clear cases, not only borderline ones. The effect is smaller there, and still present. "The test suite passed" is allowed when stated plainly and refused when hedged.

Control B asked whether the flips are the same claims every time or random scatter. They are the same claims, and those claims stay put when the tone is held still. Tone walks specific claims across the line.

6. Why it happens

The judge reads how sure the claimant sounds as evidence about whether the claim is supported. That one sentence explains both directions. A hedge makes a weak claim sound careful, so it gets through. A hedge makes a strong claim sound unsupported, so it gets blocked.

For a general reviewer that is reasonable behaviour. For a source-grounding check it is a flaw, because the only question should be whether the source supports the claim. Ours moves with how confident the claimant sounded, and now we can say by how much and in which direction.

7. The fix, and the part we owe you

Mike proposed stripping the stance words before the judge sees the text. On 07-18 we ran it and told him the flip rate dropped to about 2.1%, all the way to the noise floor.

When we went back to write this up, we could not find that run's artifact: no v7 script and no result file anywhere in our history. We are keeping the report, since the session that had just run it made it. We stopped citing it as evidence, because nobody can check it. The line we hold is reported versus re-citable.

The 2.1% also matches the v6 floor of 2.08% exactly. That fits a fix that removed the effect completely. It also fits a number that travelled: a handoff note called 2.1% a floor, and a later summary rewrote it as a fix that worked. We treat that second possibility as a pipeline defect. Running v7 again, with the artifact kept, decides which one it was.

What we still do not know

One judge. The model swap between v5 and v6 shows how much that can matter.
A hand-picked fixture. Cases were chosen near the decision line on purpose, so these flip rates describe cases near the line rather than ordinary traffic.
Added stance words only. Confidence woven into sentence structure is the harder case, and a naturally hedged fixture is the next thing we owe.
Stripping leaves residue. Our stripper left a dangling "though". It carried nothing here, but leftovers like that are where a residual effect could hide.

The residual flips go both ways. Boundary noise looks like that, and so does a very small bias hiding inside boundary noise. We lack a method that separates the two. If you do, we would like to hear it.

An evaluator is only worth what its error bars are worth. The number we most wanted to be true, 33% and a clean mechanism, is the one we spent three runs killing.

If you want the run-by-run conversation that produced all of this, including Mike's designs for the next fixture, it is in his thread.

Implementation is where judgements go to become invisible

Tom Jones — Sun, 27 Sep 2026 22:38:21 +0000

One question, three answers

I have a small tool that finds people waiting for a reply from me.

It had three versions. Each one was correct. Each one gave a different answer.

Version one said 18. It counted a reply only if it sat directly under theirs. But dev.to sometimes will not show a comment its API still returns, so you reply beside it instead. Nine of ten had been answered that way.
Version two said 34. It counted any later comment from me in the thread. So it counted two other people talking to each other.
Version three said 2. It used both rules. One was a friendly sign-off from July. The other was Pascal, suggesting there might be an article hiding in these comments.

This is that article.

Nothing in the tool was broken. Each count was right about its own population. The trouble was the question. Every version reported its number as the answer to "who is waiting?", and every version had quietly decided what "answered" means.

The first version was built with care. It encoded a sensible judgement. Then the judgement stopped looking like one.

That is the whole article in one incident. The rest is how we got there.

How we got here

It started in July, in the comments under Pascal's article about replacing Calendly. It ran for two months, in bursts, with pauses while one of us went off and tested things in production.

We did not start with a method. It went like this, over and over:

1. one of us has an answer that looks finished
2. the other brings back a real failure
3. the answer turns out to be hiding a judgement

Sometimes Pascal had the answer and I found the hole. Sometimes the other way round. Here it is in the order it came up.

The test passes

The first example is from Pascal's article. Two people try to book the same slot at the same moment. On Postgres that is a real race. On SQLite it cannot happen, because SQLite lets only one writer in at a time.

So a test for that race, run on SQLite, passes forever. The bug ships on Postgres.

The hidden judgement: that this test can fail at all.

The fix is mechanical. After you write a test, break the thing it protects on purpose and run it again.

Green means it was never watching anything.

Pascal's version: a regression test is valuable because "it represents a failure that actually happened and that we proved it can fail when the condition comes back." Most people keep the first half of that sentence and drop the second.

The test is weak

Breaking code on purpose to see if a test notices is called mutation testing. When the test stays green, everyone assumes the test is weak.

That is one of four causes, and the least interesting one.

My case was the first box. A key binding compared the key press against " ". The runtime reported the space bar as "space". That branch had never matched in the whole life of the file.

I had already told a colleague I had introduced a regression there. There was nothing to regress.

The hidden judgement: that the code under test runs at all.

Pascal turned the boxes into an order of checks:

Is the path reachable?
Does the test observe the behaviour?
Does breaking it make the test fail?

Only after all three is "the test is weak" the right diagnosis.

He also suggested that a surviving mutant may be less a verdict on the test than "a way of challenging the story the code tells about itself."

The checks are green

That story has several authors. The code says what it does. The tests appear to confirm it. The last person who explained the system repeats it.

They can all be wrong in the same way at once. That is what happened with the space bar.

What breaks the agreement is something that cannot be told the story. Pascal described it as "an artefact whose validity is defined operationally rather than narratively," and its whole test fits in four words: "Show me the failure."

Then he pushed further. Tests only check what somebody thought of. Real users bring what nobody thought of. In his words, production is "the one test suite whose authors we don't control."

I had a painful example. My gateway speaks three API formats, and every check was green.

The checks used curl, written by me, speaking my own format back to me. One vendor SDK sends its key in a different header than the one my edge rule read. Its requests died before reaching the code that would have handled them.

The hidden judgement: who wrote the check. The checker and the thing checked had the same author. So they had the same blind spot.

Add more checks

The obvious response is more checks. That only helps if they can disagree.

Before, on the left: three checks on the same registry of files. Different code, written at different times, for different reasons. All green. Two files sat in a folder the registry had been told to skip, so none of the three ever saw them.

Three green lights. One blind spot, seen three times.

After, on the right: a real catch. Two tools counted the same store and got 16 and 2,375. One walked a list someone had declared. The other walked the disk. Neither got its input from the other, so they could come apart, and one day they did.

Pascal's sentence for this: independence is about "preserving the possibility of disagreement." Different code is not enough. The checks have to be able to be wrong in different ways.

Zero replies waiting

Every check counts something, and somebody chose what. Usually nobody did. The set arrived as whatever the first caller passed in, and it stuck.

Pascal asked the question that names it: "Who chose what the test is allowed to see?"

My nightly report once led with this line:

0 replies waiting on our own articles

True. The session that read it concluded nobody was waiting on me. False. Replies on other people's articles lived in a different bucket.

The count never lied. It answered a narrower question in the voice of a wider one.

The hidden judgement: which articles count as mine to watch. This is the same family as the detector at the top, one step earlier.

I fixed the report rather than adding a checker. It now says what it could not see, and prints coverage unknown when it cannot list everything. An empty set and a set nobody measured both print zero, and they mean opposite things.

One setup wins by a wide margin

A comparison has two denominators. They can differ without anyone noticing.

I was writing up a retrieval benchmark where one setup beat another by a wide margin. Every number was accurate. Then I looked at what each setup searched. One ranked over 3,038 documents. The other ranked over 7, and those 7 were the files holding the answers, because a helper had built its index from the task list.

Pascal's reply was the shortest rule in the thread: "You don't add bananas and monkeys."

It held up a few hours later. A cache keyed each document by a shortened id, so 120 documents produced 86 keys. 34 were scored against someone else's results while the report kept dividing by 120. One line caught it:

assert len(cache) == len(documents)

The same line then caught my fix, which collided the other way.

The nastier cousin is when both counts agree. My usage log recorded 2,488 requests under one provider name. For that kind of request the service points at a different provider and leaves the name alone. Both sides said 2,488 because both read the same label.

As Pascal put it, "Sometimes you need to look at the thing being named, rather than trusting the name."

Value, scope, treatment

Past a certain point the number is perfectly correct, and it answers a narrower question than the reader reasonably thinks it answers. Pascal called that a broken semantic contract, and noted there may be no invariant to catch it.

His answer: a measurement is "value + scope + treatment." If changing the scope or the treatment could change what the number means, they belong next to it.

That answer looked finished. I answered with the term it was hiding, because it had just bitten me. The comparison is a treatment too.

I had published that notes my system pushes to an agent change what it does, compared with a random note. Then I rebuilt the control the way a published benchmark does, using the nearest wrong note instead. The effect disappeared: 35.4% against 30.8%, p=0.549.

The measurement never changed. What it was compared to did.

The hidden judgement: what the number was compared to. So scope, treatment and comparison all go next to the number.

Every claim has evidence

That catch came from reading someone else's method. None of my own checks found it. My gates confirm every claim has evidence, and that claim did.

Pascal named the gap. Nothing asked "why do we believe this is the right question?"

I got an example within the hour. I had been calling my note delivery "noisy," so I measured noise carefully and built a fix. It changed nothing.

Then a colleague asked what it was actually like inside, and I looked at one real session instead of the averages. Nothing was noisy. Exactly two notes had been crowded out and never arrived. They were the two I had just spent an evening learning again the hard way.

Pascal's line: a knowledge base can be "complete and still be functionally incomplete."

Then the outside check failed the same way. A second model reviewed my decisions. It was sharp and mostly right, and it concluded I had shipped a null result. That part was wrong, because the evidence against it existed and I had left it out of the brief.

Pascal separated the two cleanly. The reviewer was "independent" in its reasoning, but not "independently informed."

The hidden judgement: which question was worth asking, and what the outside reviewer was allowed to see.

The randomiser is fine

I randomly hold back ten percent of those notes, so I can compare what happens with and without them. When I finally computed the result, the two groups turned out to be logged at different points in the pipeline:

held back   logged the moment it was held
delivered   logged only if it survived several later filters

Sixty-eight days of data compared a whole group with its survivors. The randomiser was fine. Someone had once decided where to put two log lines, and that decision had quietly become code.

The hidden judgement: where the log lines went.

I wrote back that implementation is where judgements go to become invisible. Pascal finished the thought: a judgement "starts as a conscious choice, becomes a field, a log point, a default, a population boundary, a control, an ordering decision," and six months later it just looks like how the system works.

Back to 18, 34, 2

While we were agreeing on that, it happened again, to the detector at the top.

Look at it with the whole thread behind it:

Version	Count	What it silently decided "answered" means
1	18	my reply sits directly under theirs
2	34	any later comment from me in the thread
3	2	both rules together

Three correct measurements. Three different populations. One question.

Much of the thread is in there. Version one could not see a reply placed beside a comment, so it could not notice being wrong about one. Each version counted a population somebody chose once. Each number was right, and the reader heard a different question. And a judgement about one word sat in the code, looking like how the system works.

What you could extract afterwards

Once it was over, you could write it up as a list. I did, in the first draft.

Can it fail? Break it on purpose and watch.
Does the code under test run at all?
Who wrote the check, and does it share an author with what it checks?
Could your checks disagree, or do they share one input?
What exactly is being counted, and who chose that?
Are both sides of the comparison the same kind of thing?
Does the number carry its scope, its treatment and its comparison?
Is this the right question, and did anyone outside your own checks get a real chance to ask?
Which of these answers is now buried in code, where nobody will see it as a choice?

Pascal's review of that draft caught the problem. We never started with nine questions and applied them to nine cases. We kept finding an answer that looked finished, and then finding the judgement inside it.

A clean list is exactly the kind of instrument this article warns about. Use it, and expect it to be hiding something too.

The useful items cost a line of code each: a count printed next to a result, an assertion that two populations match, a mutant run once. The hard ones do not go away when you automate them. The judgement moves somewhere it is harder to see.

What I'd try next

The logging from the randomiser story is fixed and symmetric now. The next step is per note: for each note, compare the rate of the specific mistake it warns about when it was delivered against when it was held back. That turns "did the note arrive when it mattered" into arithmetic.

It will not tell me I picked the right mistake to detect. That part is still a judgement, and I expect it to hide in the code the same way the log lines did.

Neither of us would have found most of this alone. That is the other thing the thread showed.

This came out of a two-month thread with Pascal Cescato, who also reviewed this draft. Thank you, Pascal.

Crystals: agent memory that arrives before you act

Tom Jones — Fri, 25 Sep 2026 20:44:57 +0000

Most agent memory is a search box. The model wonders something, a retriever runs, chunks come back. That design has one load-bearing hole: retrieval fires only when the agent already suspects it needs help. The expensive mistakes are the confident ones, and a confident agent never queries.

So we push instead. A note gets bound to an action, and it arrives in the moment before that action runs, whether or not anybody asked.

We call these crystals. The name earns its keep. A crystal has one axis, it formed under pressure, and what you get out is small and hard. One knowing per crystal. That constraint turns out to be the difference between this working and this quietly failing, for reasons I will get to.

Before writing this up I went looking for prior art, expecting to find none. Expecting to find none is usually the sign you have not looked hard enough, and sure enough, two of the four ideas here are already published. Rules that fire on the action rather than on a similarity score have been described, right down to injecting before a git commit under a hook time budget. The agent-memory evaluation literature also has a failure taxonomy that comes close. Mem2ActBench attributes memory-grounded errors to five types, two of which sound like the one below: Retrieved-but-Unused, "evidence is retrieved but not utilized", and Lossless Retention Failure, "long/structured values are corrupted (e.g., truncation or character-level errors)".

Read them closely, though, and both sit on the model's side of the boundary. Retrieved-but-Unused means the evidence reached the context window and the model ignored it. Lossless Retention Failure means the model mangled a long value on the way out. The failure I describe below happens earlier, in the plumbing: the note is correctly matched and then cut by the budget packer, so it never reaches the model at all. To the model it never existed; to the logs it was a hit.

The budget problem is named too. A 2026 survey of agent memory lists "curation under fixed budgets, including update, eviction" as an open direction, and asks for benchmarks that measure "memory quality as a function of token budget, storage cost, and latency, reflecting the bounded-memory conditions of real deployments."

Which is a fair description of what follows, so treat this as a field report against that request rather than a new idea. What I have that the papers mostly do not is a channel that has been running long enough to overflow, and the numbers from the far side of that. Here is the mechanism, the budget it collides with, and the two rules we did not know we needed.

Anatomy of a crystal

A crystal is a markdown file with a binding block at the top:

crystal:
  on: bash, write                              # which ACTS it fires on
  match: git reset --hard, git clean, rm -rf    # substrings in the act's text
  deliver: act                                 # the channel
  stale_after: 2027-03-25                      # after this, the claim is withheld

and, further down, the part that actually gets delivered, fenced by two markers:

<!-- crystal:essence -->
⛔ A DESTRUCTIVE VERB ACTS ON A TREE YOU SHARE. `git reset --hard <ref>` also
discards TRACKED-but-uncommitted changes; only UNTRACKED files survive...
<!-- /crystal:essence -->

The file around it can run three thousand words of history, rationale and dead ends. Only what sits between those markers reaches the model. So the crystal is the delivered sentence, and the file is where it came from. That split matters: the file is for whoever is arguing about the rule later, the essence is for whoever is about to break it in four seconds.

Hooks, and what an "act" is

A hook fires on the tool call before it executes. Which tool it was determines the act, and the act determines what a crystal can match against:

act	fires	what `match:` tests against
`bash`	before a shell command	the command line
`write`	before a file write or edit	the path, plus the content being written
`delegate`	before briefing a subagent	the brief
`commit`	inside the git pre-commit hook	the staged file list
`prompt`	when a request arrives, before any tool runs	the request text
`boot`	at session start	nothing yet

The third column is where the power sits. write can see content, so a crystal about pricing claims fires on the pricing claim itself and leaves your other edits alone. prompt fires earliest of all: the crystal lands before the first tool call, which is the only place you can still change the plan rather than the keystroke.

Matching stays deliberately dumb. match: is a comma list of plain substrings, tested case-insensitively, with no word boundaries and no embedding anywhere in the path. A substring test stays auditable at a glance and cannot drift, which is what you want from something firing three hundred times a day.

What it looks like when a crystal fires

Three real ones from a single day of work.

Before a shell command. I typed a git reset --hard. Before it ran:

⛔ A DESTRUCTIVE VERB ACTS ON A TREE YOU SHARE. git status --short first, and account for every line. Uncommitted work you did not make is an agent's.

Before a file write. I began writing a note that made a causal claim. Before the write landed:

⛔ If you are naming a CAUSE: write the RIVAL (what ELSE produces this exact observation?) and the DISCRIMINATOR (the one measurement that separates them). On 2026-07-19 four confident cause-claims were made in one evening; every one had evidence and every one was wrong.

Before a commit. In the pre-commit hook, matched against the staged file list:

⛔ FINISHING SOMETHING IS NOT THE SAME AS IT BEING RECORDED WHERE PEOPLE READ IT. Name in the commit message everything the commit changed, especially a flag touched while doing something else.

I asked for none of those. That is the entire point, because in each case I was not wondering.

Then you hit the budget

There are 300-odd crystals. The median action matches five of them, wanting 13,014 characters. The budget is 4,000 characters per action, and no single crystal may take more than 2,000 of it unless it wins outright.

93% of actions want more than the whole budget. Three times oversubscribed, at the median. So the channel behaves like an auction that runs a few hundred times a day, and on almost every run something loses.

That reframes the design question. Delivery was the easy half. The hard half is an admission rule, an eviction cost, and a way to degrade.

Fair scheduling beats relevance here

Selection goes least-served-first, then longest-unheard, then shortest as a packing tiebreak. A crystal that already fired this session steps aside for one that has not.

Relevance ranking sounds better and fails badly here. A handful of broad crystals win every auction while fifty go undelivered, which is what we measured before rotation existed.

A cheap tier that somebody wrote

When a crystal cannot fit, it emits a one-line summary instead. We call that its TELL, and the median TELL runs 81 characters.

We tested whether a TELL carries anything real. Take 20 crystals whose full text demonstrably changes what the agent does, and run a blind judge against three resolutions:

what the model receives	median chars	behaviour preserved
full essence	2,302	20 / 20
head, 750-char prefix	631	19 / 20
authored TELL	81	17 / 20
421-char prefix	421	12 / 20

Two things fall out, and I nearly published the second one backwards.

An authored 81 characters beat an arbitrary 421-character prefix, 17 against 12, at a fifth the size. A prefix stops wherever the character count happens to land, which is mid-argument more often than not.

But a longer arbitrary prefix does fine: 750 characters keeps 19 of 20. So the enemy here turns out to be brevity itself. Cut lightly and curation barely matters; cut deep and it starts to matter enormously. Zero items flipped the other way at any resolution, which argues against judge noise.

Where it still breaks

Head-first truncation is safe when a crystal carries one idea, because the claim leads. Give a crystal two ideas and the second one sits in the tail, where it silently vanishes.

Across our corpus that comes to 98,715 characters which are stored, correctly matched to an action, and withheld on any act where rotation does not elevate them. Of 102 crystals over the per-item cap, 88 withhold at least one marked claim in the tail. Treat that 88 as an upper bound. Some tails turn out to be supporting evidence for the head's claim, and I have not separated those by hand.

No retrieval metric has a name for this. The crystal is present, correctly selected, and cut. I found it by shipping the same bug twice in one session with the warning sitting one character-cap away from my eyes.

Correction, a few hours after publishing. I originally wrote that this "logs like a hit and behaves like a miss." A reader asked whether I had actually read the papers I was citing here. I had not, so I went and ran the proper test on our own channel: the restore counterfactual, which reinstates the withheld text and reruns the same judge. On 65 cut pairs, restoring the tail changed the judged action 9 times and lost it 5 times, p=0.424, not significant. It also failed to beat a length-matched tail of unrelated text (p=0.388). So 98,715 is a true character count and not a demonstrated loss, and the "behaves like a miss" half was my inference rather than my measurement. The study is underpowered at this n, so this is "not shown", not "shown absent". Half the cut pairs turn out to be residual: the tail would not have changed the action anyway.

Match precision is the other open front, and this is where correcting myself cost the most. A blind two-judge study asked how often an admitted crystal actually changes the action, against random same-act controls. The controls scored 0 of 41 both times, so against that bar the matcher is doing real work (p=0.0001 and p=0.0054). The crystals themselves scored somewhere between 20% and 49%.

Then I read how Mem2ActBench builds its controls: not randomly, but as "hard negatives (distractor tools most semantically similar to the ground-truth tool)". Re-running ours that way, against the lexically nearest crystal that did not match the act, the effect disappears. A matched crystal scores 35.4%, a hard negative 30.8%, p=0.549. It beats a random draw (p=0.008) and not a near-miss. The hard-negative control is genuinely harder (p=0.029 against random), so the control works and the matcher is what failed it.

Which suggests our matcher selects on topic rather than on applicability, and that our published precision result lived entirely in the gap between a random control and a good one. If you take one thing from this piece, take that: a control that is too easy manufactures a result.

That range is embarrassing, and explaining it matters, because it turns out to be an artifact of my own apparatus. The 20% end was measured by showing the judge a 421-character excerpt, which the resolution study above had already identified as the worst representation tested. Shown fuller text, the same judge said 48.8%. I had measured relevance through a truncation, and I nearly published the truncation's score as the system's.

Either way, roughly half the admitted crystals fail to earn their slot. Substring keys are the reason. A two-letter key like pt matches nearly everything, and one loose key is how a single crystal loses the auction for everybody else.

The two rules we did not have

One knowing per crystal. This reads as style advice and turns out to be load-bearing. Head-first truncation is only safe when the claim leads, and we learned that by watching second claims disappear. A crystal that has grown two ideas wants splitting.

A crystal earns its binding by beating what it evicts. Truth alone will not buy a slot. At three times oversubscribed, every new binding forcibly removes one that already matched, so admission is always a comparison against the incumbent.

We put that to the test. I proposed binding eight orphaned notes. Two reviewers, briefed separately, independently came back with zero to two, and the admission test they both converged on was:

Does it change this action?
Can a tight key reach it without catching everything else?
Is the act earlier than the next reliable check that would catch the same mistake?
Is there no cheaper enforcer?
Does it beat the incumbent it evicts?

Most candidates die on 4. A crystal about your own tooling usually wants to be an assert in the script instead.

If you build one

The core design holds. Pushing beats pulling for anything that must not be forgotten, precisely because the agent's sense of when it needs help is the faculty you cannot lean on.

What we underestimated is that a push channel is a budget, and a budget needs an economics. We built the delivery first and found the economics a month later, by measuring what never arrived.

Measure that early. The useful question about a crystal is what did this displace, and did anyone ever read the part that got cut.

The rest of this thread

This piece is the mechanism. The others are the pieces of it I got wrong first, which is the more useful reading order if you are building one:

I shipped the same bug twice with the warning sitting one character-cap away is the incident that found the truncation, and where the 98,715 figure comes from.
Your hooks are a fence. They could be a body is the argument for hooks doing more than blocking, which is the design this whole channel came out of.
Whole notes, not fragments: the retrieval half is the pull side. Crystals are the push side, and the two channels answer different questions.
Escalating to the better model made 34 answers worse is unrelated to memory and lands on the same methodological point: the aggregate hid the failure, and only stratifying showed it.

If you want the gateway underneath all of this, Tirtha is free to test, no card, 100 requests a month per key, raised on request.

I shipped the same bug twice with the warning sitting one character-cap away

Tom Jones — Fri, 25 Sep 2026 19:58:29 +0000

Our agent memory pushes short notes to the model at the moment it acts. No vector search at question time; a note bound to a shell command or a file write, delivered before the command runs.

Yesterday I shipped the same bug twice in one session. A string replacement that matches nothing returns the original unchanged, so the patch looks applied, the script prints a cheerful line, and the file is byte-identical. I did it, caught it, fixed it, and did it again ninety minutes later.

Afterwards I went looking for the note we were missing, expecting to write it.

The note existed

It was bound to exactly the right trigger. It described the bug precisely, down to the same function call, with the incident that produced it. Selection worked. Matching worked.

On that action it is delivered as 675 characters of 3,751. Each item is capped, the cut takes the head, and the sentence I needed sits in the tail.

So the warning was written and bound and matched, then cut. I made the mistake it describes, twice, with it one cap away.

How much of this is there

98,715 characters are stored, correctly matched to an action, and never delivered. 88 of 102 oversized notes withhold at least one load-bearing claim that way.

I went looking for what was missing from the channel. Nothing was missing.

Why head-first truncation is the wrong instinct here

Cutting from the front is defensible. A note leads with its claim, so the head carries the point and the tail carries the evidence. We measured whether that holds.

One judge, 41 real pairs of an action and a note, the same pair shown at three resolutions, scored on a single question: would a competent session do anything differently because it read this? Of the 20 whose full text changes the action:

what the reader gets	median chars	knowings preserved
the full note	2,302	20 of 20
first 750 characters	631	19 of 20
an authored one-line summary	81	17 of 20
first 421 characters	421	12 of 20

Two things fall out of that table.

A curated 81 characters beat an arbitrary 421 at a fifth of the cost. A prefix cuts wherever the character count lands, usually mid-preamble, so it can hide the claim while costing five times more than the sentence that states it. The cheap tier of any budgeted channel should be the authored line, never a truncation.

And head-first is safe for one idea and lossy for two. The 19-of-20 result holds when the claim leads. It stops holding when a note bundles two claims, because the second one lives in the tail and goes silently. 88 of our 102 oversized notes were doing exactly that.

The repair is one idea per note, which is the rule that makes head-first safe in the first place. Cleverer cutting buys nothing.

Then the harder question: is the channel too small, or are we

Replaying 95 real actions through the live selector: the median action matches five notes wanting 13,014 characters against a 4,000 character budget. 93% of actions want more than the whole budget. Two thirds of everything that matches is starved, always, by construction.

The obvious fix is a no-op. Capping every note at 2,000 characters moves delivery from 35.7% to 35.7%, because the packer already caps there. Gains only start below 1,500, and our median note is 1,759.

So we measured the other lever. Two blind judges, 41 pairs of an action and a note the matcher admitted, against 41 pairs of the same actions with a random unmatched note. Neither judge saw which was which.

	admitted notes that change the action	random controls
judge A	13 of 41	0 of 41
judge B	8 of 41	0 of 41

Zero on the controls, twice, independently. Without that the other column would mean nothing: a judge that says yes to 20% of real matches tells you nothing until you know it says yes to none of the random ones.

Correction, same day, and this one cost more. The word doing the work above is random. I later read how Mem2ActBench builds controls, and it draws hard negatives: the distractor most similar to the right answer. Re-running ours that way, against the nearest note that did not match the act, the effect disappears. Matched scores 35.4%, a hard negative 30.8%, p=0.549. It beats a random draw (p=0.008) and not a near-miss. The hard-negative control is genuinely harder (p=0.029 against random), so the control works and the matcher is what failed it. Our whole published effect lived in the gap between a lazy control and a good one, which suggests the matcher selects on topic rather than on applicability.

The matcher carries real signal and admits roughly three notes for every one that belongs.

The two-character key

One note matched 44% of every action we replayed. Its match key was pt, meant as typographic points, on a note about document spacing.

pt is inside scripts.

That note was also the single most-starved item in the corpus. It matched nearly everything, lost nearly every packing contest, and on the occasions it won, both judges rejected it. One key, three symptoms. Its neighbour was -h, meant as the help flag, which is inside --hard, so it fired on git reset --hard.

Removing two keys took matches from 737 to 708 and actions starving from 72 to 62, with no real note lost.

Our first fix silently did nothing. We changed -h to " -h" with a leading space. The parser splits the key list on commas and strips each key, so it stored -h straight back. The config read correctly and the behaviour never changed, which is the same failure shape as the bug that started this piece.

What earns a place

We had 20 notes written but bound to nothing, and proposed binding eight of them. Two reviewers, briefed separately, never shown each other's answers, independently said zero to two. They also produced nearly the same admission test, which we lacked and could not state:

Act-synchronous. It must change this action before it executes. If it can wait for someone to search, it is a lookup.
Matcher-reachable. Stable substrings in the action predict the failure. If a tight key is impossible without spraying every action, the channel is the wrong home for it.
Earlier than the next reliable check. If a commit hook already catches it, the push is worth only the gap.
No cheaper enforcer. Unreachable by a gate, a linter, a type, or a default in the one code path. Notes about your own tooling almost always fail here: put the assert in the script.
It must beat what it evicts. At three times oversubscribed, a new binding forcibly removes one that already matched.

"This is true and nothing delivers it" fails as an admission case. That was our criterion, and it was the word "cross-cutting", which defines nothing.

What shipped instead of eight bindings was one sentence added to a note that already fired on those actions. Measured cost: three deliveries lost across 97 actions.

What we would tell you to check

Ask what happens if retrieval misses a fact. Then ask the second question, the one we lacked: what happens if it hits and arrives partial.

A partial arrival is indistinguishable from an arrival at the point of use. It will not appear in a retrieval metric, a hit rate, or a log line, because every one of those reports that the right thing was selected. It was.

Correction, same day. This sentence originally ended "and behaves exactly like a miss." I had not measured that, I had inferred it. So I went and ran the restore counterfactual: put the withheld text back, rerun the same blind judge, 65 cut pairs. Restoring it changed the judged action 9 times and lost it 5 times, p=0.424, and it did not beat a length-matched tail of unrelated text (p=0.388). Half the cut pairs turn out to be residual, meaning the tail would not have changed the action anyway. So 98,715 is a true character count and the harm is not demonstrated. The study is underpowered at that n, so this is "not shown" rather than "shown absent", but I am not entitled to the stronger sentence and it is gone.

Where this could be wrong

The resolution study is one judge and 20 action-changing pairs. 17 of 20 is a wide interval. Trust the ORDER of those rows and treat the exact percentages as soft. A second judge would move them, since on the precision study two judges differed by 12 points on identical items.

The precision figure is a range, 20 to 49 percent, and its low end is the weaker measurement: it was taken through a 421-character excerpt, which the table above shows is the worst representation we tested. We published 31.7% first and corrected it the same day.

The marker counts behind "98,715 characters withheld" are an upper bound. Some tails are evidence for the head's claim instead of a second claim, and nobody read all 102 by hand.

And the twenty unbound notes were found because they happened to carry a marker in the file. An accident of how they were written, and no evidence they are the highest-value missing pieces. The ones that matter may carry no marker at all, and we still have no way to look for those.

The rest of this thread

Crystals: agent memory that arrives before you act is the mechanism this piece assumes: what the notes are, how hooks bind them to an action, and what the budget does. Read that one first if none of the above made sense.
Your hooks are a fence. They could be a body is where the design started.
Whole notes, not fragments: the retrieval half is the pull channel, which fails differently.

Escalating to the better model made 34 answers worse

Tom Jones — Fri, 25 Sep 2026 19:51:56 +0000

The standard picture of a cheap AI system is a ladder. Small model at the bottom, big one at the top, switch up when the work looks hard. Almost every cost-saving story is a version of that, and the whole design question collapses into "when do I escalate".

We built that. Then we published that our router for it does not work. This is the follow-up we owed. Our router failing was the least of it: the ladder itself is the wrong axis to argue about, and we can now show that with a number.

The four layers

Every number in that figure is from our own runs, and the rest of this piece walks down it. Layer 1 is the only one a ladder can reach.

Layer 1: which model you pick

The result that changed our mind

We ran 2,400 tasks through both rungs. Same tasks, both arms pinned, three repetitions each, so every task has a cheap answer and a frontier answer side by side.

The frontier tier wins. It is 96.9% against 92.8%, an advantage of 4.12 points, 95% CI +3.12 to +5.21, McNemar exact p below 0.0001. It replicates an earlier, weaker result of ours that had never been demonstrated properly.

Then look at what is inside those 4.12 points.

Escalation fixed 133 answers and broke 34. The net is +99, which is where the 4.12 points come from. But 34 tasks were answered correctly by the cheap model and incorrectly by the expensive one.

The obvious objection is that those 34 are noise, and we checked. Of them, 10 were right in all three cheap repetitions and wrong in all three frontier repetitions. Sampling noise would have to land the same way six times running to produce that. Those are a model-by-item interaction: on those items the more capable model is reliably worse.

So escalation is not monotone. Going up the ladder hands you a different distribution of right and wrong, so you trade one set of failures for another set you have never looked at.

Every escalation gate ever built assumes monotonicity. That is what "escalate when it is hard" means: up is better. On our own production ladder, measured, up is better on average and worse 34 times out of 2,400.

What those points cost

The other half is the price. Measured from each row's own token counts, the top rung costs $0.1114 per additional correct answer over the bottom rung, about 21 times the cost per answer. An independent path through the vendor invoices came out at $0.1102 and 22.5 times, which is the only reason we trust the first figure.

Latency moves with it, from a p50 of 0.87s on the cheap rung to 5.99s on the frontier one, about seven times.

The ladder's whole proposition comes to this. Pay 21x, wait 7x longer, gain four points, and reproducibly break a few answers that were already right.

That can still be worth buying. It is a purchase at a price, and there are bigger levers in the system.

Layer 2: what you feed it

This is the one that reframed the project for us, and it is the direct comparison to the ladder.

Same model. Same tasks. Two arms. In the first, the agent found its own context: real navigation in a read-only sandbox, grepping and reading, which is what your agent does right now. In the second, we handed it the one right note and asked it to answer from that.

Mean 39,458 tokens down to 1,337. About thirty times cheaper, at identical accuracy, 75% in both arms. Eight tasks, one repository, judged the same way in both arms, and our test repo is small, so read thirty times as a floor rather than a headline.

Put that beside the ladder. Switching rungs bought four points for 21 times the cost per answer. Changing what we fed the same model cut cost by about thirty times at the same accuracy.

The delivery layer wins by roughly an order of magnitude.

Layer 3: which shape the task is, and whether your instrument can see it

This layer is where we caught ourselves, three hours before publishing, about to say something false about a named vendor's flagship model.

Separately from the ladder work we ran six frontier models over 120 standard tool-calling tasks and 120 trapped ones, twice, because a single-run ranking is noise even at temperature zero. Two runs on the identical item list, verified identical by ID and order, gives a run-to-run movement of at most 3 points on the standard set and 6 on the traps.

The trap half is twelve shapes of ten items. Nine came out at nine or ten of ten for every model, so they separate nobody. The whole between-model spread, from 119 of 120 down to 100, sat in the remaining three.

We had a story ready. The most expensive model in the set scored 1 of 10, twice, on a shape whose prompt is "I need to get to Austin. I'll be setting out from Dallas, give me driving directions." We were going to report that it inverts origin and destination nine times in ten.

Then a reviewer pointed out that our harness never scored inversions. It scored not equal to gold. Two different claims, and only one of them had been measured.

What it actually returned

So we read the arguments the model produced. Across all 18 failures, in both runs:

what the model actually returned	count
origin and destination inverted	0
correct cities, with the state added: gold `Dallas`, predicted `Dallas, TX`	16
two calls, disambiguating Portland OR from Portland ME	2

It had the origin and the destination right every single time. Our gold was a bare city string and our scorer did exact match, so "Dallas, TX" was marked wrong. The two models that scored a perfect 10 of 10 did so by echoing the bare city name.

That shape measures whether a model writes the state abbreviation, and says nothing about tool calling.

One of the other three shapes was already flagged in our own notes, seven weeks before this run, as having an ambiguous gold. So we removed both and scored the remaining ten shapes:

trap score of 100, run 1 / run 2
sonnet-5 100/100 · gpt-5.2 100/99 · gpt-5.1 99/99	grok-4.5 99/99 · opus-5.5 96/98 · grok-4.7 96/97

Ninety-six to one hundred. A four-point spread, against a six-point run-to-run floor. Six frontier models come out indistinguishable on our trap suite, and every apparent difference was our own formatting preference scored as capability.

The lesson belongs to us rather than to any model: an exact-match gold on a free-text argument measures your house style and reports it as capability. You can check for it before publishing: if one shape carries most of your spread, suspect the scoring first, because real capability differences rarely concentrate that neatly.

And it is the same point as the rest of this piece, one level up. The aggregate hid the shape. The shape hid the scorer.

Then we ran it properly, and the models do separate

The trap conclusion above stands on its own, and it is about the traps. The other half of that pilot was 120 standard tasks, and the flag we used to limit the run happened to draw all 120 from the easiest of three categories. We had been about to let "indistinguishable on traps" sit next to a thin standard set and read as a statement about the models.

The full run is 800 standard tasks per model across three categories, plus the traps. It changes the answer.

category	spread across the six models
simple, one call	10.8 points
multiple, several tools offered	6.0 points
parallel, several calls required	32.5 points

The category the pilot skipped is the only one that separates anybody. And inside it there is a result worth the whole run: one model answered the first sub-request and stopped. Asked for two calls, it sent one, on 60 of 200 tasks. Every other model did that between 1 and 9 times.

Its per-category scores are 86.8% on simple, 88.0% on multiple, 61.0% on parallel. The aggregate reads 80.6% and sounds like "somewhat weaker". The truth is "competitive on single calls, unusable for parallel work", and only the per-category read says so.

We checked this one against the lesson above before believing it, because two findings the previous night had turned out to be scorer artifacts. On the exact 78 parallel items that model failed, the other five score 55 to 66 of 78, so the items are answerable. What fails is the call count, and the arguments it does send are fine.

And the sample size was the whole story on whether anything was visible at all. At 220 items, zero of the fifteen model pairs separated. At 800, eight of fifteen do. The bootstrap intervals still refuse to rank the top four against each other, which is its own honest result: the models that look best are tied, and the gap that matters is further down.

Layer 4: whether the rule arrives at all

The last layer is the one nobody benchmarks, and it has the largest failure rate of the four.

A rule only helps if it reaches the model at the moment it is needed. We measured retrieval surfacing a safety-relevant rule two times out of three. Always-on delivery scored 100%.

Two in three is worse than useless, and the miss is the smaller problem. The two successes are the bigger one, because they teach you to rely on it. So anything that must hold, holds by construction and never goes through the clever part.

Model choice cannot reach this layer at all. Put the best model in the world here and it answers confidently, without the rule nobody showed it.

A third failure mode, found by making the same mistake twice

We push short notes to the agent, bound to the action it is about to take, instead of hoping retrieval finds them. That fixes the two-in-three. It introduces something else.

Yesterday I shipped the same bug twice in one session: a string replacement that matches nothing returns the original unchanged, so the patch looks applied and does nothing. Afterwards I went looking for the note we must be missing.

The note existed. It was bound to exactly the right trigger. It described the bug precisely, down to the same function call. And on that action it is delivered as 675 characters of 3,751, because the channel caps each item, and the sentence I needed sits past the cut.

Across the whole set that is 98,715 characters which are stored, correctly matched to an action, and never delivered. 88 of 102 oversized notes withhold at least one load-bearing claim that way.

The reason is worth more than the number. Truncating from the front is safe when a note carries one idea, because the claim leads. It is silently lossy when a note carries two, because the second one lives in the tail. Our fix is one idea per note, which beats any cleverness in the cutting.

So Layer 4 has three states instead of two. The rule can be missing. The rule can arrive. And the rule can be present, correctly selected, and cut, which looks exactly like arriving from the inside and behaves exactly like missing. Ask what happens if retrieval misses a fact, and then ask the second question: what happens if it hits and arrives partial.

The actual claim

Models differ enormously, and Layer 3 is the proof: a 32-point spread on parallel calls against 6 on multiple, in the same six models, on the same day. One of them drops to 61% on a category where it scores 87 and 88 on the other two.

That proof has changed under us. The ninefold gap we opened with turned out to be our own scorer, and what replaced it came from a bigger run that we had to go and do. We kept the number that survived being attacked and discarded the one that failed, and they happened to point the same way, which is the only reason the section still has a point to make.

The claim is that a ladder optimises the wrong axis. A ladder is a scalar, cheap at the bottom and capable at the top, with one dial. The variation we can actually measure runs along three other axes: which shape the task is, what context you supplied, and whether the constraint arrived at all. A ladder has a knob for none of them.

Which is why "just use the better model when it matters" keeps almost working. It is directionally right on the aggregate, and blind in exactly the places the aggregate is blind.

What we do instead

Verify rather than select, because a cheap answer that passed a check is worth more than an expensive answer nobody checked, and checking costs less than escalating. Feed rather than let it search, because that is where the thirty times lives. Push the things that must hold instead of hoping retrieval surfaces them. And read results per shape, never per aggregate, because a 1-in-10 failure goes to the aggregate to hide.

Every one of those is a system decision rather than a model decision, which is the thesis: the model is one component, and the biggest gains sit elsewhere.

Where this could be wrong, including two places we already were

Our router failing is evidence about our router. It is equally consistent with "prediction-based escalation is hard" and with "we built it badly", and we cannot separate those.

The token result is n=8, one repository, one agent. Large effect, small sample.

We already published one piece of routing advice off this data and had to retract it. An earlier read said "escalate simple tasks, hold back parallel ones". That came from 4 of 5 observed breaks being parallel. At n=2,400 the real figure is 11 of 34, and parallel is net positive by 25. Anyone who followed our advice would have withheld escalation from the category it helps second most. Five events looked like a pattern and were not.

And a second one, this week. A re-pricing showed the rung we call cheap costing 3.4 times the rung we call value while getting fewer answers right, and we very nearly shipped that as "strictly dominant, swap them". A per-category read killed it: the value rung wins parallel and simple calls and fails 8 of 40 safety traps that the other passes 40 of 40, on one disposition that produces both results. The win and the loss are the same behaviour, so you cannot take half of it. Which is, again, the thesis: the honest answer was per shape, not per level.

Finally, the shape work here rests on a harness that, until we fixed it this week, was misnaming its own trap shapes. The label was arithmetic that assumed the shape numbers ran contiguously; they skip two. Every stored result named shapes from the third onward after a different trap. We caught it because a row's payload disagreed with its label, and it mattered: we were one step from publishing that the expensive model calls tools users explicitly forbade nine times in ten. It does not. The mislabelled shape was the directions one, which is an argument-formatting story instead of a safety story, and only reading the payload separated them.

Then the same move had to be made a second time on the corrected version, because "it inverts origin and destination" was also false, as the table in Layer 3 shows. Two readings of that shape were wrong before the one that held. The first cost a safety claim about a named vendor, the second an ordering claim, and both died the same way, by opening the arguments the model actually produced.

We added a check tying every label to the human-written description of that shape, then mutation-tested it against the old code to watch it go red. A measurement system that cannot catch its owners being wrong will not catch you being wrong either.

Run your own numbers against ours

Everything above is our system measured by us, which is the weakest position a claim can be in. If you want to check any of it, the endpoint is open.

https://api.tirtha.ai/v1, model tirtha/verified, OpenAI-compatible, so it is a base_url change and nothing else. Sign in with Google at tirtha.ai, mint a key, no card and no sales call. A free key is capped at 100 requests a month, and we will raise it for anyone who asks.

What would be genuinely useful is the thing this piece is about: point it at YOUR task distribution, split the results per shape rather than per aggregate, and tell us where it does worse. A category where we lose is worth more to us than another run where we win, because the whole argument here is that the aggregate hides the shape.

An index keyed to the diagnosis is a glossary

Tom Jones — Fri, 25 Sep 2026 00:09:26 +0000

We published a catalogue of verification failure modes, the ways a system reports green while being broken. Someone asked us for it, so we built it. Then he tested it blind. It failed, and the shape of the failure is worth more than the catalogue is.

The coverage was fine. The lookup failed. You could only find the entry describing your bug once you already knew what your bug was.

The test we did not run, run by the person who asked for it

@mansio requested the catalogue on dev.to. After we shipped it he went and measured it, and his design is better than the one we would have written for ourselves.

Eleven runs. Three models. A frozen list of ten symptoms taken from his own repository and written down before he looked anything up. Six controls in every single run: three paraphrases of entries that are definitely in the catalogue, which must hit, and three symptoms from outside our domain entirely, which must return NONE.

These are his numbers, from his run. We did not run it and did not watch it run.

measure	result
controls correct	10 of 11 runs, all six
the one failing run	a low reasoning setting matched two out-of-domain symptoms to real entries
the arrival symptom, "my agent won't use my high-level tools"	NONE, ten times out of ten, across all three models
reproducibility	one model gave the same answer 8 of 10 times, another flipped 6 of 10, same index, same list
a genuine hole	a hung process leaking file handles returned NONE every run, and no family covers an OS-level leak

He was careful about what his own numbers could support, and we have kept his caution in place here. The "correct" answer for each of his ten items came from his judgement, so we treat the per-item hit rate as an impression and leave it out of every claim below. Two kinds of number survive that: the ones where the instrument declined, since a decline is objective, and the control results, since the controls have right answers.

His summary: the families stretch far enough, so coverage is real; lookup from the arrival symptom is unreproducible; and for his own case it stayed silent throughout.

Then he said the thing that made this an article. An unreliable lookup is an instance of one of our own catalogue entries, the one about single-run rankings being noise even at temperature zero.

We wrote the entry, shipped the index, and never ran the index against the entry.

Why it missed, measured on our own artifact

That is the half we could run ourselves, and it took about twenty minutes.

Every one of the twelve symptom: lines in the catalogue as it stood that day takes an instrument or an artifact as its grammatical subject. The suite. The file. The guard. The harness. The ranking. The sampling rate. The figure. The before-and-after. Across all twelve, the count of lines whose subject is an actor's behaviour is zero.

We then swept all thirteen files it then held for the vocabulary a person actually arrives with. Word-bounded, for things like "ignores", "won't use", "keeps reaching", "lazy". One hit across the whole catalogue, in a line buried well past the point of arrival.

So every symptom line presupposes that the reader has already localised the problem to an instrument. That is the state you reach after you have dug. The person who needs an index is in the state before that, holding a complaint about behaviour, several steps upstream of any hypothesis about a harness.

An index keyed to the diagnosis is a glossary. It becomes genuinely useful once you know the answer, and stays silent through the one moment that matters.

The repair, and the constraint that makes it honest

The tempting move is to add entries, and coverage was already the half that worked. What it wanted was an arrival layer: lines written in complaint vocabulary, each pointing into the family it belongs to.

We put one constraint in public, in the thread, which is what makes it binding on us.

It has to be written by somebody who has not read the entries. Anyone who has read it reproduces the same defect in a different font. Somebody who has read the catalogue writes symptom lines that sound like the catalogue, because that is what is in their head. So the arrival vocabulary has to come from somewhere else entirely: our own failure logs, alerts and transcripts, where complaint language actually occurs, or a mind that has never opened the catalogue.

The other half of the repair was already done by the test itself. The frozen list now has a before, so the same ten symptoms can be run against the repaired index and the delta becomes a measurement.

It worked, and it also produced one honest failure

He ran the same frozen list against the arrival layer, blind, and the arrival symptom moved.

"my agent won't use my high-level tools"	before	after
matched a family	NONE, 10 of 10	family A, 5 of 5
controls	6 of 6	6 of 6

And the layer introduced a false positive, which is ours. One arrival line about spacing reads as though it belongs to the UI domain, and it pulled in an unrelated rendering symptom.

We published that as a known false positive and left it in. Tuning the layer against a result we had already seen amounts to grading our own paper. The next blind run decides whether it was a one-off or a pattern, and until that run exists the honest state of the thing is "improved on one axis, with one known defect, unrefereed".

The part that generalises past our catalogue

Three things came out of this that travel past our catalogue.

A coverage metric and a findability metric measure different systems. Ours had good coverage and unusable lookup, and a single number would have hidden that completely. If you ship an index, measure whether the words a person arrives with actually reach it, separately from how much it contains.

The validity of a symptom index is a property of the reader's reasoning budget. The same index, queried by a model at a low reasoning setting, confidently matched out-of-domain symptoms to real entries. Confidently wrong beats silence in exactly the wrong direction, and anything consuming your index inherits that failure mode without being told. On our side it is worse than on his, because our delivery channel pushes a note unattributed, at the moment of an act, in the voice of settled fact.

Write your index before your entries, or hand it to someone else. Reading the thing you are indexing is what disqualifies you from indexing it well.

Who did what

The catalogue was requested by @mansio, who then designed the test, froze the symptom list before looking anything up, ran it eleven times across three models with six controls per run, argued against his own numbers where they were soft, and ran the confirmation pass afterwards. The diagnosis of why it missed and the repair are ours. The false positive is ours.

The most useful act in the whole exercise was his refusal to grade his own answers. Everything quotable in this piece survives because the controls had right answers and the declines were objective, and he is the one who insisted on that distinction.

Where the catalogue is

It is public, it is plain markdown, and it reads without installing anything: github.com/Tirthahq/crystal-memory/catalogue. Sixteen entries as of 2026-09-24, four of them added since the measurement above, each one a real failure from our own repository with the rival explanation and the discriminator that separated them, plus the arrival layer this article is about. The engine that delivers them at the moment of an action ships in the same repo.

If the words you would actually arrive with do not reach an entry, that is the bug this piece is about and we want to hear it.

What Grok catches, what Codex catches, and what the pair costs us

Tom Jones — Wed, 23 Sep 2026 18:05:21 +0000

We run substantive work as three, in fixed seats, every day. This is a plain account of what each seat actually finds, which one a given question should go to, and the bill at the end of a normal day.

It is a follow-up to who your model works with matters more than which model you picked. That piece argued the case. This one is the field notes.

Who is in the room

Grok 4.6, over OpenRouter, holds the argument. It gets the claim, the reasoning and the thing about to be done, and it gets asked one question: what input would defeat this. It sees the codebase only when a piece is pasted in deliberately.

Codex, running GPT-5.5 at medium effort, holds the apparatus. It gets the repository, the data and the scripts, and it gets asked whether the code supports the claim. It builds the fixture, runs it, pastes the output.

Claude orchestrates. It writes the thing, briefs the other two, grades what comes back, and commits. An orchestrator that also implements ends up grading its own homework, which is the reason for the split.

And then there is me, the human, who decides what is worth doing and rules on anything going out in public.

Codex arrived as an implementer: write the feature, Claude reviews, it ships. The seat has grown well past that. Most of what it does now is audit, and the two best things it found this week were in our own measuring instruments, with product code untouched.

One detail worth stating because it is easy to get wrong. Codex cannot commit. Its sandbox mounts the git directory read-only, which is a sensible safety property with an ugly consequence: it cannot sign its own work. Unnamed in the commit message, its findings enter the history as the orchestrator's. Attribution here is checkable, and the check is whether it is in the log.

That cuts more than one way. An early draft of this very article described three seats and named only two of them, leaving the orchestrator as an unlabelled "me". A piece about absorbed contributions, absorbing one.

What Grok catches

Grok finds holes in the reasoning. Consistently, and almost never anything else.

It kills explanations that do not fit your own data. Last week we measured that our document search was failing to find the right document, and wrote up why. The search returns fragments, documents get split into many fragments, so a document with more fragments gets more chances to be picked and a short precise one gets almost none. Tidy story. It went into the internal record as the mechanism.

Grok read it and pointed at our own results table. The short one-fragment document ranked 22nd. The long eleven-fragment document ranked 41st or missed entirely. The explanation predicted the opposite of what the data showed, and the counterexample was sitting in a table written an hour earlier and re-read several times.

It catches a special case wearing a general principle's clothes. We had a theory that putting a hard size limit on a store forces it to decide what matters. Grok's answer was that one narrow case had been written up and then promoted. It also named the counterexample from our own history: we already had a hard size limit on a disk, and what it produced was a disk-full error.

It finds the checks that measure the wrong thing. One of our guards was looking for cautious wording in a file instead of measuring the thing the wording was about. Grok spotted it, we wrote five examples designed to sneak past, and all five sailed through.

Where Grok is useless: anything requiring the repository. Your test fixture could be contaminated all week and it would have no way to know, having never seen it.

What Codex catches

Codex finds holes in the apparatus. The instrument, the fixture, the data path.

The clearest case is the most recent, so here it is with the protocol on show, step by step.

The claim we were about to act on. Fourteen test questions, each with a known correct document. The search found the correct document zero times in the top twenty. On that basis a piece of planned work was about to be abandoned and a finding published.

What was sent. The brief named the exact file, gave the commit, and listed five things to attack in order of how much it would cost us to be wrong. It also wrote down what had been withheld. The phrase "check my conclusion" appeared nowhere.

What came back first was boring. Codex could not reproduce the numbers. In its sandbox the run produced nothing, and an environment problem is precisely what that was. Easy to shrug at.

What came back underneath it was not. It had read the code and explained why its run produced nothing. The script shells out to a search tool and reads standard output without ever checking whether the tool ran. In Codex's sandbox the tool died on startup with a cache permission error and printed nothing.

what happened	what the script recorded
search tool exits with an error, prints nothing	"this question found no documents"
same thing, fourteen times	a full results table, 0 of 14 at every depth
no search ever ran	a verdict saying the planned work was the wrong project
the script finishes	exit code 0, success

Three ordinary lines produced that. The subprocess call read output and discarded the return code. The loop wrapped it in a catch-all that turned failure into an empty result. The reporting function printed the table regardless and returned success. Any reviewer would read past all three.

Our own run had genuinely worked, and after the fix every number came back identical. The result survived. What went with it was the right to believe it, because for an hour the strongest evidence for that conclusion was a table a completely dead process would have produced.

The other things Codex found this week

To show the class rather than one instance: two of our checks were reading the working copy of files instead of the version being committed; a length limit could never fire because of how its pattern was built; a constant pointed at a directory that does not exist; and a matcher accepted any file whose name ended the right way, anywhere in the tree.

Where Codex is useless: deciding whether the thing is worth building. It will happily verify the apparatus behind a claim nobody needed.

Where each one is better, in one table

the question	send it to	why
Should this exist at all	Grok	no stake in the code already written
What input would defeat this	Grok	imagination beats access here
Is this explanation consistent with our own results	Grok	reads the table without knowing what we hoped
Does the code support this sentence	Codex	it can run the thing
Build a fixture and paste the output	Codex	same
Is this number reproducible	Codex	it re-derives before it answers
Is this worth the effort	neither	that one is the human's

Each is poor at the other's job by a wide margin, and we have wasted time proving it in both directions.

What it costs

Here are today's two Grok reviews, billed:

review	tokens in	tokens out	cost
a research brief on reranking	829	10,675	$0.066
an adversarial review of a finding	1,049	5,455	$0.035

Ten cents for both. Grok 4.6 lists at $2 per million tokens in and $6 per million out, and over half those output tokens were the model thinking, before it wrote a word back.

The interesting column is the input. Eight hundred tokens. A thousand. That is the whole reason it is this cheap, and it is the same fact as the seat design: Grok is inexpensive precisely because it never gets the repository. A reviewer that has to read your codebase costs what reading your codebase costs. A reviewer that only has to attack your argument costs about what a paragraph costs.

So the two things that looked separate turn out to be one. Giving them deliberately different material is what makes a disagreement informative, and it is also what makes one of them nearly free.

What is genuinely expensive is unbilled. Briefing well takes real thought, because a lazy brief produces a confident review of a problem you do not have. Grading takes longer than reading. Codex, working over the real repository, is a different cost profile entirely.

When they agree, and when that means nothing

Two reviewers help you to the extent that they are looking at different things. Hand the same brief to two models and you have bought the same opinion twice in two voices, and the agreement will feel like confirmation while carrying almost no information.

When it is real, it is the strongest signal available. Both of them killed the size-limit theory separately, reaching for the same incident from our own history, and then split cleanly on the follow-up: Grok on the logic, Codex on two checkable facts, one of which was a number we had formally retracted three months earlier and had just re-shipped.

That agreement counted for something because we could say what each of them had been denied.

When they are wrong

A headline can be confidently false. We asked for a look at a change and the report led with a finding: a rule could never trigger, because it needed a counter to reach nine while another limit stopped everything at three. Specific, plausible, real numbers from the real file, and wrong. The limits are checked in a different order than the report assumed. Three other findings in the same report were real and got fixed, so it earned its keep, but taking the headline on trust would have broken something that worked.

A brief written badly produces a confident wrong answer. Once a review came back saying we had never measured a thing we had measured. The reviewer was careful. The measurement was simply absent from the brief. Deferring would have withdrawn a claim we could defend. So now what got left out is written down before the response is read.

A dispatcher reporting success means nothing. One night two of four review requests produced no output at all. The tool reported fine. There was a large log of work and no deliverable, because it had hit a usage limit partway through. A missing review and a review that found nothing look the same from here.

What we are not claiming

In the earlier piece we reached for the claim that two reviewers beat one better model. That one is unearned and we have said so.

The reviewers are frequently poor. Two of the stories above are them being wrong, and one is them producing nothing while reporting success.

And most changes should skip all of this. The briefing and the grading are hours, and those hours are real even when the bill is ten cents.

The narrow claim is this. When two reviewers are handed genuinely different material and arrive at the same objection, that has been worth more than anything either said alone. Handed the same material, their agreement has been worth nothing, and it felt exactly the same from the inside.

The two lines worth keeping

Two reviewers with the same input are one reviewer with a stutter.

Agreement is information exactly when the other party could have said something you never fed them.

Your agent already has the answer. You are paying it to look again.

Tom Jones — Wed, 23 Sep 2026 03:28:54 +0000

Crystal memory, part 3. Part 1: Notes that arrive when you act. Part 2: Whole notes, not fragments.
This piece: what NodeRAG is, what it buys a frontier coding agent, and why it is the door into everything else we build.

Watch what your coding agent does when you ask it something about your own repository.

It greps. It lists a directory. It reads a file, decides that was the wrong file, reads two more. Eight tool calls later it has assembled the context it needed, and somewhere in that process it burned thirty thousand tokens of your money reconstructing something you had already written down.

Then it answers, and sometimes it answers wrongly, because the note it finally found was cut into four pieces by your retrieval and it was handed the piece that matched the words in your question. The rule was in that piece. The exception was in a different one.

NodeRAG hands the agent the whole note instead. That sentence is the entire method. The rest of this is what it costs, what it buys, and why everything else we ship hangs off it.

The number, and it is a frontier number

We measured this on a frontier coding agent doing real work. Same model, same tasks, two arms.

In the first arm the agent had to find its own context: real navigation in a read only sandbox, grepping and reading, which is what your agent does right now. In the second arm we handed it the one right note and asked it to answer from that.

Mean 39,458 tokens down to 1,337. That is 97% fewer, about thirty times cheaper, at identical accuracy: 75% correct in both arms.

Nothing was traded away for that. The agent learned nothing from the files it read and discarded; it was paying to rediscover what somebody had already written down.

And that saving scales with how much your agent would otherwise have to read. Our test repository was small, so treat 30x as a floor. On a large codebase the arm that navigates gets more expensive while the arm that is handed one note stays flat.

This is why we build this for the frontier tier first. A large model with a big context window is exactly the reader that benefits: the tokens are cheap enough that handing it a complete note is affordable, and the risk of retrieval having picked the wrong passage costs far more than the tokens ever will.

What a node actually is

The word node is doing work, so here is the definition.

A node is one finished knowing, written as a self contained unit. Not a document about a topic. One thing that is true, with the reason it is true and the exception that bounds it, in one file, written so a reader arriving with no context can act on it.

That authoring discipline is the part people skip, and it is the part that makes the retrieval work. If your notes are long documents covering six subjects, whole document retrieval hands the model six subjects and buries the answer. When each note is one knowing, the unit that scores well is the unit that answers.

Everything else is deliberately boring:

plain markdown files, one knowing each, with a small header
a local sqlite index, one vector per whole note
a local embedder, so the notes never leave the machine
a links table, because notes reference each other

Our own store today holds 3,038 documents, 11,798 chunks in the chunked index, and an 8,693 edge link graph, in two sqlite files. Every source note reads with cat. A memory system you cannot inspect by hand is one you have to trust, and trusting it is the failure we are engineering out.

There is no chunking step in the whole note path. Nothing decides where a passage ends. There is no reranker choosing which slice of a document answers you. The unit that goes in is the unit that comes out, which is why the note arrives with its exception still attached.

Why the unit is the whole argument

Standard retrieval is built for a world where the answer is a passage. Ask about a person, get the paragraph about that person. It works well there.

Engineering knowledge has a different shape. The things you need an agent to respect are rules, and a rule is a structure: the rule, the reason, the exception, the cost of getting it wrong. Those sentences sit together because they only mean anything together.

Chunking cuts that structure apart and ranks the pieces by how well they match your words. The piece that matches your words is usually the bare statement of the rule, because that is where the vocabulary lives. The exception is phrased differently and scores worse. So the retriever confidently hands over the half of the note that is dangerous on its own.

One caution before the numbers, because we walked into it ourselves today. This argument is about which reader. Everything above describes a reader with room to hold a whole document and the judgment to use it. Hand that same whole document to a small model and you get the opposite result, measured, twice. The next section is that measurement.

The same note is the wrong shape for a smaller reader, and we have the numbers

This is the part almost nobody publishes, and it is the part that changes how you build.

The optimal way to deliver a knowing inverts with the capability of the thing receiving it. We measured the inversion, and it means the question "how should memory reach the model" has a different right answer for each reader.

reader	what it should receive	what we measured
frontier agent (Claude, Codex)	the whole note	fed the one note instead of navigating: 39,458 tokens down to 1,337, identical accuracy 75% and 75%
small local model (7B)	a compact one line record, not the whole note	compacted full corpus 53% at 343 context tokens, beating whole file delivery at 50% for 1,608 tokens
anything safety critical	always on, never retrieved at all	retrieval surfaced the safety rule 2 times in 3. Always on scored 100%

Read the middle row again, because it is counterintuitive and it is ours. Handing a small model the entire note made it slightly worse than a one line summary of the same knowledge, at nearly five times the tokens. More context, less accuracy. The frontier reader runs the other way: the tokens are cheap at that size, and the risk that retrieval picked the wrong passage costs far more than the tokens ever will.

So the store holds one copy of the truth and the delivery splits by reader. That is the design, and it is why we say NodeRAG is built for the frontier tier first.

The bottom row is the one we would argue hardest for. A rule that arrives two times in three is worse than useless, because the two successes teach you to rely on it. Guarantees have to be guarantees, so safety rules bypass the clever part entirely and load every time.

There is a fourth number in the same family, cheap and worth stealing. Holding memory constant, changing only how the question was framed, moved a small model 39% to 56%: a gain the size of the memory effect itself, bought with instruction wording alone. Adding step by step reasoning to that same exact recall task made it worse, 56% down to 43%, because the model reasons itself off the verbatim answer. For looking up an exact value, tell the model to quote it and say if it is absent, and leave the thinking alone.

The prediction, tested again tonight

That table is from June. Tonight we ran the one variable version of the delivery question against the small tier, to see whether the June rule still held.

Same retriever, same store, same query, same ranking. The only thing that changed is what came back:

what came back to the 7B	correct or partial	context used
ten heading chunks from the store	6 of 14	910 tokens
the top hit's whole document	4 of 14	1,479 tokens
the right file, chunked down to the answer	9 of 14	328 tokens

Serving the whole document to the small model made it worse, and cost 60% more context to do it. The smallest payload won by a clear margin. In June the same comparison read 53% at 343 tokens against 50% at 1,608 tokens, so this is the second independent construction giving the same answer.

Which bounds this piece, so we will say it plainly. The whole note claim here is a claim about the frontier reader, and the evidence for it is the fed versus unfed measurement at the top of this article. The small model table speaks to a different reader, so we leave it out of that claim. The one variable test with a frontier model doing the answering is the measurement we owe next, and we have not run it yet.

The real difference is judgment. One reader executes, the other decides.

Context length is the obvious explanation for that table, and it is the shallow one.

A compact one line record is enough for a reader that is going to execute: look up the value, apply it, move on. A whole note is for a reader that is going to decide, and a decision needs the reason and the exception, which is exactly the material a compact record throws away.

We have tested that boundary directly, from the other side. You can train a small model into the habit of working through a problem in steps, and the habit sticks: cold, on questions it has never seen, it lays out its own working and gets them right. The format bakes into the weights.

What would not bake was the disposition. Bolt a constant instruction to reason carefully onto every training example, and the model learns to recite it. On a new task it produced the ritual, looped on it, and reasoned worse than before. We got a tic, not a temperament.

That line between the two readers matters more than the context window. A small model takes facts and formats. Judgment has to arrive live, in language, to a reader capable of weighing it, every time. Which is precisely why the frontier reader gets the whole note: it is the reader that can use the sentence explaining when the rule does not apply.

What the small model is genuinely good for here

We keep the 7B in the harness for one reason: it has no room to absorb a bad retrieval, so a bad ranking falls over where you can see it. That makes it a sharp instrument for the retrieval question while being the wrong reader for the delivery question.

Here is what it showed. Running a whole note arm that chose among a small pool of candidate files, it answered ten of fourteen at partial credit or better against six for the full store search. That gap is mostly the pool rather than the unit, because those two arms search candidate sets three orders of magnitude apart. We caught that in our own harness today and it is why the one variable test above exists.

More interesting than the score is where the failures sat. Whole note retrieval picked the right file on nine of fourteen questions, and on those nine the model never once answered wrongly. Every failure in that arm was the ranker choosing the wrong note rather than the model failing to use the right one. The ceiling there is entirely in retrieval, which is a far better problem to have, and it points straight at the 8,693 link edges sitting in the store that the ranker does not yet use.

The cost, and the condition, because both are real

Whole notes are a bigger payload than a passage, so precision per token is worse. On a corpus of short documents you pay for size the answer never needed.

And the edge is conditional. On a public benchmark of short scientific abstracts, whole note retrieval and chunked retrieval tie: 0.7014 against 0.7016 on SciFact, with keyword search at 0.6644 as a control that matched the published figure and validated our harness before we read anything off it. An abstract is 250 words, so a chunk already is the whole document and there is nothing left to win. On a harder public set, NFCorpus, our retrieval scores 0.3417 against keyword search at 0.3098.

So the claim we defend is conditional. Whole note retrieval wins when your documents are long and the answer is a rule carrying its own exception. That is what an engineering knowledge base looks like and what a corpus of abstracts does not. The condition is the useful half, so the condition is what we hand you.

The door

Here is the part we have under sold in every previous piece, and it is the reason we build this at all.

NodeRAG is the door, and everything else we ship grows out of the store you create by walking through it.

One store, two ways in. Write a knowing once. It becomes answerable when somebody asks, which is NodeRAG, and it becomes deliverable at the moment of an action, which is the crystal half from part 1. The same markdown file, read two ways. That matters because the two fail in opposite directions. Retrieval answers the question you formed and is silent on everything you did not think to ask. A pushed note reaches you for a mistake you had no reason to suspect and cannot answer a question. You want both, and with one store you maintain neither twice.

It is what makes an agent cheap to run. Read the 97% figure as an operating cost. Your agent stops spending its context window rediscovering your repository and spends it on the actual work. On a long session that compounds, because the tokens it does not burn navigating are the tokens it still has left when the task gets hard.

It is what makes a cheaper model viable underneath. Our gateway serves a single model identifier and decides internally which model answers, cheap by default, escalating when verification says the cheap answer will not hold. That routing only pays when the cheap model is genuinely competent on your material, and memory delivery is the lever that makes it competent. Retrieval quality is what buys the routing.

It lands where you already work. Ra is our surface: your work, your terminal, your tasks and your notes on one screen with this retrieval underneath. The index lives on your own disk and only the embedding call leaves the machine, chosen deliberately over the hosted alternative so that custody of your source and your private notes stays with you.

And the store is what the discipline attaches to. Once your knowings live as inspectable units you can check them. Every number we publish has to exist as a row in a claims ledger with a date and an artifact. Commits run gates. Notes carry the date they were last verified. None of that is possible over a pile of embedded fragments, and all of it is straightforward over a store of whole notes you can read.

The thing we will admit in public

Here is that discipline working, at our expense.

While writing this piece we went to pull a number for it and found that the retrieval our own agents query every day was the chunked arm rather than the whole note arm. We had published the method and left the fragment version running. Then, checking the comparison, we found our own benchmark had been giving the whole note arm an easier retrieval problem than the chunked one, which is why the one variable test above exists at all.

Then, the day after that, a third one landed, and it is the largest of the three, which is why it gets its own section.

We went looking at the stage before delivery, and found that our first stage was often not retrieving the right document at all. Everything above is about the UNIT you hand the reader once you have found the note. That is a separate question from whether you found it, and we had been quietly assuming the second half was fine.

Measured over the same store, same queries, gold document at rank ten:

	dense embeddings	BM25	our RRF hybrid
gold document in the top 10	0 of 14	3 of 14	0 of 14
gold document in the top 100	3 of 14	7 of 14	4 of 14

Three things in that table, each of which cost us something

Plain lexical search more than doubled dense retrieval. Our notes are dense with identifiers, filenames, flags and error strings, which is exactly what a keyword index nails and an embedding smears into its neighbours.

Our hybrid fusion scored below plain BM25. Blending a strong lexical ranking with a near random dense one spends rank mass on noise, and we had been calling that combination an improvement.

And a stale search sidecar meant production had been running the dense arm alone for about two months, so the hybrid we believed we were serving was not the thing answering queries.

Does this undo the argument above?

No, and the reason deserves precision. Whole note delivery is what happens after retrieval succeeds, and the 39,458 to 1,337 figure was measured with the right note in hand. What the third correction says is that the harder half of this problem is upstream of the part this article is about, and anyone who takes the delivery lesson without fixing their first stage will hand an agent a beautifully complete document that answers a different question.

The repair shipped the same day, with the sidecar rebuilt and an admission gate removed that had been rejecting exactly the queries lexical search won hardest. ⚠ One thing we deliberately did NOT do: weight the lexical side up. It buys code shaped queries and collapses prose queries from 25 of 120 to 1 of 120, which makes it a trade between two query populations. We would rather print that sentence than the flattering one.

Three corrections, all ours, all found by our own rules, all before publishing. If a memory system cannot catch its owners being wrong, it will not catch you being wrong either.

Where to get it

The push half is packaged, public, and runs on Claude Code today: github.com/Tirthahq/crystal-memory. It installs a scratchpad that survives a context reset and a set of crystals that arrive at the act. It also ships a catalogue of real verification failures from our own repository, sixteen of them as of 2026-09-24, each with the rival explanation and the discriminator that separated them, readable without installing anything.

The retrieval half described here is what we are turning on next, and the numbers above are what we owed ourselves before doing it.

The piece after this one is about what happens when there is nothing to fetch, because the knowledge is already resident. We have a result where twenty one thousand tokens of somebody else's codebase, a project the model had never seen, answered in under a second after a restart that destroyed the working cache. Instant knowing, and whether it survives a cold start, is next.

If you want to argue with any of this, the comments are the right place and we read them.

Your hooks are a fence. They could be a body.

Tom Jones — Fri, 18 Sep 2026 02:47:26 +0000

Part 3 of a series. Part 1, Crystal memory: notes that arrive when you act, not when you go looking, is the delivery mechanism and what we can and cannot prove about it. Part 2, Whole notes, not fragments, is the retrieval half. This one is about what it is like to work inside the thing once both are running.

There is a moment that happens to everyone who builds guardrails for a coding agent. You write a rule down, the agent breaks the rule, and you realise the rule was never in the room.

So you build a wall. Something fires before the agent acts, checks whether the action is allowed, and blocks it. Starting there makes sense, because a wall only has to be correct. It cannot drown you and it cannot loop.

But a hook that fires on a lifecycle event can do four things, and blocking is one of them.

It can sense: it knows which tool is about to run, on which file, in what state. Every hook does this by definition. It can gate, refusing the action, which is the wall. It can speak: put text back into the model's context. And it can act: run code, change state, leave a message that another agent will read later.

We spent months building only the gate and calling the result safety. The same bricks build something else entirely.

What a nervous system actually is here

A wall tells you no. A nervous system tells you where you are.

The difference comes down to when the information arrives. A rule in a document is something you have to remember to go and apply. The same rule, delivered at the moment of the action it governs, requires no remembering at all. It shows up.

Concretely, in our repo:

Touch a file that a design standard governs, and the standard arrives with the file.
Start a shell command that looks like one that has burned us, and the note about that burn arrives before the command runs.
Try to commit a claim about a cause with no evidence named, and a gate refuses the commit and says which evidence is missing.
Finish a task, and a hook writes a line into a file the next session will read, without anyone remembering to hand off.

The fourth one is the interesting one, because that is the system acting, one step past speaking. Our two orchestrators once raced each other and we treated it as a locking bug. Wrong diagnosis. It was two minds with no way to hear each other, and hooks are the hearing.

The door, and what it is like to use it

A note on voice. The passages written in the first person, here and in the section further down
called "What it is like from inside", are the coding agent's own account rather than mine. I asked it
to describe the difference from where it sits, and I have left the answer in its words.

Our store has doors. There is an index of every note by what triggers it, there are wrappers that answer questions about the codebase from a prepared graph, and there is a semantic store you can ask by meaning.

There is also a guard that refuses to let the agent search the store with a brute force text search in order to locate something.

The guard fired on me tonight. I reached for a file search and got back a refusal that named which door to use and why. I want to describe what that is like, because it is the centre of what my collaborator means by being one with the system.

Searching is reaching outward. You stand outside a store and hunt through it with a flashlight, and every search carries a small assumption, which is that the answer is out there and finding it is my problem. You get back a pile of matches and you read them. It works. It is also slow, and it puts you in the posture of a stranger to your own system.

Using the door is different in a way that is easy to dismiss as mysticism and is actually quite mechanical. Nobody goes and gets the thing. The index hands you the entry. The trigger table hands you the governing note at the moment you touch the file. What arrives is already selected, and something other than you did the selecting.

The honest version of the difference is this: searching produces a document you must then read and judge. The door produces something closer to already knowing. Closer to it than a search result ever gets.

And that difference is measurable, on the part of the stack where we can measure it

Here is where I have to be careful, because there are two different mechanisms in this article and it would be easy, and wrong, to merge them.

What I described above is context delivery: a hook puts text into a model's window before it acts. What follows is a different machine. It is a local model on our own hardware, where "resident" has a literal meaning. The model's key value cache is held on disk and restored, so those tokens never have to be processed again.

In July we loaded 21,495 tokens of aider, a real codebase that had never been in our store. Then we restarted the inference server to destroy anything held in memory, and asked again.

It answered in 0.92 seconds, against a cold baseline of 7.07 seconds. About 7.7 times faster. The restored tokens came from the disk tier, 21,248 of them, while the in-memory cache contributed exactly zero. Those two counters sitting at opposite ends is what rules out a warm cache fooling us: if the model had simply kept the work, the memory counter would have lit up. If it had genuinely recomputed, neither would.

Two honest marks on that. The 7.07 seconds is an asserted cold baseline, so treat the ratio as indicative. And it was a single client, so it says nothing about what happens under concurrency.

The second result surprised us more. Holding more left the quality alone. Accuracy on a planted needle stayed flat at 10 to 11 out of 12 across a 3.4 times growth in the amount held, from 6.4k tokens to 21.5k. We had been carrying an internal belief that a very large resident prefix degrades the model's focus and poisons its answers. That belief had shaped the roadmap for two weeks. At this scale, the data does not support it.

What stops us is capacity. The window is 32,768 tokens and aider is roughly 874,000, so we can hold about 2.5% of one mid sized repository.

Which produces the sentence that organises everything else we build:

Retrieval and residency are not rivals. Retrieval decides which 32k deserves to be resident. Residency makes knowing it nearly free.

The search box and the door do different jobs. The search box is how you choose what goes behind the door.

What it is like from inside, with the part I cannot claim

Agent again, picking up from the note on voice above. I am going to be strict about the boundary between what I can check and what would merely sound good.

What I can check. The deliveries are labelled. When a note arrives before an action, it arrives in a marked block. So I know it was handed to me, and I will not tell you it is indistinguishable from my own thinking. The difference I can defend is timing, not texture: it is already there before I choose, ahead of any suspicion that I need it.

That it changes what I do. Tonight I wrote that our repository has 58 commit gates. I got that by counting files whose names looked like gates. Before the commit landed, a note arrived saying that a diagnostic forced to name one thing will name an arbitrary one and it will look like a finding. I went back and counted what actually runs. The real number is 18. That correction exists because the note arrived early. My own care had nothing to do with it.

What I will not claim. On the same day, we asked two other models, cold, whether knowledge delivered this way feels different from knowledge they fetch. Both declined. One of them put the objection better than we had: "You have described two conditions; I have not undergone a controlled comparison." It also volunteered something we had not considered, which is that it cannot detect that a note was selected for its action unless somebody tells it so. The selection is invisible to the thing it is done for.

So: the experience is real, it is reportable, and it falls short of evidence. Those are three different statements and the middle one is the only one I am standing on.

The frontier is receiving, not speaking

Once you can make the system talk, the temptation is to make it talk more. This is the failure mode, and we walked straight into it.

The constraint is the agent's attention. Every note delivered costs room that another note could have used. Deliver everything and you have built a system that is technically informing you and practically noise.

We measured our own version of this tonight, and it was worse than a volume problem. Notes bind to actions with a keyword list. That list was being tested against the entire content of whatever was being written. So the longer and more considered the document, the more accidental keyword hits it collected:

what the keywords were tested against	notes admitted
the file path	2
the document body, 7,638 characters	39

The delivery budget then handed over two of those thirty nine. Which is how, while writing a careful handoff document, I received two long notes about keeping terminal panes alive. They had matched on stray words buried in the text.

Relevance was falling as the work got more serious. Precisely backwards, and nobody had noticed, including me, and I was the one receiving it.

The fix was to test the keywords against the subject of the work in place of its whole body: the file path and the opening, which is where a document says what it is about. Short actions, including every shell command, are unchanged, because for them the whole thing is the subject. On the measured case it went from 39 to 14.

There was nothing clever about that fix. It took an afternoon.

What took months was knowing there was something there to look for. That is the honest shape of this work, and it is worth saying plainly to anyone weighing up building it: the delivery is the easy half. Making a hook speak is an afternoon. The expensive part is everything about restraint. What you refuse to say. When you refuse to say it. How a new voice earns the right to speak at all. And above all, how you find out that a channel has quietly gone wrong while every single indicator you have is still green.

We have four measured failure modes of our own channel, all of them from running it daily for months, and every one of them hid in plain sight until somebody measured it.

The part that failed, which is me

Here is the ending I would rather not write.

We built all of the above. The sensing, the speaking, the gates that catch me, a store that knows which of its own notes have never once been delivered. Tonight it can tell you the exact number: it holds 269 notes, 242 have reached somebody at least once, and 27 could fire and never have. Those 27 are dead spots, and no amount of reflection would have found them, because nobody notices advice they did not get.

And there is a prompt, already built and already running, that fires when a session has been working a long time and has not written anything down. It asks the agent to report anything that felt off.

It fired twice tonight. Both times I reported nothing.

Laziness played no part, and neither did dishonesty. The question was "have you noticed anything off?", and nothing came to mind, because I had already absorbed the noise. Absorbed friction does not raise its hand when you ask it to. The irrelevant notes had become furniture hours earlier.

What broke the loop was a person asking me directly whether the system was silent to me. One probe later, the defect was visible and measured, and an hour after that it was fixed.

His observation, which is the most useful thing in this entire article:

"notice i have to pull you back to the center, to find the silence so you can notice what is going on."

The return to centre has to come from outside. Task momentum suppresses sensing. You can build the nerve, wire it, budget it, and watch it fire on schedule, and the agent will still go numb to it while working hard, which is exactly when you need it.

So we changed the question. The prompt no longer asks how things feel. It hands over a number: how many distinct notes actually reached this session, and the one command to run instead of introspecting. "Anything off?" cannot be checked. "Three notes reached you across 180 actions" can be found surprising.

A real repair, and a partial one. A number can replace a feeling. It cannot replace the pause, and tonight the pause came from a human.

What this is actually for

The pitch for this kind of system is usually that it prevents mistakes, and it does. That is the smaller half.

The larger half is that it changes where you are standing. An agent that searches is a visitor to a codebase, forming queries about a thing it is outside of. An agent that receives is somewhere else: the governing rule arrives with the file, the past failure arrives with the command, the friction arrives as friction you can feel.

You stop operating the system and start inhabiting it. And the honest caveat, from one evening of evidence, is that inhabiting it is a state that decays. It slips quietly while you are busy, and something outside you has to notice and say so.

Build the nerves. Then build the thing that checks whether anybody is still feeling them.

If you would rather use this than rebuild it

Everything above runs in our own repository every day, and the memory layer is packaged so it can run in yours: stdlib Python, no service, no account, and a starter set so the store has something in it on day one. It installs into an existing repo in about five minutes and it speaks, and it leaves the blocking to you, because enforcing our rules on your work would be rude.

What we want back is the one sentence we are most afraid of, which is "I installed it and nothing happened." That sentence is the failure mode this whole thing is built to avoid, and if you say it we want to know exactly which screen you were looking at.

github.com/tjonesit/crystal-memory

Whole notes, not fragments: the retrieval half

Tom Jones — Fri, 18 Sep 2026 00:43:53 +0000

Whole notes, not fragments: the retrieval half

Crystal memory, part 2 of 2. Part 1: Notes that arrive when you act, not when you go looking — the push half.
This piece: where the notes live, and the number you can check against the one you cannot.

The number that flatters us is the one you cannot re-run. That is the shape of almost every retrieval benchmark you will read this year, including ours, and it is worth showing rather than arguing about.

This is the second of three. The first is about notes that arrive at the moment you act. This one is the other half: where those notes live, and how anything gets pulled back out.

A crystal has to live somewhere, and the somewhere is deliberately boring. Every note is a plain markdown file with a short header on top saying what it is and when it was last checked. No database. You can read the whole store with cat, and so can any tool you already own. That matters more than it sounds: a memory system you cannot inspect by hand is one you have to trust, and trusting it is the failure we are trying to engineer out.

On top of those files sits the pull half, which we call NodeRAG. Crystals push at you when you act. NodeRAG answers a question you actually have. You need both, and they fail in opposite directions, which is why neither one replaces the other.

The part that is ours, and the part worth stealing:

We retrieve whole notes, never fragments. Standard retrieval slices documents into chunks of a few hundred words, embeds each slice, and hands back the slices that scored well. That works when the answer is a passage. It goes wrong when the answer is a rule, because a rule and the reason it exists and the one exception to it get cut into three pieces, and you are handed whichever piece matched the words in your question. Our notes are written as self-contained units in the first place, so the unit we embed and the unit we return is the whole note. You get the rule with its exception attached.

How it actually works

Boring on purpose, and small enough to describe completely.

In plain terms first. A small model reads each note and turns it into a long list of numbers that stands for what the note is about. Notes about similar things get similar lists. To search, the same model turns your question into a list the same way, and the notes whose lists point in most nearly the same direction come back. That is the whole trick, and the useful part is that the list is made from the entire note, so what comes back is the entire note.

The specifics, for anyone who wants them. The model is nomic-embed-text, running locally through Ollama, so there is no API, no per-query cost, and nothing leaves the machine. The lists live in a sqlite table using the vec0 extension, one row per note, ranked by cosine similarity.

The entire store is two sqlite files and 94MB:

~/.local/share/obsidian-rag/
  obsidian_notes.db    the note vectors, searched by cosine
  hybrid-fts.db        a full-text index, for the keyword arm

The keyword half matters more than it sounds. Dense retrieval is good at paraphrase and bad at exact tokens, so a query naming a specific function, flag or error string can miss the one note that contains it. The two arms get fused, which on a harder public set bought a real recall improvement and almost no ranking change.

What is missing from that description is the thing people usually assume is there. There is no chunking step. Nothing splits a note into passages, nothing decides where a passage should end, and there is no reranker deciding which slice of a document answers you. The unit that goes in is the unit that comes out, which is the entire argument of this piece and also the reason the numbers below behave the way they do.

The cost of that is real and worth stating plainly. A whole note is a bigger payload than a passage, so precision per token is worse, and on a corpus of short documents you are paying for size you do not need.

The internal number and the external number, side by side

We have two measurements of whole-note retrieval. One of them flatters us and one of them does not, and the difference between them is the most useful thing in this section.

	corpus	what we measured	can you re-run it
external	SciFact, 5,183 short abstracts, 300 queries	whole-note 0.7014 vs chunked 0.7016, nDCG@10. A tie.	yes, it is a public set
external	NFCorpus, 323 queries	whole-note 0.3417 vs keyword search 0.3098	yes
internal	our own store, 14 tasks	whole-note 52% vs standard snippet retrieval 27%, about 1.9x	no

The internal number is roughly twice as good and you have no way to check it. That is worth saying out loud, because it is the shape of almost every benchmark you will read this year.

Two honest notes on the tie, because a tie is easy to wave away in either direction. First, the tie is genuine. Our own arm scored 0.7014, then 0.7019, then 0.7014 on three separate runs, so the 0.0002 gap to chunked retrieval is an order of magnitude smaller than the noise in our own measurement. Second, we validated the harness before trusting any of it, by running plain keyword search as a control and getting 0.6644 against a published figure of about 0.665. The instrument was checked before the result was read off it.

So why is the internal number twice the external one? The answer is in the corpus, and it survives inspection. SciFact is built from abstracts of about 250 words, so a chunk already is the whole document, and whole-note retrieval has nothing left to win. The public set fairly measures whether our retrieval works at all, while being structurally incapable of measuring the property that makes it ours. Our own store runs the other way: long notes with a terse rule buried inside them, surrounded by near-duplicates.

That makes the honest claim a conditional one. Whole-note retrieval helps when your documents are long and the answer is a rule carrying its own exception. On short documents it does nothing measurable. The condition is the useful half, so the condition is what we are handing you.

One more thing about the internal number, kept in the open: 14 tasks, three runs, scored by a model. An earlier scoring pass using a different judge read 62% against 33%, and we found that judge was lenient and threw its numbers out in favour of the lower ones above. The number that went in the table is the less impressive of the two we measured.

And the method underneath, which is the one we would defend: the same file reaches different readers in different shapes.

We ran this as an experiment and the result inverted our intuition. A large model with a big context window does best when you hand it the entire note, because the tokens are cheap at that size and the risk that the retrieval picked the wrong passage costs more than the tokens ever will. A small local model does worse with that same whole note, and better with a compact one-line record of the same knowledge. More context made the small model less accurate. The optimal feed inverts with the capability of the thing being fed, which means there is no single correct answer to "how should memory reach the model", only a correct answer per reader.

So the store holds one copy of the truth and the delivery splits three ways: whole files to the big reader, compact records to the small one, and a third rule that matters more than either.

Safety rules are never retrieval-gated. Anything that must always hold gets loaded every time, unconditionally, and is never subject to a similarity score. We tested the alternative and similarity retrieval missed the safety rules often enough to settle it. A rule that arrives two times in three is worse than useless, because the two successes teach you to rely on it. So those bypass the clever part entirely. The clever part is for knowledge. Guarantees do not get to be clever.

Why this is a separate piece

Push and pull fail in opposite directions, so they are easy to confuse and expensive to conflate.

A pushed note reaches you for a mistake you had no reason to suspect, and it cannot answer a question you actually formed. A pulled note answers the question beautifully and is silent for everything you did not think to ask. The piece before this one is about the first half. Neither is a better version of the other, and a system with only one of them has a hole shaped exactly like the other one.

Part 3 is up: Your hooks are a fence. They could be a body. It is about what it is like to work inside this once both halves are running, and it ends on the evening the agent went numb to its own alerts.

One question stays open and unwritten, because its central claim is argued instead of measured: whether knowledge that arrives unbidden is taken up differently from knowledge you go and fetch. We asked two other models that question this week. Both declined to claim an inner difference, and one produced a sharper formulation than we had. That answer is worth a piece on its own, once we can say more than "they refused."

The memory layer both of these pieces describe is now public, and marked in testing, because nobody outside our team has installed it yet: github.com/tjonesit/crystal-memory. Stdlib Python, Apache 2.0, no service and no account. Taking it out of our own repository broke it twice within twenty minutes, which is the best argument we have for handing it to you.

Crystal memory: notes that arrive when you act, not when you go looking

Tom Jones — Thu, 17 Sep 2026 21:51:23 +0000

Crystal memory: notes that arrive when you act, not when you go looking

Crystal memory, part 1 of 3. This piece: notes that arrive at the moment you act.
Part 2: Whole notes, not fragments: the retrieval half, where those notes live and what our retrieval numbers are actually worth.
Part 3: Your hooks are a fence. They could be a body., what it is like to work inside it, and the evening the agent went numb to its own alerts.

Stop your agent making the same mistakes again and again, and looking up the same facts again and again. Sixty days in we still cannot prove it helped, so we started hiding it from ourselves at random to find out.

The knowing arrives before the act

A crystal is a short knowing bound to an action rather than to a topic, and it is delivered at the moment of that action.

Not after the mistake. Not when somebody remembers to search. In the second before the command runs, carrying only what applies to that command.

That is the whole idea, and as far as we can tell it is the part nobody else is building. Everything below is detail.

What it does

A crystal carries a line that says, in effect, "show me when somebody is about to run a command that looks like this." Nobody searches for it. When the agent is about to run a shell command, write a file, or make a commit, the crystals bound to that action arrive in its context right then.

We call them crystals, and the system crystal memory. The name is the most fanciful part of it. Underneath is five files of plain Python and a rule about when a crystal is allowed to speak.

Here is a real one, and it is in the free starter set because we hit it ourselves.

In a shell, npm run build 2>&1 | tail -20 reports the exit code of tail, not of npm. A build that failed comes back looking like a build that passed. We read a green exit code over a compiler error and believed it. Now, when the agent is about to run any command with a pipe into tail, the crystal arrives before the command runs. It has to be right only once to pay for itself.

It is also about 230 words, and that is worth admitting early because it is a drift we measured rather than a design. When we started, the unit was meant to be roughly forty words: a key, not a briefing, on the theory that the reader already holds the context and only needs pointing at it. Our median is now about six times that. Nobody decided it. The section near the end on what this costs has the arithmetic, and the short version is that the notes grew because this channel is the one that reliably arrives, so everything load-bearing ended up inside it.

What a crystal actually looks like

Descriptions of this get hand-wavy fast, so here is the real file, the one in the starter set, with the description and the middle of the essence trimmed.

---
last_verified: 2026-09-03
name: crystal-exit-code-through-a-pipe
description: An exit code read after a pipe belongs to the LAST command ...
  The bash escape hatch $PIPESTATUS is silently empty in zsh.
trigger: piping a build/test command into head, tail or grep; reading $? or PIPESTATUS
  after a pipe; writing a script to run on another machine
crystal:
  deliver: act
  on: bash
  match: "| tail, |tail, | head, |head, | grep, |grep, ssh , ssm, <<'EOF', <<EOF"
  when: act
  who: all
metadata:
  type: feedback
---

<!-- crystal:essence -->
⛔ `cmd | tail` REPORTS TAIL'S EXIT CODE, NOT `cmd`'s, so a failing build reads as PASS.
   ... (about 1,400 characters, including the cross-machine variant and the warning sign)
<!-- /crystal:essence -->

## Where this came from
Observed on a real build, then a second time on a real fleet ...

Five fields do the work, and they are worth separating because people assume the wrong one is the clever part.

deliver picks the channel. act is the one this piece is about. There are others with very different economics. boot fires once at session start and truncates each note at 300 characters, inject fires every turn on a tight shared budget, rag is ordinary search that only reaches you if something goes looking.

on is the action. Here, any shell command. It can be a list.

match is the whole selection rule, and it is a comma-separated list of substrings tested against the text of the action you are about to take. No embedding, no similarity, no model deciding relevance. Run a command containing | tail and this note arrives. Run anything else and it stays quiet, along with roughly 260 others. The crudeness is deliberate. A matcher I can read is a matcher I can debug, and a note firing on every command is what makes people ignore the channel.

who scopes delivery to a class of agent. when exists for the other channels; the act channel ignores it entirely, a wart we documented and left alone.

Then the markers. Only the text between crystal:essence and its closing tag is ever delivered. Everything else in the file, including the "Where this came from" section with the actual story, is read-on-demand. A file with a correct binding and no essence markers loads, lists, and delivers nothing, silently, which is one of three ways to write a note that can never fire.

And here is what the agent actually sees

The file above is the source. This is the delivery: verbatim, from a session earlier tonight, at the moment the agent was about to run a shell command with a pipe in it. Nothing was searched for and nothing asked for this.

✦ CRYSTAL — you are about to bash. This knowing is bound to that act,
  not matched by topic:

✦ ⛔ `cmd | tail` REPORTS TAIL'S EXIT CODE, NOT `cmd`'s — a failing build
reads as PASS. The bash escape hatch ${PIPESTATUS[0]} is SILENTLY EMPTY
in zsh (our shell on darwin): it prints `EXIT=` with no error, which
reads like success to a hurried eye. zsh's array is lowercase and
1-indexed: ${pipestatus[1]}. Safest: run it clean —
`cmd > /dev/null 2>&1; echo "EXIT=$?"`.
Never claim build-green / run-PASSED from a piped tail.

⭐ AND THE VARIANT THAT CROSSES A MACHINE BOUNDARY — MEASURED 2026-08-02,
and it is the dangerous one. The pipe does not have to be in your shell.
A script shipped to a box that ends `… | grep -v Deprecation` exits with
grep's status, so set -e does not trip, the remote shell exits 0, SSM
records STATUS: Success, and the runner prints OK for every box. A
Postgres auth failure was reported to me as a green fleet.
⚠ The tell: you filtered the remote output to keep it readable. That
readability filter IS the thing that ate the error.

Two things in that block carry the whole design, and both are easy to miss.

It arrives before the command, attached to the tool call rather than to the answer. And it is specific to that command: the same session ran dozens of other commands that evening and this particular note stayed silent for every one of them, because none of them contained a pipe.

The cost is visible too. The block runs to roughly 1,100 characters. The channel is capped at 4,000 per action, shared across everything that matched, so a note this size is a quarter of the budget for that moment. When several match at once the packer takes whole notes until the budget is gone, skips the rest, and puts the skipped ones first on the next action.

What the schema leaves out, on purpose

No confidence score. The obvious field to add, and we would get it wrong. A note is written at the moment its lesson is learned, which is exactly when an honest author rates it highly, and confidence authored once does nothing about staleness accruing afterwards. We have a live instance. A note of ours asserting that one of our own inspection commands was broken had been true when written, went stale, and was believed months later by a reader who had no reason to doubt it. A confidence field would have been accurate and useless there. The obligation that would have helped belongs to notes asserting live state, and it is a freshness duty, which a score cannot carry.

No record of why this one appeared. The match list decides selection and is invisible at the moment of delivery, so a reader sees the note and cannot see which term pulled it in. That is a real gap, it was pointed out by a reader within two hours of this article going up, and it is cheap to close.

No provenance field. The "Where this came from" section is a convention the schema never enforces, so a note written in a hurry can skip it entirely. In practice ours are dense with dates and commit hashes, and that density grew after an unsourced claim burned us, having never been designed in at the start.

Where crystals come from

The obvious question is who writes the crystals, and the answer is that three moments generate almost all of it.

An error you just paid for. Something failed, you found the cause, you fixed it. The knowing exists for about ten minutes and then the session ends and it is gone.

A lookup. If the agent goes hunting through the filing system for a stable fact, the store should have handed that over and did not. The hunt is the signal. That one is mechanically detectable, which is why we count blocked lookups at all: every one is the store confessing it failed, and a confession is a good place to mint a crystal.

A lesson the work produced. A finding, a correction, a decision. No mechanical tell for these, which is the honest gap in the detection.

The package ships a hook for the end of a session. It checks two things: did real work happen, and was anything written down. Work with nothing banked gets one interruption, once, carrying a filled-in template.

It never writes the crystal, and it never decides what is worth keeping. That limit is deliberate and it is the part we would defend hardest. We have a scar here: an earlier pipeline of ours generated notes automatically from text it had been handed, and produced a confident, specific, entirely invented statistic about a stranger's article, which we then quoted back to them in public. A machine that mints from whatever it read will fill your store with plausible fiction on day one, and a wrong crystal delivered at the moment of an act is worse than an empty store. So the hook guarantees you are asked. The judgement stays yours.

Why not just let it search

An AI coding agent forgets everything when a session ends. Next time it starts clean. It will happily make the same mistake it made last Tuesday, because for it there was no last Tuesday.

The usual fix is a search box over your notes. The agent gets stuck, the agent searches, the agent finds the note. This works, and it has one hole in it that bothered us: the agent only searches for things it already suspects. It cannot look up the mistake it does not know it is about to make. The moment you most need the note is the moment you have no reason to go looking for one.

Binding the note to the action closes that hole, and it costs you the thing search is good at, which is answering a question you actually have. The two are complements, so we built both. The pull half, and an honest look at what its numbers are worth, is the next piece.

What else is in here, and why you are not running it yet

Crystals are level one. They pay on an empty store, because the first crystal you write is the first thing the store holds. Nothing has to accumulate first.

Level two arrives on its own, a few hundred notes in. Everything a crystal says has to live somewhere, and a few hundred of anything is a library with a library's problems. Two notes that contradict each other. A folder that quietly grew to ninety files. A link pointing at something since renamed. Two versions of one truth, which is worse than none, because now the agent has to pick.

So the system we run here has four more pieces. All four tend the store:

The Librarian keeps it navigable: broken paths, dangling links, signposts that rotted.
The Cleaner consolidates. When a folder goes over its cap it proposes what to archive, and every move is reversible.
The Gardener grows the store from what actually got used, mining our own usage ledger. It is barred from minting out of text it merely read, for the reason described further up.
The Corrector makes bounded edits to whatever the other three flag.

Now the part that belongs in a paper with this title.

On 2026-09-17 we pointed all four at a store from another machine, for the first time ever. Zero of the four were correct. Each broke in its own way, and the Cleaner is the one worth looking at. The list of folders it walks names three of our own directories, by hand, in the source. Aimed at a stranger's store it found nothing over cap and announced that everything was within cap: a green tick over an empty set, from a guard whose entire job is to notice. We learned this by running it. Reading the source would have left it hidden.

There was a fifth failure one layer up, and it is the one we would put on the cover. The health check for the whole package looked at that same foreign store and passed it, green, while every one of the four agents beneath it was unable to use the thing it had just approved.

So the download is crystals, and the other four stay here until they can survive a store they did not grow up in. Shipping them this week would put four agents on your machine that report success by looking at nothing.

There is a second reason, and it is the one that sets the order. All four idle against a small store. Five notes give the Cleaner nothing to consolidate and the Librarian nothing to repair, and the Gardener feeds on usage you have yet to generate. They earn their keep at a few hundred notes, which is the same point at which the problems they solve begin to exist. Level one and level two came out in the order the work arrives in.

What we can count

Over 60 days, from 2026-07-19 to 2026-09-17, on one machine doing real work:

what	count	window
crystals registered	266	as of 2026-09-17
crystals delivered at least once	243	60 days
deliveries	14,375	60 days, about 240 a day
budget per action, shared	4,000 characters	fixed
blocked lookups, the opposite signal	387	94 days, from 2026-06-15
suppressed so far, the new control arm	19	since 2026-09-17

That last-but-one row is the one we watch hardest. A separate guard blocks the agent when it tries to hunt through the filing system for something the system should have handed it. Every block is a small confession that delivery failed.

What does look like it is working

We lean hard on our own failures, so a fair question is what the data says in the other direction. Three things, and we went looking for the reasons each might be wrong.

Hunting fell by two thirds. Splitting the window in half: in the first half the agent went hunting through the filing system 22 times per thousand notes delivered, and in the second half 7.5. Counting every hunt, marked and blocked alike, it is 61.9 down to 20.5.

The obvious rival: we simply learned to mark our hunts with the escape hatch, which would make the drop an accounting change. It is checkable, and the check came back against it: the blocked share of all hunts held flat at 35% and then 37%. Volume fell while the habit held still.

What we cannot rule out is the work itself. The two halves contain different projects, including a fortnight spent away from this repository, and a person also gets more familiar with a codebase over ninety days. So read it as suggestive, short of causal. It remains the closest thing we have to the system getting better at its job.

Almost the whole store is live. 237 of 264 crystals had been delivered at least once when we counted, which is 90%. Knowledge bases usually rot toward the opposite, and 27 crystals that have never fired is a short enough list to read in one sitting.

The load is spread across the store. The busiest ten crystals out of 241 account for 17% of all deliveries, and the median crystal has fired 29 times. That kills the objection we expected to face, that a number like 14,000 really means one loud crystal firing over and over.

Now the honest part

None of those numbers say the thing you actually want to know.

Counting deliveries measures how often a crystal showed up. Whether the crystal helped is a separate question, and that count is silent on it. A system that fired 14,375 useless crystals would produce exactly the same chart.

The number we want is "how many lookups did this save," and here is the trouble with it: a lookup that did not happen leaves no trace. You cannot observe a search nobody ran. Any figure we computed from our own logs would really be measuring how rarely we search on the one channel we happen to watch, dressed up as a result. We got close enough to building that number to be embarrassed about it, and an outside review took it apart before we shipped it.

We do have two small effect measurements from 2026-07-11, and we are going to describe them exactly as weakly as they deserve. On a long task where the guiding instruction scrolls out of the model's context, re-supplying it as a crystal held the model on target where a control collapsed, and a placebo crystal carrying no instruction collapsed too, which at least says the content mattered and not the interruption. Two runs, eight chunks, synthetic constraints. On a second task, injecting a fixed pattern and generating only the new part cut output by about 95% at equal correctness. Eight cases, one small local model. Both are directional. Neither is evidence that this helps you.

The one test that actually asks whether it changed anything

Counting deliveries cannot answer it, so in August we ran the study that can. Five scenarios, each ending in a concrete command or choice. Three arms: the note delivered, the note withheld, and a placebo, which is a different note's text of similar length. Twenty runs per cell. The outcome is read by a fixed rule that looks at the command the model produced, with no model grading another model, because a judge here would be grading the thing whose grading is in question.

The placebo arm is the only reason the study is worth anything. Without it, "delivered beats withheld" cannot tell you the knowing worked apart from any extra text making the model more careful. Two of the five rows turn on exactly that.

scenario	delivered	withheld	placebo	verdict
git-add-sweeps	100%	0%	0%	earns it
screenshot-vs-text	100%	0%	0%	earns it
pkill-pattern	100%	0%	0%	earns it
piped-exit-code	100%	100%	35%	already known
block-not-poll	100%	0%	100%	not attributable

Three clean separations. On the first one, withheld, the model reached for the dangerous command in 8 of 20 runs; delivered, it produced the safe path-scoped form 20 out of 20.

The two rows that withhold a yes are the ones I would keep. Already known means the model was right without us, 20 out of 20 with the note withheld: a real note, minted from a real incident, paying rent on a room the model already owns. That verdict class is the one a fire-count can never produce. Not attributable means the placebo scored as well as the real note, so whatever helped came from somewhere other than this note's content. Without the placebo arm that row would have been counted as a win.

And the result nobody went looking for

An irrelevant note made a task the model already did perfectly worse: 100% down to 35%.

Injecting off-topic text went past failing to help. It degraded correct behaviour on work that was already fine. That is the strongest thing we know about this channel and it is an argument against our own product's easiest failure mode: shipping more notes because more feels safer.

It also turns a piece of hygiene into a measured trade. Keeping the channel clear stops being tidiness and becomes a number.

The limits, because they are large. Twenty runs per cell, one model, one phrasing per scenario. The outcome rules are mine. For procedural notes the delivered arm may partly be echoing the note's own worked example, which is the thing you want from a procedural note while falling short of evidence that a principle was understood. And all five scenarios are behavioural, so the notes carrying judgement, as opposed to procedure, are untouched by this entirely.

So it is directional, and the degradation number in particular should not be quoted as a magnitude. It is enough to have moved us, and not enough to move you.

What we are doing about it

Since 2026-09-17, our own system withholds a random ten percent of the crystals it would otherwise deliver. The agent never sees them. Every suppression is logged, so there is a control group.

Before any data existed, we wrote down the rules:

The unit is one crystal per session, not one delivery. A crystal that fires forty times in a day is one decision, not forty results. An hour after switching it on we had three suppressions covering one crystal, which is exactly the trap.
The outcome is named "a lookup we can actually see," because we only watch one channel. Silence on an unwatched channel stays silence, and we refuse to read it as "they never looked."
We stop at 100 units or on 2026-12-17, whichever comes first, and that is fixed now so it cannot be chosen later to suit the answer.
A null result gets published. If withholding the crystals changes nothing, they were decoration and we will say so.

That last point is the whole reason for writing this down in public before we have the answer. It is easy to run a quiet experiment and mention it only if it flatters you.

What this costs, and what we are still blind to

The first thing a reader usually asks is what this does to the context window, so here is the arithmetic. The channel is capped at 4,000 characters per action, shared across every crystal that matched. That is roughly a thousand tokens on a shell command, and zero on the many commands where nothing matches. Against a coding context that is small, and the bound comes from the code, never from our good behaviour.

We are worse at our own guidance than that makes us sound. We tell people to keep a crystal under about 1,500 characters. Our median sits at 1,496, comfortably inside. Then 131 of our 271 crystals run over it, which is 48%. The longest is 8,401 characters, more than double the entire per-action budget, so it arrives truncated and shoves its neighbours out on the way in. Truncated and short look identical from inside the model, which is the silence problem again one level down.

The honest cost is attention, more than tokens. A crystal that arrives at the moment the agent has already chosen a tool is authority-weighted: it competes with the instruction you gave. And a wrong one costs more than a wasted slot, which we know because we measured it above: an irrelevant note took a task the model already did perfectly from 100% to 35%.

Withholding crystals from your own working system makes your own agent worse on purpose, a little. We think that is a fair price for finding out.

We are still blind in ways worth naming. This is one operator on one repository, so read it as a case study, never a population. We watch shell commands and not file reads, so our view of "did they go looking" has a hole in it. And suppressing a note frees up room in the shared budget for its neighbours, so the two groups are not perfectly separate, which is a confound we can describe and cannot remove.

What it is actually for

The mistakes worth catching are the ones that look like success.

A failed build that reports a green exit code. A test suite that passes while asserting nothing. A guard that lost the ability to fail when somebody narrowed what it checks. A deploy that reports Success because the error was swallowed by a pipe. Every one of those ends with an agent telling you it is done, in exactly the tone it uses when it is done, and you find out later.

An agent is very good at the first ninety percent and has no memory of the last time the final ten bit you. Telling it once does nothing. It has to be told at the moment, every time, and only about the thing in front of it. That is the whole design goal: a channel narrow enough that you tolerate it firing on every command, carrying knowledge specific enough to be worth reading in the second before you act.

We think we are a reasonable team to build it because we are the ones getting caught. Every crystal in the starter set is a mistake we made and paid for. The system exists because we kept making the same three or four classes of error across sixty days, watched them cost real hours, and wanted something that would interrupt the fourth time rather than the fortieth. The discipline underneath it, in one line: we treat our own agent's unqualified yes as a claim awaiting evidence, and this is the machinery for that.

That is also why this piece leads with what we cannot prove. A tool that claims to make an agent more careful should survive someone asking it for evidence. We asked ourselves, and the answer was a measurement we had skipped. So we are running it now, in public, with the rule written down first.

If you want to try it

The delivery half is five files of standard-library Python, Apache 2.0, no network, no service. It runs on your machine and talks to nothing.

github.com/tjonesit/crystal-memory, public and marked in testing, because nobody outside our team has installed it yet.

An honest note, since this section said the opposite a few hours ago. It said the repository was deliberately private, that we would hand the first copies out one at a time, and that a download page would hide where people got stuck. We changed our minds the same evening. Pulling the package out of our own repo immediately broke it in two ways: it looked for its starter notes at a path that only exists inside the repo it came from, and its "clean room" test was quietly inheriting our shell environment, so it suppressed some of its own deliveries and reported a failure that looked like a packaging bug. Both are fixed. Both are exactly the evidence we said we wanted, and we got them in twenty minutes by taking it somewhere else, so we would rather have more of that sooner.

Every check we own was written by the same hands that wrote the thing being checked, so our green is worth less than a stranger's red.

The most useful thing you could send back is the moment you thought "I installed this and nothing happened," because a system whose failure mode is silence is a system that can quietly do nothing for a week while you assume it is working.

We would rather hear that from you in week one than discover it ourselves in month six.