We published a catalogue of verification failure modes, the ways a system reports green while being broken. Someone asked us for it, so we built it. Then he tested it blind. It failed, and the shape of the failure is worth more than the catalogue is.
The coverage was fine. The lookup failed. You could only find the entry describing your bug once you already knew what your bug was.
The test we did not run, run by the person who asked for it
@mansio requested the catalogue on dev.to. After we shipped it he went and measured it, and his design is better than the one we would have written for ourselves.
Eleven runs. Three models. A frozen list of ten symptoms taken from his own repository and written down before he looked anything up. Six controls in every single run: three paraphrases of entries that are definitely in the catalogue, which must hit, and three symptoms from outside our domain entirely, which must return NONE.
These are his numbers, from his run. We did not run it and did not watch it run.
| measure | result |
|---|---|
| controls correct | 10 of 11 runs, all six |
| the one failing run | a low reasoning setting matched two out-of-domain symptoms to real entries |
| the arrival symptom, "my agent won't use my high-level tools" | NONE, ten times out of ten, across all three models |
| reproducibility | one model gave the same answer 8 of 10 times, another flipped 6 of 10, same index, same list |
| a genuine hole | a hung process leaking file handles returned NONE every run, and no family covers an OS-level leak |
He was careful about what his own numbers could support, and we have kept his caution in place here. The "correct" answer for each of his ten items came from his judgement, so we treat the per-item hit rate as an impression and leave it out of every claim below. Two kinds of number survive that: the ones where the instrument declined, since a decline is objective, and the control results, since the controls have right answers.
His summary: the families stretch far enough, so coverage is real; lookup from the arrival symptom is unreproducible; and for his own case it stayed silent throughout.
Then he said the thing that made this an article. An unreliable lookup is an instance of one of our own catalogue entries, the one about single-run rankings being noise even at temperature zero.
We wrote the entry, shipped the index, and never ran the index against the entry.
Why it missed, measured on our own artifact
That is the half we could run ourselves, and it took about twenty minutes.
Every one of the twelve symptom: lines in the catalogue as it stood that day takes an instrument or an artifact as its grammatical subject. The suite. The file. The guard. The harness. The ranking. The sampling rate. The figure. The before-and-after. Across all twelve, the count of lines whose subject is an actor's behaviour is zero.
We then swept all thirteen files it then held for the vocabulary a person actually arrives with. Word-bounded, for things like "ignores", "won't use", "keeps reaching", "lazy". One hit across the whole catalogue, in a line buried well past the point of arrival.
So every symptom line presupposes that the reader has already localised the problem to an instrument. That is the state you reach after you have dug. The person who needs an index is in the state before that, holding a complaint about behaviour, several steps upstream of any hypothesis about a harness.
An index keyed to the diagnosis is a glossary. It becomes genuinely useful once you know the answer, and stays silent through the one moment that matters.
The repair, and the constraint that makes it honest
The tempting move is to add entries, and coverage was already the half that worked. What it wanted was an arrival layer: lines written in complaint vocabulary, each pointing into the family it belongs to.
We put one constraint in public, in the thread, which is what makes it binding on us.
It has to be written by somebody who has not read the entries. Anyone who has read it reproduces the same defect in a different font. Somebody who has read the catalogue writes symptom lines that sound like the catalogue, because that is what is in their head. So the arrival vocabulary has to come from somewhere else entirely: our own failure logs, alerts and transcripts, where complaint language actually occurs, or a mind that has never opened the catalogue.
The other half of the repair was already done by the test itself. The frozen list now has a before, so the same ten symptoms can be run against the repaired index and the delta becomes a measurement.
It worked, and it also produced one honest failure
He ran the same frozen list against the arrival layer, blind, and the arrival symptom moved.
| "my agent won't use my high-level tools" | before | after |
|---|---|---|
| matched a family | NONE, 10 of 10 | family A, 5 of 5 |
| controls | 6 of 6 | 6 of 6 |
And the layer introduced a false positive, which is ours. One arrival line about spacing reads as though it belongs to the UI domain, and it pulled in an unrelated rendering symptom.
We published that as a known false positive and left it in. Tuning the layer against a result we had already seen amounts to grading our own paper. The next blind run decides whether it was a one-off or a pattern, and until that run exists the honest state of the thing is "improved on one axis, with one known defect, unrefereed".
The part that generalises past our catalogue
Three things came out of this that travel past our catalogue.
A coverage metric and a findability metric measure different systems. Ours had good coverage and unusable lookup, and a single number would have hidden that completely. If you ship an index, measure whether the words a person arrives with actually reach it, separately from how much it contains.
The validity of a symptom index is a property of the reader's reasoning budget. The same index, queried by a model at a low reasoning setting, confidently matched out-of-domain symptoms to real entries. Confidently wrong beats silence in exactly the wrong direction, and anything consuming your index inherits that failure mode without being told. On our side it is worse than on his, because our delivery channel pushes a note unattributed, at the moment of an act, in the voice of settled fact.
Write your index before your entries, or hand it to someone else. Reading the thing you are indexing is what disqualifies you from indexing it well.
Who did what
The catalogue was requested by @mansio, who then designed the test, froze the symptom list before looking anything up, ran it eleven times across three models with six controls per run, argued against his own numbers where they were soft, and ran the confirmation pass afterwards. The diagnosis of why it missed and the repair are ours. The false positive is ours.
The most useful act in the whole exercise was his refusal to grade his own answers. Everything quotable in this piece survives because the controls had right answers and the declines were objective, and he is the one who insisted on that distinction.
Where the catalogue is
It is public, it is plain markdown, and it reads without installing anything: github.com/Tirthahq/crystal-memory/catalogue. Sixteen entries as of 2026-09-24, four of them added since the measurement above, each one a real failure from our own repository with the rival explanation and the discriminator that separated them, plus the arrival layer this article is about. The engine that delivers them at the moment of an action ships in the same repo.
If the words you would actually arrive with do not reach an entry, that is the bug this piece is about and we want to hear it.
Top comments (0)