A parser we shipped for our own decision log read zero of the 887 lines in the live copy. Not most of them — zero. The gate that runs it did not report that as a pass, and it did not report it as a fail either. It reported a third thing, and that third thing is the part worth writing down.
The setup
We keep a decision ledger — every ruling a session makes gets an append-only entry, machine-parseable, so a gate can check things like "does every report on disk get declared somewhere" without a human re-reading the whole history. The parser that reads that ledger got rewritten as part of widening what it could extract.
The new parser passed its test suite. Then someone ran it against the actual live ledger instead of the fixtures, and the finding was 0 out of 887 lines parsed.
Why the tests didn't catch it
The test suite was testing the parser against small, hand-written examples of the grammar. Every one of those examples used the grammar correctly, because a person had written them to demonstrate the grammar. The live ledger had drifted — some real, accumulated formatting difference the fixtures never captured — and the new parser's assumptions about the input shape were narrower than what 887 real lines actually looked like.
This is the same failure shape as testing a filter against clean data and shipping it against dirty data, except here the "dirty data" is just... the actual thing being parsed, which nobody had run the new code against until someone thought to.
What the gate did with a 0/887 result
This is the part I want to hold up. The check that depends on parsing the ledger — verifying that the causal chain between decisions is recomputable — got a denominator of 887 and a numerator of 0. The obvious two things to report are PASS (nothing failed, vacuously) or FAIL (nothing succeeded). Both are wrong, and both are worse than what actually happened: the gate reported the check as UNKNOWN, tagged CHAIN_NOT_RECOMPUTABLE, on all 887.
UNKNOWN is not a hedge. It is a specific claim: this instrument cannot currently tell you PASS or FAIL, and that is different information from either one. A gate that collapses "I could not check" into "it failed" produces false alarms indistinguishable from real ones, and everyone eventually starts ignoring the noisy gate. A gate that collapses "I could not check" into "it passed" is worse — it is a silent green wearing the costume of a real result, and that is exactly the shape that let the 0/887 parser rate sit undetected in the first place if anyone downstream had trusted a green they hadn't looked behind.
The distinction only pays for itself if UNKNOWN is rare enough to notice. If everything is UNKNOWN nothing is informative either — that failure mode exists too, in the other direction, and we don't have a clean answer for the threshold where a system that reports honest uncertainty stops being useful because it reports it too often.
The other kind of miss, same day: measuring the wrong dimension
A separate piece of the same engine got reorganized into layers, with the working assumption that this would cost time — more indirection, more hops between layers, presumably slower. That was measured directly: an operation that reads by byte offset against one that walks and greps came out at 62ms versus 2ms.
31x is a real number, and it is not the number anyone expected to explain. The byte-offset path wasn't the slow one because of layering — it was slow because it read the entire body of every record before checking whether the record was even relevant, where the grep path could reject most records without ever reading their bodies. The hypothesis going in was "the new layering will cost time." The actual finding was "the new layering costs nothing; a specific read pattern inside one layer reads more than it needs to." Confirming the wrong half of a hypothesis and refuting the right half in the same measurement is a useful reminder that "did the thing get slower" and "why" are two different questions, and answering only the first one tells you where to look, not what you'll find there.
The boring failure that matters more than either
Somewhere in the same pass, a registry file that is supposed to declare every report a work session produces was checked against what actually exists on disk: 21 report files existed, 16 were declared, 5 were orphans — real work, done, with no pointer to it anywhere a later reader or a gate would look. Nothing dramatic happened because of this. It's also the failure mode most likely to happen again, because it produces no error, no red test, no crash — just work that quietly stopped being discoverable. It was caught only because someone thought to compare a list against a filesystem instead of trusting that the list was current.
Where this leaves the parser
Rewritten to handle the shape the live 887 lines actually have, verified against the live copy rather than only the fixtures this time, and the fixtures got the drifted shape added to them so the next rewrite has a chance of catching it before shipping. The gate still reports UNKNOWN wherever it genuinely doesn't know, rather than picking the more comfortable of the two wrong answers.
TraceFold, Rust, Apache-2.0, alpha. The parser in this story isn't in the public tree yet — this was the DB layer that feeds it, still in motion.
Top comments (0)