I told the model to separate fields with <TAB>. It did exactly that, and I lost 79 percent of my data.

My extraction prompt said:

Output lines of the form <key><TAB><value>

The model output:

Elizabeth<TAB>I have not the pleasure of understanding you

Those are five literal characters — angle bracket, T, A, B, angle bracket — not
a tab. My parser looked for \t, found none, and dropped the line.

Across one document, 246 of 310 records were discarded. Only 5 were dropped
for the reason the mechanism actually existed to catch.

The model did what I asked. I asked for the wrong thing.

Why I did not notice for a whole benchmark run

This is the part worth stealing, because the bug is trivial and the reason it
survived is not.

My harness counted discarded records in a single field called rejected. That
field had one documented meaning: a record whose evidence could not be verified
against the source text. In other words, it was the hallucination-catching
counter.

So the run printed rejected=246 and I read it as the verifier working
hard. A parser failure was wearing the costume of a successful integrity
check. The number went up, and up was the direction I wanted.

The fix is not a better parser. It is that these two things must never share a
counter:

• unparseable — our format failed, the model is not at fault
• ungrounded — the model asserted something that is not in the text

One of those is my bug. The other is the model's. Summing them produces a
number that cannot be acted on, and which flatters whichever explanation you
already believe.

After splitting them, the same run reads:

records: lines=310 kept=64 | unparseable=246 ungrounded=5
separator: canonical=0 tab=64 literal_TAB=246

There is no way to misread that.

Fix one: show the delimiter, never name it

A model can reproduce a character it can see. It cannot reliably reproduce a
character you have described in prose, because your description is text and
copying text is what it does.

So instead of naming the separator, I showed a worked example using a pipe. I
picked | after checking the corpus contained zero of them in 7,987,517
characters.

Result: canonical separator compliance went from 0 to 9 out of 9 on the first
fragment, and unparseable loss went from 79 percent to zero.

Fix two: your example must be obviously not data

The first version of that fix failed differently, and worse.

I used a real line from the source novel as the worked example:

Elizabeth | I have not the pleasure of understanding you

The model copied the example verbatim and extracted nothing else. One line of
output for a 60,000-character fragment.

It looked plausible. The name was a real character, the quote was real dialogue.
The only reason I caught it was that the evidence verifier rejected it — that
sentence is not in the fragment I sent, it is from a later chapter.

The working version uses a name that cannot appear in any novel and says so:

Zolvane | we must leave before the tide turns

That line is ONLY a format example. Zolvane is not in this text; never copy
the example into your answer.

A worked example is an instruction to imitate. If your example is
indistinguishable from valid output, some fraction of the time it will be
returned as output.

Fix three: make the failure loud, at the right granularity

I added a check that aborts when more than 20 percent of output lines have no
recognised separator, printing a sample of the offending line.

My first version aborted the whole document. That was worse than the bug: the
model follows the format on most fragments and abandons it on a few, so
discarding the document threw away every good fragment along with the bad one.
Four documents produced zero records.

Now a violation fails one fragment. Good fragments survive, the failure is
still visible and still blocks a headline answer, and one document went from
zero records to 148 of 169 kept.

The forty-second diagnostic

The single highest-value thing I added was a --probe flag: run one fragment,
print the raw model reply verbatim, print the loss breakdown, exit.

It takes about seven seconds. It caught both the literal-<TAB> bug and the
example-copying bug immediately.

Before that, my shortest path to noticing a contract problem was a full
benchmark run — about forty minutes — and even then only if I read the right
counter and interpreted it correctly, which the first time around I did not.

If your pipeline sends prompts to a model in a loop, you want a way to see
exactly one of them, unedited, in under a minute. I wrote three throwaway
scripts to do that by hand before admitting it should be a flag.