I had a benchmark of retrieval questions over book-length documents, and a claim
that answering them required reading the whole document rather than the part
that fits in a context window.
I did not want to take that on faith, so I wrote a test with no model in it at
all. It scores oracles: agents with perfect extraction and perfect reasoning
that can only see the first F of each document. If a task is answerable at F=0.25,
it does not need coverage, whatever I think it needs.
The results looked exactly right:
| task type | F=1.0 | F=0.5 | F=0.25 |
|---|---|---|---|
| "who speaks last" | 1.000 | 0.423 | 0.500 |
| "does X speak in the final quarter" | 1.000 | 0.500 | 0.500 |
Exact with everything visible, collapsing to chance with half the document
hidden. That is a coverage-bound task, demonstrated rather than asserted.
It is also wrong, and the error is embarrassing once you see it.
I only ever tested prefixes
An oracle that reads the first quarter is not the only partial reader. There
is also one that reads the last quarter. I never wrote it.
| task type | prefix, F=0.25 | suffix, S=0.25 |
|---|---|---|
| "who speaks last" | 0.500 | 0.962 |
| "does X speak in the final quarter" | 0.500 | 1.000 |
| "who speaks first" | 1.000 | 0.378 |
Reading the last quarter answers both of my supposedly coverage-bound tasks
almost perfectly.
In hindsight it is obvious to the point of being painful. "Who speaks last"
has its answer in the tail by definition. "Does X speak in the final
quarter" names the region to look in. Neither task ever needed the middle of
the document. Each needs one end.
My test proved they resist truncation from the front. I read that as proving
they require the whole thing.
The empirical arm had already told me
I had also run a real baseline: the same model reading only the last window of
each document. It scored 0.852 on the final-quarter question. My full sweep,
reading everything, scored 0.852 batched and 0.870 run deterministically.
A dead heat, or close enough to one. I recorded it as "the tail baseline is
surprisingly strong" and moved on.
It was not a strong baseline. It was a task that never required what I was
selling. The oracle and the measurement agreed with each other and disagreed
with me, and I only noticed when I made the oracle symmetric.
What a coverage-bound task actually looks like
The fix took two attempts, and the first one is instructive because it
almost works.
Attempt one: widest span. "Whose speaking is spread over the largest portion
of the document, from their first line to their last?" That needs both the first
and the last line of every candidate, so neither a prefix nor a suffix reader
can have both.
Suffix oracle at S=0.25: 0.790. Not good enough.
Why: the answer is an ordering of widths, and width ordering degrades
proportionally. A character whose true span is wide tends to look comparatively
wide inside any window too. The partial reader gets the wrong magnitudes and the
right ranking, which is all the task asks for.
Attempt two: breadth. "Divide the document into ten equal parts. Who speaks
in the greatest number of them?"
| F=1.0 | prefix F=0.25 | suffix S=0.25 | |
|---|---|---|---|
| breadth | 1.000 | 0.673 | 0.612 |
This one cannot be recovered proportionally, because it is not a magnitude that
scales — it is a count of distinct regions, and an agent that sees a quarter of
the document sees at most a quarter of the regions. The true count is
unreachable, not merely noisy. And it is symmetric: neither end helps.
One caveat that matters. Counting is the single thing small models are worst at
— I have measured 0.02 accuracy on exact counts with this model class. So the
model never counts. It reports which characters spoke in the one fragment it can
see, and the code counts distinct regions afterwards. The task is a counting task
for the system and a presence task for the model.
What the claim looks like once the benchmark is honest
On breadth, paired bootstrap over 9 resampled documents, same model and same
window for every arm:
| sweep vs | difference | 95% CI |
|---|---|---|
| read the front that fits | +0.429 | [+0.160, +0.673] |
| read front and back | +0.306 | [+0.065, +0.525] |
| a 40-line regex, no model at all | +0.204 | [+0.036, +0.367] |
That is a narrower claim than the one I started with, on one task type instead
of four. It is also a better one, because the no-model baseline is in it. If I
had not run the regex, someone else would have, and the honest version of "we
didn't need the LLM for most of this" is much easier to write yourself than to
have written about you.
Then the regex took more of the claim. Those numbers come from a 20GB card with
the server running 4 parallel slots. I held the setup to the hardware the claim
is about: a 6GB card, where only 1 slot fits.
| setup | breadth | vs the regex, 95% CI |
|---|---|---|
| 20GB card, 4 slots | 0.980 | +0.204 [+0.036, +0.367] |
| 20GB card, 1 slot | 0.898 | +0.122 [+0.000, +0.276] |
| 6GB card, 1 slot | 0.816 | +0.041 [-0.109, +0.231] |
| regex, no model | 0.776 |
Half of the drop is the slot count, and half is the hardware. On the target
hardware, the system is not distinguishable from the regex. The narrower claim
did not survive either. With 9 documents, I cannot tell whether the true
difference is zero or only too small to see. The next run uses 30 documents.
This is not a new idea, only a cheaper check
Truncation checks already exist. Ada-LEval (Wang et al., 2024) calls a task
full-text if a model's score drops sharply when the input is cut. LongBench
removes the middle of each input to show that the ends are not enough. Goldman
et al. (2024) name the property: how dispersed the needed information is, and
how much of it there is. Both checks run a model, and neither compares a reader
of only the front with a reader of only the back.
The oracle above runs no model. It costs seconds, it has no model noise, and it
gives an upper bound: if a perfect reader of one quarter reaches the answer, no
model result can show that the task needs the whole document.
The transferable bit
An ablation tests the direction you thought of. Mine tested forward truncation
because truncation is what I was worried about, and my worry had a direction
baked into it that I could not see from inside.
Before trusting an ablation, ask what its opposite is. Delete the front as well
as the back. Shuffle as well as truncate. Give the degenerate baseline every
advantage rather than the least.
And when a baseline you expected to be weak comes in strong, that is data about
your benchmark, not a footnote about your baseline.
Top comments (0)