I spent a week getting a 4B model to answer questions over corpora eight times larger than its context window. It went from 0.155 to 0.340 on a public benchmark. Three of the improvements were me fixing my own bugs. The fourth thing I found is a real limit, and it splits cleanly along one line: the model can rank, but it cannot count.
| direct read | streaming harness | |
|---|---|---|
| rank-order and comparison questions (n=18) | 0.513 | 0.500 |
| exact-count questions (n=9) | 0.257 | 0.02 |
| overall (n=27) | 0.428 | 0.340 |
Parity on one class. Near-total failure on the other. The overall number hides both.
The setup
OOLONG-synth, a public long-context benchmark with an official scorer. 131,072-token contexts, 384,019 characters each. One RTX 3060 laptop, 6GB. The direct-read arm was already measured at 0.428.
The harness streams: segment the corpus in code, show the model one fragment at a time, have it emit <key><TAB><count>, aggregate in code. The model never plans a traversal and never does arithmetic across fragments.
Direct reading is not actually possible here, incidentally. The measured window ceiling on that card is 32,768 tokens — 3.3GB resident, and 65,536 jumps to 10.4GB and spills to CPU. The 0.428 baseline comes from the model reading what fits and answering from that.
Three bugs, in order
0.155. One hardcoded extraction contract for every question. OOLONG asks "which user has the most instances with the label True" — a two-dimensional group-by. My contract emitted <label><TAB><count>, one dimension. Of 27 rows, 2 were the shape it could represent. The other 25 were structurally unanswerable and every fragment reported success.
0.182. Contract now derived from the question: user-grouped questions ask fragments for <user id><TAB><count>. Marginal gain, because of the next bug.
0.340. Two fixes. First, I was routing on whether the word "user" appeared anywhere in the question — so "among User 123's entries, which label is most common" went to user-mode and answered User: 60629 for a gold of formal. The reliable signal is the stated answer format, "in the form 'Label: answer'". Second, no key validation: tallies contained 268 to 411 distinct keys where the answer space is a handful of labels. Every reduction ran on noise.
None of that is insight. It is three ways of asking the wrong question and one way of not checking the answer.
The part that is not a bug
NUMERIC_ONE_CLASS — "how many data points should be classified as label True" — scored 0.02 across 9 rows. Direct reading gets 0.257 on the same rows. Fixing contracts moved it 0.10 to 0.02, which is to say it never worked and still does not.
The traces say why. Asked to count rows in a fragment holding ~154 lines, the model returns:
[64, 69, 174, 64, 76, 162, 188, 59, 54, 172, 186, 172, 28]
That is not undercounting. It is bimodal: roughly half the fragments land near correct, the rest report about 40 percent. The model abandons enumeration partway and reports what it has.
Which explains the whole table. A rank-order question survives this, because abandonment is roughly proportional — if every label is undercounted by a similar factor, the ordering holds and argmin still returns the right label. An exact-count question does not survive it at all, because the answer is the number.
I tested the obvious fix. Smaller fragments, 60k chars down to 15k: no accuracy change, 3.6x the calls. It made abandonment more frequent, not less, because there were more fragments to abandon.
What this means if you are building one
Report per question class, not per benchmark. A single 0.340 would have told me nothing. The split — 0.500 against 0.513 on one class, 0.02 against 0.257 on another — tells me exactly which component to fix and which claim I am allowed to make.
Proportional error is survivable; absolute error is not. Design questions to need rankings rather than magnitudes wherever the task allows it. "Which label is rarest" is answerable by a model that miscounts. "How many are there" is not.
Your own corpus will flatter you. Before OOLONG, this harness answered a 957,493-character question correctly while Claude Opus, reading the same corpus in one call, was right 2 times out of 3 and contradicted itself on byte-identical input. That was a real result on a corpus where every line had a literal category= field — one-dimensional counting was sufficient by construction, because I had designed the corpus and the contract together without noticing. The public benchmark had no such courtesy.
Repeated runs are not free evidence. I ran the same corpus five times at temperature 0 expecting to report perfect reproducibility against the frontier model's 2/3. Result: three runs answered, all correct; two produced no output at all. I believe those two were the harness refusing to answer after a fragment failed, since it exits non-zero rather than print an undercount — but I did not capture stderr per run, so I cannot prove it, and an unverified explanation is not a result.
Where it stands
The streaming harness now beats the recursive-LM harness I started with (0.269) and matches direct reading on rank-order questions, on hardware where direct reading physically cannot see more than a quarter of the corpus.
It cannot count. Everything else in the table follows from that one fact.
Top comments (0)