Two results from the same system, three weeks apart, and the second one is the useful one.
Result A. A 4B model on a 6GB laptop GPU answered a 440,000-token aggregation question correctly. Claude Opus 4.8, reading the same corpus in one call, got it right 2 times out of 3 and contradicted itself on byte-identical input.
Result B. We ran the same system on OOLONG, a published long-context benchmark, against arms we had already measured. It came last by a wide margin.
| arm | n | score |
|---|---|---|
| Haiku 4.5, direct | 27 | 0.428 |
| Haiku 4.5 + recursive harness | 32 | 0.269 |
| our streaming harness | 27 | 0.155 |
Result A is the one you would put on a landing page. Result B is the one that taught us something, and it turned out not to mean what the number suggests.
What the system does
The failure we started from: a 4B asked to answer a question over a corpus 8x larger than its context window would sweep every fragment correctly and then, asked which label was least common, reply
Status: beta, Status: delta, Status: gamma, Status: alpha
It listed the candidates instead of selecting one. An earlier variant read a third of the corpus and narrated what it saw.
Neither is a knowledge problem. We were asking one model to plan a traversal, extract from text, and do arithmetic across 65 partial results, inside a loop. So we split those apart, borrowing the shape from video streaming:
| video | here |
|---|---|
| manifest | segment plan computed in code, before any model call |
| buffer | N fragments in flight |
| decoder | model sees ONE fragment, emits key<TAB>number, never prose |
| playback | reduce — aggregation in code, strategy chosen explicitly |
The model only extracts. Planning is deterministic, summing is a loop, selecting a minimum is a comparison. On our own 957,493-character corpus that produced the correct answer in 17 fragments, 611 parsed records, 3 unparseable lines, zero failures.
Then we ran it on someone else's benchmark
Self-graded results on a corpus you built yourself are worth very little. OOLONG-synth is public, published, and ships an official scorer we ported verbatim. We had two arms on it already.
We scored 0.155. Worst of the three.
The instinct is to explain that away. Here is what the trace actually said:
MOST_FREQ keys=406 answer: "Label: Male parent (also used as a term of address to your father)"
gold: 72232
Our harness emitted 406 distinct keys on a task whose answer is a user ID. Median across rows: 119 keys. It was extracting sentence fragments as if they were labels.
Then we looked at what the questions actually ask:
In the above data, which user has the most instances with the label True?
That is a two-dimensional aggregation. Group by user, filter by a label the model has to infer, count. Our extraction contract emits <key><TAB><number> — one dimension. It cannot express "user × label → count" at all.
Of the 27 rows, 10 wanted a number, 8 a comparison, 7 a user ID, and 2 were the label-frequency shape our contract was designed for.
So 0.155 does not measure "streaming is worse than direct reading." It measures one hardcoded extraction contract applied to six question types, 25 of which it structurally could not represent. That is our bug, and the benchmark found it in a way our own corpus never would have.
Why this is the more valuable result
The bespoke corpus flattered us because we had, without noticing, designed the corpus and the contract together. Every line had a literal category= field. One-dimensional counting was sufficient by construction.
A public benchmark had no such courtesy. It contains question shapes we did not anticipate, and it exposed that our "general" harness was a specialised one wearing a general interface.
The fix is not a better prompt. It is that the extraction contract has to be derived from the question — a group-by question needs a <groupkey><TAB><subkey><TAB><count> contract — and a harness that ships one fixed contract will silently score near zero on anything shaped differently. Silently, because every fragment succeeded. Coverage was 1.00 and fragment errors were 0 on every row. Nothing failed. It just answered a question nobody asked.
The other thing that did not work
We also implemented the outer loop from Meta-Harness (Lee et al., 2026): an agentic proposer that reads prior candidates' source, scores and traces from disk and proposes new harness code.
Three candidates in:
| candidate | accuracy | calls |
|---|---|---|
| baseline | 0.50 | 16 |
| smaller fragments | 0.50 | 58 |
| self-reported checksum | 0.25 | 16 |
Zero improvements. One regression. 3.6x the cost for nothing.
The traces were still worth having. The counts a fragment reported were bimodal — [64, 69, 174, 64, 76, 162, 188, 59, 54, 172, 186, 172, 28] where every fragment held ~154 lines. Not uniform undercounting: about half the fragments get abandoned partway and the model reports what it had. That is why smaller fragments did not help, and it is invisible in the score.
Which is the paper's actual claim, landing in the least flattering way available: rich access to prior experience beats compressed feedback. The score said 0.50 twice. The trace said why.
Three things worth stealing
Verify context reached the model. Our serving layer auto-sized context up to a ceiling, then silently fell back: 30,021 tokens passed intact, 50,000 and 70,000 both clipped to exactly 16,387 — no error, confident answer from the fragment it kept. Compare reported prompt tokens against what you sent, and make the mismatch fatal.
Never score with substring matching. One of our runs reported "correct" because the model dumped raw corpus rows and the dump happened to contain the gold label.
Run on a benchmark you did not build. This is the whole post. The paired OOLONG run data behind these numbers is public: huggingface.co/datasets/Rickesh/rlm-oolong-reproduction. We had a working system, a real win, and a plausible story. One public benchmark showed the generality was imaginary. That cost an afternoon and it was the cheapest thing we did.
The system genuinely does what Result A says on the workload it was built for. What we cannot yet claim is that it generalises — and we would have shipped that claim if we had stopped at our own corpus.
Top comments (0)