A model advertising a million-token context window is making a claim about what it will accept, not about what it will use. Long-context benchmarks exist to measure the gap, and they divide sharply into ones that test whether a fact can be found and ones that test whether the whole context can be reasoned over. Only the second kind measures what people actually mean by long context.
The question these benchmarks answer
There are three separable capabilities inside “handles long context”, and a benchmark usually tests one.
| Capability | Description |
|---|---|
| Retrieval | Find a specific piece of information somewhere in the context and reproduce it. One fact, one location, exact answer. This is the easiest of the three and the one needle tests measure. |
| Tracking | Follow a chain across the context: a variable assigned here, reassigned there, referenced later. Requires holding several positions in play at once. Multi-hop and variable-tracking tasks measure this. |
| Aggregation | Compute something over the whole context: count occurrences, find the most frequent item, summarise everything, answer a question no single passage contains. The hardest, and the one long documents actually demand. |
A model can be excellent at the first and poor at the third at the same length. That is not a contradiction; they are different operations. It is also why a chart showing perfect needle retrieval at 128k tells you very little about summarising a 128k document.
Needle in a haystack, and its ceiling
The needle test, popularised by Greg Kamradt in 2023, is simple enough to write in an afternoon and that is most of why it spread. Take filler text of length L. Insert one distinctive sentence at relative depth d. Ask a question whose answer is that sentence. Sweep L and d and plot a heatmap of pass or fail.
for length in [4_000, 8_000, 16_000, 32_000, 64_000, 128_000]:
for depth in [0.0, 0.1, 0.25, 0.5, 0.75, 0.9, 1.0]:
haystack = filler_of_exactly(length)
needle = "The best thing to do in San Francisco is eat a sandwich in Dolores Park on a sunny day."
prompt = insert_at(haystack, needle, depth) + "\n\nWhat is the best thing to do in San Francisco?"
record(length, depth, judge(model(prompt), needle))
It is a genuinely useful smoke test and it has a hard ceiling on what it can prove.
- One needle is a lower bound on difficulty. The answer is present verbatim and nothing else in the context resembles it. Attention has an easy target. Multi-needle variants — several facts to combine, or several similar facts of which one is asked for — are much harder and are what RULER generalises.
- The filler is not distractor text. If the haystack is essays and the needle is about sandwiches, the needle is semantically isolated. Real documents contain many passages that look relevant. Filler drawn from the same domain as the needle makes the test several times harder, and most published needle charts do not do that.
- Pass or fail hides degradation. A binary judgement cannot show a model that finds the needle but garbles a detail, which is the common real failure.
- It has been optimised against. The test is famous and the pattern is easy to train for. Perfect needle performance is now close to table stakes and correspondingly uninformative.
RULER and effective context length
RULER, published by NVIDIA researchers in 2024, is the systematic version. Instead of one synthetic task it defines a family of them across four categories, all generated procedurally so the length can be set to any value and the items are fresh every run — which incidentally makes contamination structurally impossible.
| RULER category | Description |
|---|---|
| Retrieval | Needle variants: multiple keys, multiple values per key, multiple queries at once, and needles hidden among distractors of the same type. A parameterised generalisation of the single-needle test. |
| Multi-hop tracing | Variable tracking: a chain of assignments spread through the context, where answering requires following the chain to its end. Tests the tracking capability directly. |
| Aggregation | Extract the most common or most frequent words across the entire context. There is no single passage that contains the answer; it must be computed over everything. |
| Question answering | Real QA datasets padded with distractor passages to the target length, which is the closest of the four to a realistic task. |
RULER’s headline contribution is the notion of an effective context length: the longest length at which a model still clears a threshold defined by a reference model’s performance at a short length. The finding that made it well known is a category finding rather than a per-model one, and it holds up: many models that advertise very large windows fall below the threshold at a fraction of the advertised number, and the drop-off is not gentle for the aggregation tasks.
That distinction — advertised versus effective — is developed at length in effective context length versus advertised context length, and the positional side of it in lost in the middle.
LongBench and real tasks
RULER is synthetic by design: precise, controllable, and unlike anything you would actually do. LongBench (Tsinghua, 2023) takes the other route, assembling real tasks at length — single-document and multi-document question answering, summarisation, few-shot learning, code completion, and some synthetic tasks — in English and Chinese, each with its own conventional metric.
Two construction details matter when reading its numbers. First, the metrics are per-task and mostly overlap-based (F1 for extractive QA, ROUGE for summarisation), which inherits all the known weaknesses of those measures — see what ROUGE and BLEU actually measure. Second, LongBench defines a truncation rule for models whose window is shorter than an item: cut from the middle, keeping the head and tail. That is a sensible default and it means a short-window model scores non-zero, which makes cross-model comparison at a fixed length subtly unfair unless truncation rates are reported.
The second version moves to a multiple-choice format across a very wide length range with expert human baselines, which trades some realism for unambiguous grading. Other entries in this genre — long-context suites built around retrieval-style tasks, and ones that push to extreme lengths — make similar trades. Check the version before comparing.
Position is a variable, and it must be swept
The single most important methodological point about long-context evaluation: if you place the relevant information at one position, you have measured that position.
Accuracy as a function of depth is typically U-shaped — strong at the start, strong at the end, weakest in the middle. A test that puts the answer at the top reports a number that has nothing to do with a document where the answer is at 40 per cent depth. Any long-context result without a depth sweep is reporting one cell of a table and calling it the table.
Minimum honest design for a long-context test:
lengths x depths x repeats
6 x 7 x 3 = 126 runs per condition
Report the full grid, or at minimum:
- accuracy at the worst depth (not the mean)
- the length at which worst-depth accuracy crosses your threshold
The mean over depths flatters every model, because the endpoints are easy.
Reporting worst-depth rather than mean-depth is the same argument HELM makes for worst-case robustness: the user does not get to choose where in the document their answer lives.
Designing your own long-context test
Public long-context benchmarks are more useful as templates than as verdicts, because the thing you need to know is whether your documents survive the window, and document structure varies enormously. Contracts, chat logs, codebases and transcripts degrade differently.
- Take twenty real documents at the length you care about. Not filler — real ones, with your formatting, your boilerplate and your repetition.
- Write three questions per document: one retrieval (a fact stated in one place), one tracking (requires combining two separated places), one aggregation (requires the whole document).
- For the retrieval and tracking questions, produce variants with the relevant passages relocated to at least three depths. This is the step everyone skips and it is where the finding is.
- Grade with an exact or near-exact rule where you can. Fall back on a judge only for the aggregation questions, and read a sample of its gradings by hand — judges have their own biases.
- Report accuracy by question type and by depth. A single number will hide the only interesting result, which is almost always that aggregation collapses long before retrieval does.
A depth sweep is a lot of requests: the grid above is 126 calls per model, and comparing four models at several lengths runs into thousands of long-prompt requests, which is where the cost is. Running that through one API that reports per-request token counts and cost means the sweep produces a cost curve alongside the accuracy curve, and at long context the cost curve is often the finding — input tokens dominate the bill well before quality collapses.
One caution to close on. Long-context evaluation results age faster than most, because context handling is where architectures have been changing quickest. A result from eighteen months ago about a model family is history, not evidence. Rerun it; the harness above is a day of work and it answers the question for your documents rather than for somebody’s synthetic haystack.
Top comments (0)