Every frontier LLM in 2026 advertises a 1M-token context window, but RULER, MRCR v2, and NoLiMa scores prove that "advertised" and "effective" diverge by 30-60 points for multi-fact retrieval past 200K tokens. Gemini 3.1 Pro is the only model whose 1M window holds for single-needle retrieval; Claude Opus 4.6 leads multi-needle MRCR at 1M; GPT-5.5 wins single-needle precision; DeepSeek V4 Pro lands surprisingly close at one-thirteenth the cost. Pick your model by the retrieval shape of your job, not by the headline number.
The 1M-token context window is the megapixel race of LLMs: every spec sheet has one, almost no one's job actually uses it, and the models that score highest at 1M usually trail at 128K — which is where 95% of real workloads live.
Why advertised context windows are a lie (and what benchmarks measure instead)
Long-context benchmarks split into four categories, because "long context" isn't one task.
The first category is single-needle retrieval (Needle-in-a-Haystack, NIAH). You hide one fact in a long context and ask the model to find it. This is the easy version, and almost every frontier model scores above 90% even at 1M tokens. Single-needle scores are why vendors quote "perfect recall at 1M tokens" in launch posts. They don't reflect real work.
The second is multi-needle / multi-hop retrieval, captured by Google DeepMind's MRCR v2 benchmark with 8 needles at 1M tokens, and NVIDIA's RULER suite. This is closer to RAG-over-long-docs: find six facts, in different places, and combine them. Scores collapse here. Claude Opus 4.6 currently leads MRCR v2 8-needle at 1M with around 78%, roughly quadrupling Claude Sonnet 4.5's 18.5% on the same test; DeepSeek V4 Pro reportedly hits 83.5% on the single-needle variant of MRCR at 1M, surpassing Gemini 3.1 Pro's 76.3% on that variant.
The third is inference under indirection — Adobe Research's NoLiMa benchmark, evaluated at 64K context length, strips literal keyword overlap between the question and the planted needle so the model has to reason its way to the answer. Even GPT-4o drops from 99.3% baseline to 69.7% at 32K. The 2026 frontier hasn't published much further: most vendors quietly avoid NoLiMa because results are humbling.
The fourth is downstream task performance over long context — Princeton's HELMET benchmark spans RAG, ICL, re-ranking, summarization, and instruction-following at 128K. HELMET's headline finding is the one that matters most: synthetic tasks like NIAH don't predict downstream performance. A model can ace needle-in-a-haystack and still hallucinate when asked to summarize the same haystack.
The 200K-token cliff: what RULER and effective context actually show
NVIDIA's RULER benchmark gives the cleanest answer: most frontier models reliably use only 50-65% of their advertised context window for multi-hop work. For GPT-5.5, Claude Opus 4.7, and DeepSeek V4 Pro, that means an effective context closer to 200-400K tokens for multi-needle production workloads — not the 1M they advertise.
Concretely, the 2026 NIAH-2 results at 1M tokens look like this:
| Model | NIAH-2 @ 1M tokens | Context window | Input price /MTok |
|---|---|---|---|
| Gemini 3 Deep Think | 99% | 1M | (Pro tier) |
| GPT-5.5 | 96% | 1M | $5 |
| Claude Opus 4.7 | 89% | 1M | $5 |
| DeepSeek V4 Pro | 78% | 1M | $1.74 |
But on multi-needle MRCR v2 at 128K — the band most real workloads live in — the order flips:
| Model | MRCR v2 8-needle @ 128K |
|---|---|
| Claude Opus 4.6 | 93.0% |
| Claude Sonnet 4.6 | 84.9% |
| Gemini 3.1 Pro | 84.9% |
| GPT-5.5 | 74.0% (8-needle), 41.4% on a harder multi-hop variant |
| Gemini 3.1 Flash Lite | 60.1% |
| Claude Opus 4.6 (long-context multi-hop variant) | 46.9% |
The numbers don't sort the same way at the two scales. That's the point. A model that handles single-needle retrieval at 1M tokens can collapse on multi-needle MRCR at 128K, and vice versa.
Which model wins at which context length
Under 32K tokens. Every frontier model is fine. Pick by reasoning quality, latency, or price. Long-context benchmarks don't differentiate here — this is "regular LLM" territory.
32K to 128K. The sweet spot for one-shot agent loops and medium-document analysis. Claude Opus 4.6 leads multi-needle retrieval at 93.0% on MRCR v2, with Sonnet 4.6 and Gemini 3.1 Pro tied at 84.9% just behind. If your workload is "read this 200-page contract, answer six interrelated questions" — this is your zone, and Opus 4.6 is the per-token leader on pure accuracy. The downside is cost: Opus 4.7 sits at $5/$25 per million input/output tokens on the ofox model catalog.
128K to 256K. Gemini 3.1 Pro starts pulling ahead on cost-adjusted retrieval. It ties Claude Sonnet 4.6 at 84.9% on MRCR v2 8-needle at 128K and degrades more gracefully past it than the Anthropic line; the per-MTok price drops sharply: $2 input / $12 output for Gemini 3.1 Pro on ofox vs Opus 4.7's $5/$25. For multi-document analysis up to a few hundred thousand tokens, Gemini 3.1 Pro is the budget-aware default.
256K to 1M. Only Gemini 3.1 Pro stays production-ready for retrieval. Claude Opus 4.7's 1M window is generally available at standard $5/$25 per MTok pricing (no long-context premium), but multi-needle accuracy still drops noticeably past 256K — Anthropic's own MRCR scores for Opus 4.7 at 1M trail Opus 4.6's older numbers. GPT-5.5 holds well for single-fact retrieval at 1M (96% NIAH-2) but stumbles on the harder multi-needle test. DeepSeek V4 Pro is the unexpected budget contender: $1.74/$3.48 per MTok on ofox, 1M advertised context, and competitive single-needle MRCR scores. It's the right pick when cost matters more than the last 10% of accuracy.
Above 1M tokens. Nobody. Stop stuffing context. The accuracy/cost curve makes RAG with a reranker mathematically dominant past about 500K tokens of input, even before you account for latency.
Cost reality: long-context isn't free
The price difference between models compounds quickly at long context, because input dominates output for retrieval/summarization workloads.
For a 256K-token input job (single document analysis, ~500 tokens of output) the per-call cost lands roughly:
- DeepSeek V4 Pro: 256K × $1.74/M = $0.445
- Gemini 3.1 Pro: 256K × $2/M = $0.512
- GPT-5.5: 256K × $5/M = $1.28
- Claude Opus 4.7: 256K × $5/M = $1.28
- GPT-5.4 Pro: 256K × $30/M = $7.68
At 1M tokens, multiply by 4 — and Claude Opus 4.7 also charges a 5-minute prompt cache write at $6.25/M (cache reads at $0.5/M afterward, so if you reuse the same long context across many queries, the economics flip in Anthropic's favor). For RAG-style workloads where you reuse the same retrieved chunks, Anthropic's prompt caching at the 1-hour tier ($10/M write, $0.5/M read) is the biggest cost lever in long-context production work — bigger than which model you pick.
If you're running a high-volume agent loop, you've probably already realized that paying frontier prices for retrieval is wasteful. Hybrid routing patterns — route easy chunks through DeepSeek V4 Flash at $0.14/$0.28, escalate hard ones to Opus 4.7 — get you 80% of the quality at 5-10% of the cost. The ofox API gateway handles the routing without code changes since every model speaks an OpenAI-compatible interface.
How to actually pick: a 4-step decision tree
Long-context model selection in 2026 reduces to four questions, in this order:
How long is the longest input you'll realistically see in production, p99? If the answer is under 200K, stop reading benchmark blog posts about 1M — they don't apply to you. Pick by reasoning quality at 128K, which means Claude Opus 4.6 or 4.7 for nuance, Gemini 3.1 Pro for cost, GPT-5.5 for tool use.
Do you need to find multiple facts and reason across them, or just retrieve one? Single-needle: any frontier model. Multi-needle at 128K+: Claude Opus 4.6 leads. Multi-needle at 1M: you're past where benchmarks support a confident recommendation; consider chunking instead.
Are you sending the same long context to the model repeatedly? Then prompt caching dominates the math. Claude Opus 4.7's 1-hour cache at $0.5/MTok read makes repeated queries against a 500K-token document set the cheapest option on the market — but only if you actually hit the cache.
What's your error tolerance for "model missed a fact"? Compliance, legal, medical: stay under 128K and use Claude Opus 4.7. Internal tooling, code review, exploratory summarization: Gemini 3.1 Pro at 256K-1M is fine. High-volume agent loops where rough recall is acceptable: DeepSeek V4 Pro or Gemini 3.1 Flash Lite.
For comparison shopping across models without rewiring your code, every model in this article is available on the ofox unified API under one OpenAI-compatible endpoint. Swap model: "anthropic/claude-opus-4.7" for model: "google/gemini-3.1-pro-preview" and re-run your eval set — that's the only honest way to pick.
What changed in 2026 (and what didn't)
The single biggest shift since 2025: vendors stopped competing on advertised context window and started competing on effective context. Google's own MRCR v2 results admit Gemini's 1M window degrades past 256K. Anthropic shipped Opus 4.7's 1M context at standard pricing (no long-context premium) in April 2026 — but its own MRCR v2 multi-needle scores at 1M came in lower than Opus 4.6's. OpenAI no longer leads with context length in marketing.
What didn't change: the gap between synthetic and downstream long-context tasks. The HELMET paper's finding that NIAH doesn't predict real-world performance is still the most quoted result in the long-context literature, because the 2026 generation reproduced it exactly. Score 99% on needle-in-a-haystack at 1M; still hallucinate in a legal summary at 128K.
Production long-context isn't a model problem — it's an architecture problem. The team that picks Gemini 3.1 Pro and ships RAG with reranking will outperform the team that picks Opus 4.7 and stuffs 800K tokens of unstructured noise into every call, every time.
Originally published on ofox.ai/blog.
Top comments (0)