Claude Opus 4.6 has a 1 million token context window. Gemini 2.5 Pro supports up to 1 million tokens. GPT-5 offers 256K. The numbers keep going up, and the marketing keeps implying that bigger is better. "Feed it your entire codebase!" "Drop in all your documents!"
I've been doing exactly that for months. And I'm here to tell you: large context windows are one of the most misunderstood features in AI right now. Bigger context doesn't mean better results. Often, it means worse results at higher cost.
The Research Nobody Reads
There's a growing body of research on how LLMs actually use long context. The most cited finding — sometimes called the "lost in the middle" effect — showed that models perform best on information at the beginning and end of the context, with significant degradation for content in the middle.
More recent work has refined this. A 2025 study from Microsoft Research found that effective utilization of context drops to roughly 60% beyond 100K tokens. That means if you stuff 500K tokens into a prompt, the model is effectively ignoring or poorly integrating about 200K tokens worth of information. You're paying for tokens the model isn't using well.
# Simulating the utilization curve (based on published research)
import numpy as np
def effective_utilization(context_length_tokens):
if context_length_tokens <= 32_000:
return 0.95 # Near-perfect utilization
elif context_length_tokens <= 100_000:
return 0.85 # Slight degradation
elif context_length_tokens <= 500_000:
return 0.60 # Significant drop
else:
return 0.45 # Diminishing returns
# What you pay for vs what you get:
# 32K context: paying for 32K, effectively using ~30K
# 200K context: paying for 200K, effectively using ~120K
# 1M context: paying for 1M, effectively using ~450K
This doesn't mean the model fails completely on long context. It means the quality of retrieval and reasoning degrades non-linearly. The model might correctly recall a fact from position 800K but miss a crucial detail at position 400K.
When Large Context Actually Helps
Large context windows aren't useless. They're powerful for specific use cases. The problem is people use them for everything.
Code review of a complete module. If you need to review 20 files that form a cohesive module — say, 50K tokens total — feeding the entire module into context works well. The model can see how files relate to each other, catch inconsistencies, and understand the architecture. This is a genuine win for large context.
Document Q&A with structured sources. If you have a 200-page technical document and you need to answer specific questions about it, large context with clear document structure (headers, sections, page numbers) works reasonably well. The structure gives the model anchors to navigate by.
Long conversation memory. Keeping the full conversation history in context means the model doesn't forget what you discussed 30 messages ago. For complex multi-session projects, this is valuable.
When Large Context Hurts
Here's where people get burned.
The "dump everything" antipattern. The most common misuse is throwing an entire codebase — 500 files, 800K tokens — into context and asking a question. The model has so much irrelevant information to sift through that it frequently misses the relevant parts or hallucinates connections between unrelated files.
I tested this systematically. Same question, same codebase. One approach: full 400K token codebase dump. Other approach: relevant 15K tokens selected manually.
Question: "Why does the payment webhook sometimes process duplicate events?"
Full codebase context (400K tokens):
- Response focused on the wrong webhook handler
- Mentioned idempotency but pointed to the email service, not payments
- Confidence was high but answer was wrong
- Cost: ~$4.80 in API fees
Targeted context (15K tokens):
- Correctly identified the race condition in webhook_handler.py
- Pointed to the missing Redis lock in process_payment_event()
- Suggested specific fix with code
- Cost: ~$0.18 in API fees
The targeted approach was cheaper, faster, and correct. The full-context approach was expensive, slow, and wrong. This pattern repeats consistently in my testing.
Cross-document reasoning degrades. When you need the model to synthesize information from multiple documents in a large context, accuracy drops sharply. The model is good at finding a needle in a haystack. It's bad at finding five needles and weaving them into a coherent narrative.
Latency scales linearly (or worse). More tokens in context means longer time to first token. For a 1M token context, you might wait 30-60 seconds before the model starts generating. In an interactive workflow, that latency kills productivity.
The Right Strategy
The answer isn't "avoid large context." It's "use large context strategically."
Retrieve, then reason. Use RAG (Retrieval Augmented Generation) or manual selection to pull the most relevant content into a smaller context window. A well-curated 30K token context almost always outperforms a kitchen-sink 500K token context.
Structure your context. If you must use large context, structure it clearly. Use XML tags, headers, file boundaries, and explicit labels. Give the model a map of what's where.
<!-- Good: Structured context with clear boundaries -->
<context>
<file path="src/payments/webhook_handler.py" relevance="primary">
... code here ...
</file>
<file path="src/payments/models.py" relevance="supporting">
... code here ...
</file>
<file path="tests/test_webhooks.py" relevance="reference">
... code here ...
</file>
</context>
<question>Why are payment webhooks processing duplicates?</question>
Chunk and summarize. For very large documents, summarize sections first, then dive deep into the relevant sections. Two passes with smaller context beats one pass with enormous context.
Monitor your costs. At $15 per million input tokens (a common price point for frontier models), a 1M token context costs $15 per query. If you're doing exploratory work and asking multiple questions, that adds up fast. A targeted approach at 30K tokens per query costs $0.45 — you can ask 33 questions for the price of one full-context query.
The Marketing vs Reality
Model providers want you to use large context windows because you pay per token. More tokens in, more revenue. The marketing emphasizes the capability — "feed it your entire codebase!" — without mentioning the degradation curve, the latency cost, or the financial cost.
Large context windows are a genuine technical achievement. The models really can process 1 million tokens. But "can" and "should" are different words. A car can go 150 mph. That doesn't mean you should commute at 150 mph.
Use the context window you need, not the context window you have. Your results will be better, your costs will be lower, and your latency will be tolerable. The 1M token window is there for when you genuinely need it — not as the default for every interaction.
Top comments (0)