Imagine your company has 1 lakh documents:
- Engineering docs
- Customer conversations
- Incident reports
- Product specifications
- HR policies
- Sales reports
- Meeting notes
- PDFs
Now you build an AI assistant and someone asks:
"Why did our payment service fail in Germany last November?"
You might think:
"Why do we need RAG or a vector database? Why not just give the LLM all 1 lakh documents and let it figure it out?"
It sounds reasonable. After all, newer LLMs can process surprisingly large amounts of text. But there is a difference between:
"The model can accept this much information."
and:
"The model can efficiently and reliably use all of this information."
That difference is where things get interesting.
What Happens When You Give Text to an LLM?
When you read a document, you see paragraphs, pages, sections, and diagrams. An LLM doesn't. It first converts everything into tokens.
Conceptually:
"The payment service failed"
↓
["The", " payment", " service", " failed"]
A long document becomes:
Document
↓
Text
↓
Tokens
↓
Token IDs
This matters because LLMs measure their capacity in tokens, not pages or documents.
The Context Window: An LLM's Working Space
Every LLM has a context window. Think of it as the model's working space for a single request.
Everything must fit inside it:
But let's make things more interesting.
Imagine we have a model with a 1 million token context window. Now everything fits.
Problem solved?
Not quite.
We've solved the capacity problem. We haven't solved the processing problem.
How Does an LLM Understand Context?
Consider this sentence:
The bank rejected the payment because the card had expired.
To understand the sentence, the model needs to connect:
rejected
/ \
bank payment
|
because
|
card expired
The words matter. But the relationships between the words matter even more. This is where the Transformer's attention mechanism comes in.
Attention: How the Model Decides What Matters
A useful way to think about attention is:
"Which other pieces of information should matter to me right now?"
Every token looks at other tokens and determines which ones are useful.
Technically, attention uses:
- Query (Q)
- Key (K)
- Value (V)
Think of them as:
- Q (Query) — "What am I looking for?"
- K (Key) — "What do I contain?"
- V (Value) — "What information do I carry?"
A Tiny Example
Take:
The red apple is on the table.
Suppose we're processing apple.
Very roughly, apple asks: "What helps describe me?"
Nearby words respond:
| Token | Role |
|---|---|
| The | grammatical information |
| red | descriptive/color information |
| apple | noun/entity information |
| table | location/object information |
The model computes how well the query matches each key. Simplified illustrative scores:
apple's Query × Key:
The → 1.2
red → 8.5 ← strong match
apple → 4.1
table → 0.8
These aren't actual values from a real model — they're a simple illustration.
These scores are then converted into attention weights that add up to 1:
The → 5%
red → 80%
apple → 15%
table → 0%
The model then uses those weights to pull information from the corresponding Values. So the important idea is:
Mathematically, the core operation looks like:
Attention(Q, K, V) = softmax((QKᵀ) / √dₖ) V
The important part is the flow: Q and K decide how relevant something is; V carries the information that gets passed along.
Think of attention as a very large system of information routing between tokens — much closer to what's happening than simply saying "the LLM reads the sentence."
What Changes When the Context Gets Huge?
Our example had only a few words. Now imagine:
10 tokens
↓
1,000 tokens
↓
10,000 tokens
↓
100,000 tokens
↓
1,000,000 tokens
Suddenly there are far more relationships to consider. Think about a networking event:
- 10 people → Easy to figure out who's important.
- 100 people → Still manageable.
- 10,000 people → Finding the right person becomes much harder.
The LLM faces a related scaling problem when processing long sequences. In standard self-attention, the number of token-to-token interactions grows roughly with the square of sequence length.
Very roughly:
10K tokens → N amount of attention work
20K tokens → ~4N
40K tokens → ~16N
Modern LLMs use many optimizations and architectural techniques, so these numbers should not be interpreted as the literal cost of every modern model.
The important intuition:
A larger context doesn't just mean more text to read. It also means many more relationships between tokens for the model to consider. As context grows, the amount of information the model needs to connect and reason over can grow much faster than the context itself.
This is one reason long-context inference becomes expensive.
The Bigger Problem Isn't Capacity
Let's return to our 1 lakh documents.
Suppose they contain 1,000,000 tokens — and only 10,000 tokens are actually relevant to the user's question.
┌───────────────────────────────┐
│ │
│ Mostly irrelevant content │
│ │
│ █ Relevant Data █ │
│ │
│ Mostly irrelevant content │
│ │
└───────────────────────────────┘
The model technically has access to everything. But why make it process 1 million tokens when only 10,000 matter?
Why Bigger Context Isn't Free
Consider two requests:
Request A — focused context
Question 500 tokens
Relevant Context 5,000 tokens
───────────────────────────────
Total 5,500 tokens
Request B — everything
Question 500 tokens
Entire Knowledge Base 500,000 tokens
────────────────────────────────────
Total 500,500 tokens
Request B has almost 100× more input.
The model now has to process a huge amount of additional information before it can answer. And this is important:
The LLM doesn't first magically know which 10,000 tokens matter and then ignore the rest.
The model receives the context and runs its processing over that context. If we want to reduce the amount of irrelevant information, we generally need something outside that basic generation step to help select it. That's one reason retrieval is useful. But before we get to RAG, there’s another cost.
What Happens Before the LLM Starts Answering?
When you send a long prompt, the model first processes the input. This initial stage is often called prefill.
Think of it like handing someone a 500-page report and saying:
"Read this before answering my question."
The person can’t start answering until they’ve processed the material they need. The same idea applies to LLMs.
That initial processing contributes to time-to-first-token — how long you wait before the first piece of the answer appears.
Short context → less input to process → potentially faster start
Long context → more input to process → potentially slower start
This is one reason a huge context window shouldn't be interpreted as:
"Send huge prompts everywhere."
A capability can be useful without being free.
The KV Cache: The Notes the Model Keeps
While generating text, the model needs information from earlier tokens. Recomputing everything repeatedly would be wasteful. So the model stores useful intermediate information — this is called the KV Cache.
Think of it like taking notes while reading a book. Instead of reopening page 1 every time, you keep notes on your desk.
Long Context
↓
Process
↓
KV Cache
↓
Reuse During Generation
This makes generation faster. But there's a tradeoff:
Larger contexts require larger caches — and larger caches consume more memory. This becomes a major challenge when serving thousands of users simultaneously.
Are Larger Context Windows a Bad Thing?
No. This distinction is extremely important.
A larger context window is genuinely useful. Suppose you're debugging a complicated service. You want to provide:
- 20K tokens of code
- 10K tokens of logs
- 5K tokens of architecture docs
Having a model that can consider all of this together can be extremely useful. Or you're analyzing a long contract, comparing several reports, or working through a long conversation. In these cases, a larger context window gives you something valuable:
The ability to keep more relevant information in the model's working context at once.
A larger context window is a capability, not a guarantee of better answers.
The Surprising Part: More Context Can Hurt Performance
Now imagine the answer to our question is buried somewhere in the middle of thousands of documents:
Document 1
Document 2
Document 3
…
Document 49,999
★ IMPORTANT INCIDENT REPORT ★
Document 50,001
Document 50,002
…
Document 100,000
The information is technically there. But can the model reliably find and use it?
A well-known study — "Lost in the Middle: How Language Models Use Long Contexts" — tested exactly this kind of behavior. Researchers evaluated models on tasks such as multi-document question answering and key-value retrieval.
They found that performance could degrade substantially depending on where the relevant information appeared in a long context. Models often performed better when the useful information was near the beginning or end, and worse when it was buried in the middle. This became known as the lost-in-the-middle problem.
The basic lesson is simple:
Having information available is not the same as using it effectively.
Information ≠ Useful Context
Suppose I ask:
"What is our refund policy?"
And I give the model:
- 5 relevant paragraphs
- 50,000 unrelated paragraphs
Did I give the model more information? Yes.
Did I give it better context? No.
I've actually created another job for the model: "Figure out which parts of this enormous pile of information actually matter." And that's before it even answers my original question.
More context ≠ automatically better context.
The goal isn't to maximize the amount of context. The goal is to maximize the amount of useful context.
Why Are Context Windows Getting Bigger?
Because larger context windows are genuinely useful. Think of RAM in a computer — more RAM is helpful. But:
More RAM doesn't automatically make every program better. It simply allows larger workloads when needed.
The same applies to context windows.
Bringing It All Together
Why can't we simply give an LLM all 1 lakh company documents?
- They may not fit — Too many documents → too many tokens.
- Most of them are irrelevant — Only a small portion may be relevant to the specific question.
- More context means more computation — The model has more information to process.
- More context can increase latency — Large prompts take longer to process.
- More context costs more — More tokens generally mean more compute.
- More context doesn't guarantee better understanding — The model can have access to information and still struggle to use it effectively.
- Why RAG Exists
Now the model isn't being asked:
"Read the entire company."
It's being asked:
"Here are the pieces of information relevant to this question. Use them to answer."
And that's the real idea behind RAG.
The Mental Model to Remember
If you remember only one thing from this article, remember this:
A context window tells you how much information an LLM can accept. It doesn't tell you how much information you should give it.
The goal isn't maximum context. The goal is maximum useful context.
What's Next?
Now we know the goal isn't to give an LLM everything. It's to give it the right information.
But that leads to the next question: When you have thousands or even millions of documents, how do you figure out which pieces of information are actually relevant to a user's question?
Something has to find that relevant information before it reaches the LLM. And that's where retrieval comes in. But retrieval is more than simply searching for a document.
So, how does retrieval actually work? That's what we'll explore in the next article.
Top comments (1)
if only 10,000 of the million tokens matter, why pay to process the rest?