DEV Community: Anchal Nigam

Why Can’t We Just Give an LLM All Our Company Documents? Understanding the Context Window

Anchal Nigam — Wed, 28 Oct 2026 10:08:32 +0000

Imagine your company has 1 lakh documents:

Engineering docs
Customer conversations
Incident reports
Product specifications
HR policies
Sales reports
Meeting notes
PDFs

Now you build an AI assistant and someone asks:

"Why did our payment service fail in Germany last November?"

You might think:

"Why do we need RAG or a vector database? Why not just give the LLM all 1 lakh documents and let it figure it out?"

It sounds reasonable. After all, newer LLMs can process surprisingly large amounts of text. But there is a difference between:

"The model can accept this much information."

and:

"The model can efficiently and reliably use all of this information."

That difference is where things get interesting.

What Happens When You Give Text to an LLM?

When you read a document, you see paragraphs, pages, sections, and diagrams. An LLM doesn't. It first converts everything into tokens.

Conceptually:

"The payment service failed"
           ↓
["The", " payment", " service", " failed"]

A long document becomes:

Document
   ↓
Text
   ↓
Tokens
   ↓
Token IDs

This matters because LLMs measure their capacity in tokens, not pages or documents.

The Context Window: An LLM's Working Space

Every LLM has a context window. Think of it as the model's working space for a single request.

Everything must fit inside it:

But let's make things more interesting.

Imagine we have a model with a 1 million token context window. Now everything fits.

Problem solved?

Not quite.

We've solved the capacity problem. We haven't solved the processing problem.

How Does an LLM Understand Context?

Consider this sentence:

The bank rejected the payment because the card had expired.

To understand the sentence, the model needs to connect:

             rejected
            /        \
         bank       payment
                       |
                    because
                       |
                 card expired

The words matter. But the relationships between the words matter even more. This is where the Transformer's attention mechanism comes in.

Attention: How the Model Decides What Matters

A useful way to think about attention is:

"Which other pieces of information should matter to me right now?"

Every token looks at other tokens and determines which ones are useful.

Technically, attention uses:

Query (Q)
Key (K)
Value (V)

Think of them as:

Q (Query) — "What am I looking for?"
K (Key) — "What do I contain?"
V (Value) — "What information do I carry?"

A Tiny Example

Take:

The red apple is on the table.

Suppose we're processing apple.

Very roughly, apple asks: "What helps describe me?"

Nearby words respond:

Token	Role
The	grammatical information
red	descriptive/color information
apple	noun/entity information
table	location/object information

The model computes how well the query matches each key. Simplified illustrative scores:

apple's Query × Key:
  The   → 1.2
  red   → 8.5  ← strong match
  apple → 4.1
  table → 0.8

These aren't actual values from a real model — they're a simple illustration.

These scores are then converted into attention weights that add up to 1:

The   →  5%
red   → 80%
apple → 15%
table →  0%

The model then uses those weights to pull information from the corresponding Values. So the important idea is:

Mathematically, the core operation looks like:

Attention(Q, K, V) = softmax((QKᵀ) / √dₖ) V

The important part is the flow: Q and K decide how relevant something is; V carries the information that gets passed along.

Think of attention as a very large system of information routing between tokens — much closer to what's happening than simply saying "the LLM reads the sentence."

What Changes When the Context Gets Huge?

Our example had only a few words. Now imagine:

10 tokens
     ↓
1,000 tokens
     ↓
10,000 tokens
     ↓
100,000 tokens
     ↓
1,000,000 tokens

Suddenly there are far more relationships to consider. Think about a networking event:

10 people → Easy to figure out who's important.
100 people → Still manageable.
10,000 people → Finding the right person becomes much harder.

The LLM faces a related scaling problem when processing long sequences. In standard self-attention, the number of token-to-token interactions grows roughly with the square of sequence length.

Very roughly:

10K tokens  →  N amount of attention work
20K tokens  →  ~4N
40K tokens  →  ~16N

Modern LLMs use many optimizations and architectural techniques, so these numbers should not be interpreted as the literal cost of every modern model.

The important intuition:

A larger context doesn't just mean more text to read. It also means many more relationships between tokens for the model to consider. As context grows, the amount of information the model needs to connect and reason over can grow much faster than the context itself.

This is one reason long-context inference becomes expensive.

The Bigger Problem Isn't Capacity

Let's return to our 1 lakh documents.

Suppose they contain 1,000,000 tokens — and only 10,000 tokens are actually relevant to the user's question.

┌───────────────────────────────┐
│                               │
│   Mostly irrelevant content   │
│                               │
│      █ Relevant Data █        │
│                               │
│   Mostly irrelevant content   │
│                               │
└───────────────────────────────┘

The model technically has access to everything. But why make it process 1 million tokens when only 10,000 matter?

Why Bigger Context Isn't Free

Consider two requests:

Request A — focused context

Question              500 tokens
Relevant Context    5,000 tokens
───────────────────────────────
Total               5,500 tokens

Request B — everything

Question                  500 tokens
Entire Knowledge Base  500,000 tokens
────────────────────────────────────
Total                  500,500 tokens

Request B has almost 100× more input.

The model now has to process a huge amount of additional information before it can answer. And this is important:

The LLM doesn't first magically know which 10,000 tokens matter and then ignore the rest.

The model receives the context and runs its processing over that context. If we want to reduce the amount of irrelevant information, we generally need something outside that basic generation step to help select it. That's one reason retrieval is useful. But before we get to RAG, there’s another cost.

What Happens Before the LLM Starts Answering?

When you send a long prompt, the model first processes the input. This initial stage is often called prefill.

Think of it like handing someone a 500-page report and saying:

"Read this before answering my question."

The person can’t start answering until they’ve processed the material they need. The same idea applies to LLMs.

That initial processing contributes to time-to-first-token — how long you wait before the first piece of the answer appears.

Short context → less input to process → potentially faster start
Long context  → more input to process → potentially slower start

This is one reason a huge context window shouldn't be interpreted as:

"Send huge prompts everywhere."

A capability can be useful without being free.

The KV Cache: The Notes the Model Keeps

While generating text, the model needs information from earlier tokens. Recomputing everything repeatedly would be wasteful. So the model stores useful intermediate information — this is called the KV Cache.

Think of it like taking notes while reading a book. Instead of reopening page 1 every time, you keep notes on your desk.

Long Context
      ↓
 Process
      ↓
   KV Cache
      ↓
Reuse During Generation

This makes generation faster. But there's a tradeoff:

Larger contexts require larger caches — and larger caches consume more memory. This becomes a major challenge when serving thousands of users simultaneously.

Are Larger Context Windows a Bad Thing?

No. This distinction is extremely important.

A larger context window is genuinely useful. Suppose you're debugging a complicated service. You want to provide:

20K tokens of code
10K tokens of logs
5K tokens of architecture docs

Having a model that can consider all of this together can be extremely useful. Or you're analyzing a long contract, comparing several reports, or working through a long conversation. In these cases, a larger context window gives you something valuable:

The ability to keep more relevant information in the model's working context at once.

A larger context window is a capability, not a guarantee of better answers.

The Surprising Part: More Context Can Hurt Performance

Now imagine the answer to our question is buried somewhere in the middle of thousands of documents:

Document 1
Document 2
Document 3
…
Document 49,999
★ IMPORTANT INCIDENT REPORT ★
Document 50,001
Document 50,002
…
Document 100,000

The information is technically there. But can the model reliably find and use it?

A well-known study — "Lost in the Middle: How Language Models Use Long Contexts" — tested exactly this kind of behavior. Researchers evaluated models on tasks such as multi-document question answering and key-value retrieval.

They found that performance could degrade substantially depending on where the relevant information appeared in a long context. Models often performed better when the useful information was near the beginning or end, and worse when it was buried in the middle. This became known as the lost-in-the-middle problem.

The basic lesson is simple:

Having information available is not the same as using it effectively.

Information ≠ Useful Context

Suppose I ask:

"What is our refund policy?"

And I give the model:

5 relevant paragraphs
50,000 unrelated paragraphs

Did I give the model more information? Yes.

Did I give it better context? No.

I've actually created another job for the model: "Figure out which parts of this enormous pile of information actually matter." And that's before it even answers my original question.

More context ≠ automatically better context.

The goal isn't to maximize the amount of context. The goal is to maximize the amount of useful context.

Why Are Context Windows Getting Bigger?

Because larger context windows are genuinely useful. Think of RAM in a computer — more RAM is helpful. But:

More RAM doesn't automatically make every program better. It simply allows larger workloads when needed.

The same applies to context windows.

Bringing It All Together

Why can't we simply give an LLM all 1 lakh company documents?

They may not fit — Too many documents → too many tokens.
Most of them are irrelevant — Only a small portion may be relevant to the specific question.
More context means more computation — The model has more information to process.
More context can increase latency — Large prompts take longer to process.
More context costs more — More tokens generally mean more compute.
More context doesn't guarantee better understanding — The model can have access to information and still struggle to use it effectively.
Why RAG Exists

Now the model isn't being asked:

"Read the entire company."

It's being asked:

"Here are the pieces of information relevant to this question. Use them to answer."

And that's the real idea behind RAG.

The Mental Model to Remember

If you remember only one thing from this article, remember this:

A context window tells you how much information an LLM can accept. It doesn't tell you how much information you should give it.

The goal isn't maximum context. The goal is maximum useful context.

What's Next?

Now we know the goal isn't to give an LLM everything. It's to give it the right information.

But that leads to the next question: When you have thousands or even millions of documents, how do you figure out which pieces of information are actually relevant to a user's question?

Something has to find that relevant information before it reaches the LLM. And that's where retrieval comes in. But retrieval is more than simply searching for a document.

So, how does retrieval actually work? That's what we'll explore in the next article.

What RAG Actually Is (And Why It’s More Than Search + LLM)

Anchal Nigam — Wed, 21 Oct 2026 09:01:20 +0000

In the previous article, we explored why Large Language Models hallucinate and why RAG exists.

We learned that LLMs don’t magically know everything. Their knowledge is limited to what they learned during training, and that knowledge eventually becomes outdated. RAG helps solve this problem by providing relevant external information at the time of answering a question.

But that raises another question - What actually happens inside a RAG system?

When people talk about RAG, terms like chunking, embeddings, vector databases, hybrid search, reranking, and evaluations often appear together. For someone starting their journey, it can feel overwhelming.

So before diving into each of those topics, let’s first understand the big picture.

What Does RAG Actually Mean?

RAG stands for: Retrieval + Augmentation + Generation

Each word represents a step in the process.

Retrieval — Find information relevant to the user’s question.

Augmentation — Add the retrieved information to the prompt or context sent to the model.

Generation — The model uses that context to generate a response.

At a very high level, the flow looks like this:

The Key Idea Behind RAG

One important thing often gets misunderstood. RAG does not teach new knowledge to the model. The model itself remains unchanged. Instead, RAG changes the information available during generation.

Think about asking a colleague a question. If they don’t have the required information, they may guess. If you provide them with the relevant document before they answer, the quality of the answer improves dramatically.

Their intelligence didn’t change. The information available to them changed.

RAG follows the same principle.

RAG Architecture

At first, the previous diagram made RAG look surprisingly simple. Retrieve some information. Attach it to the prompt. Generate an answer. Done.

But when I started exploring production-grade RAG systems, I realized there was much more happening behind the scenes. Most real-world systems consist of two major pipelines:

Ingestion Pipeline
Retrieval (Inference) Pipeline

Let’s look at both.

The Ingestion Pipeline

Before users can ask questions, the system must prepare its knowledge base.

A simplified ingestion pipeline looks like this:

Documents -> Chunking -> Embeddings -> Searchable Storage

Documents

These can be PDFs, support articles, product catalogs, internal documentation, knowledge-base articles, release notes, or any other source of information.

Chunking

Large documents are split into smaller pieces so that relevant information can be retrieved efficiently.

Embeddings

Each chunk is converted into a machine-understandable representation that can later be searched based on meaning.

Searchable Storage

The processed chunks are stored in a system optimized for retrieval.

At this point, the knowledge is ready to be used.

The Retrieval Pipeline

Now imagine a user asks a question. The retrieval pipeline comes into action.

User Question -> Retrieve Relevant Context ->Prompt Construction ->LLM -> Response

In production systems, however, several additional steps often exist between these stages. Questions may be transformed. Results may be filtered. Retrieved content may be reranked. Citations may be attached. Responses may be evaluated for quality and faithfulness.

A more realistic view looks something like this.

The Part That Surprised Me

At this point, I felt I finally understood RAG. Then every box in the architecture started raising new questions.

Why do we need chunking?
What exactly are embeddings?
Why are vector databases everywhere?
Why isn’t a traditional database enough?
Why do production systems use hybrid search?
Why do people talk about rerankers?
And if all the required information already exists, why do retrieval systems still fail sometimes?

The deeper I went, the more I realized that RAG is not simply “search + AI.” Each component exists because it solves a specific problem.

Next Question

While studying these architectures, one question kept coming back to me. If the goal is to provide context to the model, why don’t we simply give it all the available documents? Why retrieve only a small subset? Why not let the model read everything and figure out what’s relevant on its own? It sounds like a reasonable idea. In fact, that was my first thought as well.

As I learned later, this approach breaks surprisingly fast because of context-window limitations, cost, latency, and relevance challenges. And understanding those limitations is exactly what led me to the next topic in this series.

Why Can’t We Just Give the LLM All the Available Context Instead of Retrieving Only What’s Relevant?

That’s what we’ll explore in the next article.

Why LLMs Hallucinate — And Why RAG exists

Anchal Nigam — Mon, 12 Oct 2026 05:30:00 +0000

I remember the first time I felt something was off.

I asked an LLM to list research papers on a niche topic. The response came instantly — paper titles, author names, publication years, even DOI links.

Everything looked legitimate: structured, academic, confident. So, I searched for some of the papers online.

A few of them simply didn’t exist.

The titles sounded real. The author names looked believable. Even the citations followed proper academic formatting. But the information itself had been hallucinated.

Not obviously fake or broken. Just convincingly incorrect. And that’s a very uncomfortable place to be as an engineer because now the problem isn’t failure — it’s trust.

🤔 “But LLMs are supposed to be smart… right?”

That’s what I thought too. We hear things like:

“LLMs understand language”
“They reason like humans”
“They can answer anything”

So when they generate something incorrect, our instinct is usually:

Maybe the model just isn’t advanced enough yet.

But here’s the uncomfortable truth:

LLMs don’t store and retrieve knowledge the way humans do.

At a very basic level, an LLM is just doing one thing:

Predicting the next token.

They generate responses based on patterns learned during training and probabilities of what is most likely to come next given the context.

That means:

they don’t inherently verify truth,
they don’t automatically know your data,
and they don’t “look up” facts while generating responses.

And that distinction explains a lot.

⚠️ Where do hallucinations come from?

Think about the research paper example again.

The response looked incredibly convincing because the model had probably already seen:

thousands of research papers
academic citations
DOI formats
author naming patterns

So even when it didn’t actually know the answer, it still tried to generate the most statistically plausible continuation. That’s why:

fake paper titles started sounding real,
imaginary authors looked believable,
and nonexistent citations appeared academically correct.

The model wasn’t verifying truth. And that’s exactly why hallucinations become dangerous. Because the responses often sound:

fluent,
well-structured,
authoritative,
and highly confident.

Humans naturally associate confidence with correctness. So over time, fake information starts looking credible.

🧩 How do we fix this?

At this point, I started wondering “If LLMs are just predicting statistically probable text, how do we make them generate more accurate answers?”

Because the model itself hasn’t changed. It’s still doing the same thing — Predicting the next most probable words. So maybe the real issue wasn’t intelligence.

Maybe the issue was context.

🚧 The Actual Limitation

The model generated fake citations because it had no way to verify those citations against an authoritative source while answering. It was trying to predict answers with incomplete grounding. And that exposes one of the biggest limitations of LLMs today.

They:

don’t automatically have access to real-time information
don’t automatically have access to your company/private data
can’t access your documents unless you provide them

Your PDFs, databases, Slack discussions, APIs, internal knowledge are invisible to the model by default.

💡 The missing piece

So what if, instead of expecting the model to magically know everything, we first give it the right information before generation?

Think of it like asking a friend something from memory. Without context, they’ll answer based on vague recollection. But if they quickly open the actual document before answering, the quality of the response changes completely.

Your friend didn’t suddenly become smarter. They just got access to better context.

🚀 And this is the key insight

When we add relevant information into the prompt, we change what the model sees before predicting the next word. And that changes the direction of generation itself. In simple words:

Better context → better prediction.

This leads to something important: not a magical new model, not some hidden intelligence upgrade, but a system design shift:

Retrieval-Augmented Generation (RAG)

At its core, RAG is simply a smarter way of grounding generation using relevant external information before the model starts predicting.

What’s next?

In the next article, we’ll finally break this down properly:

What RAG actually is
How retrieval changes generation
Why “just adding a vector DB” is not enough
How modern AI systems retrieve useful context

🏁 Final thought

LLMs don’t hallucinate because they’re broken.

They hallucinate because they can generate answers that sound convincing, but they don’t automatically verify whether those answers are actually true. And until we give LLMs a way to ground their answers in reliable information, we’ll keep getting answers that sound right — but aren’t.

Next: “What RAG actually is?”