We've come a long way in this series.
In Part 1, we introduced Context Engineering and why it goes beyond writing better prompts.
In Part 2, we explored tokens, context windows and memory.
In Part 3, we looked at how AI applications get information through RAG, Tool Calling and MCP.
And in Part 4, we explored how AI agents use context and tools to execute multi-step tasks.
Now there's a problem.
A problem that becomes increasingly important as your AI application becomes more capable.
Imagine an agent working on a task for 30 minutes.
During that time, it:
- Reads 20 files.
- Calls 15 tools.
- Retrieves dozens of documents.
- Receives thousands of lines of tool output.
- Makes several decisions.
- Encounters and resolves errors.
- Maintains conversation history.
Eventually, you could end up with something like:
System instructions
+
Conversation history
+
Tool definitions
+
Retrieved documents
+
Tool results
+
Agent state
+
Memory
+
Current request
↓
LLM
You might think:
"Great. The model has everything it needs."
But there's a catch.
Having everything doesn't necessarily mean having the right context.
And that's what Part 5 is about.
🎯 The Real Goal of Context Engineering
The goal isn't:
❌ Put as much information as possible into the context window.
It's:
✅ Build the smallest useful set of information required for the model to make the current decision correctly.
Think of context as a limited working budget.
You have a task.
You have a certain amount of useful information.
Your job is to decide:
What should stay?
What should go?
What should be summarized?
What should be retrieved later?
What should be cached?
What should never enter the context?
That's Context Engineering.
🚨 More Context Can Become a Problem
Consider this request:
"Why is the payment service returning 500?"
Suppose your system retrieves:
✓ Payment service logs
✓ Recent deployment
✓ Database errors
✓ Relevant code
✓ Configuration
✓ Recent incidents
✓ Unrelated authentication logs
✓ Old deployment logs
✓ 500 pages of documentation
✓ 20 previous conversations
Technically, the model has more information.
But did we improve the context?
Not necessarily.
We introduced:
- More tokens
- More latency
- More cost
- More irrelevant information
- More opportunities for conflicting information
The useful information can become harder to distinguish from everything else.
This is why:
Context quality matters more than context quantity.
Recent production guidance and research on context management similarly emphasize that long agent trajectories can accumulate irrelevant history and tool output, making selection and compression important engineering problems.
🧩 Five Ways to Optimize Context
A practical context pipeline usually has several levers:
Information
│
▼
┌───────────┐
│ Select │
└─────┬─────┘
│
▼
┌───────────┐
│ Retrieve │
└─────┬─────┘
│
▼
┌───────────┐
│ Compress │
└─────┬─────┘
│
▼
┌───────────┐
│ Cache │
└─────┬─────┘
│
▼
┌───────────┐
│ LLM │
└───────────┘
Let's understand each one.
1️⃣ Selection — Don't Send What You Don't Need
The cheapest token is often the token you never send.
Suppose an AI coding assistant is debugging:
payment-service
Does it need the entire repository?
Probably not.
It might only need:
payment/
├── handler.go
├── service.go
├── payment.go
└── payment_test.go
Instead of:
Entire Repository
↓
LLM
Use:
User Question
↓
Identify Relevant Files
↓
Only Relevant Files
↓
LLM
This is context selection.
And it can happen before compression.
Why compress information that shouldn't have been included in the first place?
💡 A Simple Rule
Before adding something to context, ask:
"Does the model need this information to make the current decision?"
If the answer is no, don't send it.
This sounds obvious.
But it's one of the easiest things to get wrong in real systems.
2️⃣ Retrieval — Fetch Information Just in Time
Part 3 introduced RAG.
Now we can look at RAG from a different perspective.
Instead of loading everything upfront:
User Request
↓
Load 1,000 Documents
↓
LLM
retrieve information when it is actually relevant:
User Request
↓
Understand Need
↓
Retrieve Relevant Information
↓
LLM
This is especially useful for large knowledge bases.
For example:
"What's our refund policy for enterprise customers?"
There is no reason to load:
- HR policies
- Engineering documentation
- Marketing documentation
- Old product specifications
The retrieval layer should narrow the information down.
🔍 Retrieval Is a Trade-Off
There is no universal rule that says:
"Always use RAG."
If you have a short document that fits comfortably into the context window and is directly relevant, retrieving chunks may add unnecessary complexity.
For example:
Document: 2 pages
User: "Summarize this document."
Just provide the document.
But if you have:
10 million documents
you obviously cannot send them all.
Retrieval becomes valuable.
The engineering question is:
What information should be available now and what can be fetched later?
3️⃣ Compression — Keep the Meaning, Remove the Bulk
Now let's say we really do need the information.
But the information is huge.
Consider a tool response:
GET /orders/12345
{
"id": 12345,
"customer": {...},
"shipping": {...},
"billing": {...},
"items": [...],
"auditLogs": [...],
"metadata": {...},
"internalDebugInformation": {...}
}
Maybe the agent only needs:
Order ID: 12345
Status: FAILED
Payment Status: DECLINED
Failure Reason: Insufficient funds
Why send the entire response?
We can transform:
Large Tool Output
↓
Relevant Information
↓
Compact Context
This is context compression.
🗜️ Compression Is Not the Same as Deletion
This distinction matters.
Suppose we have:
50 messages
and replace them with:
"The user wants to migrate PostgreSQL
from version 14 to 16 and cannot tolerate
more than 5 minutes of downtime."
We've reduced the number of tokens.
But we also potentially lost information.
Compression is therefore lossy unless the representation preserves everything required for the task.
Recent research on long-horizon agents specifically studies this trade-off: compressing accumulated interaction history can reduce memory requirements while trying to preserve task performance.
So don't blindly summarize everything.
Instead, identify what must survive.
🧠 What Should Survive Compression?
For an agent working on a task, a useful compact state might preserve:
Goal
Current progress
Important decisions
Constraints
Known failures
Important outputs
Pending actions
Relevant identifiers
For example:
Before
User:
We need to migrate the database.
Assistant:
Let's inspect the schema.
Tool:
500 lines...
Assistant:
There are 3 large tables.
Tool:
800 lines...
Assistant:
We should migrate table A first.
Tool:
...
Assistant:
Migration failed because...
...
After compaction
Goal:
Migrate PostgreSQL database.
Progress:
Tables A and B analyzed.
Decision:
Migrate table A first.
Constraint:
Maximum downtime: 5 minutes.
Failure:
Previous migration failed due to missing index.
Next step:
Create the required index and retry.
Much smaller.
But much more useful.
⚠️ Don't Summarize Blindly
This is one of the most important practical lessons.
A generic instruction like:
"Summarize everything above."
doesn't guarantee that the summary will preserve the information your application needs later.
Imagine the original context contains:
Maximum allowed downtime: 5 minutes
If the summary accidentally removes that constraint, the agent might make a technically valid but operationally dangerous decision.
So compression should have explicit preservation rules.
For example:
Always preserve:
- User requirements
- Hard constraints
- IDs
- Decisions
- Pending actions
- Important errors
- Security restrictions
This is much safer than generic summarization.
4️⃣ Caching — Don't Recompute What Hasn't Changed
Now let's talk about something different.
Suppose your application repeatedly sends a large, mostly stable piece of context:
System Instructions
+
Tool Definitions
+
Large Reference Information
+
User Request
If the first three parts rarely change, repeatedly processing them can be wasteful.
That's where prompt caching / provider-side caching mechanisms can help, depending on the model provider.
Conceptually:
┌─────────────────────────────┐
│ Stable Context │
│ │
│ System Instructions │
│ Tool Definitions │
│ Reference Information │
└──────────────┬──────────────┘
│
CACHE
│
▼
Current User Request
│
▼
LLM
The exact caching behavior, eligibility rules, pricing and cache lifetime depend on the provider.
So don't assume:
"Caching means the model remembers everything."
It doesn't.
Caching is an infrastructure optimization, not the same thing as memory.
🧠 Memory ≠ Cache ≠ Context
This is worth clarifying because these concepts are frequently mixed together.
Context
Information available to the model for the current inference.
Current request
+
Relevant history
+
Retrieved information
Memory
Information intentionally retained for future interactions.
User preference
+
Long-term facts
+
Previous task state
Cache
Previously processed/reusable information used to reduce repeated work, depending on the implementation.
Stable prompt prefix
+
Reusable computation
They solve different problems.
5️⃣ Structure — Context Isn't Just About What You Send
There's another subtle aspect:
How you organize the information matters.
Imagine giving an engineer this:
Database:
PostgreSQL
Issue:
Connection timeout
Logs:
...
User:
...
Instructions:
...
Relevant configuration:
...
Versus:
SYSTEM INSTRUCTIONS
TASK
CONSTRAINTS
RELEVANT EVIDENCE
TOOL RESULTS
CURRENT STATE
USER REQUEST
Structured context makes it easier for both developers and models to distinguish different types of information.
A useful pattern is to separate:
Stable information
+
Dynamic information
+
Retrieved information
+
Current task
This also makes your system easier to debug.
When something goes wrong, you can inspect which context component caused the problem.
💰 Token Optimization Is More Than "Use Fewer Tokens"
Suppose an agent executes 20 steps.
And each step sends:
10,000 input tokens
That's potentially:
20 × 10,000
=
200,000 input tokens
across the run.
Now imagine that 6,000 of those tokens were:
- Old tool outputs
- Repeated instructions
- Irrelevant history
- Duplicate documents
You may have spent tokens carrying information the model didn't need.
This is why optimizing an agent requires looking at the whole execution, not just one prompt.
Recent work on context optimization explicitly treats long-running agents as a lifecycle problem involving accumulated histories, tool outputs, retrieval and repeated model calls.
📊 Think in Terms of a Context Budget
Instead of asking:
"How much context can my model handle?"
ask:
"How much context should I spend on this task?"
For example:
Context Budget: 20,000 tokens
System Instructions 2,000
Tool Definitions 3,000
Conversation 2,000
Retrieved Documents 5,000
Agent State 2,000
Current Request 1,000
Safety / Constraints 1,000
--------------------------------
Total 16,000
You still have some room.
But if retrieval suddenly returns:
15,000 tokens
you have a problem.
A context budget makes that visible.
🧹 What Should We Remove First?
When context becomes too large, don't immediately summarize everything.
Use a hierarchy.
First: Remove unnecessary information
Irrelevant tool output
Old duplicate results
Unrelated documents
Then: Reduce verbose outputs
10,000-line log
↓
Relevant error lines
Then: Retrieve only what matters
100 documents
↓
Top relevant documents
Then: Compress history
50 conversation turns
↓
Task state + important decisions
Finally: Consider architectural changes
For example:
- Separate agent contexts
- Store large artifacts externally
- Retrieve information on demand
- Split complex tasks
This approach avoids using expensive compression when simple filtering would have solved the problem.
🧰 Keep Large Artifacts Outside the Context
This is particularly important for coding agents and document-heavy systems.
Suppose a tool generates:
50,000 lines of logs
Don't necessarily place all 50,000 lines into the next model request.
Instead:
Tool
↓
Large Artifact
↓
External Storage
↓
Relevant Extract
↓
LLM
The model can receive:
"Application crashed at 14:32:08.
Relevant stack trace:
..."
while the complete artifact remains available for retrieval if needed.
This pattern is especially useful when dealing with:
- Logs
- Large files
- Database exports
- Build artifacts
- Test reports
- Large API responses
🧠 Context Engineering Is a Selection Problem
At this point, we can define the problem more precisely.
Every model call has a potential information pool:
ALL AVAILABLE INFORMATION
│
┌────────────────┼────────────────┐
▼ ▼ ▼
Memory RAG Tools
│ │ │
└────────────────┼────────────────┘
▼
SELECT / RANK
│
▼
COMPRESS
│
▼
CONTEXT BUDGET
│
▼
LLM
The LLM doesn't need everything your system knows.
It needs the right working set for the current decision.
That's perhaps the most important idea in this entire series.
🧪 But How Do You Know Your Context Strategy Is Working?
This is where many AI projects stop too early.
They measure:
Did the model produce a good answer?
But that's not enough.
You should also ask:
Retrieval
- Did we retrieve the right documents?
- How often are irrelevant documents included?
Context
- How large is the context?
- Which components consume most tokens?
- How often are contexts compressed?
Agent
- How many steps does a task require?
- How often does the agent repeat a tool call?
- How often does it need to recover from an error?
Cost
- Input tokens per request
- Output tokens
- Tokens per successful task
- Cache utilization where applicable
Quality
- Task success rate
- Incorrect answers
- Missing information
- Failures after compression
A context optimization that saves 50% of tokens but causes 20% more task failures isn't necessarily an optimization.
⚖️ Optimize for Cost and Quality
This is an important mindset.
Don't optimize:
Tokens ↓↓↓
at any cost.
Optimize:
Quality
▲
│
│ ●
│ ●
│ ●
└────────────────►
Cost
The goal is to find a useful balance between:
- Accuracy
- Reliability
- Latency
- Cost
- Context size
Sometimes sending more context is absolutely worth it.
Sometimes it isn't.
Context Engineering is about making that decision intentionally.
🚨 Common Context Engineering Mistakes
❌ 1. "Let's send everything"
More context isn't automatically better.
❌ 2. "Let's summarize everything"
Compression can remove information that becomes important later.
❌ 3. "The model has a huge context window, so we're fine"
A large context window is a capability—not a reason to fill it unnecessarily.
❌ 4. "RAG will solve it"
Poor retrieval produces poor context.
RAG doesn't magically make irrelevant documents relevant.
❌ 5. "Just truncate the oldest messages"
The oldest information isn't necessarily the least important.
A previous decision or constraint may still matter.
❌ 6. "We'll optimize tokens later"
Context architecture can affect:
- Cost
- Latency
- Retrieval
- Memory
- Agent behavior
It becomes increasingly difficult to retrofit once the application grows.
🏗️ A Practical Context Pipeline
Putting everything together:
USER REQUEST
│
▼
┌──────────────────┐
│ Understand Task │
└────────┬─────────┘
│
┌────────────────┼────────────────┐
▼ ▼ ▼
Memory RAG Tools
│ │ │
└────────────────┼────────────────┘
▼
SELECT RELEVANT
│
▼
REMOVE NOISE
│
▼
COMPRESS IF NEEDED
│
▼
CONTEXT BUDGET
│
▼
LLM
│
▼
Evaluate Result
│
┌────────┴────────┐
│ │
Continue Done
│
▼
Next Agent Step
This is the mental model I'd keep.
🚀 A Simple Production Checklist
Before shipping an AI feature, ask:
Context
- [ ] What information does the model actually need?
- [ ] Which information is unnecessary?
- [ ] What is retrieved dynamically?
- [ ] What is persistent memory?
Tokens
- [ ] Do we have a context budget?
- [ ] Are large tool outputs filtered?
- [ ] Are duplicate documents removed?
- [ ] Are long histories compressed when appropriate?
Agents
- [ ] What is the maximum number of steps?
- [ ] What happens when a tool fails?
- [ ] What happens when the model chooses the wrong tool?
- [ ] Can the agent loop indefinitely?
Cost & Performance
- [ ] Are repeated stable inputs cacheable?
- [ ] Are large artifacts kept outside the context?
- [ ] Are we measuring tokens per successful task?
- [ ] Are we measuring latency?
Quality
- [ ] Does compression preserve important constraints?
- [ ] Are retrieved documents relevant?
- [ ] Are tool results trustworthy and validated?
- [ ] Do we have evaluations for context-related failures?
🧠 The Bigger Picture
Let's step back.
When we started this series, Context Engineering sounded like:
"Writing better context for an LLM."
After five parts, we can see that it's much broader.
A production AI system may need to decide:
What should the model see?
↓
What should it retrieve?
↓
What should it remember?
↓
What should it forget?
↓
What should be compressed?
↓
What should be cached?
↓
What should be fetched only when needed?
↓
What should the model act on?
↓
What should never enter the context?
That is a very different problem from simply writing a better prompt.
🎯 The Five-Part Mental Model
Let's bring the entire series together.
Part 1 — Context Engineering
Give the model the right information.
↓
Part 2 — Tokens, Context & Memory
Understand the constraints around that information.
↓
Part 3 — RAG, Tools & MCP
Retrieve information and connect the model to external capabilities.
↓
Part 4 — AI Agents
Let the system use that context and tools across multiple steps.
↓
Part 5 — Context Optimization
Continuously decide what information should enter, stay, change or leave the context.
And that gives us the bigger picture:
USER
│
▼
┌──────────┐
│ GOAL │
└────┬─────┘
│
▼
┌───────────────┐
│ Context Layer │
│ │
│ Memory │
│ Retrieval │
│ Tools │
│ History │
└───────┬───────┘
│
▼
Select / Rank
│
▼
Compress
│
▼
Context Budget
│
▼
LLM
│
▼
Agent
│
┌─────┴─────┐
▼ ▼
Tool Done
│
▼
New Context
│
└──────────►
And that loop can continue for as long as the task requires.
🏁 Final Thoughts
One of the biggest shifts happening in AI engineering is that we're moving away from thinking:
"How do I write the perfect prompt?"
and toward:
"How do I build the right information environment for the model at every step?"
That's Context Engineering.
The model matters.
The prompt matters.
But so do:
retrieval, memory, tools, context selection, compression, caching, state, evaluation and orchestration.
And as AI systems become more agentic, this problem becomes even more important.
Because the question isn't simply:
How much can the model see?
It's:
What should the model see right now?
That is the engineering problem.
📌 Key Takeaways
- More context isn't automatically better context.
- Treat the context window as a resource that needs to be managed.
- Remove irrelevant information before trying to compress it.
- Retrieve information when it's needed instead of loading everything upfront.
- Compress long histories carefully and preserve important decisions, constraints and state.
- Keep large artifacts outside the context when possible and retrieve only the relevant portions.
- Don't confuse context, memory and caching—they solve different problems.
- Set practical token budgets for agent runs.
- Measure task success, not just token reduction.
- Context optimization is a trade-off between quality, cost, latency and reliability.
- The goal isn't to give the model everything.
- The goal is to give the model what it needs for the decision it has to make right now.
Top comments (0)