Originally published on tamiz.pro.
The promise of billion-token context windows is seductive: feed your coding agent an entire monorepo, let it "see" everything, and watch it refactor like a senior engineer. But in practice, the opposite happens. Agents with massive contexts don't get smarter — they get dumber. They hallucinate file paths, re-derive already-solved problems, and lose track of the actual task. This is the billion-token context trap: more tokens don't equal more understanding.
The root cause isn't the model's capacity to read tokens; it's its inability to prioritize them. Attention mechanisms degrade gracefully at first, but as context grows, the signal-to-noise ratio collapses. Critical instructions buried in thousands of boilerplate lines get diluted. The model's "effective" context — the portion it actually attends to with high fidelity — is far smaller than the nominal window.
This article is a deep dive into why this happens and, more importantly, how to fix it. We'll explore three concrete, production-grade strategies: context compression, durable memory, and trace-preserving error codes. These aren't theoretical; they're patterns you can implement today to keep your coding agent focused, accurate, and reliable.
- 1. The Illusion of Infinite Context
- 2. Why Agents Forget: The Attention Dilution Problem
- 3. The Three Pillars: A Practical Framework
- 4. Pillar 1: Context Compression
- 5. Pillar 2: Durable Memory
- 6. Pillar 3: Trace-Preserving Error Codes
- 7. Putting It All Together: A Reference Architecture
- 8. Frequently Asked Questions
1. The Illusion of Infinite Context
When models like GPT-4 Turbo and Claude 2.1 crossed the 100K token mark, and then Gemini 1.5 Pro hit 1M, the AI coding community celebrated. The narrative was clear: if the model can see the entire codebase, it will finally understand the project. This is the context window fallacy.
Let's do the math. A typical enterprise monorepo with 500,000 lines of code, averaging 10 tokens per line, is roughly 5 million tokens. A 1M-token context window can hold a fifth of that. But even if you could fit it all, the model doesn't process it uniformly. Research on attention patterns (e.g., "Lost in the Middle" by Liu et al., 2023) shows that models attend most strongly to the beginning and end of a prompt, with a significant degradation in the middle. This is the "lost in the middle" phenomenon.
Key Insight: A 1M-token context window doesn't give you 1M tokens of useful memory. It gives you perhaps 200K-300K tokens of high-fidelity attention, with the rest acting as background noise that can actively mislead the model.
For coding agents, this is catastrophic. An agent that "forgets" the user's actual goal because it's buried in the middle of a massive prompt will produce confident but wrong code. The problem isn't the model's intelligence — it's the context's architecture.
2. Why Agents Forget: The Attention Dilution Problem
To understand why agents forget, we need to look at how transformers process long sequences. The self-attention mechanism computes a weighted sum of all tokens, where the weights represent relevance. In theory, this is perfect. In practice, softmax over thousands of tokens leads to attention entropy: the probability mass spreads thin.
Consider a simple analogy. You're at a cocktail party with 10 people. You can easily follow a conversation. Now imagine 10,000 people all talking at once. You can't follow anyone — you just hear noise. The transformer's attention heads face the same issue. When the context contains 500,000 tokens, each attention head must distribute its focus across all of them. The result is that no single token gets enough attention weight to be reliably processed.
The Effective Context Window
We can define an effective context window (ECW) as the number of tokens that actually receive meaningful attention weight. For most models, ECW is 20-40% of the nominal window. This isn't a hard limit; it's a statistical reality. Beyond the ECW, tokens become "ghost" tokens — they're present in the prompt, but their influence on the output is negligible or even harmful.
| Context Window (tokens) | Effective Context (approx.) | Risk of Degradation |
|---|---|---|
| 8K | 6K-7K | Low |
| 32K | 15K-20K | Moderate |
| 128K | 40K-60K | High |
| 1M | 200K-400K | Severe |
The Coding Agent Failure Modes
When an agent operates beyond its ECW, we see specific, identifiable failure modes:
- Goal Drift: The agent forgets the original task and starts "improving" unrelated code.
- Hallucinated Paths: It references files that don't exist, or invents function names that were never defined.
- Re-derivation: It solves a problem it already solved 50 turns ago, because that solution fell out of the attention window.
- Contradictory Edits: It makes a change, then later makes an opposite change, because it no longer remembers its own previous action.
These aren't model flaws; they're context architecture flaws. The fix isn't a bigger window — it's better context management.
3. The Three Pillars: A Practical Framework
To escape the trap, we need to stop treating the context window as a storage device and start treating it as a working set. Just as an operating system uses virtual memory, paging, and caching to manage RAM, an AI coding agent needs its own memory hierarchy. We'll build this with three pillars:
- Context Compression: Reduce the token footprint of each piece of information without losing its semantic content.
- Durable Memory: Persist critical facts and decisions outside the prompt, so they survive context truncation.
- Trace-Preserving Error Codes: Encode the agent's reasoning and execution history into compact, machine-readable tokens that can be re-injected when needed.
Together, these form a system where the agent's "memory" is not just the prompt, but a layered architecture of compressed, durable, and traceable state.
4. Pillar 1: Context Compression
Context compression is about representing more with less. The goal is to reduce the token count of the context while preserving the information the agent actually needs to make decisions. There are three primary techniques.
4.1 Semantic Compression via Summarization
The most straightforward approach is to summarize code and docs before injecting them into the prompt. Instead of dumping 5,000 lines of a module, you inject a 200-line summary that captures its public API, key invariants, and dependencies.
Example: A summarizer service
# context_compression/summarizer.py
from transformers import pipeline
class CodeSummarizer:
def __init__(self, model_name="facebook/bart-large-cnn"):
self.summarizer = pipeline("summarization", model=model_name)
def summarize_module(self, module_path: str, max_tokens: int = 200) -> str:
"""Summarize a code module into a compact description."""
with open(module_path) as f:
code = f.read()
# Truncate extremely long files before summarization
code = code[:100_000]
summary = self.summarizer(code, max_length=max_tokens, min_length=30, do_sample=False)[0]["summary_text"]
return summary
Why this works: A summarizer is a specialized model that has been trained to distill the meaning of code. It preserves the interface and the intent while discarding implementation noise. The agent gets the "what" and "why" without the "how."
Trade-off: Summaries can lose edge-case details. To mitigate, use a layered approach: always keep the most recently touched files in full, and only summarize files that are in the background.
4.2 Structural Compression via AST Filtering
Rather than summarizing text, you can parse the code into an Abstract Syntax Tree (AST) and filter out irrelevant nodes. This is more precise than summarization because it operates on the actual structure.
Example: AST-based pruning
# context_compression/ast_filter.py
import ast
class ASTFilter:
def __init__(self, relevant_symbols: set[str]):
self.relevant_symbols = relevant_symbols
def filter(self, code: str) -> str:
tree = ast.parse(code)
# Keep only functions/classes that are relevant or directly referenced
filtered_nodes = []
for node in ast.walk(tree):
if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
if node.name in self.relevant_symbols:
filtered_nodes.append(node)
# Reconstruct a minimal module from the filtered nodes
return self._reconstruct(filtered_nodes)
def _reconstruct(self, nodes):
module = ast.Module(body=nodes, type_ignores=[])
return ast.unparse(module)
Why this works: AST filtering removes comments, blank lines, and irrelevant functions. It's like a code minifier that preserves only the semantic core. The agent sees a much smaller, but structurally accurate, view.
Trade-off: You need to know which symbols are relevant ahead of time. This is where the durable memory layer comes in — it tracks which symbols are important.
4.3 Embedding-Based Retrieval (RAG for Code)
The most powerful compression technique is to not include the code at all, but instead include retrieval pointers. You embed code chunks into vectors, store them in a vector database, and at prompt time, retrieve only the chunks most relevant to the current task.
Example: Retrieval-augmented context
# context_compression/retriever.py
import chromadb
from sentence_transformers import SentenceTransformer
class CodeRetriever:
def __init__(self, collection_name="code_chunks"):
self.client = chromadb.PersistentClient(path="./chroma_db")
self.collection = self.client.get_or_create_collection(collection_name)
self.encoder = SentenceTransformer("all-MiniLM-L6-v2")
def add_chunk(self, chunk_id: str, code: str, metadata: dict):
embedding = self.encoder.encode(code).tolist()
self.collection.add(ids=[chunk_id], embeddings=[embedding], documents=[code], metadatas=[metadata])
def retrieve_relevant(self, query: str, top_k: int = 5) -> list[str]:
query_embedding = self.encoder.encode(query).tolist()
results = self.collection.query(query_embeddings=[query_embedding], n_results=top_k)
return results["documents"][0]
Why this works: Instead of sending 500,000 tokens of code, you send a 500-token query and retrieve 5 chunks of 100 tokens each. That's a 1000x reduction in context size, with higher signal because you're retrieving exactly what's relevant.
Trade-off: Retrieval can miss context that isn't semantically similar to the query. For example, if you need to know about a global variable that's referenced indirectly, retrieval might not find it. This is why compression is a layer, not a replacement.
5. Pillar 2: Durable Memory
Context compression reduces the size of what you put in the prompt. But it doesn't solve the problem of the agent forgetting what it did across turns. That's where durable memory comes in.
Durable memory is a persistent store — a database, a file, a key-value store — that records the agent's decisions, observations, and state. Unlike the context window, it doesn't get truncated. It survives across turns, across sessions, and even across model upgrades.
5.1 The Memory Hierarchy
Just like a CPU has registers, L1/L2 cache, and RAM, an AI agent should have a memory hierarchy:
| Level | Storage | Access Speed | Capacity | Purpose |
|---|---|---|---|---|
| L0 | Prompt | Instant | ~200K tokens | Active working set |
| L1 | Short-term store | Fast (API call) | ~1M tokens | Recent turns, task state |
| L2 | Vector DB | Medium (query) | Unlimited | Semantic retrieval |
| L3 | Object store | Slow (load) | Unlimited | Full file contents |
The rule: The agent should only ever put the minimum necessary state in L0 (the prompt). Everything else lives in L1-L3 and is pulled in on demand.
5.2 Implementing a Durable Memory Store
Here's a concrete implementation of a durable memory store using a simple key-value database (e.g., Redis or SQLite). The key insight is to store facts as structured records, not raw text.
# memory/durable_store.py
import json
import sqlite3
from typing import Any
class DurableMemory:
def __init__(self, db_path="agent_memory.db"):
self.conn = sqlite3.connect(db_path)
self.conn.execute("""
CREATE TABLE IF NOT EXISTS memory (
key TEXT PRIMARY KEY,
value TEXT,
timestamp REAL
)
""")
def set(self, key: str, value: Any):
"""Store a fact. Value is JSON-serialized."""
self.conn.execute(
"INSERT OR REPLACE INTO memory (key, value, timestamp) VALUES (?, ?, ?)",
(key, json.dumps(value), time.time())
)
self.conn.commit()
def get(self, key: str) -> Any | None:
"""Retrieve a fact, or None if not found."""
row = self.conn.execute("SELECT value FROM memory WHERE key = ?", (key,)).fetchone()
return json.loads(row[0]) if row else None
def search(self, query: str) -> list[tuple[str, Any]]:
"""Search facts by key prefix (simple heuristic)."""
rows = self.conn.execute(
"SELECT key, value FROM memory WHERE key LIKE ?", (f"{query}%",)
).fetchall()
return [(k, json.loads(v)) for k, v in rows]
What to store in durable memory:
- Task Intent: The original user goal, rewritten as a crisp, machine-readable objective.
- Decision Log: Every significant choice the agent made (e.g., "Used async/await instead of callbacks because...").
- File Map: A mapping of file paths to their purpose, updated as the agent explores.
- Constraint Set: Immutable rules like "Never modify generated files" or "Use the project's existing error handling pattern."
5.3 The "Memory Injection" Pattern
Durable memory is useless if the agent never reads it. The critical pattern is memory injection: at the start of each turn, the agent reads the relevant facts from durable memory and injects them into the prompt as a compact, structured preamble.
# memory/injection.py
from durable_store import DurableMemory
class MemoryInjector:
def __init__(self, memory: DurableMemory):
self.memory = memory
def build_preamble(self, task_id: str) -> str:
"""Construct a compact memory preamble for the prompt."""
task = self.memory.get(f"task:{task_id}")
decisions = self.memory.search(f"decision:{task_id}")
constraints = self.memory.search(f"constraint:{task_id}")
preamble = []
if task:
preamble.append(f"## TASK INTENT\n{task['description']}")
if decisions:
preamble.append("## DECISIONS")
for key, val in decisions[:10]: # Limit to last 10
preamble.append(f"- {val['reason']}: {val['choice']}")
if constraints:
preamble.append("## CONSTRAINTS")
for key, val in constraints[:5]:
preamble.append(f"- {val['rule']}")
return "\n".join(preamble)
Why this works: The preamble is short (maybe 500-1000 tokens), but it carries the highest-priority information. Because it's at the top of the prompt, it benefits from the model's positional bias toward the beginning. The agent starts every turn knowing exactly what it's supposed to do and what rules it must follow.
6. Pillar 3: Trace-Preserving Error Codes
The third pillar addresses the most insidious failure mode: when the agent makes a mistake, it can't diagnose it because the error trace is lost. In a long-running agent session, an error might occur in turn 10, but the agent doesn't see the fix until turn 50. By then, the original stack trace is long gone from the context.
Trace-preserving error codes are a way to compress an execution trace into a stable, human-readable, machine-parseable identifier that can be stored in durable memory and re-injected when needed.
6.1 The Problem with Raw Stack Traces
A typical Python stack trace is 50-100 lines. A Java stack trace can be 200+ lines. When you inject these into a prompt, they consume valuable context and often get truncated. Worse, the agent might interpret the error text incorrectly because it doesn't have the full context.
6.2 Designing a Trace-Preserving Error Code
We can design a structured error code that captures the essence of the trace without the verbosity. The code should include:
-
Error Class: The type of error (e.g.,
NullPointer,TypeMismatch). - File Path Hash: A short hash of the file where the error originated.
- Line Number: The exact line.
- Call Chain Hash: A hash of the function call chain (to identify the path).
- Semantic Description: A brief, LLM-friendly description of what went wrong.
Example: Generating a trace code
# trace/trace_code.py
import hashlib
import traceback
def generate_trace_code(exc: Exception) -> str:
"""Generate a compact, stable error code from a traceback."""
tb = traceback.extract_tb(exc.__traceback__)
if not tb:
return "ERR-UNKNOWN"
# Get the last frame (where the error occurred)
last_frame = tb[-1]
file_hash = hashlib.md5(last_frame.filename.encode()).hexdigest()[:8]
line_num = last_frame.lineno
# Hash the call chain (all function names)
call_chain = "->".join(frame.name for frame in tb)
chain_hash = hashlib.sha256(call_chain.encode()).hexdigest()[:12]
# Build a semantic description
error_class = type(exc).__name__
description = f"{error_class} in {last_frame.name} at line {line_num}"
# Combine into a stable code
code = f"ERR-{error_class}-{file_hash}-{line_num}-{chain_hash}"
return code
6.3 Using Trace Codes in the Agent Loop
The trace code isn't just a string; it's a pointer into durable memory. When an error occurs, the agent:
- Generates the trace code.
- Stores the full trace, the code, and the attempted fix in durable memory.
- Continues working.
Later, when the agent needs to revisit the error (e.g., because the same error occurs again), it retrieves the trace code from memory and injects a summary into the prompt:
## RECURRING ERROR
Code: ERR-ValueError-a1b2c3d4-42-9f8e7d6c5b4a
Previous Fix: Changed input validation to reject empty strings.
Status: Fixed but not verified.
Why this works: The trace code is only ~50 tokens, but it carries the full semantic weight of a 200-line stack trace. It also creates a stable identity for the error, so the agent can track whether it has seen this error before, what it tried, and whether the fix worked.
6.4 Trace Code as a Verification Mechanism
Beyond diagnosis, trace codes enable a powerful verification pattern: idempotent error handling. If the agent generates the same trace code twice, it knows it's dealing with the same underlying issue. It can then avoid repeating the same failed fix and instead try an alternative approach.
# trace/verification.py
class TraceRegistry:
def __init__(self, memory):
self.memory = memory
def register_attempt(self, trace_code: str, fix_description: str, success: bool):
"""Record an attempt to fix an error identified by trace_code."""
attempts = self.memory.get(f"trace:{trace_code}") or []
attempts.append({"fix": fix_description, "success": success})
self.memory.set(f"trace:{trace_code}", attempts)
def get_attempts(self, trace_code: str) -> list[dict]:
return self.memory.get(f"trace:{trace_code}") or []
def should_retry(self, trace_code: str) -> bool:
"""Return True if we should try a different fix (previous attempts failed)."""
attempts = self.get_attempts(trace_code)
return len(attempts) > 0 and not attempts[-1]["success"]
7. Putting It All Together: A Reference Architecture
Now let's see how these three pillars integrate into a single, coherent agent loop. The architecture is a layered memory system with a compression pipeline and a trace registry.
┌─────────────────────────────────────────────────────────┐
│ USER TASK │
└──────────────────────┬──────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────┐
│ AGENT ORCHESTRATOR │
│ - Parse task intent │
│ - Load durable memory (L1) │
│ - Build compressed context (L0) │
│ - Execute LLM call │
│ - Post-process: update memory, register traces │
└───────────────┬──────────────────────────┬──────────────┘
│ │
▼ ▼
┌─────────────────────────┐ ┌─────────────────────────┐
│ CONTEXT COMPRESSION │ │ DURABLE MEMORY │
│ - Summarizer (L3) │ │ - Task Intent │
│ - AST Filter (L2) │ │ - Decision Log │
│ - Retriever (L1) │ │ - Constraint Set │
└─────────────────────────┘ └─────────────────────────┘
│ │
└──────────┬───────────────┘
│
▼
┌─────────────────────────────────────────────────────────┐
│ TRACE REGISTRY │
│ - Generate trace code on error │
│ - Store fix attempts │
│ - Verify idempotency │
└─────────────────────────────────────────────────────────┘
7.1 The Agent Loop Pseudocode
Here's a simplified version of the agent loop, showing how the three pillars interact.
# agent_loop.py
from context_compression import CodeSummarizer, ASTFilter, CodeRetriever
from memory import DurableMemory, MemoryInjector
from trace import TraceRegistry
class CodingAgent:
def __init__(self, llm):
self.llm = llm
self.memory = DurableMemory()
self.injector = MemoryInjector(self.memory)
self.summarizer = CodeSummarizer()
self.ast_filter = ASTFilter()
self.retriever = CodeRetriever()
self.trace_registry = TraceRegistry(self.memory)
def run(self, task: str):
# 1. Store task intent in durable memory
task_id = hash(task)
self.memory.set(f"task:{task_id}", {"description": task})
# 2. Build the initial prompt with memory preamble
preamble = self.injector.build_preamble(task_id)
prompt = f"{preamble}\n\nTASK: {task}\n\n"
# 3. Execute the agent loop
for turn in range(MAX_TURNS):
# 4. Retrieve relevant code chunks (compression)
relevant_code = self.retriever.retrieve_relevant(prompt, top_k=5)
prompt += "\n\nRELEVANT CODE:\n" + "\n".join(relevant_code)
# 5. Call the LLM
response = self.llm.generate(prompt)
# 6. Process the response
if response.has_error:
# 7. Generate a trace code and register it
trace_code = generate_trace_code(response.error)
self.trace_registry.register_attempt(trace_code, response.fix, success=False)
# 8. Check if we should retry with a different fix
if self.trace_registry.should_retry(trace_code):
prompt += f"\n\nPREVIOUS FIX FAILED. TRY DIFFERENT APPROACH.\nError Code: {trace_code}"
else:
prompt += f"\n\nNEW ERROR: {trace_code}"
else:
# 9. Update durable memory with new decisions
self.memory.set(f"decision:{task_id}:{turn}", response.decision)
# 10. Check for task completion
if response.is_complete:
break
return response
7.2 Why This Architecture Works
This architecture attacks the context trap from three angles simultaneously:
- Compression reduces the input size, so the model operates well within its effective context window.
- Durable memory preserves critical state across turns, so the agent doesn't forget its goals or constraints.
- Trace codes provide a compact, stable identity for errors, enabling the agent to learn from past failures without re-reading massive stack traces.
Together, they transform the agent from a "stateless predictor" into a "stateful engineer" — one that can reason about a project over hours, not just seconds.
7.3 Practical Implementation Notes
- Start with durable memory. It's the easiest to implement and yields immediate improvements. Use a simple SQLite or Redis store.
- Add compression incrementally. Begin with AST filtering (fast, deterministic), then add summarization, then retrieval.
- Use trace codes only for recurring errors. Don't clutter the memory with one-off errors.
- Measure your effective context window. Run a simple benchmark: send a prompt with a known fact in the middle and see if the model recalls it. This tells you your ECW.
8. Frequently Asked Questions
Q1: Is a bigger context window always better?
No. As we discussed, attention degrades with context size. A 1M-token window is only useful if you have a way to prioritize which tokens matter. Without compression and memory, a bigger window often makes agents worse because they waste attention on irrelevant code. The goal should be to keep the prompt small and focused, not to maximize tokens.
Q2: Can I use these techniques with any LLM, or only specific models?
These techniques are model-agnostic. They work with any LLM that supports a standard chat or completion API. The compression and memory layers are external to the model — they're part of your agent's architecture. The only model-specific consideration is the effective context window, which you should measure empirically for your chosen model.
Q3: How do I handle the trade-off between compression and fidelity?
The key is to use a layered approach. Always keep the most recent and most relevant files in full (high fidelity). Compress only the background context (low fidelity). And use retrieval to pull in specific chunks when needed. This is similar to how a developer works: you have the file you're editing open in full, but you only look at other files when you need them. For more on building robust systems, see Tamiz's Insights.
This article was written for software engineers building production-grade AI coding agents. The patterns described here — context compression, durable memory, and trace-preserving error codes — are the building blocks of a reliable agentic system. Start small, measure your agent's actual performance, and iterate. The billion-token trap is real, but it's escapable.
Top comments (0)