Your AI assistant gives sharp, accurate answers at the start of a session. An hour later, it's hallucinating function names and confusing files.
This isn't a model problem. It's context rot.
What Is Context Rot?
Context rot is the gradual degradation of AI answer quality during a long session. The context window fills with irrelevant material from earlier turns, and the model can't pay attention to what actually matters.
Two related problems:
- Quality rot - accuracy drops as irrelevant content crowds out critical evidence
- Economic rot - token costs compound because every request re-sends the bloated history
Why It Happens
Transformers distribute attention across ALL input tokens. As a session grows:
| Content | Relevance | Token Cost |
|---|---|---|
| Early conversation turns | Usually stale | High |
| Imported dependency files | Partially relevant | Very high |
| Shell command output logs | Usually irrelevant | Medium-High |
| Duplicate file reads | Redundant | High |
| The actual bug/critical file | Critical | Low - crowded out |
The critical file gets pushed to the middle of the context window where attention is weakest. Research backs this up - the "Lost in the Middle" paper (Liu et al., 2023) showed LLMs perform significantly worse at retrieving information placed in the middle of long contexts.
The 4 Stages
Stage 1 - Fresh Session: Context mostly empty. Sharp, specific answers. Low cost.
Stage 2 - Accumulation: 30-40% context used. Model occasionally confuses variable names. You think it's a bad prompt.
Stage 3 - Congestion: 60-80% used. Model references wrong functions, suggests already-tried fixes. You start restarting sessions.
Stage 4 - Overflow: Context full. Tool truncates history or forces new session. Answers become generic. Hallucinations spike.
The Hidden Cost
Fresh start context: ~2,000 tokens = $0.006/request
Session turn 50: ~80,000 tokens = $0.24/request
Session turn 100: ~180,000 tokens = $0.54/request
Developer doing 60+ requests/day at stage 3:
Monthly cost: $350-800
With properly selected context (2,000 relevant tokens):
Monthly cost: $9-18
You're paying $350-800/month to send irrelevant code to Claude. The model isn't using it. And context rot is making it give worse answers.
The Fix: Information-Theoretic Selection
The right fix isn't "start a new session" (you lose all progress). It's mathematically ranking every fragment by relevance to the current query:
Bad: All files -> LLM call (irrelevant files included)
Bad: All files -> Compress -> LLM call (wrong starting set)
Good: All files -> Rank by query relevance -> Select under budget -> LLM call
The ranking uses three signals:
- BM25 relevance - lexical match between query and each fragment
- Shannon entropy scoring - high-entropy fragments (errors, anomalies) selected over boilerplate
- Dependency graph traversal - if the buggy function is selected, its callees are included
Budget allocation uses knapsack dynamic programming - the theoretically optimal solution.
Results
I built this into an open-source tool called Entroly. On real workloads:
| Budget | Token Reduction | Accuracy |
|---|---|---|
| 8K tokens | 99.1% | 100% (NeedleInAHaystack) |
| 32K tokens | 96.7% | 103% (LongBench HotpotQA) |
| Average | 87.0% | Maintained or improved |
103% accuracy on HotpotQA means the model actually gives better answers with selected context than full context - because it's no longer distracted.
The tool also includes a local hallucination detector (WITNESS) that achieves 0.844 AUROC on HaluEval-QA for $0 in ~3ms - statistically ties GPT-4o-mini as a judge.
pip install entroly
entroly simulate # see savings estimate, no API key needed
Works with Claude Code, Cursor, Aider, Codex, and 35+ others. Apache-2.0. Local-first.
Has anyone else noticed context rot in their sessions? Curious how you deal with it.
Top comments (0)