What happened: As context windows grew to hundreds of thousands of tokens, teams stopped obsessing over the perfect single prompt and started treating everything a model sees — system instructions, retrieved documents, tool results, conversation history, memory — as one deliberate budget to design.
Why it matters: A bigger window makes it tempting to dump everything in "just in case." That backfires: irrelevant context dilutes attention and measurably hurts accuracy, even when the answer is technically present in the window. More space isn't the same as better use of it.
What's actually in the context budget
- System prompt: role, constraints, output format — set once, reused every call
- Retrieved documents: the RAG chunks pulled for this specific question — relevance beats volume
- Tool results: what a function call or search actually returned — often the noisiest part of the window
- Conversation history: prior turns — prune aggressively in long agent loops, don't just append forever
- Memory: facts carried across sessions — curated, not a dump of every past interaction
Common mistakes
Pasting full documents instead of the relevant chunk. Letting tool output accumulate unpruned across a long agent loop until it crowds out the actual task. Assuming the model "sees" everything with equal weight — position and relevance both matter, not just presence.
Who should care
Anyone building RAG or agents past the demo stage. If your system prompt keeps growing and accuracy keeps dropping, the fix is usually curation, not a bigger model.
Originally published on ayraix.com, practical AI for enterprise builders.
Top comments (0)