A language model has access to exactly one thing: the tokens sitting in its context window on this call. It does not remember your last session, it cannot see your database, and it has no idea what a tool returned unless that text is in the prompt right now. Everything the model appears to know is there because some part of your system decided to put it there.
Context engineering is the discipline of making that decision well, on every request, under a hard token budget. It has quietly become the main thing separating an AI demo that impresses from a system that holds up with real users over months.
Why Context Beats Prompt Wording
Model quality is rarely the bottleneck anymore. Frontier models are extremely capable when the right information is in front of them, and they fail in predictable ways when it is not.
A support agent that gives a wrong refund answer usually had the correct policy sitting in its knowledge base and simply never retrieved it into context. A coding assistant that edits the wrong function usually was never shown the file that mattered. Those are not reasoning failures, they are context failures, and no amount of rewriting the instruction fixes a window that is missing the one document the answer depended on.
Prompt engineering is a real skill, and it is a subset of this. The prompt is static text you write once and reuse. The context is assembled at runtime out of instructions, conversation history, retrieved knowledge, long term memory, tool definitions and tool results, and the right mix changes with every single request.
The Token Budget as a Design Constraint
The instinct when answers go wrong is to put more into the window. Larger windows make that easy and it is usually the wrong move.
Even models advertising very large windows degrade as the window fills. Relevant tokens get diluted by irrelevant ones and attention spreads thinner across the whole. You also pay for every token in money and latency, on every call, forever.
So the useful framing is a budget rather than a storage locker. System rules get an allotment. Live conversation gets an allotment. Retrieved knowledge, memory and tool output get theirs. Anything that does not fit its allotment gets summarized or dropped before the call goes out. The number worth optimizing is relevance density, the fraction of tokens in the window that actually bear on the current request. A window padded with an entire manual to answer one shipping question has low relevance density and will produce a worse answer than a window holding just the shipping section, even though the padded one technically contains the answer too.
Where Context Fails in Agents
Agent systems make all of this harder because the window is a moving target. Every tool call adds output. Every step adds history. A loop that runs twenty steps has a very different window at step twenty than at step one.
The failures this produces are the frustrating kind, because they are intermittent. The same question works on Monday and fails on Friday because the retrieval index grew and a near duplicate document now outranks the right one. A long conversation quietly pushes the system instructions out of the window and the agent starts ignoring rules it followed an hour ago. Nothing errors. The output just gets worse.
An explicit context pipeline, with stated rules for what gets included and a fixed budget per part of the window, turns those silent failures into something you can measure, test and debug.
Memory as the Context Layer
Retrieval answers the question "what documents relate to this query". Memory answers a different one: "what do we already know about this user, this project, this task, that should shape the answer".
Treating memory as just another retrieval index is where a lot of systems go wrong. Memory needs scoping so one team's facts do not bleed into another team's answers, it needs a way to supersede stale entries rather than accumulate contradictions, and it needs to earn its place in the budget on each call like everything else. Done well, it is what stops your system from making the user re-explain their situation every session, which is the single most common reason people quietly stop using an assistant.
The Takeaway
Context engineering is not prompt tuning with a new name. It is the runtime system that decides what the model sees, and its objective is simple to state and hard to hit: the highest possible fraction of tokens that matter, inside a fixed budget, fast enough to run in production.
If you are debugging bad outputs, start by asking what was actually in the window on the call that failed. Most of the time the answer explains everything.
The full breakdown, including the four context strategies, the token budget and the common failure modes, is here: https://www.adaptiverecall.com/context-engineering/
Top comments (0)