You wrote a great prompt. It worked beautifully in the playground — one question, one clean answer. Then you wired the same model into an agent that runs twenty steps, calls six tools, and reads back their output, and somewhere around step twelve it started forgetting the goal, calling the wrong tool, or confidently acting on something it misread three steps ago. The prompt didn't get worse. The context did.
This is the gap that context engineering fills. Prompt engineering is about writing one good instruction. Context engineering is about managing the entire set of tokens a model sees at inference time — across a long, multi-step run — so the signal stays high and the model keeps making good decisions. Anthropic frames it as the natural progression of prompt engineering, and if you're building anything agentic in 2026, it's the discipline that separates a demo from a system.
What context engineering actually is
Start with a precise definition. Context is the full set of tokens you include when you sample from a large language model. Not just your prompt — the system instructions, the tool definitions, the examples, the running message history, the retrieved documents, the tool results fed back in. Everything in the window.
Context engineering is the set of strategies for curating and maintaining the optimal set of those tokens during inference. The goal, in one line: find the smallest set of high-signal tokens that reliably produces the outcome you want.
The reason this is a distinct discipline from prompt engineering is the shape of the problem. A prompt is something you write once and it stays put. Context in an agent is dynamic — it grows on every turn as the model reads files, calls tools, and accumulates history. You're not authoring a static string anymore; you're managing a budget that fills up on its own, and deciding continuously what earns a place in it and what gets thrown out. That's an engineering problem, and it's the foundation the whole prompt-to-production journey builds on.
Why "just add more context" is the wrong instinct
The intuitive move, when an agent makes a mistake, is to give it more: more instructions, more examples, more history, more retrieved documents. Sometimes that helps. Very often it makes things worse, and here's why.
A model's effective attention is a finite resource. Every token you add competes with every other token for the model's limited ability to attend to what matters. Past a certain point, adding context doesn't add capability — it dilutes it. The relevant fact is now buried among ten irrelevant ones, and the model attends to the wrong thing.
This shows up empirically. The "lost in the middle" effect — documented by Liu et al. — found that models attend most reliably to information at the start and end of a long context, and least reliably to what's stuck in the middle. As context grows, retrieval of any single fact inside it gets less reliable, a degradation sometimes called context rot. A 200K-token window does not mean you should put 200K tokens in it. Capacity is not the same as attention.
So the mental model to internalize: context is a budget, not a bucket. You're not trying to fill it. You're trying to spend it on the highest-signal tokens available and refuse everything else. Every technique below is a way to enforce that discipline.
The four things competing for your window
Before the techniques, know your spenders. In a running agent, four categories of tokens fight for the same finite budget:
- The system prompt — your instructions, role, constraints. Usually small, high-value, and stable.
- Tool definitions — the schemas describing every tool the agent can call. These are sneakily expensive: each tool's description sits in context on every turn, whether or not it's used.
- Message history — the accumulating transcript of the conversation and the agent's own steps. This is the one that grows without bound and quietly eats the window.
- Retrieved / external data — documents, search results, file contents, database rows pulled in to ground the model.
The overall guidance from Anthropic is worth memorizing: keep each of these informative yet tight. Not empty — an under-specified system prompt or a missing tool leaves the model guessing. But not bloated either. The art is the calibration, and it's different for each category. Let's work through the techniques that manage them.
Technique 1: Compaction — summarize the history before it drowns you
Message history is the runaway spender. A long agent run accumulates hundreds of turns of "called tool, got 4KB of JSON back, reasoned about it, called the next tool." Most of those raw tool outputs are dead weight three steps later — you needed the conclusion, not the 4KB.
Compaction is the fix: periodically replace a chunk of verbose history with a tight summary that preserves the decisions, the key facts, and the current state, while dropping the raw noise. When the agent has finished investigating something, you don't need the full transcript of the investigation in context — you need "here's what I found and what it means for the task."
Practical rules:
- Compact at natural boundaries, not mid-reasoning — after a sub-task completes, when a phase ends.
- Preserve the load-bearing details: open questions, decisions made, constraints discovered, current state. Drop the intermediate chatter and the raw dumps.
- Keep the goal pinned. The single most common long-run failure is the agent losing the plot on the original objective. The goal should survive every compaction.
Done well, compaction is what lets an agent run for hundreds of steps without either overflowing its window or forgetting why it started.
Technique 2: External memory — let the agent write things down
The window is not the only place to keep information. The most effective long-running agents treat the context window as working memory and offload durable state to external memory — a file, a scratchpad, a structured store the agent reads from and writes to deliberately.
Instead of carrying every fact in-context forever, the agent writes a note ("the auth module uses JWT with 15-minute expiry; the bug is in the refresh path") to a persistent store, and pulls it back only when relevant. The context window stays lean; the knowledge doesn't get lost. This is exactly how a human engineer works — you don't hold the entire codebase in your head, you keep notes and open the file when you need it.
This pattern — persistent, deliberately-managed memory outside the window — is the core of building agents that hold state over long horizons, and it's what turns a stateless model into something that can work a problem across a session without drowning.
Technique 3: Sub-agents — isolate context so the main thread stays clean
Here's a structural technique that most people never turn on. When a sub-task is going to burn a lot of tokens — "find every call site of this function across the repo," "research these five libraries" — don't do it in the main agent's context. Delegate it to a sub-agent: a separate agent instance with its own isolated window that does the noisy work and reports back a clean result.
The win is context hygiene. The thousands of tokens of file contents and search output that the investigation churns through stay in the sub-agent's context and die with it. The main agent gets back a two-paragraph summary, and its own window stays focused on the actual task instead of silting up with intermediate noise. As a bonus, independent sub-tasks run in parallel instead of serially.
The rule of thumb: delegate when work is independent, parallelizable, or context-heavy; keep it inline when it's sequential and cheap. Knowing when to fan out to sub-agents and how to orchestrate them without stepping on each other is a central skill in building AI agents and automation, and it's one of the highest-leverage context moves you have.
Technique 4: Just-in-time retrieval — pull data when needed, not upfront
There are two ways to get external data into an agent. The naive way is to preload: at the start, dump everything the agent might conceivably need into context — the whole document, all the schemas, every config file. The problem is obvious once you see it: you're spending your budget on maybes, and most of it goes unused while crowding out what matters.
The better pattern is just-in-time retrieval: give the agent the ability to fetch data (a search tool, a file reader, a database query) and let it pull exactly what it needs, exactly when it needs it. Instead of "here are all 40 files," it's "here's a tool to read a file — go get the one you need." The agent loads the relevant chunk into context at the moment of use, acts on it, and (with compaction) lets it fall away afterward.
This mirrors how retrieval-augmented systems already work, and getting the retrieval layer right — what to fetch, how to rank it, how much to bring back — is where advanced LLM integration earns its keep. Preloading feels safer; just-in-time is what actually scales.
Technique 5: Tool curation — the failure mode hiding in plain sight
Remember that tool definitions sit in context on every turn. That makes a bloated tool set a double tax: it burns tokens continuously, and it degrades decisions. Anthropic calls out one of the most common failure modes directly — tool sets that cover too much functionality or create ambiguous, overlapping choices about which tool to use.
The tell is a sharp one: if a human engineer can't say for certain which tool should be used in a given situation, an AI agent can't either. Fifteen tools with fuzzy, overlapping responsibilities will produce worse behavior than six sharp, non-overlapping ones — and cost more tokens doing it.
So curate the toolbox like you'd curate an API:
- Fewer, sharper tools. Each with a clear, distinct job and an unambiguous "use this when…"
- No overlap. Two tools that could both plausibly handle the same request is a decision point where the agent will sometimes pick wrong.
- Prune ruthlessly. A tool that's rarely the right choice is paying rent in your context on every single turn. Cut it.
Tool curation is the least glamorous technique here and often the highest-ROI. It's pure subtraction, and subtraction is exactly what context engineering rewards.
How you know it's working: measure it
Every technique above is a change to your context, and changes to context are exactly the kind of thing that feels better while being worse — or vice versa. If your only signal is "I ran a few queries and it seemed fine," you're tuning blind, and you'll ship a regression the day you compact one turn too aggressively and the agent starts forgetting a constraint.
The fix is the same as anywhere in production ML: an evaluation set. Assemble representative tasks with known good outcomes, and re-run them every time you change how context is managed. Then you can say "compaction at phase boundaries held task success at 0.9 while cutting average tokens 40%" instead of "I think it's better now." Treating evaluation as first-class rather than an afterthought is what turns context engineering from a craft into engineering — a number that moves when you change something, not a vibe.
Putting it together
Context engineering isn't a framework you install; it's a posture you adopt toward the model's window. The whole discipline collapses to one principle applied relentlessly: spend the finite budget on the smallest set of high-signal tokens that does the job.
In practice, for a real agent, that means:
- Compact the history at natural boundaries so a long run doesn't drown in its own transcript.
- Offload durable state to external memory instead of carrying it forever.
- Delegate noisy, independent work to sub-agents so the main window stays clean.
- Retrieve just-in-time instead of preloading everything you might need.
- Curate the tools hard — fewer, sharper, no overlap.
- Measure with evals so every change is verified, not hoped.
None of these are exotic, and that's the point. The model is already capable. What makes the capability hold up over a twenty-step run isn't a cleverer prompt — it's disciplined management of everything the model reads along the way.
Conclusion
Prompt engineering taught us to write one good instruction. Context engineering is what you need the moment that instruction has to survive a long, tool-using, self-accumulating agent run — which is to say, the moment you build anything real. The failure you saw at step twelve was never the model getting dumber. It was the context getting noisier, and no one curating it.
Adopt the budget mindset, apply the five techniques, and put an eval set behind every change. Do that, and the same model that fell apart at step twelve will run to step fifty and still know exactly what it's doing — because you engineered what it was looking at the whole way.
The courses linked throughout are part of Cursuri-AI.ro, an AI-learning platform with hands-on, current tracks on context engineering, AI agents, LLM integration, and evaluating AI systems in production.
Sources & further reading:
- Anthropic — Effective context engineering for AI agents (definition, the four context components, tool-set failure modes, compaction and memory)
- Anthropic — Effective harnesses for long-running agents
- Liu et al. — Lost in the Middle: How Language Models Use Long Contexts
This article is educational content. Model behavior, context limits, and tooling evolve quickly; validate approaches against your own workloads and current official documentation.
Top comments (0)