Context Engineering: Feeding AI Agents the Right Data

Last month I debugged a support agent for a client that kept quoting a refund policy the company had retired eight months earlier. The prompt was fine. The model was fine. The problem was the retrieval step: it stuffed twelve old policy documents into the context window, and the outdated one sat right in the middle. The agent was not stupid. It was badly fed. That is the difference between prompt engineering and context engineering, and it is where most agent projects break.

Prompt engineering gets the words right. Context engineering gets the information right. It is the discipline of deciding what enters the context window: the system prompt, the retrieved documents, the tool results, and the memory from past turns. Get it wrong and your agent hallucinates with confidence, burns tokens, and fails in ways that look like model problems but are not.

What Context Engineering Actually Is

Every agent call assembles a payload before the model sees anything: a system prompt, conversation history, retrieved documents, tool results, and structured state like the user's account record. Context engineering is the work of curating each of those inputs so the model sees exactly what it needs and nothing it does not.

I think of it as mise en place for agents: someone portions and arranges ingredients before service, and your agent runtime should do the same. Prompt wording still matters, but it is the garnish, not the meal. Context engineering is the layer underneath: which facts, how many, in what order, and at what level of detail.

Anthropic's engineering team laid out a solid framing in their context engineering write-up, and it matches what I see in client work: agents fail from context problems far more than from prompt problems.

The Context Budget and the Lost-in-the-Middle Problem

Bigger context windows created a bad habit. When your model accepts 200,000 tokens, it is tempting to throw everything in and let the model sort it out. Do not do this. Two things go wrong.

First, cost and latency scale with input tokens. A 100,000-token input on a frontier model can run 25 to 40 cents per call plus several seconds of latency. Multiply by a few thousand agent runs a day and you have a real line item, and input tokens are where bloated agents bleed money. OpenAI's token pricing page makes the per-million-token math easy to check against your own payload sizes, and our guide on LLM token cost optimization breaks down how runaway prompt inflation compounds across multi-turn sessions.

Second, and worse, models do not attend to long contexts evenly. The Lost in the Middle paper from Liu et al. showed that recall for facts buried in the middle of a long context drops sharply compared to facts near the beginning or end. That was 2023, and the U-shaped attention curve has not disappeared. In my own evals, a fact at position 60% of a 90,000-token prompt still gets missed more often than the same fact in the first or last 10%.

So set a budget. For a support agent, I cap retrieval at 3,000 to 5,000 tokens, conversation history at the last six to ten turns, and I put the single most important fact (the current policy, the active order, the user's account status) near the end of the context, right before the user's message. That placement alone fixed the refund policy bug from the intro.

Retrieval: Select Less, Chunk Smarter

Most RAG failures I audit are selection failures. The vector store returns the top ten chunks because someone set k=10 in a tutorial and never revisited it. Half those chunks are near-duplicates, two are stale, and one is from the wrong product line entirely.

My default recipe for a production agent:

1. Retrieve wide, then rerank. Pull 20 to 40 candidates from the vector store, run them through a cross-encoder reranker, and keep the top three to five.
2. Deduplicate by source. If two chunks come from the same document section, keep the better one.
3. Filter by freshness. Stamp every chunk with an updated date at ingest time and hard-exclude anything past its TTL for policy-like content.
4. Chunk by structure, not by fixed size. Split on headings and semantic boundaries so a chunk is a complete thought. A 400-token chunk that ends mid-sentence confuses the model more than a 900-token chunk that covers one full topic.

If you are building the retrieval layer from scratch, the RAG system build guide walks through the pipeline. The context engineering twist is what happens after retrieval: you are not returning chunks to the user, you are composing them into a ranked, deduplicated, budget-capped section of the prompt. Label it clearly; I wrap retrieved material in <retrieved_context> markers so the model can tell vetted facts from conversation noise.

Tool Results: Return Less, Summarize More

Tools are the biggest source of context bloat. An agent calls your order lookup API and the endpoint returns a 4,000-token JSON blob with every field the warehouse tracks. The agent needed two: status and delivery date.

Shape tool results at the boundary, not in the prompt, with a thin adapter between the raw API and the agent:

def lookup_order_for_agent(order_id: str) -> str:
    raw = orders_api.get(order_id)  # full internal payload

    if raw["status"] == "exception":
        # Small result: return detail, the agent needs it.
        return (
            f"Order {order_id}: EXCEPTION - {raw['exception_reason']}. "
            f"Last scan: {raw['last_scan_location']} at {raw['last_scan_at']}."
        )

    # Normal path: return only what answers the question.
    return (
        f"Order {order_id}: {raw['status']}, "
        f"estimated delivery {raw['eta']}, "
        f"carrier {raw['carrier']}."
    )

Two rules here. First, the tool returns a short string instead of raw JSON, cutting tokens and parsing errors, since models read prose more reliably than nested JSON. Second, verbosity is conditional: happy paths get one line, failure states get detail, because failures are where the agent needs context to act.

Apply the same idea to search tools, database queries, and web fetches. A web fetch tool should return a 500-word extraction, not the full page HTML. Cap every tool result at about 1,000 tokens and note when truncation happened so the model does not assume it saw everything.

Memory Tiers: Working, Episodic, and Long-Term

Memory is context that persists, and it needs the same discipline as retrieval. I use three tiers.

Working memory is the current conversation. Keep the last six to ten turns verbatim. Older turns get summarized into a running digest of two to four sentences, refreshed every few turns. This keeps tasks coherent without dragging twenty stale messages into every call.

Episodic memory covers past sessions with the same user. Store one compact record per session: what the user wanted, what happened, what was resolved. Retrieve the two or three most relevant episodes at session start, not all of them. A customer who contacts you monthly does not need their entire history in the window; they need the open issue and the last outcome.

Long-term memory holds stable facts: the user's plan tier, their preferred contact channel, their company's billing contact. These belong in a structured profile, injected as a few lines near the top of the context, and updated only by explicit writes, never by the agent silently appending to a growing string. Silent memory growth is how agents end up with 8,000 tokens of contradictory user facts after three months.

Standardized protocols like the Model Context Protocol for small business help keep agent inputs consistent across vendors, and the hallucination detection guide covers how to catch the wrong answers that slip through when context goes stale. Add memory tiers one at a time. I have watched teams bolt on full memory systems before the agent could finish a two-step task; debugging became impossible because nobody could tell which tier supplied the bad fact.

Debugging Agent Failures Caused by Bad Context

When an agent gives a wrong answer, resist rewriting the prompt. First log the full context payload for every run: in my stack, one JSON file with the system prompt, each retrieved chunk and its score and date, every tool call with raw and shaped results, and the memory blocks injected. Ninety percent of the time the failure is visible in that log within five minutes.

The failure patterns repeat. Stale context: a document or memory record that should have been expired. Collision: two retrieved chunks that contradict each other, where the model splits the difference into a confident wrong answer. Drowning: the right fact is present but buried under 30,000 tokens of noise, lost in the middle. Absence: the fact was never retrieved, and the model filled the gap from training data.

Each pattern has a different fix, which is why "the model hallucinated" is a useless diagnosis. Stale context needs TTLs. Collision needs deduplication and source ranking. Drowning needs a tighter budget and better placement. Absence needs retrieval tuning or a clarifying question back to the user.

To catch these before users do, wire context logging into your eval loop. Anthropic's agent evaluation guidance is a solid reference for building that harness. In runaway scenarios, context bloat can spiral into thousands of dollars in wasted API spend, as we documented in our post-mortem on unsupervised AI agent runaway loops. My addition: write one eval case per failure pattern seen in production, with the exact context payload that caused it, so a retrieval change cannot reintroduce an old bug.

When Context Engineering Is Overkill

Not every agent needs this machinery. If your agent handles a single narrow task, like classifying inbound emails into four buckets, a fixed system prompt plus the email body is the whole context, and there is nothing to engineer. If your retrieval corpus is fifty documents, you can skip reranking and just return all matches under the token cap. If the agent runs five times a day, a sloppy 20,000-token context costs you pennies and nobody cares about the extra second of latency.

Build the tiers when volume, task length, or failure cost justifies it. A support agent handling hundreds of conversations a day earns its context pipeline in the first week through token savings alone. An agent that touches refunds, account changes, or anything a lawyer might read later earns it through error prevention.

What Most Guides Get Wrong

Most guides treat context engineering as a retrieval tuning exercise, as if the whole problem is picking a better embedding model or bumping the reranker. In practice, retrieval quality is rarely the bottleneck; I have audited agents with flawless retrieval that still failed because nobody set a TTL on the documents being retrieved. Another common claim is that bigger context windows make this discipline obsolete, which is backwards: larger windows make drowning cheaper to cause and harder to notice. Guides also love to prescribe fixed chunk sizes like 512 tokens, ignoring that a chunk ending mid-thought is worse than a chunk twice as long that stays coherent. Many treat tool outputs as untouchable, dumping raw API responses into the window and then blaming the model for fixating on an irrelevant field. And almost none of them talk about placement, even though moving one critical fact from position 50% to position 95% has fixed more production bugs for me than any reranking upgrade ever has. The unglamorous truth is that freshness rules, output shaping, and budget caps do more than any model swap.

Conclusion

Models are good enough. The agents that fail in production are fed badly: stale documents, unshaped tool dumps, memory that grows without rules, and critical facts buried in the middle of bloated prompts. Context engineering is the fix, and it is unglamorous plumbing work: budgets, reranking, adapters, TTLs, and logs.

Start with three changes. Cap and label your retrieved context. Make every tool return less, with detail only on failure paths. Log the full payload for every run so failures become diagnosable in minutes. Do those three and your agent gets cheaper, faster, and right more often, without touching the model at all.